Maintenance jobs and job runner

From: Derrick Stolee <[email protected]>

The 'git run-job' command will be used to execute a short-lived set of maintenance activities by a background job manager. The intention is to perform small batches of work that reduce the foreground time taken by repository maintenance such as 'git gc --auto'.

This change does the absolute minimum to create the builtin and show the usage output.

Provide an explicit warning that this command is experimental. The set of jobs may change, and each job could alter its behavior in future versions.

RFC QUESTION: This builtin is based on the background maintenance in Scalar. Specifically, this builtin is based on the "scalar run <job>" command [1] [2]. My default thought was to make this a "git run <job>" command to maximize similarity. However, it seems like "git run" is too generic. Or, am I being overly verbose for no reason?

[1] https://github.com/microsoft/scalar/blob/master/docs/advanced.md#run-maintenance-in-the-foreground
[2] https://github.com/microsoft/scalar/blob/master/Scalar/CommandLine/RunVerb.cs

Signed-off-by Derrick Stolee <[email protected]>

---
  .gitignore                    |  1 +
  Documentation/git-run-job.txt | 36 +++++++++++++++++++++++++++++++++++
  Makefile                      |  1 +
  builtin.h                     |  1 +
  builtin/run-job.c             | 28 +++++++++++++++++++++++++++
  command-list.txt              |  1 +
  git.c                         |  1 +
  t/t7900-run-job.sh            | 15 +++++++++++++++
  8 files changed, 84 insertions(+)

create mode 100644 Documentation/git-run-job.txt create mode 100644 builtin/run-job.c create mode 100755 t/t7900-run-job.sh

.gitignore +1 -0

index 188bd1c3de1..5dea9d3b96b 100644
@@ -144,6 +144,7 @@
  /git-rev-parse
  /git-revert
  /git-rm
+/git-run-job
  /git-send-email
  /git-send-pack
  /git-serve

Documentation/git-run-job.txt new file +36 -0

new file mode 100644
index 00000000000..0627b3ed259
@@ -0,0 +1,36 @@
+git-run-job(1)
+==============
+
+NAME
+----
+git-run-job - Run a maintenance job. Intended for background operation.
+
+
+SYNOPSIS
+--------
+[verse]
+'git run-job <job-name>'
+
+
+DESCRIPTION
+-----------
+
+Run a maintenance job on the current repository. This is available as a
+command for a few reasons. First, the background job feature can launch
+these commands on a schedule and each process will completely clear its
+memory when complete. Second, an expert user could create their own job
+schedule by running these jobs themselves.
+
+THIS COMMAND IS EXPERIMENTAL. THE SET OF AVAILABLE JOBS OR THEIR EXACT
+BEHAVIOR MAY BE ALTERED IN THE FUTURE.
+
+
+JOBS
+----
+
+TBD
+
+
+GIT
+---
+Part of the linkgit:git[1] suite

Makefile +1 -0

index ef1ff2228f0..f5f9c4d9e94 100644
@@ -1125,6 +1125,7 @@ BUILTIN_OBJS += builtin/rev-list.o
  BUILTIN_OBJS += builtin/rev-parse.o
  BUILTIN_OBJS += builtin/revert.o
  BUILTIN_OBJS += builtin/rm.o
+BUILTIN_OBJS += builtin/run-job.o
  BUILTIN_OBJS += builtin/send-pack.o
  BUILTIN_OBJS += builtin/shortlog.o
  BUILTIN_OBJS += builtin/show-branch.o

builtin.h +1 -0

index 2b25a80cde3..3e0ddaaf67f 100644
@@ -220,6 +220,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix);
  int cmd_rev_parse(int argc, const char **argv, const char *prefix);
  int cmd_revert(int argc, const char **argv, const char *prefix);
  int cmd_rm(int argc, const char **argv, const char *prefix);
+int cmd_run_job(int argc, const char **argv, const char *prefix);
  int cmd_send_pack(int argc, const char **argv, const char *prefix);
  int cmd_shortlog(int argc, const char **argv, const char *prefix);
  int cmd_show(int argc, const char **argv, const char *prefix);

builtin/run-job.c new file +28 -0

new file mode 100644
index 00000000000..2c78d053aa4
@@ -0,0 +1,28 @@
+#include "builtin.h"
+#include "config.h"
+#include "parse-options.h"
+
+static char const * const builtin_run_job_usage[] = {
+	N_("git run-job"),
+	NULL
+};
+
+int cmd_run_job(int argc, const char **argv, const char *prefix)
+{
+	static struct option builtin_run_job_options[] = {
+		OPT_END(),
+	};
+
+	if (argc == 2 && !strcmp(argv[1], "-h"))
+		usage_with_options(builtin_run_job_usage,
+				   builtin_run_job_options);
+
+	git_config(git_default_config, NULL);
+	argc = parse_options(argc, argv, prefix,
+			     builtin_run_job_options,
+			     builtin_run_job_usage,
+			     PARSE_OPT_KEEP_UNKNOWN);
+
+	usage_with_options(builtin_run_job_usage,
+			   builtin_run_job_options);
+}

command-list.txt +1 -0

index 20878946558..1cd2b415e46 100644
@@ -156,6 +156,7 @@ git-revert                              mainporcelain
  git-rev-list                            plumbinginterrogators
  git-rev-parse                           plumbinginterrogators
  git-rm                                  mainporcelain           worktree
+git-run-job                             plumbingmanipulators
  git-send-email                          foreignscminterface             complete
  git-send-pack                           synchingrepositories
  git-shell                               synchelpers

git.c +1 -0

index b07198fe036..db5a43c8687 100644
@@ -566,6 +566,7 @@ static struct cmd_struct commands[] = {
  	{ "rev-parse", cmd_rev_parse, NO_PARSEOPT },
  	{ "revert", cmd_revert, RUN_SETUP | NEED_WORK_TREE },
  	{ "rm", cmd_rm, RUN_SETUP },
+	{ "run-job", cmd_run_job, RUN_SETUP },
  	{ "send-pack", cmd_send_pack, RUN_SETUP },
  	{ "shortlog", cmd_shortlog, RUN_SETUP_GENTLY | USE_PAGER },
  	{ "show", cmd_show, RUN_SETUP },

t/t7900-run-job.sh new file +15 -0

new file mode 100755
index 00000000000..1eac80b7ed3
@@ -0,0 +1,15 @@
+#!/bin/sh
+
+test_description='git run-job
+
+Testing the background jobs, in the foreground
+'
+
+. ./test-lib.sh
+
+test_expect_success 'help text' '
+	test_must_fail git run-job -h 2>err &&
+	test_i18ngrep "usage: git run-job" err
+'
+
+test_done

Hi Stolee

Quoted 14 lines · Derrick Stolee via GitGitGadget

From: Derrick Stolee <[email protected]>

The 'git run-job' command will be used to execute a short-lived set of maintenance activities by a background job manager. The intention is to perform small batches of work that reduce the foreground time taken by repository maintenance such as 'git gc --auto'.

This change does the absolute minimum to create the builtin and show the usage output.

Provide an explicit warning that this command is experimental. The set of jobs may change, and each job could alter its behavior in future versions.

Quoted 5 lines

RFC QUESTION: This builtin is based on the background maintenance in Scalar. Specifically, this builtin is based on the "scalar run <job>" command [1] [2]. My default thought was to make this a "git run <job>" command to maximize similarity. However, it seems like "git run" is too generic. Or, am I being overly verbose for no reason?

Having read through this series I wondered if we wanted a single git command such as 'git maintenance' (suggestions of better names welcome) and then 'git run-job' could become 'git maintenance run', 'git job-runner' would become another subcommand (run-jobs or schedule-jobs?) and the 'git please-run-maintenance-on-this-repo' you mentioned in you email to Junio could become 'git maintenance init' (or maybe setup)

Quoted 23 lines · Phillip Wood

Hi Stolee

Quoted 14 lines · Derrick Stolee via GitGitGadget

From: Derrick Stolee <[email protected]>

The 'git run-job' command will be used to execute a short-lived set of maintenance activities by a background job manager. The intention is to perform small batches of work that reduce the foreground time taken by repository maintenance such as 'git gc --auto'.

This change does the absolute minimum to create the builtin and show the usage output.

Provide an explicit warning that this command is experimental. The set of jobs may change, and each job could alter its behavior in future versions.

Quoted 5 lines

RFC QUESTION: This builtin is based on the background maintenance in Scalar. Specifically, this builtin is based on the "scalar run <job>" command [1] [2]. My default thought was to make this a "git run <job>" command to maximize similarity. However, it seems like "git run" is too generic. Or, am I being overly verbose for no reason?

Quoted 7 lines

Having read through this series I wondered if we wanted a single git command such as 'git maintenance' (suggestions of better names welcome) and then 'git run-job' could become 'git maintenance run', 'git job-runner' would become another subcommand (run-jobs or schedule-jobs?) and the 'git please-run-maintenance-on-this-repo' you mentioned in you email to Junio could become 'git maintenance init' (or maybe setup)

I had a very similar impression. In addition to what you already said, a few more were:

- Why the existing "git repack" isn't such "maintenance" command? IOW why do we even need [01/15]? After all, "repack" may have started its life as a tool to reorganize the PACKFILES, but it is no longer limited to 'git/objects/pack/*.pack' files with its knowledge about the loose object files and the "--prune" option. Consolidating pieces of information spread across multiple .idx files, reachability bitmaps and commit graph files, into a newer and more performant forms can just be part of "packing the pieces of information in a repository for optimum performance", which is a better way to understand why "repack" has a word 'pack' in its name.

- Many of the "maintenance" operations this series proposes do make sense, just like other "maintenance" operations we already have in "repack", "prune", "prune-packed" etc., which are welcome additions.

- Like the individual steps that appear in e.g. "repack", however, some of the individual steps in this series can be triggered by calling underlying tools directly, allowing scripted maintenance commands that suit individual needs better than the canned invocation of "run-job", but I didn't get the impression that the series strives to make sure that all knobs of these individual steps are available to scripters who want to deviate from what "run-job" prescribes. If it is not doing so, we probably should.

- Again, I do not think we want a reimplementation of cron, at or inetd that is not specific to "git" at all.

Of course, not every platform has "cron" but that just means we need a cross-platform way to launch Git processes on some schedule. That could be a command that creates a cron job on platforms that have it, and on

Windows it could create a scheduled task instead.

But what should we launch? It should probably be a Git command that checks config for a list of repositories, then runs "the maintenance command" on each of those repos.

I'm inserting a break here to draw the eye to a new proposed design:

---

Create a "git maintenance" builtin. This has a few subcommands:

1. "run" will run the configured maintenance on the current repo. This should become the single entry point for users to say "please clean up my repo." What _exactly_ it does can be altered with config. I'll list some possibilities after listing the subcommands.

2. "run-on-repos" uses command-line arguments or config to launch "git -C <dir> maintenance run" for all configured directories. The intention is that this is launched on some schedule by a platform- specific scheduling mechanism (i.e. cron).

(This subcommand could use a better name.)

3. "schedule" adds the current repository to the configured list of repositories for running with "run-on-repos". It will also initialize the platform-specific scheduling mechanism. This may be to start the schedule for the first time OR to update how frequent "run-on-repos" is run, as appropriate.

4. (OPTIONAL) "mode <mode>" adjusts the config for the current repo to change the type of maintenance requested for this repo. For example, "simple" could just run "git gc --auto" using a normal range. "incremental" could run the maintenance tasks from this series. Finally, "server" could run maintenance tasks as if we are serving the repo to others, so we repack aggressively with full bitmaps, and more frequently.

Quoted 3 lines · Derrick Stolee

Of course, not every platform has "cron" but that just means we need a cross-platform way to launch Git processes on some schedule. That could be a command that creates a cron job on platforms that have it, and on

There's Unix system that doesn't have cron.
People could use other scheduler mechanism.

A lot of systemd users uses systemd-timer.
I'm using snooze.

Each of those set of utilities have different grammar and configuration.

Quoted 1 line

Windows it could create a scheduled task instead.

Quoted 4 lines

2. "run-on-repos" uses command-line arguments or config to launch "git -C <dir> maintenance run" for all configured directories. The intention is that this is launched on some schedule by a platform- specific scheduling mechanism (i.e. cron).

So, IIUC, Git will have a _hard_ dependencies on cron on *nix? Else, we're gonna received a bug-report that some tools doesn't work?

I've seen some bug report in our distro that "git add -p" doesn't work like documented, because it's in "git-perl" packages.
When we merge "git-perl" back to git, other people (who never use "git add -p" and git-sendemail) complain why does we add a hard dependencies on perl to git.

Quoted 5 lines · Danh Doan

Quoted 3 lines · Derrick Stolee

Of course, not every platform has "cron" but that just means we need a cross-platform way to launch Git processes on some schedule. That could be a command that creates a cron job on platforms that have it, and on

Quoted 5 lines

There's Unix system that doesn't have cron.
People could use other scheduler mechanism.

A lot of systemd users uses systemd-timer.
I'm using snooze.

Thanks for listing some alternatives. I'll look into these.

Quoted 10 lines

Each of those set of utilities have different grammar and configuration.

Quoted 6 lines

Windows it could create a scheduled task instead.

2. "run-on-repos" uses command-line arguments or config to launch "git -C <dir> maintenance run" for all configured directories. The intention is that this is launched on some schedule by a platform- specific scheduling mechanism (i.e. cron).

Quoted 2 lines

So, IIUC, Git will have a _hard_ dependencies on cron on *nix? Else, we're gonna received a bug-report that some tools doesn't work?

No. Such a dependency would be unacceptable. I'm just using cron as an example when available.

Quoted 5 lines

I've seen some bug report in our distro that "git add -p" doesn't work like documented, because it's in "git-perl" packages.
When we merge "git-perl" back to git, other people (who never use "git add -p" and git-sendemail) complain why does we add a hard dependencies on perl to git.

Good news: "git add -p" is becoming a builtin with a lot of work by some determined contributors.

Quoted 9 lines · Derrick Stolee

Quoted 8 lines · Danh Doan

Quoted 3 lines · Derrick Stolee

Of course, not every platform has "cron" but that just means we need a cross-platform way to launch Git processes on some schedule. That could be a command that creates a cron job on platforms that have it, and on

There's Unix system that doesn't have cron.
People could use other scheduler mechanism.

Quoted 4 lines

Quoted 2 lines

A lot of systemd users uses systemd-timer.
I'm using snooze.

Thanks for listing some alternatives. I'll look into these.

I didn't mean to list those alternatives as only possible alternatives.

The point is people have their own preference to choose a scheduler that suites their need.

Someone could use their own supervisor system with things like:

	#/bin/sh

	sleep 3600 # 1 hour
	exec git cmd

When "git cmd" exit, the supervisor will start the job again (because it's down and it needs to be run).

From: Derrick Stolee <[email protected]>

The first job to implement in the 'git run-job' command is the 'commit-graph' job. It is based on the sequence of events in the 'commit-graph' job in Scalar [1]. This sequence is as follows:

1. git commit-graph write --reachable --split
2. git commit-graph verify --shallow
3. If the verify succeeds, stop.
4. Delete the commit-graph-chain file.
5. git commit-graph write --reachable --split

By writing an incremental commit-graph file using the "--split" option we minimize the disruption from this operation. The default behavior is to merge layers until the new "top" layer is less than half the size of the layer below. This provides quick writes "most" of the time, with the longer writes following a power law distribution.

Most importantly, concurrent Git processes only look at the commit-graph-chain file for a very short amount of time, so they will verly likely not be holding a handle to the file when we try to replace it. (This only matters on Windows.)

If a concurrent process reads the old commit-graph-chain file, but our job expires some of the .graph files before they can be read, then those processes will see a warning message (but not fail). This could be avoided by a future update to use the --expire-time argument when writing the commit-graph.

By using 'git commit-graph verify --shallow' we can ensure that the file we just wrote is valid. This is an extra safety precaution that is faster than our 'write' subcommand. In the rare situation that the newest layer of the commit-graph is corrupt, we can "fix" the corruption by deleting the commit-graph-chain file and rewrite the full commit-graph as a new one-layer commit graph. This does not completely prevent _that_ file from being corrupt, but it does recompute the commit-graph by parsing commits from the object database. In our use of this step in Scalar and VFS for Git, we have only seen this issue arise because our microsoft/git fork reverted 43d3561 ("commit-graph write: don't die if the existing graph is corrupt" 2019-03-25) for a while to keep commit-graph writes very fast. We dropped the revert when updating to v2.23.0. The verify still has potential for catching corrupt data across the layer boundary: if the new file has commit X with parent Y in an old file but the commit ID for Y in the old file had a bitswap, then we will notice that in the 'verify' command.

[1] https://github.com/microsoft/scalar/blob/master/Scalar.Common/Maintenance/CommitGraphStep.cs

RFC QUESTIONS:

1. Are links like [1] helpful?

2. Can anyone think of a way to test the rewrite fallback? It requires corrupting the latest file between two steps of this one command, so that is a tricky spot to inject a fault.

Signed-off-by Derrick Stolee <[email protected]>

---
 Documentation/git-run-job.txt | 21 ++++++++++--
 builtin/run-job.c             | 60 ++++++++++++++++++++++++++++++++++-
 commit-graph.c                |  2 +-
 commit-graph.h                |  1 +
 t/t7900-run-job.sh            | 32 +++++++++++++++++++
 5 files changed, 112 insertions(+), 4 deletions(-)

Documentation/git-run-job.txt +19 -2

index 0627b3ed259..8bf2762d650 100644
@@ -9,7 +9,7 @@ git-run-job - Run a maintenance job. Intended for background operation.
 SYNOPSIS
 --------
 [verse]
-'git run-job <job-name>'
+'git run-job commit-graph'
 
 
 DESCRIPTION

@@ -28,7 +28,24 @@ BEHAVIOR MAY BE ALTERED IN THE FUTURE.
 JOBS
 ----
 
-TBD
+'commit-graph'::
+
+The `commit-graph` job updates the `commit-graph` files incrementally,
+then verifies that the written data is correct. If the new layer has an
+issue, then the chain file is removed and the `commit-graph` is
+rewritten from scratch.
++
+The verification only checks the top layer of the `commit-graph` chain.
+If the incremental write merged the new commits with at least one
+existing layer, then there is potential for on-disk corruption being
+carried forward into the new file. This will be noticed and the new
+commit-graph file will be clean as Git reparses the commit data from
+the object database.
++
+The incremental write is safe to run alongside concurrent Git processes
+since it will not expire `.graph` files that were in the previous
+`commit-graph-chain` file. They will be deleted by a later run based on
+the expiration delay.
 
 
 GIT

builtin/run-job.c +31 -1

index 2c78d053aa4..dd7709952d3 100644
@@ -1,12 +1,65 @@
 #include "builtin.h"
 #include "config.h"
+#include "commit-graph.h"
+#include "object-store.h"
 #include "parse-options.h"
+#include "repository.h"
+#include "run-command.h"
 
 static char const * const builtin_run_job_usage[] = {
-	N_("git run-job"),
+	N_("git run-job commit-graph"),
 	NULL
 };
 
+static int run_write_commit_graph(void)
+{
+	struct argv_array cmd = ARGV_ARRAY_INIT;
+
+	argv_array_pushl(&cmd, "commit-graph", "write",
+			 "--split", "--reachable", "--no-progress",
+			 NULL);
+
+	return run_command_v_opt(cmd.argv, RUN_GIT_CMD);
+}
+
+static int run_verify_commit_graph(void)
+{
+	struct argv_array cmd = ARGV_ARRAY_INIT;
+
+	argv_array_pushl(&cmd, "commit-graph", "verify",
+			 "--shallow", "--no-progress", NULL);
+
+	return run_command_v_opt(cmd.argv, RUN_GIT_CMD);
+}
+
+static int run_commit_graph_job(void)
+{
+	char *chain_path;
+
+	if (run_write_commit_graph()) {

builtin/run-job.c +10 -0

+
+	error(_("failed to rewrite commit-graph"));
+	return 1;
+}
+
 int cmd_run_job(int argc, const char **argv, const char *prefix)
 {
 	static struct option builtin_run_job_options[] = {

@@ -23,6 +76,11 @@ int cmd_run_job(int argc, const char **argv, const char *prefix)
 			     builtin_run_job_usage,
 			     PARSE_OPT_KEEP_UNKNOWN);
 
+	if (argc > 0) {
+		if (!strcmp(argv[0], "commit-graph"))
+			return run_commit_graph_job();
+	}
+
 	usage_with_options(builtin_run_job_usage,
 			   builtin_run_job_options);
 }

commit-graph.c +1 -1

index f013a84e294..6867f92d04a 100644
@@ -56,7 +56,7 @@ static char *get_split_graph_filename(struct object_directory *odb,
 		       oid_hex);
 }
 
-static char *get_chain_filename(struct object_directory *odb)
+char *get_chain_filename(struct object_directory *odb)
 {
 	return xstrfmt("%s/info/commit-graphs/commit-graph-chain", odb->path);
 }

commit-graph.h +1 -0

index e87a6f63600..8b7bd5a6cb1 100644
@@ -13,6 +13,7 @@
 struct commit;
 
 char *get_commit_graph_filename(struct object_directory *odb);
+char *get_chain_filename(struct object_directory *odb);
 int open_commit_graph(const char *graph_file, int *fd, struct stat *st);
 
 /*

t/t7900-run-job.sh +32 -0

index 1eac80b7ed3..18b9bd26b3a 100755
@@ -5,6 +5,8 @@ test_description='git run-job
 Testing the background jobs, in the foreground
 '
 
+GIT_TEST_COMMIT_GRAPH=0
+
 . ./test-lib.sh
 
 test_expect_success 'help text' '

@@ -12,4 +14,34 @@ test_expect_success 'help text' '
 	test_i18ngrep "usage: git run-job" err
 '
 
+test_expect_success 'commit-graph job' '
+	git init server &&
+	(
+		cd server &&
+		chain=.git/objects/info/commit-graphs/commit-graph-chain &&
+		git checkout -b master &&
+		for i in $(test_seq 1 15)
+		do
+			test_commit $i || return 1
+		done &&
+		git run-job commit-graph 2>../err &&
+		test_must_be_empty ../err &&
+		test_line_count = 1 $chain &&
+		for i in $(test_seq 16 19)
+		do
+			test_commit $i || return 1
+		done &&
+		git run-job commit-graph 2>../err &&
+		test_must_be_empty ../err &&
+		test_line_count = 2 $chain &&
+		for i in $(test_seq 20 23)
+		do
+			test_commit $i || return 1
+		done &&
+		git run-job commit-graph 2>../err &&
+		test_must_be_empty ../err &&
+		test_line_count = 1 $chain
+	)
+'
+
 test_done

Signature · 1 line

gitgitgadget

From: Derrick Stolee <[email protected]>

When working with very large repositories, an incremental 'git fetch' command can download a large amount of data. If there are many other users pushing to a common repo, then this data can rival the initial pack-file size of a 'git clone' of a medium-size repo.

Users may want to keep the data on their local repos as close as possible to the data on the remote repos by fetching periodically in the background. This can break up a large daily fetch into several smaller hourly fetches.

However, if we simply ran 'git fetch <remote>' in the background, then the user running a foregroudn 'git fetch <remote>' would lose some important feedback when a new branch appears or an existing branch updates. This is especially true if a remote branch is force-updated and this isn't noticed by the user because it occurred in the background. Further, the functionality of 'git push --force-with-lease' becomes suspect.

When running 'git fetch <remote> <options>' in the background, use the following options for careful updating:

1. --no-tags prevents getting a new tag when a user wants to see the new tags appear in their foreground fetches.

2. --refmap= removes the configured refspec which usually updates refs/remotes/<remote>/* with the refs advertised by the remote.

3. By adding a new refspec "+refs/heads/*:refs/hidden/<remote>/*" we can ensure that we actually load the new values somewhere in our refspace while not updating refs/heads or refs/remotes. By storing these refs here, the commit-graph job will update the commit-graph with the commits from these hidden refs.

4. --prune will delete the refs/hidden/<remote> refs that no longer appear on the remote.

We've been using this step as a critical background job in Scalar
[1] (and VFS for Git). This solved a pain point that was showing up in user reports: fetching was a pain! Users do not like waiting to download the data that was created while they were away from their machines. After implementing background fetch, the foreground fetch commands sped up significantly because they mostly just update refs and download a small amount of new data. The effect is especially dramatic when paried with --no-show-forced-udpates (through fetch.showForcedUpdates=false).

[1] https://github.com/microsoft/scalar/blob/master/Scalar.Common/Maintenance/FetchStep.cs

RFC QUESTIONS:

1. One downside of the refs/hidden pattern is that 'git log' will decorate commits with twice as many refs if they appear at a remote ref (<remote>/<ref> _and_ refs/hidden/<remote>/<ref>). Is there an easy way to exclude a refspace from decorations? Should we make refs/hidden/* a "special" refspace that is excluded from decorations?

2. This feature is designed for a desktop machine or equivalent that has a permanent wired network connection, and the machine stays on while the user is not present. For things like laptops with inconsistent WiFi connections (that may be metered) the feature can use the less stable connection more than the user wants. Of course, this feature is opt-in for Git, but in Scalar we have a "scalar pause" command [2] that pauses all maintenance for some amount of time. We should consider a similar mechanism for Git, but for the point of this series the user needs to set up the "background" part of these jobs manually.

[2] https://github.com/microsoft/scalar/blob/master/Scalar/CommandLine/PauseVerb.cs
[3] https://github.com/microsoft/scalar/blob/master/docs/advanced.md#controlling-background-maintenance

Signed-off-by Derrick Stolee <[email protected]>

---
  Documentation/git-run-job.txt | 13 ++++-
  builtin/run-job.c             | 89 ++++++++++++++++++++++++++++++++++-
  t/t7900-run-job.sh            | 22 +++++++++
  3 files changed, 122 insertions(+), 2 deletions(-)

Documentation/git-run-job.txt +12 -1

index 8bf2762d650..eb92e891915 100644
@@ -9,7 +9,7 @@ git-run-job - Run a maintenance job. Intended for background operation.
  SYNOPSIS
  --------
  [verse]
-'git run-job commit-graph'
+'git run-job (commit-graph|fetch)'
 
 
  DESCRIPTION

@@ -47,6 +47,17 @@ since it will not expire `.graph` files that were in the previous
  `commit-graph-chain` file. They will be deleted by a later run based on
  the expiration delay.
 
+'fetch'::
+
+The `fetch` job updates the object directory with the latest objects
+from all registered remotes. For each remote, a `git fetch` command is
+run. The refmap is custom to avoid updating local or remote branches
+(those in `refs/heads` or `refs/remotes`). Instead, the remote refs are
+stored in `refs/hidden/<remote>/`. Also, no tags are updated.
++
+This means that foreground fetches are still required to update the
+remote refs, but the users is notified when the branches and tags are
+updated on the remote.
 
  GIT
  ---

builtin/run-job.c +10 -1

index dd7709952d3..e59056b2918 100644
@@ -7,7 +7,7 @@
  #include "run-command.h"
 
  static char const * const builtin_run_job_usage[] = {
-	N_("git run-job commit-graph"),
+	N_("git run-job (commit-graph|fetch)"),
  	NULL
  };
 

@@ -60,6 +60,91 @@ static int run_commit_graph_job(void)
  	return 1;
  }
 
+static int fetch_remote(const char *remote)
+{
+	int result;
+	struct argv_array cmd = ARGV_ARRAY_INIT;
+	struct strbuf refmap = STRBUF_INIT;
+
+	argv_array_pushl(&cmd, "fetch", remote, "--quiet", "--prune",
+			 "--no-tags", "--refmap=", NULL);
+

builtin/run-job.c +68 -0

+{
+	int result = 0;
+	FILE *proc_out;
+	struct strbuf line = STRBUF_INIT;
+	struct child_process *remote_proc = xmalloc(sizeof(*remote_proc));
+
+	child_process_init(remote_proc);
+
+	argv_array_pushl(&remote_proc->args, "git", "remote", NULL);
+
+	remote_proc->out = -1;
+
+	if (start_command(remote_proc)) {
+		error(_("failed to start 'git remote' process"));
+		result = 1;
+		goto cleanup;
+	}
+
+	proc_out = xfdopen(remote_proc->out, "r");
+
+	/* if there is no line, leave the value as given */
+	while (!strbuf_getline(&line, proc_out))
+		string_list_append(remotes, line.buf);
+
+	strbuf_release(&line);
+
+	fclose(proc_out);
+
+	if (finish_command(remote_proc)) {
+		error(_("failed to finish 'git remote' process"));
+		result = 1;
+	}
+
+cleanup:
+	free(remote_proc);
+	return result;
+}
+
+static int run_fetch_job(void)
+{
+	int result = 0;
+	struct string_list_item *item;
+	struct string_list remotes = STRING_LIST_INIT_DUP;
+
+	if (fill_remotes(&remotes)) {
+		error(_("failed to fill remotes"));
+		result = 1;
+		goto cleanup;
+	}
+
+	/*
+	 * Do not modify the result based on the success of the 'fetch'
+	 * operation, as a loss of network could cause 'fetch' to fail
+	 * quickly. We do not want that to stop the rest of our
+	 * background operations.
+	 */
+	for (item = remotes.items;
+	     item && item < remotes.items + remotes.nr;
+	     item++)
+		fetch_remote(item->string);
+
+cleanup:
+	string_list_clear(&remotes, 0);
+	return result;
+}
+
  int cmd_run_job(int argc, const char **argv, const char *prefix)
  {
  	static struct option builtin_run_job_options[] = {

@@ -79,6 +164,8 @@ int cmd_run_job(int argc, const char **argv, const char *prefix)
  	if (argc > 0) {
  		if (!strcmp(argv[0], "commit-graph"))
  			return run_commit_graph_job();
+		if (!strcmp(argv[0], "fetch"))
+			return run_fetch_job();
  	}
 
  	usage_with_options(builtin_run_job_usage,

t/t7900-run-job.sh +22 -0

index 18b9bd26b3a..d3faeba135b 100755
@@ -44,4 +44,26 @@ test_expect_success 'commit-graph job' '
  	)
  '
 
+test_expect_success 'fetch job' '
+	git clone "file://$(pwd)/server" client &&
+
+	# Before fetching, build a client commit-graph
+	git -C client run-job commit-graph &&
+	chain=client/.git/objects/info/commit-graphs/commit-graph-chain &&
+	test_line_count = 1 $chain &&
+
+	git -C client branch -v --remotes >before-refs &&
+	test_commit -C server 24 &&
+
+	git -C client run-job fetch &&
+	git -C client branch -v --remotes >after-refs &&
+	test_cmp before-refs after-refs &&
+	test_cmp server/.git/refs/heads/master \
+		 client/.git/refs/hidden/origin/master &&
+
+	# the hidden ref should trigger a new layer in the commit-graph
+	git -C client run-job commit-graph &&
+	test_line_count = 2 $chain
+'
+
  test_done

Hi Stolee

Quoted 3 lines · Derrick Stolee via GitGitGadget

From: Derrick Stolee <[email protected]>
RFC QUESTIONS:

Quoted 6 lines

1. One downside of the refs/hidden pattern is that 'git log' will decorate commits with twice as many refs if they appear at a remote ref (<remote>/<ref> _and_ refs/hidden/<remote>/<ref>). Is there an easy way to exclude a refspace from decorations? Should we make refs/hidden/* a "special" refspace that is excluded from decorations?

Having some way to specify which refs outside of refs/{heads,remote,tags}/ to show or exclude from decorations would be useful I think. Fetching to a hidden ref is a good idea (as are the other steps you outline above) but as you say we don't want it to show up in the output of 'git log' etc.

Quoted 1 line

+static int fill_remotes(struct string_list *remotes)

Isn't there a easy way to get this using the config api rather than forking 'git remote'?

1. One downside of the refs/hidden pattern is that 'git log' will decorate commits with twice as many refs if they appear at a remote ref (<remote>/<ref> _and_ refs/hidden/<remote>/<ref>). Is there an easy way to exclude a refspace from decorations?

Should
we make refs/hidden/* a "special" refspace that is excluded from decorations?

Quoted 4 lines · Derrick Stolee via GitGitGadget

1. One downside of the refs/hidden pattern is that 'git log' will decorate commits with twice as many refs if they appear at a remote ref (<remote>/<ref> _and_ refs/hidden/<remote>/<ref>). Is there an easy way to exclude a refspace from decorations?

I do not think there is, but it makes sense to teach the decoration machinery to either use only certain refs hierarchies or use all hierarchies except for certain ones; if we want to make sure we won't break existing workflows, we should by defautlt use all the refs we currently use and nothing else, but over time we probably would want to migrate the default to cover only the local and remote-tracking branches and tags (and at that point, refs/hidden would be outside the decoration source).

By the way, I have a moderately strong opinion against the use of "refs/hidden" for the purpose of "prefetch in advance, without disrupting refs/remotes". There may be more than one reason why some refs want to be "hidden", and depending on the purose, the exact refs that one workflow (e.g. "decorate") wants to hide may be the ones that want to be exposed.

If we rename it to "refs/prefetch/", would it make the purpose of the hierarchy clearer without squatting on a vague (because it does not tell why it is hidden) name "hidden" that other people might want to use to hide their stuff for different reasons?

Quoted 3 lines

Should
we make refs/hidden/* a "special" refspace that is excluded from decorations?

See above.

From: Derrick Stolee <[email protected]>

When working with very large repositories, an incremental 'git fetch' command can download a large amount of data. If there are many other users pushing to a common repo, then this data can rival the initial pack-file size of a 'git clone' of a medium-size repo.

Users may want to keep the data on their local repos as close as possible to the data on the remote repos by fetching periodically in the background. This can break up a large daily fetch into several smaller hourly fetches.

However, if we simply ran 'git fetch <remote>' in the background, then the user running a foregroudn 'git fetch <remote>' would lose some important feedback when a new branch appears or an existing branch updates. This is especially true if a remote branch is force-updated and this isn't noticed by the user because it occurred in the background. Further, the functionality of 'git push --force-with-lease' becomes suspect.

When running 'git fetch <remote> <options>' in the background, use the following options for careful updating:

1. --no-tags prevents getting a new tag when a user wants to see the new tags appear in their foreground fetches.

2. --refmap= removes the configured refspec which usually updates refs/remotes/<remote>/* with the refs advertised by the remote.

3. By adding a new refspec "+refs/heads/*:refs/hidden/<remote>/*" we can ensure that we actually load the new values somewhere in our refspace while not updating refs/heads or refs/remotes. By storing these refs here, the commit-graph job will update the commit-graph with the commits from these hidden refs.

4. --prune will delete the refs/hidden/<remote> refs that no longer appear on the remote.

We've been using this step as a critical background job in Scalar
[1] (and VFS for Git). This solved a pain point that was showing up in user reports: fetching was a pain! Users do not like waiting to download the data that was created while they were away from their machines. After implementing background fetch, the foreground fetch commands sped up significantly because they mostly just update refs and download a small amount of new data. The effect is especially dramatic when paried with --no-show-forced-udpates (through fetch.showForcedUpdates=false).

[1] https://github.com/microsoft/scalar/blob/master/Scalar.Common/Maintenance/FetchStep.cs

RFC QUESTIONS:

1. One downside of the refs/hidden pattern is that 'git log' will decorate commits with twice as many refs if they appear at a remote ref (<remote>/<ref> _and_ refs/hidden/<remote>/<ref>). Is there an easy way to exclude a refspace from decorations? Should we make refs/hidden/* a "special" refspace that is excluded from decorations?

2. This feature is designed for a desktop machine or equivalent that has a permanent wired network connection, and the machine stays on while the user is not present. For things like laptops with inconsistent WiFi connections (that may be metered) the feature can use the less stable connection more than the user wants. Of course, this feature is opt-in for Git, but in Scalar we have a "scalar pause" command [2] that pauses all maintenance for some amount of time. We should consider a similar mechanism for Git, but for the point of this series the user needs to set up the "background" part of these jobs manually.

[2] https://github.com/microsoft/scalar/blob/master/Scalar/CommandLine/PauseVerb.cs
[3] https://github.com/microsoft/scalar/blob/master/docs/advanced.md#controlling-background-maintenance

Signed-off-by Derrick Stolee <[email protected]>

---
 Documentation/git-run-job.txt | 13 ++++-
 builtin/run-job.c             | 89 ++++++++++++++++++++++++++++++++++-
 t/t7900-run-job.sh            | 22 +++++++++
 3 files changed, 122 insertions(+), 2 deletions(-)

Documentation/git-run-job.txt +12 -1

index 8bf2762d650..eb92e891915 100644
@@ -9,7 +9,7 @@ git-run-job - Run a maintenance job. Intended for background operation.
 SYNOPSIS
 --------
 [verse]
-'git run-job commit-graph'
+'git run-job (commit-graph|fetch)'
 
 
 DESCRIPTION

@@ -47,6 +47,17 @@ since it will not expire `.graph` files that were in the previous
 `commit-graph-chain` file. They will be deleted by a later run based on
 the expiration delay.
 
+'fetch'::
+
+The `fetch` job updates the object directory with the latest objects
+from all registered remotes. For each remote, a `git fetch` command is
+run. The refmap is custom to avoid updating local or remote branches
+(those in `refs/heads` or `refs/remotes`). Instead, the remote refs are
+stored in `refs/hidden/<remote>/`. Also, no tags are updated.
++
+This means that foreground fetches are still required to update the
+remote refs, but the users is notified when the branches and tags are
+updated on the remote.
 
 GIT
 ---

builtin/run-job.c +15 -1

index dd7709952d3..e59056b2918 100644
@@ -7,7 +7,7 @@
 #include "run-command.h"
 
 static char const * const builtin_run_job_usage[] = {
-	N_("git run-job commit-graph"),
+	N_("git run-job (commit-graph|fetch)"),
 	NULL
 };
 

@@ -60,6 +60,91 @@ static int run_commit_graph_job(void)
 	return 1;
 }
 
+static int fetch_remote(const char *remote)
+{
+	int result;
+	struct argv_array cmd = ARGV_ARRAY_INIT;
+	struct strbuf refmap = STRBUF_INIT;
+
+	argv_array_pushl(&cmd, "fetch", remote, "--quiet", "--prune",
+			 "--no-tags", "--refmap=", NULL);
+
+	strbuf_addf(&refmap, "+refs/heads/*:refs/hidden/%s/*", remote);
+	argv_array_push(&cmd, refmap.buf);
+
+	result = run_command_v_opt(cmd.argv, RUN_GIT_CMD);
+

builtin/run-job.c +63 -0

+
+	child_process_init(remote_proc);
+
+	argv_array_pushl(&remote_proc->args, "git", "remote", NULL);
+
+	remote_proc->out = -1;
+
+	if (start_command(remote_proc)) {
+		error(_("failed to start 'git remote' process"));
+		result = 1;
+		goto cleanup;
+	}
+
+	proc_out = xfdopen(remote_proc->out, "r");
+
+	/* if there is no line, leave the value as given */
+	while (!strbuf_getline(&line, proc_out))
+		string_list_append(remotes, line.buf);
+
+	strbuf_release(&line);
+
+	fclose(proc_out);
+
+	if (finish_command(remote_proc)) {
+		error(_("failed to finish 'git remote' process"));
+		result = 1;
+	}
+
+cleanup:
+	free(remote_proc);
+	return result;
+}
+
+static int run_fetch_job(void)
+{
+	int result = 0;
+	struct string_list_item *item;
+	struct string_list remotes = STRING_LIST_INIT_DUP;
+
+	if (fill_remotes(&remotes)) {
+		error(_("failed to fill remotes"));
+		result = 1;
+		goto cleanup;
+	}
+
+	/*
+	 * Do not modify the result based on the success of the 'fetch'
+	 * operation, as a loss of network could cause 'fetch' to fail
+	 * quickly. We do not want that to stop the rest of our
+	 * background operations.
+	 */
+	for (item = remotes.items;
+	     item && item < remotes.items + remotes.nr;
+	     item++)
+		fetch_remote(item->string);
+
+cleanup:
+	string_list_clear(&remotes, 0);
+	return result;
+}
+
 int cmd_run_job(int argc, const char **argv, const char *prefix)
 {
 	static struct option builtin_run_job_options[] = {

@@ -79,6 +164,8 @@ int cmd_run_job(int argc, const char **argv, const char *prefix)
 	if (argc > 0) {
 		if (!strcmp(argv[0], "commit-graph"))
 			return run_commit_graph_job();
+		if (!strcmp(argv[0], "fetch"))
+			return run_fetch_job();
 	}
 
 	usage_with_options(builtin_run_job_usage,

t/t7900-run-job.sh +22 -0

index 18b9bd26b3a..d3faeba135b 100755
@@ -44,4 +44,26 @@ test_expect_success 'commit-graph job' '
 	)
 '
 
+test_expect_success 'fetch job' '
+	git clone "file://$(pwd)/server" client &&
+
+	# Before fetching, build a client commit-graph
+	git -C client run-job commit-graph &&
+	chain=client/.git/objects/info/commit-graphs/commit-graph-chain &&
+	test_line_count = 1 $chain &&
+
+	git -C client branch -v --remotes >before-refs &&
+	test_commit -C server 24 &&
+
+	git -C client run-job fetch &&
+	git -C client branch -v --remotes >after-refs &&
+	test_cmp before-refs after-refs &&
+	test_cmp server/.git/refs/heads/master \
+		 client/.git/refs/hidden/origin/master &&
+
+	# the hidden ref should trigger a new layer in the commit-graph
+	git -C client run-job commit-graph &&
+	test_line_count = 2 $chain
+'
+
 test_done

Signature · 1 line

gitgitgadget

Create a 'loose-objects' job for the 'git run-job' command. This helps clean up loose objects without disrupting concurrent Git commands using the following sequence of events:

From: Derrick Stolee <[email protected]>

The previous change cleaned up loose objects using the 'loose-objects' that can be run safely in the background. Add a similar job that performs similar cleanups for pack-files.

One issue with running 'git repack' is that it is designed to repack all pack-files into a single pack-file. While this is the most space-efficient way to store object data, it is not time or memory efficient. This becomes extremely important if the repo is so large that a user struggles to store two copies of the pack on their disk.

Instead, perform an "incremental" repack by collecting a few small pack-files into a new pack-file. The multi-pack-index facilitates this process ever since 'git multi-pack-index expire' was added in 19575c7 (multi-pack-index: implement 'expire' subcommand, 2019-06-10) and 'git multi-pack-index repack' was added in ce1e4a1 (midx: implement midx_repack(), 2019-06-10).

The 'pack-files' job runs the following steps:

1. 'git multi-pack-index write' creates a multi-pack-index file if one did not exist, and otherwise will update the multi-pack-index with any new pack-files that appeared since the last write. This is particularly relevant with the background fetch job.

When the multi-pack-index sees two copies of the same object, it stores the offset data into the newer pack-file. This means that some old pack-files could become "unreferenced" which I will use to mean "a pack-file that is in the pack-file list of the multi-pack-index but none of the objects in the multi-pack-index reference a location inside that pack-file."

2. 'git multi-pack-index expire' deletes any unreferenced pack-files and updaes the multi-pack-index to drop those pack-files from the

list. This is safe to do as concurrent Git processes will see the multi-pack-index and not open those packs when looking for object contents. (Similar to the 'loose-objects' job, there are some Git

commands that open pack-files regardless of the multi-pack-index, but they are rarely used. Further, a user that self-selects to use background operations would likely refrain from using those commands.)

3. 'git multi-pack-index repack --bacth-size=<size>' collects a set

of pack-files that are listed in the multi-pack-index and creates a new pack-file containing the objects whose offsets are listed by the multi-pack-index to be in those objects. The set of pack- files is selected greedily by sorting the pack-files by modified time and adding a pack-file to the set if its "expected size" is smaller than the batch size until the total expected size of the selected pack-files is at least the batch size. The "expected size" is calculated by taking the size of the pack-file divided by the number of objects in the pack-file and multiplied by the number of objects from the multi-pack-index with offset in that pack-file. The expected size approximats how much data from that

pack-file will contribute to the resulting pack-file size. The intention is that the resulting pack-file will be close in size to the provided batch size.

The next run of the pack-files job will delete these repacked pack-files during the 'expire' step.

In this version, the batch size is set to "0" which ignores the size restrictions when selecting the pack-files. It instead selects all pack-files and repacks all packed objects into a single pack-file. This will be updated in the next change, but it requires doing some calculations that are better isolated to a separate change.

Each of the above steps update the multi-pack-index file. After each step, we verify the new multi-pack-index. If the new multi-pack-index is corrupt, then delete the multi-pack-index, rewrite it from scratch, and stop doing the later steps of the job. This is intended to be an extra-safe check without leaving a repo with many pack-files without a multi-pack-index.

These steps are based on a similar background maintenance step in Scalar (and VFS for Git) [1]. This was incredibly effective for users of the Windows OS repository. After using the same VFS for Git repository for over a year, some users had _thousands_ of pack-files that combined to up to 250 GB of data. We noticed a few users were running into the open file descriptor limits (due in part to a bug in the multi-pack-index fixed by af96fe3392 (midx: add packs to packed_git linked list, 2019-04-29).

These pack-files were mostly small since they contained the commits and trees that were pushed to the origin in a given hour. The GVFS protocol includes a "prefetch" step that asks for pre-computed pack- files containing commits and trees by timestamp. These pack-files were grouped into "daily" pack-files once a day for up to 30 days. If a user did not request prefetch packs for over 30 days, then they would get the entire history of commits and trees in a new, large pack-file. This led to a large number of pack-files that had poor delta compression.

By running this pack-file maintenance step once per day, these repos with thousands of packs spanning 200+ GB dropped to dozens of pack- files spanning 30-50 GB. This was done all without removing objects from the system and using a constant batch size of two gigabytes. Once the work was done to reduce the pack-files to small sizes, the batch size of two gigabytes means that not every run triggers a repack operation, so the following run will not expire a pack-file. This has kept these repos in a "clean" state.

[1] https://github.com/microsoft/scalar/blob/master/Scalar.Common/Maintenance/PackfileMaintenanceStep.cs

Signed-off-by Derrick Stolee <[email protected]>

---
 Documentation/git-run-job.txt | 18 ++++++-
 builtin/run-job.c             | 90 ++++++++++++++++++++++++++++++++++-
 midx.c                        |  2 +-
 midx.h                        |  1 +
 t/t7900-run-job.sh            | 39 +++++++++++++++
 5 files changed, 147 insertions(+), 3 deletions(-)

Documentation/git-run-job.txt +17 -1

index 43ca1160b5a..108ed25b8bd 100644
@@ -9,7 +9,7 @@ git-run-job - Run a maintenance job. Intended for background operation.
 SYNOPSIS
 --------
 [verse]
-'git run-job (commit-graph|fetch|loose-objects)'
+'git run-job (commit-graph|fetch|loose-objects|pack-files)'
 
 
 DESCRIPTION

@@ -71,6 +71,22 @@ a batch of loose objects. The batch size is limited to 50 thousand
 objects to prevent the job from taking too long on a repository with
 many loose objects.
 
+'pack-files'::
+
+The `pack-files` job incrementally repacks the object directory using
+the `multi-pack-index` feature. In order to prevent race conditions with
+concurrent Git commands, it follows a two-step process. First, it
+deletes any pack-files included in the `multi-pack-index` where none of
+the objects in the `multi-pack-index` reference those pack-files; this
+only happens if all objects in the pack-file are also stored in a newer
+pack-file. Second, it selects a group of pack-files whose "expected
+size" is below the batch size until the group has total expected size at
+least the batch size; see the `--batch-size` option for the `repack`
+subcommand in linkgit:git-multi-pack-index[1]. The default batch-size is
+zero, which is a special case that attempts to repack all pack-files
+into a single pack-file.
+
+
 GIT
 ---
 Part of the linkgit:git[1] suite

builtin/run-job.c +89 -1

index cecf9058c51..d3543f7ccb9 100644
@@ -1,13 +1,14 @@
 #include "builtin.h"
 #include "config.h"
 #include "commit-graph.h"
+#include "midx.h"
 #include "object-store.h"
 #include "parse-options.h"
 #include "repository.h"
 #include "run-command.h"
 
 static char const * const builtin_run_job_usage[] = {
-	N_("git run-job (commit-graph|fetch|loose-objects)"),
+	N_("git run-job (commit-graph|fetch|loose-objects|pack-files)"),
 	NULL
 };
 

@@ -238,6 +239,91 @@ static int run_loose_objects_job(void)
 	return prune_packed() || pack_loose();
 }
 
+static int multi_pack_index_write(void)
+{
+	struct argv_array cmd = ARGV_ARRAY_INIT;
+	argv_array_pushl(&cmd, "multi-pack-index", "write",
+			 "--no-progress", NULL);
+	return run_command_v_opt(cmd.argv, RUN_GIT_CMD);
+}
+
+static int rewrite_multi_pack_index(void)
+{
+	char *midx_name = get_midx_filename(the_repository->objects->odb->path);
+
+	unlink(midx_name);
+	free(midx_name);
+
+	if (multi_pack_index_write()) {
+		error(_("failed to rewrite multi-pack-index"));
+		return 1;
+	}
+
+	return 0;
+}
+
+static int multi_pack_index_verify(void)
+{
+	struct argv_array cmd = ARGV_ARRAY_INIT;
+	argv_array_pushl(&cmd, "multi-pack-index", "verify",
+			 "--no-progress", NULL);
+	return run_command_v_opt(cmd.argv, RUN_GIT_CMD);
+}
+
+static int multi_pack_index_expire(void)
+{
+	struct argv_array cmd = ARGV_ARRAY_INIT;
+	argv_array_pushl(&cmd, "multi-pack-index", "expire",
+			 "--no-progress", NULL);
+	return run_command_v_opt(cmd.argv, RUN_GIT_CMD);
+}
+
+static int multi_pack_index_repack(void)
+{
+	int result;
+	struct argv_array cmd = ARGV_ARRAY_INIT;
+	argv_array_pushl(&cmd, "multi-pack-index", "repack",
+			 "--no-progress", "--batch-size=0", NULL);
+	result = run_command_v_opt(cmd.argv, RUN_GIT_CMD);
+
+	if (result && multi_pack_index_verify()) {
+		warning(_("multi-pack-index verify failed after repack"));
+		result = rewrite_multi_pack_index();
+	}
+
+	return result;
+}
+
+static int run_pack_files_job(void)
+{
+	if (multi_pack_index_write()) {
+		error(_("failed to write multi-pack-index"));
+		return 1;
+	}
+
+	if (multi_pack_index_verify()) {
+		warning(_("multi-pack-index verify failed after initial write"));
+		return rewrite_multi_pack_index();
+	}
+
+	if (multi_pack_index_expire()) {
+		error(_("multi-pack-index expire failed"));
+		return 1;
+	}
+
+	if (multi_pack_index_verify()) {
+		warning(_("multi-pack-index verify failed after expire"));
+		return rewrite_multi_pack_index();
+	}
+
+	if (multi_pack_index_repack()) {
+		error(_("multi-pack-index repack failed"));
+		return 1;
+	}
+
+	return 0;
+}
+
 int cmd_run_job(int argc, const char **argv, const char *prefix)
 {
 	static struct option builtin_run_job_options[] = {

@@ -261,6 +347,8 @@ int cmd_run_job(int argc, const char **argv, const char *prefix)
 			return run_fetch_job();
 		if (!strcmp(argv[0], "loose-objects"))
 			return run_loose_objects_job();
+		if (!strcmp(argv[0], "pack-files"))
+			return run_pack_files_job();
 	}
 
 	usage_with_options(builtin_run_job_usage,

midx.c +1 -1

index 1527e464a7b..0f0d0a38812 100644
@@ -36,7 +36,7 @@
 
 #define PACK_EXPIRED UINT_MAX
 
-static char *get_midx_filename(const char *object_dir)
+char *get_midx_filename(const char *object_dir)
 {
 	return xstrfmt("%s/pack/multi-pack-index", object_dir);
 }

midx.h +1 -0

index e6fa356b5ca..cf2c09dffc2 100644
@@ -39,6 +39,7 @@ struct multi_pack_index {
 
 #define MIDX_PROGRESS     (1 << 0)
 
+char *get_midx_filename(const char *object_dir);
 struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);
 int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);
 int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);

t/t7900-run-job.sh +39 -0

index 41da083257b..416ba04989d 100755
@@ -6,6 +6,7 @@ Testing the background jobs, in the foreground
 '
 
 GIT_TEST_COMMIT_GRAPH=0
+GIT_TEST_MULTI_PACK_INDEX=0
 
 . ./test-lib.sh
 

@@ -93,4 +94,42 @@ test_expect_success 'loose-objects job' '
 	test_cmp packs-between packs-after
 '
 
+test_expect_success 'pack-files job' '
+	packDir=.git/objects/pack &&
+
+	# Create three disjoint pack-files with size BIG, small, small.
+
+	echo HEAD~2 | git -C client pack-objects --revs $packDir/test-1 &&
+
+	test_tick &&
+	git -C client pack-objects --revs $packDir/test-2 <<-\EOF &&
+	HEAD~1
+	^HEAD~2
+	EOF
+
+	test_tick &&
+	git -C client pack-objects --revs $packDir/test-3 <<-\EOF &&
+	HEAD
+	^HEAD~1
+	EOF
+
+	rm -f client/$packDir/pack-* &&
+	rm -f client/$packDir/loose-* &&
+
+	ls client/$packDir/*.pack >packs-before &&
+	test_line_count = 3 packs-before &&
+
+	# the job repacks the two into a new pack, but does not
+	# delete the old ones.
+	git -C client run-job pack-files &&
+	ls client/$packDir/*.pack >packs-between &&
+	test_line_count = 4 packs-between &&
+
+	# the job deletes the two old packs, and does not write
+	# a new one because only one pack remains.
+	git -C client run-job pack-files &&
+	ls client/.git/objects/pack/*.pack >packs-after &&
+	test_line_count = 1 packs-after
+'
+
 test_done

Signature · 1 line

gitgitgadget

From: Derrick Stolee <[email protected]>

The 'git job-runner' command was introduced with a '--repo=<path>' option so a user could start a process with an explicit list of repos. However, it is more likely that we will want 'git job-runner' to start without any arguments and discover the set of repos from another source.

A natural source is the Git config. The 'git job-runner' does not need to run in a Git repository, but the config could be located in the global or system config.

Create the job.repo config setting as a multi-valued config option.

Read all values for job.repo whenever triggering an iteration of the job loop. This allows a user to add or remove repos dynamically without restarting the job-runner.

Signed-off-by Derrick Stolee <[email protected]>

---
  Documentation/config/job.txt |  7 +++++++
  builtin/job-runner.c         | 20 ++++++++++++++++++++
  2 files changed, 27 insertions(+)

Documentation/config/job.txt +7 -0

index efdb76afad3..2dfed3935fa 100644
@@ -4,3 +4,10 @@ job.pack-files.batchSize::
  	part of `git run-job pack-files`. If not specified, then a
  	dynamic size calculation is run. See linkgit:git-run-job[1]
  	for more details.
+
+job.repo::
+	Store an absolute path to a repository that wants background
+	jobs run by `git job-runner`. This is a multi-valued config
+	option, but it must be stored in a config file seen by the
+	background runner. This may be the global or system config
+	depending on how `git job-runner` is launched on your system.

builtin/job-runner.c +3 -0

index 135288bcaae..bed401f94bf 100644
@@ -20,6 +20,9 @@ static int arg_repos_append(const struct option *opt,
 
  static int load_active_repos(struct string_list *repos)
  {
+	struct string_list_item *item;
+	const struct string_list *config_repos;
+
  	if (arg_repos.nr) {
  		struct string_list_item *item;
  		for (item = arg_repos.items;

builtin/job-runner.c +16 -0

+
+	for (item = config_repos->items;
+	     item && item < config_repos->items + config_repos->nr;
+	     item++) {
+		DIR *dir = opendir(item->string);
+
+		if (!dir)
+			continue;
+
+		closedir(dir);
+		string_list_append(repos, item->string);
+	}
+
+	string_list_sort(repos);
+	string_list_remove_duplicates(repos, 0);
+
  	return 0;
  }
 

Hi Stolee

Quoted 1 line · Derrick Stolee via GitGitGadget

  +    config_repos = git_config_get_value_multi("job.repo");

Does this really re-read the config files as the commit message suggests or just return the cached values? Perhaps the runner could exec itself with a --run-jobs option to actually run the jobs so that it sees any updated config values.

From: Derrick Stolee <[email protected]>

The 'git job-runner' command was introduced with a '--repo=<path>' option so a user could start a process with an explicit list of repos. However, it is more likely that we will want 'git job-runner' to start without any arguments and discover the set of repos from another source.

A natural source is the Git config. The 'git job-runner' does not need to run in a Git repository, but the config could be located in the global or system config.

Create the job.repo config setting as a multi-valued config option.

Read all values for job.repo whenever triggering an iteration of the job loop. This allows a user to add or remove repos dynamically without restarting the job-runner.

Signed-off-by Derrick Stolee <[email protected]>

---
  Documentation/config/job.txt |  7 +++++++
  builtin/job-runner.c         | 20 ++++++++++++++++++++
  2 files changed, 27 insertions(+)

Documentation/config/job.txt +7 -0

index efdb76afad3..2dfed3935fa 100644
@@ -4,3 +4,10 @@ job.pack-files.batchSize::
  	part of `git run-job pack-files`. If not specified, then a
  	dynamic size calculation is run. See linkgit:git-run-job[1]
  	for more details.
+
+job.repo::
+	Store an absolute path to a repository that wants background
+	jobs run by `git job-runner`. This is a multi-valued config
+	option, but it must be stored in a config file seen by the
+	background runner. This may be the global or system config
+	depending on how `git job-runner` is launched on your system.

builtin/job-runner.c +20 -0

index 135288bcaae..bed401f94bf 100644
@@ -20,6 +20,9 @@ static int arg_repos_append(const struct option *opt,
 
  static int load_active_repos(struct string_list *repos)
  {
+	struct string_list_item *item;
+	const struct string_list *config_repos;
+
  	if (arg_repos.nr) {
  		struct string_list_item *item;
  		for (item = arg_repos.items;

@@ -29,6 +32,23 @@ static int load_active_repos(struct string_list *repos)
  		return 0;
  	}
 
+	config_repos = git_config_get_value_multi("job.repo");
+
+	for (item = config_repos->items;
+	     item && item < config_repos->items + config_repos->nr;
+	     item++) {
+		DIR *dir = opendir(item->string);
+
+		if (!dir)
+			continue;
+
+		closedir(dir);
+		string_list_append(repos, item->string);
+	}
+
+	string_list_sort(repos);
+	string_list_remove_duplicates(repos, 0);
+
  	return 0;
  }
 

Hi Stolee

Quoted 7 lines · Derrick Stolee via GitGitGadget

From: Derrick Stolee <[email protected]>

The 'git job-runner' command was introduced with a '--repo=<path>' option so a user could start a process with an explicit list of repos. However, it is more likely that we will want 'git job-runner' to start without any arguments and discover the set of repos from another source.

One thought that occurred to me was that it might be convenient to put 'git job-runner $HOME &' in my .bashrc to have it start on login and find all the repositories under $HOME without me having to list each one (and remember to update the list each time a clone a new repository). There are a couple of issues with this
1 - We only want to run the jobs once per repo, not for each worktree. That should be easy enough to implement by checking if we're in the main worktree.

2 - If I start several shells I only want one instance of 'git job-runner' to start. One way to handle that would be for the runner to hold a lock file in the directory it's given and quit if the lock is already taken. It should probably walk up the directory hierarchy to check for lock files as well in case I try to start another job-runner instance in a subdirectory.

From: Derrick Stolee <[email protected]>

We want to run different maintenance tasks at different rates. That means we cannot rely only on the time between job-runner loop pauses to reduce the frequency of specific jobs. This means we need to persist a timestamp for the previous run somewhere. A natural place is the Git config file for that repo.

Create the job.<job-namme>.lastRun config option to store the timestamp of the previous run of that job. Set this config option after a successful run of 'git -C <repo> run-job <job-name>'.

To maximize code reuse, we dynamically construct the config key and parse the config value into a timestamp in a generic way. This makes the introduction of another config option extremely trivial:

The job.<job-name>.interval option allows a user to specify a minimum number of seconds between two calls to 'git run-job <job-name>' on a given repo. This could be stored in the global or system config to provide an update on the default for all repos, or could be updated on a per-repo basis. This is checked on every iteration of the job loop, so a user could update this and see the effect without restarting the job-runner process.

RFC QUESTION: I'm using a 'git -C <repo> config <option>' process to test if enough time has elapsed before running the 'run-job' process. These 'config' processes are pretty light-weight, so hopefully they are not too noisy. An alternative would be to always run 'git -C <repo> run-job <job-name>' and rely on that process to no-op based on the configured values and how recently they were run.

However, that will likely interrupt users who want
to test the jobs in the foreground. Finally, that user disruption would be mitigated by adding a "--check-latest" option. A user running a job manually would not include that argument by default and would succeed. However, any logging that we might do for the job-runner would make it look like we are running the run-job commands all the time even if they are no-ops. Advice welcome!

Signed-off-by Derrick Stolee <[email protected]>

---
  Documentation/config/job.txt  |  10 ++
  Documentation/git-run-job.txt |   3 +
  builtin/job-runner.c          | 177 +++++++++++++++++++++++++++++++++-
  3 files changed, 189 insertions(+), 1 deletion(-)

Documentation/config/job.txt +10 -0

index 2dfed3935fa..7c799d66221 100644
@@ -1,3 +1,13 @@
+job.<job-name>.interval::
+	The minimum number of seconds between runs of
+	`git run-job <job-name>` when running `git job-runner`.
+
+job.<job-name>.lastRun::
+	The Unix epoch for the timestamp of the previous run of
+	`git run-job <job-name>` when running `git job-runner`. You
+	can manually update this to a later time to delay a specific
+	job on this repository.
+
  job.pack-files.batchSize::
  	This string value `<size>` will be passed to the
  	`git multi-pack-index repack --batch-size=<size>` command as

Documentation/git-run-job.txt +3 -0

index cdd6417f7c9..c6d5674d699 100644
@@ -90,6 +90,9 @@ pack-files, the job does not attempt to repack. Otherwise, the batch
  size is the sum of all pack-file sizes minus the largest pack-file size.
  The batch size is capped at two gigabytes. This intends to pack all
  small pack-files into a single pack-file.
++
+The `--batch-size=<size>` option will override the dynamic or configured
+batch size.
 
 
  GIT

builtin/job-runner.c +176 -1

index bed401f94bf..aee55c106e8 100644
@@ -4,11 +4,175 @@
  #include "run-command.h"
  #include "string-list.h"
 
+#define MAX_TIMESTAMP ((timestamp_t)-1)
+
  static char const * const builtin_job_runner_usage[] = {
  	N_("git job-runner [<options>]"),
  	NULL
  };
 
+static char *config_name(const char *prefix,
+			 const char *job,
+			 const char *postfix)
+{
+	int postfix_dot = 0;
+	struct strbuf name = STRBUF_INIT;
+
+	if (prefix) {
+		strbuf_addstr(&name, prefix);
+		strbuf_addch(&name, '.');
+	}
+
+	if (job) {
+		strbuf_addstr(&name, job);
+		postfix_dot = 1;
+	}
+
+	if (postfix) {
+		if (postfix_dot)
+			strbuf_addch(&name, '.');
+		strbuf_addstr(&name, postfix);
+	}
+
+	return strbuf_detach(&name, NULL);
+}
+
+static int try_get_config(const char *job,
+			  const char *repo,
+			  const char *postfix,
+			  char **value)
+{
+	int result = 0;
+	FILE *proc_out;
+	struct strbuf line = STRBUF_INIT;
+	char *cname = config_name("job", job, postfix);
+	struct child_process *config_proc = xmalloc(sizeof(*config_proc));
+
+	child_process_init(config_proc);
+
+	argv_array_push(&config_proc->args, "git");
+	argv_array_push(&config_proc->args, "-C");
+	argv_array_push(&config_proc->args, repo);
+	argv_array_push(&config_proc->args, "config");
+	argv_array_push(&config_proc->args, cname);
+	free(cname);
+
+	config_proc->out = -1;
+
+	if (start_command(config_proc)) {
+		warning(_("failed to start process for repo '%s'"),
+			repo);
+		result = 1;
+		goto cleanup;
+	}
+
+	proc_out = xfdopen(config_proc->out, "r");
+
+	/* if there is no line, leave the value as given */
+	if (!strbuf_getline(&line, proc_out))
+		*value = strbuf_detach(&line, NULL);
+	else
+		*value = NULL;
+
+	strbuf_release(&line);
+
+	fclose(proc_out);
+
+	result = finish_command(config_proc);
+
+cleanup:
+	free(config_proc);
+	return result;
+}
+
+static int try_get_timestamp(const char *job,
+			     const char *repo,
+			     const char *postfix,
+			     timestamp_t *t)
+{
+	char *value;
+	int result = try_get_config(job, repo, postfix, &value);
+
+	if (!result) {
+		*t = atol(value);
+		free(value);
+	}
+
+	return result;
+}
+
+static timestamp_t get_last_run(const char *job,
+				const char *repo)
+{
+	timestamp_t last_run = 0;
+
+	/* In an error state, do not run the job */
+	if (try_get_timestamp(job, repo, "lastrun", &last_run))
+		return MAX_TIMESTAMP;
+
+	return last_run;
+}
+
+static timestamp_t get_interval(const char *job,
+				const char *repo)
+{
+	timestamp_t interval = MAX_TIMESTAMP;
+
+	/* In an error state, do not run the job */
+	if (try_get_timestamp(job, repo, "interval", &interval))
+		return MAX_TIMESTAMP;
+
+	if (interval == MAX_TIMESTAMP) {
+		/* load defaults for each job */
+		if (!strcmp(job, "commit-graph") || !strcmp(job, "fetch"))
+			interval = 60 * 60; /* 1 hour */
+		else
+			interval = 24 * 60 * 60; /* 1 day */
+	}
+
+	return interval;
+}
+
+static int set_last_run(const char *job,
+			const char *repo,
+			timestamp_t last_run)
+{
+	int result = 0;
+	struct strbuf last_run_string = STRBUF_INIT;
+	struct child_process *config_proc = xmalloc(sizeof(*config_proc));
+	char *cname = config_name("job", job, "lastrun");
+
+	strbuf_addf(&last_run_string, "%"PRItime, last_run);
+
+	child_process_init(config_proc);
+
+	argv_array_push(&config_proc->args, "git");
+	argv_array_push(&config_proc->args, "-C");
+	argv_array_push(&config_proc->args, repo);
+	argv_array_push(&config_proc->args, "config");
+	argv_array_push(&config_proc->args, cname);
+	argv_array_push(&config_proc->args, last_run_string.buf);
+	free(cname);
+	strbuf_release(&last_run_string);
+
+	if (start_command(config_proc)) {
+		warning(_("failed to start process for repo '%s'"),
+			repo);
+		result = 1;
+		goto cleanup;
+	}
+
+	if (finish_command(config_proc)) {
+		warning(_("failed to finish process for repo '%s'"),
+			repo);
+		result = 1;
+	}
+
+cleanup:
+	free(config_proc);
+	return result;
+}
+
  static struct string_list arg_repos = STRING_LIST_INIT_DUP;
 
  static int arg_repos_append(const struct option *opt,

@@ -54,9 +218,20 @@ static int load_active_repos(struct string_list *repos)
 
  static int run_job(const char *job, const char *repo)
  {
+	int result;
  	struct argv_array cmd = ARGV_ARRAY_INIT;
+	timestamp_t now = time(NULL);
+	timestamp_t last_run = get_last_run(job, repo);
+	timestamp_t interval = get_interval(job, repo);
+
+	if (last_run + interval > now)
+		return 0;
+
  	argv_array_pushl(&cmd, "-C", repo, "run-job", job, NULL);
-	return run_command_v_opt(cmd.argv, RUN_GIT_CMD);
+	result = run_command_v_opt(cmd.argv, RUN_GIT_CMD);
+
+	set_last_run(job, repo, now);
+	return result;
  }
 
  static int run_job_loop_step(struct string_list *list)

* git run-job <job-name>: This builtin will run a single instance of a maintenance job.

* git job-runner [--repo=<path>]: This builtin will run an infinite loop that executes git run-job as a subcommand.

Quoted 5 lines · Derrick Stolee via GitGitGadget

* git run-job <job-name>: This builtin will run a single instance of a maintenance job.

* git job-runner [--repo=<path>]: This builtin will run an infinite loop that executes git run-job as a subcommand.

What does this have to do with "git", though? IOW, why does this have to be part of Git, so that those who would benefit from having a mechanism that makes it easy to run regular maintenance tasks but are not Git users (or those that want to do such maintenance tasks that are not necessarily tied to "git") must use "git" to do so?

I'll find out later why it is so after reading thru 15 patches myself, so no need to give a quick answer to the above; it was just my knee-jerk reaction.

Quoted 5 lines · Junio C Hamano

Quoted 3 lines · Derrick Stolee via GitGitGadget

* git run-job <job-name>: This builtin will run a single instance of a maintenance job.

Quoted 8 lines

Quoted 2 lines

* git job-runner [--repo=<path>]: This builtin will run an infinite loop that executes git run-job as a subcommand.

What does this have to do with "git", though? IOW, why does this have to be part of Git, so that those who would benefit from having a mechanism that makes it easy to run regular maintenance tasks but are not Git users (or those that want to do such maintenance tasks that are not necessarily tied to "git") must use "git" to do so?

That's a reasonable reaction. The short version of my reasoning is that many many people _use_ Git but are not Git experts. While a Git expert could find the right set of commands to run and at what frequency to keep their repo clean, most users do not want to spend time learning these commands. It's also worth our time as contributors to select what

Quoted 6 lines · Derrick Stolee

Quoted 5 lines · Junio C Hamano

Quoted 3 lines · Derrick Stolee via GitGitGadget

* git run-job <job-name>: This builtin will run a single instance of a maintenance job.

Quoted 8 lines

Quoted 8 lines

Quoted 2 lines

* git job-runner [--repo=<path>]: This builtin will run an infinite loop that executes git run-job as a subcommand.

What does this have to do with "git", though? IOW, why does this have to be part of Git, so that those who would benefit from having a mechanism that makes it easy to run regular maintenance tasks but are not Git users (or those that want to do such maintenance tasks that are not necessarily tied to "git") must use "git" to do so?

I also agree with Junio,
I don't think Git should be responsible to be a scheduler. It's the job of either tranditional crontab, at on *nix, or scheduler on Windows.

Quoted 5 lines

That's a reasonable reaction. The short version of my reasoning is that many many people _use_ Git but are not Git experts. While a Git expert could find the right set of commands to run and at what frequency to keep their repo clean, most users do not want to spend time learning these commands. It's also worth our time as contributors to select what

And now, people will need to learn _both_ Git existing maintainance command, and new scheduler (Do I understand it right?, I haven't go through all patches)

Yes, it could be a setup it once and forget, but, if there's a problem with their local repo, they will scratch their head to understand what wrong with them.

It's easier to destroy their repo, and it's harder to know what operation is running in their computer.

Quoted 27 lines · Danh Doan

Quoted 6 lines · Derrick Stolee

Quoted 5 lines · Junio C Hamano

Quoted 3 lines · Derrick Stolee via GitGitGadget

* git run-job <job-name>: This builtin will run a single instance of a maintenance job.

Quoted 8 lines

Quoted 8 lines

Quoted 2 lines

* git job-runner [--repo=<path>]: This builtin will run an infinite loop that executes git run-job as a subcommand.

What does this have to do with "git", though? IOW, why does this have to be part of Git, so that those who would benefit from having a mechanism that makes it easy to run regular maintenance tasks but are not Git users (or those that want to do such maintenance tasks that are not necessarily tied to "git") must use "git" to do so?

I also agree with Junio,
I don't think Git should be responsible to be a scheduler. It's the job of either tranditional crontab, at on *nix, or scheduler on Windows.

Quoted 5 lines

That's a reasonable reaction. The short version of my reasoning is that many many people _use_ Git but are not Git experts. While a Git expert could find the right set of commands to run and at what frequency to keep their repo clean, most users do not want to spend time learning these commands. It's also worth our time as contributors to select what

Quoted 3 lines

And now, people will need to learn _both_ Git existing maintainance command, and new scheduler (Do I understand it right?, I haven't go through all patches)

The point is that they would not need to learn the existing commands. They could accept the community's "best practices" by running the simple command to start background maintenance.

In an "enterprise" environment, the users would not even need to learn the command in the first place, because the engineering tools team could configure the maintenance tools using the necessary setup scripts to built the repo.

Quoted 6 lines

Yes, it could be a setup it once and forget, but, if there's a problem with their local repo, they will scratch their head to understand what wrong with them.

It's easier to destroy their repo, and it's harder to know what operation is running in their computer.

That's why we need to be careful. Luckily, these steps have been tested in the wild for over a year with great success (as part of VFS for Git).

Adding this set of commands to Git gonna made Git over-complicated, IMHO.

Quoted 14 lines · Junio C Hamano

Quoted 5 lines · Derrick Stolee via GitGitGadget

* git run-job <job-name>: This builtin will run a single instance of a maintenance job.

* git job-runner [--repo=<path>]: This builtin will run an infinite loop that executes git run-job as a subcommand.

What does this have to do with "git", though? IOW, why does this have to be part of Git, so that those who would benefit from having a mechanism that makes it easy to run regular maintenance tasks but are not Git users (or those that want to do such maintenance tasks that are not necessarily tied to "git") must use "git" to do so?

Quoted 3 lines

I'll find out later why it is so after reading thru 15 patches myself, so no need to give a quick answer to the above; it was just my knee-jerk reaction.

That's a reasonable reaction. The short version of my reasoning is that many many people _use_ Git but are not Git experts. While a Git expert could find the right set of commands to run and at what frequency to keep their repo clean, most users do not want to spend time learning these commands. It's also worth our time as contributors to select what a good set of non-intrusive maintenance tasks could be, and make them easily accessible to users.

This series gets us half of the way there: a user interested in doing background maintenance could figure out how to launch "git run-job" on a schedule for their platform, or to launch "git job-runner" at start- up. That's a lot simpler than learning how the commit-graph, multi-pack-index, prune-packed, pack-objects, and fetch builtins work with the complicated sets of arguments.

The second half would be to create a command such as

git please-run-maintenance-on-this-repo

that initializes the background jobs and enables them on the repo they are using. This allows the most casual of Git user to work efficiently on very large repositories.

If there are periodic tasks that should be done, even if only on large repos, then let's have a git gc --periodic that does them. I'm not sure that fetch should be in that set, but nothing prevents users from doing "git fetch origin && git gc --periodic". Let's make it as simple and straightforward as possible.

I'm not opposed to seeing a tool that can schedule periodic maintenance jobs, perhaps in contrib, depending on whether other people think it should go. However, I think running periodic jobs is best handled on Unix with cron or anacron and not a custom tool or a command in Git.

I've dealt with systems that implemented periodic tasks without using the existing tools for doing that, and I've found that usually that's a mistake. Despite seeming straightforward, there are a lot of tricky edge cases to deal with and it's easy to get wrong.

We also don't have to reimplement all the features in the system scheduler and can let expert users use a different tool of their choice instead if cron (or the Windows equivalent) is not to their liking.

Quoted 27 lines · Derrick Stolee

Quoted 14 lines · Junio C Hamano

Quoted 5 lines · Derrick Stolee via GitGitGadget

* git run-job <job-name>: This builtin will run a single instance of a maintenance job.

* git job-runner [--repo=<path>]: This builtin will run an infinite loop that executes git run-job as a subcommand.

What does this have to do with "git", though? IOW, why does this have to be part of Git, so that those who would benefit from having a mechanism that makes it easy to run regular maintenance tasks but are not Git users (or those that want to do such maintenance tasks that are not necessarily tied to "git") must use "git" to do so?

Quoted 3 lines

I'll find out later why it is so after reading thru 15 patches myself, so no need to give a quick answer to the above; it was just my knee-jerk reaction.

That's a reasonable reaction. The short version of my reasoning is that many many people _use_ Git but are not Git experts. While a Git expert could find the right set of commands to run and at what frequency to keep their repo clean, most users do not want to spend time learning these commands. It's also worth our time as contributors to select what a good set of non-intrusive maintenance tasks could be, and make them easily accessible to users.

Quoted 6 lines

This series gets us half of the way there: a user interested in doing background maintenance could figure out how to launch "git run-job" on a schedule for their platform, or to launch "git job-runner" at start- up. That's a lot simpler than learning how the commit-graph, multi-pack-index, prune-packed, pack-objects, and fetch builtins work with the complicated sets of arguments.

If there are periodic tasks that should be done, even if only on large repos, then let's have a git gc --periodic that does them. I'm not sure that fetch should be in that set, but nothing prevents users from doing "git fetch origin && git gc --periodic".

Let's make it as simple and straightforward as possible.

As for handling multiple repositories, the tool to do that could be as simple as a shell script which reads from ~/.config/git/repo-maintenance (or whatever) and runs the same command on all of the repos it finds there, possibly with a subcommand to add and remove repos.

Quoted 4 lines

The second half would be to create a command such as

git please-run-maintenance-on-this-repo

Quoted 3 lines

that initializes the background jobs and enables them on the repo they are using. This allows the most casual of Git user to work efficiently on very large repositories.

I'm not opposed to seeing a tool that can schedule periodic maintenance jobs, perhaps in contrib, depending on whether other people think it should go. However, I think running periodic jobs is best handled on Unix with cron or anacron and not a custom tool or a command in Git.

I've dealt with systems that implemented periodic tasks without using the existing tools for doing that, and I've found that usually that's a mistake. Despite seeming straightforward, there are a lot of tricky edge cases to deal with and it's easy to get wrong.

We also don't have to reimplement all the features in the system scheduler and can let expert users use a different tool of their choice instead if cron (or the Windows equivalent) is not to their liking.

Quoted 4 lines

If there are periodic tasks that should be done, even if only on large repos, then let's have a git gc --periodic that does them. I'm not sure that fetch should be in that set, but nothing prevents users from doing "git fetch origin && git gc --periodic".

Hmm. Who says that maintenance tasks are essentially only gc? With _maaaaaybe_ a fetch thrown in?

Quoted 1 line

Let's make it as simple and straightforward as possible.

I get the impression, however, that many reviewers here seem to favor the goal of making the _patches_ as simple and straightforward as possible, however, at the expense of the original goal. Like, totally sacrificing the ease of use in return for "just use a shell script" advice.

Quoted 4 lines

As for handling multiple repositories, the tool to do that could be as simple as a shell script which reads from ~/.config/git/repo-maintenance (or whatever) and runs the same command on all of the repos it finds there, possibly with a subcommand to add and remove repos.

Sure, that is flexible.

And it requires a ton of Git expertise to know what to put into those scripts. And Git updates cannot deliver more value to those scripts.

Quoted 4 lines

I'm not opposed to seeing a tool that can schedule periodic maintenance jobs, perhaps in contrib, depending on whether other people think it should go. However, I think running periodic jobs is best handled on Unix with cron or anacron and not a custom tool or a command in Git.

Okay, here is a challenge for you: design this such that the Windows experience does _not_ feel like a 3rd-class citizen. Go ahead. Yes, there is a scheduler. Yep, it does not do cron-like things. Precisely: you have to feed it an XML to make use of the "advanced" features. Yeah, I also cannot remember what the semantics are regarding missed jobs due to shutdown cycles. Nope, you cannot rely on the XML being an option, that would require Windows 10. The list goes on.

Quoted 4 lines

I've dealt with systems that implemented periodic tasks without using the existing tools for doing that, and I've found that usually that's a mistake. Despite seeming straightforward, there are a lot of tricky edge cases to deal with and it's easy to get wrong.

But maybe you found one of those issues in Stolee's patches? If so, please do contribute your experience there to point out those issues, so that they can be addressed.

Quoted 3 lines

We also don't have to reimplement all the features in the system scheduler and can let expert users use a different tool of their choice instead if cron (or the Windows equivalent) is not to their liking.

Do we really want to start relying on cron, when the major platform used by the target audience (enterprise software engineers who deal with rather larger repositories than git.git or linux.git) quite obviously _lacks_ support for that?

Hey,

Quoted 7 lines · Johannes Schindelin

Quoted 4 lines

If there are periodic tasks that should be done, even if only on large repos, then let's have a git gc --periodic that does them. I'm not sure that fetch should be in that set, but nothing prevents users from doing "git fetch origin && git gc --periodic".

Hmm. Who says that maintenance tasks are essentially only gc? With _maaaaaybe_ a fetch thrown in?

What I'm saying is that we have a tool to run maintenance tasks on the repository. If we need to perform additional maintenance tasks, let's put them in the same place as the ones we have now. I realize "gc" may become a less accurate name, but oh, well.

Quoted 6 lines

Quoted 1 line

Let's make it as simple and straightforward as possible.

I get the impression, however, that many reviewers here seem to favor the goal of making the _patches_ as simple and straightforward as possible, however, at the expense of the original goal. Like, totally sacrificing the ease of use in return for "just use a shell script" advice.

I think we can have both. They are not mutually exclusive, and I've proposed a suggestion for both.

Quoted 9 lines

Quoted 4 lines

As for handling multiple repositories, the tool to do that could be as simple as a shell script which reads from ~/.config/git/repo-maintenance (or whatever) and runs the same command on all of the repos it finds there, possibly with a subcommand to add and remove repos.

Sure, that is flexible.

And it requires a ton of Git expertise to know what to put into those scripts. And Git updates cannot deliver more value to those scripts.

Perhaps I was unclear what I thought could be the design of this. My proposal is something like the following:

  git schedule-gc add [--period=TIME] [--fetch=REMOTE | --fetch-all] REPO
  git schedule-gc remove REPO

The actual command invoked by the system scheduler would be something like the following:

git schedule-gc run

It would work as I proposed under the hood, but it would be relatively straightforward to use.

Quoted 5 lines

Quoted 4 lines

I'm not opposed to seeing a tool that can schedule periodic maintenance jobs, perhaps in contrib, depending on whether other people think it should go. However, I think running periodic jobs is best handled on Unix with cron or anacron and not a custom tool or a command in Git.

Quoted 7 lines

Okay, here is a challenge for you: design this such that the Windows experience does _not_ feel like a 3rd-class citizen. Go ahead. Yes, there is a scheduler. Yep, it does not do cron-like things. Precisely: you have to feed it an XML to make use of the "advanced" features. Yeah, I also cannot remember what the semantics are regarding missed jobs due to shutdown cycles. Nope, you cannot rely on the XML being an option, that would require Windows 10. The list goes on.

I will freely admit that I know next to nothing about Windows. I have used it only incidentally, if at all, for at least two decades. It is not a platform I generally have an interest in developing for, although I try to make it work as well as possible when I am working on a project which supports it.

It is, in general, my assumption, based on its wide usage, that it is a powerful and robust operating system with many features, but I have little actual knowledge about how it functions or the exact features it provides.

I want a solution that builds on the existing Unix tools for Unix, because that is least surprising to users and it is how Unix tools are supposed to work. I think we can agree that Git was designed with the Unix philosophy in mind.

I also want a solution that works on Windows. Ideally that solution would build on existing components that are part of Windows, because it reduces the maintenance burden on all of us. But unfortunately, I know next to nothing about how to build such a solution.

Quoted 8 lines

Quoted 4 lines

I've dealt with systems that implemented periodic tasks without using the existing tools for doing that, and I've found that usually that's a mistake. Despite seeming straightforward, there are a lot of tricky edge cases to deal with and it's easy to get wrong.

But maybe you found one of those issues in Stolee's patches? If so, please do contribute your experience there to point out those issues, so that they can be addressed.

One of the benefits of using anacron on Unix is that it can skip running tasks when the user is on battery. This is not anything we can portably do across systems, nor is it something that Git should need to know about.

Quoted 8 lines

Quoted 3 lines

We also don't have to reimplement all the features in the system scheduler and can let expert users use a different tool of their choice instead if cron (or the Windows equivalent) is not to their liking.

Do we really want to start relying on cron, when the major platform used by the target audience (enterprise software engineers who deal with rather larger repositories than git.git or linux.git) quite obviously _lacks_ support for that?

Unix users will be unhappy with us if we use our own scheduling system when cron is available. They will expect us to reimplement those features and they will complain if we do not. While I cannot name names, there are a nontrivial number of large, enterprise monorepos that run only on macOS and Linux.

That doesn't prevent us from building tooling that does the scheduling on Windows if we can't use the system scheduler, but it would be nice to try to present a relatively unified interface across the two platforms.

Signature · 2 lines

brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204

Regardless of what happens with the job-runner, I would like to see a top-level command that performs a single iteration of all the recommended maintenance steps, with zero configuration required, on a single repo. This gives an entry point for users who want to manage their own maintenance schedule without running a background process.
...

Quoted 5 lines

Unix users will be unhappy with us if we use our own scheduling system when cron is available. They will expect us to reimplement those features and they will complain if we do not. While I cannot name names, there are a nontrivial number of large, enterprise monorepos that run only on macOS and Linux.

Speaking purely as a user, I agree with this point. This is why I want a single-iteration top-level maintenance command.

Quoted 6 lines · Josh Steadmon

Regardless of what happens with the job-runner, I would like to see a top-level command that performs a single iteration of all the recommended maintenance steps, with zero configuration required, on a single repo. This gives an entry point for users who want to manage their own maintenance schedule without running a background process.
...

Quoted 8 lines

Quoted 5 lines

Unix users will be unhappy with us if we use our own scheduling system when cron is available. They will expect us to reimplement those features and they will complain if we do not. While I cannot name names, there are a nontrivial number of large, enterprise monorepos that run only on macOS and Linux.

Speaking purely as a user, I agree with this point. This is why I want a single-iteration top-level maintenance command.

Yes, well said.

It exactly is what "git gc" was meant to be. To put it differently, if you asked any non-novice end-user if there is one single command that s/he would use to keep a repository healthy, it is very likely that the answer would be "git gc".

And having such a single point of entry would be a good thing.

Quoted 4 lines

It exactly is what "git gc" was meant to be. To put it differently, if you asked any non-novice end-user if there is one single command that s/he would use to keep a repository healthy, it is very likely that the answer would be "git gc".

The biggest problem with bringing up git gc in this context (and it is actually a quite big problem) is that "gc" stands for ...

So while git gc looks like a good candidate on its technical merits, the usability/discoverability point of view paints a very different picture.

I guess I would argue for the introduction of a new command, like git maintenance, which could potentially trigger a git gc --auto, but is primarily intended to run _outside_ of the users' work hours.

Once we have that, we can always figure out whether there is a convenient way to register this via crontab or Windows Task Scheduler, without asking the users to do all of these tedious steps manually.

Therefore, I would like very much to have a git maintenance --schedule (or something like that), even if only on Windows, on the grounds alone that it is even more tedious to work with the Windows Task Scheduler. But I would prefer to have that also in my Linux setup. The convenience for the users (myself included) is just too compelling.

I think it's worth discussing naming, but it's kind of a distraction from what Brian brought up. The real question is, do we consider the existing "git gc" infrastructure such a lost cause that we should touch it as little as possible?

Quoted 3 lines · Jonathan Nieder

The real question is, do we consider the
existing "git gc" infrastructure such a lost cause that we should touch it as little as possible?

I am fine with that, as long as the "new" thing will take over what "git gc" currently does.

Quoted 6 lines · Junio C Hamano

Quoted 3 lines · Jonathan Nieder

The real question is, do we consider the
existing "git gc" infrastructure such a lost cause that we should touch it as little as possible?

Quoted 2 lines

I am fine with that, as long as the "new" thing will take over what "git gc" currently does.

Good reminder, thank you.

Yes, as long as we end up replacing the old thing, making a parallel new thing (e.g. with a config option for switching between during a transition period) can be a fine approach.

Quoted 12 lines · Jonathan Nieder

Quoted 8 lines · Junio C Hamano

Quoted 3 lines · Jonathan Nieder

The real question is, do we consider the
existing "git gc" infrastructure such a lost cause that we should touch it as little as possible?

I am fine with that, as long as the "new" thing will take over what "git gc" currently does.

Good reminder, thank you.

Quoted 3 lines

Yes, as long as we end up replacing the old thing, making a parallel new thing (e.g. with a config option for switching between during a transition period) can be a fine approach.

Thanks for the discussion, everyone. I'm sorry that I'm very late in providing a new RFC that takes this approach, but yes I intend to create the "single entry point" for maintenance activities, and incorporate auto-GC as a default option there.

Something that is a good long-term goal is to have the new maintenance entry point replace the "git gc --auto" calls, so we have better customization of post-command "automatic" maintenance. This can be done without any of the "background" part of my original RFC.

I've just been to busy with other tasks to devote the deep focus that this feature deserves. Thanks for your patience.

-Stolee