New sparse-checkout builtin and "cone" mode

Thanks. I was taking your comment to mean "this builtin is experimental" but what you really mean is "the sparse-checkout feature is (still) experimental".

-Stolee

You need to add an entry for the new command to 'command-list.txt' as well, so it will be included in 'git help -a' and completion (since it's intended to be a porcelain), etc.

Interesting. GitGitGadget runs a documentation build, and it has never failed for me. Will fix.

The content of the "a" file is not important at all to the tests. It only matters that there is a blob in each of these trees.

That's fine. We explicitly set the contents in the line above.

From: Derrick Stolee <[email protected]>

Getting started with a sparse-checkout file can be daunting. Help users start their sparse enlistment using 'git sparse-checkout init'. This will set 'core.sparseCheckout=true' in their config, write an initial set of patterns to the sparse-checkout file, and update their working directory.

Make sure to use the extensions.worktreeConfig setting and write the sparse checkout config to the worktree-specific config file. This avoids confusing interactions with other worktrees.

The use of running another process for 'git read-tree' is sub- optimal. This will be removed in a later change.

---
 Documentation/git-sparse-checkout.txt | 11 ++++
 builtin/sparse-checkout.c             | 79 ++++++++++++++++++++++++++-
 t/t1091-sparse-checkout-builtin.sh    | 41 ++++++++++++++
 3 files changed, 130 insertions(+), 1 deletion(-)

index 46d3dc3cb1..d5fbbf17a0 100644
@@ -28,6 +28,17 @@ COMMANDS
 'list'::
 	Provide a list of the contents in the sparse-checkout file.
 
+'init'::
+	Enable the `core.sparseCheckout` setting. If the
+	sparse-checkout file does not exist, then populate it with
+	patterns that match every file in the root directory and
+	no other directories, then will remove all directories tracked
+	by Git. Add patterns to the sparse-checkout file to
+	repopulate the working directory.
++
+To avoid interfering with other worktrees, it first enables the
+`extensions.worktreeConfig` setting and makes sure to set the
+`core.sparseCheckout` setting in the worktree-specific config file.
 
 SPARSE CHECKOUT
 ----------------

gitgitgadget

From: Derrick Stolee <[email protected]>

When someone wants to clone a large repository, but plans to work using a sparse-checkout file, they either need to do a full checkout first and then reduce the patterns they included, or clone with --no-checkout, set up their patterns, and then run a checkout manually. This requires knowing a lot about the repo shape and how sparse-checkout works.

Add a new '--sparse' option to 'git clone' that initializes the sparse-checkout file to include the following patterns:

	/*
	!/*/

These patterns include every file in the root directory, but no directories. This allows a repo to include files like a README or a bootstrapping script to grow enlistments from that point.

During the 'git sparse-checkout init' call, we must first look to see if HEAD is valid, since 'git clone' does not have a valid HEAD at the point where it initializes the sparse-checkout. The following checkout within the clone command will create the HEAD ref and update the working directory correctly.

---
 Documentation/git-clone.txt        |  8 +++++++-
 builtin/clone.c                    | 27 +++++++++++++++++++++++++++
 builtin/sparse-checkout.c          |  6 ++++++
 t/t1091-sparse-checkout-builtin.sh | 13 +++++++++++++
 4 files changed, 53 insertions(+), 1 deletion(-)

index 34011c2940..0fe91d2f04 100644
@@ -15,7 +15,7 @@ SYNOPSIS
 	  [--dissociate] [--separate-git-dir <git dir>]
 	  [--depth <depth>] [--[no-]single-branch] [--no-tags]
 	  [--recurse-submodules[=<pathspec>]] [--[no-]shallow-submodules]
-	  [--[no-]remote-submodules] [--jobs <n>] [--] <repository>
+	  [--[no-]remote-submodules] [--jobs <n>] [--sparse] [--] <repository>
 	  [<directory>]
 
 DESCRIPTION

@@ -156,6 +156,12 @@ objects from the source repository into a pack in the cloned repository.
 	used, neither remote-tracking branches nor the related
 	configuration variables are created.
 
+--sparse::
+	Initialize the sparse-checkout file so the working
+	directory starts with only the files in the root
+	of the repository. The sparse-checkout file can be
+	modified to grow the working directory as needed.
+
 --mirror::
 	Set up a mirror of the source repository.  This implies `--bare`.
 	Compared to `--bare`, `--mirror` not only maps local branches of the

index c46ee29f0a..4348d962c9 100644
@@ -59,6 +59,7 @@ static const char *real_git_dir;
 static char *option_upload_pack = "git-upload-pack";
 static int option_verbosity;
 static int option_progress = -1;
+static int option_sparse_checkout;
 static enum transport_family family;
 static struct string_list option_config = STRING_LIST_INIT_NODUP;
 static struct string_list option_required_reference = STRING_LIST_INIT_NODUP;

@@ -146,6 +147,8 @@ static struct option builtin_clone_options[] = {
 	OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
 	OPT_BOOL(0, "remote-submodules", &option_remote_submodules,
 		    N_("any cloned submodules will use their remote-tracking branch")),
+	OPT_BOOL(0, "sparse", &option_sparse_checkout,
+		    N_("initialize sparse-checkout file to include only files at root")),
 	OPT_END()
 };
 

@@ -733,6 +736,27 @@ static void update_head(const struct ref *our, const struct ref *remote,
 	}
 }
 
+static int git_sparse_checkout_init(const char *repo)
+{
+	struct argv_array argv = ARGV_ARRAY_INIT;
+	int result = 0;
+	argv_array_pushl(&argv, "-C", repo, "sparse-checkout", "init", NULL);
+
+	/*
+	 * We must apply the setting in the current process
+	 * for the later checkout to use the sparse-checkout file.
+	 */
+	core_apply_sparse_checkout = 1;
+
+	if (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {
+		error(_("failed to initialize sparse-checkout"));
+		result = 1;
+	}
+
+	argv_array_clear(&argv);
+	return result;
+}
+
 static int checkout(int submodule_progress)
 {
 	struct object_id oid;

@@ -1106,6 +1130,9 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 	if (option_required_reference.nr || option_optional_reference.nr)
 		setup_reference();
 
+	if (option_sparse_checkout && git_sparse_checkout_init(repo))
+		return 1;
+
 	remote = remote_get(option_origin);
 
 	strbuf_addf(&default_refspec, "+%s*:%s*", src_ref_prefix,

index 1d2327111a..4198995d46 100644
@@ -109,6 +109,7 @@ static int sparse_checkout_init(int argc, const char **argv)
 	char *sparse_filename;
 	FILE *fp;
 	int res;
+	struct object_id oid;
 
 	if (sc_set_config(MODE_ALL_PATTERNS))
 		return 1;

@@ -130,6 +131,11 @@ static int sparse_checkout_init(int argc, const char **argv)
 	fprintf(fp, "/*\n!/*/\n");
 	fclose(fp);
 
+	if (get_oid("HEAD", &oid)) {
+		/* assume we are in a fresh repo */
+		return 0;
+	}
+
 reset_dir:
 	return update_working_directory();
 }

index c70085a759..d4c145a3af 100755
@@ -88,4 +88,17 @@ test_expect_success 'init with existing sparse-checkout' '
 	test_cmp expect dir
 '
 
+test_expect_success 'clone --sparse' '
+	git clone --sparse repo clone &&
+	git -C clone sparse-checkout list >actual &&
+	cat >expect <<-EOF &&
+		/*
+		!/*/
+	EOF
+	test_cmp expect actual &&
+	ls clone >dir &&
+	echo a >expect &&
+	test_cmp expect dir
+'
+
 test_done

gitgitgadget

From: Derrick Stolee <[email protected]>

The 'git sparse-checkout set' subcommand takes a list of patterns as arguments and writes them to the sparse-checkout file. Then, it updates the working directory using 'git read-tree -mu HEAD'.

The 'set' subcommand will replace the entire contents of the sparse-checkout file. The write_patterns_and_update() method is extracted from cmd_sparse_checkout() to make it easier to implement 'add' and/or 'remove' subcommands in the future.

---
 Documentation/git-sparse-checkout.txt |  5 ++++
 builtin/sparse-checkout.c             | 35 ++++++++++++++++++++++++++-
 t/t1091-sparse-checkout-builtin.sh    | 19 +++++++++++++++
 3 files changed, 58 insertions(+), 1 deletion(-)

index d5fbbf17a0..163f6c8db0 100644
@@ -40,6 +40,11 @@ To avoid interfering with other worktrees, it first enables the
 `extensions.worktreeConfig` setting and makes sure to set the
 `core.sparseCheckout` setting in the worktree-specific config file.
 
+'set'::
+	Write a set of patterns to the sparse-checkout file, as given as
+	a list of arguments following the 'set' subcommand. Update the
+	working directory to match the new patterns.
+
 SPARSE CHECKOUT
 ----------------
 

index 4198995d46..2103cbe00c 100644
@@ -8,7 +8,7 @@
 #include "strbuf.h"
 
 static char const * const builtin_sparse_checkout_usage[] = {
-	N_("git sparse-checkout [init|list]"),
+	N_("git sparse-checkout [init|list|set] <options>"),
 	NULL
 };
 

@@ -140,6 +140,37 @@ static int sparse_checkout_init(int argc, const char **argv)
 	return update_working_directory();
 }
 
+static int write_patterns_and_update(struct pattern_list *pl)
+{
+	char *sparse_filename;
+	FILE *fp;
+
+	sparse_filename = get_sparse_checkout_filename();
+	fp = fopen(sparse_filename, "w");
+	write_patterns_to_file(fp, pl);
+	fclose(fp);
+	free(sparse_filename);
+
+	return update_working_directory();
+}
+
+static int sparse_checkout_set(int argc, const char **argv, const char *prefix)
+{
+	static const char *empty_base = "";
+	int i;
+	struct pattern_list pl;
+	int result;
+	memset(&pl, 0, sizeof(pl));
+
+	for (i = 1; i < argc; i++)
+		add_pattern(argv[i], empty_base, 0, &pl, 0);
+
+	result = write_patterns_and_update(&pl);
+
+	clear_pattern_list(&pl);
+	return result;
+}
+
 int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)
 {
 	static struct option builtin_sparse_checkout_options[] = {

@@ -162,6 +193,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)
 			return sparse_checkout_list(argc, argv);
 		if (!strcmp(argv[0], "init"))
 			return sparse_checkout_init(argc, argv);
+		if (!strcmp(argv[0], "set"))
+			return sparse_checkout_set(argc, argv, prefix);
 	}
 
 	usage_with_options(builtin_sparse_checkout_usage,

index d4c145a3af..19e8673c6b 100755
@@ -101,4 +101,23 @@ test_expect_success 'clone --sparse' '
 	test_cmp expect dir
 '
 
+test_expect_success 'set sparse-checkout using builtin' '
+	git -C repo sparse-checkout set "/*" "!/*/" "*folder*" &&
+	cat >expect <<-EOF &&
+		/*
+		!/*/
+		*folder*
+	EOF
+	git -C repo sparse-checkout list >actual &&
+	test_cmp expect actual &&
+	test_cmp expect repo/.git/info/sparse-checkout &&
+	ls repo >dir  &&
+	cat >expect <<-EOF &&
+		a
+		folder1
+		folder2
+	EOF
+	test_cmp expect dir
+'
+
 test_done

gitgitgadget

From: Derrick Stolee <[email protected]>

The 'git sparse-checkout set' subcommand takes a list of patterns and places them in the sparse-checkout file. Then, it updates the working directory to match those patterns. For a large list of patterns, the command-line call can get very cumbersome.

Add a '--stdin' option to instead read patterns over standard in.

---
 builtin/sparse-checkout.c          | 40 ++++++++++++++++++++++++++++--
 t/t1091-sparse-checkout-builtin.sh | 27 ++++++++++++++++++++
 2 files changed, 65 insertions(+), 2 deletions(-)

index 2103cbe00c..b747b78d34 100644
@@ -145,6 +145,11 @@ static int write_patterns_and_update(struct pattern_list *pl)
 	char *sparse_filename;
 	FILE *fp;
 
+	if (!core_apply_sparse_checkout) {
+		warning(_("core.sparseCheckout is disabled, so changes to the sparse-checkout file will have no effect"));
+		warning(_("run 'git sparse-checkout init' to enable the sparse-checkout feature"));
+	}
+
 	sparse_filename = get_sparse_checkout_filename();
 	fp = fopen(sparse_filename, "w");
 	write_patterns_to_file(fp, pl);

@@ -154,16 +159,47 @@ static int write_patterns_and_update(struct pattern_list *pl)
 	return update_working_directory();
 }
 
+static char const * const builtin_sparse_checkout_set_usage[] = {
+	N_("git sparse-checkout set [--stdin|<patterns>]"),
+	NULL
+};
+
+static struct sparse_checkout_set_opts {
+	int use_stdin;
+} set_opts;
+
 static int sparse_checkout_set(int argc, const char **argv, const char *prefix)
 {
 	static const char *empty_base = "";
 	int i;
 	struct pattern_list pl;
 	int result;
+
+	static struct option builtin_sparse_checkout_set_options[] = {
+		OPT_BOOL(0, "stdin", &set_opts.use_stdin,
+			 N_("read patterns from standard in")),
+		OPT_END(),
+	};
+
 	memset(&pl, 0, sizeof(pl));
 
-	for (i = 1; i < argc; i++)
-		add_pattern(argv[i], empty_base, 0, &pl, 0);
+	argc = parse_options(argc, argv, prefix,
+			     builtin_sparse_checkout_set_options,
+			     builtin_sparse_checkout_set_usage,
+			     PARSE_OPT_KEEP_UNKNOWN);
+
+	if (set_opts.use_stdin) {
+		struct strbuf line = STRBUF_INIT;
+
+		while (!strbuf_getline(&line, stdin)) {
+			size_t len;
+			char *buf = strbuf_detach(&line, &len);
+			add_pattern(buf, empty_base, 0, &pl, 0);
+		}
+	} else {
+		for (i = 0; i < argc; i++)
+			add_pattern(argv[i], empty_base, 0, &pl, 0);
+	}
 
 	result = write_patterns_and_update(&pl);
 

index 19e8673c6b..2a0137fde3 100755
@@ -101,6 +101,13 @@ test_expect_success 'clone --sparse' '
 	test_cmp expect dir
 '
 
+test_expect_success 'warn if core.sparseCheckout is disabled' '
+	test_when_finished git -C repo config --worktree core.sparseCheckout true &&
+	git -C repo config --worktree core.sparseCheckout false &&
+	git -C repo sparse-checkout set folder1 2>err &&
+	test_i18ngrep "core.sparseCheckout is disabled" err
+'
+
 test_expect_success 'set sparse-checkout using builtin' '
 	git -C repo sparse-checkout set "/*" "!/*/" "*folder*" &&
 	cat >expect <<-EOF &&

@@ -120,4 +127,24 @@ test_expect_success 'set sparse-checkout using builtin' '
 	test_cmp expect dir
 '
 
+test_expect_success 'set sparse-checkout using --stdin' '
+	cat >expect <<-EOF &&
+		/*
+		!/*/
+		/folder1/
+		/folder2/
+	EOF
+	git -C repo sparse-checkout set --stdin <expect &&
+	git -C repo sparse-checkout list >actual &&
+	test_cmp expect actual &&
+	test_cmp expect repo/.git/info/sparse-checkout &&
+	ls repo >dir  &&
+	cat >expect <<-EOF &&
+		a
+		folder1
+		folder2
+	EOF
+	test_cmp expect dir
+'
+
 test_done

gitgitgadget

From: Derrick Stolee <[email protected]>

The instructions for disabling a sparse-checkout to a full working directory are complicated and non-intuitive. Add a subcommand, 'git sparse-checkout disable', to perform those steps for the user.

---
 Documentation/git-sparse-checkout.txt | 27 ++++++++++++---------------
 builtin/sparse-checkout.c             | 26 +++++++++++++++++++++++++-
 t/t1091-sparse-checkout-builtin.sh    | 15 +++++++++++++++
 3 files changed, 52 insertions(+), 16 deletions(-)

index 163f6c8db0..b0c141b582 100644
@@ -45,6 +45,10 @@ To avoid interfering with other worktrees, it first enables the
 	a list of arguments following the 'set' subcommand. Update the
 	working directory to match the new patterns.
 
+'disable'::
+	Remove the sparse-checkout file, set `core.sparseCheckout` to
+	`false`, and restore the working directory to include all files.
+
 SPARSE CHECKOUT
 ----------------
 

@@ -62,6 +66,14 @@ directory, it updates the skip-worktree bits in the index based
 on this file. The files matching the patterns in the file will
 appear in the working directory, and the rest will not.
 
+To enable the sparse-checkout feature, run `git sparse-checkout init` to
+initialize a simple sparse-checkout file and enable the `core.sparseCheckout`
+config setting. Then, run `git sparse-checkout set` to modify the patterns in
+the sparse-checkout file.
+
+To repopulate the working directory with all files, use the
+`git sparse-checkout disable` command.
+
 ## FULL PATTERN SET
 
 By default, the sparse-checkout file uses the same syntax as `.gitignore`

@@ -76,21 +88,6 @@ using negative patterns. For example, to remove the file `unwanted`:
 !unwanted
 ----------------
 
-Another tricky thing is fully repopulating the working directory when you
-no longer want sparse checkout. You cannot just disable "sparse
-checkout" because skip-worktree bits are still in the index and your working
-directory is still sparsely populated. You should re-populate the working
-directory with the `$GIT_DIR/info/sparse-checkout` file content as
-follows:
-
-----------------
-/*
-----------------
-
-Then you can disable sparse checkout. Sparse checkout support in 'git
-checkout' and similar commands is disabled by default. You need to
-set `core.sparseCheckout` to `true` in order to have sparse checkout
-support.
 
 SEE ALSO
 --------

index b747b78d34..78a80ce119 100644
@@ -8,7 +8,7 @@
 #include "strbuf.h"
 
 static char const * const builtin_sparse_checkout_usage[] = {
-	N_("git sparse-checkout [init|list|set] <options>"),
+	N_("git sparse-checkout [init|list|set|disable] <options>"),
 	NULL
 };
 

@@ -207,6 +207,28 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)
 	return result;
 }
 
+static int sparse_checkout_disable(int argc, const char **argv)
+{
+	char *sparse_filename;
+	FILE *fp;
+
+	if (sc_set_config(MODE_ALL_PATTERNS))
+		die(_("failed to change config"));
+
+	sparse_filename = get_sparse_checkout_filename();
+	fp = fopen(sparse_filename, "w");
+	fprintf(fp, "/*\n");
+	fclose(fp);
+
+	if (update_working_directory())
+		die(_("error while refreshing working directory"));
+
+	unlink(sparse_filename);
+	free(sparse_filename);
+
+	return sc_set_config(MODE_NO_PATTERNS);
+}
+
 int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)
 {
 	static struct option builtin_sparse_checkout_options[] = {

@@ -231,6 +253,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)
 			return sparse_checkout_init(argc, argv);
 		if (!strcmp(argv[0], "set"))
 			return sparse_checkout_set(argc, argv, prefix);
+		if (!strcmp(argv[0], "disable"))
+			return sparse_checkout_disable(argc, argv);
 	}
 
 	usage_with_options(builtin_sparse_checkout_usage,

index 2a0137fde3..52d24c66ba 100755
@@ -147,4 +147,19 @@ test_expect_success 'set sparse-checkout using --stdin' '
 	test_cmp expect dir
 '
 
+test_expect_success 'sparse-checkout disable' '
+	git -C repo sparse-checkout disable &&
+	test_path_is_missing repo/.git/info/sparse-checkout &&
+	git -C repo config --list >config &&
+	test_i18ngrep "core.sparsecheckout=false" config &&
+	ls repo >dir &&
+	cat >expect <<-EOF &&
+		a
+		deep
+		folder1
+		folder2
+	EOF
+	test_cmp expect dir
+'
+
 test_done

gitgitgadget

From: Jeff Hostetler <[email protected]>

When Git updates the working directory with the sparse-checkout feature enabled, the unpack_trees() method calls clear_ce_flags() to update the skip-wortree bits on the cache entries. This check can be expensive, depending on the patterns used.

Add trace2 regions around the method, including some flag information, so we can get granular performance data during experiments. This data will be used to measure improvements to the pattern-matching algorithms for sparse-checkout.

---
 unpack-trees.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

index 33ea7810d8..01a05ff66d 100644
@@ -1407,15 +1407,23 @@ static int clear_ce_flags(struct index_state *istate,
 			  struct pattern_list *pl)
 {
 	static struct strbuf prefix = STRBUF_INIT;
+	char label[100];
+	int rval;
 
 	strbuf_reset(&prefix);
 
-	return clear_ce_flags_1(istate,
+	xsnprintf(label, sizeof(label), "clear_ce_flags(0x%08lx,0x%08lx)",
+		  (unsigned long)select_mask, (unsigned long)clear_mask);
+	trace2_region_enter("unpack_trees", label, the_repository);
+	rval = clear_ce_flags_1(istate,
 				istate->cache,
 				istate->cache_nr,
 				&prefix,
 				select_mask, clear_mask,
 				pl, 0);
+	trace2_region_leave("unpack_trees", label, the_repository);
+
+	return rval;
 }
 
 /*

gitgitgadget

From: Derrick Stolee <[email protected]>

The sparse-checkout feature can have quadratic performance as the number of patterns and number of entries in the index grow. If there are 1,000 patterns and 1,000,000 entries, this time can be very significant.

Create a new Boolean config option, core.sparseCheckoutCone, to indicate that we expect the sparse-checkout file to contain a more limited set of patterns. This is a separate config setting from core.sparseCheckout to avoid breaking older clients by introducing a tri-state option.

The config option does nothing right now, but will be expanded upon in a later commit.

---
 Documentation/config/core.txt         | 10 ++++--
 Documentation/git-sparse-checkout.txt | 50 +++++++++++++++++++++++++++
 cache.h                               |  4 ++-
 config.c                              |  5 +++
 environment.c                         |  1 +
 t/t1091-sparse-checkout-builtin.sh    | 14 ++++++++
 6 files changed, 81 insertions(+), 3 deletions(-)

index 852d2ba37a..bdbbee58b9 100644
@@ -593,8 +593,14 @@ core.multiPackIndex::
 	multi-pack-index design document].
 
 core.sparseCheckout::
-	Enable "sparse checkout" feature. See section "Sparse checkout" in
-	linkgit:git-read-tree[1] for more information.
+	Enable "sparse checkout" feature. See linkgit:git-sparse-checkout[1]
+	for more information.
+
+core.sparseCheckoutCone::
+	Enables the "cone mode" of the sparse checkout feature. When the
+	sparse-checkout file contains a limited set of patterns, then this
+	mode provides significant performance advantages. See
+	linkgit:git-sparse-checkout[1] for more information.
 
 core.abbrev::
 	Set the length object names are abbreviated to.  If

index 04cabaac11..4980ee198e 100644
@@ -918,12 +918,14 @@ extern char *git_replace_ref_base;
 
 extern int fsync_object_files;
 extern int core_preload_index;
-extern int core_apply_sparse_checkout;
 extern int precomposed_unicode;
 extern int protect_hfs;
 extern int protect_ntfs;
 extern const char *core_fsmonitor;
 
+int core_apply_sparse_checkout;
+int core_sparse_checkout_cone;
+
 /*
  * Include broken refs in all ref iterations, which will
  * generally choke dangerous operations rather than letting

index e7052b3977..d75f88ca0c 100644
@@ -1364,6 +1364,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
 		return 0;
 	}
 
+	if (!strcmp(var, "core.sparsecheckoutcone")) {
+		core_sparse_checkout_cone = git_config_bool(var, value);
+		return 0;
+	}
+
 	if (!strcmp(var, "core.precomposeunicode")) {
 		precomposed_unicode = git_config_bool(var, value);
 		return 0;

index efa072680a..2a1a866659 100644
@@ -67,6 +67,7 @@ enum object_creation_mode object_creation_mode = OBJECT_CREATION_MODE;
 char *notes_ref_name;
 int grafts_replace_parents = 1;
 int core_apply_sparse_checkout;
+int core_sparse_checkout_cone;
 int merge_log_config = -1;
 int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */
 unsigned long pack_size_limit_cfg;

index 52d24c66ba..36fda5907b 100755
@@ -147,6 +147,20 @@ test_expect_success 'set sparse-checkout using --stdin' '
 	test_cmp expect dir
 '
 
+test_expect_success 'cone mode: match patterns' '
+	git -C repo config --worktree core.sparseCheckoutCone true &&
+	rm -rf repo/a repo/folder1 repo/folder2 &&
+	git -C repo read-tree -mu HEAD &&
+	git -C repo reset --hard &&
+	ls repo >dir  &&
+	cat >expect <<-EOF &&
+		a
+		folder1
+		folder2
+	EOF
+	test_cmp expect dir
+'
+
 test_expect_success 'sparse-checkout disable' '
 	git -C repo sparse-checkout disable &&
 	test_path_is_missing repo/.git/info/sparse-checkout &&

gitgitgadget

From: Derrick Stolee <[email protected]>

The parent and recursive patterns allowed by the "cone mode" option in sparse-checkout are restrictive enough that we can avoid using the regex parsing. Everything is based on prefix matches, so we can use hashsets to store the prefixes from the sparse-checkout file. When checking a path, we can strip path entries from the path and check the hashset for an exact match.

As a test, I created a cone-mode sparse-checkout file for the Linux repository that actually includes every file. This was constructed by taking every folder in the Linux repo and creating the pattern pairs here:

	/$folder/
	!/$folder/*/

This resulted in a sparse-checkout file sith 8,296 patterns. Running 'git read-tree -mu HEAD' on this file had the following performance:

	core.sparseCheckout=false: 0.21 s (0.00 s)
	 core.sparseCheckout=true: 3.75 s (3.50 s)
	 core.sparseCheckout=cone: 0.23 s (0.01 s)

The times in parentheses above correspond to the time spent in the first clear_ce_flags() call, according to the trace2 performance traces.

While this example is contrived, it demonstrates how these patterns can slow the sparse-checkout feature.

---
 dir.c                              | 207 +++++++++++++++++++++++++++--
 dir.h                              |  31 +++++
 t/t1091-sparse-checkout-builtin.sh |  11 +-
 unpack-trees.c                     |   1 +
 4 files changed, 241 insertions(+), 9 deletions(-)

index 61f559f980..dfabf9982f 100644
@@ -611,6 +611,150 @@ void parse_path_pattern(const char **pattern,
 	*patternlen = len;
 }
 
+static int pl_hashmap_cmp(const void *unused_cmp_data,
+			  const struct hashmap_entry *a,
+			  const struct hashmap_entry *b,
+			  const void *key)
+{
+	const struct pattern_entry *ee1 =
+			container_of(a, struct pattern_entry, ent);
+	const struct pattern_entry *ee2 =
+			container_of(b, struct pattern_entry, ent);
+
+	size_t min_len = ee1->patternlen <= ee2->patternlen
+			 ? ee1->patternlen
+			 : ee2->patternlen;
+
+	return strncmp(ee1->pattern, ee2->pattern, min_len);
+}
+
+static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern *given)
+{
+	struct pattern_entry *translated;
+	char *truncated;
+	char *data = NULL;
+
+	if (!pl->use_cone_patterns)
+		return;
+
+	if (given->flags & PATTERN_FLAG_NEGATIVE &&
+	    given->flags & PATTERN_FLAG_MUSTBEDIR &&
+	    !strcmp(given->pattern, "/*")) {
+		pl->full_cone = 0;
+		return;
+	}
+
+	if (!given->flags && !strcmp(given->pattern, "/*")) {
+		pl->full_cone = 1;
+		return;
+	}
+
+	if (given->patternlen > 2 &&
+	    !strcmp(given->pattern + given->patternlen - 2, "/*")) {
+		if (!(given->flags & PATTERN_FLAG_NEGATIVE)) {
+			/* Not a cone pattern. */
+			pl->use_cone_patterns = 0;
+			warning(_("unrecognized pattern: '%s'"), given->pattern);
+			goto clear_hashmaps;
+		}
+
+		truncated = xstrdup(given->pattern);
+		truncated[given->patternlen - 2] = 0;
+
+		translated = xmalloc(sizeof(struct pattern_entry));
+		translated->pattern = truncated;
+		translated->patternlen = given->patternlen - 2;
+		hashmap_entry_init(&translated->ent,
+				   memhash(translated->pattern, translated->patternlen));
+
+		if (!hashmap_get_entry(&pl->recursive_hashmap,
+				       translated, ent, NULL)) {
+			/* We did not see the "parent" included */
+			warning(_("unrecognized negative pattern: '%s'"),
+				given->pattern);
+			free(truncated);
+			free(translated);
+			goto clear_hashmaps;
+		}
+
+		hashmap_add(&pl->parent_hashmap, &translated->ent);
+		hashmap_remove(&pl->recursive_hashmap, &translated->ent, &data);
+		free(data);
+		return;
+	}
+
+	if (given->flags & PATTERN_FLAG_NEGATIVE) {
+		warning(_("unrecognized negative pattern: '%s'"),
+			given->pattern);
+		goto clear_hashmaps;
+	}
+
+	translated = xmalloc(sizeof(struct pattern_entry));
+
+	translated->pattern = xstrdup(given->pattern);
+	translated->patternlen = given->patternlen;
+	hashmap_entry_init(&translated->ent,
+			   memhash(translated->pattern, translated->patternlen));
+
+	hashmap_add(&pl->recursive_hashmap, &translated->ent);
+
+	if (hashmap_get_entry(&pl->parent_hashmap, translated, ent, NULL)) {
+		/* we already included this at the parent level */
+		warning(_("your sparse-checkout file may have issues: pattern '%s' is repeated"),
+			given->pattern);
+		hashmap_remove(&pl->parent_hashmap, &translated->ent, &data);
+		free(data);
+		free(translated);
+	}
+
+	return;
+
+clear_hashmaps:
+	warning(_("disabling cone pattern matching"));
+	hashmap_free_entries(&pl->parent_hashmap, struct pattern_entry, ent);
+	hashmap_free_entries(&pl->recursive_hashmap, struct pattern_entry, ent);
+	pl->use_cone_patterns = 0;
+}
+
+static int hashmap_contains_path(struct hashmap *map,
+				 struct strbuf *pattern)
+{
+	struct pattern_entry p;
+
+	/* Check straight mapping */
+	p.pattern = pattern->buf;
+	p.patternlen = pattern->len;
+	hashmap_entry_init(&p.ent, memhash(p.pattern, p.patternlen));
+	return !!hashmap_get_entry(map, &p, ent, NULL);
+}
+
+int hashmap_contains_parent(struct hashmap *map,
+			    const char *path,
+			    struct strbuf *buffer)
+{
+	char *slash_pos;
+
+	strbuf_setlen(buffer, 0);
+
+	if (path[0] != '/')
+		strbuf_addch(buffer, '/');
+
+	strbuf_addstr(buffer, path);
+
+	slash_pos = strrchr(buffer->buf, '/');
+
+	while (slash_pos > buffer->buf) {
+		strbuf_setlen(buffer, slash_pos - buffer->buf);
+
+		if (hashmap_contains_path(map, buffer))
+			return 1;
+
+		slash_pos = strrchr(buffer->buf, '/');
+	}
+
+	return 0;
+}
+
 void add_pattern(const char *string, const char *base,
 		 int baselen, struct pattern_list *pl, int srcpos)
 {

@@ -635,6 +779,8 @@ void add_pattern(const char *string, const char *base,
 	ALLOC_GROW(pl->patterns, pl->nr + 1, pl->alloc);
 	pl->patterns[pl->nr++] = pattern;
 	pattern->pl = pl;
+
+	add_pattern_to_hashsets(pl, pattern);
 }
 
 static int read_skip_worktree_file_from_index(const struct index_state *istate,

@@ -860,6 +1006,9 @@ static int add_patterns_from_buffer(char *buf, size_t size,
 	int i, lineno = 1;
 	char *entry;
 
+	hashmap_init(&pl->recursive_hashmap, pl_hashmap_cmp, NULL, 0);
+	hashmap_init(&pl->parent_hashmap, pl_hashmap_cmp, NULL, 0);
+
 	pl->filebuf = buf;
 
 	if (skip_utf8_bom(&buf, size))

@@ -1096,16 +1245,58 @@ enum pattern_match_result path_matches_pattern_list(
 				struct index_state *istate)
 {
 	struct path_pattern *pattern;
-	pattern = last_matching_pattern_from_list(pathname, pathlen, basename,
-						  dtype, pl, istate);
-	if (pattern) {
-		if (pattern->flags & PATTERN_FLAG_NEGATIVE)
-			return NOT_MATCHED;
-		else
-			return MATCHED;
+	struct strbuf parent_pathname = STRBUF_INIT;
+	int result = NOT_MATCHED;
+	const char *slash_pos;
+
+	if (!pl->use_cone_patterns) {
+		pattern = last_matching_pattern_from_list(pathname, pathlen, basename,
+							dtype, pl, istate);
+		if (pattern) {
+			if (pattern->flags & PATTERN_FLAG_NEGATIVE)
+				return NOT_MATCHED;
+			else
+				return MATCHED;
+		}
+
+		return UNDECIDED;
+	}
+
+	if (pl->full_cone)
+		return MATCHED;
+
+	strbuf_addch(&parent_pathname, '/');
+	strbuf_add(&parent_pathname, pathname, pathlen);
+
+	if (hashmap_contains_path(&pl->recursive_hashmap,
+				  &parent_pathname)) {
+		result = MATCHED;
+		goto done;
+	}
+
+	slash_pos = strrchr(parent_pathname.buf, '/');
+
+	if (slash_pos == parent_pathname.buf) {
+		/* include every file in root */
+		result = MATCHED;
+		goto done;
 	}
 
-	return UNDECIDED;
+	strbuf_setlen(&parent_pathname, slash_pos - parent_pathname.buf);
+
+	if (hashmap_contains_path(&pl->parent_hashmap, &parent_pathname)) {
+		result = MATCHED;
+		goto done;
+	}
+
+	if (hashmap_contains_parent(&pl->recursive_hashmap,
+				    pathname,
+				    &parent_pathname))
+		result = MATCHED;
+
+done:
+	strbuf_release(&parent_pathname);
+	return result;
 }
 
 static struct path_pattern *last_matching_pattern_from_lists(

index 2fbdef014f..f8edbca72b 100644
@@ -4,6 +4,7 @@
 /* See Documentation/technical/api-directory-listing.txt */
 
 #include "cache.h"
+#include "hashmap.h"
 #include "strbuf.h"
 
 struct dir_entry {

@@ -37,6 +38,13 @@ struct path_pattern {
 	int srcpos;
 };
 
+/* used for hashmaps for cone patterns */
+struct pattern_entry {
+	struct hashmap_entry ent;
+	char *pattern;
+	size_t patternlen;
+};
+
 /*
  * Each excludes file will be parsed into a fresh exclude_list which
  * is appended to the relevant exclude_list_group (either EXC_DIRS or

@@ -55,6 +63,26 @@ struct pattern_list {
 	const char *src;
 
 	struct path_pattern **patterns;
+
+	/*
+	 * While scanning the excludes, we attempt to match the patterns
+	 * with a more restricted set that allows us to use hashsets for
+	 * matching logic, which is faster than the linear lookup in the
+	 * excludes array above. If non-zero, that check succeeded.
+	 */
+	unsigned use_cone_patterns;
+	unsigned full_cone;
+
+	/*
+	 * Stores paths where everything starting with those paths
+	 * is included.
+	 */
+	struct hashmap recursive_hashmap;
+
+	/*
+	 * Used to check single-level parents of blobs.
+	 */
+	struct hashmap parent_hashmap;
 };
 
 /*

@@ -271,6 +299,9 @@ int is_excluded(struct dir_struct *dir,
 		struct index_state *istate,
 		const char *name, int *dtype);
 
+int hashmap_contains_parent(struct hashmap *map,
+			    const char *path,
+			    struct strbuf *buffer);
 struct pattern_list *add_pattern_list(struct dir_struct *dir,
 				      int group_type, const char *src);
 int add_patterns_from_file_to_list(const char *fname, const char *base, int baselen,

index 36fda5907b..b0d5aeb33a 100755
@@ -150,7 +150,8 @@ test_expect_success 'set sparse-checkout using --stdin' '
 test_expect_success 'cone mode: match patterns' '
 	git -C repo config --worktree core.sparseCheckoutCone true &&
 	rm -rf repo/a repo/folder1 repo/folder2 &&
-	git -C repo read-tree -mu HEAD &&
+	git -C repo read-tree -mu HEAD 2>err &&
+	test_i18ngrep ! "disabling cone patterns" err &&
 	git -C repo reset --hard &&
 	ls repo >dir  &&
 	cat >expect <<-EOF &&

@@ -161,6 +162,14 @@ test_expect_success 'cone mode: match patterns' '
 	test_cmp expect dir
 '
 
+test_expect_success 'cone mode: warn on bad pattern' '
+	test_when_finished mv sparse-checkout repo/.git/info/ &&
+	cp repo/.git/info/sparse-checkout . &&
+	echo "!/deep/deeper/*" >>repo/.git/info/sparse-checkout &&
+	git -C repo read-tree -mu HEAD 2>err &&
+	test_i18ngrep "unrecognized negative pattern" err
+'
+
 test_expect_success 'sparse-checkout disable' '
 	git -C repo sparse-checkout disable &&
 	test_path_is_missing repo/.git/info/sparse-checkout &&

index 01a05ff66d..a90d71845d 100644
@@ -1482,6 +1482,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options
 		o->skip_sparse_checkout = 1;
 	if (!o->skip_sparse_checkout) {
 		char *sparse = git_pathdup("info/sparse-checkout");
+		pl.use_cone_patterns = core_sparse_checkout_cone;
 		if (add_patterns_from_file_to_list(sparse, "", 0, &pl, NULL) < 0)
 			o->skip_sparse_checkout = 1;
 		else

gitgitgadget

The previous patch added 'core.sparseCheckoutCone', so this last line should be 'core.sparseCheckoutCone=true', shouldn't it?

Good catch! Thanks.

-Stolee

From: Derrick Stolee <[email protected]>

To make the cone pattern set easy to use, update the behavior of 'git sparse-checkout [init|set]'.

Add '--cone' flag to 'git sparse-checkout init' to set the config option 'core.sparseCheckoutCone=true'.

When running 'git sparse-checkout set' in cone mode, a user only needs to supply a list of recursive folder matches. Git will automatically add the necessary parent matches for the leading directories.

When testing 'git sparse-checkout set' in cone mode, check the error stream to ensure we do not see any errors. Specifically, we want to avoid the warning that the patterns do not match the cone-mode patterns.

---
 builtin/sparse-checkout.c          | 159 +++++++++++++++++++++++++++--
 dir.c                              |   8 +-
 dir.h                              |   4 +
 t/t1091-sparse-checkout-builtin.sh |  51 +++++++++
 4 files changed, 208 insertions(+), 14 deletions(-)

index 78a80ce119..95d9a90b84 100644
@@ -6,6 +6,7 @@
 #include "repository.h"
 #include "run-command.h"
 #include "strbuf.h"
+#include "string-list.h"
 
 static char const * const builtin_sparse_checkout_usage[] = {
 	N_("git sparse-checkout [init|list|set|disable] <options>"),

@@ -77,11 +78,13 @@ static int update_working_directory(void)
 enum sparse_checkout_mode {
 	MODE_NO_PATTERNS = 0,
 	MODE_ALL_PATTERNS = 1,
+	MODE_CONE_PATTERNS = 2,
 };
 
 static int sc_set_config(enum sparse_checkout_mode mode)
 {
 	struct argv_array argv = ARGV_ARRAY_INIT;
+	struct argv_array cone_argv = ARGV_ARRAY_INIT;
 
 	if (git_config_set_gently("extensions.worktreeConfig", "true")) {
 		error(_("failed to set extensions.worktreeConfig setting"));

@@ -100,9 +103,31 @@ static int sc_set_config(enum sparse_checkout_mode mode)
 		return 1;
 	}
 
+	argv_array_pushl(&cone_argv, "config", "--worktree",
+			 "core.sparseCheckoutCone", NULL);
+
+	if (mode == MODE_CONE_PATTERNS)
+		argv_array_push(&cone_argv, "true");
+	else
+		argv_array_push(&cone_argv, "false");
+
+	if (run_command_v_opt(cone_argv.argv, RUN_GIT_CMD)) {
+		error(_("failed to enable core.sparseCheckoutCone"));
+		return 1;
+	}
+
 	return 0;
 }
 
+static char const * const builtin_sparse_checkout_init_usage[] = {
+	N_("git sparse-checkout init [--cone]"),
+	NULL
+};
+
+static struct sparse_checkout_init_opts {
+	int cone_mode;
+} init_opts;
+
 static int sparse_checkout_init(int argc, const char **argv)
 {
 	struct pattern_list pl;

@@ -110,8 +135,21 @@ static int sparse_checkout_init(int argc, const char **argv)
 	FILE *fp;
 	int res;
 	struct object_id oid;
+	int mode;
 
-	if (sc_set_config(MODE_ALL_PATTERNS))
+	static struct option builtin_sparse_checkout_init_options[] = {
+		OPT_BOOL(0, "cone", &init_opts.cone_mode,
+			 N_("initialize the sparse-checkout in cone mode")),
+		OPT_END(),
+	};
+
+	argc = parse_options(argc, argv, NULL,
+			     builtin_sparse_checkout_init_options,
+			     builtin_sparse_checkout_init_usage, 0);
+
+	mode = init_opts.cone_mode ? MODE_CONE_PATTERNS : MODE_ALL_PATTERNS;
+
+	if (sc_set_config(mode))
 		return 1;
 
 	memset(&pl, 0, sizeof(pl));

@@ -140,6 +178,70 @@ static int sparse_checkout_init(int argc, const char **argv)
 	return update_working_directory();
 }
 
+static void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *path)
+{
+	struct pattern_entry *e = xmalloc(sizeof(*e));
+	e->patternlen = path->len;
+	e->pattern = strbuf_detach(path, NULL);
+	hashmap_entry_init(&e->ent, memhash(e->pattern, e->patternlen));
+
+	hashmap_add(&pl->recursive_hashmap, &e->ent);
+
+	while (e->patternlen) {
+		char *slash = strrchr(e->pattern, '/');
+		char *oldpattern = e->pattern;
+		size_t newlen;
+
+		if (slash == e->pattern)
+			break;
+
+		newlen = slash - e->pattern;
+		e = xmalloc(sizeof(struct pattern_entry));
+		e->patternlen = newlen;
+		e->pattern = xstrndup(oldpattern, newlen);
+		hashmap_entry_init(&e->ent, memhash(e->pattern, e->patternlen));
+
+		if (!hashmap_get_entry(&pl->parent_hashmap, e, ent, NULL))
+			hashmap_add(&pl->parent_hashmap, &e->ent);
+	}
+}
+
+static void write_cone_to_file(FILE *fp, struct pattern_list *pl)
+{
+	int i;
+	struct pattern_entry *pe;
+	struct hashmap_iter iter;
+	struct string_list sl = STRING_LIST_INIT_DUP;
+
+	hashmap_for_each_entry(&pl->parent_hashmap, &iter, pe, ent)
+		string_list_insert(&sl, pe->pattern);
+
+	string_list_sort(&sl);
+	string_list_remove_duplicates(&sl, 0);
+
+	fprintf(fp, "/*\n!/*/\n");
+
+	for (i = 0; i < sl.nr; i++) {
+		char *pattern = sl.items[i].string;
+
+		if (strlen(pattern))
+			fprintf(fp, "%s/\n!%s/*/\n", pattern, pattern);
+	}
+
+	string_list_clear(&sl, 0);
+
+	hashmap_for_each_entry(&pl->recursive_hashmap, &iter, pe, ent)
+		string_list_insert(&sl, pe->pattern);
+
+	string_list_sort(&sl);
+	string_list_remove_duplicates(&sl, 0);
+
+	for (i = 0; i < sl.nr; i++) {
+		char *pattern = sl.items[i].string;
+		fprintf(fp, "%s/\n", pattern);
+	}
+}
+
 static int write_patterns_and_update(struct pattern_list *pl)
 {
 	char *sparse_filename;

@@ -152,13 +254,33 @@ static int write_patterns_and_update(struct pattern_list *pl)
 
 	sparse_filename = get_sparse_checkout_filename();
 	fp = fopen(sparse_filename, "w");
-	write_patterns_to_file(fp, pl);
+
+	if (core_sparse_checkout_cone)
+		write_cone_to_file(fp, pl);
+	else
+		write_patterns_to_file(fp, pl);
+
 	fclose(fp);
 	free(sparse_filename);
 
 	return update_working_directory();
 }
 
+static void strbuf_to_cone_pattern(struct strbuf *line, struct pattern_list *pl)
+{
+	strbuf_trim(line);
+
+	strbuf_trim_trailing_dir_sep(line);
+
+	if (!line->len)
+		return;
+
+	if (line->buf[0] != '/')
+		strbuf_insert(line, 0, "/", 1);
+
+	insert_recursive_pattern(pl, line);
+}
+
 static char const * const builtin_sparse_checkout_set_usage[] = {
 	N_("git sparse-checkout set [--stdin|<patterns>]"),
 	NULL

@@ -188,17 +310,34 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)
 			     builtin_sparse_checkout_set_usage,
 			     PARSE_OPT_KEEP_UNKNOWN);
 
-	if (set_opts.use_stdin) {
+	if (core_sparse_checkout_cone) {
 		struct strbuf line = STRBUF_INIT;
-
-		while (!strbuf_getline(&line, stdin)) {
-			size_t len;
-			char *buf = strbuf_detach(&line, &len);
-			add_pattern(buf, empty_base, 0, &pl, 0);
+		hashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0);
+		hashmap_init(&pl.parent_hashmap, pl_hashmap_cmp, NULL, 0);
+
+		if (set_opts.use_stdin) {
+			while (!strbuf_getline(&line, stdin))
+				strbuf_to_cone_pattern(&line, &pl);
+		} else {
+			for (i = 0; i < argc; i++) {
+				strbuf_setlen(&line, 0);
+				strbuf_addstr(&line, argv[i]);
+				strbuf_to_cone_pattern(&line, &pl);
+			}
 		}
 	} else {
-		for (i = 0; i < argc; i++)
-			add_pattern(argv[i], empty_base, 0, &pl, 0);
+		if (set_opts.use_stdin) {
+			struct strbuf line = STRBUF_INIT;
+
+			while (!strbuf_getline(&line, stdin)) {
+				size_t len;
+				char *buf = strbuf_detach(&line, &len);
+				add_pattern(buf, empty_base, 0, &pl, 0);
+			}
+		} else {
+			for (i = 0; i < argc; i++)
+				add_pattern(argv[i], empty_base, 0, &pl, 0);
+		}
 	}
 
 	result = write_patterns_and_update(&pl);

index dfabf9982f..35c1ca9e24 100644
@@ -611,10 +611,10 @@ void parse_path_pattern(const char **pattern,
 	*patternlen = len;
 }
 
-static int pl_hashmap_cmp(const void *unused_cmp_data,
-			  const struct hashmap_entry *a,
-			  const struct hashmap_entry *b,
-			  const void *key)
+int pl_hashmap_cmp(const void *unused_cmp_data,
+		   const struct hashmap_entry *a,
+		   const struct hashmap_entry *b,
+		   const void *key)
 {
 	const struct pattern_entry *ee1 =
 			container_of(a, struct pattern_entry, ent);

index f8edbca72b..8e232085cd 100644
@@ -299,6 +299,10 @@ int is_excluded(struct dir_struct *dir,
 		struct index_state *istate,
 		const char *name, int *dtype);
 
+int pl_hashmap_cmp(const void *unused_cmp_data,
+		   const struct hashmap_entry *a,
+		   const struct hashmap_entry *b,
+		   const void *key);
 int hashmap_contains_parent(struct hashmap *map,
 			    const char *path,
 			    struct strbuf *buffer);

gitgitgadget

From: Derrick Stolee <[email protected]>

The sparse-checkout feature in "cone mode" can use the fact that the recursive patterns are "connected" to the root via parent patterns to decide if a directory is entirely contained in the sparse-checkout or entirely removed.

In these cases, we can skip hashing the paths within those directories and simply set the skipworktree bit to the correct value.

---
 dir.c          |  4 ++--
 dir.h          |  1 +
 unpack-trees.c | 38 +++++++++++++++++++++++---------------
 3 files changed, 26 insertions(+), 17 deletions(-)

index 35c1ca9e24..2ef92a50a0 100644
@@ -1270,7 +1270,7 @@ enum pattern_match_result path_matches_pattern_list(
 
 	if (hashmap_contains_path(&pl->recursive_hashmap,
 				  &parent_pathname)) {
-		result = MATCHED;
+		result = MATCHED_RECURSIVE;
 		goto done;
 	}
 

@@ -1292,7 +1292,7 @@ enum pattern_match_result path_matches_pattern_list(
 	if (hashmap_contains_parent(&pl->recursive_hashmap,
 				    pathname,
 				    &parent_pathname))
-		result = MATCHED;
+		result = MATCHED_RECURSIVE;
 
 done:
 	strbuf_release(&parent_pathname);

index 8e232085cd..77a43dbf89 100644
@@ -264,6 +264,7 @@ enum pattern_match_result {
 	UNDECIDED = -1,
 	NOT_MATCHED = 0,
 	MATCHED = 1,
+	MATCHED_RECURSIVE = 2,
 };
 
 /*

index a90d71845d..c0dca20865 100644
@@ -1283,15 +1283,17 @@ static int clear_ce_flags_dir(struct index_state *istate,
 	struct cache_entry **cache_end;
 	int dtype = DT_DIR;
 	int rc;
-	enum pattern_match_result ret;
-	ret = path_matches_pattern_list(prefix->buf, prefix->len,
-					basename, &dtype, pl, istate);
+	enum pattern_match_result ret, orig_ret;
+	orig_ret = path_matches_pattern_list(prefix->buf, prefix->len,
+					     basename, &dtype, pl, istate);
 
 	strbuf_addch(prefix, '/');
 
 	/* If undecided, use matching result of parent dir in defval */
-	if (ret == UNDECIDED)
+	if (orig_ret == UNDECIDED)
 		ret = default_match;
+	else
+		ret = orig_ret;
 
 	for (cache_end = cache; cache_end != cache + nr; cache_end++) {
 		struct cache_entry *ce = *cache_end;

@@ -1299,17 +1301,23 @@ static int clear_ce_flags_dir(struct index_state *istate,
 			break;
 	}
 
-	/*
-	 * TODO: check pl, if there are no patterns that may conflict
-	 * with ret (iow, we know in advance the incl/excl
-	 * decision for the entire directory), clear flag here without
-	 * calling clear_ce_flags_1(). That function will call
-	 * the expensive path_matches_pattern_list() on every entry.
-	 */
-	rc = clear_ce_flags_1(istate, cache, cache_end - cache,
-			      prefix,
-			      select_mask, clear_mask,
-			      pl, ret);
+	if (pl->use_cone_patterns && orig_ret == MATCHED_RECURSIVE) {
+		struct cache_entry **ce = cache;
+		rc = (cache_end - cache) / sizeof(struct cache_entry *);
+
+		while (ce < cache_end) {
+			(*ce)->ce_flags &= ~clear_mask;
+			ce++;
+		}
+	} else if (pl->use_cone_patterns && orig_ret == NOT_MATCHED) {
+		rc = (cache_end - cache) / sizeof(struct cache_entry *);
+	} else {
+		rc = clear_ce_flags_1(istate, cache, cache_end - cache,
+				      prefix,
+				      select_mask, clear_mask,
+				      pl, ret);
+	}
+
 	strbuf_setlen(prefix, prefix->len - 1);
 	return rc;
 }

gitgitgadget

From: Derrick Stolee <[email protected]>

When a large repository has many sparse-checkout patterns, the process for updating the skip-worktree bits can take long enough that a user gets confused why nothing is happening. Update the clear_ce_flags() method to write progress.

---
 cache.h        |  2 ++
 unpack-trees.c | 56 ++++++++++++++++++++++++++++++++++++--------------
 2 files changed, 43 insertions(+), 15 deletions(-)

index 4980ee198e..d3c89e7a53 100644
@@ -304,6 +304,7 @@ static inline unsigned int canon_mode(unsigned int mode)
 
 struct split_index;
 struct untracked_cache;
+struct progress;
 
 struct index_state {
 	struct cache_entry **cache;

@@ -326,6 +327,7 @@ struct index_state {
 	uint64_t fsmonitor_last_update;
 	struct ewah_bitmap *fsmonitor_dirty;
 	struct mem_pool *ce_mem_pool;
+	struct progress *progress;
 };
 
 /* Name hashing */

gitgitgadget

From: Derrick Stolee <[email protected]>

The read-tree builtin has a --verbose option that signals to show progress and other data while updating the index. Update this to be on by default when stderr is a terminal window.

This will help tools like 'git sparse-checkout' to automatically benefit from progress indicators when a user runs these commands.

---
 builtin/read-tree.c | 1 +
 1 file changed, 1 insertion(+)

index ca5e655d2f..69963d83dc 100644
@@ -162,6 +162,7 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)
 	opts.head_idx = -1;
 	opts.src_index = &the_index;
 	opts.dst_index = &the_index;
+	opts.verbose_update = isatty(2);
 
 	git_config(git_read_tree_config, NULL);
 

gitgitgadget

From: Derrick Stolee <[email protected]>

If a user provides folders A/ and A/B/ for inclusion in a cone-mode sparse-checkout file, the parsing logic will notice that A/ appears both as a "parent" type pattern and as a "recursive" type pattern. This is unexpected and hence will complain via a warning and revert to the old logic for checking sparse-checkout patterns.

Prevent this from happening accidentally by sanitizing the folders for this type of inclusion in the 'git sparse-checkout' builtin. This happens in two ways:

1. Do not include any parent patterns that also appear as recursive patterns.

2. Do not include any recursive patterns deeper than other recursive patterns.

In order to minimize duplicate code for scanning parents, create hashmap_contains_parent() method. It takes a strbuf buffer to avoid reallocating a buffer when calling in a tight loop.

---
 builtin/sparse-checkout.c          | 22 ++++++++++++++++++----
 t/t1091-sparse-checkout-builtin.sh | 11 +++++++++++
 2 files changed, 29 insertions(+), 4 deletions(-)

index 95d9a90b84..40b6c599dc 100644
@@ -212,9 +212,17 @@ static void write_cone_to_file(FILE *fp, struct pattern_list *pl)
 	struct pattern_entry *pe;
 	struct hashmap_iter iter;
 	struct string_list sl = STRING_LIST_INIT_DUP;
+	struct strbuf parent_pattern = STRBUF_INIT;
 
-	hashmap_for_each_entry(&pl->parent_hashmap, &iter, pe, ent)
-		string_list_insert(&sl, pe->pattern);
+	hashmap_for_each_entry(&pl->parent_hashmap, &iter, pe, ent) {
+		if (hashmap_get_entry(&pl->recursive_hashmap, pe, ent, NULL))
+			continue;
+
+		if (!hashmap_contains_parent(&pl->recursive_hashmap,
+					     pe->pattern,
+					     &parent_pattern))
+			string_list_insert(&sl, pe->pattern);
+	}
 
 	string_list_sort(&sl);
 	string_list_remove_duplicates(&sl, 0);

@@ -230,8 +238,14 @@ static void write_cone_to_file(FILE *fp, struct pattern_list *pl)
 
 	string_list_clear(&sl, 0);
 
-	hashmap_for_each_entry(&pl->recursive_hashmap, &iter, pe, ent)
-		string_list_insert(&sl, pe->pattern);
+	hashmap_for_each_entry(&pl->recursive_hashmap, &iter, pe, ent) {
+		if (!hashmap_contains_parent(&pl->recursive_hashmap,
+					     pe->pattern,
+					     &parent_pattern))
+			string_list_insert(&sl, pe->pattern);
+	}
+
+	strbuf_release(&parent_pattern);
 
 	string_list_sort(&sl);
 	string_list_remove_duplicates(&sl, 0);

index 3a4c0e6f5d..e752f3ba0f 100755
@@ -236,4 +236,15 @@ test_expect_success 'cone mode: init and set' '
 	test_cmp expect dir
 '
 
+test_expect_success 'cone mode: set with nested folders' '
+	git -C repo sparse-checkout set deep deep/deeper1/deepest 2>err &&
+	test_line_count = 0 err &&
+	cat >expect <<-EOF &&
+		/*
+		!/*/
+		/deep/
+	EOF
+	test_cmp repo/.git/info/sparse-checkout expect
+'
+
 test_done

gitgitgadget

From: Derrick Stolee <[email protected]>

The sparse-checkout builtin used 'git read-tree -mu HEAD' to update the skip-worktree bits in the index and to update the working directory. This extra process is overly complex, and prone to failure. It also requires that we write our changes to the sparse-checkout file before trying to update the index.

Remove this extra process call by creating a direct call to unpack_trees() in the same way 'git read-tree -mu HEAD' does. In addition, provide an in-memory list of patterns so we can avoid reading from the sparse-checkout file. This allows us to test a proposed change to the file before writing to it.

---
 builtin/read-tree.c                |  2 +-
 builtin/sparse-checkout.c          | 80 ++++++++++++++++++++++++++----
 t/t1091-sparse-checkout-builtin.sh | 15 ++++++
 unpack-trees.c                     |  5 +-
 unpack-trees.h                     |  3 +-
 5 files changed, 90 insertions(+), 15 deletions(-)

index 69963d83dc..d7eeaa26ec 100644
@@ -186,7 +186,7 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)
 
 	if (opts.reset || opts.merge || opts.prefix) {
 		if (read_cache_unmerged() && (opts.prefix || opts.merge))
-			die("You need to resolve your current index first");
+			die(_("You need to resolve your current index first"));
 		stage = opts.merge = 1;
 	}
 	resolve_undo_clear();

index e752f3ba0f..ecb5a63671 100755
@@ -247,4 +247,19 @@ test_expect_success 'cone mode: set with nested folders' '
 	test_cmp repo/.git/info/sparse-checkout expect
 '
 
+test_expect_success 'revert to old sparse-checkout on bad update' '
+	echo update >repo/deep/deeper2/a &&
+	cp repo/.git/info/sparse-checkout expect &&
+	test_must_fail git -C repo sparse-checkout set deep/deeper1 2>err &&
+	test_i18ngrep "Cannot update sparse checkout" err &&
+	test_cmp repo/.git/info/sparse-checkout expect &&
+	ls repo/deep >dir &&
+	cat >expect <<-EOF &&
+		a
+		deeper1
+		deeper2
+	EOF
+	test_cmp dir expect
+'
+
 test_done

index 8bb684ad62..3789a22cf0 100644
@@ -1511,7 +1511,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options
 	memset(&pl, 0, sizeof(pl));
 	if (!core_apply_sparse_checkout || !o->update)
 		o->skip_sparse_checkout = 1;
-	if (!o->skip_sparse_checkout) {
+	if (!o->skip_sparse_checkout && !o->pl) {
 		char *sparse = git_pathdup("info/sparse-checkout");
 		pl.use_cone_patterns = core_sparse_checkout_cone;
 		if (add_patterns_from_file_to_list(sparse, "", 0, &pl, NULL) < 0)

@@ -1684,7 +1684,8 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options
 
 done:
 	trace_performance_leave("unpack_trees");
-	clear_pattern_list(&pl);
+	if (!o->keep_pattern_list)
+		clear_pattern_list(&pl);
 	return ret;
 
 return_failed:

index f2eee0c7c5..ca94a421a5 100644
@@ -59,7 +59,8 @@ struct unpack_trees_options {
 		     quiet,
 		     exiting_early,
 		     show_all_errors,
-		     dry_run;
+		     dry_run,
+		     keep_pattern_list;
 	const char *prefix;
 	int cache_bottom;
 	struct dir_struct *dir;

gitgitgadget

Starting with this patch there is an issue with locking the index:

  $ git init
  Initialized empty Git repository in /home/szeder/src/git/tmp/SC/.git/
  $ >file
  $ git commit -m initial
  [master (root-commit) 5d80b9c] initial
   1 file changed, 0 insertions(+), 0 deletions(-)
   create mode 100644 file
  $ ls .git/index.lock
  ls: cannot access '.git/index.lock': No such file or directory
  $ git sparse-checkout set nope
  warning: core.sparseCheckout is disabled, so changes to the sparse-checkout file will have no effect
  warning: run 'git sparse-checkout init' to enable the sparse-checkout feature
  error: Sparse checkout leaves no entry on working directory
  fatal: Unable to create '/home/szeder/src/git/tmp/SC/.git/index.lock':
  File exists.

Another git process seems to be running in this repository, e.g. an editor opened by 'git commit'. Please make sure all processes are terminated then try again. If it still fails, a git process may have crashed in this repository earlier:
remove the file manually to continue.

  $ ls .git/index.lock
  ls: cannot access '.git/index.lock': No such file or directory

  $ git add file

Forgot to copy that command...

I would add that building the previous patch and running the same sequence of commands works, in the sense that 'git sparse-checkout set' writes the non-existing filename to the 'sparse-checkout' file and it prints the same two warnings, and doesn't (seem to) attempt to update the working tree and the index.

Thank you for catching this! The issue is that I was not rolling the index back on this kind of failed update, and then trying to replay the "old" sparse-checkout hit the existing .git/index.lock file.

Here is the test I added that breaks on the current patch, but passes when adding a rollback_lock_file() call:

test_expect_success 'revert to old sparse-checkout on empty update' '

	git init empty-test &&
	(
		echo >file &&
		git add file &&
		git commit -m "test" &&
		test_must_fail git sparse-checkout set nothing 2>err &&
		test_i18ngrep "Sparse checkout leaves no entry on working directory" err &&
		test_i18ngrep ! ".git/index.lock" err &&
		git sparse-checkout set file
	)

'

It also has the following problem: because we are setting the patterns in-memory, this update acts like core.sparseCheckout is enabled, even when it is not. So, I'm finally convinced that the 'set' command should enable the config setting if it is called even without 'init'.

Thanks,
-Stolee

From: Derrick Stolee <[email protected]>

If two 'git sparse-checkout set' subcommands are launched at the same time, the behavior can be unexpected as they compete to write the sparse-checkout file and update the working directory.

Take a lockfile around the writes to the sparse-checkout file. In addition, acquire this lock around the working directory update to avoid two commands updating the working directory in different ways.

---
 builtin/sparse-checkout.c          | 15 ++++++++++++---
 t/t1091-sparse-checkout-builtin.sh |  7 +++++++
 2 files changed, 19 insertions(+), 3 deletions(-)

index 595955d4fc..b319966a79 100644
@@ -305,6 +305,8 @@ static int write_patterns_and_update(struct pattern_list *pl)
 {
 	char *sparse_filename;
 	FILE *fp;
+	int fd;
+	struct lock_file lk = LOCK_INIT;
 	int result;
 
 	if (!core_apply_sparse_checkout) {

@@ -314,21 +316,28 @@ static int write_patterns_and_update(struct pattern_list *pl)
 
 	result = update_working_directory(pl);
 
+	sparse_filename = get_sparse_checkout_filename();
+	fd = hold_lock_file_for_update(&lk, sparse_filename,
+				      LOCK_DIE_ON_ERROR);
+
+	result = update_working_directory(pl);
 	if (result) {
+		rollback_lock_file(&lk);
+		free(sparse_filename);
 		clear_pattern_list(pl);
 		update_working_directory(NULL);
 		return result;
 	}
 
-	sparse_filename = get_sparse_checkout_filename();
-	fp = fopen(sparse_filename, "w");
+	fp = fdopen(fd, "w");
 
 	if (core_sparse_checkout_cone)
 		write_cone_to_file(fp, pl);
 	else
 		write_patterns_to_file(fp, pl);
 
-	fclose(fp);
+	fflush(fp);
+	commit_lock_file(&lk);
 
 	free(sparse_filename);
 	clear_pattern_list(pl);

index ecb5a63671..48493e1012 100755
@@ -262,4 +262,11 @@ test_expect_success 'revert to old sparse-checkout on bad update' '
 	test_cmp dir expect
 '
 
+test_expect_success 'fail when lock is taken' '
+	test_when_finished rm -rf repo/.git/info/sparse-checkout.lock &&
+	touch repo/.git/info/sparse-checkout.lock &&
+	test_must_fail git -C repo sparse-checkout set deep 2>err &&
+	test_i18ngrep "File exists" err
+'
+
 test_done

gitgitgadget

From: Derrick Stolee <[email protected]>

During the development of the sparse-checkout "cone mode" feature, an incorrect placement of the initializer for "use_cone_patterns = 1" caused warnings to show up when a .gitignore file was present with non-cone-mode patterns. This was fixed in the original commit introducing the cone mode, but now we should add a test to avoid hitting this problem again in the future.

---
 t/t1091-sparse-checkout-builtin.sh | 7 +++++++
 1 file changed, 7 insertions(+)

index 48493e1012..14141b8f54 100755
@@ -269,4 +269,11 @@ test_expect_success 'fail when lock is taken' '
 	test_i18ngrep "File exists" err
 '
 
+test_expect_success '.gitignore should not warn about cone mode' '
+	git -C repo config --worktree core.sparseCheckoutCone true &&
+	echo "**/bin/*" >repo/.gitignore &&
+	git -C repo reset --hard 2>err &&
+	test_i18ngrep ! "disabling cone patterns" err
+'
+
 test_done

gitgitgadget

I read through the range-diff, and it all looks good to me other than one issue I flagged on patch 1.

Nice work!

I did a partial review of the v4 patches out of curiosity and I notice in sparse-checkout.c there are a couple of unchecked fopen call sites that could be converted to xfopen to die gracefully upon error, and one unchecked fdopen that can likewise be xfdopen.

-Jon

Thanks! Fixed for v5.

-Stolee

V4 UPDATE: Rebased on latest master to include ew/hashmap and ds/include-exclude in the base.

This series makes the sparse-checkout feature more user-friendly. While there, I also present a way to use a limited set of patterns to gain a significant performance boost in very large repositories.

Sparse-checkout is only documented as a subsection of the read-tree docs [1], which makes the feature hard to discover. Users have trouble navigating the feature, especially at clone time [2], and have even resorted to creating their own helper tools [3].

This series attempts to solve these problems using a new builtin. Here is a sample workflow to give a feeling for how it can work:

In an existing repo:

$ git sparse-checkout init
$ ls
myFile1.txt myFile2.txt
$ git sparse-checkout set "/*" "!/*/" /myFolder/
$ ls
myFile1.txt myFile2.txt myFolder
$ ls myFolder
a.c a.h
$ git sparse-checkout disable
$ ls
hiddenFolder myFile1.txt myFile2.txt myFolder

At clone time:

$ git clone --sparse origin repo
$ cd repo
$ ls
myFile1.txt myFile2.txt
$ git sparse-checkout set "/*" "!/*/" /myFolder/
$ ls
myFile1.txt myFile2.txt myFolder

Here are some more specific details:

* git sparse-checkout init enables core.sparseCheckout and populates the sparse-checkout file with patterns that match only the files at root.

* git clone learns the --sparse argument to run git sparse-checkout init before the first checkout.

* git sparse-checkout set reads patterns from the arguments, or with --stdin reads patterns from stdin one per line, then writes them to the sparse-checkout file and refreshes the working directory.

* git sparse-checkout disable removes the patterns from the sparse-checkout file, disables core.sparseCheckout, and refills the working directory.

* git sparse-checkout list lists the contents of the sparse-checkout file.

The documentation for the sparse-checkout feature can now live primarily with the git-sparse-checkout documentation.

Cone Mode
=========

What really got me interested in this area is a performance problem. If we have N patterns in the sparse-checkout file and M entries in the index, then we can perform up to O(N * M) pattern checks in clear_ce_flags(). This quadratic growth is not sustainable in a repo with 1,000+ patterns and 1,000,000+ index entries.

To solve this problem, I propose a new, more restrictive mode to
sparse-checkout: "cone mode". In this mode, all patterns are based on prefix matches at a directory level. This can then use hashsets for fast performance -- O(M) instead of O(N*M). My hashset implementation is based on the virtual filesystem hook in the VFS for Git custom code [4].

In cone mode, a user specifies a list of folders which the user wants every file inside. In addition, the cone adds all blobs that are siblings of the folders in the directory path to that folder. This makes the directories look "hydrated" as a user drills down to those recursively-closed folders. These directories are called "parent" folders, as a file matches them only if the file's immediate parent is that directory.

When building a prototype of this feature, I used a separate file to contain the list of recursively-closed folders and built the hashsets dynamically based on that file. In this implementation, I tried to maximize the amount of backwards-compatibility by storing all data in the sparse-checkout file using patterns recognized by earlier Git versions.

For example, if we add A/B/C as a recursive folder, then we add the following patterns to the sparse-checkout file:

/*
!/*/
/A/
!/A/*/
/A/B/
!/A/B/*/
/A/B/C/

The alternating positive/negative patterns say "include everything in this folder, but exclude everything another level deeper". The final pattern has no matching negation, so is a recursively closed pattern.

Note that I have some basic warnings to try and check that the sparse-checkout file doesn't match what would be written by a cone-mode add. In such a case, Git writes a warning to stderr and continues with the old pattern matching algorithm. These checks are currently very barebones, and would need to be updated with more robust checks for things like regex characters in the middle of the pattern. As review moves forward (and if we don't change the data storage) then we could spend more time on this.

Thanks, -Stolee

Updates in v2, relative to the RFC:

* Instead of an 'add' subcommand, use a 'set' subcommand. We can consider adding 'add' and/or 'remove' subcommands later.

* 'set' reads from the arguments by default. '--stdin' option is available.

* A new performance-oriented commit is added at the end.

* Patterns no longer end with a trailing asterisk except for the first "/*" pattern.

* References to a "bug" (that was really a strange GVFS interaction in microsoft/git) around deleting outside the cone are removed.

Updates in v3:

* The bad interaction with "cone mode" and .gitignore files is fixed. A test is added in the last patch.

* Several patches are added that make the feature more robust. One sanitizes user input, another few add progress indicators, and another more prevent users from getting in bad states due to working directory changes or concurrent processes.

* Updated several docs and commit messages according to feedback. Thanks, Elijah!

Updates in V4:

* Updated hashmap API usage to respond to ew/hashmap

* Responded to detailed review by Elijah. Thanks!

* Marked the feature as experimental in git-sparse-checkout.txt the same way that git-switch.txt does.

Updates in V5:

* The 'set' subcommand now enables the core.sparseCheckout config setting (unless the checkout fails).

* If the in-process unpack_trees() fails with the new patterns, the index.lock file is rolled back before the replay of the old sparse-checkout patterns.

* Some documentation fixes, f(d)open->xf(d)open calls, and other nits. Thanks everyone!

Things to leave for future patches:

1. Integrate in 'git worktree add' to copy the sparse-checkout file to a worktree-specific file.

2. More robustness around detecting non-cone patterns with wildcards in the middle of the line.

3. 'git clone --sparse-cone' to clone into "cone mode" sparse-checkouts (i.e. set 'core.sparseCheckoutCone=true'). This may not be super-valuable, as it only starts changing behavior when someone calls 'git sparse-checkout set', but may be interesting.

[1] https://git-scm.com/docs/git-read-tree#_sparse_checkoutSparse-checkout
documentation in git-read-tree.

[2] https://stackoverflow.com/a/4909267/127088Is it possible to do a sparse checkout without checking out the whole repository first?

[3] http://www.marcoyuen.com/articles/2016/06/07/git-sparse.htmlA blog post of a user's extra "git-sparse" helper.

[4]
https://github.com/git/git/compare/fc5fd706ff733392053e6180086a4d7f96acc2af...01204f24c5349aa2fb0c474546d768946d315dab
The virtual filesystem hook in microsoft/git.

Derrick Stolee (16):
sparse-checkout: create builtin with 'list' subcommand
sparse-checkout: create 'init' subcommand
clone: add --sparse mode
sparse-checkout: 'set' subcommand
sparse-checkout: add '--stdin' option to set subcommand
sparse-checkout: create 'disable' subcommand
sparse-checkout: add 'cone' mode
sparse-checkout: use hashmaps for cone patterns
sparse-checkout: init and set in cone mode
unpack-trees: hash less in cone mode
unpack-trees: add progress to clear_ce_flags()
read-tree: show progress by default
sparse-checkout: sanitize for nested folders
sparse-checkout: update working directory in-process
sparse-checkout: write using lockfile
sparse-checkout: cone mode should not interact with .gitignore

Jeff Hostetler (1):
trace2: add region in clear_ce_flags

 .gitignore                            |   1 +
 Documentation/config/core.txt         |  10 +-
 Documentation/git-clone.txt           |   8 +-
 Documentation/git-read-tree.txt       |   2 +-
 Documentation/git-sparse-checkout.txt | 153 ++++++++
 Makefile                              |   1 +
 builtin.h                             |   1 +
 builtin/clone.c                       |  27 ++
 builtin/read-tree.c                   |   3 +-
 builtin/sparse-checkout.c             | 488 ++++++++++++++++++++++++++
 cache.h                               |   6 +-
 command-list.txt                      |   1 +
 config.c                              |   5 +
 dir.c                                 | 207 ++++++++++-
 dir.h                                 |  36 ++
 environment.c                         |   1 +
 git.c                                 |   1 +
 t/t1091-sparse-checkout-builtin.sh    | 293 ++++++++++++++++
 unpack-trees.c                        | 110 ++++--
 unpack-trees.h                        |   3 +-
 20 files changed, 1309 insertions(+), 48 deletions(-)
 create mode 100644 Documentation/git-sparse-checkout.txt
 create mode 100644 builtin/sparse-checkout.c
 create mode 100755 t/t1091-sparse-checkout-builtin.sh

base-commit: 108b97dc372828f0e72e56bbb40cae8e1e83ece6
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-316%2Fderrickstolee%2Fsparse-checkout%2Fupstream-v5
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-316/derrickstolee/sparse-checkout/upstream-v5
Pull-Request: https://github.com/gitgitgadget/git/pull/316

Range-diff vs v4:

1: 63626e1097 ! 1: 55c306a73a sparse-checkout: create builtin with 'list' subcommand

     @@ -54,7 +54,7 @@
       +++ b/Documentation/git-sparse-checkout.txt
      @@
      +git-sparse-checkout(1)
     -+=======================
     ++======================
      +
      +NAME
      +----
     @@ -75,7 +75,9 @@
      +Initialize and modify the sparse-checkout configuration, which reduces
      +the checkout to a set of directories given by a list of prefixes.
      +
     -+THIS COMMAND IS EXPERIMENTAL. THE BEHAVIOR MAY CHANGE.
     ++THIS COMMAND IS EXPERIMENTAL. ITS BEHAVIOR, AND THE BEHAVIOR OF OTHER
     ++COMMANDS IN THE PRESENCE OF SPARSE-CHECKOUTS, WILL LIKELY CHANGE IN
     ++THE FUTURE.
      +
      +
      +COMMANDS
     @@ -85,7 +87,7 @@
      +
      +
      +SPARSE CHECKOUT
     -+----------------
     ++---------------
      +
      +"Sparse checkout" allows populating the working directory sparsely.
      +It uses the skip-worktree bit (see linkgit:git-update-index[1]) to tell
     @@ -180,7 +182,7 @@
      +#include "strbuf.h"
      +
      +static char const * const builtin_sparse_checkout_usage[] = {
     -+	N_("git sparse-checkout [list]"),
     ++	N_("git sparse-checkout list"),
      +	NULL
      +};
      +
     @@ -257,6 +259,18 @@
      +			   builtin_sparse_checkout_options);
      +}
      
     + diff --git a/command-list.txt b/command-list.txt
     + --- a/command-list.txt
     + +++ b/command-list.txt
     +@@
     + git-show-ref                            plumbinginterrogators
     + git-sh-i18n                             purehelpers
     + git-sh-setup                            purehelpers
     ++git-sparse-checkout                     mainporcelain           worktree
     + git-stash                               mainporcelain
     + git-stage                                                               complete
     + git-status                              mainporcelain           info
     +
       diff --git a/git.c b/git.c
       --- a/git.c
       +++ b/git.c
     @@ -301,7 +315,7 @@
      +
      +test_expect_success 'git sparse-checkout list (empty)' '
      +	git -C repo sparse-checkout list >list 2>err &&
     -+	test_line_count = 0 list &&
     ++	test_must_be_empty list &&
      +	test_i18ngrep "this worktree is not sparse (sparse-checkout file may not exist)" err
      +'
      +
     @@ -313,13 +327,8 @@
      +		**/a
      +		!*bin*
      +	EOF
     ++	cp repo/.git/info/sparse-checkout expect &&
      +	git -C repo sparse-checkout list >list &&
     -+	cat >expect <<-EOF &&
     -+		/folder1/*
     -+		/deep/
     -+		**/a
     -+		!*bin*
     -+	EOF
      +	test_cmp expect list
      +'
      +
  2:  65d26de1c2 !  2:  a161cee0df sparse-checkout: create 'init' subcommand
     @@ -37,7 +37,7 @@
      +`core.sparseCheckout` setting in the worktree-specific config file.
       
       SPARSE CHECKOUT
     - ----------------
     + ---------------
      
       diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c
       --- a/builtin/sparse-checkout.c
     @@ -46,8 +46,8 @@
       #include "strbuf.h"
       
       static char const * const builtin_sparse_checkout_usage[] = {
     --	N_("git sparse-checkout [list]"),
     -+	N_("git sparse-checkout [init|list]"),
     +-	N_("git sparse-checkout list"),
     ++	N_("git sparse-checkout (init|list)"),
       	NULL
       };
       
     @@ -121,7 +121,7 @@
      +	}
      +
      +	/* initial mode: all blobs at root */
     -+	fp = fopen(sparse_filename, "w");
     ++	fp = xfopen(sparse_filename, "w");
      +	free(sparse_filename);
      +	fprintf(fp, "/*\n!/*/\n");
      +	fclose(fp);
  3:  e59ed7128f =  3:  cd150853b7 clone: add --sparse mode
  4:  502b4b08f0 !  4:  7d9d66a89f sparse-checkout: 'set' subcommand
     @@ -11,6 +11,11 @@
          extracted from cmd_sparse_checkout() to make it easier to implement
          'add' and/or 'remove' subcommands in the future.
      
     +    If the core.sparseCheckout config setting is disabled, then enable
     +    the config setting in the worktree config. If we set the config
     +    this way and the sparse-checkout fails, then re-disable the config
     +    setting.
     +
          Signed-off-by: Derrick Stolee <[email protected]>
      
       diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt
     @@ -23,10 +28,11 @@
      +'set'::
      +	Write a set of patterns to the sparse-checkout file, as given as
      +	a list of arguments following the 'set' subcommand. Update the
     -+	working directory to match the new patterns.
     ++	working directory to match the new patterns. Enable the
     ++	core.sparseCheckout config setting if it is not already enabled.
      +
       SPARSE CHECKOUT
     - ----------------
     + ---------------
       
      
       diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c
     @@ -36,8 +42,8 @@
       #include "strbuf.h"
       
       static char const * const builtin_sparse_checkout_usage[] = {
     --	N_("git sparse-checkout [init|list]"),
     -+	N_("git sparse-checkout [init|list|set] <options>"),
     +-	N_("git sparse-checkout (init|list)"),
     ++	N_("git sparse-checkout (init|list|set) <options>"),
       	NULL
       };
       
     @@ -65,13 +71,23 @@
      +	int i;
      +	struct pattern_list pl;
      +	int result;
     ++	int set_config = 0;
      +	memset(&pl, 0, sizeof(pl));
      +
      +	for (i = 1; i < argc; i++)
      +		add_pattern(argv[i], empty_base, 0, &pl, 0);
      +
     ++	if (!core_apply_sparse_checkout) {
     ++		sc_set_config(MODE_ALL_PATTERNS);
     ++		core_apply_sparse_checkout = 1;
     ++		set_config = 1;
     ++	}
     ++
      +	result = write_patterns_and_update(&pl);
      +
     ++	if (result && set_config)
     ++		sc_set_config(MODE_NO_PATTERNS);
     ++
      +	clear_pattern_list(&pl);
      +	return result;
      +}
     @@ -96,6 +112,19 @@
       	test_cmp expect dir
       '
       
     ++test_expect_success 'set enables config' '
     ++	git init empty-config &&
     ++	(
     ++		cd empty-config &&
     ++		test_commit test file &&
     ++		test_path_is_missing .git/config.worktree &&
     ++		test_must_fail git sparse-checkout set nothing &&
     ++		test_i18ngrep "sparseCheckout = false" .git/config.worktree &&
     ++		git sparse-checkout set "/*" &&
     ++		test_i18ngrep "sparseCheckout = true" .git/config.worktree
     ++	)
     ++'
     ++
      +test_expect_success 'set sparse-checkout using builtin' '
      +	git -C repo sparse-checkout set "/*" "!/*/" "*folder*" &&
      +	cat >expect <<-EOF &&
  5:  2852cf8e11 !  5:  0e08898dcb sparse-checkout: add '--stdin' option to set subcommand
     @@ -14,18 +14,6 @@
       diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c
       --- a/builtin/sparse-checkout.c
       +++ b/builtin/sparse-checkout.c
     -@@
     - 	char *sparse_filename;
     - 	FILE *fp;
     - 
     -+	if (!core_apply_sparse_checkout) {
     -+		warning(_("core.sparseCheckout is disabled, so changes to the sparse-checkout file will have no effect"));
     -+		warning(_("run 'git sparse-checkout init' to enable the sparse-checkout feature"));
     -+	}
     -+
     - 	sparse_filename = get_sparse_checkout_filename();
     - 	fp = fopen(sparse_filename, "w");
     - 	write_patterns_to_file(fp, pl);
      @@
       	return update_working_directory();
       }
     @@ -42,9 +30,10 @@
       static int sparse_checkout_set(int argc, const char **argv, const char *prefix)
       {
       	static const char *empty_base = "";
     - 	int i;
     +@@
       	struct pattern_list pl;
       	int result;
     + 	int set_config = 0;
      +
      +	static struct option builtin_sparse_checkout_set_options[] = {
      +		OPT_BOOL(0, "stdin", &set_opts.use_stdin,
     @@ -74,8 +63,8 @@
      +			add_pattern(argv[i], empty_base, 0, &pl, 0);
      +	}
       
     - 	result = write_patterns_and_update(&pl);
     - 
     + 	if (!core_apply_sparse_checkout) {
     + 		sc_set_config(MODE_ALL_PATTERNS);
      
       diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh
       --- a/t/t1091-sparse-checkout-builtin.sh
     @@ -84,20 +73,6 @@
       	test_cmp expect dir
       '
       
     -+test_expect_success 'warn if core.sparseCheckout is disabled' '
     -+	test_when_finished git -C repo config --worktree core.sparseCheckout true &&
     -+	git -C repo config --worktree core.sparseCheckout false &&
     -+	git -C repo sparse-checkout set folder1 2>err &&
     -+	test_i18ngrep "core.sparseCheckout is disabled" err
     -+'
     -+
     - test_expect_success 'set sparse-checkout using builtin' '
     - 	git -C repo sparse-checkout set "/*" "!/*/" "*folder*" &&
     - 	cat >expect <<-EOF &&
     -@@
     - 	test_cmp expect dir
     - '
     - 
      +test_expect_success 'set sparse-checkout using --stdin' '
      +	cat >expect <<-EOF &&
      +		/*
  6:  55f95f290e !  6:  c48535cd5c sparse-checkout: create 'disable' subcommand
     @@ -13,15 +13,15 @@
       --- a/Documentation/git-sparse-checkout.txt
       +++ b/Documentation/git-sparse-checkout.txt
      @@
     - 	a list of arguments following the 'set' subcommand. Update the
     - 	working directory to match the new patterns.
     + 	working directory to match the new patterns. Enable the
     + 	core.sparseCheckout config setting if it is not already enabled.
       
      +'disable'::
      +	Remove the sparse-checkout file, set `core.sparseCheckout` to
      +	`false`, and restore the working directory to include all files.
      +
       SPARSE CHECKOUT
     - ----------------
     + ---------------
       
      @@
       on this file. The files matching the patterns in the file will
     @@ -68,8 +68,8 @@
       #include "strbuf.h"
       
       static char const * const builtin_sparse_checkout_usage[] = {
     --	N_("git sparse-checkout [init|list|set] <options>"),
     -+	N_("git sparse-checkout [init|list|set|disable] <options>"),
     +-	N_("git sparse-checkout (init|list|set) <options>"),
     ++	N_("git sparse-checkout (init|list|set|disable) <options>"),
       	NULL
       };
       
     @@ -86,7 +86,7 @@
      +		die(_("failed to change config"));
      +
      +	sparse_filename = get_sparse_checkout_filename();
     -+	fp = fopen(sparse_filename, "w");
     ++	fp = xfopen(sparse_filename, "w");
      +	fprintf(fp, "/*\n");
      +	fclose(fp);
      +
  7:  3d8f2f2007 =  7:  44b0245e65 trace2: add region in clear_ce_flags
  8:  03dc0ed716 =  8:  121d878882 sparse-checkout: add 'cone' mode
  9:  28606a152c !  9:  ff2a59bfff sparse-checkout: use hashmaps for cone patterns

@@ -22,9 +22,9 @@
Running 'git read-tree -mu HEAD' on this file had the following performance:

- core.sparseCheckout=false: 0.21 s (0.00 s)
- core.sparseCheckout=true: 3.75 s (3.50 s)
- core.sparseCheckout=cone: 0.23 s (0.01 s)
+ core.sparseCheckout=false: 0.21 s (0.00 s)
+ core.sparseCheckout=true: 3.75 s (3.50 s)
+ core.sparseCheckoutCone=true: 0.23 s (0.01 s)

The times in parentheses above correspond to the time spent in the first clear_ce_flags() call, according to the trace2
10: 2ea3d8819b ! 10: 0258ee8026 sparse-checkout: init and set in cone mode

     @@ -32,7 +32,7 @@
      +#include "string-list.h"
       
       static char const * const builtin_sparse_checkout_usage[] = {
     - 	N_("git sparse-checkout [init|list|set|disable] <options>"),
     + 	N_("git sparse-checkout (init|list|set|disable) <options>"),
      @@
       enum sparse_checkout_mode {
       	MODE_NO_PATTERNS = 0,
     @@ -250,7 +250,7 @@
      +		}
       	}
       
     - 	result = write_patterns_and_update(&pl);
     + 	if (!core_apply_sparse_checkout) {
      
       diff --git a/dir.c b/dir.c
       --- a/dir.c
 11:  fefd1e1744 = 11:  7b5c5bad2d unpack-trees: hash less in cone mode
 12:  034fbd31bd = 12:  386f4f8eb5 unpack-trees: add progress to clear_ce_flags()
 13:  27aa9d22f0 = 13:  a229e1ee0c read-tree: show progress by default
 14:  44c3078029 = 14:  f03bb23305 sparse-checkout: sanitize for nested folders
 15:  9ccec3ca9a ! 15:  d7af75672b sparse-checkout: update working directory in-process
     @@ -14,6 +14,12 @@
          reading from the sparse-checkout file. This allows us to test a
          proposed change to the file before writing to it.
      
     +    An earlier version of this patch included a bug when the 'set' command
     +    failed due to the "Sparse checkout leaves no entry on working directory"
     +    error. It would not rollback the index.lock file, so the replay of the
     +    old sparse-checkout specification would fail. A test in t1091 now
     +    covers that scenario.
     +
          Signed-off-by: Derrick Stolee <[email protected]>
      
       diff --git a/builtin/read-tree.c b/builtin/read-tree.c
     @@ -43,7 +49,7 @@
      +#include "unpack-trees.h"
       
       static char const * const builtin_sparse_checkout_usage[] = {
     - 	N_("git sparse-checkout [init|list|set|disable] <options>"),
     + 	N_("git sparse-checkout (init|list|set|disable) <options>"),
      @@
       	return 0;
       }
     @@ -64,6 +70,7 @@
      -	if (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {
      -		error(_("failed to update index with new sparse-checkout paths"));
      -		result = 1;
     +-	}
      +	if (repo_read_index_unmerged(r))
      +		die(_("You need to resolve your current index first"));
      +
     @@ -99,7 +106,8 @@
      +	if (!result) {
      +		prime_cache_tree(r, r->index, tree);
      +		write_locked_index(r->index, &lock_file, COMMIT_LOCK);
     - 	}
     ++	} else
     ++		rollback_lock_file(&lock_file);
       
      -	argv_array_clear(&argv);
       	return result;
     @@ -133,12 +141,7 @@
       	char *sparse_filename;
       	FILE *fp;
      +	int result;
     - 
     - 	if (!core_apply_sparse_checkout) {
     - 		warning(_("core.sparseCheckout is disabled, so changes to the sparse-checkout file will have no effect"));
     - 		warning(_("run 'git sparse-checkout init' to enable the sparse-checkout feature"));
     - 	}
     - 
     ++
      +	result = update_working_directory(pl);
      +
      +	if (result) {
     @@ -146,10 +149,9 @@
      +		update_working_directory(NULL);
      +		return result;
      +	}
     -+
     + 
       	sparse_filename = get_sparse_checkout_filename();
       	fp = fopen(sparse_filename, "w");
     - 
      @@
       		write_patterns_to_file(fp, pl);
       
     @@ -203,6 +205,19 @@
      +	EOF
      +	test_cmp dir expect
      +'
     ++
     ++test_expect_success 'revert to old sparse-checkout on empty update' '
     ++	git init empty-test &&
     ++	(
     ++		echo >file &&
     ++		git add file &&
     ++		git commit -m "test" &&
     ++		test_must_fail git sparse-checkout set nothing 2>err &&
     ++		test_i18ngrep "Sparse checkout leaves no entry on working directory" err &&
     ++		test_i18ngrep ! ".git/index.lock" err &&
     ++		git sparse-checkout set file
     ++	)
     ++'
      +
       test_done

16: d0421ef7b2 ! 16: 88eff318e0 sparse-checkout: write using lockfile

     @@ -24,9 +24,6 @@
      +	struct lock_file lk = LOCK_INIT;
       	int result;
       
     - 	if (!core_apply_sparse_checkout) {
     -@@
     - 
       	result = update_working_directory(pl);
       
      +	sparse_filename = get_sparse_checkout_filename();
     @@ -44,7 +41,7 @@
       
      -	sparse_filename = get_sparse_checkout_filename();
      -	fp = fopen(sparse_filename, "w");
     -+	fp = fdopen(fd, "w");
     ++	fp = xfdopen(fd, "w");
       
       	if (core_sparse_checkout_cone)
       		write_cone_to_file(fp, pl);
     @@ -62,7 +59,7 @@
       --- a/t/t1091-sparse-checkout-builtin.sh
       +++ b/t/t1091-sparse-checkout-builtin.sh
      @@
     - 	test_cmp dir expect
     + 	)
       '
       
      +test_expect_success 'fail when lock is taken' '

17: ed1f148763 = 17: ab20c11d71 sparse-checkout: cone mode should not interact with .gitignore

gitgitgadget

From: Derrick Stolee <[email protected]>

The sparse-checkout feature is mostly hidden to users, as its only documentation is supplementary information in the docs for 'git read-tree'. In addition, users need to know how to edit the .git/info/sparse-checkout file with the right patterns, then run the appropriate 'git read-tree -mu HEAD' command. Keeping the working directory in sync with the sparse-checkout file requires care.

Begin an effort to make the sparse-checkout feature a porcelain feature by creating a new 'git sparse-checkout' builtin. This builtin will be the preferred mechanism for manipulating the sparse-checkout file and syncing the working directory.

The documentation provided is adapted from the "git read-tree" documentation with a few edits for clarity in the new context. Extra sections are added to hint toward a future change to a more restricted pattern set.

---
 .gitignore                            |  1 +
 Documentation/git-read-tree.txt       |  2 +-
 Documentation/git-sparse-checkout.txt | 89 +++++++++++++++++++++++++++
 Makefile                              |  1 +
 builtin.h                             |  1 +
 builtin/sparse-checkout.c             | 86 ++++++++++++++++++++++++++
 command-list.txt                      |  1 +
 git.c                                 |  1 +
 t/t1091-sparse-checkout-builtin.sh    | 45 ++++++++++++++
 9 files changed, 226 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/git-sparse-checkout.txt
 create mode 100644 builtin/sparse-checkout.c
 create mode 100755 t/t1091-sparse-checkout-builtin.sh

index 89b3b79c1a..aebe7c0908 100644
@@ -158,6 +158,7 @@
 /git-show-branch
 /git-show-index
 /git-show-ref
+/git-sparse-checkout
 /git-stage
 /git-stash
 /git-status

index d271842608..da33f84f33 100644
@@ -436,7 +436,7 @@ support.
 SEE ALSO
 --------
 linkgit:git-write-tree[1]; linkgit:git-ls-files[1];
-linkgit:gitignore[5]
+linkgit:gitignore[5]; linkgit:git-sparse-checkout[1];
 
 GIT
 ---

index de60c8e7aa..adefc229fe 100644
@@ -1125,6 +1125,7 @@ BUILTIN_OBJS += builtin/shortlog.o
 BUILTIN_OBJS += builtin/show-branch.o
 BUILTIN_OBJS += builtin/show-index.o
 BUILTIN_OBJS += builtin/show-ref.o
+BUILTIN_OBJS += builtin/sparse-checkout.o
 BUILTIN_OBJS += builtin/stash.o
 BUILTIN_OBJS += builtin/stripspace.o
 BUILTIN_OBJS += builtin/submodule--helper.o

index 5cf5df69f7..2b25a80cde 100644
@@ -225,6 +225,7 @@ int cmd_shortlog(int argc, const char **argv, const char *prefix);
 int cmd_show(int argc, const char **argv, const char *prefix);
 int cmd_show_branch(int argc, const char **argv, const char *prefix);
 int cmd_show_index(int argc, const char **argv, const char *prefix);
+int cmd_sparse_checkout(int argc, const char **argv, const char *prefix);
 int cmd_status(int argc, const char **argv, const char *prefix);
 int cmd_stash(int argc, const char **argv, const char *prefix);
 int cmd_stripspace(int argc, const char **argv, const char *prefix);

index a9ac72bef4..d3d28252b3 100644
@@ -166,6 +166,7 @@ git-show-index                          plumbinginterrogators
 git-show-ref                            plumbinginterrogators
 git-sh-i18n                             purehelpers
 git-sh-setup                            purehelpers
+git-sparse-checkout                     mainporcelain           worktree
 git-stash                               mainporcelain
 git-stage                                                               complete
 git-status                              mainporcelain           info

index ce6ab0ece2..7be7ad34bd 100644
@@ -572,6 +572,7 @@ static struct cmd_struct commands[] = {
 	{ "show-branch", cmd_show_branch, RUN_SETUP },
 	{ "show-index", cmd_show_index },
 	{ "show-ref", cmd_show_ref, RUN_SETUP },
+	{ "sparse-checkout", cmd_sparse_checkout, RUN_SETUP | NEED_WORK_TREE },
 	{ "stage", cmd_add, RUN_SETUP | NEED_WORK_TREE },
 	/*
 	 * NEEDSWORK: Until the builtin stash is thoroughly robust and no

gitgitgadget

From: Derrick Stolee <[email protected]>

Getting started with a sparse-checkout file can be daunting. Help users start their sparse enlistment using 'git sparse-checkout init'. This will set 'core.sparseCheckout=true' in their config, write an initial set of patterns to the sparse-checkout file, and update their working directory.

Make sure to use the extensions.worktreeConfig setting and write the sparse checkout config to the worktree-specific config file. This avoids confusing interactions with other worktrees.

The use of running another process for 'git read-tree' is sub- optimal. This will be removed in a later change.

---
 Documentation/git-sparse-checkout.txt | 11 ++++
 builtin/sparse-checkout.c             | 79 ++++++++++++++++++++++++++-
 t/t1091-sparse-checkout-builtin.sh    | 41 ++++++++++++++
 3 files changed, 130 insertions(+), 1 deletion(-)

index 9d6ca22917..930a361567 100644
@@ -30,6 +30,17 @@ COMMANDS
 'list'::
 	Provide a list of the contents in the sparse-checkout file.
 
+'init'::
+	Enable the `core.sparseCheckout` setting. If the
+	sparse-checkout file does not exist, then populate it with
+	patterns that match every file in the root directory and
+	no other directories, then will remove all directories tracked
+	by Git. Add patterns to the sparse-checkout file to
+	repopulate the working directory.
++
+To avoid interfering with other worktrees, it first enables the
+`extensions.worktreeConfig` setting and makes sure to set the
+`core.sparseCheckout` setting in the worktree-specific config file.
 
 SPARSE CHECKOUT
 ---------------

gitgitgadget

Reading this I was wandering what those "initial set of patterns" might be ...

                                "core.sparseCheckout",
                                mode ? "true" : "false")

Having slept on it, I think it would be even better to pass NULL instead of "false". "false", well, just sets the configuration variable to false, but a NULL would properly unset it.

Enabling sparse-checkout can remove modified files:

  $ mkdir dir
  $ touch foo dir/bar
  $ git add .
  $ git commit -m Initial
  [master (root-commit) ecc81bd] Initial
   2 files changed, 0 insertions(+), 0 deletions(-)
   create mode 100644 dir/bar
   create mode 100644 foo
  $ echo changes >dir/bar
  $ ~/src/git/git sparse-checkout init
  error: Entry 'dir/bar' not uptodate. Cannot update sparse checkout.
  error: failed to update index with new sparse-checkout paths
  $ cat dir/bar 
  changes

So far so good, my changes are still there. Unfortunately, however:

  $ git add -u
  $ ~/src/git/git sparse-checkout init
  $ echo $?
  0
  $ ls
  foo

Uh-oh, my changes are gone.

And after this it leaves the 'sparse-checkout' file behind, which may or may not be desired:

  $ cat .git/info/sparse-checkout 
  /*
  !/*/

Here, your changes are not "lost", they are staged, correct? A "git status" should report that your changes are ready for committing. This seems to be the expected behavior.

As to your other message about leaving the sparse-checkout file even when the 'init' command fails, I'll fix that by using the in-process mechanism.

-Stolee

Well, yes, the changes were staged, so they must be in the object database somewhere, the user just has to go through the dangling objects reported by 'git fsck' to find and restore it... So from the perspective of an ordinary user they are lost.

No, unfortunately enabling sparse-checkout did throw the staged changes away:

~/test (master +)$ ~/src/git/git sparse-checkout init ~/test (master)$ git status
On branch master
nothing to commit, working tree clean

Note also the shell prompt going from "you have staged changes" to "working tree is clean".

But wait, I thought that only changes to files that are excluded from the sparse-checkout are thrown away, but as it turns out it throws away changes to files that are included in the sparse-checkout:

~/test (master #)$ echo original >foo
~/test (master #%)$ git add .
~/test (master +)$ git commit -m Initial
[master (root-commit) 04cb2a2] Initial

   1 file changed, 1 insertion(+)

create mode 100644 foo
~/test (master)$ echo changes >foo
~/test (master *)$ ~/src/git/git sparse-checkout init ~/test (master *)$ cat foo
changes

So far so good, but:

~/test (master *)$ ~/src/git/git sparse-checkout disable ~/test (master *)$ git add .
~/test (master +)$ ~/src/git/git sparse-checkout init ~/test (master)$ cat foo
original

This is not really sparse-checkout-specific, because this is what 'git read-tree -um HEAD' always does on its own:

~/test (master)$ echo changes >foo
~/test (master *)$ git read-tree -um HEAD
~/test (master *)$ cat foo
changes
~/test (master *)$ git add -u
~/test (master +)$ git read-tree -um HEAD
~/test (master)$ cat foo
original

These issues are present at the end of the patch series as well, but that is sort-of expected, because the later commit message states that:

Remove this extra process call by creating a direct call to unpack_trees() in the same way 'git read-tree -mu HEAD' does.

Thanks for the additional details.

This series intended to make the existing sparse-checkout behavior more useful to users by not requiring manual edits of the sparse-checkout file followed by 'git read-tree' commands. However, there do appear to be some serious improvements that we can make in the future.

Keeping staged changes seems important, and we can address that in the near future.

-Stolee

For completeness, 'git sparse-checkout disable' throws away staged changes as well, as it, too, runs 'git read-tree -um HEAD' (or its equivalent).

I think that at least for now 'git sparse-checkout' should flat out refuse to init/set/disable if the working tree is not clean (but still allow 'list', as that's a read-only operation), like the patch below. Yeah, that way it wouldn't work in cases that appear to be safe (unstaged changes), but it would prevent the data loss until we carefully consider the circumstances under which these operations can be safely allowed.

-- >8 --

index 76f65d8edd..4b05625c4c 100644
@@ -12,6 +12,7 @@
 #include "lockfile.h"
 #include "resolve-undo.h"
 #include "unpack-trees.h"
+#include "wt-status.h"
 
 static char const * const builtin_sparse_checkout_usage[] = {
 	N_("git sparse-checkout (init|list|set|disable) <options>"),

@@ -188,6 +189,10 @@ static int sparse_checkout_init(int argc, const char **argv)
 			     builtin_sparse_checkout_init_options,
 			     builtin_sparse_checkout_init_usage, 0);
 
+	repo_read_index(the_repository);
+	require_clean_work_tree(the_repository,
+				N_("initialize sparse-checkout"), NULL, 1, 0);
+
 	if (init_opts.cone_mode) {
 		mode = MODE_CONE_PATTERNS;
 		core_sparse_checkout_cone = 1;

@@ -386,6 +391,10 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)
 			     builtin_sparse_checkout_set_usage,
 			     PARSE_OPT_KEEP_UNKNOWN);
 
+	repo_read_index(the_repository);
+	require_clean_work_tree(the_repository,
+				N_("set up sparse-checkout"), NULL, 1, 0);
+
 	if (core_sparse_checkout_cone) {
 		struct strbuf line = STRBUF_INIT;
 		hashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0);

@@ -437,6 +446,10 @@ static int sparse_checkout_disable(int argc, const char **argv)
 	char *sparse_filename;
 	FILE *fp;
 
+	repo_read_index(the_repository);
+	require_clean_work_tree(the_repository,
+				N_("disable sparse-checkout"), NULL, 1, 0);
+
 	if (sc_set_config(MODE_ALL_PATTERNS))
 		die(_("failed to change config"));
 

A big +1 for this from me.

We had an unfortunately large number of mis-merging and dataloss bugs in the merge machinery that were slowly fixed over the course of more than a decade[1], due to the fact that builtin/merge required index==HEAD and did so by placing a comment in the code notifying folks that the individual merge strategies were responsible to enforce it -- and, in practice, they *all* forgot to do so unless and until we discovered bugs. So, count me as a strongly in favor of just preventatively enforcing safe conditions and then later relaxing them in special conditions if it can be proven safe.

[1] https://public-inbox.org/git/[email protected]/

Sounds good. Thanks, both.

-Stolee

From: Derrick Stolee <[email protected]>

When someone wants to clone a large repository, but plans to work using a sparse-checkout file, they either need to do a full checkout first and then reduce the patterns they included, or clone with --no-checkout, set up their patterns, and then run a checkout manually. This requires knowing a lot about the repo shape and how sparse-checkout works.

Add a new '--sparse' option to 'git clone' that initializes the sparse-checkout file to include the following patterns:

	/*
	!/*/

These patterns include every file in the root directory, but no directories. This allows a repo to include files like a README or a bootstrapping script to grow enlistments from that point.

During the 'git sparse-checkout init' call, we must first look to see if HEAD is valid, since 'git clone' does not have a valid HEAD at the point where it initializes the sparse-checkout. The following checkout within the clone command will create the HEAD ref and update the working directory correctly.

---
 Documentation/git-clone.txt        |  8 +++++++-
 builtin/clone.c                    | 27 +++++++++++++++++++++++++++
 builtin/sparse-checkout.c          |  6 ++++++
 t/t1091-sparse-checkout-builtin.sh | 13 +++++++++++++
 4 files changed, 53 insertions(+), 1 deletion(-)

index 34011c2940..0fe91d2f04 100644
@@ -15,7 +15,7 @@ SYNOPSIS
 	  [--dissociate] [--separate-git-dir <git dir>]
 	  [--depth <depth>] [--[no-]single-branch] [--no-tags]
 	  [--recurse-submodules[=<pathspec>]] [--[no-]shallow-submodules]
-	  [--[no-]remote-submodules] [--jobs <n>] [--] <repository>
+	  [--[no-]remote-submodules] [--jobs <n>] [--sparse] [--] <repository>
 	  [<directory>]
 
 DESCRIPTION

@@ -156,6 +156,12 @@ objects from the source repository into a pack in the cloned repository.
 	used, neither remote-tracking branches nor the related
 	configuration variables are created.
 
+--sparse::
+	Initialize the sparse-checkout file so the working
+	directory starts with only the files in the root
+	of the repository. The sparse-checkout file can be
+	modified to grow the working directory as needed.
+
 --mirror::
 	Set up a mirror of the source repository.  This implies `--bare`.
 	Compared to `--bare`, `--mirror` not only maps local branches of the

index c46ee29f0a..4348d962c9 100644
@@ -59,6 +59,7 @@ static const char *real_git_dir;
 static char *option_upload_pack = "git-upload-pack";
 static int option_verbosity;
 static int option_progress = -1;
+static int option_sparse_checkout;
 static enum transport_family family;
 static struct string_list option_config = STRING_LIST_INIT_NODUP;
 static struct string_list option_required_reference = STRING_LIST_INIT_NODUP;

@@ -146,6 +147,8 @@ static struct option builtin_clone_options[] = {
 	OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
 	OPT_BOOL(0, "remote-submodules", &option_remote_submodules,
 		    N_("any cloned submodules will use their remote-tracking branch")),
+	OPT_BOOL(0, "sparse", &option_sparse_checkout,
+		    N_("initialize sparse-checkout file to include only files at root")),
 	OPT_END()
 };
 

@@ -733,6 +736,27 @@ static void update_head(const struct ref *our, const struct ref *remote,
 	}
 }
 
+static int git_sparse_checkout_init(const char *repo)
+{
+	struct argv_array argv = ARGV_ARRAY_INIT;
+	int result = 0;
+	argv_array_pushl(&argv, "-C", repo, "sparse-checkout", "init", NULL);
+
+	/*
+	 * We must apply the setting in the current process
+	 * for the later checkout to use the sparse-checkout file.
+	 */
+	core_apply_sparse_checkout = 1;
+
+	if (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {
+		error(_("failed to initialize sparse-checkout"));
+		result = 1;
+	}
+
+	argv_array_clear(&argv);
+	return result;
+}
+
 static int checkout(int submodule_progress)
 {
 	struct object_id oid;

@@ -1106,6 +1130,9 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 	if (option_required_reference.nr || option_optional_reference.nr)
 		setup_reference();
 
+	if (option_sparse_checkout && git_sparse_checkout_init(repo))
+		return 1;
+
 	remote = remote_get(option_origin);
 
 	strbuf_addf(&default_refspec, "+%s*:%s*", src_ref_prefix,

index 77aa52ca01..6c336b7ab3 100644
@@ -109,6 +109,7 @@ static int sparse_checkout_init(int argc, const char **argv)
 	char *sparse_filename;
 	FILE *fp;
 	int res;
+	struct object_id oid;
 
 	if (sc_set_config(MODE_ALL_PATTERNS))
 		return 1;

@@ -130,6 +131,11 @@ static int sparse_checkout_init(int argc, const char **argv)
 	fprintf(fp, "/*\n!/*/\n");
 	fclose(fp);
 
+	if (get_oid("HEAD", &oid)) {
+		/* assume we are in a fresh repo */
+		return 0;
+	}
+
 reset_dir:
 	return update_working_directory();
 }

index cd56cc384b..cb74715ca6 100755
@@ -83,4 +83,17 @@ test_expect_success 'init with existing sparse-checkout' '
 	test_cmp expect dir
 '
 
+test_expect_success 'clone --sparse' '
+	git clone --sparse repo clone &&
+	git -C clone sparse-checkout list >actual &&
+	cat >expect <<-EOF &&
+		/*
+		!/*/
+	EOF
+	test_cmp expect actual &&
+	ls clone >dir &&
+	echo a >expect &&
+	test_cmp expect dir
+'
+
 test_done

gitgitgadget

From: Derrick Stolee <[email protected]>

The 'git sparse-checkout set' subcommand takes a list of patterns as arguments and writes them to the sparse-checkout file. Then, it updates the working directory using 'git read-tree -mu HEAD'.

The 'set' subcommand will replace the entire contents of the sparse-checkout file. The write_patterns_and_update() method is extracted from cmd_sparse_checkout() to make it easier to implement 'add' and/or 'remove' subcommands in the future.

If the core.sparseCheckout config setting is disabled, then enable the config setting in the worktree config. If we set the config this way and the sparse-checkout fails, then re-disable the config setting.

---
 Documentation/git-sparse-checkout.txt |  6 ++++
 builtin/sparse-checkout.c             | 45 ++++++++++++++++++++++++++-
 t/t1091-sparse-checkout-builtin.sh    | 32 +++++++++++++++++++
 3 files changed, 82 insertions(+), 1 deletion(-)

index 930a361567..b933043b3d 100644
@@ -42,6 +42,12 @@ To avoid interfering with other worktrees, it first enables the
 `extensions.worktreeConfig` setting and makes sure to set the
 `core.sparseCheckout` setting in the worktree-specific config file.
 
+'set'::
+	Write a set of patterns to the sparse-checkout file, as given as
+	a list of arguments following the 'set' subcommand. Update the
+	working directory to match the new patterns. Enable the
+	core.sparseCheckout config setting if it is not already enabled.
+
 SPARSE CHECKOUT
 ---------------
 

index cb74715ca6..bf2dc55bb1 100755
@@ -96,4 +96,36 @@ test_expect_success 'clone --sparse' '
 	test_cmp expect dir
 '
 
+test_expect_success 'set enables config' '
+	git init empty-config &&
+	(
+		cd empty-config &&
+		test_commit test file &&
+		test_path_is_missing .git/config.worktree &&
+		test_must_fail git sparse-checkout set nothing &&
+		test_i18ngrep "sparseCheckout = false" .git/config.worktree &&
+		git sparse-checkout set "/*" &&
+		test_i18ngrep "sparseCheckout = true" .git/config.worktree
+	)
+'
+
+test_expect_success 'set sparse-checkout using builtin' '
+	git -C repo sparse-checkout set "/*" "!/*/" "*folder*" &&
+	cat >expect <<-EOF &&
+		/*
+		!/*/
+		*folder*
+	EOF
+	git -C repo sparse-checkout list >actual &&
+	test_cmp expect actual &&
+	test_cmp expect repo/.git/info/sparse-checkout &&
+	ls repo >dir  &&
+	cat >expect <<-EOF &&
+		a
+		folder1
+		folder2
+	EOF
+	test_cmp expect dir
+'
+
 test_done

gitgitgadget

Yeah, an 'add' subcommand would be great, because 'set' throwing away all the existing patterns can lead to some frustration:

# Let's clone with sparse checkout

  $ git clone --sparse ...
  $ cd clone
  $ less README
  $ git sparse-checkout '/some/dir/*'
  # Erm, what was the next step?
  $ less README
  README: No such file or directory
  # Uhoh.

Having said that, being forced to use 'git sparse-checkout set' and specify all patterns at once does have its own benefits: I needed like 6 subdirectories to build that subset of a big project that I was interested in, and now all the necessary patterns are saved in my Bash history, and I will be able to easily dig out the complete command in the future. That wouldn't work with using the 'add' subcommand to add one pattern at a time.

I plan to extend this feature when this series lands.

In general, this "set" command is created first because tools can interact with it more easily than "add" and "remove". Users would probably prefer the "add" and "remove" way to interact.

At this commit, I do not see two identical lines, but instead the second line says

error: failed to update index with new sparse-checkout paths

(that "paths" should probably be "patterns")

Thanks,
-Stolee

Makes sense.

However, I'd like to add that in the meantime I got to like the 'set' subcommand more and more. I enabled-disabled sparse checkout a lot of times while testing and trying to poke holes, and to be able to set up everything with only one single command that I can easily recall from the shell's history was a great help.

You're right, I must have looked at and copied the results from a test run of the full patch series or even from 'next'. The error message changes with commit efd9c53b6d (sparse-checkout: update working directory in-process, 2019-10-21) to what I shown above.

Interestingly, at one point in my testing I managed to get a different error three times:

  $ ~/src/git/git sparse-checkout set foo
  error: Entry 'dir/bar' not uptodate. Cannot update sparse checkout.
  error: Entry 'dir/bar' not uptodate. Cannot update sparse checkout.
  error: Entry 'dir/bar' not uptodate. Cannot update sparse checkout.

However, when I later re-run the same sequence of commands that lead to that three errors I got that error only twice, and couldn't reproduce it since.

Style nit, seeing "error: Sparse..." and "error: Entry...": error messages should start with a lower-case letter.

That would be better indeed, but that error message goes away by the end of the series anyway...

From: Derrick Stolee <[email protected]>

The 'git sparse-checkout set' subcommand takes a list of patterns and places them in the sparse-checkout file. Then, it updates the working directory to match those patterns. For a large list of patterns, the command-line call can get very cumbersome.

Add a '--stdin' option to instead read patterns over standard in.

---
 builtin/sparse-checkout.c          | 35 ++++++++++++++++++++++++++++--
 t/t1091-sparse-checkout-builtin.sh | 20 +++++++++++++++++
 2 files changed, 53 insertions(+), 2 deletions(-)

index 834ee421f0..f2e2bd772d 100644
@@ -154,6 +154,15 @@ static int write_patterns_and_update(struct pattern_list *pl)
 	return update_working_directory();
 }
 
+static char const * const builtin_sparse_checkout_set_usage[] = {
+	N_("git sparse-checkout set [--stdin|<patterns>]"),
+	NULL
+};
+
+static struct sparse_checkout_set_opts {
+	int use_stdin;
+} set_opts;
+
 static int sparse_checkout_set(int argc, const char **argv, const char *prefix)
 {
 	static const char *empty_base = "";

@@ -161,10 +170,32 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)
 	struct pattern_list pl;
 	int result;
 	int set_config = 0;
+
+	static struct option builtin_sparse_checkout_set_options[] = {
+		OPT_BOOL(0, "stdin", &set_opts.use_stdin,
+			 N_("read patterns from standard in")),
+		OPT_END(),
+	};
+
 	memset(&pl, 0, sizeof(pl));
 
-	for (i = 1; i < argc; i++)
-		add_pattern(argv[i], empty_base, 0, &pl, 0);
+	argc = parse_options(argc, argv, prefix,
+			     builtin_sparse_checkout_set_options,
+			     builtin_sparse_checkout_set_usage,
+			     PARSE_OPT_KEEP_UNKNOWN);
+
+	if (set_opts.use_stdin) {
+		struct strbuf line = STRBUF_INIT;
+
+		while (!strbuf_getline(&line, stdin)) {
+			size_t len;
+			char *buf = strbuf_detach(&line, &len);
+			add_pattern(buf, empty_base, 0, &pl, 0);
+		}
+	} else {
+		for (i = 0; i < argc; i++)
+			add_pattern(argv[i], empty_base, 0, &pl, 0);
+	}
 
 	if (!core_apply_sparse_checkout) {
 		sc_set_config(MODE_ALL_PATTERNS);

index bf2dc55bb1..a9ff5eb9ec 100755
@@ -128,4 +128,24 @@ test_expect_success 'set sparse-checkout using builtin' '
 	test_cmp expect dir
 '
 
+test_expect_success 'set sparse-checkout using --stdin' '
+	cat >expect <<-EOF &&
+		/*
+		!/*/
+		/folder1/
+		/folder2/
+	EOF
+	git -C repo sparse-checkout set --stdin <expect &&
+	git -C repo sparse-checkout list >actual &&
+	test_cmp expect actual &&
+	test_cmp expect repo/.git/info/sparse-checkout &&
+	ls repo >dir  &&
+	cat >expect <<-EOF &&
+		a
+		folder1
+		folder2
+	EOF
+	test_cmp expect dir
+'
+
 test_done

gitgitgadget

No documentation update.

  $ git sparse-checkout set
  error: Sparse checkout leaves no entry on working directory
  error: Sparse checkout leaves no entry on working directory
  $ echo $?
  255

  $ touch foo bar $'foo\nbar'
  $ git add .
  $ git commit -m 'filename with newline'
  [master (root-commit) 5cd7369] filename with newline
   3 files changed, 0 insertions(+), 0 deletions(-)
   create mode 100644 bar
   create mode 100644 foo
   create mode 100644 "foo\nbar"
  $ git sparse-checkout set foo
  $ ls
  foo
  $ git sparse-checkout set 'foo*'
  $ ls
  foo  foo?bar
  $ git sparse-checkout set $'foo\nbar'
  $ ls
  foo?bar
  # Looks good so far, but...
  $ cat .git/info/sparse-checkout 
  foo
  bar
  $ git read-tree -um HEAD
  $ ls
  bar  foo
  # Not so good after all.
  # Let's try to hand-edit the sparse-checkout file.
  $ echo $'"foo\\nbar"' >.git/info/sparse-checkout 
  $ git read-tree -um HEAD
  error: Sparse checkout leaves no entry on working directory
  $ echo $'foo\\nbar'

  $ git read-tree -um HEAD
  error: Sparse checkout leaves no entry on working directory
  $ echo $'foo\\\nbar'

  $ git read-tree -um HEAD
  $ ls
  bar
  # OK, I give up :)

So, it seems that the sparse-checkout file format doesn't support paths/patterns containing a newline character (or if it does, I couldn't figure out how), and thus there is no use for a '-z' option.

However, as shown above a newline in a pattern given as parameter eventually leads to undesired behavior, so I think 'git sparse-checkout set $'foo\nbar' should error out upfront.

  $ echo README | git sparse-checkout set --stdin Makefile
  $ echo $?
  0
  $ find . -name README |wc -l
  51
  $ find . -name Makefile |wc -l
  0

I will add this to the list of things to do as a follow-up. There are lots of ways users can abuse the command-line right now, especially in cone mode.

The goal right now is to get the typical usage figured out, then we can look for atypical cases (such as newlines in filenames, which...who even does that?).

I'll add this to the list, too.

Thanks,
-Stolee

From: Derrick Stolee <[email protected]>

The instructions for disabling a sparse-checkout to a full working directory are complicated and non-intuitive. Add a subcommand, 'git sparse-checkout disable', to perform those steps for the user.

---
 Documentation/git-sparse-checkout.txt | 27 ++++++++++++---------------
 builtin/sparse-checkout.c             | 26 +++++++++++++++++++++++++-
 t/t1091-sparse-checkout-builtin.sh    | 15 +++++++++++++++
 3 files changed, 52 insertions(+), 16 deletions(-)

index b933043b3d..f794d4797a 100644
@@ -48,6 +48,10 @@ To avoid interfering with other worktrees, it first enables the
 	working directory to match the new patterns. Enable the
 	core.sparseCheckout config setting if it is not already enabled.
 
+'disable'::
+	Remove the sparse-checkout file, set `core.sparseCheckout` to
+	`false`, and restore the working directory to include all files.
+
 SPARSE CHECKOUT
 ---------------
 

@@ -65,6 +69,14 @@ directory, it updates the skip-worktree bits in the index based
 on this file. The files matching the patterns in the file will
 appear in the working directory, and the rest will not.
 
+To enable the sparse-checkout feature, run `git sparse-checkout init` to
+initialize a simple sparse-checkout file and enable the `core.sparseCheckout`
+config setting. Then, run `git sparse-checkout set` to modify the patterns in
+the sparse-checkout file.
+
+To repopulate the working directory with all files, use the
+`git sparse-checkout disable` command.
+
 ## FULL PATTERN SET
 
 By default, the sparse-checkout file uses the same syntax as `.gitignore`