Sparse-checkout: modify 'git add', 'git rm', and 'git add' behavior

As requested, this series looks to update the behavior of git add, git rm, and git mv when they attempt to modify paths outside of the sparse-checkout cone. In particular, this care is expanded to not just cache entries with the SKIP_WORKTREE bit, but also paths that do not match the sparse-checkout definition.

This means that commands that worked before this series can now fail. In particular, if 'git merge' results in a conflict outside of the sparse-checkout cone, then 'git add ' will now fail.

In order to allow users to circumvent these protections, a new '--sparse' option is added that ignores the sparse-checkout patterns and the SKIP_WORKTREE bit. The message for advice.updateSparsePath is adjusted to assist with discovery of this option.

There is a subtle issue with git mv in that it does not check the index until it discovers a directory and then uses the index to find the contained entries. This means that in non-cone-mode patterns, a pattern such as "sub/dir" will not match the path "sub" and this can cause an issue.

In order to allow for checking arbitrary paths against the sparse-checkout patterns, some changes to the underlying pattern matching code is required. It turns out that there are some bugs in the methods as advertised, but these bugs were never discovered because of the way methods like unpack_trees() will check a directory for a pattern match before checking its contained paths. Our new "check patterns on-demand" approach pokes holes in that approach, specifically with patterns that match entire directories.

I'm open to alternative designs here, too.

The patches are based on v2 of the integration with merge, cherry-pick, and rebase. I'm sending the RFC really early in case it needs to be significantly reworked.

Thanks, -Stolee

Derrick Stolee (13):
t1092: behavior for adding sparse files
dir: extract directory-matching logic
dir: select directories correctly
dir: fix pattern matching on dirs
add: fail when adding an untracked sparse file
add: skip paths that are outside sparse-checkout cone
add: implement the --sparse option
add: prevent adding sparse conflict files
rm: add --sparse option
rm: skip sparse paths with missing SKIP_WORKTREE
mv: refuse to move sparse paths
mv: add '--sparse' option to ignore sparse-checkout
advice: update message to suggest '--sparse'

 Documentation/git-add.txt                |   9 +-
 Documentation/git-rm.txt                 |   6 ++
 advice.c                                 |   3 +-
 builtin/add.c                            |  22 ++++-
 builtin/mv.c                             |  39 ++++++--
 builtin/rm.c                             |  10 +-
 dir.c                                    |  54 ++++++++--
 pathspec.c                               |   5 +-
 t/t1091-sparse-checkout-builtin.sh       |   4 +-
 t/t1092-sparse-checkout-compatibility.sh |  47 ++++++---
 t/t3602-rm-sparse-checkout.sh            |  27 ++++-
 t/t3705-add-sparse-checkout.sh           |  10 +-
 t/t7002-mv-sparse-checkout.sh            | 121 +++++++++++++++++++++++
 13 files changed, 316 insertions(+), 41 deletions(-)
 create mode 100755 t/t7002-mv-sparse-checkout.sh

base-commit: df4bbec744f5cd4a060082212d95a36b812fa50b
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1018%2Fderrickstolee%2Fsparse-index%2Fadd-rm-mv-behavior-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1018/derrickstolee/sparse-index/add-rm-mv-behavior-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1018

gitgitgadget

From: Derrick Stolee <[email protected]>

Add some tests to demonstrate the current behavior around adding files outside of the sparse-checkout cone. Currently, untracked files are handled differently from tracked files. A futuer change will make these cases be handled the same way.

---
 t/t1092-sparse-checkout-compatibility.sh | 12 ++++++++++++
 1 file changed, 12 insertions(+)

index bbc6de712c4..23bee918260 100755
@@ -291,6 +291,18 @@ test_expect_success 'add, commit, checkout' '
 	test_all_match git checkout -
 '
 
+# NEEDSWORK: This documents current behavior, but is not a desirable
+# behavior (untracked files are handled differently than tracked).
+test_expect_success 'add outside sparse cone' '
+	init_repos &&
+
+	run_on_sparse mkdir folder1 &&
+	run_on_sparse ../edit-contents folder1/a &&
+	run_on_sparse ../edit-contents folder1/newfile &&
+	test_sparse_match test_must_fail git add folder1/a &&
+	test_sparse_match git add folder1/newfile
+'
+
 test_expect_success 'commit including unstaged changes' '
 	init_repos &&
 

gitgitgadget

From: Derrick Stolee <[email protected]>

The last_matching_pattern_from_list() logic performs some checks on the filetype of a path within the index when the PATTERN_FLAG_MUSTBEDIR flag is set. This works great when setting SKIP_WORKTREE bits within unpack_trees(), but doesn't work well when passing an arbitrary path such as a file within a matching directory.

This change only rearranges the logic but does not change its functionality.

We will expand the path_matches_dir_pattern() method in a following change.

---
 dir.c | 22 +++++++++++++++++-----
 1 file changed, 17 insertions(+), 5 deletions(-)

index 86afa2eae00..652135df896 100644
@@ -1303,6 +1303,19 @@ int match_pathname(const char *pathname, int pathlen,
 				 WM_PATHNAME) == 0;
 }
 
+static int path_matches_dir_pattern(const char *pathname,
+				    int pathlen,
+				    int *dtype,
+				    struct path_pattern *pattern,
+				    struct index_state *istate)
+{
+	*dtype = resolve_dtype(*dtype, istate, pathname, pathlen);
+	if (*dtype != DT_DIR)
+		return 0;
+
+	return 1;
+}
+
 /*
  * Scan the given exclude list in reverse to see whether pathname
  * should be ignored.  The first match (i.e. the last on the list), if

@@ -1327,11 +1340,10 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname
 		const char *exclude = pattern->pattern;
 		int prefix = pattern->nowildcardlen;
 
-		if (pattern->flags & PATTERN_FLAG_MUSTBEDIR) {
-			*dtype = resolve_dtype(*dtype, istate, pathname, pathlen);
-			if (*dtype != DT_DIR)
-				continue;
-		}
+		if ((pattern->flags & PATTERN_FLAG_MUSTBEDIR) &&
+		    !path_matches_dir_pattern(pathname, pathlen,
+					      dtype, pattern, istate))
+			continue;
 
 		if (pattern->flags & PATTERN_FLAG_NODIR) {
 			if (match_basename(basename,

gitgitgadget

From: Derrick Stolee <[email protected]>

When matching a path against a list of patterns, the ones that require a directory match previously did not work when a filename is specified. This was fine when all pattern-matching was done within methods such as unpack_trees() that check a directory before recursing into the contained files. However, other commands will start matching individual files against pattern lists without that recursive approach.

We modify path_matches_dir_pattern() to take a strbuf 'path_parent' that is used to store the parent directory of 'pathname' between multiple pattern matching tests. This is loaded lazily, only on the first pattern it finds that has the PATTERN_FLAG_MUSTBEDIR flag.

If we find that a path has a parent directory, we start by checking to see if that parent directory matches the pattern. If so, then we do not need to query the index for the type (which can be expensive). If we find that the parent does not match, then we still must check the type from the index for the given pathname.

Note that this does not affect cone mode pattern matching, but instead the more general -- and slower -- full pattern set. Thus, this does not affect the sparse index.

---
 dir.c | 34 ++++++++++++++++++++++++++++++++--
 1 file changed, 32 insertions(+), 2 deletions(-)

index 652135df896..fe5ee87bb5f 100644
@@ -1305,10 +1305,38 @@ int match_pathname(const char *pathname, int pathlen,
 
 static int path_matches_dir_pattern(const char *pathname,
 				    int pathlen,
+				    struct strbuf *path_parent,
 				    int *dtype,
 				    struct path_pattern *pattern,
 				    struct index_state *istate)
 {
+	/*
+	 * Use 'alloc' as an indicator that the string has not been
+	 * initialized, in case the parent is the root directory.
+	 */
+	if (!path_parent->alloc) {
+		char *slash;
+		strbuf_addstr(path_parent, pathname);
+		slash = find_last_dir_sep(path_parent->buf);
+
+		if (slash)
+			*slash = '\0';
+		else
+			strbuf_setlen(path_parent, 0);
+	}
+
+	/*
+	 * If the parent directory matches the pattern, then we do not
+	 * need to check for dtype.
+	 */
+	if (path_parent->len &&
+	    match_pathname(path_parent->buf, path_parent->len,
+			   pattern->base,
+			   pattern->baselen ? pattern->baselen - 1 : 0,
+			   pattern->pattern, pattern->nowildcardlen,
+			   pattern->patternlen, pattern->flags))
+		return 1;
+
 	*dtype = resolve_dtype(*dtype, istate, pathname, pathlen);
 	if (*dtype != DT_DIR)
 		return 0;

@@ -1331,6 +1359,7 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname
 {
 	struct path_pattern *res = NULL; /* undecided */
 	int i;
+	struct strbuf path_parent = STRBUF_INIT;
 
 	if (!pl->nr)
 		return NULL;	/* undefined */

@@ -1340,8 +1369,8 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname
 		const char *exclude = pattern->pattern;
 		int prefix = pattern->nowildcardlen;
 
-		if ((pattern->flags & PATTERN_FLAG_MUSTBEDIR) &&
-		    !path_matches_dir_pattern(pathname, pathlen,
+		if (pattern->flags & PATTERN_FLAG_MUSTBEDIR &&
+		    !path_matches_dir_pattern(pathname, pathlen, &path_parent,
 					      dtype, pattern, istate))
 			continue;
 

@@ -1367,6 +1396,7 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname
 			break;
 		}
 	}
+	strbuf_release(&path_parent);
 	return res;
 }
 

gitgitgadget

From: Derrick Stolee <[email protected]>

Within match_pathname(), one successful matching category happens when the pattern is equal to its non-wildcard prefix. At this point, we have checked that the input 'pathname' matches the pattern up to the prefix length, and then we subtraced that length from both 'patternlen' and 'namelen'.

In the case of a directory match, this prefix match should be sufficient. However, the success condition only cared about _exact_ equality here. Instead, we should allow any path that agrees on this prefix in the case of PATTERN_FLAG_MUSTBEDIR.

This case was not tested before because of the way unpack_trees() would match a parent directory before visiting the contained paths. This approach is changing, so we must change this comparison.

---
 dir.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index fe5ee87bb5f..80fd0ad2fc0 100644
@@ -1294,7 +1294,7 @@ int match_pathname(const char *pathname, int pathlen,
 		 * then our prefix match is all we need; we
 		 * do not need to call fnmatch at all.
 		 */
-		if (!patternlen && !namelen)
+		if (!patternlen && (!namelen || (flags & PATTERN_FLAG_MUSTBEDIR)))
 			return 1;
 	}
 

gitgitgadget

From: Derrick Stolee <[email protected]>

The add_files() method in builtin/add.c takes a set of untracked files that are being added by the input pathspec and inserts them into the index. If these files are outside of the sparse-checkout cone, then they gain the SKIP_WORKTREE bit at some point. However, this was not checked before inserting into the index, so these files are added even though we want to avoid modifying the index outside of the sparse-checkout cone.

Add a check within add_files() for these files and write the advice about files outside of the sprase-checkout cone.

This behavior change modifies some existing tests within t1092. These tests intended to document how a user could interact with the existing behavior in place. Many of these tests need to be marked as expecting failure. A future change will allow these tests to pass by adding a flag to 'git add' that allows users to modify index entries outside of the sparse-checkout cone.

The 'submodule handling' test is intended to document what happens to directories that contain a submodule when the sparse index is enabled. It is not trying to say that users should be able to add submodules outside of the sparse-checkout cone, so that test can be modified to avoid that operation.

---
 builtin/add.c                            | 14 ++++++++++
 t/t1092-sparse-checkout-compatibility.sh | 33 +++++++++++++++++-------
 2 files changed, 38 insertions(+), 9 deletions(-)

index 88a6c0c69fb..3a109276b74 100644
@@ -443,6 +443,7 @@ static void check_embedded_repo(const char *path)
 static int add_files(struct dir_struct *dir, int flags)
 {
 	int i, exit_status = 0;
+	struct string_list only_match_skip_worktree = STRING_LIST_INIT_NODUP;
 
 	if (dir->ignored_nr) {
 		fprintf(stderr, _(ignore_error));

@@ -456,6 +457,11 @@ static int add_files(struct dir_struct *dir, int flags)
 	}
 
 	for (i = 0; i < dir->nr; i++) {
+		if (!path_in_sparse_checkout(dir->entries[i]->name, &the_index)) {
+			string_list_append(&only_match_skip_worktree,
+					   dir->entries[i]->name);
+			continue;
+		}
 		if (add_file_to_index(&the_index, dir->entries[i]->name, flags)) {
 			if (!ignore_add_errors)
 				die(_("adding files failed"));

@@ -464,6 +470,14 @@ static int add_files(struct dir_struct *dir, int flags)
 			check_embedded_repo(dir->entries[i]->name);
 		}
 	}
+
+	if (only_match_skip_worktree.nr) {
+		advise_on_updating_sparse_paths(&only_match_skip_worktree);
+		exit_status = 1;
+	}
+
+	string_list_clear(&only_match_skip_worktree, 0);
+
 	return exit_status;
 }
 

index 23bee918260..962bece03e1 100755
@@ -291,8 +291,6 @@ test_expect_success 'add, commit, checkout' '
 	test_all_match git checkout -
 '
 
-# NEEDSWORK: This documents current behavior, but is not a desirable
-# behavior (untracked files are handled differently than tracked).
 test_expect_success 'add outside sparse cone' '
 	init_repos &&
 

@@ -300,7 +298,7 @@ test_expect_success 'add outside sparse cone' '
 	run_on_sparse ../edit-contents folder1/a &&
 	run_on_sparse ../edit-contents folder1/newfile &&
 	test_sparse_match test_must_fail git add folder1/a &&
-	test_sparse_match git add folder1/newfile
+	test_sparse_match test_must_fail git add folder1/newfile
 '
 
 test_expect_success 'commit including unstaged changes' '

@@ -331,7 +329,11 @@ test_expect_success 'commit including unstaged changes' '
 	test_all_match git status --porcelain=v2
 '
 
-test_expect_success 'status/add: outside sparse cone' '
+# NEEDSWORK: Now that 'git add folder1/new' fails, the changes being
+# attempted here fail for the sparse-checkout and sparse-index repos.
+# We must enable a way for adding files outside the sparse-checkout
+# done, even if it is by an optional flag.
+test_expect_failure 'status/add: outside sparse cone' '
 	init_repos &&
 
 	# folder1 is at HEAD, but outside the sparse cone

@@ -352,10 +354,9 @@ test_expect_success 'status/add: outside sparse cone' '
 	# Adding the path outside of the sparse-checkout cone should fail.
 	test_sparse_match test_must_fail git add folder1/a &&
 	test_sparse_match test_must_fail git add --refresh folder1/a &&
+	test_sparse_match test_must_fail git add folder1/new &&
 
-	# NEEDSWORK: Adding a newly-tracked file outside the cone succeeds
-	test_sparse_match git add folder1/new &&
-
+	# NEEDSWORK: behavior begins to deviate here.
 	test_all_match git add . &&
 	test_all_match git status --porcelain=v2 &&
 	test_all_match git commit -m folder1/new &&

@@ -511,7 +512,7 @@ test_expect_success 'merge, cherry-pick, and rebase' '
 # Right now, users might be using this flow to work through conflicts,
 # so any solution should present advice to users who try this sequence
 # of commands to follow whatever new method we create.
-test_expect_success 'merge with conflict outside cone' '
+test_expect_failure 'merge with conflict outside cone' '
 	init_repos &&
 
 	test_all_match git checkout -b merge-tip merge-left &&

@@ -525,12 +526,18 @@ test_expect_success 'merge with conflict outside cone' '
 	test_all_match git status --porcelain=v2 &&
 
 	# 2. Add the file with conflict markers
+	# NEEDSWORK: Even though the merge conflict removed the
+	# SKIP_WORKTREE bit from the index entry for folder1/a, we should
+	# warn that this is a problematic add.
 	test_all_match git add folder1/a &&
 	test_all_match git status --porcelain=v2 &&
 
 	# 3. Rename the file to another sparse filename and
 	#    accept conflict markers as resolved content.
 	run_on_all mv folder2/a folder2/z &&
+	# NEEDSWORK: This mode now fails, because folder2/z is
+	# outside of the sparse-checkout cone and does not match an
+	# existing index entry with the SKIP_WORKTREE bit cleared.
 	test_all_match git add folder2 &&
 	test_all_match git status --porcelain=v2 &&
 

@@ -539,7 +546,7 @@ test_expect_success 'merge with conflict outside cone' '
 	test_all_match git rev-parse HEAD^{tree}
 '
 
-test_expect_success 'cherry-pick/rebase with conflict outside cone' '
+test_expect_failure 'cherry-pick/rebase with conflict outside cone' '
 	init_repos &&
 
 	for OPERATION in cherry-pick rebase

@@ -556,11 +563,17 @@ test_expect_success 'cherry-pick/rebase with conflict outside cone' '
 		test_all_match git status --porcelain=v2 &&
 
 		# 2. Add the file with conflict markers
+		# NEEDSWORK: Even though the merge conflict removed the
+		# SKIP_WORKTREE bit from the index entry for folder1/a, we should
+		# warn that this is a problematic add.
 		test_all_match git add folder1/a &&
 		test_all_match git status --porcelain=v2 &&
 
 		# 3. Rename the file to another sparse filename and
 		#    accept conflict markers as resolved content.
+		# NEEDSWORK: This mode now fails, because folder2/z is
+		# outside of the sparse-checkout cone and does not match an
+		# existing index entry with the SKIP_WORKTREE bit cleared.
 		run_on_all mv folder2/a folder2/z &&
 		test_all_match git add folder2 &&
 		test_all_match git status --porcelain=v2 &&

@@ -638,6 +651,7 @@ test_expect_success 'clean' '
 test_expect_success 'submodule handling' '
 	init_repos &&
 
+	test_sparse_match git sparse-checkout add modules &&
 	test_all_match mkdir modules &&
 	test_all_match touch modules/a &&
 	test_all_match git add modules &&

@@ -647,6 +661,7 @@ test_expect_success 'submodule handling' '
 	test_all_match git commit -m "add submodule" &&
 
 	# having a submodule prevents "modules" from collapse
+	test_sparse_match git sparse-checkout set deep/deeper1 &&
 	test-tool -C sparse-index read-cache --table >cache &&
 	grep "100644 blob .*	modules/a" cache &&
 	grep "160000 commit $(git -C initial-repo rev-parse HEAD)	modules/sub" cache

gitgitgadget

From: Derrick Stolee <[email protected]>

When 'git add' adds a tracked file that is outside of the sparse-checkout cone, it checks the SKIP_WORKTREE bit to see if the file exists outside of the sparse-checkout cone. This is usually correct, except in the case of a merge conflict outside of the cone.

Modify add_pathspec_matched_against_index() to be more careful about pathes by checking the sparse-checkout patterns in addition to the SKIP_WORKTREE bit. This causes 'git add' to no longer allow files outside of the cone that removed the SKIP_WORKTREE bit due to a merge conflict.

With only this change, users will only be able to add the file after adding the file to the sparse-checkout cone. A later change will allow users to force adding even though the file is outside of the sparse-checkout cone.

---
 pathspec.c                               | 3 ++-
 t/t1092-sparse-checkout-compatibility.sh | 8 ++++----
 2 files changed, 6 insertions(+), 5 deletions(-)

index 44306fdaca2..0e6e60fdc5a 100644
@@ -39,7 +39,8 @@ void add_pathspec_matches_against_index(const struct pathspec *pathspec,
 		return;
 	for (i = 0; i < istate->cache_nr; i++) {
 		const struct cache_entry *ce = istate->cache[i];
-		if (sw_action == PS_IGNORE_SKIP_WORKTREE && ce_skip_worktree(ce))
+		if (sw_action == PS_IGNORE_SKIP_WORKTREE &&
+		    (ce_skip_worktree(ce) || !path_in_sparse_checkout(ce->name, istate)))
 			continue;
 		ce_path_match(istate, ce, pathspec, seen);
 	}

index 962bece03e1..c2a4eec548d 100755
@@ -529,7 +529,7 @@ test_expect_failure 'merge with conflict outside cone' '
 	# NEEDSWORK: Even though the merge conflict removed the
 	# SKIP_WORKTREE bit from the index entry for folder1/a, we should
 	# warn that this is a problematic add.
-	test_all_match git add folder1/a &&
+	test_sparse_match test_must_fail git add folder1/a &&
 	test_all_match git status --porcelain=v2 &&
 
 	# 3. Rename the file to another sparse filename and

@@ -538,7 +538,7 @@ test_expect_failure 'merge with conflict outside cone' '
 	# NEEDSWORK: This mode now fails, because folder2/z is
 	# outside of the sparse-checkout cone and does not match an
 	# existing index entry with the SKIP_WORKTREE bit cleared.
-	test_all_match git add folder2 &&
+	test_sparse_match test_must_fail git add folder2 &&
 	test_all_match git status --porcelain=v2 &&
 
 	test_all_match git merge --continue &&

@@ -566,7 +566,7 @@ test_expect_failure 'cherry-pick/rebase with conflict outside cone' '
 		# NEEDSWORK: Even though the merge conflict removed the
 		# SKIP_WORKTREE bit from the index entry for folder1/a, we should
 		# warn that this is a problematic add.
-		test_all_match git add folder1/a &&
+		test_sparse_match test_must_fail git add folder1/a &&
 		test_all_match git status --porcelain=v2 &&
 
 		# 3. Rename the file to another sparse filename and

@@ -575,7 +575,7 @@ test_expect_failure 'cherry-pick/rebase with conflict outside cone' '
 		# outside of the sparse-checkout cone and does not match an
 		# existing index entry with the SKIP_WORKTREE bit cleared.
 		run_on_all mv folder2/a folder2/z &&
-		test_all_match git add folder2 &&
+		test_sparse_match test_must_fail git add folder2 &&
 		test_all_match git status --porcelain=v2 &&
 
 		test_all_match git $OPERATION --continue &&

gitgitgadget

From: Derrick Stolee <[email protected]>

We previously modified 'git add' to refuse updating index entries outside of the sparse-checkout cone. This is justified to prevent users from accidentally getting into a confusing state when Git removes those files from the working tree at some later point.

Unfortunately, this caused some workflows that were previously possible to become impossible, especially around merge conflicts outside of the sparse-checkout cone. These were documented in tests within t1092.

We now re-enable these workflows using a new '--sparse' option to 'git add'. This allows users to signal "Yes, I do know what I'm doing with these files," and accept the consequences of the files leaving the worktree later.

We delay updating the advice message until implementing a similar option in 'git rm' and 'git mv'.

---
 Documentation/git-add.txt                |  9 +++++++-
 builtin/add.c                            | 10 ++++++---
 t/t1092-sparse-checkout-compatibility.sh | 28 ++++++++----------------
 t/t3705-add-sparse-checkout.sh           |  8 +++++++
 4 files changed, 32 insertions(+), 23 deletions(-)

index be5e3ac54b8..bb79016d2ca 100644
@@ -9,7 +9,7 @@ SYNOPSIS
 --------
 [verse]
 'git add' [--verbose | -v] [--dry-run | -n] [--force | -f] [--interactive | -i] [--patch | -p]
-	  [--edit | -e] [--[no-]all | --[no-]ignore-removal | [--update | -u]]
+	  [--edit | -e] [--[no-]all | --[no-]ignore-removal | [--update | -u]] [--sparse]
 	  [--intent-to-add | -N] [--refresh] [--ignore-errors] [--ignore-missing] [--renormalize]
 	  [--chmod=(+|-)x] [--pathspec-from-file=<file> [--pathspec-file-nul]]
 	  [--] [<pathspec>...]

@@ -79,6 +79,13 @@ in linkgit:gitglossary[7].
 --force::
 	Allow adding otherwise ignored files.
 
+--sparse::
+	Allow updating index entries outside of the sparse-checkout cone.
+	Normally, `git add` refuses to update index entries whose paths do
+	not fit within the sparse-checkout cone, since those files might
+	be removed from the working tree without warning. See
+	linkgit:git-sparse-checkout[1] for more.
+
 -i::
 --interactive::
 	Add modified contents in the working tree interactively to

index 3a109276b74..68f2de80594 100644
@@ -30,6 +30,7 @@ static int patch_interactive, add_interactive, edit_interactive;
 static int take_worktree_changes;
 static int add_renormalize;
 static int pathspec_file_nul;
+static int include_sparse;
 static const char *pathspec_from_file;
 static int legacy_stash_p; /* support for the scripted `git stash` */
 

@@ -46,7 +47,7 @@ static int chmod_pathspec(struct pathspec *pathspec, char flip, int show_only)
 		struct cache_entry *ce = active_cache[i];
 		int err;
 
-		if (ce_skip_worktree(ce))
+		if (!include_sparse && ce_skip_worktree(ce))
 			continue;
 
 		if (pathspec && !ce_path_match(&the_index, ce, pathspec, NULL))

@@ -379,6 +380,7 @@ static struct option builtin_add_options[] = {
 	OPT_BOOL( 0 , "refresh", &refresh_only, N_("don't add, only refresh the index")),
 	OPT_BOOL( 0 , "ignore-errors", &ignore_add_errors, N_("just skip files which cannot be added because of errors")),
 	OPT_BOOL( 0 , "ignore-missing", &ignore_missing, N_("check if - even missing - files are ignored in dry run")),
+	OPT_BOOL(0, "sparse", &include_sparse, N_("allow updating entries outside of the sparse-checkout cone")),
 	OPT_STRING(0, "chmod", &chmod_arg, "(+|-)x",
 		   N_("override the executable bit of the listed files")),
 	OPT_HIDDEN_BOOL(0, "warn-embedded-repo", &warn_on_embedded_repo,

@@ -457,7 +459,8 @@ static int add_files(struct dir_struct *dir, int flags)
 	}
 
 	for (i = 0; i < dir->nr; i++) {
-		if (!path_in_sparse_checkout(dir->entries[i]->name, &the_index)) {
+		if (!include_sparse &&
+		    !path_in_sparse_checkout(dir->entries[i]->name, &the_index)) {
 			string_list_append(&only_match_skip_worktree,
 					   dir->entries[i]->name);
 			continue;

@@ -642,7 +645,8 @@ int cmd_add(int argc, const char **argv, const char *prefix)
 			if (seen[i])
 				continue;
 
-			if (matches_skip_worktree(&pathspec, i, &skip_worktree_seen)) {
+			if (!include_sparse &&
+			    matches_skip_worktree(&pathspec, i, &skip_worktree_seen)) {
 				string_list_append(&only_match_skip_worktree,
 						   pathspec.items[i].original);
 				continue;

index c2a4eec548d..1e7799fd76a 100755
@@ -329,11 +329,7 @@ test_expect_success 'commit including unstaged changes' '
 	test_all_match git status --porcelain=v2
 '
 
-# NEEDSWORK: Now that 'git add folder1/new' fails, the changes being
-# attempted here fail for the sparse-checkout and sparse-index repos.
-# We must enable a way for adding files outside the sparse-checkout
-# done, even if it is by an optional flag.
-test_expect_failure 'status/add: outside sparse cone' '
+test_expect_success 'status/add: outside sparse cone' '
 	init_repos &&
 
 	# folder1 is at HEAD, but outside the sparse cone

@@ -355,15 +351,16 @@ test_expect_failure 'status/add: outside sparse cone' '
 	test_sparse_match test_must_fail git add folder1/a &&
 	test_sparse_match test_must_fail git add --refresh folder1/a &&
 	test_sparse_match test_must_fail git add folder1/new &&
+	test_sparse_match git add --sparse folder1/a &&
+	test_sparse_match git add --sparse folder1/new &&
 
-	# NEEDSWORK: behavior begins to deviate here.
-	test_all_match git add . &&
+	test_all_match git add --sparse . &&
 	test_all_match git status --porcelain=v2 &&
 	test_all_match git commit -m folder1/new &&
 	test_all_match git rev-parse HEAD^{tree} &&
 
 	run_on_all ../edit-contents folder1/newer &&
-	test_all_match git add folder1/ &&
+	test_all_match git add --sparse folder1/ &&
 	test_all_match git status --porcelain=v2 &&
 	test_all_match git commit -m folder1/newer &&
 	test_all_match git rev-parse HEAD^{tree}

@@ -507,12 +504,7 @@ test_expect_success 'merge, cherry-pick, and rebase' '
 	done
 '
 
-# NEEDSWORK: This test is documenting current behavior, but that
-# behavior can be confusing to users so there is desire to change it.
-# Right now, users might be using this flow to work through conflicts,
-# so any solution should present advice to users who try this sequence
-# of commands to follow whatever new method we create.
-test_expect_failure 'merge with conflict outside cone' '
+test_expect_success 'merge with conflict outside cone' '
 	init_repos &&
 
 	test_all_match git checkout -b merge-tip merge-left &&

@@ -528,17 +520,15 @@ test_expect_failure 'merge with conflict outside cone' '
 	# 2. Add the file with conflict markers
 	# NEEDSWORK: Even though the merge conflict removed the
 	# SKIP_WORKTREE bit from the index entry for folder1/a, we should
-	# warn that this is a problematic add.
-	test_sparse_match test_must_fail git add folder1/a &&
+	# warn that this is a problematic add when --sparse is not set.
+	test_all_match git add --sparse folder1/a &&
 	test_all_match git status --porcelain=v2 &&
 
 	# 3. Rename the file to another sparse filename and
 	#    accept conflict markers as resolved content.
 	run_on_all mv folder2/a folder2/z &&
-	# NEEDSWORK: This mode now fails, because folder2/z is
-	# outside of the sparse-checkout cone and does not match an
-	# existing index entry with the SKIP_WORKTREE bit cleared.
 	test_sparse_match test_must_fail git add folder2 &&
+	test_all_match git add --sparse folder2 &&
 	test_all_match git status --porcelain=v2 &&
 
 	test_all_match git merge --continue &&

index 2b1fd0d0eef..99c27dd9fb7 100755
@@ -152,4 +152,12 @@ test_expect_success 'add obeys advice.updateSparsePath' '
 
 '
 
+test_expect_success 'add allows sparse entries with --sparse' '
+	git sparse-checkout set a &&
+	echo modified >sparse_entry &&
+	test_must_fail git add sparse_entry &&
+	git add --sparse sparse_entry 2>stderr &&
+	test_must_be_empty stderr
+'
+
 test_done

gitgitgadget

From: Derrick Stolee <[email protected]>

When a merge results in a conflict outside of the sparse-checkout cone, the conflicted file is written to the working tree and the index entry loses the SKIP_WORKTREE bit. This allows users to add the file to the index without realizing that the file might leave the working tree in a later Git command.

Block this behavior, but keep in mind that the user can override the failure using the '--sparse' option.

---
 pathspec.c                               | 2 +-
 t/t1091-sparse-checkout-builtin.sh       | 4 +++-
 t/t1092-sparse-checkout-compatibility.sh | 6 +++---
 3 files changed, 7 insertions(+), 5 deletions(-)

index 0e6e60fdc5a..ddeeba79114 100644
@@ -71,7 +71,7 @@ char *find_pathspecs_matching_skip_worktree(const struct pathspec *pathspec)
 
 	for (i = 0; i < istate->cache_nr; i++) {
 		struct cache_entry *ce = istate->cache[i];
-		if (ce_skip_worktree(ce))
+		if (ce_skip_worktree(ce) || !path_in_sparse_checkout(ce->name, istate))
 		    ce_path_match(istate, ce, pathspec, seen);
 	}
 

index e0f31186d89..b6efdb3c52f 100755
@@ -406,7 +406,7 @@ test_expect_success 'sparse-checkout (init|set|disable) warns with unmerged stat
 	git -C unmerged sparse-checkout disable
 '
 
-test_expect_success 'sparse-checkout reapply' '
+test_expect_failure 'sparse-checkout reapply' '
 	git clone repo tweak &&
 
 	echo dirty >tweak/deep/deeper2/a &&

@@ -438,6 +438,8 @@ test_expect_success 'sparse-checkout reapply' '
 	test_i18ngrep "warning.*The following paths are unmerged" err &&
 	test_path_is_file tweak/folder1/a &&
 
+	# NEEDSWORK: We are asking to update a file outside of the
+	# sparse-checkout cone, but this is no longer allowed.
 	git -C tweak add folder1/a &&
 	git -C tweak sparse-checkout reapply 2>err &&
 	test_must_be_empty err &&

index 1e7799fd76a..65998e664a9 100755
@@ -518,9 +518,8 @@ test_expect_success 'merge with conflict outside cone' '
 	test_all_match git status --porcelain=v2 &&
 
 	# 2. Add the file with conflict markers
-	# NEEDSWORK: Even though the merge conflict removed the
-	# SKIP_WORKTREE bit from the index entry for folder1/a, we should
-	# warn that this is a problematic add when --sparse is not set.
+	test_sparse_match test_must_fail git add folder1/a &&
+	test_i18ngrep "Disable or modify the sparsity rules" sparse-checkout-err &&
 	test_all_match git add --sparse folder1/a &&
 	test_all_match git status --porcelain=v2 &&
 

@@ -528,6 +527,7 @@ test_expect_success 'merge with conflict outside cone' '
 	#    accept conflict markers as resolved content.
 	run_on_all mv folder2/a folder2/z &&
 	test_sparse_match test_must_fail git add folder2 &&
+	test_i18ngrep "Disable or modify the sparsity rules" sparse-checkout-err &&
 	test_all_match git add --sparse folder2 &&
 	test_all_match git status --porcelain=v2 &&
 

gitgitgadget

From: Derrick Stolee <[email protected]>

---
 Documentation/git-rm.txt      |  6 ++++++
 builtin/rm.c                  |  8 ++++++--
 t/t3602-rm-sparse-checkout.sh | 14 +++++++++++++-
 3 files changed, 25 insertions(+), 3 deletions(-)

index 26e9b284704..81bc23f3cdb 100644
@@ -72,6 +72,12 @@ For more details, see the 'pathspec' entry in linkgit:gitglossary[7].
 --ignore-unmatch::
 	Exit with a zero status even if no files matched.
 
+--sparse::
+	Allow updating index entries outside of the sparse-checkout cone.
+	Normally, `git rm` refuses to update index entries whose paths do
+	not fit within the sparse-checkout cone. See
+	linkgit:git-sparse-checkout[1] for more.
+
 -q::
 --quiet::
 	`git rm` normally outputs one line (in the form of an `rm` command)

index 8a24c715e02..4208f3f9a5f 100644
@@ -237,6 +237,7 @@ static int check_local_mod(struct object_id *head, int index_only)
 
 static int show_only = 0, force = 0, index_only = 0, recursive = 0, quiet = 0;
 static int ignore_unmatch = 0, pathspec_file_nul;
+static int include_sparse;
 static char *pathspec_from_file;
 
 static struct option builtin_rm_options[] = {

@@ -247,6 +248,7 @@ static struct option builtin_rm_options[] = {
 	OPT_BOOL('r', NULL,             &recursive,  N_("allow recursive removal")),
 	OPT_BOOL( 0 , "ignore-unmatch", &ignore_unmatch,
 				N_("exit with a zero status even if nothing matched")),
+	OPT_BOOL(0, "sparse", &include_sparse, N_("allow updating entries outside of the sparse-checkout cone")),
 	OPT_PATHSPEC_FROM_FILE(&pathspec_from_file),
 	OPT_PATHSPEC_FILE_NUL(&pathspec_file_nul),
 	OPT_END(),

@@ -298,7 +300,8 @@ int cmd_rm(int argc, const char **argv, const char *prefix)
 	ensure_full_index(&the_index);
 	for (i = 0; i < active_nr; i++) {
 		const struct cache_entry *ce = active_cache[i];
-		if (ce_skip_worktree(ce))
+
+		if (!include_sparse && ce_skip_worktree(ce))
 			continue;
 		if (!ce_path_match(&the_index, ce, &pathspec, seen))
 			continue;

@@ -322,7 +325,8 @@ int cmd_rm(int argc, const char **argv, const char *prefix)
 				seen_any = 1;
 			else if (ignore_unmatch)
 				continue;
-			else if (matches_skip_worktree(&pathspec, i, &skip_worktree_seen))
+			else if (!include_sparse &&
+				 matches_skip_worktree(&pathspec, i, &skip_worktree_seen))
 				string_list_append(&only_match_skip_worktree, original);
 			else
 				die(_("pathspec '%s' did not match any files"), original);

index e9e9a15c74c..a34b978bfd8 100755
@@ -36,13 +36,25 @@ done
 
 test_expect_success 'recursive rm does not remove sparse entries' '
 	git reset --hard &&
-	git sparse-checkout set sub/dir &&
+	git sparse-checkout set sub/dir/ &&
 	git rm -r sub &&
 	git status --porcelain -uno >actual &&
 	echo "D  sub/dir/e" >expected &&
 	test_cmp expected actual
 '
 
+test_expect_success 'recursive rm --sparse removes sparse entries' '
+	git reset --hard &&
+	git sparse-checkout set "sub/dir" &&
+	git rm --sparse -r sub &&
+	git status --porcelain -uno >actual &&
+	cat >expected <<-\EOF &&
+	D  sub/d
+	D  sub/dir/e
+	EOF
+	test_cmp expected actual
+'
+
 test_expect_success 'rm obeys advice.updateSparsePath' '
 	git reset --hard &&
 	git sparse-checkout set a &&

gitgitgadget

From: Derrick Stolee <[email protected]>

If a path does not match the sparse-checkout cone but is somehow missing the SKIP_WORKTREE bit, then 'git rm' currently succeeds in removing the file. One reason a user might be in this situation is a merge conflict outside of the sparse-checkout cone. Removing such a file might be problematic for users who are not sure what they are doing.

Add a check to path_in_sparse_checkout() when 'git rm' is checking if a path should be considered for deletion. Of course, this check is ignored if the '--sparse' option is specified, allowing users who accept the risks to continue with the removal.

---
 builtin/rm.c                  |  4 +++-
 t/t3602-rm-sparse-checkout.sh | 11 +++++++++++
 2 files changed, 14 insertions(+), 1 deletion(-)

index 4208f3f9a5f..a6da03da2be 100644
@@ -301,7 +301,9 @@ int cmd_rm(int argc, const char **argv, const char *prefix)
 	for (i = 0; i < active_nr; i++) {
 		const struct cache_entry *ce = active_cache[i];
 
-		if (!include_sparse && ce_skip_worktree(ce))
+		if (!include_sparse &&
+		    (ce_skip_worktree(ce) ||
+		     !path_in_sparse_checkout(ce->name, &the_index)))
 			continue;
 		if (!ce_path_match(&the_index, ce, &pathspec, seen))
 			continue;

index a34b978bfd8..44f3e923164 100755
@@ -87,4 +87,15 @@ test_expect_success 'do not warn about sparse entries with --ignore-unmatch' '
 	git ls-files --error-unmatch b
 '
 
+test_expect_success 'refuse to rm a non-skip-worktree path outside sparse cone' '
+	git reset --hard &&
+	git sparse-checkout set a &&
+	git update-index --no-skip-worktree b &&
+	test_must_fail git rm b 2>stderr &&
+	test_cmp b_error_and_hint stderr &&
+	git rm --sparse b 2>stderr &&
+	test_must_be_empty stderr &&
+	test_path_is_missing b
+'
+
 test_done

gitgitgadget

From: Derrick Stolee <[email protected]>

Since cmd_mv() does not operate on cache entries and instead directly checks the filesystem, we can only use path_in_sparse_checkout() as a mechanism for seeing if a path is sparse or not. Be sure to skip returning a failure if '-k' is specified.

---
 builtin/mv.c                  | 19 +++++++
 t/t7002-mv-sparse-checkout.sh | 99 +++++++++++++++++++++++++++++++++++
 2 files changed, 118 insertions(+)
 create mode 100755 t/t7002-mv-sparse-checkout.sh

index c2f96c8e895..b58fd4ce5ba 100644
@@ -133,6 +133,7 @@ int cmd_mv(int argc, const char **argv, const char *prefix)
 	struct string_list src_for_dst = STRING_LIST_INIT_NODUP;
 	struct lock_file lock_file = LOCK_INIT;
 	struct cache_entry *ce;
+	struct string_list only_match_skip_worktree = STRING_LIST_INIT_NODUP;
 
 	git_config(git_default_config, NULL);
 

@@ -176,10 +177,22 @@ int cmd_mv(int argc, const char **argv, const char *prefix)
 		const char *src = source[i], *dst = destination[i];
 		int length, src_is_dir;
 		const char *bad = NULL;
+		int skip_sparse = 0;
 
 		if (show_only)
 			printf(_("Checking rename of '%s' to '%s'\n"), src, dst);
 
+		if (!path_in_sparse_checkout(src, &the_index)) {
+			string_list_append(&only_match_skip_worktree, src);
+			skip_sparse = 1;
+		}
+		if (!path_in_sparse_checkout(dst, &the_index)) {
+			string_list_append(&only_match_skip_worktree, dst);
+			skip_sparse = 1;
+		}
+		if (skip_sparse)
+			continue;
+
 		length = strlen(src);
 		if (lstat(src, &st) < 0)
 			bad = _("bad source");

@@ -266,6 +279,12 @@ int cmd_mv(int argc, const char **argv, const char *prefix)
 		}
 	}
 
+	if (only_match_skip_worktree.nr) {
+		advise_on_updating_sparse_paths(&only_match_skip_worktree);
+		if (!ignore_errors)
+			return 1;
+	}
+
 	for (i = 0; i < argc; i++) {
 		const char *src = source[i], *dst = destination[i];
 		enum update_mode mode = modes[i];

gitgitgadget

From: Derrick Stolee <[email protected]>

Users can get into strange situations if 'git mv' allows moving files into, out of, or around the sparse-checkout cone. However, some users may still want to do it. Allow knowledgeable users to do so via a new '--sparse' option.

There are some special cases that occur in this change, such as the case of a directory that doesn't match the sparse-checkout cone, but exists in the working tree because a subset of its contents do match. We need to communicate that index entries with the SKIP_WORKTREE bit are not expected to be in the working directory and hence are not needed when moving the contents. This is only a check for the existence of the source file. The call to rename_cache_entry_at() still changes the index appropriately in these cases.

---
 builtin/mv.c                  | 24 ++++++++++++++----------
 t/t7002-mv-sparse-checkout.sh | 34 ++++++++++++++++++++++++++++------
 2 files changed, 42 insertions(+), 16 deletions(-)

index b58fd4ce5ba..92ea9f0ca37 100644
@@ -118,17 +118,18 @@ static int index_range_of_same_dir(const char *src, int length,
 int cmd_mv(int argc, const char **argv, const char *prefix)
 {
 	int i, flags, gitmodules_modified = 0;
-	int verbose = 0, show_only = 0, force = 0, ignore_errors = 0;
+	int verbose = 0, show_only = 0, force = 0, ignore_errors = 0, ignore_sparse = 0;
 	struct option builtin_mv_options[] = {
 		OPT__VERBOSE(&verbose, N_("be verbose")),
 		OPT__DRY_RUN(&show_only, N_("dry run")),
 		OPT__FORCE(&force, N_("force move/rename even if target exists"),
 			   PARSE_OPT_NOCOMPLETE),
 		OPT_BOOL('k', NULL, &ignore_errors, N_("skip move/rename errors")),
+		OPT_BOOL(0, "sparse", &ignore_sparse, N_("allow updating entries outside of the sparse-checkout cone")),
 		OPT_END(),
 	};
 	const char **source, **destination, **dest_path, **submodule_gitfile;
-	enum update_mode { BOTH = 0, WORKING_DIRECTORY, INDEX } *modes;
+	enum update_mode { BOTH = 0, WORKING_DIRECTORY, INDEX, SPARSE } *modes;
 	struct stat st;
 	struct string_list src_for_dst = STRING_LIST_INIT_NODUP;
 	struct lock_file lock_file = LOCK_INIT;

@@ -182,11 +183,11 @@ int cmd_mv(int argc, const char **argv, const char *prefix)
 		if (show_only)
 			printf(_("Checking rename of '%s' to '%s'\n"), src, dst);
 
-		if (!path_in_sparse_checkout(src, &the_index)) {
+		if (!ignore_sparse && !path_in_sparse_checkout(src, &the_index)) {
 			string_list_append(&only_match_skip_worktree, src);
 			skip_sparse = 1;
 		}
-		if (!path_in_sparse_checkout(dst, &the_index)) {
+		if (!ignore_sparse && !path_in_sparse_checkout(dst, &the_index)) {
 			string_list_append(&only_match_skip_worktree, dst);
 			skip_sparse = 1;
 		}

@@ -194,9 +195,11 @@ int cmd_mv(int argc, const char **argv, const char *prefix)
 			continue;
 
 		length = strlen(src);
-		if (lstat(src, &st) < 0)
-			bad = _("bad source");
-		else if (!strncmp(src, dst, length) &&
+		if (lstat(src, &st) < 0) {
+			/* only error if existence is expected. */
+			if (modes[i] != SPARSE)
+				bad = _("bad source");
+		} else if (!strncmp(src, dst, length) &&
 				(dst[length] == 0 || dst[length] == '/')) {
 			bad = _("can not move directory into itself");
 		} else if ((src_is_dir = S_ISDIR(st.st_mode))

@@ -225,11 +228,12 @@ int cmd_mv(int argc, const char **argv, const char *prefix)
 				dst_len = strlen(dst);
 
 				for (j = 0; j < last - first; j++) {
-					const char *path = active_cache[first + j]->name;
+					const struct cache_entry *ce = active_cache[first + j];
+					const char *path = ce->name;
 					source[argc + j] = path;
 					destination[argc + j] =
 						prefix_path(dst, dst_len, path + length + 1);
-					modes[argc + j] = INDEX;
+					modes[argc + j] = ce_skip_worktree(ce) ? SPARSE : INDEX;
 					submodule_gitfile[argc + j] = NULL;
 				}
 				argc += last - first;

@@ -293,7 +297,7 @@ int cmd_mv(int argc, const char **argv, const char *prefix)
 			printf(_("Renaming %s to %s\n"), src, dst);
 		if (show_only)
 			continue;
-		if (mode != INDEX && rename(src, dst) < 0) {
+		if (mode != INDEX && mode != SPARSE && rename(src, dst) < 0) {
 			if (ignore_errors)
 				continue;
 			die_errno(_("renaming '%s' failed"), src);

index 5397c6d07bd..517fd587fa8 100755
@@ -31,7 +31,9 @@ test_expect_success 'mv refuses to move sparse-to-sparse' '
 	echo b >>expect &&
 	echo e >>expect &&
 	cat sparse_hint >>expect &&
-	test_cmp expect stderr
+	test_cmp expect stderr &&
+	git mv --sparse b e 2>stderr &&
+	test_must_be_empty stderr
 '
 
 test_expect_success 'mv refuses to move sparse-to-sparse, ignores failure' '

@@ -44,7 +46,9 @@ test_expect_success 'mv refuses to move sparse-to-sparse, ignores failure' '
 	echo b >>expect &&
 	echo e >>expect &&
 	cat sparse_hint >>expect &&
-	test_cmp expect stderr
+	test_cmp expect stderr &&
+	git mv --sparse -k b e 2>stderr &&
+	test_must_be_empty stderr
 '
 
 test_expect_success 'mv refuses to move non-sparse-to-sparse' '

@@ -55,7 +59,9 @@ test_expect_success 'mv refuses to move non-sparse-to-sparse' '
 	cat sparse_error_header >expect &&
 	echo e >>expect &&
 	cat sparse_hint >>expect &&
-	test_cmp expect stderr
+	test_cmp expect stderr &&
+	git mv --sparse a e 2>stderr &&
+	test_must_be_empty stderr
 '
 
 test_expect_success 'mv refuses to move sparse-to-non-sparse' '

@@ -67,7 +73,9 @@ test_expect_success 'mv refuses to move sparse-to-non-sparse' '
 	cat sparse_error_header >expect &&
 	echo b >>expect &&
 	cat sparse_hint >>expect &&
-	test_cmp expect stderr
+	test_cmp expect stderr &&
+	git mv --sparse b e 2>stderr &&
+	test_must_be_empty stderr
 '
 
 test_expect_success 'recursive mv refuses to move (possible) sparse' '

@@ -80,7 +88,14 @@ test_expect_success 'recursive mv refuses to move (possible) sparse' '
 	echo sub >>expect &&
 	echo sub2 >>expect &&
 	cat sparse_hint >>expect &&
-	test_cmp expect stderr
+	test_cmp expect stderr &&
+	git mv --sparse sub sub2 2>stderr &&
+	test_must_be_empty stderr &&
+	git commit -m "moved sub to sub2" &&
+	git rev-parse HEAD~1:sub >expect &&
+	git rev-parse HEAD:sub2 >actual &&
+	test_cmp expect actual &&
+	git reset --hard HEAD~1
 '
 
 test_expect_success 'recursive mv refuses to move sparse' '

@@ -93,7 +108,14 @@ test_expect_success 'recursive mv refuses to move sparse' '
 	echo sub/dir2/e >>expect &&
 	echo sub2/dir2/e >>expect &&
 	cat sparse_hint >>expect &&
-	test_cmp expect stderr
+	test_cmp expect stderr &&
+	git mv --sparse sub sub2 2>stderr &&
+	test_must_be_empty stderr &&
+	git commit -m "moved sub to sub2" &&
+	git rev-parse HEAD~1:sub >expect &&
+	git rev-parse HEAD:sub2 >actual &&
+	test_cmp expect actual &&
+	git reset --hard HEAD~1
 '
 
 test_done

gitgitgadget

From: Derrick Stolee <[email protected]>

The previous changes modified the behavior of 'git add', 'git rm', and 'git mv' to not adjust paths outside the sparse-checkout cone, even if they exist in the working tree and their cache entries lack the SKIP_WORKTREE bit. The intention is to warn users that they are doing something potentially dangerous. The '--sparse' option was added to each command to allow careful users the same ability they had before.

To improve the discoverability of this new functionality, add a message to advice.updateSparsePath that mentions the existence of the option.

---
 advice.c                       | 3 ++-
 t/t3602-rm-sparse-checkout.sh  | 2 +-
 t/t3705-add-sparse-checkout.sh | 2 +-
 t/t7002-mv-sparse-checkout.sh  | 2 +-
 4 files changed, 5 insertions(+), 4 deletions(-)

index 0b9c89c48ab..695693e5fa8 100644
@@ -300,7 +300,8 @@ void advise_on_updating_sparse_paths(struct string_list *pathspec_list)
 		fprintf(stderr, "%s\n", item->string);
 
 	advise_if_enabled(ADVICE_UPDATE_SPARSE_PATH,
-			  _("Disable or modify the sparsity rules if you intend"
+			  _("Disable or modify the sparsity rules or"
+			    " use the --sparse option if you intend"
 			    " to update such entries."));
 }
 

index 44f3e923164..22170feef62 100755
@@ -16,7 +16,7 @@ test_expect_success 'setup' "
 	EOF
 
 	cat >sparse_hint <<-EOF &&
-	hint: Disable or modify the sparsity rules if you intend to update such entries.
+	hint: Disable or modify the sparsity rules or use the --sparse option if you intend to update such entries.
 	hint: Disable this message with \"git config advice.updateSparsePath false\"
 	EOF
 

index 99c27dd9fb7..0a8071370d3 100755
@@ -43,7 +43,7 @@ test_expect_success 'setup' "
 	EOF
 
 	cat >sparse_hint <<-EOF &&
-	hint: Disable or modify the sparsity rules if you intend to update such entries.
+	hint: Disable or modify the sparsity rules or use the --sparse option if you intend to update such entries.
 	hint: Disable this message with \"git config advice.updateSparsePath false\"
 	EOF
 

index 517fd587fa8..b6e9ea0b2c8 100755
@@ -16,7 +16,7 @@ test_expect_success 'setup' "
 	EOF
 
 	cat >sparse_hint <<-EOF
-	hint: Disable or modify the sparsity rules if you intend to update such entries.
+	hint: Disable or modify the sparsity rules or use the --sparse option if you intend to update such entries.
 	hint: Disable this message with \"git config advice.updateSparsePath false\"
 	EOF
 "

gitgitgadget

s/sprase/sparse/

While I was playing with this patch, I did the following:

echo a >a
echo b >b
git add .
git commit -m files
git sparse-checkout set a
echo c >c
git add c

And the last git add was successful in adding the untracked c file which is outside the sparse checkout. I'm not sure if I'm doing something wrong, but it seems that path_in_sparse_checkout() returns UNDECIDED for c. Is it because there was no pattern in the list explicitly excluding it? And if so, should we consider UNDECIDED as NOT_MATCHED for path_in_sparse_checkout()?

Perhaps this could be "skip _tracked_ paths that ..." (to help differentiate the end goal of this patch from the previous one).

index f9eadc4fc1..4245d7ead5 100644
@@ -198,7 +198,8 @@ int run_diff_files(struct rev_info *revs, unsigned int option)
 				continue;
 		}
 
-		if (ce_uptodate(ce) || ce_skip_worktree(ce))
+		if (ce_uptodate(ce) || ce_skip_worktree(ce) ||
+		    !path_in_sparse_checkout(ce->name, istate))
 			continue;
 
 		/*

index f675bdeae4..3d7762aac2 100644
@@ -94,6 +94,10 @@ static void update_callback(struct diff_queue_struct *q,
 	for (i = 0; i < q->nr; i++) {
 		struct diff_filepair *p = q->queue[i];
 		const char *path = p->one->path;
+
+		if (!path_in_sparse_checkout(path, &the_index))
+			continue;
+
 		switch (fix_unmerged_status(p, data)) {
 		default:
 			die(_("unexpected diff status %c"), p->status);

Hmm, didn't we already block this behavior at patch 6?

Nevertheless, as I mentioned there, I think the change to find_pathspecs_matching_skip_worktree() from this patch should be together with the other changes from 6.

Maybe mention in the commit message that, for now, rm's --sparse only affects entries with the skip_worktree bit set? (Which will be changed in the following patch.)

Please disconsider this, It was my fault indeed. I had applied the patches onto the wrong base. Now I fetched them again but from the GGG tag, and my manual test worked as expected.

Another problem is that the displayed message will say that the pathspecs "match index entries outside sparse checkout" even when the path given to mv doesn't really exist:

git sparse-checkout set some/dir/
git mv nonexistent-file foo

The following pathspecs didn't match any eligible path, but they do match index entries outside the current sparse checkout:
nonexistent-file
hint: Disable or modify the sparsity rules if you intend to update such entries.
hint: Disable this message with "git config advice.updateSparsePath false"

Thanks for adding your review. I'm sorry I'm so late getting back to it.

Thanks.

How about "matched_sparse_paths" as a whole name swap? The earlier uses cared only if every match was sparse, but here we are actually looking at cases that are untracked, and the pathspec could also match other non-sparse cases.

I think the advice should be modified to refer to paths and/or pathspecs, and also it is not completely correct anymore.

Instead of

The following pathspecs didn't match any eligible path, but they do match index entries outside the current sparse checkout:

perhaps

The following paths and/or pathspecs matched paths that exist outside of your sparse-checkout definition, so will not be updated in the index:

I'm going to save this one for a new patch at the end to make sure it handles all of the cases involved in this series.

Thanks,
-Stolee

I will expand with these details.

No, it is not. Thanks.

-Stolee

I will move the logic to the last possible place before "accepting" the move, then add a comment detailing why it should be there.

...

test_when_finished is cleaner.

I can expand these tests to include tracked and untracked targets.

Good idea.

In fact, there is a problem that the '-k' gets around the protections because it doesn't return 1 early. I'll fix this by jumping to the end of the loop which removes the entries from the arrays.

Thanks,
-Stolee

You are absolutely right to point this out. I had missed this interaction. But, this is also already broken. The patch below adds a check to show that 'git add' does not add the sparse_entry, but it does (even when applied before any patch in this series). That is: all the modified tests fail after this change. I'll work to fix this issue before the next version of this series.

Thanks,
-Stolee

--- >8 ---

From 21dab466d221e8632d98553f5f1fa900a2d47c7f Mon Sep 17 00:00:00 2001
From: Derrick Stolee <[email protected]>
Date: Wed, 8 Sep 2021 15:40:32 -0400
Subject: [PATCH] t3705: test that 'sparse_entry' is unstaged

The tests in t3705-add-sparse-checkout.sh check to see how 'git add' behaves with paths outside the sparse-checkout definition. These currently check to see if a given warning is present but not that the index is not updated with the sparse entries. Add a new 'test_sparse_entry_unstaged' helper to be sure 'git add' is behaving correctly.

---
 t/t3705-add-sparse-checkout.sh | 13 +++++++++++++
 1 file changed, 13 insertions(+)

index 2b1fd0d0eef..d31a0d4f550 100755
@@ -36,6 +36,11 @@ setup_gitignore () {
 	EOF
 }
 
+test_sparse_entry_unstaged () {
+	git status --porcelain >actual &&
+	! grep "A  sparse_entry" actual
+}
+
 test_expect_success 'setup' "
 	cat >sparse_error_header <<-EOF &&
 	The following pathspecs didn't match any eligible path, but they do match index

@@ -55,6 +60,7 @@ test_expect_success 'git add does not remove sparse entries' '
 	setup_sparse_entry &&
 	rm sparse_entry &&
 	test_must_fail git add sparse_entry 2>stderr &&
+	test_sparse_entry_unstaged &&
 	test_cmp error_and_hint stderr &&
 	test_sparse_entry_unchanged
 '

@@ -73,6 +79,7 @@ test_expect_success 'git add . does not remove sparse entries' '
 	rm sparse_entry &&
 	setup_gitignore &&
 	test_must_fail git add . 2>stderr &&
+	test_sparse_entry_unstaged &&
 
 	cat sparse_error_header >expect &&
 	echo . >>expect &&

@@ -88,6 +95,7 @@ do
 		setup_sparse_entry &&
 		echo modified >sparse_entry &&
 		test_must_fail git add $opt sparse_entry 2>stderr &&
+		test_sparse_entry_unstaged &&
 		test_cmp error_and_hint stderr &&
 		test_sparse_entry_unchanged
 	'

@@ -98,6 +106,7 @@ test_expect_success 'git add --refresh does not update sparse entries' '
 	git ls-files --debug sparse_entry | grep mtime >before &&
 	test-tool chmtime -60 sparse_entry &&
 	test_must_fail git add --refresh sparse_entry 2>stderr &&
+	test_sparse_entry_unstaged &&
 	test_cmp error_and_hint stderr &&
 	git ls-files --debug sparse_entry | grep mtime >after &&
 	test_cmp before after

@@ -106,6 +115,7 @@ test_expect_success 'git add --refresh does not update sparse entries' '
 test_expect_success 'git add --chmod does not update sparse entries' '
 	setup_sparse_entry &&
 	test_must_fail git add --chmod=+x sparse_entry 2>stderr &&
+	test_sparse_entry_unstaged &&
 	test_cmp error_and_hint stderr &&
 	test_sparse_entry_unchanged &&
 	! test -x sparse_entry

@@ -116,6 +126,7 @@ test_expect_success 'git add --renormalize does not update sparse entries' '
 	setup_sparse_entry "LINEONE\r\nLINETWO\r\n" &&
 	echo "sparse_entry text=auto" >.gitattributes &&
 	test_must_fail git add --renormalize sparse_entry 2>stderr &&
+	test_sparse_entry_unstaged &&
 	test_cmp error_and_hint stderr &&
 	test_sparse_entry_unchanged
 '

@@ -124,6 +135,7 @@ test_expect_success 'git add --dry-run --ignore-missing warn on sparse path' '
 	setup_sparse_entry &&
 	rm sparse_entry &&
 	test_must_fail git add --dry-run --ignore-missing sparse_entry 2>stderr &&
+	test_sparse_entry_unstaged &&
 	test_cmp error_and_hint stderr &&
 	test_sparse_entry_unchanged
 '

@@ -148,6 +160,7 @@ test_expect_success 'do not warn when pathspec matches dense entries' '
 test_expect_success 'add obeys advice.updateSparsePath' '
 	setup_sparse_entry &&
 	test_must_fail git -c advice.updateSparsePath=false add sparse_entry 2>stderr &&
+	test_sparse_entry_unstaged &&
 	test_cmp sparse_entry_error stderr
 
 '

2.33.0.vfs.signtest.55.g904c8365e91

Of course, the reason for the failures is because the 'sparse_entry' is staged as part of setup_sparse_entry. Not a bug.

This makes things more difficult to test, so I'll look around for an alternative way to test that 'git add' is behaving correctly.

Thanks,
-Stolee

Ok, this sent me off on a tangent (see other replies) trying to show that 'git add <sparse-path>' is still modifying index entries. I finally added enough checks that this fails in my merge/cherry-pick/rebase tests for conflicts outside of the sparse-checkout cone. Here is a test that I can add to t3705 to get a failure:

test_expect_success 'git add fails outside of sparse-checkout definition' '

	test_when_finished git sparse-checkout disable &&
	test_commit a &&
	git sparse-checkout init &&
	git sparse-checkout set a &&

	git update-index --no-skip-worktree sparse_entry &&
	test_must_fail git add sparse_entry &&
	test_sparse_entry_unstaged

'

OK, I'll make sure to include and test these cases.

I'll merge the patches to avoid confusion.

Thanks,
-Stolee

This series is based on ds/mergies-with-sparse-index.

As requested, this series looks to update the behavior of git add, git rm, and git mv when they attempt to modify paths outside of the sparse-checkout cone. In particular, this care is expanded to not just cache entries with the SKIP_WORKTREE bit, but also paths that do not match the sparse-checkout definition.

This means that commands that worked before this series can now fail. In particular, if 'git merge' results in a conflict outside of the sparse-checkout cone, then 'git add ' will now fail.

In order to allow users to circumvent these protections, a new '--sparse' option is added that ignores the sparse-checkout patterns and the SKIP_WORKTREE bit. The message for advice.updateSparsePath is adjusted to assist with discovery of this option.

There is a subtle issue with git mv in that it does not check the index until it discovers a directory and then uses the index to find the contained entries. This means that in non-cone-mode patterns, a pattern such as "sub/dir" will not match the path "sub" and this can cause an issue.

In order to allow for checking arbitrary paths against the sparse-checkout patterns, some changes to the underlying pattern matching code is required. It turns out that there are some bugs in the methods as advertised, but these bugs were never discovered because of the way methods like unpack_trees() will check a directory for a pattern match before checking its contained paths. Our new "check patterns on-demand" approach pokes holes in that approach, specifically with patterns that match entire directories.

Updates in v2
=============

* I got no complaints about these restrictions, so this is now a full series, not RFC.

* Thanks to Matheus, several holes are filled with extra testing and bugfixes.

* New patches add --chmod and --renormalize improvements. These are added after the --sparse option to make them be one change each.

Thanks, -Stolee

Derrick Stolee (14):
t3705: test that 'sparse_entry' is unstaged
t1092: behavior for adding sparse files
dir: extract directory-matching logic
dir: select directories correctly
dir: fix pattern matching on dirs
add: fail when adding an untracked sparse file
add: skip tracked paths outside sparse-checkout cone
add: implement the --sparse option
add: update --chmod to skip sparse paths
add: update --renormalize to skip sparse paths
rm: add --sparse option
rm: skip sparse paths with missing SKIP_WORKTREE
mv: refuse to move sparse paths
advice: update message to suggest '--sparse'

 Documentation/git-add.txt                |   9 +-
 Documentation/git-rm.txt                 |   6 +
 advice.c                                 |  10 +-
 builtin/add.c                            |  32 +++-
 builtin/mv.c                             |  52 +++++--
 builtin/rm.c                             |  10 +-
 dir.c                                    |  54 ++++++-
 pathspec.c                               |   5 +-
 t/t1091-sparse-checkout-builtin.sh       |   4 +-
 t/t1092-sparse-checkout-compatibility.sh |  75 +++++++--
 t/t3602-rm-sparse-checkout.sh            |  37 ++++-
 t/t3705-add-sparse-checkout.sh           |  65 +++++++-
 t/t7002-mv-sparse-checkout.sh            | 186 +++++++++++++++++++++++
 13 files changed, 494 insertions(+), 51 deletions(-)
 create mode 100755 t/t7002-mv-sparse-checkout.sh

base-commit: 516680ba7704c473bb21628aa19cabbd787df4db
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1018%2Fderrickstolee%2Fsparse-index%2Fadd-rm-mv-behavior-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1018/derrickstolee/sparse-index/add-rm-mv-behavior-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/1018

Range-diff vs v1:

  -:  ----------- >  1:  8aefce6254c t3705: test that 'sparse_entry' is unstaged
  1:  69ca54877b4 !  2:  61c23dc59a6 t1092: behavior for adding sparse files
     @@ Commit message
      
          Add some tests to demonstrate the current behavior around adding files
          outside of the sparse-checkout cone. Currently, untracked files are
     -    handled differently from tracked files. A futuer change will make these
     +    handled differently from tracked files. A future change will make these
          cases be handled the same way.
      
     +    Further expand checking that a failed 'git add' does not stage changes
     +    to the index.
     +
          Signed-off-by: Derrick Stolee <[email protected]>
      
       ## t/t1092-sparse-checkout-compatibility.sh ##
     +@@ t/t1092-sparse-checkout-compatibility.sh: test_sparse_match () {
     + 	test_cmp sparse-checkout-err sparse-index-err
     + }
     + 
     ++test_sparse_unstaged () {
     ++	file=$1 &&
     ++	for repo in sparse-checkout sparse-index
     ++	do
     ++		git -C $repo status --porcelain >$repo-out &&
     ++		! grep "^A  $file\$" $repo-out &&
     ++		! grep "^M  $file\$" $repo-out || return 1
     ++	done
     ++}
     ++
     + test_expect_success 'sparse-index contents' '
     + 	init_repos &&
     + 
      @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'add, commit, checkout' '
       	test_all_match git checkout -
       '
     @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'add, commit, chec
      +	run_on_sparse ../edit-contents folder1/a &&
      +	run_on_sparse ../edit-contents folder1/newfile &&
      +	test_sparse_match test_must_fail git add folder1/a &&
     ++	test_i18ngrep "Disable or modify the sparsity rules" sparse-checkout-err &&
     ++	test_sparse_unstaged folder1/a &&
      +	test_sparse_match git add folder1/newfile
      +'
      +
       test_expect_success 'commit including unstaged changes' '
       	init_repos &&
       
     +@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'status/add: outside sparse cone' '
     + 
     + 	# Adding the path outside of the sparse-checkout cone should fail.
     + 	test_sparse_match test_must_fail git add folder1/a &&
     ++	test_i18ngrep "Disable or modify the sparsity rules" sparse-checkout-err &&
     ++	test_sparse_unstaged folder1/a &&
     + 	test_sparse_match test_must_fail git add --refresh folder1/a &&
     ++	test_i18ngrep "Disable or modify the sparsity rules" sparse-checkout-err &&
     ++	test_sparse_unstaged folder1/a &&
     + 
     + 	# NEEDSWORK: Adding a newly-tracked file outside the cone succeeds
     + 	test_sparse_match git add folder1/new &&
  2:  8cca5bcf405 =  3:  747b52e7bde dir: extract directory-matching logic
  3:  d47c7a1cf2a =  4:  723de4e4258 dir: select directories correctly
  4:  58e7df44982 =  5:  ed2472d33f1 dir: fix pattern matching on dirs
  5:  e80fcfa932c !  6:  bac6cedfc8d add: fail when adding an untracked sparse file
     @@ Commit message
          want to avoid modifying the index outside of the sparse-checkout cone.
      
          Add a check within add_files() for these files and write the advice
     -    about files outside of the sprase-checkout cone.
     +    about files outside of the sparse-checkout cone.
      
          This behavior change modifies some existing tests within t1092. These
          tests intended to document how a user could interact with the existing
     @@ builtin/add.c: static void check_embedded_repo(const char *path)
       static int add_files(struct dir_struct *dir, int flags)
       {
       	int i, exit_status = 0;
     -+	struct string_list only_match_skip_worktree = STRING_LIST_INIT_NODUP;
     ++	struct string_list matched_sparse_paths = STRING_LIST_INIT_NODUP;
       
       	if (dir->ignored_nr) {
       		fprintf(stderr, _(ignore_error));
     @@ builtin/add.c: static int add_files(struct dir_struct *dir, int flags)
       
       	for (i = 0; i < dir->nr; i++) {
      +		if (!path_in_sparse_checkout(dir->entries[i]->name, &the_index)) {
     -+			string_list_append(&only_match_skip_worktree,
     ++			string_list_append(&matched_sparse_paths,
      +					   dir->entries[i]->name);
      +			continue;
      +		}
     @@ builtin/add.c: static int add_files(struct dir_struct *dir, int flags)
       		}
       	}
      +
     -+	if (only_match_skip_worktree.nr) {
     -+		advise_on_updating_sparse_paths(&only_match_skip_worktree);
     ++	if (matched_sparse_paths.nr) {
     ++		advise_on_updating_sparse_paths(&matched_sparse_paths);
      +		exit_status = 1;
      +	}
      +
     -+	string_list_clear(&only_match_skip_worktree, 0);
     ++	string_list_clear(&matched_sparse_paths, 0);
      +
       	return exit_status;
       }
     @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'add, commit, chec
       	init_repos &&
       
      @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'add outside sparse cone' '
     - 	run_on_sparse ../edit-contents folder1/a &&
     - 	run_on_sparse ../edit-contents folder1/newfile &&
       	test_sparse_match test_must_fail git add folder1/a &&
     + 	test_i18ngrep "Disable or modify the sparsity rules" sparse-checkout-err &&
     + 	test_sparse_unstaged folder1/a &&
      -	test_sparse_match git add folder1/newfile
     -+	test_sparse_match test_must_fail git add folder1/newfile
     ++	test_sparse_match test_must_fail git add folder1/newfile &&
     ++	test_i18ngrep "Disable or modify the sparsity rules" sparse-checkout-err &&
     ++	test_sparse_unstaged folder1/newfile
       '
       
       test_expect_success 'commit including unstaged changes' '
     @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'commit including
       
       	# folder1 is at HEAD, but outside the sparse cone
      @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'status/add: outside sparse cone' '
     - 	# Adding the path outside of the sparse-checkout cone should fail.
     - 	test_sparse_match test_must_fail git add folder1/a &&
       	test_sparse_match test_must_fail git add --refresh folder1/a &&
     + 	test_i18ngrep "Disable or modify the sparsity rules" sparse-checkout-err &&
     + 	test_sparse_unstaged folder1/a &&
      +	test_sparse_match test_must_fail git add folder1/new &&
     ++	test_i18ngrep "Disable or modify the sparsity rules" sparse-checkout-err &&
     ++	test_sparse_unstaged folder1/new &&
       
      -	# NEEDSWORK: Adding a newly-tracked file outside the cone succeeds
      -	test_sparse_match git add folder1/new &&
  6:  eeba97ad492 <  -:  ----------- add: skip paths that are outside sparse-checkout cone
  8:  f1764f9ed18 !  7:  d4706e17cae add: prevent adding sparse conflict files

@@ Metadata
Author: Derrick Stolee <[email protected]>

## Commit message ##
- add: prevent adding sparse conflict files
+ add: skip tracked paths outside sparse-checkout cone

- When a merge results in a conflict outside of the sparse-checkout cone,
- the conflicted file is written to the working tree and the index entry
- loses the SKIP_WORKTREE bit. This allows users to add the file to the
- index without realizing that the file might leave the working tree in a
- later Git command.
+ When 'git add' adds a tracked file that is outside of the
+ sparse-checkout cone, it checks the SKIP_WORKTREE bit to see if the file
+ exists outside of the sparse-checkout cone. This is usually correct,
+ except in the case of a merge conflict outside of the cone.

- Block this behavior, but keep in mind that the user can override the
- failure using the '--sparse' option.
+ Modify add_pathspec_matched_against_index() to be more careful about
+ paths by checking the sparse-checkout patterns in addition to the
+ SKIP_WORKTREE bit. This causes 'git add' to no longer allow files
+ outside of the cone that removed the SKIP_WORKTREE bit due to a merge
+ conflict.
+
+ With only this change, users will only be able to add the file after
+ adding the file to the sparse-checkout cone. A later change will allow
+ users to force adding even though the file is outside of the
+ sparse-checkout cone.

+ ## builtin/add.c ##
+@@ builtin/add.c: static void update_callback(struct diff_queue_struct *q,
+ for (i = 0; i < q->nr; i++) {
+ struct diff_filepair *p = q->queue[i];
+ const char *path = p->one->path;
++
++ if (!path_in_sparse_checkout(path, &the_index))
++ continue;
++
+ switch (fix_unmerged_status(p, data)) {
+ default:
+ die(_("unexpected diff status %c"), p->status);
+
## pathspec.c ##
+@@ pathspec.c: void add_pathspec_matches_against_index(const struct pathspec *pathspec,
+ return;
+ for (i = 0; i < istate->cache_nr; i++) {
+ const struct cache_entry *ce = istate->cache[i];
+- if (sw_action == PS_IGNORE_SKIP_WORKTREE && ce_skip_worktree(ce))
++ if (sw_action == PS_IGNORE_SKIP_WORKTREE &&
++ (ce_skip_worktree(ce) || !path_in_sparse_checkout(ce->name, istate)))
+ continue;
+ ce_path_match(istate, ce, pathspec, seen);
+ }
@@ pathspec.c: char *find_pathspecs_matching_skip_worktree(const struct pathspec *pathspec)

for (i = 0; i < istate->cache_nr; i++) {
@@ t/t1091-sparse-checkout-builtin.sh: test_expect_success 'sparse-checkout reapply
test_must_be_empty err &&

## t/t1092-sparse-checkout-compatibility.sh ##
-@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'merge with conflict outside cone' ' +@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'merge with conflict outside cone' '
test_all_match git status --porcelain=v2 &&

# 2. Add the file with conflict markers
- # NEEDSWORK: Even though the merge conflict removed the
- # SKIP_WORKTREE bit from the index entry for folder1/a, we should
-- # warn that this is a problematic add when --sparse is not set.
+- # warn that this is a problematic add.
+- test_all_match git add folder1/a &&
+ test_sparse_match test_must_fail git add folder1/a &&
+ test_i18ngrep "Disable or modify the sparsity rules" sparse-checkout-err &&
- test_all_match git add --sparse folder1/a &&
++ test_sparse_unstaged folder1/a &&
++ test_all_match git add --sparse folder1/a &&
test_all_match git status --porcelain=v2 &&

-@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'merge with conflict outside cone' '
- # accept conflict markers as resolved content.
- run_on_all mv folder2/a folder2/z &&
- test_sparse_match test_must_fail git add folder2 &&
+ # 3. Rename the file to another sparse filename and
+@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'merge with conflict outside cone' '
+ # NEEDSWORK: This mode now fails, because folder2/z is
+ # outside of the sparse-checkout cone and does not match an
+ # existing index entry with the SKIP_WORKTREE bit cleared.
+- test_all_match git add folder2 &&
++ test_sparse_match test_must_fail git add folder2 &&
+ test_i18ngrep "Disable or modify the sparsity rules" sparse-checkout-err &&
- test_all_match git add --sparse folder2 &&
++ test_sparse_unstaged folder2/z &&
++ test_all_match git add --sparse folder2 &&
test_all_match git status --porcelain=v2 &&

+ test_all_match git merge --continue &&
+@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'cherry-pick/rebase with conflict outside cone' '
+ # NEEDSWORK: Even though the merge conflict removed the
+ # SKIP_WORKTREE bit from the index entry for folder1/a, we should
+ # warn that this is a problematic add.
+- test_all_match git add folder1/a &&
++ test_sparse_match test_must_fail git add folder1/a &&
++ test_i18ngrep "Disable or modify the sparsity rules" sparse-checkout-err &&
++ test_sparse_unstaged folder1/a &&
+ test_all_match git status --porcelain=v2 &&
+
+ # 3. Rename the file to another sparse filename and
+@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'cherry-pick/rebase with conflict outside cone' '
+ # outside of the sparse-checkout cone and does not match an
+ # existing index entry with the SKIP_WORKTREE bit cleared.
+ run_on_all mv folder2/a folder2/z &&
+- test_all_match git add folder2 &&
++ test_sparse_match test_must_fail git add folder2 &&
++ test_i18ngrep "Disable or modify the sparsity rules" sparse-checkout-err &&
++ test_sparse_unstaged folder2/z &&
+ test_all_match git status --porcelain=v2 &&
+
+ test_all_match git $OPERATION --continue &&
+
+ ## t/t3705-add-sparse-checkout.sh ##
+@@ t/t3705-add-sparse-checkout.sh: test_expect_success 'do not warn when pathspec matches dense entries' '
+ git ls-files --error-unmatch dense_entry
+ '
+
++test_expect_success 'git add fails outside of sparse-checkout definition' '
++ test_when_finished git sparse-checkout disable &&
++ test_commit a &&
++ git sparse-checkout init &&
++ git sparse-checkout set a &&
++ echo >>sparse_entry &&
++
++ git update-index --no-skip-worktree sparse_entry &&
++ test_must_fail git add sparse_entry &&
++ test_sparse_entry_unstaged
++'
++
+ test_expect_success 'add obeys advice.updateSparsePath' '
+ setup_sparse_entry &&
+ test_must_fail git -c advice.updateSparsePath=false add sparse_entry 2>stderr &&
7: 6edadbc2000 ! 8: 944ae2cffa8 add: implement the --sparse option
@@ builtin/add.c: static int chmod_pathspec(struct pathspec *pathspec, char flip, i
continue;

if (pathspec && !ce_path_match(&the_index, ce, pathspec, NULL))
+@@ builtin/add.c: static void update_callback(struct diff_queue_struct *q,
+ struct diff_filepair *p = q->queue[i];
+ const char *path = p->one->path;
+
+- if (!path_in_sparse_checkout(path, &the_index))
++ if (!include_sparse && !path_in_sparse_checkout(path, &the_index))
+ continue;
+
+ switch (fix_unmerged_status(p, data)) {
@@ builtin/add.c: static struct option builtin_add_options[] = {
OPT_BOOL( 0 , "refresh", &refresh_only, N_("don't add, only refresh the index")),
OPT_BOOL( 0 , "ignore-errors", &ignore_add_errors, N_("just skip files which cannot be added because of errors")),
@@ builtin/add.c: static int add_files(struct dir_struct *dir, int flags)
- if (!path_in_sparse_checkout(dir->entries[i]->name, &the_index)) {
+ if (!include_sparse &&
+ !path_in_sparse_checkout(dir->entries[i]->name, &the_index)) {
- string_list_append(&only_match_skip_worktree,
+ string_list_append(&matched_sparse_paths,
dir->entries[i]->name);
continue;
@@ builtin/add.c: int cmd_add(int argc, const char **argv, const char *prefix)
@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'commit including

# folder1 is at HEAD, but outside the sparse cone
@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'status/add: outside sparse cone' '
- test_sparse_match test_must_fail git add folder1/a &&
- test_sparse_match test_must_fail git add --refresh folder1/a &&
test_sparse_match test_must_fail git add folder1/new &&
+ test_i18ngrep "Disable or modify the sparsity rules" sparse-checkout-err &&
+ test_sparse_unstaged folder1/new &&
+ test_sparse_match git add --sparse folder1/a &&
+ test_sparse_match git add --sparse folder1/new &&

@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'merge, cherry-pic

test_all_match git checkout -b merge-tip merge-left &&
@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'merge with conflict outside cone' '
- # 2. Add the file with conflict markers
- # NEEDSWORK: Even though the merge conflict removed the
- # SKIP_WORKTREE bit from the index entry for folder1/a, we should
-- # warn that this is a problematic add.
-- test_sparse_match test_must_fail git add folder1/a &&
-+ # warn that this is a problematic add when --sparse is not set.
-+ test_all_match git add --sparse folder1/a &&
- test_all_match git status --porcelain=v2 &&
-
# 3. Rename the file to another sparse filename and
# accept conflict markers as resolved content.
run_on_all mv folder2/a folder2/z &&
@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'merge with confli
- # outside of the sparse-checkout cone and does not match an
- # existing index entry with the SKIP_WORKTREE bit cleared.
test_sparse_match test_must_fail git add folder2 &&
-+ test_all_match git add --sparse folder2 &&
- test_all_match git status --porcelain=v2 &&
+ test_i18ngrep "Disable or modify the sparsity rules" sparse-checkout-err &&
+ test_sparse_unstaged folder2/z &&
+@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'merge with conflict outside cone' '
+ test_all_match git rev-parse HEAD^{tree}
+ '
+
+-test_expect_failure 'cherry-pick/rebase with conflict outside cone' ' ++test_expect_success 'cherry-pick/rebase with conflict outside cone' '
+ init_repos &&

- test_all_match git merge --continue &&
+ for OPERATION in cherry-pick rebase
+@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'cherry-pick/rebase with conflict outside cone' '
+ test_sparse_match test_must_fail git add folder1/a &&
+ test_i18ngrep "Disable or modify the sparsity rules" sparse-checkout-err &&
+ test_sparse_unstaged folder1/a &&
++ test_all_match git add --sparse folder1/a &&
+ test_all_match git status --porcelain=v2 &&
+
+ # 3. Rename the file to another sparse filename and
+@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'cherry-pick/rebase with conflict outside cone' '
+ test_sparse_match test_must_fail git add folder2 &&
+ test_i18ngrep "Disable or modify the sparsity rules" sparse-checkout-err &&
+ test_sparse_unstaged folder2/z &&
++ test_all_match git add --sparse folder2 &&
+ test_all_match git status --porcelain=v2 &&
+
+ test_all_match git $OPERATION --continue &&

## t/t3705-add-sparse-checkout.sh ##
+@@ t/t3705-add-sparse-checkout.sh: test_expect_success 'git add fails outside of sparse-checkout definition' ' +
+ git update-index --no-skip-worktree sparse_entry &&
+ test_must_fail git add sparse_entry &&
+- test_sparse_entry_unstaged
++ test_sparse_entry_unstaged &&
++
++ # Avoid munging CRLFs to avoid an error message
++ git -c core.autocrlf=input add --sparse sparse_entry 2>stderr &&
++ test_must_be_empty stderr &&
++ test-tool read-cache --table >actual &&
++ grep "^100644 blob.*sparse_entry\$" actual
+ '
+
+ test_expect_success 'add obeys advice.updateSparsePath' '
@@ t/t3705-add-sparse-checkout.sh: test_expect_success 'add obeys advice.updateSparsePath' '

'
@@ t/t3705-add-sparse-checkout.sh: test_expect_success 'add obeys advice.updateSpar
+ git sparse-checkout set a &&
+ echo modified >sparse_entry &&
+ test_must_fail git add sparse_entry &&
++ test_sparse_entry_unchanged &&
+ git add --sparse sparse_entry 2>stderr &&
+ test_must_be_empty stderr
+'

  -:  ----------- >  9:  7ace030c709 add: update --chmod to skip sparse paths
  -:  ----------- > 10:  f34be166097 add: update --renormalize to skip sparse paths
  9:  6c9c986ff43 ! 11:  fb3f86efa9b rm: add --sparse option

@@ Metadata
## Commit message ##
rm: add --sparse option

+ As we did previously in 'git add', add a '--sparse' option to 'git rm'
+ that allows modifying paths outside of the sparse-checkout definition.
+ The existing checks in 'git rm' are restricted to tracked files that
+ have the SKIP_WORKTREE bit in the current index. Future changes will
+ cause 'git rm' to reject removing paths outside of the sparse-checkout
+ definition, even if they are untracked or do not have the SKIP_WORKTREE
+ bit.
+

## Documentation/git-rm.txt ##
@@ builtin/rm.c: int cmd_rm(int argc, const char **argv, const char *prefix)
die(_("pathspec '%s' did not match any files"), original);

## t/t3602-rm-sparse-checkout.sh ##
-@@ t/t3602-rm-sparse-checkout.sh: done
-
- test_expect_success 'recursive rm does not remove sparse entries' '
- git reset --hard &&
-- git sparse-checkout set sub/dir &&
-+ git sparse-checkout set sub/dir/ &&
- git rm -r sub &&
- git status --porcelain -uno >actual &&
- echo "D sub/dir/e" >expected &&
+@@ t/t3602-rm-sparse-checkout.sh: test_expect_success 'recursive rm does not remove sparse entries' '
test_cmp expected actual
'

10: 5153accded4 ! 12: de0c9d09ef7 rm: skip sparse paths with missing SKIP_WORKTREE

     @@ Commit message
          if the '--sparse' option is specified, allowing users who accept the
          risks to continue with the removal.
      
     +    This also removes a confusing behavior where a user asks for a directory
     +    to be removed, but only the entries that are within the sparse-checkout
     +    definition are removed. Now, 'git rm <dir>' will fail without '--sparse'
     +    and will succeed in removing all contained paths with '--sparse'.
     +
          Signed-off-by: Derrick Stolee <[email protected]>
      
       ## builtin/rm.c ##
     @@ builtin/rm.c: int cmd_rm(int argc, const char **argv, const char *prefix)
       			continue;
      
       ## t/t3602-rm-sparse-checkout.sh ##
     +@@ t/t3602-rm-sparse-checkout.sh: done
     + test_expect_success 'recursive rm does not remove sparse entries' '
     + 	git reset --hard &&
     + 	git sparse-checkout set sub/dir &&
     +-	git rm -r sub &&
     ++	test_must_fail git rm -r sub &&
     ++	git rm --sparse -r sub &&
     + 	git status --porcelain -uno >actual &&
     +-	echo "D  sub/dir/e" >expected &&
     ++	cat >expected <<-\EOF &&
     ++	D  sub/d
     ++	D  sub/dir/e
     ++	EOF
     + 	test_cmp expected actual
     + '
     + 
      @@ t/t3602-rm-sparse-checkout.sh: test_expect_success 'do not warn about sparse entries with --ignore-unmatch' '
       	git ls-files --error-unmatch b
       '
 11:  d31c6411806 ! 13:  e17a529332d mv: refuse to move sparse paths
     @@ Commit message
          mechanism for seeing if a path is sparse or not. Be sure to skip
          returning a failure if '-k' is specified.
      
     +    To ensure that the advice around sparse paths is the only reason a move
     +    failed, be sure to check this as the very last thing before inserting
     +    into the src_for_dst list.
     +
     +    The tests cover a variety of cases such as whether the target is tracked
     +    or untracked, and whether the source or destination are in or outside of
     +    the sparse-checkout definition.
     +
     +    Helped-by: Matheus Tavares Bernardino <[email protected]>
          Signed-off-by: Derrick Stolee <[email protected]>
      
       ## builtin/mv.c ##
     -@@ builtin/mv.c: int cmd_mv(int argc, const char **argv, const char *prefix)
     +@@ builtin/mv.c: static int index_range_of_same_dir(const char *src, int length,
     + int cmd_mv(int argc, const char **argv, const char *prefix)
     + {
     + 	int i, flags, gitmodules_modified = 0;
     +-	int verbose = 0, show_only = 0, force = 0, ignore_errors = 0;
     ++	int verbose = 0, show_only = 0, force = 0, ignore_errors = 0, ignore_sparse = 0;
     + 	struct option builtin_mv_options[] = {
     + 		OPT__VERBOSE(&verbose, N_("be verbose")),
     + 		OPT__DRY_RUN(&show_only, N_("dry run")),
     + 		OPT__FORCE(&force, N_("force move/rename even if target exists"),
     + 			   PARSE_OPT_NOCOMPLETE),
     + 		OPT_BOOL('k', NULL, &ignore_errors, N_("skip move/rename errors")),
     ++		OPT_BOOL(0, "sparse", &ignore_sparse, N_("allow updating entries outside of the sparse-checkout cone")),
     + 		OPT_END(),
     + 	};
     + 	const char **source, **destination, **dest_path, **submodule_gitfile;
     +-	enum update_mode { BOTH = 0, WORKING_DIRECTORY, INDEX } *modes;
     ++	enum update_mode { BOTH = 0, WORKING_DIRECTORY, INDEX, SPARSE } *modes;
     + 	struct stat st;
       	struct string_list src_for_dst = STRING_LIST_INIT_NODUP;
       	struct lock_file lock_file = LOCK_INIT;
       	struct cache_entry *ce;
     @@ builtin/mv.c: int cmd_mv(int argc, const char **argv, const char *prefix)
       		if (show_only)
       			printf(_("Checking rename of '%s' to '%s'\n"), src, dst);
       
     -+		if (!path_in_sparse_checkout(src, &the_index)) {
     -+			string_list_append(&only_match_skip_worktree, src);
     -+			skip_sparse = 1;
     -+		}
     -+		if (!path_in_sparse_checkout(dst, &the_index)) {
     -+			string_list_append(&only_match_skip_worktree, dst);
     -+			skip_sparse = 1;
     -+		}
     -+		if (skip_sparse)
     -+			continue;
     -+
       		length = strlen(src);
     - 		if (lstat(src, &st) < 0)
     - 			bad = _("bad source");
     +-		if (lstat(src, &st) < 0)
     +-			bad = _("bad source");
     +-		else if (!strncmp(src, dst, length) &&
     ++		if (lstat(src, &st) < 0) {
     ++			/* only error if existence is expected. */
     ++			if (modes[i] != SPARSE)
     ++				bad = _("bad source");
     ++		} else if (!strncmp(src, dst, length) &&
     + 				(dst[length] == 0 || dst[length] == '/')) {
     + 			bad = _("can not move directory into itself");
     + 		} else if ((src_is_dir = S_ISDIR(st.st_mode))
     +@@ builtin/mv.c: int cmd_mv(int argc, const char **argv, const char *prefix)
     + 				dst_len = strlen(dst);
     + 
     + 				for (j = 0; j < last - first; j++) {
     +-					const char *path = active_cache[first + j]->name;
     ++					const struct cache_entry *ce = active_cache[first + j];
     ++					const char *path = ce->name;
     + 					source[argc + j] = path;
     + 					destination[argc + j] =
     + 						prefix_path(dst, dst_len, path + length + 1);
     +-					modes[argc + j] = INDEX;
     ++					modes[argc + j] = ce_skip_worktree(ce) ? SPARSE : INDEX;
     + 					submodule_gitfile[argc + j] = NULL;
     + 				}
     + 				argc += last - first;
     +@@ builtin/mv.c: int cmd_mv(int argc, const char **argv, const char *prefix)
     + 			bad = _("multiple sources for the same target");
     + 		else if (is_dir_sep(dst[strlen(dst) - 1]))
     + 			bad = _("destination directory does not exist");
     +-		else
     ++		else {
     ++			/*
     ++			 * We check if the paths are in the sparse-checkout
     ++			 * definition as a very final check, since that
     ++			 * allows us to point the user to the --sparse
     ++			 * option as a way to have a successful run.
     ++			 */
     ++			if (!ignore_sparse &&
     ++			    !path_in_sparse_checkout(src, &the_index)) {
     ++				string_list_append(&only_match_skip_worktree, src);
     ++				skip_sparse = 1;
     ++			}
     ++			if (!ignore_sparse &&
     ++			    !path_in_sparse_checkout(dst, &the_index)) {
     ++				string_list_append(&only_match_skip_worktree, dst);
     ++				skip_sparse = 1;
     ++			}
     ++
     ++			if (skip_sparse)
     ++				goto remove_entry;
     ++
     + 			string_list_insert(&src_for_dst, dst);
     ++		}
     + 
     + 		if (!bad)
     + 			continue;
     + 		if (!ignore_errors)
     + 			die(_("%s, source=%s, destination=%s"),
     + 			     bad, src, dst);
     ++remove_entry:
     + 		if (--argc > 0) {
     + 			int n = argc - i;
     + 			memmove(source + i, source + i + 1,
      @@ builtin/mv.c: int cmd_mv(int argc, const char **argv, const char *prefix)
       		}
       	}
     @@ builtin/mv.c: int cmd_mv(int argc, const char **argv, const char *prefix)
       	for (i = 0; i < argc; i++) {
       		const char *src = source[i], *dst = destination[i];
       		enum update_mode mode = modes[i];
     +@@ builtin/mv.c: int cmd_mv(int argc, const char **argv, const char *prefix)
     + 			printf(_("Renaming %s to %s\n"), src, dst);
     + 		if (show_only)
     + 			continue;
     +-		if (mode != INDEX && rename(src, dst) < 0) {
     ++		if (mode != INDEX && mode != SPARSE && rename(src, dst) < 0) {
     + 			if (ignore_errors)
     + 				continue;
     + 			die_errno(_("renaming '%s' failed"), src);
      
       ## t/t7002-mv-sparse-checkout.sh (new) ##
      @@
     @@ t/t7002-mv-sparse-checkout.sh (new)
      +"
      +
      +test_expect_success 'mv refuses to move sparse-to-sparse' '
     -+	rm -f e &&
     ++	test_when_finished rm -f e &&
      +	git reset --hard &&
      +	git sparse-checkout set a &&
      +	touch b &&
     @@ t/t7002-mv-sparse-checkout.sh (new)
      +	echo b >>expect &&
      +	echo e >>expect &&
      +	cat sparse_hint >>expect &&
     -+	test_cmp expect stderr
     ++	test_cmp expect stderr &&
     ++	git mv --sparse b e 2>stderr &&
     ++	test_must_be_empty stderr
      +'
      +
      +test_expect_success 'mv refuses to move sparse-to-sparse, ignores failure' '
     -+	rm -f e &&
     ++	test_when_finished rm -f b c e &&
      +	git reset --hard &&
      +	git sparse-checkout set a &&
     ++
     ++	# tracked-to-untracked
      +	touch b &&
      +	git mv -k b e 2>stderr &&
     ++	test_path_exists b &&
     ++	test_path_is_missing e &&
      +	cat sparse_error_header >expect &&
      +	echo b >>expect &&
      +	echo e >>expect &&
      +	cat sparse_hint >>expect &&
     -+	test_cmp expect stderr
     ++	test_cmp expect stderr &&
     ++
     ++	git mv --sparse b e 2>stderr &&
     ++	test_must_be_empty stderr &&
     ++	test_path_is_missing b &&
     ++	test_path_exists e &&
     ++
     ++	# tracked-to-tracked
     ++	git reset --hard &&
     ++	touch b &&
     ++	git mv -k b c 2>stderr &&
     ++	test_path_exists b &&
     ++	test_path_is_missing c &&
     ++	cat sparse_error_header >expect &&
     ++	echo b >>expect &&
     ++	echo c >>expect &&
     ++	cat sparse_hint >>expect &&
     ++	test_cmp expect stderr &&
     ++
     ++	git mv --sparse b c 2>stderr &&
     ++	test_must_be_empty stderr &&
     ++	test_path_is_missing b &&
     ++	test_path_exists c
      +'
      +
      +test_expect_success 'mv refuses to move non-sparse-to-sparse' '
     -+	rm -f e &&
     ++	test_when_finished rm -f b c e &&
      +	git reset --hard &&
      +	git sparse-checkout set a &&
     ++
     ++	# tracked-to-untracked
      +	test_must_fail git mv a e 2>stderr &&
     ++	test_path_exists a &&
     ++	test_path_is_missing e &&
      +	cat sparse_error_header >expect &&
      +	echo e >>expect &&
      +	cat sparse_hint >>expect &&
     -+	test_cmp expect stderr
     ++	test_cmp expect stderr &&
     ++	git mv --sparse a e 2>stderr &&
     ++	test_must_be_empty stderr &&
     ++	test_path_is_missing a &&
     ++	test_path_exists e &&
     ++
     ++	# tracked-to-tracked
     ++	rm e &&
     ++	git reset --hard &&
     ++	test_must_fail git mv a c 2>stderr &&
     ++	test_path_exists a &&
     ++	test_path_is_missing c &&
     ++	cat sparse_error_header >expect &&
     ++	echo c >>expect &&
     ++	cat sparse_hint >>expect &&
     ++	test_cmp expect stderr &&
     ++	git mv --sparse a c 2>stderr &&
     ++	test_must_be_empty stderr &&
     ++	test_path_is_missing a &&
     ++	test_path_exists c
      +'
      +
      +test_expect_success 'mv refuses to move sparse-to-non-sparse' '
     -+	rm -f e &&
     ++	test_when_finished rm -f b c e &&
      +	git reset --hard &&
      +	git sparse-checkout set a e &&
     ++
     ++	# tracked-to-untracked
      +	touch b &&
      +	test_must_fail git mv b e 2>stderr &&
      +	cat sparse_error_header >expect &&
      +	echo b >>expect &&
      +	cat sparse_hint >>expect &&
     -+	test_cmp expect stderr
     ++	test_cmp expect stderr &&
     ++	git mv --sparse b e 2>stderr &&
     ++	test_must_be_empty stderr
      +'
      +
      +test_expect_success 'recursive mv refuses to move (possible) sparse' '
     -+	rm -f e &&
     ++	test_when_finished rm -rf b c e sub2 &&
      +	git reset --hard &&
      +	# Without cone mode, "sub" and "sub2" do not match
      +	git sparse-checkout set sub/dir sub2/dir &&
     ++
     ++	# Add contained contents to ensure we avoid non-existence errors
     ++	mkdir sub/dir2 &&
     ++	touch sub/d sub/dir2/e &&
     ++
      +	test_must_fail git mv sub sub2 2>stderr &&
      +	cat sparse_error_header >expect &&
     -+	echo sub >>expect &&
     -+	echo sub2 >>expect &&
     ++	cat >>expect <<-\EOF &&
     ++	sub/d
     ++	sub2/d
     ++	sub/dir/e
     ++	sub2/dir/e
     ++	sub/dir2/e
     ++	sub2/dir2/e
     ++	EOF
      +	cat sparse_hint >>expect &&
     -+	test_cmp expect stderr
     ++	test_cmp expect stderr &&
     ++	git mv --sparse sub sub2 2>stderr &&
     ++	test_must_be_empty stderr &&
     ++	git commit -m "moved sub to sub2" &&
     ++	git rev-parse HEAD~1:sub >expect &&
     ++	git rev-parse HEAD:sub2 >actual &&
     ++	test_cmp expect actual &&
     ++	git reset --hard HEAD~1
      +'
      +
      +test_expect_success 'recursive mv refuses to move sparse' '
     @@ t/t7002-mv-sparse-checkout.sh (new)
      +	# Use cone mode so "sub/" matches the sparse-checkout patterns
      +	git sparse-checkout init --cone &&
      +	git sparse-checkout set sub/dir sub2/dir &&
     ++
     ++	# Add contained contents to ensure we avoid non-existence errors
     ++	mkdir sub/dir2 &&
     ++	touch sub/dir2/e &&
     ++
      +	test_must_fail git mv sub sub2 2>stderr &&
      +	cat sparse_error_header >expect &&
     -+	echo sub/dir2/e >>expect &&
     -+	echo sub2/dir2/e >>expect &&
     ++	cat >>expect <<-\EOF &&
     ++	sub/dir2/e
     ++	sub2/dir2/e
     ++	EOF
      +	cat sparse_hint >>expect &&
     -+	test_cmp expect stderr
     ++	test_cmp expect stderr &&
     ++	git mv --sparse sub sub2 2>stderr &&
     ++	test_must_be_empty stderr &&
     ++	git commit -m "moved sub to sub2" &&
     ++	git rev-parse HEAD~1:sub >expect &&
     ++	git rev-parse HEAD:sub2 >actual &&
     ++	test_cmp expect actual &&
     ++	git reset --hard HEAD~1
      +'
      +
      +test_done
 12:  f6c0d4e3a06 <  -:  ----------- mv: add '--sparse' option to ignore sparse-checkout
 13:  7749a69a38f ! 14:  f2abc387822 advice: update message to suggest '--sparse'

@@ Commit message
To improve the discoverability of this new functionality, add a message to advice.updateSparsePath that mentions the existence of the option.

+ The previous set of changes also modified the purpose of this message to
+ include possibly a list of paths instead of only a list of pathspecs.
+ Make the warning message more clear about this new behavior.
+

## advice.c ##
@@ advice.c: void advise_on_updating_sparse_paths(struct string_list *pathspec_list)
+ if (!pathspec_list->nr)
+ return;
+
+- fprintf(stderr, _("The following pathspecs didn't match any"
+- " eligible path, but they do match index\n"
+- "entries outside the current sparse checkout:\n"));
++ fprintf(stderr, _("The following paths and/or pathspecs matched "
++ "paths that exist outside of your\n"
++ "sparse-checkout definition, so will not be "
++ "updated in the index:\n"));
+ for_each_string_list_item(item, pathspec_list)
fprintf(stderr, "%s\n", item->string);

advise_if_enabled(ADVICE_UPDATE_SPARSE_PATH,
@@ advice.c: void advise_on_updating_sparse_paths(struct string_list *pathspec_list

## t/t3602-rm-sparse-checkout.sh ##
@@ t/t3602-rm-sparse-checkout.sh: test_expect_success 'setup' "
+ git commit -m files &&
+
+ cat >sparse_error_header <<-EOF &&
+- The following pathspecs didn't match any eligible path, but they do match index
+- entries outside the current sparse checkout:
++ The following paths and/or pathspecs matched paths that exist outside of your
++ sparse-checkout definition, so will not be updated in the index:
EOF

cat >sparse_hint <<-EOF &&
@@ t/t3602-rm-sparse-checkout.sh: test_expect_success 'setup' "

## t/t3705-add-sparse-checkout.sh ##
-@@ t/t3705-add-sparse-checkout.sh: test_expect_success 'setup' " +@@ t/t3705-add-sparse-checkout.sh: test_sparse_entry_unstaged () { +
+ test_expect_success 'setup' "
+ cat >sparse_error_header <<-EOF &&
+- The following pathspecs didn't match any eligible path, but they do match index
+- entries outside the current sparse checkout:
++ The following paths and/or pathspecs matched paths that exist outside of your
++ sparse-checkout definition, so will not be updated in the index:
EOF

cat >sparse_hint <<-EOF &&
@@ t/t3705-add-sparse-checkout.sh: test_expect_success 'setup' "

## t/t7002-mv-sparse-checkout.sh ##
@@ t/t7002-mv-sparse-checkout.sh: test_expect_success 'setup' "
+ git commit -m files &&
+
+ cat >sparse_error_header <<-EOF &&
+- The following pathspecs didn't match any eligible path, but they do match index
+- entries outside the current sparse checkout:
++ The following paths and/or pathspecs matched paths that exist outside of your
++ sparse-checkout definition, so will not be updated in the index:
EOF

cat >sparse_hint <<-EOF

gitgitgadget

From: Derrick Stolee <[email protected]>

The tests in t3705-add-sparse-checkout.sh check to see how 'git add' behaves with paths outside the sparse-checkout definition. These currently check to see if a given warning is present but not that the index is not updated with the sparse entries. Add a new 'test_sparse_entry_unstaged' helper to be sure 'git add' is behaving correctly.

We need to modify setup_sparse_entry to actually commit the sparse_entry file so it exists at HEAD but is not already staged in the index.

---
 t/t3705-add-sparse-checkout.sh | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

index 2b1fd0d0eef..af81b4b6846 100755
@@ -19,6 +19,7 @@ setup_sparse_entry () {
 	fi &&
 	git add sparse_entry &&
 	git update-index --skip-worktree sparse_entry &&
+	git commit --allow-empty -m "ensure sparse_entry exists at HEAD" &&
 	SPARSE_ENTRY_BLOB=$(git rev-parse :sparse_entry)
 }
 

@@ -36,6 +37,11 @@ setup_gitignore () {
 	EOF
 }
 
+test_sparse_entry_unstaged () {
+	git status --porcelain >actual &&
+	! grep "^M  sparse_entry\$" actual
+}
+
 test_expect_success 'setup' "
 	cat >sparse_error_header <<-EOF &&
 	The following pathspecs didn't match any eligible path, but they do match index

@@ -55,6 +61,7 @@ test_expect_success 'git add does not remove sparse entries' '
 	setup_sparse_entry &&
 	rm sparse_entry &&
 	test_must_fail git add sparse_entry 2>stderr &&
+	test_sparse_entry_unstaged &&
 	test_cmp error_and_hint stderr &&
 	test_sparse_entry_unchanged
 '

@@ -73,6 +80,7 @@ test_expect_success 'git add . does not remove sparse entries' '
 	rm sparse_entry &&
 	setup_gitignore &&
 	test_must_fail git add . 2>stderr &&
+	test_sparse_entry_unstaged &&
 
 	cat sparse_error_header >expect &&
 	echo . >>expect &&

@@ -88,6 +96,7 @@ do
 		setup_sparse_entry &&
 		echo modified >sparse_entry &&
 		test_must_fail git add $opt sparse_entry 2>stderr &&
+		test_sparse_entry_unstaged &&
 		test_cmp error_and_hint stderr &&
 		test_sparse_entry_unchanged
 	'

@@ -98,6 +107,7 @@ test_expect_success 'git add --refresh does not update sparse entries' '
 	git ls-files --debug sparse_entry | grep mtime >before &&
 	test-tool chmtime -60 sparse_entry &&
 	test_must_fail git add --refresh sparse_entry 2>stderr &&
+	test_sparse_entry_unstaged &&
 	test_cmp error_and_hint stderr &&
 	git ls-files --debug sparse_entry | grep mtime >after &&
 	test_cmp before after

@@ -106,6 +116,7 @@ test_expect_success 'git add --refresh does not update sparse entries' '
 test_expect_success 'git add --chmod does not update sparse entries' '
 	setup_sparse_entry &&
 	test_must_fail git add --chmod=+x sparse_entry 2>stderr &&
+	test_sparse_entry_unstaged &&
 	test_cmp error_and_hint stderr &&
 	test_sparse_entry_unchanged &&
 	! test -x sparse_entry

@@ -116,6 +127,7 @@ test_expect_success 'git add --renormalize does not update sparse entries' '
 	setup_sparse_entry "LINEONE\r\nLINETWO\r\n" &&
 	echo "sparse_entry text=auto" >.gitattributes &&
 	test_must_fail git add --renormalize sparse_entry 2>stderr &&
+	test_sparse_entry_unstaged &&
 	test_cmp error_and_hint stderr &&
 	test_sparse_entry_unchanged
 '

@@ -124,6 +136,7 @@ test_expect_success 'git add --dry-run --ignore-missing warn on sparse path' '
 	setup_sparse_entry &&
 	rm sparse_entry &&
 	test_must_fail git add --dry-run --ignore-missing sparse_entry 2>stderr &&
+	test_sparse_entry_unstaged &&
 	test_cmp error_and_hint stderr &&
 	test_sparse_entry_unchanged
 '

@@ -148,6 +161,7 @@ test_expect_success 'do not warn when pathspec matches dense entries' '
 test_expect_success 'add obeys advice.updateSparsePath' '
 	setup_sparse_entry &&
 	test_must_fail git -c advice.updateSparsePath=false add sparse_entry 2>stderr &&
+	test_sparse_entry_unstaged &&
 	test_cmp sparse_entry_error stderr
 
 '

gitgitgadget

From: Derrick Stolee <[email protected]>

Add some tests to demonstrate the current behavior around adding files outside of the sparse-checkout cone. Currently, untracked files are handled differently from tracked files. A future change will make these cases be handled the same way.

Further expand checking that a failed 'git add' does not stage changes to the index.

---
 t/t1092-sparse-checkout-compatibility.sh | 28 ++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

index 886e78715fe..5b3f55e355e 100755
@@ -187,6 +187,16 @@ test_sparse_match () {
 	test_cmp sparse-checkout-err sparse-index-err
 }
 
+test_sparse_unstaged () {
+	file=$1 &&
+	for repo in sparse-checkout sparse-index
+	do
+		git -C $repo status --porcelain >$repo-out &&
+		! grep "^A  $file\$" $repo-out &&
+		! grep "^M  $file\$" $repo-out || return 1
+	done
+}
+
 test_expect_success 'sparse-index contents' '
 	init_repos &&
 

@@ -291,6 +301,20 @@ test_expect_success 'add, commit, checkout' '
 	test_all_match git checkout -
 '
 
+# NEEDSWORK: This documents current behavior, but is not a desirable
+# behavior (untracked files are handled differently than tracked).
+test_expect_success 'add outside sparse cone' '
+	init_repos &&
+
+	run_on_sparse mkdir folder1 &&
+	run_on_sparse ../edit-contents folder1/a &&
+	run_on_sparse ../edit-contents folder1/newfile &&
+	test_sparse_match test_must_fail git add folder1/a &&
+	test_i18ngrep "Disable or modify the sparsity rules" sparse-checkout-err &&
+	test_sparse_unstaged folder1/a &&
+	test_sparse_match git add folder1/newfile
+'
+
 test_expect_success 'commit including unstaged changes' '
 	init_repos &&
 

@@ -339,7 +363,11 @@ test_expect_success 'status/add: outside sparse cone' '
 
 	# Adding the path outside of the sparse-checkout cone should fail.
 	test_sparse_match test_must_fail git add folder1/a &&
+	test_i18ngrep "Disable or modify the sparsity rules" sparse-checkout-err &&
+	test_sparse_unstaged folder1/a &&
 	test_sparse_match test_must_fail git add --refresh folder1/a &&
+	test_i18ngrep "Disable or modify the sparsity rules" sparse-checkout-err &&
+	test_sparse_unstaged folder1/a &&
 
 	# NEEDSWORK: Adding a newly-tracked file outside the cone succeeds
 	test_sparse_match git add folder1/new &&

gitgitgadget

From: Derrick Stolee <[email protected]>

The last_matching_pattern_from_list() logic performs some checks on the filetype of a path within the index when the PATTERN_FLAG_MUSTBEDIR flag is set. This works great when setting SKIP_WORKTREE bits within unpack_trees(), but doesn't work well when passing an arbitrary path such as a file within a matching directory.

This change only rearranges the logic but does not change its functionality.

We will expand the path_matches_dir_pattern() method in a following change.

---
 dir.c | 22 +++++++++++++++++-----
 1 file changed, 17 insertions(+), 5 deletions(-)

index 86afa2eae00..652135df896 100644
@@ -1303,6 +1303,19 @@ int match_pathname(const char *pathname, int pathlen,
 				 WM_PATHNAME) == 0;
 }
 
+static int path_matches_dir_pattern(const char *pathname,
+				    int pathlen,
+				    int *dtype,
+				    struct path_pattern *pattern,
+				    struct index_state *istate)
+{
+	*dtype = resolve_dtype(*dtype, istate, pathname, pathlen);
+	if (*dtype != DT_DIR)
+		return 0;
+
+	return 1;
+}
+
 /*
  * Scan the given exclude list in reverse to see whether pathname
  * should be ignored.  The first match (i.e. the last on the list), if

@@ -1327,11 +1340,10 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname
 		const char *exclude = pattern->pattern;
 		int prefix = pattern->nowildcardlen;
 
-		if (pattern->flags & PATTERN_FLAG_MUSTBEDIR) {
-			*dtype = resolve_dtype(*dtype, istate, pathname, pathlen);
-			if (*dtype != DT_DIR)
-				continue;
-		}
+		if ((pattern->flags & PATTERN_FLAG_MUSTBEDIR) &&
+		    !path_matches_dir_pattern(pathname, pathlen,
+					      dtype, pattern, istate))
+			continue;
 
 		if (pattern->flags & PATTERN_FLAG_NODIR) {
 			if (match_basename(basename,

gitgitgadget

From: Derrick Stolee <[email protected]>

When matching a path against a list of patterns, the ones that require a directory match previously did not work when a filename is specified. This was fine when all pattern-matching was done within methods such as unpack_trees() that check a directory before recursing into the contained files. However, other commands will start matching individual files against pattern lists without that recursive approach.

We modify path_matches_dir_pattern() to take a strbuf 'path_parent' that is used to store the parent directory of 'pathname' between multiple pattern matching tests. This is loaded lazily, only on the first pattern it finds that has the PATTERN_FLAG_MUSTBEDIR flag.

If we find that a path has a parent directory, we start by checking to see if that parent directory matches the pattern. If so, then we do not need to query the index for the type (which can be expensive). If we find that the parent does not match, then we still must check the type from the index for the given pathname.

Note that this does not affect cone mode pattern matching, but instead the more general -- and slower -- full pattern set. Thus, this does not affect the sparse index.

---
 dir.c | 34 ++++++++++++++++++++++++++++++++--
 1 file changed, 32 insertions(+), 2 deletions(-)

index 652135df896..fe5ee87bb5f 100644
@@ -1305,10 +1305,38 @@ int match_pathname(const char *pathname, int pathlen,
 
 static int path_matches_dir_pattern(const char *pathname,
 				    int pathlen,
+				    struct strbuf *path_parent,
 				    int *dtype,
 				    struct path_pattern *pattern,
 				    struct index_state *istate)
 {
+	/*
+	 * Use 'alloc' as an indicator that the string has not been
+	 * initialized, in case the parent is the root directory.
+	 */
+	if (!path_parent->alloc) {
+		char *slash;
+		strbuf_addstr(path_parent, pathname);
+		slash = find_last_dir_sep(path_parent->buf);
+
+		if (slash)
+			*slash = '\0';
+		else
+			strbuf_setlen(path_parent, 0);
+	}
+
+	/*
+	 * If the parent directory matches the pattern, then we do not
+	 * need to check for dtype.
+	 */
+	if (path_parent->len &&
+	    match_pathname(path_parent->buf, path_parent->len,
+			   pattern->base,
+			   pattern->baselen ? pattern->baselen - 1 : 0,
+			   pattern->pattern, pattern->nowildcardlen,
+			   pattern->patternlen, pattern->flags))
+		return 1;
+
 	*dtype = resolve_dtype(*dtype, istate, pathname, pathlen);
 	if (*dtype != DT_DIR)
 		return 0;

@@ -1331,6 +1359,7 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname
 {
 	struct path_pattern *res = NULL; /* undecided */
 	int i;
+	struct strbuf path_parent = STRBUF_INIT;
 
 	if (!pl->nr)
 		return NULL;	/* undefined */

@@ -1340,8 +1369,8 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname
 		const char *exclude = pattern->pattern;
 		int prefix = pattern->nowildcardlen;
 
-		if ((pattern->flags & PATTERN_FLAG_MUSTBEDIR) &&
-		    !path_matches_dir_pattern(pathname, pathlen,
+		if (pattern->flags & PATTERN_FLAG_MUSTBEDIR &&
+		    !path_matches_dir_pattern(pathname, pathlen, &path_parent,
 					      dtype, pattern, istate))
 			continue;
 

@@ -1367,6 +1396,7 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname
 			break;
 		}
 	}
+	strbuf_release(&path_parent);
 	return res;
 }
 

gitgitgadget

From: Derrick Stolee <[email protected]>

Within match_pathname(), one successful matching category happens when the pattern is equal to its non-wildcard prefix. At this point, we have checked that the input 'pathname' matches the pattern up to the prefix length, and then we subtraced that length from both 'patternlen' and 'namelen'.

In the case of a directory match, this prefix match should be sufficient. However, the success condition only cared about _exact_ equality here. Instead, we should allow any path that agrees on this prefix in the case of PATTERN_FLAG_MUSTBEDIR.

This case was not tested before because of the way unpack_trees() would match a parent directory before visiting the contained paths. This approach is changing, so we must change this comparison.

---
 dir.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index fe5ee87bb5f..80fd0ad2fc0 100644
@@ -1294,7 +1294,7 @@ int match_pathname(const char *pathname, int pathlen,
 		 * then our prefix match is all we need; we
 		 * do not need to call fnmatch at all.
 		 */
-		if (!patternlen && !namelen)
+		if (!patternlen && (!namelen || (flags & PATTERN_FLAG_MUSTBEDIR)))
 			return 1;
 	}
 

gitgitgadget

From: Derrick Stolee <[email protected]>

The add_files() method in builtin/add.c takes a set of untracked files that are being added by the input pathspec and inserts them into the index. If these files are outside of the sparse-checkout cone, then they gain the SKIP_WORKTREE bit at some point. However, this was not checked before inserting into the index, so these files are added even though we want to avoid modifying the index outside of the sparse-checkout cone.

Add a check within add_files() for these files and write the advice about files outside of the sparse-checkout cone.

This behavior change modifies some existing tests within t1092. These tests intended to document how a user could interact with the existing behavior in place. Many of these tests need to be marked as expecting failure. A future change will allow these tests to pass by adding a flag to 'git add' that allows users to modify index entries outside of the sparse-checkout cone.

The 'submodule handling' test is intended to document what happens to directories that contain a submodule when the sparse index is enabled. It is not trying to say that users should be able to add submodules outside of the sparse-checkout cone, so that test can be modified to avoid that operation.

---
 builtin/add.c                            | 14 +++++++++
 t/t1092-sparse-checkout-compatibility.sh | 37 ++++++++++++++++++------
 2 files changed, 42 insertions(+), 9 deletions(-)

index 88a6c0c69fb..8ea9cae0e7a 100644
@@ -443,6 +443,7 @@ static void check_embedded_repo(const char *path)
 static int add_files(struct dir_struct *dir, int flags)
 {
 	int i, exit_status = 0;
+	struct string_list matched_sparse_paths = STRING_LIST_INIT_NODUP;
 
 	if (dir->ignored_nr) {
 		fprintf(stderr, _(ignore_error));

@@ -456,6 +457,11 @@ static int add_files(struct dir_struct *dir, int flags)
 	}
 
 	for (i = 0; i < dir->nr; i++) {
+		if (!path_in_sparse_checkout(dir->entries[i]->name, &the_index)) {
+			string_list_append(&matched_sparse_paths,
+					   dir->entries[i]->name);
+			continue;
+		}
 		if (add_file_to_index(&the_index, dir->entries[i]->name, flags)) {
 			if (!ignore_add_errors)
 				die(_("adding files failed"));

@@ -464,6 +470,14 @@ static int add_files(struct dir_struct *dir, int flags)
 			check_embedded_repo(dir->entries[i]->name);
 		}
 	}
+
+	if (matched_sparse_paths.nr) {
+		advise_on_updating_sparse_paths(&matched_sparse_paths);
+		exit_status = 1;
+	}
+
+	string_list_clear(&matched_sparse_paths, 0);
+
 	return exit_status;
 }
 

index 5b3f55e355e..997442b13e3 100755
@@ -301,8 +301,6 @@ test_expect_success 'add, commit, checkout' '
 	test_all_match git checkout -
 '
 
-# NEEDSWORK: This documents current behavior, but is not a desirable
-# behavior (untracked files are handled differently than tracked).
 test_expect_success 'add outside sparse cone' '
 	init_repos &&
 

@@ -312,7 +310,9 @@ test_expect_success 'add outside sparse cone' '
 	test_sparse_match test_must_fail git add folder1/a &&
 	test_i18ngrep "Disable or modify the sparsity rules" sparse-checkout-err &&
 	test_sparse_unstaged folder1/a &&
-	test_sparse_match git add folder1/newfile
+	test_sparse_match test_must_fail git add folder1/newfile &&
+	test_i18ngrep "Disable or modify the sparsity rules" sparse-checkout-err &&
+	test_sparse_unstaged folder1/newfile
 '
 
 test_expect_success 'commit including unstaged changes' '

@@ -343,7 +343,11 @@ test_expect_success 'commit including unstaged changes' '
 	test_all_match git status --porcelain=v2
 '
 
-test_expect_success 'status/add: outside sparse cone' '
+# NEEDSWORK: Now that 'git add folder1/new' fails, the changes being
+# attempted here fail for the sparse-checkout and sparse-index repos.
+# We must enable a way for adding files outside the sparse-checkout
+# done, even if it is by an optional flag.
+test_expect_failure 'status/add: outside sparse cone' '
 	init_repos &&
 
 	# folder1 is at HEAD, but outside the sparse cone

@@ -368,10 +372,11 @@ test_expect_success 'status/add: outside sparse cone' '
 	test_sparse_match test_must_fail git add --refresh folder1/a &&
 	test_i18ngrep "Disable or modify the sparsity rules" sparse-checkout-err &&
 	test_sparse_unstaged folder1/a &&
+	test_sparse_match test_must_fail git add folder1/new &&
+	test_i18ngrep "Disable or modify the sparsity rules" sparse-checkout-err &&
+	test_sparse_unstaged folder1/new &&
 
-	# NEEDSWORK: Adding a newly-tracked file outside the cone succeeds
-	test_sparse_match git add folder1/new &&
-
+	# NEEDSWORK: behavior begins to deviate here.
 	test_all_match git add . &&
 	test_all_match git status --porcelain=v2 &&
 	test_all_match git commit -m folder1/new &&

@@ -527,7 +532,7 @@ test_expect_success 'merge, cherry-pick, and rebase' '
 # Right now, users might be using this flow to work through conflicts,
 # so any solution should present advice to users who try this sequence
 # of commands to follow whatever new method we create.
-test_expect_success 'merge with conflict outside cone' '
+test_expect_failure 'merge with conflict outside cone' '
 	init_repos &&
 
 	test_all_match git checkout -b merge-tip merge-left &&

@@ -541,12 +546,18 @@ test_expect_success 'merge with conflict outside cone' '
 	test_all_match git status --porcelain=v2 &&
 
 	# 2. Add the file with conflict markers
+	# NEEDSWORK: Even though the merge conflict removed the
+	# SKIP_WORKTREE bit from the index entry for folder1/a, we should
+	# warn that this is a problematic add.
 	test_all_match git add folder1/a &&
 	test_all_match git status --porcelain=v2 &&
 
 	# 3. Rename the file to another sparse filename and
 	#    accept conflict markers as resolved content.
 	run_on_all mv folder2/a folder2/z &&
+	# NEEDSWORK: This mode now fails, because folder2/z is
+	# outside of the sparse-checkout cone and does not match an
+	# existing index entry with the SKIP_WORKTREE bit cleared.
 	test_all_match git add folder2 &&
 	test_all_match git status --porcelain=v2 &&
 

@@ -555,7 +566,7 @@ test_expect_success 'merge with conflict outside cone' '
 	test_all_match git rev-parse HEAD^{tree}
 '
 
-test_expect_success 'cherry-pick/rebase with conflict outside cone' '
+test_expect_failure 'cherry-pick/rebase with conflict outside cone' '
 	init_repos &&
 
 	for OPERATION in cherry-pick rebase

@@ -572,11 +583,17 @@ test_expect_success 'cherry-pick/rebase with conflict outside cone' '
 		test_all_match git status --porcelain=v2 &&
 
 		# 2. Add the file with conflict markers
+		# NEEDSWORK: Even though the merge conflict removed the
+		# SKIP_WORKTREE bit from the index entry for folder1/a, we should
+		# warn that this is a problematic add.
 		test_all_match git add folder1/a &&
 		test_all_match git status --porcelain=v2 &&
 
 		# 3. Rename the file to another sparse filename and
 		#    accept conflict markers as resolved content.
+		# NEEDSWORK: This mode now fails, because folder2/z is
+		# outside of the sparse-checkout cone and does not match an
+		# existing index entry with the SKIP_WORKTREE bit cleared.
 		run_on_all mv folder2/a folder2/z &&
 		test_all_match git add folder2 &&
 		test_all_match git status --porcelain=v2 &&

@@ -654,6 +671,7 @@ test_expect_success 'clean' '
 test_expect_success 'submodule handling' '
 	init_repos &&
 
+	test_sparse_match git sparse-checkout add modules &&
 	test_all_match mkdir modules &&
 	test_all_match touch modules/a &&
 	test_all_match git add modules &&

@@ -663,6 +681,7 @@ test_expect_success 'submodule handling' '
 	test_all_match git commit -m "add submodule" &&
 
 	# having a submodule prevents "modules" from collapse
+	test_sparse_match git sparse-checkout set deep/deeper1 &&
 	test-tool -C sparse-index read-cache --table >cache &&
 	grep "100644 blob .*	modules/a" cache &&
 	grep "160000 commit $(git -C initial-repo rev-parse HEAD)	modules/sub" cache

gitgitgadget

From: Derrick Stolee <[email protected]>

When 'git add' adds a tracked file that is outside of the sparse-checkout cone, it checks the SKIP_WORKTREE bit to see if the file exists outside of the sparse-checkout cone. This is usually correct, except in the case of a merge conflict outside of the cone.

Modify add_pathspec_matched_against_index() to be more careful about paths by checking the sparse-checkout patterns in addition to the SKIP_WORKTREE bit. This causes 'git add' to no longer allow files outside of the cone that removed the SKIP_WORKTREE bit due to a merge conflict.

With only this change, users will only be able to add the file after adding the file to the sparse-checkout cone. A later change will allow users to force adding even though the file is outside of the sparse-checkout cone.

---
 builtin/add.c                            |  4 ++++
 pathspec.c                               |  5 +++--
 t/t1091-sparse-checkout-builtin.sh       |  4 +++-
 t/t1092-sparse-checkout-compatibility.sh | 21 ++++++++++++++-------
 t/t3705-add-sparse-checkout.sh           | 12 ++++++++++++
 5 files changed, 36 insertions(+), 10 deletions(-)

index 8ea9cae0e7a..09c3fad6321 100644
@@ -94,6 +94,10 @@ static void update_callback(struct diff_queue_struct *q,
 	for (i = 0; i < q->nr; i++) {
 		struct diff_filepair *p = q->queue[i];
 		const char *path = p->one->path;
+
+		if (!path_in_sparse_checkout(path, &the_index))
+			continue;
+
 		switch (fix_unmerged_status(p, data)) {
 		default:
 			die(_("unexpected diff status %c"), p->status);

index 44306fdaca2..ddeeba79114 100644
@@ -39,7 +39,8 @@ void add_pathspec_matches_against_index(const struct pathspec *pathspec,
 		return;
 	for (i = 0; i < istate->cache_nr; i++) {
 		const struct cache_entry *ce = istate->cache[i];
-		if (sw_action == PS_IGNORE_SKIP_WORKTREE && ce_skip_worktree(ce))
+		if (sw_action == PS_IGNORE_SKIP_WORKTREE &&
+		    (ce_skip_worktree(ce) || !path_in_sparse_checkout(ce->name, istate)))
 			continue;
 		ce_path_match(istate, ce, pathspec, seen);
 	}

@@ -70,7 +71,7 @@ char *find_pathspecs_matching_skip_worktree(const struct pathspec *pathspec)
 
 	for (i = 0; i < istate->cache_nr; i++) {
 		struct cache_entry *ce = istate->cache[i];
-		if (ce_skip_worktree(ce))
+		if (ce_skip_worktree(ce) || !path_in_sparse_checkout(ce->name, istate))
 		    ce_path_match(istate, ce, pathspec, seen);
 	}
 

index 71236981e64..af99ae81b1d 100755
@@ -406,7 +406,7 @@ test_expect_success 'sparse-checkout (init|set|disable) warns with unmerged stat
 	git -C unmerged sparse-checkout disable
 '
 
-test_expect_success 'sparse-checkout reapply' '
+test_expect_failure 'sparse-checkout reapply' '
 	git clone repo tweak &&
 
 	echo dirty >tweak/deep/deeper2/a &&

@@ -438,6 +438,8 @@ test_expect_success 'sparse-checkout reapply' '
 	test_i18ngrep "warning.*The following paths are unmerged" err &&
 	test_path_is_file tweak/folder1/a &&
 
+	# NEEDSWORK: We are asking to update a file outside of the
+	# sparse-checkout cone, but this is no longer allowed.
 	git -C tweak add folder1/a &&
 	git -C tweak sparse-checkout reapply 2>err &&
 	test_must_be_empty err &&

index 997442b13e3..0fdc5c7098c 100755
@@ -546,10 +546,10 @@ test_expect_failure 'merge with conflict outside cone' '
 	test_all_match git status --porcelain=v2 &&
 
 	# 2. Add the file with conflict markers
-	# NEEDSWORK: Even though the merge conflict removed the
-	# SKIP_WORKTREE bit from the index entry for folder1/a, we should
-	# warn that this is a problematic add.
-	test_all_match git add folder1/a &&
+	test_sparse_match test_must_fail git add folder1/a &&
+	test_i18ngrep "Disable or modify the sparsity rules" sparse-checkout-err &&
+	test_sparse_unstaged folder1/a &&
+	test_all_match git add --sparse folder1/a &&
 	test_all_match git status --porcelain=v2 &&
 
 	# 3. Rename the file to another sparse filename and

@@ -558,7 +558,10 @@ test_expect_failure 'merge with conflict outside cone' '
 	# NEEDSWORK: This mode now fails, because folder2/z is
 	# outside of the sparse-checkout cone and does not match an
 	# existing index entry with the SKIP_WORKTREE bit cleared.
-	test_all_match git add folder2 &&
+	test_sparse_match test_must_fail git add folder2 &&
+	test_i18ngrep "Disable or modify the sparsity rules" sparse-checkout-err &&
+	test_sparse_unstaged folder2/z &&
+	test_all_match git add --sparse folder2 &&
 	test_all_match git status --porcelain=v2 &&
 
 	test_all_match git merge --continue &&

@@ -586,7 +589,9 @@ test_expect_failure 'cherry-pick/rebase with conflict outside cone' '
 		# NEEDSWORK: Even though the merge conflict removed the
 		# SKIP_WORKTREE bit from the index entry for folder1/a, we should
 		# warn that this is a problematic add.
-		test_all_match git add folder1/a &&
+		test_sparse_match test_must_fail git add folder1/a &&
+		test_i18ngrep "Disable or modify the sparsity rules" sparse-checkout-err &&
+		test_sparse_unstaged folder1/a &&
 		test_all_match git status --porcelain=v2 &&
 
 		# 3. Rename the file to another sparse filename and

@@ -595,7 +600,9 @@ test_expect_failure 'cherry-pick/rebase with conflict outside cone' '
 		# outside of the sparse-checkout cone and does not match an
 		# existing index entry with the SKIP_WORKTREE bit cleared.
 		run_on_all mv folder2/a folder2/z &&
-		test_all_match git add folder2 &&
+		test_sparse_match test_must_fail git add folder2 &&
+		test_i18ngrep "Disable or modify the sparsity rules" sparse-checkout-err &&
+		test_sparse_unstaged folder2/z &&
 		test_all_match git status --porcelain=v2 &&
 
 		test_all_match git $OPERATION --continue &&

index af81b4b6846..678be1acbf9 100755
@@ -158,6 +158,18 @@ test_expect_success 'do not warn when pathspec matches dense entries' '
 	git ls-files --error-unmatch dense_entry
 '
 
+test_expect_success 'git add fails outside of sparse-checkout definition' '
+	test_when_finished git sparse-checkout disable &&
+	test_commit a &&
+	git sparse-checkout init &&
+	git sparse-checkout set a &&
+	echo >>sparse_entry &&
+
+	git update-index --no-skip-worktree sparse_entry &&
+	test_must_fail git add sparse_entry &&
+	test_sparse_entry_unstaged
+'
+
 test_expect_success 'add obeys advice.updateSparsePath' '
 	setup_sparse_entry &&
 	test_must_fail git -c advice.updateSparsePath=false add sparse_entry 2>stderr &&

gitgitgadget

From: Derrick Stolee <[email protected]>

We added checks for path_in_sparse_checkout() to portions of 'git add' that add warnings and prevent staging a modification, but we skipped the --chmod mode. Update chmod_pathspec() to ignore cache entries whose path is outside of the sparse-checkout cone (unless --sparse is provided). Add a test in t3705.

---
 builtin/add.c                  |  4 +++-
 t/t3705-add-sparse-checkout.sh | 12 ++++++++++--
 2 files changed, 13 insertions(+), 3 deletions(-)

index f8e3930608d..f87b8134b67 100644
@@ -47,7 +47,9 @@ static int chmod_pathspec(struct pathspec *pathspec, char flip, int show_only)
 		struct cache_entry *ce = active_cache[i];
 		int err;
 
-		if (!include_sparse && ce_skip_worktree(ce))
+		if (!include_sparse &&
+		    (ce_skip_worktree(ce) ||
+		     !path_in_sparse_checkout(ce->name, &the_index)))
 			continue;
 
 		if (pathspec && !ce_path_match(&the_index, ce, pathspec, NULL))

index 0f7e03b5326..6fc49a1845f 100755
@@ -39,7 +39,7 @@ setup_gitignore () {
 
 test_sparse_entry_unstaged () {
 	git status --porcelain >actual &&
-	! grep "^M  sparse_entry\$" actual
+	! grep "^M[ M] sparse_entry\$" actual
 }
 
 test_expect_success 'setup' "

@@ -169,11 +169,19 @@ test_expect_success 'git add fails outside of sparse-checkout definition' '
 	test_must_fail git add sparse_entry &&
 	test_sparse_entry_unstaged &&
 
+	test_must_fail git add --chmod=+x sparse_entry &&
+	test_sparse_entry_unstaged &&
+
 	# Avoid munging CRLFs to avoid an error message
 	git -c core.autocrlf=input add --sparse sparse_entry 2>stderr &&
 	test_must_be_empty stderr &&
 	test-tool read-cache --table >actual &&
-	grep "^100644 blob.*sparse_entry\$" actual
+	grep "^100644 blob.*sparse_entry\$" actual &&
+
+	git add --sparse --chmod=+x sparse_entry 2>stderr &&
+	test_must_be_empty stderr &&
+	test-tool read-cache --table >actual &&
+	grep "^100755 blob.*sparse_entry\$" actual
 '
 
 test_expect_success 'add obeys advice.updateSparsePath' '

gitgitgadget

From: Derrick Stolee <[email protected]>

We previously modified 'git add' to refuse updating index entries outside of the sparse-checkout cone. This is justified to prevent users from accidentally getting into a confusing state when Git removes those files from the working tree at some later point.

Unfortunately, this caused some workflows that were previously possible to become impossible, especially around merge conflicts outside of the sparse-checkout cone. These were documented in tests within t1092.

We now re-enable these workflows using a new '--sparse' option to 'git add'. This allows users to signal "Yes, I do know what I'm doing with these files," and accept the consequences of the files leaving the worktree later.

We delay updating the advice message until implementing a similar option in 'git rm' and 'git mv'.

---
 Documentation/git-add.txt                |  9 +++++++-
 builtin/add.c                            | 12 +++++++----
 t/t1092-sparse-checkout-compatibility.sh | 27 ++++++++----------------
 t/t3705-add-sparse-checkout.sh           | 17 ++++++++++++++-
 4 files changed, 41 insertions(+), 24 deletions(-)

index be5e3ac54b8..bb79016d2ca 100644
@@ -9,7 +9,7 @@ SYNOPSIS
 --------
 [verse]
 'git add' [--verbose | -v] [--dry-run | -n] [--force | -f] [--interactive | -i] [--patch | -p]
-	  [--edit | -e] [--[no-]all | --[no-]ignore-removal | [--update | -u]]
+	  [--edit | -e] [--[no-]all | --[no-]ignore-removal | [--update | -u]] [--sparse]
 	  [--intent-to-add | -N] [--refresh] [--ignore-errors] [--ignore-missing] [--renormalize]
 	  [--chmod=(+|-)x] [--pathspec-from-file=<file> [--pathspec-file-nul]]
 	  [--] [<pathspec>...]

@@ -79,6 +79,13 @@ in linkgit:gitglossary[7].
 --force::
 	Allow adding otherwise ignored files.
 
+--sparse::
+	Allow updating index entries outside of the sparse-checkout cone.
+	Normally, `git add` refuses to update index entries whose paths do
+	not fit within the sparse-checkout cone, since those files might
+	be removed from the working tree without warning. See
+	linkgit:git-sparse-checkout[1] for more.
+
 -i::
 --interactive::
 	Add modified contents in the working tree interactively to

index 09c3fad6321..f8e3930608d 100644
@@ -30,6 +30,7 @@ static int patch_interactive, add_interactive, edit_interactive;
 static int take_worktree_changes;
 static int add_renormalize;
 static int pathspec_file_nul;
+static int include_sparse;
 static const char *pathspec_from_file;
 static int legacy_stash_p; /* support for the scripted `git stash` */
 

@@ -46,7 +47,7 @@ static int chmod_pathspec(struct pathspec *pathspec, char flip, int show_only)
 		struct cache_entry *ce = active_cache[i];
 		int err;
 
-		if (ce_skip_worktree(ce))
+		if (!include_sparse && ce_skip_worktree(ce))
 			continue;
 
 		if (pathspec && !ce_path_match(&the_index, ce, pathspec, NULL))

@@ -95,7 +96,7 @@ static void update_callback(struct diff_queue_struct *q,
 		struct diff_filepair *p = q->queue[i];
 		const char *path = p->one->path;
 
-		if (!path_in_sparse_checkout(path, &the_index))
+		if (!include_sparse && !path_in_sparse_checkout(path, &the_index))
 			continue;
 
 		switch (fix_unmerged_status(p, data)) {

@@ -383,6 +384,7 @@ static struct option builtin_add_options[] = {
 	OPT_BOOL( 0 , "refresh", &refresh_only, N_("don't add, only refresh the index")),
 	OPT_BOOL( 0 , "ignore-errors", &ignore_add_errors, N_("just skip files which cannot be added because of errors")),
 	OPT_BOOL( 0 , "ignore-missing", &ignore_missing, N_("check if - even missing - files are ignored in dry run")),
+	OPT_BOOL(0, "sparse", &include_sparse, N_("allow updating entries outside of the sparse-checkout cone")),
 	OPT_STRING(0, "chmod", &chmod_arg, "(+|-)x",
 		   N_("override the executable bit of the listed files")),
 	OPT_HIDDEN_BOOL(0, "warn-embedded-repo", &warn_on_embedded_repo,

@@ -461,7 +463,8 @@ static int add_files(struct dir_struct *dir, int flags)
 	}
 
 	for (i = 0; i < dir->nr; i++) {
-		if (!path_in_sparse_checkout(dir->entries[i]->name, &the_index)) {
+		if (!include_sparse &&
+		    !path_in_sparse_checkout(dir->entries[i]->name, &the_index)) {
 			string_list_append(&matched_sparse_paths,
 					   dir->entries[i]->name);
 			continue;

@@ -646,7 +649,8 @@ int cmd_add(int argc, const char **argv, const char *prefix)
 			if (seen[i])
 				continue;
 
-			if (matches_skip_worktree(&pathspec, i, &skip_worktree_seen)) {
+			if (!include_sparse &&
+			    matches_skip_worktree(&pathspec, i, &skip_worktree_seen)) {
 				string_list_append(&only_match_skip_worktree,
 						   pathspec.items[i].original);
 				continue;

index 0fdc5c7098c..7d64d9deb22 100755
@@ -343,11 +343,7 @@ test_expect_success 'commit including unstaged changes' '
 	test_all_match git status --porcelain=v2
 '
 
-# NEEDSWORK: Now that 'git add folder1/new' fails, the changes being
-# attempted here fail for the sparse-checkout and sparse-index repos.
-# We must enable a way for adding files outside the sparse-checkout
-# done, even if it is by an optional flag.
-test_expect_failure 'status/add: outside sparse cone' '
+test_expect_success 'status/add: outside sparse cone' '
 	init_repos &&
 
 	# folder1 is at HEAD, but outside the sparse cone

@@ -375,15 +371,16 @@ test_expect_failure 'status/add: outside sparse cone' '
 	test_sparse_match test_must_fail git add folder1/new &&
 	test_i18ngrep "Disable or modify the sparsity rules" sparse-checkout-err &&
 	test_sparse_unstaged folder1/new &&
+	test_sparse_match git add --sparse folder1/a &&
+	test_sparse_match git add --sparse folder1/new &&
 
-	# NEEDSWORK: behavior begins to deviate here.
-	test_all_match git add . &&
+	test_all_match git add --sparse . &&
 	test_all_match git status --porcelain=v2 &&
 	test_all_match git commit -m folder1/new &&
 	test_all_match git rev-parse HEAD^{tree} &&
 
 	run_on_all ../edit-contents folder1/newer &&
-	test_all_match git add folder1/ &&
+	test_all_match git add --sparse folder1/ &&
 	test_all_match git status --porcelain=v2 &&
 	test_all_match git commit -m folder1/newer &&
 	test_all_match git rev-parse HEAD^{tree}

@@ -527,12 +524,7 @@ test_expect_success 'merge, cherry-pick, and rebase' '
 	done
 '
 
-# NEEDSWORK: This test is documenting current behavior, but that
-# behavior can be confusing to users so there is desire to change it.
-# Right now, users might be using this flow to work through conflicts,
-# so any solution should present advice to users who try this sequence
-# of commands to follow whatever new method we create.
-test_expect_failure 'merge with conflict outside cone' '
+test_expect_success 'merge with conflict outside cone' '
 	init_repos &&
 
 	test_all_match git checkout -b merge-tip merge-left &&

@@ -555,9 +547,6 @@ test_expect_failure 'merge with conflict outside cone' '
 	# 3. Rename the file to another sparse filename and
 	#    accept conflict markers as resolved content.
 	run_on_all mv folder2/a folder2/z &&
-	# NEEDSWORK: This mode now fails, because folder2/z is
-	# outside of the sparse-checkout cone and does not match an
-	# existing index entry with the SKIP_WORKTREE bit cleared.
 	test_sparse_match test_must_fail git add folder2 &&
 	test_i18ngrep "Disable or modify the sparsity rules" sparse-checkout-err &&
 	test_sparse_unstaged folder2/z &&

@@ -569,7 +558,7 @@ test_expect_failure 'merge with conflict outside cone' '
 	test_all_match git rev-parse HEAD^{tree}
 '
 
-test_expect_failure 'cherry-pick/rebase with conflict outside cone' '
+test_expect_success 'cherry-pick/rebase with conflict outside cone' '
 	init_repos &&
 
 	for OPERATION in cherry-pick rebase

@@ -592,6 +581,7 @@ test_expect_failure 'cherry-pick/rebase with conflict outside cone' '
 		test_sparse_match test_must_fail git add folder1/a &&
 		test_i18ngrep "Disable or modify the sparsity rules" sparse-checkout-err &&
 		test_sparse_unstaged folder1/a &&
+		test_all_match git add --sparse folder1/a &&
 		test_all_match git status --porcelain=v2 &&
 
 		# 3. Rename the file to another sparse filename and

@@ -603,6 +593,7 @@ test_expect_failure 'cherry-pick/rebase with conflict outside cone' '
 		test_sparse_match test_must_fail git add folder2 &&
 		test_i18ngrep "Disable or modify the sparsity rules" sparse-checkout-err &&
 		test_sparse_unstaged folder2/z &&
+		test_all_match git add --sparse folder2 &&
 		test_all_match git status --porcelain=v2 &&
 
 		test_all_match git $OPERATION --continue &&

index 678be1acbf9..0f7e03b5326 100755
@@ -167,7 +167,13 @@ test_expect_success 'git add fails outside of sparse-checkout definition' '
 
 	git update-index --no-skip-worktree sparse_entry &&
 	test_must_fail git add sparse_entry &&
-	test_sparse_entry_unstaged
+	test_sparse_entry_unstaged &&
+
+	# Avoid munging CRLFs to avoid an error message
+	git -c core.autocrlf=input add --sparse sparse_entry 2>stderr &&
+	test_must_be_empty stderr &&
+	test-tool read-cache --table >actual &&
+	grep "^100644 blob.*sparse_entry\$" actual
 '
 
 test_expect_success 'add obeys advice.updateSparsePath' '

@@ -178,4 +184,13 @@ test_expect_success 'add obeys advice.updateSparsePath' '
 
 '
 
+test_expect_success 'add allows sparse entries with --sparse' '
+	git sparse-checkout set a &&
+	echo modified >sparse_entry &&
+	test_must_fail git add sparse_entry &&
+	test_sparse_entry_unchanged &&
+	git add --sparse sparse_entry 2>stderr &&
+	test_must_be_empty stderr
+'
+
 test_done

gitgitgadget

From: Derrick Stolee <[email protected]>

We added checks for path_in_sparse_checkout() to portions of 'git add' that add warnings and prevent stagins a modification, but we skipped the --renormalize mode. Update renormalize_tracked_files() to ignore cache entries whose path is outside of the sparse-checkout cone (unless --sparse is provided). Add a test in t3705.

---
 builtin/add.c                  |  4 +++-
 t/t3705-add-sparse-checkout.sh | 12 +++++++++++-
 2 files changed, 14 insertions(+), 2 deletions(-)

index f87b8134b67..f8f0dfa4046 100644
@@ -154,7 +154,9 @@ static int renormalize_tracked_files(const struct pathspec *pathspec, int flags)
 	for (i = 0; i < active_nr; i++) {
 		struct cache_entry *ce = active_cache[i];
 
-		if (ce_skip_worktree(ce))
+		if (!include_sparse &&
+		    (ce_skip_worktree(ce) ||
+		     !path_in_sparse_checkout(ce->name, &the_index)))
 			continue;
 		if (ce_stage(ce))
 			continue; /* do not touch unmerged paths */

index 6fc49a1845f..cf2ccb87cf2 100755
@@ -172,6 +172,9 @@ test_expect_success 'git add fails outside of sparse-checkout definition' '
 	test_must_fail git add --chmod=+x sparse_entry &&
 	test_sparse_entry_unstaged &&
 
+	test_must_fail git add --renormalize sparse_entry &&
+	test_sparse_entry_unstaged &&
+
 	# Avoid munging CRLFs to avoid an error message
 	git -c core.autocrlf=input add --sparse sparse_entry 2>stderr &&
 	test_must_be_empty stderr &&

@@ -181,7 +184,14 @@ test_expect_success 'git add fails outside of sparse-checkout definition' '
 	git add --sparse --chmod=+x sparse_entry 2>stderr &&
 	test_must_be_empty stderr &&
 	test-tool read-cache --table >actual &&
-	grep "^100755 blob.*sparse_entry\$" actual
+	grep "^100755 blob.*sparse_entry\$" actual &&
+
+	git reset &&
+
+	# This will print a message over stderr on Windows.
+	git add --sparse --renormalize sparse_entry &&
+	git status --porcelain >actual &&
+	grep "^M  sparse_entry\$" actual
 '
 
 test_expect_success 'add obeys advice.updateSparsePath' '

gitgitgadget

From: Derrick Stolee <[email protected]>

As we did previously in 'git add', add a '--sparse' option to 'git rm' that allows modifying paths outside of the sparse-checkout definition. The existing checks in 'git rm' are restricted to tracked files that have the SKIP_WORKTREE bit in the current index. Future changes will cause 'git rm' to reject removing paths outside of the sparse-checkout definition, even if they are untracked or do not have the SKIP_WORKTREE bit.

---
 Documentation/git-rm.txt      |  6 ++++++
 builtin/rm.c                  |  8 ++++++--
 t/t3602-rm-sparse-checkout.sh | 12 ++++++++++++
 3 files changed, 24 insertions(+), 2 deletions(-)

index 26e9b284704..81bc23f3cdb 100644
@@ -72,6 +72,12 @@ For more details, see the 'pathspec' entry in linkgit:gitglossary[7].
 --ignore-unmatch::
 	Exit with a zero status even if no files matched.
 
+--sparse::
+	Allow updating index entries outside of the sparse-checkout cone.
+	Normally, `git rm` refuses to update index entries whose paths do
+	not fit within the sparse-checkout cone. See
+	linkgit:git-sparse-checkout[1] for more.
+
 -q::
 --quiet::
 	`git rm` normally outputs one line (in the form of an `rm` command)

index 8a24c715e02..4208f3f9a5f 100644
@@ -237,6 +237,7 @@ static int check_local_mod(struct object_id *head, int index_only)
 
 static int show_only = 0, force = 0, index_only = 0, recursive = 0, quiet = 0;
 static int ignore_unmatch = 0, pathspec_file_nul;
+static int include_sparse;
 static char *pathspec_from_file;
 
 static struct option builtin_rm_options[] = {

@@ -247,6 +248,7 @@ static struct option builtin_rm_options[] = {
 	OPT_BOOL('r', NULL,             &recursive,  N_("allow recursive removal")),
 	OPT_BOOL( 0 , "ignore-unmatch", &ignore_unmatch,
 				N_("exit with a zero status even if nothing matched")),
+	OPT_BOOL(0, "sparse", &include_sparse, N_("allow updating entries outside of the sparse-checkout cone")),
 	OPT_PATHSPEC_FROM_FILE(&pathspec_from_file),
 	OPT_PATHSPEC_FILE_NUL(&pathspec_file_nul),
 	OPT_END(),

@@ -298,7 +300,8 @@ int cmd_rm(int argc, const char **argv, const char *prefix)
 	ensure_full_index(&the_index);
 	for (i = 0; i < active_nr; i++) {
 		const struct cache_entry *ce = active_cache[i];
-		if (ce_skip_worktree(ce))
+
+		if (!include_sparse && ce_skip_worktree(ce))
 			continue;
 		if (!ce_path_match(&the_index, ce, &pathspec, seen))
 			continue;

@@ -322,7 +325,8 @@ int cmd_rm(int argc, const char **argv, const char *prefix)
 				seen_any = 1;
 			else if (ignore_unmatch)
 				continue;
-			else if (matches_skip_worktree(&pathspec, i, &skip_worktree_seen))
+			else if (!include_sparse &&
+				 matches_skip_worktree(&pathspec, i, &skip_worktree_seen))
 				string_list_append(&only_match_skip_worktree, original);
 			else
 				die(_("pathspec '%s' did not match any files"), original);

index e9e9a15c74c..493c8f636b8 100755
@@ -43,6 +43,18 @@ test_expect_success 'recursive rm does not remove sparse entries' '
 	test_cmp expected actual
 '
 
+test_expect_success 'recursive rm --sparse removes sparse entries' '
+	git reset --hard &&
+	git sparse-checkout set "sub/dir" &&
+	git rm --sparse -r sub &&
+	git status --porcelain -uno >actual &&
+	cat >expected <<-\EOF &&
+	D  sub/d
+	D  sub/dir/e
+	EOF
+	test_cmp expected actual
+'
+
 test_expect_success 'rm obeys advice.updateSparsePath' '
 	git reset --hard &&
 	git sparse-checkout set a &&

gitgitgadget

From: Derrick Stolee <[email protected]>

The previous changes modified the behavior of 'git add', 'git rm', and 'git mv' to not adjust paths outside the sparse-checkout cone, even if they exist in the working tree and their cache entries lack the SKIP_WORKTREE bit. The intention is to warn users that they are doing something potentially dangerous. The '--sparse' option was added to each command to allow careful users the same ability they had before.

To improve the discoverability of this new functionality, add a message to advice.updateSparsePath that mentions the existence of the option.

The previous set of changes also modified the purpose of this message to include possibly a list of paths instead of only a list of pathspecs. Make the warning message more clear about this new behavior.

---
 advice.c                       | 10 ++++++----
 t/t3602-rm-sparse-checkout.sh  |  6 +++---
 t/t3705-add-sparse-checkout.sh |  6 +++---
 t/t7002-mv-sparse-checkout.sh  |  6 +++---
 4 files changed, 15 insertions(+), 13 deletions(-)

index 0b9c89c48ab..0deaf111795 100644
@@ -293,14 +293,16 @@ void advise_on_updating_sparse_paths(struct string_list *pathspec_list)
 	if (!pathspec_list->nr)
 		return;
 
-	fprintf(stderr, _("The following pathspecs didn't match any"
-			  " eligible path, but they do match index\n"
-			  "entries outside the current sparse checkout:\n"));
+	fprintf(stderr, _("The following paths and/or pathspecs matched "
+			  "paths that exist outside of your\n"
+			  "sparse-checkout definition, so will not be "
+			  "updated in the index:\n"));
 	for_each_string_list_item(item, pathspec_list)
 		fprintf(stderr, "%s\n", item->string);
 
 	advise_if_enabled(ADVICE_UPDATE_SPARSE_PATH,
-			  _("Disable or modify the sparsity rules if you intend"
+			  _("Disable or modify the sparsity rules or"
+			    " use the --sparse option if you intend"
 			    " to update such entries."));
 }
 

index 5f92b60a56a..712ddae2b15 100755
@@ -11,12 +11,12 @@ test_expect_success 'setup' "
 	git commit -m files &&
 
 	cat >sparse_error_header <<-EOF &&
-	The following pathspecs didn't match any eligible path, but they do match index
-	entries outside the current sparse checkout:
+	The following paths and/or pathspecs matched paths that exist outside of your
+	sparse-checkout definition, so will not be updated in the index:
 	EOF
 
 	cat >sparse_hint <<-EOF &&
-	hint: Disable or modify the sparsity rules if you intend to update such entries.
+	hint: Disable or modify the sparsity rules or use the --sparse option if you intend to update such entries.
 	hint: Disable this message with \"git config advice.updateSparsePath false\"
 	EOF
 

index cf2ccb87cf2..d51c96f8d72 100755
@@ -44,12 +44,12 @@ test_sparse_entry_unstaged () {
 
 test_expect_success 'setup' "
 	cat >sparse_error_header <<-EOF &&
-	The following pathspecs didn't match any eligible path, but they do match index
-	entries outside the current sparse checkout:
+	The following paths and/or pathspecs matched paths that exist outside of your
+	sparse-checkout definition, so will not be updated in the index:
 	EOF
 
 	cat >sparse_hint <<-EOF &&
-	hint: Disable or modify the sparsity rules if you intend to update such entries.
+	hint: Disable or modify the sparsity rules or use the --sparse option if you intend to update such entries.
 	hint: Disable this message with \"git config advice.updateSparsePath false\"
 	EOF
 

index 07dbfeb6d17..3db7ddaba9a 100755
@@ -11,12 +11,12 @@ test_expect_success 'setup' "
 	git commit -m files &&
 
 	cat >sparse_error_header <<-EOF &&
-	The following pathspecs didn't match any eligible path, but they do match index
-	entries outside the current sparse checkout:
+	The following paths and/or pathspecs matched paths that exist outside of your
+	sparse-checkout definition, so will not be updated in the index:
 	EOF
 
 	cat >sparse_hint <<-EOF
-	hint: Disable or modify the sparsity rules if you intend to update such entries.
+	hint: Disable or modify the sparsity rules or use the --sparse option if you intend to update such entries.
 	hint: Disable this message with \"git config advice.updateSparsePath false\"
 	EOF
 "

gitgitgadget

From: Derrick Stolee <[email protected]>

If a path does not match the sparse-checkout cone but is somehow missing the SKIP_WORKTREE bit, then 'git rm' currently succeeds in removing the file. One reason a user might be in this situation is a merge conflict outside of the sparse-checkout cone. Removing such a file might be problematic for users who are not sure what they are doing.

Add a check to path_in_sparse_checkout() when 'git rm' is checking if a path should be considered for deletion. Of course, this check is ignored if the '--sparse' option is specified, allowing users who accept the risks to continue with the removal.

This also removes a confusing behavior where a user asks for a directory to be removed, but only the entries that are within the sparse-checkout definition are removed. Now, 'git rm <dir>' will fail without '--sparse' and will succeed in removing all contained paths with '--sparse'.

---
 builtin/rm.c                  |  4 +++-
 t/t3602-rm-sparse-checkout.sh | 19 +++++++++++++++++--
 2 files changed, 20 insertions(+), 3 deletions(-)

index 4208f3f9a5f..a6da03da2be 100644
@@ -301,7 +301,9 @@ int cmd_rm(int argc, const char **argv, const char *prefix)
 	for (i = 0; i < active_nr; i++) {
 		const struct cache_entry *ce = active_cache[i];
 
-		if (!include_sparse && ce_skip_worktree(ce))
+		if (!include_sparse &&
+		    (ce_skip_worktree(ce) ||
+		     !path_in_sparse_checkout(ce->name, &the_index)))
 			continue;
 		if (!ce_path_match(&the_index, ce, &pathspec, seen))
 			continue;

index 493c8f636b8..5f92b60a56a 100755
@@ -37,9 +37,13 @@ done
 test_expect_success 'recursive rm does not remove sparse entries' '
 	git reset --hard &&
 	git sparse-checkout set sub/dir &&
-	git rm -r sub &&
+	test_must_fail git rm -r sub &&
+	git rm --sparse -r sub &&
 	git status --porcelain -uno >actual &&
-	echo "D  sub/dir/e" >expected &&
+	cat >expected <<-\EOF &&
+	D  sub/d
+	D  sub/dir/e
+	EOF
 	test_cmp expected actual
 '
 

@@ -87,4 +91,15 @@ test_expect_success 'do not warn about sparse entries with --ignore-unmatch' '
 	git ls-files --error-unmatch b
 '
 
+test_expect_success 'refuse to rm a non-skip-worktree path outside sparse cone' '
+	git reset --hard &&
+	git sparse-checkout set a &&
+	git update-index --no-skip-worktree b &&
+	test_must_fail git rm b 2>stderr &&
+	test_cmp b_error_and_hint stderr &&
+	git rm --sparse b 2>stderr &&
+	test_must_be_empty stderr &&
+	test_path_is_missing b
+'
+
 test_done

gitgitgadget

From: Derrick Stolee <[email protected]>

Since cmd_mv() does not operate on cache entries and instead directly checks the filesystem, we can only use path_in_sparse_checkout() as a mechanism for seeing if a path is sparse or not. Be sure to skip returning a failure if '-k' is specified.

To ensure that the advice around sparse paths is the only reason a move failed, be sure to check this as the very last thing before inserting into the src_for_dst list.

The tests cover a variety of cases such as whether the target is tracked or untracked, and whether the source or destination are in or outside of the sparse-checkout definition.

---
 builtin/mv.c                  |  52 ++++++++--
 t/t7002-mv-sparse-checkout.sh | 186 ++++++++++++++++++++++++++++++++++
 2 files changed, 229 insertions(+), 9 deletions(-)
 create mode 100755 t/t7002-mv-sparse-checkout.sh

index c2f96c8e895..83a465ba831 100644
@@ -118,21 +118,23 @@ static int index_range_of_same_dir(const char *src, int length,
 int cmd_mv(int argc, const char **argv, const char *prefix)
 {
 	int i, flags, gitmodules_modified = 0;
-	int verbose = 0, show_only = 0, force = 0, ignore_errors = 0;
+	int verbose = 0, show_only = 0, force = 0, ignore_errors = 0, ignore_sparse = 0;
 	struct option builtin_mv_options[] = {
 		OPT__VERBOSE(&verbose, N_("be verbose")),
 		OPT__DRY_RUN(&show_only, N_("dry run")),
 		OPT__FORCE(&force, N_("force move/rename even if target exists"),
 			   PARSE_OPT_NOCOMPLETE),
 		OPT_BOOL('k', NULL, &ignore_errors, N_("skip move/rename errors")),
+		OPT_BOOL(0, "sparse", &ignore_sparse, N_("allow updating entries outside of the sparse-checkout cone")),
 		OPT_END(),
 	};
 	const char **source, **destination, **dest_path, **submodule_gitfile;
-	enum update_mode { BOTH = 0, WORKING_DIRECTORY, INDEX } *modes;
+	enum update_mode { BOTH = 0, WORKING_DIRECTORY, INDEX, SPARSE } *modes;
 	struct stat st;
 	struct string_list src_for_dst = STRING_LIST_INIT_NODUP;
 	struct lock_file lock_file = LOCK_INIT;
 	struct cache_entry *ce;
+	struct string_list only_match_skip_worktree = STRING_LIST_INIT_NODUP;
 
 	git_config(git_default_config, NULL);
 

@@ -176,14 +178,17 @@ int cmd_mv(int argc, const char **argv, const char *prefix)
 		const char *src = source[i], *dst = destination[i];
 		int length, src_is_dir;
 		const char *bad = NULL;
+		int skip_sparse = 0;
 
 		if (show_only)
 			printf(_("Checking rename of '%s' to '%s'\n"), src, dst);
 
 		length = strlen(src);
-		if (lstat(src, &st) < 0)
-			bad = _("bad source");
-		else if (!strncmp(src, dst, length) &&
+		if (lstat(src, &st) < 0) {
+			/* only error if existence is expected. */
+			if (modes[i] != SPARSE)
+				bad = _("bad source");
+		} else if (!strncmp(src, dst, length) &&
 				(dst[length] == 0 || dst[length] == '/')) {
 			bad = _("can not move directory into itself");
 		} else if ((src_is_dir = S_ISDIR(st.st_mode))

@@ -212,11 +217,12 @@ int cmd_mv(int argc, const char **argv, const char *prefix)
 				dst_len = strlen(dst);
 
 				for (j = 0; j < last - first; j++) {
-					const char *path = active_cache[first + j]->name;
+					const struct cache_entry *ce = active_cache[first + j];
+					const char *path = ce->name;
 					source[argc + j] = path;
 					destination[argc + j] =
 						prefix_path(dst, dst_len, path + length + 1);
-					modes[argc + j] = INDEX;
+					modes[argc + j] = ce_skip_worktree(ce) ? SPARSE : INDEX;
 					submodule_gitfile[argc + j] = NULL;
 				}
 				argc += last - first;

@@ -244,14 +250,36 @@ int cmd_mv(int argc, const char **argv, const char *prefix)
 			bad = _("multiple sources for the same target");
 		else if (is_dir_sep(dst[strlen(dst) - 1]))
 			bad = _("destination directory does not exist");
-		else
+		else {
+			/*
+			 * We check if the paths are in the sparse-checkout
+			 * definition as a very final check, since that
+			 * allows us to point the user to the --sparse
+			 * option as a way to have a successful run.
+			 */
+			if (!ignore_sparse &&
+			    !path_in_sparse_checkout(src, &the_index)) {
+				string_list_append(&only_match_skip_worktree, src);
+				skip_sparse = 1;
+			}
+			if (!ignore_sparse &&
+			    !path_in_sparse_checkout(dst, &the_index)) {
+				string_list_append(&only_match_skip_worktree, dst);
+				skip_sparse = 1;
+			}
+
+			if (skip_sparse)
+				goto remove_entry;
+
 			string_list_insert(&src_for_dst, dst);
+		}
 
 		if (!bad)
 			continue;
 		if (!ignore_errors)
 			die(_("%s, source=%s, destination=%s"),
 			     bad, src, dst);
+remove_entry:
 		if (--argc > 0) {
 			int n = argc - i;
 			memmove(source + i, source + i + 1,

@@ -266,6 +294,12 @@ int cmd_mv(int argc, const char **argv, const char *prefix)
 		}
 	}
 
+	if (only_match_skip_worktree.nr) {
+		advise_on_updating_sparse_paths(&only_match_skip_worktree);
+		if (!ignore_errors)
+			return 1;
+	}
+
 	for (i = 0; i < argc; i++) {
 		const char *src = source[i], *dst = destination[i];
 		enum update_mode mode = modes[i];

@@ -274,7 +308,7 @@ int cmd_mv(int argc, const char **argv, const char *prefix)
 			printf(_("Renaming %s to %s\n"), src, dst);
 		if (show_only)
 			continue;
-		if (mode != INDEX && rename(src, dst) < 0) {
+		if (mode != INDEX && mode != SPARSE && rename(src, dst) < 0) {
 			if (ignore_errors)
 				continue;
 			die_errno(_("renaming '%s' failed"), src);

gitgitgadget