Skip to contentSkip to conversation

stash: avoid sparse-index expansion for in-cone paths

PATCH In review [email protected] 3 participants 6 messages Jul 20 – Jul 21

lore
Share & tools
lore
#0 lore raw

cover stash: avoid sparse-index expansion for in-cone paths

From: Ted Nyman <[email protected]>

git stash push -- <pathspec> expands a sparse index before checking whether the pathspec matches a tracked path. A pathspec wholly inside the sparse-checkout cone cannot match part of a sparse-directory entry, so that expansion needlessly makes the command proportional to the full index size.

The first patch fixes the pathspec helper to use the parsed, prefixed path consistently. The existing code can read past the end of the unprefixed path for a wildcard passed to git rm or git reset from a subdirectory; AddressSanitizer reports a heap-buffer-overflow in that case.

The second patch uses the helper in git stash push, following the same approach as bcf96cfca6 ("rm: expand the index only when necessary", 2022-08-07). It adds compatibility coverage for the supported pathspec forms and a path-limited stash case to p2000.

On a cone-mode repository with 349,525 tracked paths and 49 sparse-index entries, the best of three runs was:

before: 18.87s (2.93s user + 15.62s system), 4 expansions
after: 0.06s (0.01s user + 0.02s system), 0 expansions

A full-index control was unchanged (1.62s before, 1.65s after).

The series is based on 48bbf81c29 ("The 5th batch", 2026-07-19), the current master. Focused sparse-index, stash, pathspec, rm, reset, SHA-256, and unit-test coverage passes. Clang, GCC, and sanitizer builds also pass.

Ted Nyman (2):
pathspec: use match for sparse-index expansion checks
stash: avoid sparse-index expansion for in-cone paths

 builtin/stash.c                          |  4 +-
 pathspec.c                               | 12 ++---
 t/perf/p2000-sparse-operations.sh        |  1 +
 t/t1092-sparse-checkout-compatibility.sh | 62 ++++++++++++++++++++++++
 4 files changed, 71 insertions(+), 8 deletions(-)

base-commit: 48bbf81c29ca9a4479ec7850fe206518682cdb2f

#1 lore raw

PATCH 1/2 pathspec: use match for sparse-index expansion checks

From: Ted Nyman <[email protected]>

The pathspec parser computes len and nowildcard_len from item.match, which includes any prefix added when a command is run from a subdirectory. item.original can still contain the shorter, unprefixed argument.

Using item.original + item.nowildcard_len in pathspec_needs_expanded_index() can therefore read past the end of the allocation. AddressSanitizer reports a heap-buffer-overflow for prefixed wildcard pathspecs passed to git rm and git reset with a sparse index.

The mismatch dates back to 4d1cfc1351 ("reset: make --mixed sparse-aware", 2021-11-29), which introduced the helper using item.original. b29ad38322 ("pathspec.h: move pathspec_needs_expanded_index() from reset.c to here", 2022-08-07) later moved it to pathspec.c and preserved the affected comparisons.

Use item.match consistently when checking whether a pathspec can match a sparse-directory entry. Add coverage for prefixed wildcard pathspecs so both commands keep the index sparse.

---
 pathspec.c                               | 12 ++++++------
 t/t1092-sparse-checkout-compatibility.sh |  7 +++++++
 2 files changed, 13 insertions(+), 6 deletions(-)
pathspec.c +6 -6
index f78b22709ccb67..281858f21f9c59 100644
@@ -847,9 +847,9 @@ int pathspec_needs_expanded_index(struct index_state *istate,
 			 * - not-in-cone/bar*: may need expanded index
 			 * - **.c: may need expanded index
 			 */
-			if (strspn(item.original + item.nowildcard_len, "*") ==
+			if (strspn(item.match + item.nowildcard_len, "*") ==
 				    (unsigned int)(item.len - item.nowildcard_len) &&
-			    path_in_cone_mode_sparse_checkout(item.original, istate))
+			    path_in_cone_mode_sparse_checkout(item.match, istate))
 				continue;
 
 			for (pos = 0; pos < istate->cache_nr; pos++) {
@@ -865,7 +865,7 @@ int pathspec_needs_expanded_index(struct index_state *istate,
 				 */
 				if ((unsigned int)item.nowildcard_len >
 					    ce_namelen(ce) &&
-				    !strncmp(item.original, ce->name,
+				    !strncmp(item.match, ce->name,
 					     ce_namelen(ce))) {
 					res = 1;
 					break;
@@ -876,13 +876,13 @@ int pathspec_needs_expanded_index(struct index_state *istate,
 				 * directory and the pathspec does not match the whole
 				 * directory, need to expand the index.
 				 */
-				if (!strncmp(item.original, ce->name, item.nowildcard_len) &&
-				    wildmatch(item.original, ce->name, 0)) {
+				if (!strncmp(item.match, ce->name, item.nowildcard_len) &&
+				    wildmatch(item.match, ce->name, 0)) {
 					res = 1;
 					break;
 				}
 			}
-		} else if (!path_in_cone_mode_sparse_checkout(item.original, istate) &&
+		} else if (!path_in_cone_mode_sparse_checkout(item.match, istate) &&
 			   !matches_skip_worktree(pathspec, i, &skip_worktree_seen))
 			res = 1;
 
t/t1092-sparse-checkout-compatibility.sh +7 -0
index 9814431cd74aff..d0b42371663f9d 100755
@@ -2119,6 +2119,13 @@ test_expect_success 'sparse index is not expanded: rm' '
 	ensure_not_expanded rm -r deep
 '
 
+test_expect_success 'sparse index is not expanded: prefixed wildcard pathspec' '
+	init_repos &&
+
+	ensure_not_expanded -C deep rm --dry-run -- "a*" &&
+	ensure_not_expanded -C deep reset base -- "a*"
+'
+
 test_expect_success 'grep with and --cached' '
 	init_repos &&
 
#2 lore raw

Re: [PATCH 1/2] pathspec: use match for sparse-index expansion checks

Quoted 11 lines · On Mon, Jul 20, 2026 at 03:31:20PM -0700, [email protected] wrote:

Using item.original + item.nowildcard_len in pathspec_needs_expanded_index() can therefore read past the end of the allocation. AddressSanitizer reports a heap-buffer-overflow for prefixed wildcard pathspecs passed to git rm and git reset with a sparse index.

The mismatch dates back to 4d1cfc1351 ("reset: make --mixed sparse-aware", 2021-11-29), which introduced the helper using item.original. b29ad38322 ("pathspec.h: move pathspec_needs_expanded_index() from reset.c to here", 2022-08-07) later moved it to pathspec.c and preserved the affected comparisons.

Nice find. I can reliably reproduce the ASan failure you described above like so:

repo=$(mktemp -d /tmp/pathspec-asan.XXXXXX)
trap 'rm -rf "$repo"' EXIT

git init "$repo"

cd "$repo"

mkdir -p deep outside
: >deep/a
: >outside/file
git add .
git commit -q -m base

git sparse-checkout init --cone --sparse-index git sparse-checkout set deep

# From deep/, match is "deep/a*" while original is only "a*". git.compile -C deep reset HEAD -- 'a*'

(where 'git.compile' points at my build, which in this case was compiled with "make SANITIZE=address"), and results in

    ==89470==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000001f36 at pc 0x000106a69430 bp 0x00016b07c530 sp 0x00016b07bce0
    READ of size 1 at 0x602000001f36 thread T0
        #0 0x000106a6942c in strspn+0x3f0 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x1942c)
        #1 0x0001054a35b0 in pathspec_needs_expanded_index pathspec.c:850
        #2 0x000104fe6c24 in read_from_tree reset.c:214
        #3 0x000104fe5774 in cmd_reset reset.c:495
    [...]

It made me wonder whether or not this bug was trigger-able back in 4d1cfc1351. After checking out that version, I re-ran the same script and got an identical buffer overflow in the 'strspn()' call.

Applying your patch and repeating the same steps results in a clean exit.

pathspec.c int pathspec_needs_expanded_index(struct index_state *istate, · line 850

Quoted 10 lines
pathspec.c +1 -1
index f78b22709ccb67..281858f21f9c59 100644
@@ -847,9 +847,9 @@ int pathspec_needs_expanded_index(struct index_state *istate,
 			 * - not-in-cone/bar*: may need expanded index
 			 * - **.c: may need expanded index
 			 */
-			if (strspn(item.original + item.nowildcard_len, "*") ==
+			if (strspn(item.match + item.nowildcard_len, "*") ==

OK. The comment above is elided from the diff context, but is useful IMHO during review. Here we want to make sure that the remaining wildcard-ed portion of the pathspec element is only "*", which may need to expand the index only if we are not inside of the existing sparse checkout.

But 'item.nowildcard_len' bytes ahead of 'item.original' may (at worst) point into uninitialized memory, or (at best) point at a portion of the string that is not in fact a wildcard (even if the pathspec item would not otherwise require us to expand the sparse checkout).

So this makes sense.

pathspec.c int pathspec_needs_expanded_index(struct index_state *istate, · line 852

Quoted 3 lines
pathspec.c +1 -1
 				    (unsigned int)(item.len - item.nowildcard_len) &&
-			    path_in_cone_mode_sparse_checkout(item.original, istate))
+			    path_in_cone_mode_sparse_checkout(item.match, istate))

Likewise. Here I think we *might* actually be OK, but I haven't read 'path_in_cone_mode_sparse_checkout()' to know whether that's (a) true, and (b) if so, whether it's true by accident or intention.

Regardless, 'item.match' makes sense here as well for the same reason. Likewise with the rest of the patch.

Quoted 5 lines
t/t1092-sparse-checkout-compatibility.sh +0 -0
index 9814431cd74aff..d0b42371663f9d 100755

t/t1092-sparse-checkout-compatibility.sh test_expect_success 'sparse index is not expanded: rm' ' · line 2127

Quoted 9 lines
t/t1092-sparse-checkout-compatibility.sh +6 -0
@@ -2119,6 +2119,13 @@ test_expect_success 'sparse index is not expanded: rm' '
 	ensure_not_expanded rm -r deep
 '
 
+test_expect_success 'sparse index is not expanded: prefixed wildcard pathspec' '
+	init_repos &&
+
+	ensure_not_expanded -C deep rm --dry-run -- "a*" &&
+	ensure_not_expanded -C deep reset base -- "a*"
+'

Looks good, this is effectively the same thing as I ran in the reproduction script above.

Thanks,
Taylor

#3 lore raw

PATCH 2/2 stash: avoid sparse-index expansion for in-cone paths

In reply to [email protected]

From: Ted Nyman <[email protected]>

git stash push -- <pathspec> expands a sparse index before checking whether the pathspec matches any tracked paths. This is unnecessary when the pathspec is wholly inside the sparse-checkout cone and makes a path-limited stash proportional to the size of the full index.

Use pathspec_needs_expanded_index() to expand only when a pathspec can match part of a sparse-directory entry, as git rm and git reset already do. Keep the full-index behavior for pathspecs that need it.

Add compatibility coverage for literal, prefixed, wildcard, file, multiple, staged, and missing pathspecs. Add the corresponding path-limited stash case to p2000.

On a cone-mode repository with 349,525 tracked paths and 49 sparse index entries, the best of three runs changed from 18.87s to 0.06s. Trace2 reported four index expansions before this change and none after it.

---
 builtin/stash.c                          |  4 +-
 t/perf/p2000-sparse-operations.sh        |  1 +
 t/t1092-sparse-checkout-compatibility.sh | 55 ++++++++++++++++++++++++
 3 files changed, 58 insertions(+), 2 deletions(-)
builtin/stash.c +2 -2
index c4809f299a313b..72c52571f8c06c 100644
@@ -1702,8 +1702,8 @@ static int do_push_stash(const struct pathspec *ps, const char *stash_msg, int q
 	if (!include_untracked && ps->nr) {
 		char *ps_matched = xcalloc(ps->nr, 1);
 
-		/* TODO: audit for interaction with sparse-index. */
-		ensure_full_index(the_repository->index);
+		if (pathspec_needs_expanded_index(the_repository->index, ps))
+			ensure_full_index(the_repository->index);
 		for (size_t i = 0; i < the_repository->index->cache_nr; i++)
 			ce_path_match(the_repository->index, the_repository->index->cache[i], ps,
 				      ps_matched);
t/perf/p2000-sparse-operations.sh +1 -0
index aadf22bc2f0bb2..548a61cd9064bc 100755
@@ -108,6 +108,7 @@ test_perf_on_all () {
 
 test_perf_on_all git status
 test_perf_on_all 'git stash && git stash pop'
+test_perf_on_all "git stash push -- $SPARSE_CONE/a && git stash pop"
 test_perf_on_all 'echo >>new && git stash -u && git stash pop'
 test_perf_on_all git add -A
 test_perf_on_all git add .
t/t1092-sparse-checkout-compatibility.sh +55 -0

61 lines not on this page · raw message

Reviewed-by on v1 2/2
#4 lore raw
Quoted 6 lines · On Mon, Jul 20, 2026 at 03:31:21PM -0700, [email protected] wrote:
---
 builtin/stash.c                          |  4 +-
 t/perf/p2000-sparse-operations.sh        |  1 +
 t/t1092-sparse-checkout-compatibility.sh | 55 ++++++++++++++++++++++++
 3 files changed, 58 insertions(+), 2 deletions(-)

All looks reasonable, and it's very nice indeed to see another one of these /* TODO */ comments go away ;-).

Very pleasant read, this series is

, and looks good to me.

Thanks,
Taylor

#5 lore raw
Quoted 11 lines · Taylor Blau
Quoted 6 lines · On Mon, Jul 20, 2026 at 03:31:21PM -0700, [email protected] wrote:
---
 builtin/stash.c                          |  4 +-
 t/perf/p2000-sparse-operations.sh        |  1 +
 t/t1092-sparse-checkout-compatibility.sh | 55 ++++++++++++++++++++++++
 3 files changed, 58 insertions(+), 2 deletions(-)

All looks reasonable, and it's very nice indeed to see another one of these /* TODO */ comments go away ;-).

Quoted 8 lines

Very pleasant read, this series is

, and looks good to me.

Thanks,
Taylor

Thanks, both of you. Let me mark the topic for 'next'.