t7300-clean: demonstrate deleting nested repo with an ignored file breakage
PATCH 1/1 t7300-clean: demonstrate deleting nested repo with an ignored file breakage
'git clean -fd' must not delete an untracked directory if it belongs to a different Git repository or worktree. Unfortunately, if a '.gitignore' rule in the outer repository happens to match a file in a nested repository or worktree, then something goes awry and 'git clean -fd' does delete the content of the nested repository's worktree except that ignored file, potentially leading to data loss.
Add a test to 't7300-clean.sh' to demonstrate this breakage.
This issue is a regression introduced in 6b1db43109 (clean: teach clean -d to preserve ignored paths, 2017-05-23).
---
BEWARE: Our toplevel '.gitignore' currently contains the '*.manifest' rule [1], which ignores the file 'compat/win32/git.manifest' [2], so if you use nested worktrees in your git repo, then a 'git clean -fd' will delete them.
[1] 516dfb8416 (.gitignore: touch up the entries regarding Visual
Studio, 2019-07-29)
[2] fe90397604 (mingw: embed a manifest to trick UAC into Doing The
Right Thing, 2019-06-27)t/t7300-clean.sh | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
t/t7300-clean.sh +22 -0
@@ -669,6 +669,28 @@ test_expect_success 'git clean -d skips untracked dirs containing ignored files' test_path_is_missing foo/b/bb ' +test_expect_failure 'git clean -d skips nested repo containing ignored files' ' + test_when_finished "rm -rf nested-repo-with-ignored-file" && + + git init nested-repo-with-ignored-file && + ( + cd nested-repo-with-ignored-file && + >file && + git add file && + git commit -m Initial && + + # This file is ignored by a .gitignore rule in the outer repo + # added in the previous test. + >ignoreme + ) && + + git clean -fd && + + test_path_is_file nested-repo-with-ignored-file/.git/index && + test_path_is_file nested-repo-with-ignored-file/ignoreme && + test_path_is_file nested-repo-with-ignored-file/file +' + test_expect_success MINGW 'handle clean & core.longpaths = false nicely' ' test_config core.longpaths false && a50=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa &&
Signature · 1 line
2.23.0.331.g4e51dcdf11
Quoted 12 lines · SZEDER Gábor
'git clean -fd' must not delete an untracked directory if it belongs to a different Git repository or worktree. Unfortunately, if a '.gitignore' rule in the outer repository happens to match a file in a nested repository or worktree, then something goes awry and 'git clean -fd' does delete the content of the nested repository's worktree except that ignored file, potentially leading to data loss.
Add a test to '
t7300-clean.sh' to demonstrate this breakage.This issue is a regression introduced in
6b1db43109(clean: teach clean -d to preserve ignored paths, 2017-05-23).
Quoted 7 lines
Signed-off-by SZEDER Gábor <[email protected]>---
BEWARE: Our toplevel '.gitignore' currently contains the '*.manifest' rule [1], which ignores the file 'compat/win32/git.manifest' [2], so if you use nested worktrees in your git repo, then a 'git clean -fd' will delete them.
OK, singling out that manifest file is just nonsense, any object file, etc... in the nested worktree/repo can trigger the same issue just as well.
(It just so happened that when I ran 'git clean -fd' I had a nested worktree where I haven't build anything yet, and besides the .git file only that 'git.manifest' file survived in the nested worktree, and then I got misled by it.)
Hi Szeder,
Quoted 1 line · SZEDER Gábor
'git clean -fd' must not delete an untracked directory if it belongs
s/untracked//
I don't believe it should matter either way for a sub-module (sub-directory).
Quoted 1 line
to a different Git repository or worktree.
msybr split the assertion from the fault explanation.
Quoted 4 lines
Unfortunately, if a
'.gitignore' rule in the outer repository happens to match a file in a nested repository or worktree, then something goes awry and 'git clean -fd' does delete the content of the nested repository's worktree
good so far.
Quoted 1 line
except that ignored file, potentially leading to data loss.
this appears at cross purposes as the description has changed from 'ignored/untracked directory' to 'ignored file'. I'm not sure which part the data loss is meant to refer to.
Quoted 56 lines
Add a test to '
t7300-clean.sh' to demonstrate this breakage.This issue is a regression introduced in
6b1db43109(clean: teach clean -d to preserve ignored paths, 2017-05-23).Signed-off-by SZEDER Gábor <[email protected]>---
BEWARE: Our toplevel '.gitignore' currently contains the '*.manifest' rule [1], which ignores the file 'compat/win32/git.manifest' [2], so if you use nested worktrees in your git repo, then a 'git clean -fd' will delete them.
[1]
516dfb8416(.gitignore: touch up the entries regarding VisualStudio, 2019-07-29) [2] fe90397604 (mingw: embed a manifest to trick UAC into Doing The Right Thing, 2019-06-27)t/t7300-clean.sh | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)t/t7300-clean.sh +22
-0@@ -669,6 +669,28 @@ test_expect_success 'git clean -d skips untracked dirs containing ignored files' test_path_is_missing foo/b/bb ' +test_expect_failure 'git clean -d skips nested repo containing ignored files' ' + test_when_finished "rm -rf nested-repo-with-ignored-file" && + + git init nested-repo-with-ignored-file && + ( + cd nested-repo-with-ignored-file && + >file && + git add file && + git commit -m Initial && + + # This file is ignored by a .gitignore rule in the outer repo + # added in the previous test. + >ignoreme + ) && + + git clean -fd && + + test_path_is_file nested-repo-with-ignored-file/.git/index && + test_path_is_file nested-repo-with-ignored-file/ignoreme && + test_path_is_file nested-repo-with-ignored-file/file +' + test_expect_success MINGW 'handle clean & core.longpaths = false nicely' ' test_config core.longpaths false && a50=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa &&
Quoted 7 lines · Philip Oakley
Hi Szeder,
Quoted 1 line · SZEDER Gábor
'git clean -fd' must not delete an untracked directory if it belongs
s/untracked//
I don't believe it should matter either way for a sub-module (sub-directory).
I just paraphrased the documentation of the '-d' option for a bit of context.
Remove untracked directories in addition to untracked files. If an untracked directory is managed by a different Git repository, it is not removed by default. Use -f option twice if you really want to remove such a directory.
Quoted 10 lines
Quoted 1 line
to a different Git repository or worktree.
msybr split the assertion from the fault explanation.
Quoted 4 lines
Unfortunately, if a
'.gitignore' rule in the outer repository happens to match a file in a nested repository or worktree, then something goes awry and 'git clean -fd' does delete the content of the nested repository's worktreegood so far.
Quoted 1 line
except that ignored file, potentially leading to data loss.
this appears at cross purposes as the description has changed from 'ignored/untracked directory' to 'ignored file'.
The description does not mention any ignored directories.
Quoted 2 lines
I'm not sure which part the
data loss is meant to refer to.
Well, there is only one part where the description talks about stuff getting deleted... and that's what it refers to :)
Quoted 7 lines
Quoted 7 lines
Add a test to '
t7300-clean.sh' to demonstrate this breakage.This issue is a regression introduced in
6b1db43109(clean: teach clean -d to preserve ignored paths, 2017-05-23).Signed-off-by SZEDER Gábor <[email protected]>---
On a related note, 'git clean -fdx' does leave the nested repository or worktree intact in the same situation, as it should.
Quoted 36 lines
Quoted 35 lines
t/t7300-clean.sh | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)t/t7300-clean.sh +22
-0@@ -669,6 +669,28 @@ test_expect_success 'git clean -d skips untracked dirs containing ignored files' test_path_is_missing foo/b/bb ' +test_expect_failure 'git clean -d skips nested repo containing ignored files' ' + test_when_finished "rm -rf nested-repo-with-ignored-file" && + + git init nested-repo-with-ignored-file && + ( + cd nested-repo-with-ignored-file && + >file && + git add file && + git commit -m Initial && + + # This file is ignored by a .gitignore rule in the outer repo + # added in the previous test. + >ignoreme + ) && + + git clean -fd && + + test_path_is_file nested-repo-with-ignored-file/.git/index && + test_path_is_file nested-repo-with-ignored-file/ignoreme && + test_path_is_file nested-repo-with-ignored-file/file +' + test_expect_success MINGW 'handle clean & core.longpaths = false nicely' ' test_config core.longpaths false && a50=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa &&
v2 cover Fix some git clean issues
In reply to SZEDER GáborNOTE: This series builds on sg/clean-nested-repo-with-ignored, as it
(among other things) modifies his testcase from expect_failure to expect_success. Also, Peff is probably the only one who remembers v1 (and even he may have forgotten it): v1 was posted a year and a half ago.
This patch series fixes a few issues with git-clean:
* Failure to clean when multiple pathspecs are specified, reported both
in April 2018[1] and again in May 2019[2].
* Failure to preserve both tracked and untracked files within a nested Git repository reported a few weeks ago by SZEDER[3].
[1] https://public-inbox.org/git/[email protected]/
[2] https://public-inbox.org/git/[email protected]/
[3] https://public-inbox.org/git/[email protected]/
I still never got answers to some questions in v1 of my RFC, so after considerable thought I eventually decided to:
* Declare the existing documentation to be ambiguous and hard to interpret correctly; modified the documentation to clearly document 'correct behavior' with how different pieces interact.
* Overrule four regression tests as having the wrong *expectation*, and modify them to have a correct one. That sounds like a backward compatibility issue BUT: The tests were written to check for issues that were orthogonal to the pieces that mattered in this series and thus couldn't be viewed as actually having an opinion on correct behavior on my issues; rather, they were simply reinforcing existing (buggy) implementation results.
* Add a few tests which actually check relevant interactions of parameters and setup, to make this area less ambiguous. (Though one of them was added by SZEDER before my patches, and I should probably add a couple more tests...)
Help from reviewers:
The biggest area I need help from reviewers is to look at the commit messages for patches 9 and 10, to see if folks agree with my declaration of 'correct behavior' and my changes to the regression tests. If those are good, this series can proceed. If they aren't, and someone else can't provide an alternate easy-to-explain 'correct behavior' that we should implement and which is devoid of ugly edge cases for users, then this patch series may languish for another few years.
Other notes:
* Patches 1-6 were included in v1 and have almost no changes (just one fix pointed out by Peff).
* Patch 6's commit message has some additional RFC-related comments and questions, one of which ties in with Patch 9.
* Patch 7 was added as per (old) conversation with Peff.
* Patch 9 & 10 are in most need of review (see above); each has lengthy commit messages.
* It would be nice if someone knows whether the codepath edited in Patch 12 is dead code. If so, we could change that patch to just drop that if-check block. If it's not dead code, that patch fixes what is probably a rare but ugly bug.
Elijah Newren (12):
t7300: Add some testcases showing failure to clean specified pathspecs
dir: fix typo in comment
dir: fix off-by-one error in match_pathspec_item
dir: Directories should be checked for matching pathspecs too
dir: Make the DO_MATCH_SUBMODULE code reusable for a non-submodule
case
dir: If our pathspec might match files under a dir, recurse into it
dir: add commentary explaining match_pathspec_item's return value git-clean.txt: do not claim we will delete files with -n/--dry-run
clean: disambiguate the definition of -d
clean: avoid removing untracked files in a nested git repository
clean: rewrap overly long line
clean: fix theoretical path corruption
Documentation/git-clean.txt | 16 +++++----- builtin/clean.c | 17 ++++++++-- dir.c | 63 +++++++++++++++++++++++++++---------- dir.h | 8 +++-- t/t7300-clean.sh | 44 +++++++++++++++++++++++--- 5 files changed, 114 insertions(+), 34 deletions(-)
Signature · 1 line
2.22.1.11.g45a39ee867
RFC PATCH v2 1/12 t7300: Add some testcases showing failure to clean specified pathspecs
Someone brought me a testcase where multiple git-clean invocations were required to clean out unwanted files:
mkdir d{1,2}
touch d{1,2}/ut
touch d1/t && git add d1/t
With this setup, the user would need to run
git clean -ffd */ut
twice to delete both ut files.
A little testing showed some interesting variants:
* If only one of those two ut files existed (either one), then only one clean command would be necessary.
* If both directories had tracked files, then only one git clean would be necessary to clean both files.
* If both directories had no tracked files then the clean command above would never clean either of the untracked files despite the pathspec explicitly calling both of them out.
A bisect showed that the failure to clean out the files started with commit cf424f5fd89b ("clean: respect pathspecs with "-d", 2014-03-10). However, that pointed to a separate issue: while the "-d" flag was used by the original user who showed me this problem, that flag should have been irrelevant to this problem. Testing again without the "-d" flag showed that the same buggy behavior exists without using that flag, and has in fact existed since before cf424f5fd89b.
Add testcases showing that multiple untracked files within entirely untracked directories cannot be cleaned when specifying these files to git clean via pathspecs.
--- t/t7300-clean.sh | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+)
t/t7300-clean.sh +32 -0
@@ -691,6 +691,38 @@ test_expect_failure 'git clean -d skips nested repo containing ignored files' ' test_path_is_file nested-repo-with-ignored-file/file ' +test_expect_failure 'git clean handles being told what to clean' ' + mkdir -p d1 d2 && + touch d1/ut d2/ut && + git clean -f */ut && + test_path_is_missing d1/ut && + test_path_is_missing d2/ut +' + +test_expect_failure 'git clean handles being told what to clean, with -d' ' + mkdir -p d1 d2 && + touch d1/ut d2/ut && + git clean -ffd */ut && + test_path_is_missing d1/ut && + test_path_is_missing d2/ut +' + +test_expect_failure 'git clean works if a glob is passed without -d' ' + mkdir -p d1 d2 && + touch d1/ut d2/ut && + git clean -f "*ut" && + test_path_is_missing d1/ut && + test_path_is_missing d2/ut +' + +test_expect_failure 'git clean works if a glob is passed with -d' ' + mkdir -p d1 d2 && + touch d1/ut d2/ut && + git clean -ffd "*ut" && + test_path_is_missing d1/ut && + test_path_is_missing d2/ut +' + test_expect_success MINGW 'handle clean & core.longpaths = false nicely' ' test_config core.longpaths false && a50=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa &&
Signature · 1 line
2.22.1.11.g45a39ee867
RFC PATCH v2 2/12 dir: fix typo in comment
In reply to Elijah Newren--- dir.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
dir.c +1 -1
@@ -139,7 +139,7 @@ static size_t common_prefix_len(const struct pathspec *pathspec)* ":(icase)path" is treated as a pathspec full of * wildcard. In other words, only prefix is considered common * prefix. If the pathspec is abc/foo abc/bar, running in -* subdir xyz, the common prefix is still xyz, not xuz/abc as +* subdir xyz, the common prefix is still xyz, not xyz/abc as * in non-:(icase). */ GUARD_PATHSPEC(pathspec,
Signature · 1 line
2.22.1.11.g45a39ee867
RFC PATCH v2 3/12 dir: fix off-by-one error in match_pathspec_item
In reply to Elijah NewrenFor a pathspec like 'foo/bar' comparing against a path named "foo/", namelen will be 4, and match[namelen] will be 'b'. The correct location of the directory separator is namelen-1.
The reason the code worked anyway was that the following code immediately checked whether the first matchlen characters matched (which they do) and then bailed and return MATCHED_RECURSIVELY anyway since wildmatch doesn't have the ability to check if "name" can be matched as a directory (or prefix) against the pathspec.
--- dir.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
dir.c +2 -1
@@ -356,8 +356,9 @@ static int match_pathspec_item(const struct index_state *istate,/* Perform checks to see if "name" is a super set of the pathspec */ if (flags &DO_MATCH_SUBMODULE ) {/* name is a literal prefix of the pathspec */ +int offset = name[namelen-1 ] =='/' ?1 :0 ;if ((namelen < matchlen) && - (match[namelen] =='/' ) && + (match[namelen-offset] =='/' ) && !ps_strncmp (item, match, name, namelen))return MATCHED_RECURSIVELY ;
Signature · 1 line
2.22.1.11.g45a39ee867
RFC PATCH v2 4/12 dir: Directories should be checked for matching pathspecs too
In reply to Elijah NewrenEven if a directory doesn't match a pathspec, it is possible, depending on the precise pathspecs, that some file underneath it might. So we special case and recurse into the directory for such situations. However, we previously always added any untracked directory that we recursed into to the list of untracked paths, regardless of whether the directory itself matched the pathspec.
For the case of git-clean and a set of pathspecs of "dir/file" and "more", this caused a problem because we'd end up with dir entries for both of
"dir"
"dir/file"
Then correct_untracked_entries() would try to helpfully prune duplicates for us by removing "dir/file" since it's under "dir", leaving us with
"dir"
Since the original pathspec only had "dir/file", the only entry left doesn't match and leaves nothing to be removed. (Note that if only one pathspec was specified, e.g. only "dir/file", then the common_prefix_len optimizations in fill_directory would cause us to bypass this problem, making it appear in simple tests that we could correctly remove manually specified pathspecs.)
Fix this by actually checking whether the directory we are about to add to the list of dir entries actually matches the pathspec; only do this matching check after we have already returned from recursing into the directory.
--- dir.c | 5 +++++ t/t7300-clean.sh | 4 ++-- 2 files changed, 7 insertions(+), 2 deletions(-)
dir.c +5 -0
@@ -1951,6 +1951,11 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir, check_only, stop_at_first_file, pathspec);if (subdir_state > dir_state) dir_state = subdir_state; + +if (!match_pathspec (istate, pathspec, path.buf, path.len, +0 /* prefix */ ,NULL , +0 /* do NOT special case dirs */ )) + state = path_none; }if (check_only) {
t/t7300-clean.sh +2 -2
@@ -699,7 +699,7 @@ test_expect_failure 'git clean handles being told what to clean' ' test_path_is_missing d2/ut ' -test_expect_failure 'git clean handles being told what to clean, with -d' ' +test_expect_success 'git clean handles being told what to clean, with -d' ' mkdir -p d1 d2 && touch d1/ut d2/ut && git clean -ffd */ut &&
@@ -715,7 +715,7 @@ test_expect_failure 'git clean works if a glob is passed without -d' ' test_path_is_missing d2/ut ' -test_expect_failure 'git clean works if a glob is passed with -d' ' +test_expect_success 'git clean works if a glob is passed with -d' ' mkdir -p d1 d2 && touch d1/ut d2/ut && git clean -ffd "*ut" &&
Signature · 1 line
2.22.1.11.g45a39ee867
RFC PATCH v2 5/12 dir: Make the DO_MATCH_SUBMODULE code reusable for a non-submodule case
In reply to Elijah NewrenThe specific checks done in match_pathspec_item for the DO_MATCH_SUBMODULE case are useful for other cases which have nothing to do with submodules. Rename this constant; a subsequent commit will make use of this change.
--- dir.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
dir.c +3 -3
@@ -273,7 +273,7 @@ static int do_read_blob(const struct object_id *oid, struct oid_stat *oid_stat,#define DO_MATCH_EXCLUDE (1<<0)#define DO_MATCH_DIRECTORY (1<<1) -#define DO_MATCH_SUBMODULE (1<<2) +#define DO_MATCH_LEADING_PATHSPEC (1<<2)/* * Does 'match' match the given name?
@@ -354,7 +354,7 @@ static int match_pathspec_item(const struct index_state *istate,return MATCHED_FNMATCH ;/* Perform checks to see if "name" is a super set of the pathspec */ -if (flags &DO_MATCH_SUBMODULE ) { +if (flags &DO_MATCH_LEADING_PATHSPEC ) {/* name is a literal prefix of the pathspec */ int offset = name[namelen-1 ] =='/' ?1 :0 ; if ((namelen < matchlen) &&
@@ -498,7 +498,7 @@ int submodule_path_match(const struct index_state *istate,strlen (submodule_name),0 , seen,DO_MATCH_DIRECTORY | - DO_MATCH_SUBMODULE); + DO_MATCH_LEADING_PATHSPEC);return matched; }
Signature · 1 line
2.22.1.11.g45a39ee867
RFC PATCH v2 6/12 dir: If our pathspec might match files under a dir, recurse into it
In reply to Elijah NewrenFor git clean, if a directory is entirely untracked and the user did not specify -d (corresponding to DIR_SHOW_IGNORED_TOO), then we usually do not want to remove that directory and thus do not recurse into it. However, if the user manually specified specific (or even globbed) paths somewhere under that directory to remove, then we need to recurse into the directory to make sure we remove the relevant paths under that directory as the user requested.
Note that this does not mean that the recursed-into directory will be added to dir->entries for later removal; as of a few commits earlier in this series, there is another more strict match check that is run after returning from a recursed-into directory before deciding to add it to the list of entries. Therefore, this will only result in files underneath the given directory which match one of the pathspecs being added to the entries list.
Two particular considerations for this patch:
* If we want to only recurse into a directory when it is specifically matched rather than matched-via-glob (e.g. '*.c'), then we could do so via making the final non-zero return in match_pathspec_item be MATCHED_RECURSIVELY instead of MATCHED_RECURSIVELY_LEADING_PATHSPEC. (See final patch of this RFC series for details; note that the relative order of MATCHED_RECURSIVELY_LEADING_PATHSPEC and MATCHED_RECURSIVELY are important for such a change.))
* There is a growing amount of logic in read_directory_recursive() for deciding whether to recurse into a subdirectory. However, there is a comment immediately preceding this logic that says to recurse if
instructed by treat_path(). It may be better for the logic in
read_directory_recursive() to be moved to treat_path() (or another function it calls, such as treat_directory()), but I did not feel strongly about this and just left the logic where it was while adding to it. Do others have strong opinions on this?
--- dir.c | 10 ++++++---- dir.h | 5 +++-- t/t7300-clean.sh | 4 ++-- 3 files changed, 11 insertions(+), 8 deletions(-)
dir.c +6 -4
@@ -360,7 +360,7 @@ static int match_pathspec_item(const struct index_state *istate,if ((namelen < matchlen) && (match[namelen-offset] =='/' ) && !ps_strncmp (item, match, name, namelen)) -return MATCHED_RECURSIVELY ; +return MATCHED_RECURSIVELY_LEADING_PATHSPEC ;/* name" doesn't match up to the first wild character */ if (item->nowildcard_len < item->len &&
@@ -377,7 +377,7 @@ static int match_pathspec_item(const struct index_state *istate,* The submodules themselves will be able to perform more * accurate matching to determine if the pathspec matches. */ -return MATCHED_RECURSIVELY ; +return MATCHED_RECURSIVELY_LEADING_PATHSPEC ; }return 0 ;
@@ -1939,8 +1939,10 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,/* recurse into subdir if instructed by treat_path */ if ((state == path_recurse) || ((state == path_untracked) && - (dir->flags &DIR_SHOW_IGNORED_TOO ) && - (get_dtype (cdir.de, istate, path.buf, path.len) ==DT_DIR ))) { + (get_dtype (cdir.de, istate, path.buf, path.len) ==DT_DIR ) && + ((dir->flags &DIR_SHOW_IGNORED_TOO ) || +do_match_pathspec (istate, pathspec, path.buf, path.len, + baselen,NULL ,DO_MATCH_LEADING_PATHSPEC ) ==MATCHED_RECURSIVELY_LEADING_PATHSPEC ))) {struct untracked_cache_dir *ud; ud = lookup_untracked(dir->untracked, untracked, path.buf + baselen,
dir.h +3 -2
@@ -211,8 +211,9 @@ int count_slashes(const char *s);* when populating the seen[] array. */ #define MATCHED_RECURSIVELY 1 -#define MATCHED_FNMATCH 2 -#define MATCHED_EXACTLY 3 +#define MATCHED_RECURSIVELY_LEADING_PATHSPEC 2 +#define MATCHED_FNMATCH 3 +#define MATCHED_EXACTLY 4int simple_length (const char *match);int no_wildcard (const char *string);char *common_prefix (const struct pathspec *pathspec);
t/t7300-clean.sh +2 -2
@@ -691,7 +691,7 @@ test_expect_failure 'git clean -d skips nested repo containing ignored files' ' test_path_is_file nested-repo-with-ignored-file/file ' -test_expect_failure 'git clean handles being told what to clean' ' +test_expect_success 'git clean handles being told what to clean' ' mkdir -p d1 d2 && touch d1/ut d2/ut && git clean -f */ut &&
@@ -707,7 +707,7 @@ test_expect_success 'git clean handles being told what to clean, with -d' ' test_path_is_missing d2/ut ' -test_expect_failure 'git clean works if a glob is passed without -d' ' +test_expect_success 'git clean works if a glob is passed without -d' ' mkdir -p d1 d2 && touch d1/ut d2/ut && git clean -f "*ut" &&
Signature · 1 line
2.22.1.11.g45a39ee867
RFC PATCH v2 7/12 dir: add commentary explaining match_pathspec_item's return value
In reply to Elijah NewrenThe way match_pathspec_item() handles names and pathspecs with trailing slash characters, in conjunction with special options like DO_MATCH_DIRECTORY and DO_MATCH_LEADING_PATHSPEC were non-obvious, and broken until this patch series. Add a table in a comment explaining the intent of how these work.
--- dir.c | 27 +++++++++++++++++++-------- 1 file changed, 19 insertions(+), 8 deletions(-)
dir.c +19 -8
@@ -276,16 +276,27 @@ static int do_read_blob(const struct object_id *oid, struct oid_stat *oid_stat,#define DO_MATCH_LEADING_PATHSPEC (1<<2)/* -* Does 'match' match the given name? -* A match is found if +* Does the given pathspec match the given name? A match is found if * -* (1) the 'match' string is leading directory of 'name', or -* (2) the 'match' string is a wildcard and matches 'name', or -* (3) the 'match' string is exactly the same as 'name'. +* (1) the pathspec string is leading directory of 'name' ("RECURSIVELY"), or +* (2) the pathspec string has a leading part matching 'name' ("LEADING"), or +* (3) the pathspec string is a wildcard and matches 'name' ("WILDCARD"), or +* (4) the pathspec string is exactly the same as 'name' ("EXACT"). * -* and the return value tells which case it was. +* Return value tells which case it was (1-4), or 0 when there is no match. * -* It returns 0 when there is no match. +* It may be instructive to look at a small table of concrete examples +* to understand the differences between 1, 2, and 4: +* +* Pathspecs +* | a/b | a/b/ | a/b/c +* ------+-----------+-----------+------------ +* a/b | EXACT | EXACT[1] | LEADING[2] +* Names a/b/ | RECURSIVE | EXACT | LEADING[2] +* a/b/c | RECURSIVE | RECURSIVE | EXACT +* +* [1] Only if DO_MATCH_DIRECTORY is passed; otherwise, this is NOT a match. +* [2] Only if DO_MATCH_LEADING_PATHSPEC is passed; otherwise, not a match. */ static int match_pathspec_item(const struct index_state *istate, const struct pathspec_item *item, int prefix,
@@ -353,7 +364,7 @@ static int match_pathspec_item(const struct index_state *istate, item->nowildcard_len - prefix))return MATCHED_FNMATCH ; -/* Perform checks to see if "name" is a super set of the pathspec */ +/* Perform checks to see if "name" is a leading string of the pathspec */ if (flags &DO_MATCH_LEADING_PATHSPEC ) {/* name is a literal prefix of the pathspec */ int offset = name[namelen-1 ] =='/' ?1 :0 ;
Signature · 1 line
2.22.1.11.g45a39ee867
RFC PATCH v2 8/12 git-clean.txt: do not claim we will delete files with -n/--dry-run
In reply to Elijah NewrenIt appears that the wrong option got included in the list of what will cause git-clean to actually take action. Correct the list.
--- Documentation/git-clean.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Documentation/git-clean.txt +1 -1
@@ -35,7 +35,7 @@ OPTIONS --force:: If the Git configuration variable clean.requireForce is not set to false, 'git clean' will refuse to delete files or directories - unless given -f, -n or -i. Git will refuse to delete directories + unless given -f or -i. Git will refuse to delete directories with .git sub directory or file unless a second -f is given.
Signature · 1 line
2.22.1.11.g45a39ee867
RFC PATCH v2 9/12 clean: disambiguate the definition of -d
In reply to Elijah NewrenThe -d flag pre-dated git-clean's ability to have paths specified. As such, the default for git-clean was to only remove untracked files in the current directory, and -d existed to allow it to recurse into subdirectories.
The interaction of paths and the -d option appears to not have been carefully considered, as evidenced by numerous bugs and a dearth of tests covering such pairings in the testsuite. The definition turns out to be important, so let's look at some of the various ways one could interpret the -d option:
A) Without -d, only look in subdirectories which contain tracked files under them; with -d, also look in subdirectories which are untracked for files to clean.
B) Without specified paths from the user for us to delete, we need to have some kind of default, so...without -d, only look in subdirectories which contain tracked files under them; with -d, also look in subdirectories which are untracked for files to clean.
The important distinction here is that choice B says that the presence or absence of '-d' is irrelevant if paths are specified. The logic behind option B is that if a user explicitly asked us to clean a specified pathspec, then we should clean anything that matches that pathspec. Some examples may clarify. Should
git clean -f untracked_dir/file
remove untracked_dir/file or not? It seems crazy not to, but a strict reading of option A says it shouldn't be removed. How about
git clean -f untracked_dir/file1 tracked_dir/file2
or
git clean -f untracked_dir_1/file1 untracked_dir_2/file2
? Should it remove either or both of these files? Should it require multiple runs to remove both the files listed? (If this sounds like a crazy question to even ask, see the commit message of "t7300: Add some testcases showing failure to clean specified pathspecs" added earlier in this patch series.) What if -ffd were used instead of -f -- should that allow these to be removed? Should it take multiple invocations with -ffd? What if a glob (such as '*tracked*') were used instead of spelling out the directory names? What if the filenames involved globs, such as
git clean -f '*.o'
or
git clean -f '*/*.o'
?
The current documentation actually suggests a definition that is slightly different than choice A, and the implementation prior to this series provided something radically different than either choices A or B. (The implementation, though, was clearly just buggy). There may be other choices as well. However, for almost any given choice of definition for -d that I can think of, some of the examples above will appear buggy to the user. The only case that doesn't have negative surprises is choice B: treat a user-specified path as a request to clean all untracked files which match that path specification, including recursing into any untracked directories.
Change the documentation and basic implementation to use this definition.
There were two regression tests that indirectly depended on the current implementation, but neither was about subdirectory handling. These two tests were introduced in commit 5b7570cfb41c ("git-clean: add tests for relative path", 2008-03-07) which was solely created to add coverage for the changes in commit fb328947c8e ("git-clean: correct printing relative path", 2008-03-07). Both tests specified a directory that happened to have an untracked subdirectory, but both were only checking that the resulting printout of a file that was removed was shown with a relative path. Update these tests appropriately.
--- Documentation/git-clean.txt | 10 ++++++---- builtin/clean.c | 8 ++++++++ t/t7300-clean.sh | 2 ++ 3 files changed, 16 insertions(+), 4 deletions(-)
Documentation/git-clean.txt +6 -4
@@ -26,10 +26,12 @@ are affected. OPTIONS ------- -d:: - Remove untracked directories in addition to untracked files. - If an untracked directory is managed by a different Git - repository, it is not removed by default. Use -f option twice - if you really want to remove such a directory. + Normally, when no <path> is specified, git clean will not + recurse into untracked directories to avoid removing too much. + Specify -d to have it recurse into such directories as well. + If any paths are specified, -d is irrelevant; all untracked + files matching the specified paths (with exceptions for nested + git directories mentioned under `--force`) will be removed. -f:: --force::
builtin/clean.c +8 -0
@@ -949,6 +949,14 @@ int cmd_clean(int argc, const char **argv, const char *prefix) dir.flags |=DIR_SHOW_OTHER_DIRECTORIES ; +if (argc) { +/* +* Remaining args implies pathspecs specified, and we should +* recurse within those. +*/ + remove_directories =1 ; + } +if (remove_directories) dir.flags |=DIR_SHOW_IGNORED_TOO |DIR_KEEP_UNTRACKED_CONTENTS ;
t/t7300-clean.sh +2 -0
@@ -117,6 +117,7 @@ test_expect_success C_LOCALE_OUTPUT 'git clean with relative prefix' ' would_clean=$( cd docs && git clean -n ../src | + grep part3 | sed -n -e "s|^Would remove ||p" ) && verbose test "$would_clean" = ../src/part3.c
@@ -129,6 +130,7 @@ test_expect_success C_LOCALE_OUTPUT 'git clean with absolute path' ' would_clean=$( cd docs && git clean -n "$(pwd)/../src" | + grep part3 | sed -n -e "s|^Would remove ||p" ) && verbose test "$would_clean" = ../src/part3.c
Signature · 1 line
2.22.1.11.g45a39ee867
RFC PATCH v2 10/12 clean: avoid removing untracked files in a nested git repository
In reply to Elijah NewrenUsers expect files in a nested git repository to be left alone unless sufficiently forced (with two -f's). Unfortunately, in certain circumstances, git would delete both tracked (and possibly dirty) files and untracked files within a nested repository. To explain how this happens, let's contrast a couple cases. First, take the following example setup (which assumes we are already within a git repo):
git init nested
cd nested
Quoted 1 line
tracked
git add tracked
git commit -m init
Quoted 1 line
untracked
cd ..
In this setup, everything works as expected; running 'git clean -fd' will result in fill_directory() returning the following paths:
nested/
nested/tracked
nested/untracked
and then correct_untracked_entries() would notice this can be compressed to
nested/
and then since "nested/" is a directory, we would call remove_dirs("nested/", ...), which would
check is_nonbare_repository_dir() and then decide to skip it.
However, if someone also creates an ignored file:
Quoted 1 line
nested/ignored
then running 'git clean -fd' would result in fill_directory() returning the same paths:
nested/
nested/tracked
nested/untracked
but correct_untracked_entries() will notice that we had ignored entries under nested/ and thus simplify this list to
nested/tracked
nested/untracked
Since these are not directories, we do not call remove_dirs() which was the only place that had the is_nonbare_repository_dir() safety check -- resulting in us deleting both the untracked file and the tracked (and possibly dirty) file.
One possible fix for this issue would be walking the parent directories of each path and checking if they represent nonbare repositories, but that would be wasteful. Even if we added caching of some sort, it's still a waste because we should have been able to check that "nested/" represented a nonbare repository before even descending into it in the first place. Add a DIR_SKIP_NESTED_GIT flag to dir_struct.flags and use it to prevent fill_directory() and friends from descending into nested git repos.
With this change, we also modify two regression tests added in commit 91479b9c72f1 ("t7300: add tests to document behavior of clean and nested git", 2015-06-15). That commit, nor its series, nor the six previous iterations of that series on the mailing list discussed why those tests coded the expectation they did. In fact, it appears their purpose was simply to test _existing_ behavior to make sure that the performance changes didn't change the behavior. However, these two tests directly contradicted the manpage's claims that two -f's were required to delete files/directories under a nested git repository. While one could argue that the user gave an explicit path which matched files/directories that were within a nested repository, there's a slippery slope that becomes very difficult for users to understand once you go down that route (e.g. what if they specified "git clean -f -d '*.c'"?) It would also be hard to explain what the exact behavior was; avoid such problems by making it really simple.
Also, clean up some grammar errors describing this functionality in the git-clean manpage.
Finally, there is one somewhat related bug which this patch does not fix, coming from the opposite angle. If the user runs
git clean -ffd
to force deletion of untracked nested repositories, and within an untracked nested repo the user has ignored files (according to the inner OR outer repositories' .gitignore), then not only will those ignored files be left alone but the .git/ subdirectory of the nested repo will be left alone too. I am not completely sure if this should be considered a bug (though it seems like it since the lack of the untracked file would result in the .git/ subdirectory being deleted), but in any event it is very minor compared to accidentally deleting user data and I did not dive into it.
--- Documentation/git-clean.txt | 6 +++--- builtin/clean.c | 2 ++ dir.c | 10 ++++++++++ dir.h | 3 ++- t/t7300-clean.sh | 10 +++++----- 5 files changed, 22 insertions(+), 9 deletions(-)
Documentation/git-clean.txt +3 -3
@@ -37,9 +37,9 @@ OPTIONS --force:: If the Git configuration variable clean.requireForce is not set to false, 'git clean' will refuse to delete files or directories - unless given -f or -i. Git will refuse to delete directories - with .git sub directory or file unless a second -f - is given. + unless given -f or -i. Git will refuse to modify untracked + nested git repositories (directories with a .git subdirectory) + unless a second -f is given. -i:: --interactive::
builtin/clean.c +2 -0
@@ -946,6 +946,8 @@ int cmd_clean(int argc, const char **argv, const char *prefix)if (force >1 ) rm_flags =0 ; +else + dir.flags |=DIR_SKIP_NESTED_GIT ; dir.flags |=DIR_SHOW_OTHER_DIRECTORIES ;
dir.c +10 -0
@@ -1451,6 +1451,16 @@ static enum path_treatment treat_directory(struct dir_struct *dir,return path_none;case index_nonexistent: +if (dir->flags &DIR_SKIP_NESTED_GIT ) { +int nested_repo; +struct strbuf sb =STRBUF_INIT ; +strbuf_addstr (&sb, dirname); + nested_repo =is_nonbare_repository_dir (&sb); +strbuf_release (&sb); +if (nested_repo) +return path_none; + } +if (dir->flags &DIR_SHOW_OTHER_DIRECTORIES )break ; if (exclude &&
dir.h +2 -1
@@ -156,7 +156,8 @@ struct dir_struct {DIR_SHOW_IGNORED_TOO =1 <<5 ,DIR_COLLECT_KILLED_ONLY =1 <<6 ,DIR_KEEP_UNTRACKED_CONTENTS =1 <<7 , - DIR_SHOW_IGNORED_TOO_MODE_MATCHING = 1<<8 +DIR_SHOW_IGNORED_TOO_MODE_MATCHING =1 <<8 , + DIR_SKIP_NESTED_GIT = 1<<9 } flags;struct dir_entry **entries;struct dir_entry **ignored;
t/t7300-clean.sh +5 -5
@@ -549,7 +549,7 @@ test_expect_failure 'nested (non-empty) bare repositories should be cleaned even test_path_is_missing strange_bare ' -test_expect_success 'giving path in nested git work tree will remove it' ' +test_expect_success 'giving path in nested git work tree will NOT remove it' ' rm -fr repo && mkdir repo && (
@@ -561,7 +561,7 @@ test_expect_success 'giving path in nested git work tree will remove it' ' git clean -f -d repo/bar/baz && test_path_is_file repo/.git/HEAD && test_path_is_dir repo/bar/ && - test_path_is_missing repo/bar/baz + test_path_is_file repo/bar/baz/hello.world ' test_expect_success 'giving path to nested .git will not remove it' '
@@ -579,7 +579,7 @@ test_expect_success 'giving path to nested .git will not remove it' ' test_path_is_dir untracked/ ' -test_expect_success 'giving path to nested .git/ will remove contents' ' +test_expect_success 'giving path to nested .git/ will NOT remove contents' ' rm -fr repo untracked && mkdir repo untracked && (
@@ -589,7 +589,7 @@ test_expect_success 'giving path to nested .git/ will remove contents' ' ) && git clean -f -d repo/.git/ && test_path_is_dir repo/.git && - test_dir_is_empty repo/.git && + test_path_is_file repo/.git/HEAD && test_path_is_dir untracked/ '
@@ -671,7 +671,7 @@ test_expect_success 'git clean -d skips untracked dirs containing ignored files' test_path_is_missing foo/b/bb ' -test_expect_failure 'git clean -d skips nested repo containing ignored files' ' +test_expect_success 'git clean -d skips nested repo containing ignored files' ' test_when_finished "rm -rf nested-repo-with-ignored-file" && git init nested-repo-with-ignored-file &&
Signature · 1 line
2.22.1.11.g45a39ee867
Re: [RFC PATCH v2 10/12] clean: avoid removing untracked files in a nested git repository
Quoted 52 lines · Elijah Newren
Users expect files in a nested git repository to be left alone unless sufficiently forced (with two -f's). Unfortunately, in certain circumstances, git would delete both tracked (and possibly dirty) files and untracked files within a nested repository. To explain how this happens, let's contrast a couple cases. First, take the following example setup (which assumes we are already within a git repo):
git init nested
cd nestedQuoted 1 line
tracked
git add tracked
git commit -m initQuoted 1 line
untracked
cd ..
In this setup, everything works as expected; running 'git clean -fd' will result in
fill_directory()returning the following paths:
nested/
nested/tracked
nested/untracked
and thencorrect_untracked_entries()would notice this can be compressed to
nested/
and then since "nested/" is a directory, we would callremove_dirs("nested/", ...), which would
checkis_nonbare_repository_dir()and then decide to skip it.However, if someone also creates an ignored file:
Quoted 1 line
nested/ignored
then running 'git clean -fd' would result in
fill_directory()returning the same paths:
nested/
nested/tracked
nested/untracked
butcorrect_untracked_entries()will notice that we had ignored entries under nested/ and thus simplify this list to
nested/tracked
nested/untracked
Since these are not directories, we do not callremove_dirs()which was the only place that had theis_nonbare_repository_dir()safety check -- resulting in us deleting both the untracked file and the tracked (and possibly dirty) file.One possible fix for this issue would be walking the parent directories of each path and checking if they represent nonbare repositories, but that would be wasteful. Even if we added caching of some sort, it's still a waste because we should have been able to check that "nested/" represented a nonbare repository before even descending into it in the first place. Add a
DIR_SKIP_NESTED_GITflag to dir_struct.flags and use it to preventfill_directory()and friends from descending into nested git repos.
Quoted 12 lines
Finally, there is one somewhat related bug which this patch does not fix, coming from the opposite angle. If the user runs
git clean -ffd
to force deletion of untracked nested repositories, and within an untracked nested repo the user has ignored files (according to the inner OR outer repositories' .gitignore), then not only will those ignored files be left alone but the .git/ subdirectory of the nested repo will be left alone too. I am not completely sure if this should be considered a bug (though it seems like it since the lack of the untracked file would result in the .git/ subdirectory being deleted), but in any event it is very minor compared to accidentally deleting user data and I did not dive into it.
We briefly mentioned this "ignored file in a nested repo fools 'git clean -d'" issue in an unrelated thread as well, where Philip suggested that the gitignore of the outer repository should not have any effect on the nested repository. I'm inclined to agree.
https://public-inbox.org/git/[email protected]/
Now, 'git clean -X' is supposed to "Remove only files ignored by Git.". I'm not entirely sure what 'git clean -ffdX' is supposed to do (or whether it makes any sense in the first place), but it does delete files in the nested repository that are ignored only in the outer repository, both tracked (and possibly dirty) and untracked, even with this patch series. Without this series '-fdX' is just as bad, but with this patch (i.e. by not descending into nested repositories) '-fdX' becomes sensible and leaves the nested repository alone.
Quoted 5 lines
Documentation/git-clean.txt +0
-0
Documentation/git-clean.txt OPTIONS · line 42
Quoted 9 lines
Documentation/git-clean.txt +3
-3@@ -37,9 +37,9 @@ OPTIONS --force:: If the Git configuration variable clean.requireForce is not set to false, 'git clean' will refuse to delete files or directories - unless given -f or -i. Git will refuse to delete directories - with .git sub directory or file unless a second -f - is given. + unless given -f or -i. Git will refuse to modify untracked + nested git repositories (directories with a .git subdirectory) + unless a second -f is given.
I like this wording.
RFC PATCH v2 11/12 clean: rewrap overly long line
In reply to Elijah Newren--- builtin/clean.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
builtin/clean.c +2 -1
@@ -158,7 +158,8 @@ static int remove_dirs(struct strbuf *path, const char *prefix, int force_flag, *dir_gone =1 ; -if ((force_flag &REMOVE_DIR_KEEP_NESTED_GIT ) &&is_nonbare_repository_dir (path)) { +if ((force_flag &REMOVE_DIR_KEEP_NESTED_GIT ) && +is_nonbare_repository_dir (path)) {if (!quiet) {quote_path_relative (path->buf, prefix, "ed); printf(dry_run ? _(msg_would_skip_git_dir) : _(msg_skip_git_dir),
Signature · 1 line
2.22.1.11.g45a39ee867
RFC PATCH v2 12/12 clean: fix theoretical path corruption
In reply to Elijah Newrencmd_clean() had the following code structure:
struct strbuf abs_path = STRBUF_INIT;
for_each_string_list_item(item, &del_list) {
strbuf_addstr(&abs_path, prefix);
strbuf_addstr(&abs_path, item->string);
PROCESS(&abs_path);
strbuf_reset(&abs_path);
}where I've elided a bunch of unnecessary details and PROCESS(&abs_path) represents a big chunk of code rather than an actual function call. One piece of PROCESS was:
if (lstat(abs_path.buf, &st))
continue;which would cause the strbuf_reset() to be missed -- meaning that the next path to be handled would have two paths concatenated. This path used to use die_errno() instead of continue prior to commit 396049e5fb62 ("git-clean: refactor git-clean into two phases", 2013-06-25), but my understanding of how correct_untracked_entries() works is that it will prevent both dir/ and dir/file from being in the list to clean so this should be dead code and the die_errno() should be safe. But I hesitate to remove it since I am not certain. Instead, just fix it to avoid path corruption in case it is possible to reach this continue statement.
--- builtin/clean.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
builtin/clean.c +3 -1
@@ -1028,8 +1028,10 @@ int cmd_clean(int argc, const char **argv, const char *prefix)* recursive directory removal, so lstat() here could * fail with ENOENT. */ -if (lstat (abs_path.buf, &st)) +if (lstat (abs_path.buf, &st)) { +strbuf_reset (&abs_path);continue ; + }if (S_ISDIR (st.st_mode)) {if (remove_dirs (&abs_path, prefix, rm_flags, dry_run, quiet, &gone))
Signature · 1 line
2.22.1.11.g45a39ee867
Re: [RFC PATCH v2 12/12] clean: fix theoretical path corruption
Quoted 37 lines · Elijah Newren
cmd_clean()had the following code structure:struct strbuf abs_path = STRBUF_INIT; for_each_string_list_item(item, &del_list) { strbuf_addstr(&abs_path, prefix); strbuf_addstr(&abs_path, item->string); PROCESS(&abs_path); strbuf_reset(&abs_path); }where I've elided a bunch of unnecessary details and
PROCESS(&abs_path)represents a big chunk of code rather than an actual function call. One piece of PROCESS was:if (lstat(abs_path.buf, &st)) continue;which would cause the
strbuf_reset()to be missed -- meaning that the next path to be handled would have two paths concatenated. This path used to usedie_errno()instead of continue prior to commit396049e5fb62("git-clean: refactor git-clean into two phases", 2013-06-25), but my understanding of howcorrect_untracked_entries()works is that it will prevent both dir/ and dir/file from being in the list to clean so this should be dead code and thedie_errno()should be safe. But I hesitate to remove it since I am not certain. Instead, just fix it to avoid path corruption in case it is possible to reach this continue statement.Signed-off-by Elijah Newren <[email protected]>--- builtin/clean.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)builtin/clean.c +0
-0
builtin/clean.c int cmd_clean(int argc, const char **argv, const char *prefix) · line 1034
Quoted 8 lines
builtin/clean.c +3
-1@@ -1028,8 +1028,10 @@ int cmd_clean(int argc, const char **argv, const char *prefix)* recursive directory removal, so lstat() here could * fail with ENOENT. */ -if (lstat (abs_path.buf, &st)) +if (lstat (abs_path.buf, &st)) { +strbuf_reset (&abs_path);continue ; + }
I wonder whether it would be safer to call strbuf_reset() at the start of each loop iteration instead of before 'continue'. That way we wouldn't have to worry about another 'continue' statements forgetting about it.
It probably doesn't really matter in this particular case (considering that it's potentially dead code to begin with), but have a look at e.g. diff.c:show_stats() and its several strbuf_reset(&out) calls preceeding continue statements.
Quoted 5 lines
builtin/clean.c +0
-0if (S_ISDIR (st.st_mode)) {if (remove_dirs (&abs_path, prefix, rm_flags, dry_run, quiet, &gone))Signature · 1 line
2.22.1.11.g45a39ee867
Re: [RFC PATCH v2 12/12] clean: fix theoretical path corruption
Quoted 53 lines · SZEDER Gábor
Quoted 37 lines · Elijah Newren
cmd_clean()had the following code structure:struct strbuf abs_path = STRBUF_INIT; for_each_string_list_item(item, &del_list) { strbuf_addstr(&abs_path, prefix); strbuf_addstr(&abs_path, item->string); PROCESS(&abs_path); strbuf_reset(&abs_path); }where I've elided a bunch of unnecessary details and
PROCESS(&abs_path)represents a big chunk of code rather than an actual function call. One piece of PROCESS was:if (lstat(abs_path.buf, &st)) continue;which would cause the
strbuf_reset()to be missed -- meaning that the next path to be handled would have two paths concatenated. This path used to usedie_errno()instead of continue prior to commit396049e5fb62("git-clean: refactor git-clean into two phases", 2013-06-25), but my understanding of howcorrect_untracked_entries()works is that it will prevent both dir/ and dir/file from being in the list to clean so this should be dead code and thedie_errno()should be safe. But I hesitate to remove it since I am not certain. Instead, just fix it to avoid path corruption in case it is possible to reach this continue statement.Signed-off-by Elijah Newren <[email protected]>--- builtin/clean.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)builtin/clean.c +0
-0builtin/clean.c int cmd_clean(int argc, const char **argv, const char *prefix) · line 1034
Quoted 8 lines
builtin/clean.c +3
-1@@ -1028,8 +1028,10 @@ int cmd_clean(int argc, const char **argv, const char *prefix)* recursive directory removal, so lstat() here could * fail with ENOENT. */ -if (lstat (abs_path.buf, &st)) +if (lstat (abs_path.buf, &st)) { +strbuf_reset (&abs_path);continue ; + }I wonder whether it would be safer to call
strbuf_reset()at the start of each loop iteration instead of before 'continue'. That way we wouldn't have to worry about another 'continue' statements forgetting about it.
Quoted 4 lines
It probably doesn't really matter in this particular case (considering that it's potentially dead code to begin with), but have a look at e.g. diff.c:show_stats() and its several
strbuf_reset(&out)calls preceeding continue statements.
Ooh, I like that idea. I think I'll apply that here. I'll probably leave diff.c:show_stats() as #leftoverbits for someone else, though I really like the idea of fixing up other issues like this as you suggest.
Re: [RFC PATCH v2 00/12] Fix some git clean issues
In reply to Elijah NewrenQuoted 4 lines · Elijah Newren
This patch series fixes a few issues with git-clean:
* Failure to preserve both tracked and untracked files within a nested Git repository reported a few weeks ago by SZEDER[3].
Wow, I didn't expect a 12 patch series to fix that issue... Thanks.
Quoted 8 lines
Elijah Newren (12):
t7300: Add some testcases showing failure to clean specified pathspecs
dir: fix typo in comment
dir: fix off-by-one error inmatch_pathspec_item
dir: Directories should be checked for matching pathspecs too
dir: Make theDO_MATCH_SUBMODULEcode reusable for a non-submodule
case
dir: If our pathspec might match files under a dir, recurse into it
Nit: no capital letters after the '<area>:' prefix.
Quoted 17 lines
dir: add commentary explaining match_pathspec_item's return value git-clean.txt: do not claim we will delete files with -n/--dry-run
clean: disambiguate the definition of -d
clean: avoid removing untracked files in a nested git repository
clean: rewrap overly long line
clean: fix theoretical path corruptionDocumentation/git-clean.txt | 16 +++++----- builtin/clean.c | 17 ++++++++-- dir.c | 63 +++++++++++++++++++++++++++---------- dir.h | 8 +++-- t/t7300-clean.sh | 44 +++++++++++++++++++++++--- 5 files changed, 114 insertions(+), 34 deletions(-)Signature · 1 line
2.22.1.11.g45a39ee867
Re: [RFC PATCH v2 00/12] Fix some git clean issues
Quoted 9 lines · SZEDER Gábor
Quoted 4 lines · Elijah Newren
This patch series fixes a few issues with git-clean:
* Failure to preserve both tracked and untracked files within a nested Git repository reported a few weeks ago by SZEDER[3].
Wow, I didn't expect a 12 patch series to fix that issue... Thanks.
Well, to be fair, only the last three patches were about that issue. The first 9 were about the other issues. It's just that your testcase reminded me of my old series and gave me another nudge to dig it out and see if it helped with your problem. I had to rebase it and look back over it, and then found it didn't help with your problem, but by then I had refamiliarized myself with the code so...
Quoted 10 lines
Quoted 8 lines
Elijah Newren (12):
t7300: Add some testcases showing failure to clean specified pathspecs
dir: fix typo in comment
dir: fix off-by-one error inmatch_pathspec_item
dir: Directories should be checked for matching pathspecs too
dir: Make theDO_MATCH_SUBMODULEcode reusable for a non-submodule
case
dir: If our pathspec might match files under a dir, recurse into itNit: no capital letters after the '<area>:' prefix.
Gah, I should know that any patch series I submitted from a year and a half ago probably made that mistake. I've mostly trained myself out of it now, but I certainly hadn't back then.
Thanks for pointing it out; will fix.
v3 cover Fix some git clean issues
In reply to Elijah NewrenNOTE: This series builds on sg/clean-nested-repo-with-ignored, as it
(among other things) modifies his testcase from expect_failure to expect_success. Also, Peff is probably the only one who remembers v1 (and even he may have forgotten it): v1 was posted a year and a half ago.
This patch series fixes a few issues with git-clean:
* Failure to clean when multiple pathspecs are specified, reported both
in April 2018[1] and again in May 2019[2].
* Failure to preserve both tracked and untracked files within a nested Git repository reported a few weeks ago by SZEDER[3].
[1] https://public-inbox.org/git/[email protected]/
[2] https://public-inbox.org/git/[email protected]/
[3] https://public-inbox.org/git/[email protected]/
Changes since v2:
* Removed the RFC label
* Fixed up a few things SZEDER pointed out in v2 -- some commit message improvements, and an extra safety precaution for the final patch.
Stuff I'd like reviewer to focus on:
* Read the (long) commit messages for patches 9 & 10; do folks agree with my declaration of "correct" behavior and my changes to the few regression tests in those patches?
Elijah Newren (12):
t7300: add testcases showing failure to clean specified pathspecs
dir: fix typo in comment
dir: fix off-by-one error in match_pathspec_item
dir: also check directories for matching pathspecs
dir: make the DO_MATCH_SUBMODULE code reusable for a non-submodule
case
dir: if our pathspec might match files under a dir, recurse into it
dir: add commentary explaining match_pathspec_item's return value git-clean.txt: do not claim we will delete files with -n/--dry-run
clean: disambiguate the definition of -d
clean: avoid removing untracked files in a nested git repository
clean: rewrap overly long line
clean: fix theoretical path corruption
Documentation/git-clean.txt | 16 +++++----- builtin/clean.c | 15 +++++++-- dir.c | 63 +++++++++++++++++++++++++++---------- dir.h | 8 +++-- t/t7300-clean.sh | 44 +++++++++++++++++++++++--- 5 files changed, 112 insertions(+), 34 deletions(-)
Range-diff:
1: 82328e2033 ! 1: fe35ab8cc3 t7300: Add some testcases showing failure to clean specified pathspecs
@@ Metadata
Author: Elijah Newren <[email protected]>
## Commit message ##
- t7300: Add some testcases showing failure to clean specified pathspecs
+ t7300: add testcases showing failure to clean specified pathspecs
Someone brought me a testcase where multiple git-clean invocations were required to clean out unwanted files:
2: 5c1f58fd9d = 2: 707d287d79 dir: fix typo in comment 3: 0e8b415af3 = 3: bb316e82b2 dir: fix off-by-one error in match_pathspec_item 4: 30b3ede443 ! 4: 56319f934a dir: Directories should be checked for matching pathspecs too
@@ Metadata
Author: Elijah Newren <[email protected]>
## Commit message ##
- dir: Directories should be checked for matching pathspecs too
+ dir: also check directories for matching pathspecs
Even if a directory doesn't match a pathspec, it is possible, depending
on the precise pathspecs, that some file underneath it might. So we
5: bab01f4cda ! 5: 81593a565c dir: Make the DO_MATCH_SUBMODULE code reusable for a non-submodule case
@@ Metadata
Author: Elijah Newren <[email protected]>
## Commit message ##
- dir: Make the DO_MATCH_SUBMODULE code reusable for a non-submodule case
+ dir: make the DO_MATCH_SUBMODULE code reusable for a non-submodule case
The specific checks done in match_pathspec_item for the DO_MATCH_SUBMODULE case are useful for other cases which have nothing to do with submodules.
6: c619ab4b3e ! 6: 9566823a0f dir: If our pathspec might match files under a dir, recurse into it
@@ Metadata
Author: Elijah Newren <[email protected]>
## Commit message ##
- dir: If our pathspec might match files under a dir, recurse into it
+ dir: if our pathspec might match files under a dir, recurse into it
For git clean, if a directory is entirely untracked and the user did not specify -d (corresponding to DIR_SHOW_IGNORED_TOO), then we usually do
@@ Commit message
the given directory which match one of the pathspecs being added to the entries list.
- Two particular considerations for this patch:
+ Two notes of potential interest to future readers:
- * If we want to only recurse into a directory when it is specifically
+ * If we wanted to only recurse into a directory when it is specifically
matched rather than matched-via-glob (e.g. '*.c'), then we could do so via making the final non-zero return in match_pathspec_item be MATCHED_RECURSIVELY instead of MATCHED_RECURSIVELY_LEADING_PATHSPEC.
- (See final patch of this RFC series for details; note that the
- relative order of MATCHED_RECURSIVELY_LEADING_PATHSPEC and
- MATCHED_RECURSIVELY are important for such a change.))
+ (Note that the relative order of MATCHED_RECURSIVELY_LEADING_PATHSPEC
+ and MATCHED_RECURSIVELY are important for such a change.) I was
+ leaving open that possibility while writing an RFC asking for the
+ behavior we want, but even though we don't want it, that knowledge
+ might help you understand the code flow better.
* There is a growing amount of logic in read_directory_recursive() for
- deciding whether to recurse into a subdirectory. However, there is
- a comment immediately preceding this logic that says to recurse if
+ deciding whether to recurse into a subdirectory. However, there is a
+ comment immediately preceding this logic that says to recurse if
instructed by treat_path(). It may be better for the logic in
- read_directory_recursive() to be moved to treat_path() (or another
- function it calls, such as treat_directory()), but I did not feel
- strongly about this and just left the logic where it was while
- adding to it. Do others have strong opinions on this?
+ read_directory_recursive() to ultimately be moved to treat_path() (or
+ another function it calls, such as treat_directory()), but I have
+ left that for someone else to tackle in the future.
7: 5e1686a30e = 7: 7821898ba7 dir: add commentary explaining match_pathspec_item's return value 8: d92637f961 = 8: 13def5df57 git-clean.txt: do not claim we will delete files with -n/--dry-run 9: 4550a30df8 = 9: e6b274abf7 clean: disambiguate the definition of -d 10: 0985be2793 ! 10: 5f4ef14765 clean: avoid removing untracked files in a nested git repository
@@ Commit message
Also, clean up some grammar errors describing this functionality in the git-clean manpage.
- Finally, there is one somewhat related bug which this patch does not
- fix, coming from the opposite angle. If the user runs
- git clean -ffd
- to force deletion of untracked nested repositories, and within an
- untracked nested repo the user has ignored files (according to the inner
- OR outer repositories' .gitignore), then not only will those ignored
- files be left alone but the .git/ subdirectory of the nested repo will
- be left alone too. I am not completely sure if this should be
- considered a bug (though it seems like it since the lack of the
- untracked file would result in the .git/ subdirectory being deleted),
- but in any event it is very minor compared to accidentally deleting user
- data and I did not dive into it.
+ Finally, there are still a couple bugs with -ffd not cleaning out enough
+ (e.g. missing the nested .git) and with -ffdX possibly cleaning out the
+ wrong files (paying attention to outer .gitignore instead of inner).
+ This patch does not address these cases at all (and does not change the
+ behavior relative to those flags), it only fixes the handling when given
+ a single -f. See
+ https://public-inbox.org/git/[email protected]/ for more
+ discussion of the -ffd[X?] bugs.
11: 2d36e3f7cb = 11: 4e30e62eb1 clean: rewrap overly long line 12: 3c5d1ff16a < -: ---------- clean: fix theoretical path corruption -: ---------- > 12: de2444f7cb clean: fix theoretical path corruption
Signature · 1 line
2.23.0.173.gad11b3a635.dirty
PATCH v3 1/12 t7300: add testcases showing failure to clean specified pathspecs
Someone brought me a testcase where multiple git-clean invocations were required to clean out unwanted files:
mkdir d{1,2}
touch d{1,2}/ut
touch d1/t && git add d1/t
With this setup, the user would need to run
git clean -ffd */ut
twice to delete both ut files.
A little testing showed some interesting variants:
* If only one of those two ut files existed (either one), then only one clean command would be necessary.
* If both directories had tracked files, then only one git clean would be necessary to clean both files.
* If both directories had no tracked files then the clean command above would never clean either of the untracked files despite the pathspec explicitly calling both of them out.
A bisect showed that the failure to clean out the files started with commit cf424f5fd89b ("clean: respect pathspecs with "-d", 2014-03-10). However, that pointed to a separate issue: while the "-d" flag was used by the original user who showed me this problem, that flag should have been irrelevant to this problem. Testing again without the "-d" flag showed that the same buggy behavior exists without using that flag, and has in fact existed since before cf424f5fd89b.
Add testcases showing that multiple untracked files within entirely untracked directories cannot be cleaned when specifying these files to git clean via pathspecs.
--- t/t7300-clean.sh | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+)
t/t7300-clean.sh +32 -0
@@ -691,6 +691,38 @@ test_expect_failure 'git clean -d skips nested repo containing ignored files' ' test_path_is_file nested-repo-with-ignored-file/file ' +test_expect_failure 'git clean handles being told what to clean' ' + mkdir -p d1 d2 && + touch d1/ut d2/ut && + git clean -f */ut && + test_path_is_missing d1/ut && + test_path_is_missing d2/ut +' + +test_expect_failure 'git clean handles being told what to clean, with -d' ' + mkdir -p d1 d2 && + touch d1/ut d2/ut && + git clean -ffd */ut && + test_path_is_missing d1/ut && + test_path_is_missing d2/ut +' + +test_expect_failure 'git clean works if a glob is passed without -d' ' + mkdir -p d1 d2 && + touch d1/ut d2/ut && + git clean -f "*ut" && + test_path_is_missing d1/ut && + test_path_is_missing d2/ut +' + +test_expect_failure 'git clean works if a glob is passed with -d' ' + mkdir -p d1 d2 && + touch d1/ut d2/ut && + git clean -ffd "*ut" && + test_path_is_missing d1/ut && + test_path_is_missing d2/ut +' + test_expect_success MINGW 'handle clean & core.longpaths = false nicely' ' test_config core.longpaths false && a50=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa &&
Signature · 1 line
2.23.0.173.gad11b3a635.dirty
Re: [PATCH v3 01/12] t7300: add testcases showing failure to clean specified pathspecs
t/t7300-clean.sh test_expect_failure 'git clean -d skips nested repo containing ignored files' ' · line 700
Quoted 7 lines · Elijah Newren
t/t7300-clean.sh +7
-0+test_expect_failure 'git clean handles being told what to clean' ' + mkdir -p d1 d2 && + touch d1/ut d2/ut && + git clean -f */ut && + test_path_is_missing d1/ut && + test_path_is_missing d2/ut +'
Looks like d1 and d2 are new directories and the paths we see in the test are the only ones that are involved (i.e. we do not rely on any leftover cruft in d[12]/ from previous tests). If so, perhaps it is easier to follow by starting the tests with "rm -fr d1 d2 &&" or something to assure the readers of the script (not this patch, but the resulting file down the road) about the isolation? The same comment applies to the remainder.
Also, you talked about tracked paths in the proposed log message; do they not participate in reproducing the issue(s)?
Thanks.
Quoted 27 lines
t/t7300-clean.sh +24
-0+test_expect_failure 'git clean handles being told what to clean, with -d' ' + mkdir -p d1 d2 && + touch d1/ut d2/ut && + git clean -ffd */ut && + test_path_is_missing d1/ut && + test_path_is_missing d2/ut +' + +test_expect_failure 'git clean works if a glob is passed without -d' ' + mkdir -p d1 d2 && + touch d1/ut d2/ut && + git clean -f "*ut" && + test_path_is_missing d1/ut && + test_path_is_missing d2/ut +' + +test_expect_failure 'git clean works if a glob is passed with -d' ' + mkdir -p d1 d2 && + touch d1/ut d2/ut && + git clean -ffd "*ut" && + test_path_is_missing d1/ut && + test_path_is_missing d2/ut +' + test_expect_success MINGW 'handle clean & core.longpaths = false nicely' ' test_config core.longpaths false && a50=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa &&
Re: [PATCH v3 01/12] t7300: add testcases showing failure to clean specified pathspecs
Quoted 11 lines · Junio C Hamano
t/t7300-clean.sh test_expect_failure 'git clean -d skips nested repo containing ignored files' ' · line 700
Quoted 7 lines · Elijah Newren
t/t7300-clean.sh +7
-0+test_expect_failure 'git clean handles being told what to clean' ' + mkdir -p d1 d2 && + touch d1/ut d2/ut && + git clean -f */ut && + test_path_is_missing d1/ut && + test_path_is_missing d2/ut +'
Quoted 7 lines
Looks like d1 and d2 are new directories and the paths we see in the test are the only ones that are involved (i.e. we do not rely on any leftover cruft in d[12]/ from previous tests). If so, perhaps it is easier to follow by starting the tests with "rm -fr d1 d2 &&" or something to assure the readers of the script (not this patch, but the resulting file down the road) about the isolation? The same comment applies to the remainder.
Makes sense.
Quoted 2 lines
Also, you talked about tracked paths in the proposed log message; do they not participate in reproducing the issue(s)?
If there is only one directory which has no tracked files, then the user can clean up the files -- but confusingly, they have to issue the same git-clean command multiple times. If multiple directories have no untracked files, git-clean will never clean them out. I probably didn't do a very good job explaining that although I started with the case with one tracked, that I view the case without any as the more general case -- and that solving it solves both problems. I could probably make that clearer in the commit message. (Or maybe just add more testcases even if slightly duplicative, I guess.)
Re: [PATCH v3 01/12] t7300: add testcases showing failure to clean specified pathspecs
Quoted 12 lines · Elijah Newren
Quoted 2 lines
Also, you talked about tracked paths in the proposed log message; do they not participate in reproducing the issue(s)?
If there is only one directory which has no tracked files, then the user can clean up the files -- but confusingly, they have to issue the same git-clean command multiple times. If multiple directories have no untracked files, git-clean will never clean them out. I probably didn't do a very good job explaining that although I started with the case with one tracked, that I view the case without any as the more general case -- and that solving it solves both problems. I could probably make that clearer in the commit message. (Or maybe just add more testcases even if slightly duplicative, I guess.)
My comment/puzzlement indeed was the lack of any tracked file in your tests, even though the log message did talk about one's presence making a difference in the outcome.
Thanks.
PATCH v3 2/12 dir: fix typo in comment
In reply to Elijah Newren--- dir.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
dir.c +1 -1
@@ -139,7 +139,7 @@ static size_t common_prefix_len(const struct pathspec *pathspec)* ":(icase)path" is treated as a pathspec full of * wildcard. In other words, only prefix is considered common * prefix. If the pathspec is abc/foo abc/bar, running in -* subdir xyz, the common prefix is still xyz, not xuz/abc as +* subdir xyz, the common prefix is still xyz, not xyz/abc as * in non-:(icase). */ GUARD_PATHSPEC(pathspec,
Signature · 1 line
2.23.0.173.gad11b3a635.dirty
PATCH v3 3/12 dir: fix off-by-one error in match_pathspec_item
In reply to Elijah NewrenFor a pathspec like 'foo/bar' comparing against a path named "foo/", namelen will be 4, and match[namelen] will be 'b'. The correct location of the directory separator is namelen-1.
The reason the code worked anyway was that the following code immediately checked whether the first matchlen characters matched (which they do) and then bailed and return MATCHED_RECURSIVELY anyway since wildmatch doesn't have the ability to check if "name" can be matched as a directory (or prefix) against the pathspec.
--- dir.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
dir.c +2 -1
@@ -356,8 +356,9 @@ static int match_pathspec_item(const struct index_state *istate,/* Perform checks to see if "name" is a super set of the pathspec */ if (flags &DO_MATCH_SUBMODULE ) {/* name is a literal prefix of the pathspec */ +int offset = name[namelen-1 ] =='/' ?1 :0 ;if ((namelen < matchlen) && - (match[namelen] =='/' ) && + (match[namelen-offset] =='/' ) && !ps_strncmp (item, match, name, namelen))return MATCHED_RECURSIVELY ;
Signature · 1 line
2.23.0.173.gad11b3a635.dirty
Re: [PATCH v3 03/12] dir: fix off-by-one error in match_pathspec_item
Quoted 3 lines · Elijah Newren
For a pathspec like 'foo/bar' comparing against a path named "foo/", namelen will be 4, and match[namelen] will be 'b'. The correct location of the directory separator is namelen-1.
And the reason why name[namelen-1] may not be slash, in which case your new code makes offset 0, is because we need to handle what case? When path is "foo" (not "foo/")? Just makes me wonder why this callee allows the caller(s) to be inconsistent, sometimes including the trailing slash in <name, nemelen> tuple, sometimes not.
Quoted 5 lines
The reason the code worked anyway was that the following code immediately checked whether the first matchlen characters matched (which they do) and then bailed and return
MATCHED_RECURSIVELYanyway since wildmatch doesn't have the ability to check if "name" can be matched as a directory (or prefix) against the pathspec.
Nicely spotted and explained.
Quoted 20 lines
Signed-off-by Elijah Newren <[email protected]>--- dir.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)dir.c +2
-1@@ -356,8 +356,9 @@ static int match_pathspec_item(const struct index_state *istate,/* Perform checks to see if "name" is a super set of the pathspec */ if (flags &DO_MATCH_SUBMODULE ) {/* name is a literal prefix of the pathspec */ +int offset = name[namelen-1 ] =='/' ?1 :0 ;if ((namelen < matchlen) && - (match[namelen] =='/' ) && + (match[namelen-offset] =='/' ) && !ps_strncmp (item, match, name, namelen))return MATCHED_RECURSIVELY ;
PATCH v3 4/12 dir: also check directories for matching pathspecs
In reply to Elijah NewrenEven if a directory doesn't match a pathspec, it is possible, depending on the precise pathspecs, that some file underneath it might. So we special case and recurse into the directory for such situations. However, we previously always added any untracked directory that we recursed into to the list of untracked paths, regardless of whether the directory itself matched the pathspec.
For the case of git-clean and a set of pathspecs of "dir/file" and "more", this caused a problem because we'd end up with dir entries for both of
"dir"
"dir/file"
Then correct_untracked_entries() would try to helpfully prune duplicates for us by removing "dir/file" since it's under "dir", leaving us with
"dir"
Since the original pathspec only had "dir/file", the only entry left doesn't match and leaves nothing to be removed. (Note that if only one pathspec was specified, e.g. only "dir/file", then the common_prefix_len optimizations in fill_directory would cause us to bypass this problem, making it appear in simple tests that we could correctly remove manually specified pathspecs.)
Fix this by actually checking whether the directory we are about to add to the list of dir entries actually matches the pathspec; only do this matching check after we have already returned from recursing into the directory.
--- dir.c | 5 +++++ t/t7300-clean.sh | 4 ++-- 2 files changed, 7 insertions(+), 2 deletions(-)
dir.c +5 -0
@@ -1951,6 +1951,11 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir, check_only, stop_at_first_file, pathspec);if (subdir_state > dir_state) dir_state = subdir_state; + +if (!match_pathspec (istate, pathspec, path.buf, path.len, +0 /* prefix */ ,NULL , +0 /* do NOT special case dirs */ )) + state = path_none; }if (check_only) {
t/t7300-clean.sh +2 -2
@@ -699,7 +699,7 @@ test_expect_failure 'git clean handles being told what to clean' ' test_path_is_missing d2/ut ' -test_expect_failure 'git clean handles being told what to clean, with -d' ' +test_expect_success 'git clean handles being told what to clean, with -d' ' mkdir -p d1 d2 && touch d1/ut d2/ut && git clean -ffd */ut &&
@@ -715,7 +715,7 @@ test_expect_failure 'git clean works if a glob is passed without -d' ' test_path_is_missing d2/ut ' -test_expect_failure 'git clean works if a glob is passed with -d' ' +test_expect_success 'git clean works if a glob is passed with -d' ' mkdir -p d1 d2 && touch d1/ut d2/ut && git clean -ffd "*ut" &&
Signature · 1 line
2.23.0.173.gad11b3a635.dirty
PATCH v3 5/12 dir: make the DO_MATCH_SUBMODULE code reusable for a non-submodule case
In reply to Elijah NewrenThe specific checks done in match_pathspec_item for the DO_MATCH_SUBMODULE case are useful for other cases which have nothing to do with submodules. Rename this constant; a subsequent commit will make use of this change.
--- dir.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
dir.c +3 -3
@@ -273,7 +273,7 @@ static int do_read_blob(const struct object_id *oid, struct oid_stat *oid_stat,#define DO_MATCH_EXCLUDE (1<<0)#define DO_MATCH_DIRECTORY (1<<1) -#define DO_MATCH_SUBMODULE (1<<2) +#define DO_MATCH_LEADING_PATHSPEC (1<<2)/* * Does 'match' match the given name?
@@ -354,7 +354,7 @@ static int match_pathspec_item(const struct index_state *istate,return MATCHED_FNMATCH ;/* Perform checks to see if "name" is a super set of the pathspec */ -if (flags &DO_MATCH_SUBMODULE ) { +if (flags &DO_MATCH_LEADING_PATHSPEC ) {/* name is a literal prefix of the pathspec */ int offset = name[namelen-1 ] =='/' ?1 :0 ; if ((namelen < matchlen) &&
@@ -498,7 +498,7 @@ int submodule_path_match(const struct index_state *istate,strlen (submodule_name),0 , seen,DO_MATCH_DIRECTORY | - DO_MATCH_SUBMODULE); + DO_MATCH_LEADING_PATHSPEC);return matched; }
Signature · 1 line
2.23.0.173.gad11b3a635.dirty
PATCH v3 6/12 dir: if our pathspec might match files under a dir, recurse into it
In reply to Elijah NewrenFor git clean, if a directory is entirely untracked and the user did not specify -d (corresponding to DIR_SHOW_IGNORED_TOO), then we usually do not want to remove that directory and thus do not recurse into it. However, if the user manually specified specific (or even globbed) paths somewhere under that directory to remove, then we need to recurse into the directory to make sure we remove the relevant paths under that directory as the user requested.
Note that this does not mean that the recursed-into directory will be added to dir->entries for later removal; as of a few commits earlier in this series, there is another more strict match check that is run after returning from a recursed-into directory before deciding to add it to the list of entries. Therefore, this will only result in files underneath the given directory which match one of the pathspecs being added to the entries list.
Two notes of potential interest to future readers:
* If we wanted to only recurse into a directory when it is specifically matched rather than matched-via-glob (e.g. '*.c'), then we could do so via making the final non-zero return in match_pathspec_item be MATCHED_RECURSIVELY instead of MATCHED_RECURSIVELY_LEADING_PATHSPEC. (Note that the relative order of MATCHED_RECURSIVELY_LEADING_PATHSPEC and MATCHED_RECURSIVELY are important for such a change.) I was leaving open that possibility while writing an RFC asking for the behavior we want, but even though we don't want it, that knowledge might help you understand the code flow better.
* There is a growing amount of logic in read_directory_recursive() for deciding whether to recurse into a subdirectory. However, there is a comment immediately preceding this logic that says to recurse if
instructed by treat_path(). It may be better for the logic in
read_directory_recursive() to ultimately be moved to treat_path() (or another function it calls, such as treat_directory()), but I have left that for someone else to tackle in the future.
--- dir.c | 10 ++++++---- dir.h | 5 +++-- t/t7300-clean.sh | 4 ++-- 3 files changed, 11 insertions(+), 8 deletions(-)
dir.c +6 -4
@@ -360,7 +360,7 @@ static int match_pathspec_item(const struct index_state *istate,if ((namelen < matchlen) && (match[namelen-offset] =='/' ) && !ps_strncmp (item, match, name, namelen)) -return MATCHED_RECURSIVELY ; +return MATCHED_RECURSIVELY_LEADING_PATHSPEC ;/* name" doesn't match up to the first wild character */ if (item->nowildcard_len < item->len &&
@@ -377,7 +377,7 @@ static int match_pathspec_item(const struct index_state *istate,* The submodules themselves will be able to perform more * accurate matching to determine if the pathspec matches. */ -return MATCHED_RECURSIVELY ; +return MATCHED_RECURSIVELY_LEADING_PATHSPEC ; }return 0 ;
@@ -1939,8 +1939,10 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,/* recurse into subdir if instructed by treat_path */ if ((state == path_recurse) || ((state == path_untracked) && - (dir->flags &DIR_SHOW_IGNORED_TOO ) && - (get_dtype (cdir.de, istate, path.buf, path.len) ==DT_DIR ))) { + (get_dtype (cdir.de, istate, path.buf, path.len) ==DT_DIR ) && + ((dir->flags &DIR_SHOW_IGNORED_TOO ) || +do_match_pathspec (istate, pathspec, path.buf, path.len, + baselen,NULL ,DO_MATCH_LEADING_PATHSPEC ) ==MATCHED_RECURSIVELY_LEADING_PATHSPEC ))) {struct untracked_cache_dir *ud; ud = lookup_untracked(dir->untracked, untracked, path.buf + baselen,
dir.h +3 -2
@@ -211,8 +211,9 @@ int count_slashes(const char *s);* when populating the seen[] array. */ #define MATCHED_RECURSIVELY 1 -#define MATCHED_FNMATCH 2 -#define MATCHED_EXACTLY 3 +#define MATCHED_RECURSIVELY_LEADING_PATHSPEC 2 +#define MATCHED_FNMATCH 3 +#define MATCHED_EXACTLY 4int simple_length (const char *match);int no_wildcard (const char *string);char *common_prefix (const struct pathspec *pathspec);
t/t7300-clean.sh +2 -2
@@ -691,7 +691,7 @@ test_expect_failure 'git clean -d skips nested repo containing ignored files' ' test_path_is_file nested-repo-with-ignored-file/file ' -test_expect_failure 'git clean handles being told what to clean' ' +test_expect_success 'git clean handles being told what to clean' ' mkdir -p d1 d2 && touch d1/ut d2/ut && git clean -f */ut &&
@@ -707,7 +707,7 @@ test_expect_success 'git clean handles being told what to clean, with -d' ' test_path_is_missing d2/ut ' -test_expect_failure 'git clean works if a glob is passed without -d' ' +test_expect_success 'git clean works if a glob is passed without -d' ' mkdir -p d1 d2 && touch d1/ut d2/ut && git clean -f "*ut" &&
Signature · 1 line
2.23.0.173.gad11b3a635.dirty
Re: [PATCH v3 06/12] dir: if our pathspec might match files under a dir, recurse into it
Quoted 3 lines · Elijah Newren
For git clean, if a directory is entirely untracked and the user did not specify -d (corresponding to
DIR_SHOW_IGNORED_TOO), then we usually do not want to remove that directory and thus do not recurse into it.
Makes sense. To clean named paths in such a directory, we'd need an option to recurse into it to find them, yet make sure the directory itself does not get removed.
Quoted 4 lines
However, if the user manually specified specific (or even globbed) paths somewhere under that directory to remove, then we need to recurse into the directory to make sure we remove the relevant paths under that directory as the user requested.
Surely.
dir.c static enum path_treatment read_directory_recursive(struct dir_struct *dir, · line 1948
Quoted 13 lines
dir.c +4
-2@@ -1939,8 +1939,10 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,/* recurse into subdir if instructed by treat_path */ if ((state == path_recurse) || ((state == path_untracked) && - (dir->flags &DIR_SHOW_IGNORED_TOO ) && - (get_dtype (cdir.de, istate, path.buf, path.len) ==DT_DIR ))) { + (get_dtype (cdir.de, istate, path.buf, path.len) ==DT_DIR ) && + ((dir->flags &DIR_SHOW_IGNORED_TOO ) || +do_match_pathspec (istate, pathspec, path.buf, path.len, + baselen,NULL ,DO_MATCH_LEADING_PATHSPEC ) ==MATCHED_RECURSIVELY_LEADING_PATHSPEC ))) {struct untracked_cache_dir *ud; ud = lookup_untracked(dir->untracked, untracked, path.buf + baselen,
OK.
Quoted 14 lines
t/t7300-clean.sh +1
-1@@ -691,7 +691,7 @@ test_expect_failure 'git clean -d skips nested repo containing ignored files' ' test_path_is_file nested-repo-with-ignored-file/file ' -test_expect_failure 'git clean handles being told what to clean' ' +test_expect_success 'git clean handles being told what to clean' ' mkdir -p d1 d2 && touch d1/ut d2/ut && git clean -f */ut &&
t/t7300-clean.sh test_expect_success 'git clean handles being told what to clean, with -d' ' · line 713
Quoted 8 lines
t/t7300-clean.sh +1
-1@@ -707,7 +707,7 @@ test_expect_success 'git clean handles being told what to clean, with -d' ' test_path_is_missing d2/ut ' -test_expect_failure 'git clean works if a glob is passed without -d' ' +test_expect_success 'git clean works if a glob is passed without -d' ' mkdir -p d1 d2 && touch d1/ut d2/ut && git clean -f "*ut" &&
Nice.
Thanks.
PATCH v3 7/12 dir: add commentary explaining match_pathspec_item's return value
In reply to Elijah NewrenThe way match_pathspec_item() handles names and pathspecs with trailing slash characters, in conjunction with special options like DO_MATCH_DIRECTORY and DO_MATCH_LEADING_PATHSPEC were non-obvious, and broken until this patch series. Add a table in a comment explaining the intent of how these work.
--- dir.c | 27 +++++++++++++++++++-------- 1 file changed, 19 insertions(+), 8 deletions(-)
dir.c +19 -8
@@ -276,16 +276,27 @@ static int do_read_blob(const struct object_id *oid, struct oid_stat *oid_stat,#define DO_MATCH_LEADING_PATHSPEC (1<<2)/* -* Does 'match' match the given name? -* A match is found if +* Does the given pathspec match the given name? A match is found if * -* (1) the 'match' string is leading directory of 'name', or -* (2) the 'match' string is a wildcard and matches 'name', or -* (3) the 'match' string is exactly the same as 'name'. +* (1) the pathspec string is leading directory of 'name' ("RECURSIVELY"), or +* (2) the pathspec string has a leading part matching 'name' ("LEADING"), or +* (3) the pathspec string is a wildcard and matches 'name' ("WILDCARD"), or +* (4) the pathspec string is exactly the same as 'name' ("EXACT"). * -* and the return value tells which case it was. +* Return value tells which case it was (1-4), or 0 when there is no match. * -* It returns 0 when there is no match. +* It may be instructive to look at a small table of concrete examples +* to understand the differences between 1, 2, and 4: +* +* Pathspecs +* | a/b | a/b/ | a/b/c +* ------+-----------+-----------+------------ +* a/b | EXACT | EXACT[1] | LEADING[2] +* Names a/b/ | RECURSIVE | EXACT | LEADING[2] +* a/b/c | RECURSIVE | RECURSIVE | EXACT +* +* [1] Only if DO_MATCH_DIRECTORY is passed; otherwise, this is NOT a match. +* [2] Only if DO_MATCH_LEADING_PATHSPEC is passed; otherwise, not a match. */ static int match_pathspec_item(const struct index_state *istate, const struct pathspec_item *item, int prefix,
@@ -353,7 +364,7 @@ static int match_pathspec_item(const struct index_state *istate, item->nowildcard_len - prefix))return MATCHED_FNMATCH ; -/* Perform checks to see if "name" is a super set of the pathspec */ +/* Perform checks to see if "name" is a leading string of the pathspec */ if (flags &DO_MATCH_LEADING_PATHSPEC ) {/* name is a literal prefix of the pathspec */ int offset = name[namelen-1 ] =='/' ?1 :0 ;
Signature · 1 line
2.23.0.173.gad11b3a635.dirty
Re: [PATCH v3 07/12] dir: add commentary explaining match_pathspec_item's return value
Quoted 5 lines · Elijah Newren
The way
match_pathspec_item()handles names and pathspecs with trailing slash characters, in conjunction with special options likeDO_MATCH_DIRECTORYandDO_MATCH_LEADING_PATHSPECwere non-obvious, and broken until this patch series. Add a table in a comment explaining the intent of how these work.
Thanks.
Quoted 54 lines
Signed-off-by Elijah Newren <[email protected]>--- dir.c | 27 +++++++++++++++++++-------- 1 file changed, 19 insertions(+), 8 deletions(-)dir.c +19
-8@@ -276,16 +276,27 @@ static int do_read_blob(const struct object_id *oid, struct oid_stat *oid_stat,#define DO_MATCH_LEADING_PATHSPEC (1<<2)/* -* Does 'match' match the given name? -* A match is found if +* Does the given pathspec match the given name? A match is found if * -* (1) the 'match' string is leading directory of 'name', or -* (2) the 'match' string is a wildcard and matches 'name', or -* (3) the 'match' string is exactly the same as 'name'. +* (1) the pathspec string is leading directory of 'name' ("RECURSIVELY"), or +* (2) the pathspec string has a leading part matching 'name' ("LEADING"), or +* (3) the pathspec string is a wildcard and matches 'name' ("WILDCARD"), or +* (4) the pathspec string is exactly the same as 'name' ("EXACT"). * -* and the return value tells which case it was. +* Return value tells which case it was (1-4), or 0 when there is no match. * -* It returns 0 when there is no match. +* It may be instructive to look at a small table of concrete examples +* to understand the differences between 1, 2, and 4: +* +* Pathspecs +* | a/b | a/b/ | a/b/c +* ------+-----------+-----------+------------ +* a/b | EXACT | EXACT[1] | LEADING[2] +* Names a/b/ | RECURSIVE | EXACT | LEADING[2] +* a/b/c | RECURSIVE | RECURSIVE | EXACT +* +* [1] Only if DO_MATCH_DIRECTORY is passed; otherwise, this is NOT a match. +* [2] Only if DO_MATCH_LEADING_PATHSPEC is passed; otherwise, not a match. */ static int match_pathspec_item(const struct index_state *istate, const struct pathspec_item *item, int prefix,@@ -353,7 +364,7 @@ static int match_pathspec_item(const struct index_state *istate, item->nowildcard_len - prefix))return MATCHED_FNMATCH ; -/* Perform checks to see if "name" is a super set of the pathspec */ +/* Perform checks to see if "name" is a leading string of the pathspec */ if (flags &DO_MATCH_LEADING_PATHSPEC ) {/* name is a literal prefix of the pathspec */ int offset = name[namelen-1 ] =='/' ?1 :0 ;
PATCH v3 8/12 git-clean.txt: do not claim we will delete files with -n/--dry-run
In reply to Elijah NewrenIt appears that the wrong option got included in the list of what will cause git-clean to actually take action. Correct the list.
--- Documentation/git-clean.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Documentation/git-clean.txt +1 -1
@@ -35,7 +35,7 @@ OPTIONS --force:: If the Git configuration variable clean.requireForce is not set to false, 'git clean' will refuse to delete files or directories - unless given -f, -n or -i. Git will refuse to delete directories + unless given -f or -i. Git will refuse to delete directories with .git sub directory or file unless a second -f is given.
Signature · 1 line
2.23.0.173.gad11b3a635.dirty
PATCH v3 9/12 clean: disambiguate the definition of -d
In reply to Elijah NewrenThe -d flag pre-dated git-clean's ability to have paths specified. As such, the default for git-clean was to only remove untracked files in the current directory, and -d existed to allow it to recurse into subdirectories.
The interaction of paths and the -d option appears to not have been carefully considered, as evidenced by numerous bugs and a dearth of tests covering such pairings in the testsuite. The definition turns out to be important, so let's look at some of the various ways one could interpret the -d option:
A) Without -d, only look in subdirectories which contain tracked files under them; with -d, also look in subdirectories which are untracked for files to clean.
B) Without specified paths from the user for us to delete, we need to have some kind of default, so...without -d, only look in subdirectories which contain tracked files under them; with -d, also look in subdirectories which are untracked for files to clean.
The important distinction here is that choice B says that the presence or absence of '-d' is irrelevant if paths are specified. The logic behind option B is that if a user explicitly asked us to clean a specified pathspec, then we should clean anything that matches that pathspec. Some examples may clarify. Should
git clean -f untracked_dir/file
remove untracked_dir/file or not? It seems crazy not to, but a strict reading of option A says it shouldn't be removed. How about
git clean -f untracked_dir/file1 tracked_dir/file2
or
git clean -f untracked_dir_1/file1 untracked_dir_2/file2
? Should it remove either or both of these files? Should it require multiple runs to remove both the files listed? (If this sounds like a crazy question to even ask, see the commit message of "t7300: Add some testcases showing failure to clean specified pathspecs" added earlier in this patch series.) What if -ffd were used instead of -f -- should that allow these to be removed? Should it take multiple invocations with -ffd? What if a glob (such as '*tracked*') were used instead of spelling out the directory names? What if the filenames involved globs, such as
git clean -f '*.o'
or
git clean -f '*/*.o'
?
The current documentation actually suggests a definition that is slightly different than choice A, and the implementation prior to this series provided something radically different than either choices A or B. (The implementation, though, was clearly just buggy). There may be other choices as well. However, for almost any given choice of definition for -d that I can think of, some of the examples above will appear buggy to the user. The only case that doesn't have negative surprises is choice B: treat a user-specified path as a request to clean all untracked files which match that path specification, including recursing into any untracked directories.
Change the documentation and basic implementation to use this definition.
There were two regression tests that indirectly depended on the current implementation, but neither was about subdirectory handling. These two tests were introduced in commit 5b7570cfb41c ("git-clean: add tests for relative path", 2008-03-07) which was solely created to add coverage for the changes in commit fb328947c8e ("git-clean: correct printing relative path", 2008-03-07). Both tests specified a directory that happened to have an untracked subdirectory, but both were only checking that the resulting printout of a file that was removed was shown with a relative path. Update these tests appropriately.
--- Documentation/git-clean.txt | 10 ++++++---- builtin/clean.c | 8 ++++++++ t/t7300-clean.sh | 2 ++ 3 files changed, 16 insertions(+), 4 deletions(-)
Documentation/git-clean.txt +6 -4
@@ -26,10 +26,12 @@ are affected. OPTIONS ------- -d:: - Remove untracked directories in addition to untracked files. - If an untracked directory is managed by a different Git - repository, it is not removed by default. Use -f option twice - if you really want to remove such a directory. + Normally, when no <path> is specified, git clean will not + recurse into untracked directories to avoid removing too much. + Specify -d to have it recurse into such directories as well. + If any paths are specified, -d is irrelevant; all untracked + files matching the specified paths (with exceptions for nested + git directories mentioned under `--force`) will be removed. -f:: --force::
builtin/clean.c +8 -0
@@ -949,6 +949,14 @@ int cmd_clean(int argc, const char **argv, const char *prefix) dir.flags |=DIR_SHOW_OTHER_DIRECTORIES ; +if (argc) { +/* +* Remaining args implies pathspecs specified, and we should +* recurse within those. +*/ + remove_directories =1 ; + } +if (remove_directories) dir.flags |=DIR_SHOW_IGNORED_TOO |DIR_KEEP_UNTRACKED_CONTENTS ;
t/t7300-clean.sh +2 -0
@@ -117,6 +117,7 @@ test_expect_success C_LOCALE_OUTPUT 'git clean with relative prefix' ' would_clean=$( cd docs && git clean -n ../src | + grep part3 | sed -n -e "s|^Would remove ||p" ) && verbose test "$would_clean" = ../src/part3.c
@@ -129,6 +130,7 @@ test_expect_success C_LOCALE_OUTPUT 'git clean with absolute path' ' would_clean=$( cd docs && git clean -n "$(pwd)/../src" | + grep part3 | sed -n -e "s|^Would remove ||p" ) && verbose test "$would_clean" = ../src/part3.c
Signature · 1 line
2.23.0.173.gad11b3a635.dirty
PATCH v3 10/12 clean: avoid removing untracked files in a nested git repository
In reply to Elijah NewrenUsers expect files in a nested git repository to be left alone unless sufficiently forced (with two -f's). Unfortunately, in certain circumstances, git would delete both tracked (and possibly dirty) files and untracked files within a nested repository. To explain how this happens, let's contrast a couple cases. First, take the following example setup (which assumes we are already within a git repo):
git init nested
cd nested
Quoted 1 line
tracked
git add tracked
git commit -m init
Quoted 1 line
untracked
cd ..
In this setup, everything works as expected; running 'git clean -fd' will result in fill_directory() returning the following paths:
nested/
nested/tracked
nested/untracked
and then correct_untracked_entries() would notice this can be compressed to
nested/
and then since "nested/" is a directory, we would call remove_dirs("nested/", ...), which would
check is_nonbare_repository_dir() and then decide to skip it.
However, if someone also creates an ignored file:
Quoted 1 line
nested/ignored
then running 'git clean -fd' would result in fill_directory() returning the same paths:
nested/
nested/tracked
nested/untracked
but correct_untracked_entries() will notice that we had ignored entries under nested/ and thus simplify this list to
nested/tracked
nested/untracked
Since these are not directories, we do not call remove_dirs() which was the only place that had the is_nonbare_repository_dir() safety check -- resulting in us deleting both the untracked file and the tracked (and possibly dirty) file.
One possible fix for this issue would be walking the parent directories of each path and checking if they represent nonbare repositories, but that would be wasteful. Even if we added caching of some sort, it's still a waste because we should have been able to check that "nested/" represented a nonbare repository before even descending into it in the first place. Add a DIR_SKIP_NESTED_GIT flag to dir_struct.flags and use it to prevent fill_directory() and friends from descending into nested git repos.
With this change, we also modify two regression tests added in commit 91479b9c72f1 ("t7300: add tests to document behavior of clean and nested git", 2015-06-15). That commit, nor its series, nor the six previous iterations of that series on the mailing list discussed why those tests coded the expectation they did. In fact, it appears their purpose was simply to test _existing_ behavior to make sure that the performance changes didn't change the behavior. However, these two tests directly contradicted the manpage's claims that two -f's were required to delete files/directories under a nested git repository. While one could argue that the user gave an explicit path which matched files/directories that were within a nested repository, there's a slippery slope that becomes very difficult for users to understand once you go down that route (e.g. what if they specified "git clean -f -d '*.c'"?) It would also be hard to explain what the exact behavior was; avoid such problems by making it really simple.
Also, clean up some grammar errors describing this functionality in the git-clean manpage.
Finally, there are still a couple bugs with -ffd not cleaning out enough (e.g. missing the nested .git) and with -ffdX possibly cleaning out the wrong files (paying attention to outer .gitignore instead of inner). This patch does not address these cases at all (and does not change the behavior relative to those flags), it only fixes the handling when given a single -f. See
https://public-inbox.org/git/[email protected]/ for more discussion of the -ffd[X?] bugs.
--- Documentation/git-clean.txt | 6 +++--- builtin/clean.c | 2 ++ dir.c | 10 ++++++++++ dir.h | 3 ++- t/t7300-clean.sh | 10 +++++----- 5 files changed, 22 insertions(+), 9 deletions(-)
Documentation/git-clean.txt +3 -3
@@ -37,9 +37,9 @@ OPTIONS --force:: If the Git configuration variable clean.requireForce is not set to false, 'git clean' will refuse to delete files or directories - unless given -f or -i. Git will refuse to delete directories - with .git sub directory or file unless a second -f - is given. + unless given -f or -i. Git will refuse to modify untracked + nested git repositories (directories with a .git subdirectory) + unless a second -f is given. -i:: --interactive::
builtin/clean.c +2 -0
@@ -946,6 +946,8 @@ int cmd_clean(int argc, const char **argv, const char *prefix)if (force >1 ) rm_flags =0 ; +else + dir.flags |=DIR_SKIP_NESTED_GIT ; dir.flags |=DIR_SHOW_OTHER_DIRECTORIES ;
dir.c +10 -0
@@ -1451,6 +1451,16 @@ static enum path_treatment treat_directory(struct dir_struct *dir,return path_none;case index_nonexistent: +if (dir->flags &DIR_SKIP_NESTED_GIT ) { +int nested_repo; +struct strbuf sb =STRBUF_INIT ; +strbuf_addstr (&sb, dirname); + nested_repo =is_nonbare_repository_dir (&sb); +strbuf_release (&sb); +if (nested_repo) +return path_none; + } +if (dir->flags &DIR_SHOW_OTHER_DIRECTORIES )break ; if (exclude &&
dir.h +2 -1
@@ -156,7 +156,8 @@ struct dir_struct {DIR_SHOW_IGNORED_TOO =1 <<5 ,DIR_COLLECT_KILLED_ONLY =1 <<6 ,DIR_KEEP_UNTRACKED_CONTENTS =1 <<7 , - DIR_SHOW_IGNORED_TOO_MODE_MATCHING = 1<<8 +DIR_SHOW_IGNORED_TOO_MODE_MATCHING =1 <<8 , + DIR_SKIP_NESTED_GIT = 1<<9 } flags;struct dir_entry **entries;struct dir_entry **ignored;
t/t7300-clean.sh +5 -5
@@ -549,7 +549,7 @@ test_expect_failure 'nested (non-empty) bare repositories should be cleaned even test_path_is_missing strange_bare ' -test_expect_success 'giving path in nested git work tree will remove it' ' +test_expect_success 'giving path in nested git work tree will NOT remove it' ' rm -fr repo && mkdir repo && (
@@ -561,7 +561,7 @@ test_expect_success 'giving path in nested git work tree will remove it' ' git clean -f -d repo/bar/baz && test_path_is_file repo/.git/HEAD && test_path_is_dir repo/bar/ && - test_path_is_missing repo/bar/baz + test_path_is_file repo/bar/baz/hello.world ' test_expect_success 'giving path to nested .git will not remove it' '
@@ -579,7 +579,7 @@ test_expect_success 'giving path to nested .git will not remove it' ' test_path_is_dir untracked/ ' -test_expect_success 'giving path to nested .git/ will remove contents' ' +test_expect_success 'giving path to nested .git/ will NOT remove contents' ' rm -fr repo untracked && mkdir repo untracked && (
@@ -589,7 +589,7 @@ test_expect_success 'giving path to nested .git/ will remove contents' ' ) && git clean -f -d repo/.git/ && test_path_is_dir repo/.git && - test_dir_is_empty repo/.git && + test_path_is_file repo/.git/HEAD && test_path_is_dir untracked/ '
@@ -671,7 +671,7 @@ test_expect_success 'git clean -d skips untracked dirs containing ignored files' test_path_is_missing foo/b/bb ' -test_expect_failure 'git clean -d skips nested repo containing ignored files' ' +test_expect_success 'git clean -d skips nested repo containing ignored files' ' test_when_finished "rm -rf nested-repo-with-ignored-file" && git init nested-repo-with-ignored-file &&
Signature · 1 line
2.23.0.173.gad11b3a635.dirty
PATCH v3 11/12 clean: rewrap overly long line
In reply to Elijah Newren--- builtin/clean.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
builtin/clean.c +2 -1
@@ -158,7 +158,8 @@ static int remove_dirs(struct strbuf *path, const char *prefix, int force_flag, *dir_gone =1 ; -if ((force_flag &REMOVE_DIR_KEEP_NESTED_GIT ) &&is_nonbare_repository_dir (path)) { +if ((force_flag &REMOVE_DIR_KEEP_NESTED_GIT ) && +is_nonbare_repository_dir (path)) {if (!quiet) {quote_path_relative (path->buf, prefix, "ed); printf(dry_run ? _(msg_would_skip_git_dir) : _(msg_skip_git_dir),
Signature · 1 line
2.23.0.173.gad11b3a635.dirty
PATCH v3 12/12 clean: fix theoretical path corruption
In reply to Elijah Newrencmd_clean() had the following code structure:
struct strbuf abs_path = STRBUF_INIT;
for_each_string_list_item(item, &del_list) {
strbuf_addstr(&abs_path, prefix);
strbuf_addstr(&abs_path, item->string);
PROCESS(&abs_path);
strbuf_reset(&abs_path);
}where I've elided a bunch of unnecessary details and PROCESS(&abs_path) represents a big chunk of code rather than an actual function call. One piece of PROCESS was:
if (lstat(abs_path.buf, &st))
continue;which would cause the strbuf_reset() to be missed -- meaning that the next path to be handled would have two paths concatenated. This path used to use die_errno() instead of continue prior to commit 396049e5fb62 ("git-clean: refactor git-clean into two phases", 2013-06-25), but my understanding of how correct_untracked_entries() works is that it will prevent both dir/ and dir/file from being in the list to clean so this should be dead code and the die_errno() should be safe. But I hesitate to remove it since I am not certain.
However, we can fix both this bug and possible similar future bugs by simply moving the strbuf_reset(&abs_path) to the beginning of the loop. It'll result in N calls to strbuf_reset() instead of N-1, but that's a small price to pay to avoid sneaky bugs like this.
--- builtin/clean.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
builtin/clean.c +1 -1
@@ -1018,6 +1018,7 @@ int cmd_clean(int argc, const char **argv, const char *prefix)for_each_string_list_item (item, &del_list) {struct stat st; +strbuf_reset (&abs_path);if (prefix)strbuf_addstr (&abs_path, prefix);
@@ -1051,7 +1052,6 @@ int cmd_clean(int argc, const char **argv, const char *prefix)printf (dry_run ?_ (msg_would_remove) :_ (msg_remove), qname); } } -strbuf_reset (&abs_path); }strbuf_release (&abs_path);
Signature · 1 line
2.23.0.173.gad11b3a635.dirty
v4 cover Fix some git clean issues
In reply to Elijah NewrenThis patch series fixes a few issues with git-clean:
* Failure to clean when multiple pathspecs are specified, reported both
in April 2018[1] and again in May 2019[2].
* Failure to preserve both tracked and untracked files within a nested Git repository reported a few weeks ago by SZEDER[3].
It builds on sg/clean-nested-repo-with-ignored.
[1] https://public-inbox.org/git/[email protected]/
[2] https://public-inbox.org/git/[email protected]/
[3] https://public-inbox.org/git/[email protected]/
Changes since v3:
* Clarified a couple commit messages highlighted by Junio.
Elijah Newren (12):
t7300: add testcases showing failure to clean specified pathspecs
dir: fix typo in comment
dir: fix off-by-one error in match_pathspec_item
dir: also check directories for matching pathspecs
dir: make the DO_MATCH_SUBMODULE code reusable for a non-submodule
case
dir: if our pathspec might match files under a dir, recurse into it
dir: add commentary explaining match_pathspec_item's return value git-clean.txt: do not claim we will delete files with -n/--dry-run
clean: disambiguate the definition of -d
clean: avoid removing untracked files in a nested git repository
clean: rewrap overly long line
clean: fix theoretical path corruption
Documentation/git-clean.txt | 16 +++++----- builtin/clean.c | 15 +++++++-- dir.c | 63 +++++++++++++++++++++++++++---------- dir.h | 8 +++-- t/t7300-clean.sh | 44 +++++++++++++++++++++++--- 5 files changed, 112 insertions(+), 34 deletions(-)
Range-diff:
1: fe35ab8cc3 ! 1: a48d4e7faf t7300: add testcases showing failure to clean specified pathspecs
@@ -28,9 +28,15 @@
showed that the same buggy behavior exists without using that flag, and has in fact existed since before cf424f5fd89b.
- Add testcases showing that multiple untracked files within entirely
- untracked directories cannot be cleaned when specifying these files to
- git clean via pathspecs.
+ Although these problems at first are perceived to be different (e.g.
+ never clearing out the requested files vs. taking multiple invocations
+ to get everything cleared out), they are actually just different
+ manifestations of the same problem. The case with multiple directories
+ that have no tracked files is the more general case; solving it will
+ solve all the others. So, I concentrate on it. Add testcases showing
+ that multiple untracked files within entirely untracked directories
+ cannot be cleaned when specifying these files to git clean via
+ pathspecs.
2: 707d287d79 = 2: eb00b46822 dir: fix typo in comment 3: bb316e82b2 ! 3: c0e5b820a9 dir: fix off-by-one error in match_pathspec_item
@@ -6,11 +6,22 @@
namelen will be 4, and match[namelen] will be 'b'. The correct location of the directory separator is namelen-1.
- The reason the code worked anyway was that the following code immediately
- checked whether the first matchlen characters matched (which they do) and
- then bailed and return MATCHED_RECURSIVELY anyway since wildmatch doesn't
- have the ability to check if "name" can be matched as a directory (or
- prefix) against the pathspec.
+ However, other callers of match_pathspec_item() such as builtin/grep.c's
+ submodule_path_match() will compare against a path named "foo" instead of
+ "foo/". It might be better to change all the callers to be consistent,
+ as discussed at
+ https://public-inbox.org/git/[email protected]/
+ and
+ https://public-inbox.org/git/CABPp-BERWUPCPq-9fVW1LNocqkrfsoF4BPj3gJd9+En43vEkTQ@mail.gmail.com/
+ but there are many cases to audit, so for now just make sure we handle
+ both cases with and without a trailing slash.
+
+ The reason the code worked despite this sometimes-off-by-one error was
+ that the subsequent code immediately checked whether the first matchlen
+ characters matched (which they do) and then bailed and return
+ MATCHED_RECURSIVELY anyway since wildmatch doesn't have the ability to
+ check if "name" can be matched as a directory (or prefix) against the
+ pathspec.
4: 56319f934a = 4: 397775ec35 dir: also check directories for matching pathspecs 5: 81593a565c = 5: b836de82c0 dir: make the DO_MATCH_SUBMODULE code reusable for a non-submodule case 6: 9566823a0f = 6: feb317d090 dir: if our pathspec might match files under a dir, recurse into it 7: 7821898ba7 = 7: 0a574d6779 dir: add commentary explaining match_pathspec_item's return value 8: 13def5df57 = 8: 0eaa08537c git-clean.txt: do not claim we will delete files with -n/--dry-run 9: e6b274abf7 = 9: a1438301bb clean: disambiguate the definition of -d 10: 5f4ef14765 = 10: 8dc21923ee clean: avoid removing untracked files in a nested git repository 11: 4e30e62eb1 = 11: 707b6a5509 clean: rewrap overly long line 12: de2444f7cb = 12: 84a90010ed clean: fix theoretical path corruption
Signature · 1 line
2.22.1.17.g6e632477f7.dirty
PATCH v4 1/12 t7300: add testcases showing failure to clean specified pathspecs
Someone brought me a testcase where multiple git-clean invocations were required to clean out unwanted files:
mkdir d{1,2}
touch d{1,2}/ut
touch d1/t && git add d1/t
With this setup, the user would need to run
git clean -ffd */ut
twice to delete both ut files.
A little testing showed some interesting variants:
* If only one of those two ut files existed (either one), then only one clean command would be necessary.
* If both directories had tracked files, then only one git clean would be necessary to clean both files.
* If both directories had no tracked files then the clean command above would never clean either of the untracked files despite the pathspec explicitly calling both of them out.
A bisect showed that the failure to clean out the files started with commit cf424f5fd89b ("clean: respect pathspecs with "-d", 2014-03-10). However, that pointed to a separate issue: while the "-d" flag was used by the original user who showed me this problem, that flag should have been irrelevant to this problem. Testing again without the "-d" flag showed that the same buggy behavior exists without using that flag, and has in fact existed since before cf424f5fd89b.
Although these problems at first are perceived to be different (e.g. never clearing out the requested files vs. taking multiple invocations to get everything cleared out), they are actually just different manifestations of the same problem. The case with multiple directories that have no tracked files is the more general case; solving it will solve all the others. So, I concentrate on it. Add testcases showing that multiple untracked files within entirely untracked directories cannot be cleaned when specifying these files to git clean via pathspecs.
--- t/t7300-clean.sh | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+)
t/t7300-clean.sh +32 -0
@@ -691,6 +691,38 @@ test_expect_failure 'git clean -d skips nested repo containing ignored files' ' test_path_is_file nested-repo-with-ignored-file/file ' +test_expect_failure 'git clean handles being told what to clean' ' + mkdir -p d1 d2 && + touch d1/ut d2/ut && + git clean -f */ut && + test_path_is_missing d1/ut && + test_path_is_missing d2/ut +' + +test_expect_failure 'git clean handles being told what to clean, with -d' ' + mkdir -p d1 d2 && + touch d1/ut d2/ut && + git clean -ffd */ut && + test_path_is_missing d1/ut && + test_path_is_missing d2/ut +' + +test_expect_failure 'git clean works if a glob is passed without -d' ' + mkdir -p d1 d2 && + touch d1/ut d2/ut && + git clean -f "*ut" && + test_path_is_missing d1/ut && + test_path_is_missing d2/ut +' + +test_expect_failure 'git clean works if a glob is passed with -d' ' + mkdir -p d1 d2 && + touch d1/ut d2/ut && + git clean -ffd "*ut" && + test_path_is_missing d1/ut && + test_path_is_missing d2/ut +' + test_expect_success MINGW 'handle clean & core.longpaths = false nicely' ' test_config core.longpaths false && a50=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa &&
Signature · 1 line
2.22.1.17.g6e632477f7.dirty
PATCH v4 2/12 dir: fix typo in comment
In reply to Elijah Newren--- dir.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
dir.c +1 -1
@@ -139,7 +139,7 @@ static size_t common_prefix_len(const struct pathspec *pathspec)* ":(icase)path" is treated as a pathspec full of * wildcard. In other words, only prefix is considered common * prefix. If the pathspec is abc/foo abc/bar, running in -* subdir xyz, the common prefix is still xyz, not xuz/abc as +* subdir xyz, the common prefix is still xyz, not xyz/abc as * in non-:(icase). */ GUARD_PATHSPEC(pathspec,
Signature · 1 line
2.22.1.17.g6e632477f7.dirty
PATCH v4 3/12 dir: fix off-by-one error in match_pathspec_item
In reply to Elijah NewrenFor a pathspec like 'foo/bar' comparing against a path named "foo/", namelen will be 4, and match[namelen] will be 'b'. The correct location of the directory separator is namelen-1.
However, other callers of match_pathspec_item() such as builtin/grep.c's submodule_path_match() will compare against a path named "foo" instead of "foo/". It might be better to change all the callers to be consistent, as discussed at
https://public-inbox.org/git/[email protected]/
and
https://public-inbox.org/git/CABPp-BERWUPCPq-9fVW1LNocqkrfsoF4BPj3gJd9+En43vEkTQ@mail.gmail.com/
but there are many cases to audit, so for now just make sure we handle both cases with and without a trailing slash.
The reason the code worked despite this sometimes-off-by-one error was that the subsequent code immediately checked whether the first matchlen characters matched (which they do) and then bailed and return MATCHED_RECURSIVELY anyway since wildmatch doesn't have the ability to check if "name" can be matched as a directory (or prefix) against the pathspec.
--- dir.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
dir.c +2 -1
@@ -356,8 +356,9 @@ static int match_pathspec_item(const struct index_state *istate,/* Perform checks to see if "name" is a super set of the pathspec */ if (flags &DO_MATCH_SUBMODULE ) {/* name is a literal prefix of the pathspec */ +int offset = name[namelen-1 ] =='/' ?1 :0 ;if ((namelen < matchlen) && - (match[namelen] =='/' ) && + (match[namelen-offset] =='/' ) && !ps_strncmp (item, match, name, namelen))return MATCHED_RECURSIVELY ;
Signature · 1 line
2.22.1.17.g6e632477f7.dirty
PATCH v4 4/12 dir: also check directories for matching pathspecs
In reply to Elijah NewrenEven if a directory doesn't match a pathspec, it is possible, depending on the precise pathspecs, that some file underneath it might. So we special case and recurse into the directory for such situations. However, we previously always added any untracked directory that we recursed into to the list of untracked paths, regardless of whether the directory itself matched the pathspec.
For the case of git-clean and a set of pathspecs of "dir/file" and "more", this caused a problem because we'd end up with dir entries for both of
"dir"
"dir/file"
Then correct_untracked_entries() would try to helpfully prune duplicates for us by removing "dir/file" since it's under "dir", leaving us with
"dir"
Since the original pathspec only had "dir/file", the only entry left doesn't match and leaves nothing to be removed. (Note that if only one pathspec was specified, e.g. only "dir/file", then the common_prefix_len optimizations in fill_directory would cause us to bypass this problem, making it appear in simple tests that we could correctly remove manually specified pathspecs.)
Fix this by actually checking whether the directory we are about to add to the list of dir entries actually matches the pathspec; only do this matching check after we have already returned from recursing into the directory.
--- dir.c | 5 +++++ t/t7300-clean.sh | 4 ++-- 2 files changed, 7 insertions(+), 2 deletions(-)
dir.c +5 -0
@@ -1951,6 +1951,11 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir, check_only, stop_at_first_file, pathspec);if (subdir_state > dir_state) dir_state = subdir_state; + +if (!match_pathspec (istate, pathspec, path.buf, path.len, +0 /* prefix */ ,NULL , +0 /* do NOT special case dirs */ )) + state = path_none; }if (check_only) {
t/t7300-clean.sh +2 -2
@@ -699,7 +699,7 @@ test_expect_failure 'git clean handles being told what to clean' ' test_path_is_missing d2/ut ' -test_expect_failure 'git clean handles being told what to clean, with -d' ' +test_expect_success 'git clean handles being told what to clean, with -d' ' mkdir -p d1 d2 && touch d1/ut d2/ut && git clean -ffd */ut &&
@@ -715,7 +715,7 @@ test_expect_failure 'git clean works if a glob is passed without -d' ' test_path_is_missing d2/ut ' -test_expect_failure 'git clean works if a glob is passed with -d' ' +test_expect_success 'git clean works if a glob is passed with -d' ' mkdir -p d1 d2 && touch d1/ut d2/ut && git clean -ffd "*ut" &&
Signature · 1 line
2.22.1.17.g6e632477f7.dirty
[BUG] git is segfaulting, was [PATCH v4 04/12] dir: also check directories for matching pathspecs
Hi Elijah,
I ran into a segfault on MacOS. I managed to bisect it down to 404ebceda0 (dir: also check directories for matching pathspecs, 2019-09-17), which should be the patch in the parent thread. The test case below works fine without this patch applied but segfaults once it is applied.
#!/bin/sh
git worktree add testdir
git -C testdir checkout master
git -C testdir fetch https://github.com/git/git.git todo bin-wrappers/git -C testdir checkout FETCH_HEAD # segfault here
Note that the worktree part isn't necessary to reproduce the problem but I didn't want my files to be constantly refreshed, triggering a rebuild each time.
I also managed to get this backtrace from running lldb at the segfault but it is based on the latest "jch" commit, 1cc52d20df (Merge branch 'jt/merge-recursive-symlink-is-not-a-dir-in-way' into jch, 2019-09-20).
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x8) * frame #0: 0x00000001000f63a0 git`do_match_pathspec(istate=0x0000000100299940, ps=0x000000010200aa80, name="Gitweb/static/js/lib/", namelen=21, prefix=0, seen=0x0000000000000000, flags=0) at dir.c:420:2 [opt] frame #1: 0x00000001000f632c git`match_pathspec(istate=0x0000000100299940, ps=0x0000000000000000, name="Gitweb/static/js/lib/", namelen=21, prefix=0, seen=0x0000000000000000, is_dir=0) at dir.c:490:13 [opt] frame #2: 0x00000001000f8315 git`read_directory_recursive(dir=0x00007ffeefbfe278, istate=0x0000000100299940, base=<unavailable>, baselen=17, untracked=<unavailable>, check_only=0, stop_at_first_file=0, pathspec=0x0000000000000000) at dir.c:1990:9 [opt] frame #3: 0x00000001000f82e9 git`read_directory_recursive(dir=0x00007ffeefbfe278, istate=0x0000000100299940, base=<unavailable>, baselen=14, untracked=<unavailable>, check_only=0, stop_at_first_file=0, pathspec=0x0000000000000000) at dir.c:1984:5 [opt] frame #4: 0x00000001000f82e9 git`read_directory_recursive(dir=0x00007ffeefbfe278, istate=0x0000000100299940, base=<unavailable>, baselen=7, untracked=<unavailable>, check_only=0, stop_at_first_file=0, pathspec=0x0000000000000000) at dir.c:1984:5 [opt] frame #5: 0x00000001000f60d1 git`read_directory(dir=0x00007ffeefbfe278, istate=0x0000000100299940, path="Gitweb/", len=7, pathspec=0x0000000000000000) at dir.c:2298:3 [opt] frame #6: 0x00000001001bded1 git`verify_clean_subdirectory(ce=<unavailable>, o=0x00007ffeefbfe8c0) at unpack-trees.c:1846:6 [opt] frame #7: 0x00000001001bdc1d git`check_ok_to_remove(name="Gitweb", len=6, dtype=4, ce=0x0000000103e70de0, st=0x00007ffeefbfe438, error_type=ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN, o=0x00007ffeefbfe8c0) at unpack-trees.c:1901:7 [opt] frame #8: 0x00000001001bdb01 git`verify_absent_1(ce=<unavailable>, error_type=<unavailable>, o=<unavailable>) at unpack-trees.c:1964:10 [opt] frame #9: 0x00000001001bafc0 git`verify_absent(ce=<unavailable>, error_type=<unavailable>, o=<unavailable>) at unpack-trees.c:1052:11 [opt] [artificial] frame #10: 0x00000001001bbc3c git`merged_entry(ce=0x0000000100605fb0, old=0x0000000000000000, o=0x00007ffeefbfe8c0) at unpack-trees.c:2013:7 [opt] frame #11: 0x00000001001bd2b7 git`call_unpack_fn(src=<unavailable>, o=<unavailable>) at unpack-trees.c:522:12 [opt] frame #12: 0x00000001001bca16 git`unpack_nondirectories(n=2, mask=2, dirmask=<unavailable>, src=0x00007ffeefbfe5d0, names=<unavailable>, info=0x00007ffeefbfe718) at unpack-trees.c:1029:12 [opt] frame #13: 0x00000001001bad1a git`unpack_callback(n=2, mask=2, dirmask=0, names=0x0000000102007390, info=0x00007ffeefbfe718) at unpack-trees.c:1229:6 [opt] frame #14: 0x00000001001b8be2 git`traverse_trees(istate=0x0000000100299940, n=2, t=<unavailable>, info=<unavailable>) at tree-walk.c:497:17 [opt] frame #15: 0x00000001001ba80f git`unpack_trees(len=2, t=0x00007ffeefbfebe0, o=0x00007ffeefbfe8c0) at unpack-trees.c:1546:9 [opt] frame #16: 0x000000010001a443 git`merge_working_tree(opts=0x00007ffeefbfee38, old_branch_info=0x00007ffeefbfeca0, new_branch_info=0x00007ffeefbfeda0, writeout_error=0x00007ffeefbfeccc) at checkout.c:704:9 [opt] frame #17: 0x000000010001a08c git`switch_branches(opts=0x00007ffeefbfee38, new_branch_info=0x00007ffeefbfeda0) at checkout.c:1057:9 [opt] frame #18: 0x0000000100018df0 git`checkout_branch(opts=<unavailable>, new_branch_info=<unavailable>) at checkout.c:1426:9 [opt] frame #19: 0x0000000100017b90 git`checkout_main(argc=0, argv=0x00007ffeefbff570, prefix=0x0000000000000000, opts=0x00007ffeefbfee38, options=<unavailable>, usagestr=<unavailable>) at checkout.c:1682:10 [opt] frame #20: 0x0000000100016f2d git`cmd_checkout(argc=2, argv=0x00007ffeefbff568, prefix=0x0000000000000000) at checkout.c:1731:8 [opt] frame #21: 0x00000001000026f6 git`run_builtin(p=0x000000010024c710, argc=2, argv=0x00007ffeefbff568) at git.c:444:11 [opt] frame #22: 0x0000000100001a36 git`handle_builtin(argc=2, argv=0x00007ffeefbff568) at git.c:673:3 [opt] frame #23: 0x000000010000235c git`run_argv(argcp=0x00007ffeefbff4ec, argv=0x00007ffeefbff4d8) at git.c:740:4 [opt] frame #24: 0x0000000100001794 git`cmd_main(argc=2, argv=0x00007ffeefbff568) at git.c:871:19 [opt] frame #25: 0x00000001000a4405 git`main(argc=<unavailable>, argv=0x00007ffeefbff560) at common-main.c:52:11 [opt] frame #26: 0x00007fff783053d5 libdyld.dylib`start + 1
Sorry for the information dump, I haven't had the time to properly look into the issue but I just wanted to make sure that you're aware.
Thanks and hope this helps,
Denton
Re: [BUG] git is segfaulting, was [PATCH v4 04/12] dir: also check directories for matching pathspecs
Hi Denton,
Quoted 53 lines · Denton Liu
Hi Elijah,
I ran into a segfault on MacOS. I managed to bisect it down to
404ebceda0(dir: also check directories for matching pathspecs, 2019-09-17), which should be the patch in the parent thread. The test case below works fine without this patch applied but segfaults once it is applied.#!/bin/sh
git worktree add testdir
git -C testdir checkout master
git -C testdir fetch https://github.com/git/git.git todo bin-wrappers/git -C testdir checkoutFETCH_HEAD# segfault hereNote that the worktree part isn't necessary to reproduce the problem but I didn't want my files to be constantly refreshed, triggering a rebuild each time.
I also managed to get this backtrace from running lldb at the segfault but it is based on the latest "jch" commit,
1cc52d20df(Merge branch 'jt/merge-recursive-symlink-is-not-a-dir-in-way' into jch, 2019-09-20).* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x8) * frame #0: 0x00000001000f63a0 git`do_match_pathspec(istate=0x0000000100299940, ps=0x000000010200aa80, name="Gitweb/static/js/lib/", namelen=21, prefix=0, seen=0x0000000000000000, flags=0) at dir.c:420:2 [opt] frame #1: 0x00000001000f632c git`match_pathspec(istate=0x0000000100299940, ps=0x0000000000000000, name="Gitweb/static/js/lib/", namelen=21, prefix=0, seen=0x0000000000000000, is_dir=0) at dir.c:490:13 [opt] frame #2: 0x00000001000f8315 git`read_directory_recursive(dir=0x00007ffeefbfe278, istate=0x0000000100299940, base=<unavailable>, baselen=17, untracked=<unavailable>, check_only=0, stop_at_first_file=0, pathspec=0x0000000000000000) at dir.c:1990:9 [opt] frame #3: 0x00000001000f82e9 git`read_directory_recursive(dir=0x00007ffeefbfe278, istate=0x0000000100299940, base=<unavailable>, baselen=14, untracked=<unavailable>, check_only=0, stop_at_first_file=0, pathspec=0x0000000000000000) at dir.c:1984:5 [opt] frame #4: 0x00000001000f82e9 git`read_directory_recursive(dir=0x00007ffeefbfe278, istate=0x0000000100299940, base=<unavailable>, baselen=7, untracked=<unavailable>, check_only=0, stop_at_first_file=0, pathspec=0x0000000000000000) at dir.c:1984:5 [opt] frame #5: 0x00000001000f60d1 git`read_directory(dir=0x00007ffeefbfe278, istate=0x0000000100299940, path="Gitweb/", len=7, pathspec=0x0000000000000000) at dir.c:2298:3 [opt] frame #6: 0x00000001001bded1 git`verify_clean_subdirectory(ce=<unavailable>, o=0x00007ffeefbfe8c0) at unpack-trees.c:1846:6 [opt] frame #7: 0x00000001001bdc1d git`check_ok_to_remove(name="Gitweb", len=6, dtype=4, ce=0x0000000103e70de0, st=0x00007ffeefbfe438, error_type=ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN, o=0x00007ffeefbfe8c0) at unpack-trees.c:1901:7 [opt] frame #8: 0x00000001001bdb01 git`verify_absent_1(ce=<unavailable>, error_type=<unavailable>, o=<unavailable>) at unpack-trees.c:1964:10 [opt] frame #9: 0x00000001001bafc0 git`verify_absent(ce=<unavailable>, error_type=<unavailable>, o=<unavailable>) at unpack-trees.c:1052:11 [opt] [artificial] frame #10: 0x00000001001bbc3c git`merged_entry(ce=0x0000000100605fb0, old=0x0000000000000000, o=0x00007ffeefbfe8c0) at unpack-trees.c:2013:7 [opt] frame #11: 0x00000001001bd2b7 git`call_unpack_fn(src=<unavailable>, o=<unavailable>) at unpack-trees.c:522:12 [opt] frame #12: 0x00000001001bca16 git`unpack_nondirectories(n=2, mask=2, dirmask=<unavailable>, src=0x00007ffeefbfe5d0, names=<unavailable>, info=0x00007ffeefbfe718) at unpack-trees.c:1029:12 [opt] frame #13: 0x00000001001bad1a git`unpack_callback(n=2, mask=2, dirmask=0, names=0x0000000102007390, info=0x00007ffeefbfe718) at unpack-trees.c:1229:6 [opt] frame #14: 0x00000001001b8be2 git`traverse_trees(istate=0x0000000100299940, n=2, t=<unavailable>, info=<unavailable>) at tree-walk.c:497:17 [opt] frame #15: 0x00000001001ba80f git`unpack_trees(len=2, t=0x00007ffeefbfebe0, o=0x00007ffeefbfe8c0) at unpack-trees.c:1546:9 [opt] frame #16: 0x000000010001a443 git`merge_working_tree(opts=0x00007ffeefbfee38, old_branch_info=0x00007ffeefbfeca0, new_branch_info=0x00007ffeefbfeda0, writeout_error=0x00007ffeefbfeccc) at checkout.c:704:9 [opt] frame #17: 0x000000010001a08c git`switch_branches(opts=0x00007ffeefbfee38, new_branch_info=0x00007ffeefbfeda0) at checkout.c:1057:9 [opt] frame #18: 0x0000000100018df0 git`checkout_branch(opts=<unavailable>, new_branch_info=<unavailable>) at checkout.c:1426:9 [opt] frame #19: 0x0000000100017b90 git`checkout_main(argc=0, argv=0x00007ffeefbff570, prefix=0x0000000000000000, opts=0x00007ffeefbfee38, options=<unavailable>, usagestr=<unavailable>) at checkout.c:1682:10 [opt] frame #20: 0x0000000100016f2d git`cmd_checkout(argc=2, argv=0x00007ffeefbff568, prefix=0x0000000000000000) at checkout.c:1731:8 [opt] frame #21: 0x00000001000026f6 git`run_builtin(p=0x000000010024c710, argc=2, argv=0x00007ffeefbff568) at git.c:444:11 [opt] frame #22: 0x0000000100001a36 git`handle_builtin(argc=2, argv=0x00007ffeefbff568) at git.c:673:3 [opt] frame #23: 0x000000010000235c git`run_argv(argcp=0x00007ffeefbff4ec, argv=0x00007ffeefbff4d8) at git.c:740:4 [opt] frame #24: 0x0000000100001794 git`cmd_main(argc=2, argv=0x00007ffeefbff568) at git.c:871:19 [opt] frame #25: 0x00000001000a4405 git`main(argc=<unavailable>, argv=0x00007ffeefbff560) at common-main.c:52:11 [opt] frame #26: 0x00007fff783053d5 libdyld.dylib`start + 1
Quoted 2 lines
Sorry for the information dump, I haven't had the time to properly look into the issue but I just wanted to make sure that you're aware.
Thanks for testing and sending the heads up. Unfortunately, I cannot reproduce on either Linux or Mac. Do you have some special ignore files or sparse-checkout paths that are important to triggering? What's in your config.mak? What compiler and version?
Here's what I did, just to verify:
cd ~/floss/git
git checkout 404ebceda0
NO_GETTEXT=1 make DEVELOPER=1 -j8 # I leave off the NO_GETTEXT=1 on linux
git worktree add testdir
git -C testdir checkout master
git -C testdir fetch https://github.com/git/git.git todo bin-wrappers/git -C testdir checkout FETCH_HEAD
Did I get any of those steps wrong?
Thanks,
Elijah
Re: [BUG] git is segfaulting, was [PATCH v4 04/12] dir: also check directories for matching pathspecs
On Wed, Sep 25, 2019 at 02:28:15PM -0700, Elijah Newren wrote:
[...]
Quoted 7 lines
Quoted 2 lines
Sorry for the information dump, I haven't had the time to properly look into the issue but I just wanted to make sure that you're aware.
Thanks for testing and sending the heads up. Unfortunately, I cannot reproduce on either Linux or Mac. Do you have some special ignore files or sparse-checkout paths that are important to triggering? What's in your config.mak?
Before, I had an empty config.mak and I also had the following .git/info/exclude (these are two worktrees I have checked out):
/jch
/patches
aside from that, I don't think I've changed anything else. Anyway, to double-check that it wasn't my setup that was broken, I ran
cd .. git clone git git2 cd git2 make configure ./configure
and then followed the rest the steps and I could still reproduce it.
Quoted 1 line
What compiler and version?
$ cc --version Apple LLVM version 10.0.1 (clang-1001.0.46.4) Target: x86_64-apple-darwin18.7.0 Thread model: posix InstalledDir: /Library/Developer/CommandLineTools/usr/bin
Quoted 6 lines
Here's what I did, just to verify:
cd ~/floss/git
git checkout404ebceda0
NO_GETTEXT=1 make DEVELOPER=1 -j8 # I leave off the NO_GETTEXT=1 on linux
I don't have NO_GETTEXT on Mac but I don't think it affects anything.
Quoted 6 lines
git worktree add testdir
git -C testdir checkout master
git -C testdir fetch https://github.com/git/git.git todo bin-wrappers/git -C testdir checkoutFETCH_HEADDid I get any of those steps wrong?
Looks correct to me. I don't see why this wouldn't reproduce. I'll send you more information if I figure anything else out.
Thanks,
Denton
Quoted 4 lines
Thanks,
Elijah
Re: [BUG] git is segfaulting, was [PATCH v4 04/12] dir: also check directories for matching pathspecs
Quoted 2 lines · Denton Liu
Looks correct to me. I don't see why this wouldn't reproduce. I'll send you more information if I figure anything else out.
I looked into it a little more and I think I know why it's being triggered.
When we checkout 'todo' from 'master', since they're completely different trees, all of git's source files need to be removed. As a result, the checkout process at some point invokes check_ok_to_remove().
This kicks off the following call chain:
check_ok_to_remove()
verify_clean_subdirectory()
read_directory()
read_directory_recursive() (this is called recursively, of course) match_pathspec()
do_match_pathspec()
Where we segfault in do_match_pathspec() because ps is NULL:
GUARD_PATHSPEC(ps, PATHSPEC_FROMTOP | PATHSPEC_MAXDEPTH | PATHSPEC_LITERAL | PATHSPEC_GLOB | PATHSPEC_ICASE | PATHSPEC_EXCLUDE | PATHSPEC_ATTR);
So why is ps == NULL? In verify_clean_subdirectory(), we call read_directory() like this:
i = read_directory(&d, o->src_index, pathbuf, namelen+1, NULL);
where we explictly pass in a NULL and it is handed down the callstack. I guess this means that we should be expecting that pathspecs can be NULL in this path. So I've applied the patch at the bottom and it fixes the problem.
I was wondering if we should stick a
if (!ps)
BUG("ps is NULL");into do_match_pathspec(), though, so we can avoid these situations in the future.
Also, I'm still not sure why the issue wasn't reproducible on your side... I'm not too familiar with this area of the code, though.
-- >8 --
dir.c +1 -1
@@ -1952,7 +1952,7 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,if (subdir_state > dir_state) dir_state = subdir_state; -if (!match_pathspec (istate, pathspec, path.buf, path.len, +if (pathspec && !match_pathspec (istate, pathspec, path.buf, path.len,0 /* prefix */ ,NULL ,0 /* do NOT special case dirs */ )) state = path_none;
Re: [BUG] git is segfaulting, was [PATCH v4 04/12] dir: also check directories for matching pathspecs
Hi Denton,
Quoted 59 lines · Denton Liu
Quoted 2 lines · Denton Liu
Looks correct to me. I don't see why this wouldn't reproduce. I'll send you more information if I figure anything else out.
I looked into it a little more and I think I know why it's being triggered.
When we checkout 'todo' from 'master', since they're completely different trees, all of git's source files need to be removed. As a result, the checkout process at some point invokes
check_ok_to_remove().This kicks off the following call chain:
check_ok_to_remove()
verify_clean_subdirectory()
read_directory()
read_directory_recursive()(this is called recursively, of course)match_pathspec()
do_match_pathspec()Where we segfault in
do_match_pathspec()because ps is NULL:GUARD_PATHSPEC(ps, PATHSPEC_FROMTOP | PATHSPEC_MAXDEPTH | PATHSPEC_LITERAL | PATHSPEC_GLOB | PATHSPEC_ICASE | PATHSPEC_EXCLUDE | PATHSPEC_ATTR);So why is ps == NULL? In
verify_clean_subdirectory(), we callread_directory()like this:i = read_directory(&d, o->src_index, pathbuf, namelen+1, NULL);where we explictly pass in a NULL and it is handed down the callstack. I guess this means that we should be expecting that pathspecs can be NULL in this path. So I've applied the patch at the bottom and it fixes the problem.
I was wondering if we should stick a
if (!ps) BUG("ps is NULL");into
do_match_pathspec(), though, so we can avoid these situations in the future.Also, I'm still not sure why the issue wasn't reproducible on your side... I'm not too familiar with this area of the code, though.
-- >8 --
dir.c +0
-0
dir.c static enum path_treatment read_directory_recursive(struct dir_struct *dir, · line 1958
Quoted 8 lines
dir.c +1
-1@@ -1952,7 +1952,7 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,if (subdir_state > dir_state) dir_state = subdir_state; -if (!match_pathspec (istate, pathspec, path.buf, path.len, +if (pathspec && !match_pathspec (istate, pathspec, path.buf, path.len,0 /* prefix */ ,NULL ,0 /* do NOT special case dirs */ )) state = path_none;
The patch makes sense...but I'd really like to add a test, and understand it better so I can check to see if there are any other bad codepaths. Sadly, I still have no idea how to reproduce the bug. I can put
char *oopsies = NULL;
printf("oopsies = %s\n", oopsies);at the beginning of check_ok_to_remove() to verify that function is never called and run the steps you gave with no problem. However, I do notice that your reproduction steps involve 'master' which may have local changes for you that I don't have. Is there any chance you can reproduce this using a commit id that is already upstream instead of 'master'? I've been poking around unpack-trees.c for a bit but I'm having a hard time reversing out of it what's different about our setups and how to trigger.
Re: [BUG] git is segfaulting, was [PATCH v4 04/12] dir: also check directories for matching pathspecs
In reply to Denton LiuQuoted 23 lines · Denton Liu
Hi Elijah,
I ran into a segfault on MacOS. I managed to bisect it down to
404ebceda0(dir: also check directories for matching pathspecs, 2019-09-17), which should be the patch in the parent thread. The test case below works fine without this patch applied but segfaults once it is applied.#!/bin/sh
git worktree add testdir
git -C testdir checkout master
git -C testdir fetch https://github.com/git/git.git todo bin-wrappers/git -C testdir checkoutFETCH_HEAD# segfault hereNote that the worktree part isn't necessary to reproduce the problem but I didn't want my files to be constantly refreshed, triggering a rebuild each time.
I also managed to get this backtrace from running lldb at the segfault but it is based on the latest "jch" commit,
1cc52d20df(Merge branch 'jt/merge-recursive-symlink-is-not-a-dir-in-way' into jch, 2019-09-20).
Quoted 9 lines
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x8) * frame #0: 0x00000001000f63a0 git`do_match_pathspec(istate=0x0000000100299940, ps=0x000000010200aa80, name="Gitweb/static/js/lib/", namelen=21, prefix=0, seen=0x0000000000000000, flags=0) at dir.c:420:2 [opt] frame #1: 0x00000001000f632c git`match_pathspec(istate=0x0000000100299940, ps=0x0000000000000000, name="Gitweb/static/js/lib/", namelen=21, prefix=0, seen=0x0000000000000000, is_dir=0) at dir.c:490:13 [opt] frame #2: 0x00000001000f8315 git`read_directory_recursive(dir=0x00007ffeefbfe278, istate=0x0000000100299940, base=<unavailable>, baselen=17, untracked=<unavailable>, check_only=0, stop_at_first_file=0, pathspec=0x0000000000000000) at dir.c:1990:9 [opt] frame #3: 0x00000001000f82e9 git`read_directory_recursive(dir=0x00007ffeefbfe278, istate=0x0000000100299940, base=<unavailable>, baselen=14, untracked=<unavailable>, check_only=0, stop_at_first_file=0, pathspec=0x0000000000000000) at dir.c:1984:5 [opt] frame #4: 0x00000001000f82e9 git`read_directory_recursive(dir=0x00007ffeefbfe278, istate=0x0000000100299940, base=<unavailable>, baselen=7, untracked=<unavailable>, check_only=0, stop_at_first_file=0, pathspec=0x0000000000000000) at dir.c:1984:5 [opt] frame #5: 0x00000001000f60d1 git`read_directory(dir=0x00007ffeefbfe278, istate=0x0000000100299940, path="Gitweb/", len=7, pathspec=0x0000000000000000) at dir.c:2298:3 [opt] frame #6: 0x00000001001bded1 git`verify_clean_subdirectory(ce=<unavailable>, o=0x00007ffeefbfe8c0) at unpack-trees.c:1846:6 [opt] frame #7: 0x00000001001bdc1d git`check_ok_to_remove(name="Gitweb", len=6, dtype=4, ce=0x0000000103e70de0, st=0x00007ffeefbfe438, error_type=ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN, o=0x00007ffeefbfe8c0) at unpack-trees.c:1901:7 [opt]
That 'name="Gitweb" parameter caught my eye. origin/todo contains a 'Gitweb' file, with upper case 'G', while master contains a 'gitweb' directory, with lower case 'g'.
Could it be that case (in)sensitivity plays a crucial rule in triggering the segfault? FWIW I could reproduce it following Denton's description on Travis CI's macOS VM with the debug shell access, and it uses case insensitive file system.
Re: [BUG] git is segfaulting, was [PATCH v4 04/12] dir: also check directories for matching pathspecs
Quoted 34 lines · SZEDER Gábor
Quoted 23 lines · Denton Liu
Hi Elijah,
I ran into a segfault on MacOS. I managed to bisect it down to
404ebceda0(dir: also check directories for matching pathspecs, 2019-09-17), which should be the patch in the parent thread. The test case below works fine without this patch applied but segfaults once it is applied.#!/bin/sh
git worktree add testdir
git -C testdir checkout master
git -C testdir fetch https://github.com/git/git.git todo bin-wrappers/git -C testdir checkoutFETCH_HEAD# segfault hereNote that the worktree part isn't necessary to reproduce the problem but I didn't want my files to be constantly refreshed, triggering a rebuild each time.
I also managed to get this backtrace from running lldb at the segfault but it is based on the latest "jch" commit,
1cc52d20df(Merge branch 'jt/merge-recursive-symlink-is-not-a-dir-in-way' into jch, 2019-09-20).Quoted 9 lines
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x8) * frame #0: 0x00000001000f63a0 git`do_match_pathspec(istate=0x0000000100299940, ps=0x000000010200aa80, name="Gitweb/static/js/lib/", namelen=21, prefix=0, seen=0x0000000000000000, flags=0) at dir.c:420:2 [opt] frame #1: 0x00000001000f632c git`match_pathspec(istate=0x0000000100299940, ps=0x0000000000000000, name="Gitweb/static/js/lib/", namelen=21, prefix=0, seen=0x0000000000000000, is_dir=0) at dir.c:490:13 [opt] frame #2: 0x00000001000f8315 git`read_directory_recursive(dir=0x00007ffeefbfe278, istate=0x0000000100299940, base=<unavailable>, baselen=17, untracked=<unavailable>, check_only=0, stop_at_first_file=0, pathspec=0x0000000000000000) at dir.c:1990:9 [opt] frame #3: 0x00000001000f82e9 git`read_directory_recursive(dir=0x00007ffeefbfe278, istate=0x0000000100299940, base=<unavailable>, baselen=14, untracked=<unavailable>, check_only=0, stop_at_first_file=0, pathspec=0x0000000000000000) at dir.c:1984:5 [opt] frame #4: 0x00000001000f82e9 git`read_directory_recursive(dir=0x00007ffeefbfe278, istate=0x0000000100299940, base=<unavailable>, baselen=7, untracked=<unavailable>, check_only=0, stop_at_first_file=0, pathspec=0x0000000000000000) at dir.c:1984:5 [opt] frame #5: 0x00000001000f60d1 git`read_directory(dir=0x00007ffeefbfe278, istate=0x0000000100299940, path="Gitweb/", len=7, pathspec=0x0000000000000000) at dir.c:2298:3 [opt] frame #6: 0x00000001001bded1 git`verify_clean_subdirectory(ce=<unavailable>, o=0x00007ffeefbfe8c0) at unpack-trees.c:1846:6 [opt] frame #7: 0x00000001001bdc1d git`check_ok_to_remove(name="Gitweb", len=6, dtype=4, ce=0x0000000103e70de0, st=0x00007ffeefbfe438, error_type=ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN, o=0x00007ffeefbfe8c0) at unpack-trees.c:1901:7 [opt]
Quoted 8 lines
That 'name="Gitweb" parameter caught my eye. origin/todo contains a 'Gitweb' file, with upper case 'G', while master contains a 'gitweb' directory, with lower case 'g'.
Could it be that case (in)sensitivity plays a crucial rule in triggering the segfault? FWIW I could reproduce it following Denton's description on Travis CI's macOS VM with the debug shell access, and it uses case insensitive file system.
Indeed, with 404ebceda0 the test below segfaults on case insensitive fs, but not on a case sensitive one.
t/t0050-filesystem.sh +23 -0
@@ -131,4 +131,27 @@ $test_unicode 'merge (silent unicode normalization)' ' git merge topic ' +test_expect_success CASE_INSENSITIVE_FS "Denton's segfault" ' + git init repo && + ( + cd repo && + + echo foo >Gitweb && + git add Gitweb && + git commit -m "add Gitweb" && + + git checkout --orphan todo && + git reset --hard && + # the subdir is crucial, without it there is no segfault + mkdir -p gitweb/subdir && + echo bar >gitweb/subdir/file && + # it is not strictly necessary to add and commit the + # gitweb directory, its presence is sufficient + git add gitweb && + git commit -m "add gitweb/subdir/file" && + + git checkout master + ) +' + test_done
The end of its trace:
++git checkout master
./test-lib.sh: line 910: 11220 Segmentation fault: 11 git checkout master
error: last command exited with $?=139
Case insensitivity is important because check_ok_to_remove() is invoked from verify_absent_1(), which looks like this:
if (...)
....
else if (...)
....
else if (lstat(ce->name, &st))
// That lstat() checked whether 'Gitweb' is absent. On a case
// sensitive fs it's absent, so it returns. On a case
// insensitive fs it finds 'master's 'gitweb' directory, so it
// goes on to the else below, and eventually segfaults.
return; else
check_ok_to_remove()
Good night :)
Re: [BUG] git is segfaulting, was [PATCH v4 04/12] dir: also check directories for matching pathspecs
Quoted 39 lines · SZEDER Gábor
Quoted 38 lines · SZEDER Gábor
Quoted 23 lines · Denton Liu
Hi Elijah,
I ran into a segfault on MacOS. I managed to bisect it down to
404ebceda0(dir: also check directories for matching pathspecs, 2019-09-17), which should be the patch in the parent thread. The test case below works fine without this patch applied but segfaults once it is applied.#!/bin/sh
git worktree add testdir
git -C testdir checkout master
git -C testdir fetch https://github.com/git/git.git todo bin-wrappers/git -C testdir checkoutFETCH_HEAD# segfault hereNote that the worktree part isn't necessary to reproduce the problem but I didn't want my files to be constantly refreshed, triggering a rebuild each time.
I also managed to get this backtrace from running lldb at the segfault but it is based on the latest "jch" commit,
1cc52d20df(Merge branch 'jt/merge-recursive-symlink-is-not-a-dir-in-way' into jch, 2019-09-20).Quoted 9 lines
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x8) * frame #0: 0x00000001000f63a0 git`do_match_pathspec(istate=0x0000000100299940, ps=0x000000010200aa80, name="Gitweb/static/js/lib/", namelen=21, prefix=0, seen=0x0000000000000000, flags=0) at dir.c:420:2 [opt] frame #1: 0x00000001000f632c git`match_pathspec(istate=0x0000000100299940, ps=0x0000000000000000, name="Gitweb/static/js/lib/", namelen=21, prefix=0, seen=0x0000000000000000, is_dir=0) at dir.c:490:13 [opt] frame #2: 0x00000001000f8315 git`read_directory_recursive(dir=0x00007ffeefbfe278, istate=0x0000000100299940, base=<unavailable>, baselen=17, untracked=<unavailable>, check_only=0, stop_at_first_file=0, pathspec=0x0000000000000000) at dir.c:1990:9 [opt] frame #3: 0x00000001000f82e9 git`read_directory_recursive(dir=0x00007ffeefbfe278, istate=0x0000000100299940, base=<unavailable>, baselen=14, untracked=<unavailable>, check_only=0, stop_at_first_file=0, pathspec=0x0000000000000000) at dir.c:1984:5 [opt] frame #4: 0x00000001000f82e9 git`read_directory_recursive(dir=0x00007ffeefbfe278, istate=0x0000000100299940, base=<unavailable>, baselen=7, untracked=<unavailable>, check_only=0, stop_at_first_file=0, pathspec=0x0000000000000000) at dir.c:1984:5 [opt] frame #5: 0x00000001000f60d1 git`read_directory(dir=0x00007ffeefbfe278, istate=0x0000000100299940, path="Gitweb/", len=7, pathspec=0x0000000000000000) at dir.c:2298:3 [opt] frame #6: 0x00000001001bded1 git`verify_clean_subdirectory(ce=<unavailable>, o=0x00007ffeefbfe8c0) at unpack-trees.c:1846:6 [opt] frame #7: 0x00000001001bdc1d git`check_ok_to_remove(name="Gitweb", len=6, dtype=4, ce=0x0000000103e70de0, st=0x00007ffeefbfe438, error_type=ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN, o=0x00007ffeefbfe8c0) at unpack-trees.c:1901:7 [opt]That 'name="Gitweb" parameter caught my eye. origin/todo contains a 'Gitweb' file, with upper case 'G', while master contains a 'gitweb' directory, with lower case 'g'.
Quoted 7 lines
Quoted 4 lines
Could it be that case (in)sensitivity plays a crucial rule in triggering the segfault? FWIW I could reproduce it following Denton's description on Travis CI's macOS VM with the debug shell access, and it uses case insensitive file system.
Indeed, with
404ebceda0the test below segfaults on case insensitive fs, but not on a case sensitive one.
Wow, good catch. I didn't even notice that in the backtrace.
Quoted 24 lines
t/t0050-filesystem.sh +14
-0@@ -131,4 +131,27 @@ $test_unicode 'merge (silent unicode normalization)' ' git merge topic ' +test_expect_success CASE_INSENSITIVE_FS "Denton's segfault" ' + git init repo && + ( + cd repo && + + echo foo >Gitweb && + git add Gitweb && + git commit -m "add Gitweb" && + + git checkout --orphan todo && + git reset --hard && + # the subdir is crucial, without it there is no segfault + mkdir -p gitweb/subdir && + echo bar >gitweb/subdir/file &&
t/t0050-filesystem.sh $test_unicode 'merge (silent unicode normalization)' ' · line 157
Quoted 10 lines
t/t0050-filesystem.sh +9
-0+ # it is not strictly necessary to add and commit the + # gitweb directory, its presence is sufficient + git add gitweb && + git commit -m "add gitweb/subdir/file" && + + git checkout master + ) +' + test_done
I can confirm that this test case reproduces for me. Thanks for writing this.
Quoted 22 lines
The end of its trace:
++git checkout master
./test-lib.sh: line 910: 11220 Segmentation fault: 11 git checkout master
error: last command exited with $?=139Case insensitivity is important because
check_ok_to_remove()is invoked fromverify_absent_1(), which looks like this:if (...)
.... else if (...) .... else if (lstat(ce->name, &st)) // That lstat() checked whether 'Gitweb' is absent. On a case // sensitive fs it's absent, so it returns. On a case // insensitive fs it finds 'master's 'gitweb' directory, so it // goes on to the else below, and eventually segfaults. return;
Quoted 5 lines
else
check_ok_to_remove()Good night :)
Thanks for your help!
PATCH 1/1 dir: special case check for the possibility that pathspec is NULL
Commits 404ebceda01c ("dir: also check directories for matching pathspecs", 2019-09-17) and 89a1f4aaf765 ("dir: if our pathspec might match files under a dir, recurse into it", 2019-09-17) added calls to match_pathspec() and do_match_pathspec() passing along their pathspec parameter. Both match_pathspec() and do_match_pathspec() assume the pathspec argument they are given is non-NULL. It turns out that unpack-tree.c's verify_clean_subdirectory() calls read_directory() with pathspec == NULL, and it is possible on case insensitive filesystems for that NULL to make it to these new calls to match_pathspec() and do_match_pathspec(). Add appropriate checks on the NULLness of pathspec to avoid a segfault.
In case the negation throws anyone off (one of the calls was to do_match_pathspec() while the other was to !match_pathspec(), yet no negation of the NULLness of pathspec is used), there are two ways to understand the differences:
* The code already handled the pathspec == NULL cases before this series, and this series only tried to change behavior when there was a pathspec, thus we only want to go into the if-block if pathspec is non-NULL.
* One of the calls is for whether to recurse into a subdirectory, the other is for after we've recursed into it for whether we want to remove the subdirectory itself (i.e. the subdirectory didn't match but something under it could have). That difference in situation leads to the slight differences in logic used (well, that and the slightly unusual fact that we don't want empty pathspecs to remove untracked directories by default).
---
This patch applies on top of en/clean-nested-with-ignored, which is now in next.
Denton found and analyzed one issue and provided the patch for the match_pathspec() call, SZEDER figured out why the issue only reproduced for some folks and not others and provided the testcase, and I looked through the remainder of the series and noted the do_match_pathspec() call that should have the same check.
So, I'm not sure who should be author and who should be helped-by; I feel like their contributions are possibly bigger than mine. While I tried to reproduce and debug, they ended up doing the work, and I just looked through the rest of the series for similar issues and wrote up a commit message. *shrug*
dir.c | 8 +++++--- t/t0050-filesystem.sh | 23 +++++++++++++++++++++++ 2 files changed, 28 insertions(+), 3 deletions(-)
dir.c +5 -3
@@ -1962,8 +1962,9 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir, ((state == path_untracked) && (get_dtype(cdir.de, istate, path.buf, path.len) == DT_DIR) && ((dir->flags & DIR_SHOW_IGNORED_TOO) || - do_match_pathspec(istate, pathspec, path.buf, path.len, - baselen, NULL, DO_MATCH_LEADING_PATHSPEC) == MATCHED_RECURSIVELY_LEADING_PATHSPEC))) { + (pathspec && + do_match_pathspec(istate, pathspec, path.buf, path.len, + baselen, NULL, DO_MATCH_LEADING_PATHSPEC) == MATCHED_RECURSIVELY_LEADING_PATHSPEC)))) {struct untracked_cache_dir *ud; ud = lookup_untracked(dir->untracked, untracked, path.buf + baselen,
@@ -1975,7 +1976,8 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,if (subdir_state > dir_state) dir_state = subdir_state; -if (!match_pathspec (istate, pathspec, path.buf, path.len, +if (pathspec && + !match_pathspec (istate, pathspec, path.buf, path.len,0 /* prefix */ ,NULL ,0 /* do NOT special case dirs */ )) state = path_none;
t/t0050-filesystem.sh +23 -0
@@ -131,4 +131,27 @@ $test_unicode 'merge (silent unicode normalization)' ' git merge topic ' +test_expect_success CASE_INSENSITIVE_FS 'checkout with no pathspec and a case insensitive fs' ' + git init repo && + ( + cd repo && + + >Gitweb && + git add Gitweb && + git commit -m "add Gitweb" && + + git checkout --orphan todo && + git reset --hard && + # the subdir is crucial, without it there is no segfault + mkdir -p gitweb/subdir && + >gitweb/subdir/file && + # it is not strictly necessary to add and commit the + # gitweb directory, its presence is sufficient + git add gitweb && + git commit -m "add gitweb/subdir/file" && + + git checkout master + ) +' + test_done
Signature · 1 line
2.22.1.14.g885c22d24b
Re: [PATCH] dir: special case check for the possibility that pathspec is NULL
Hi Elijah,
Quoted 35 lines · Elijah Newren
Commits
404ebceda01c("dir: also check directories for matching pathspecs", 2019-09-17) and89a1f4aaf765("dir: if our pathspec might match files under a dir, recurse into it", 2019-09-17) added calls tomatch_pathspec()anddo_match_pathspec()passing along their pathspec parameter. Bothmatch_pathspec()anddo_match_pathspec()assume the pathspec argument they are given is non-NULL. It turns out that unpack-tree.c'sverify_clean_subdirectory()callsread_directory()with pathspec == NULL, and it is possible on case insensitive filesystems for that NULL to make it to these new calls tomatch_pathspec()anddo_match_pathspec(). Add appropriate checks on the NULLness of pathspec to avoid a segfault.In case the negation throws anyone off (one of the calls was to
do_match_pathspec()while the other was to !match_pathspec(), yet no negation of the NULLness of pathspec is used), there are two ways to understand the differences:
* The code already handled the pathspec == NULL cases before this series, and this series only tried to change behavior when there was a pathspec, thus we only want to go into the if-block if pathspec is non-NULL.
* One of the calls is for whether to recurse into a subdirectory, the other is for after we've recursed into it for whether we want to remove the subdirectory itself (i.e. the subdirectory didn't match but something under it could have). That difference in situation leads to the slight differences in logic used (well, that and the slightly unusual fact that we don't want empty pathspecs to remove untracked directories by default).Helped-by Denton Liu <[email protected]>Helped-by SZEDER Gábor <[email protected]>Signed-off-by Elijah Newren <[email protected]>---
This patch applies on top of en/clean-nested-with-ignored, which is now in next.
Quoted 5 lines
Denton found and analyzed one issue and provided the patch for the
match_pathspec()call, SZEDER figured out why the issue only reproduced for some folks and not others and provided the testcase, and I looked through the remainder of the series and noted thedo_match_pathspec()call that should have the same check.
Thanks for catching what I missed.
Quoted 6 lines
So, I'm not sure who should be author and who should be helped-by; I feel like their contributions are possibly bigger than mine. While I tried to reproduce and debug, they ended up doing the work, and I just looked through the rest of the series for similar issues and wrote up a commit message. *shrug*
Eh, it doesn't really matter to me. GitHub appears to have de facto standardised the Co-authored-by: trailer to allow credit to be split amonst multiple authors so _maybe_ we could use that, but I'm pretty impartial.
Quoted 41 lines
dir.c | 8 +++++--- t/t0050-filesystem.sh | 23 +++++++++++++++++++++++ 2 files changed, 28 insertions(+), 3 deletions(-)dir.c +5
-3@@ -1962,8 +1962,9 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir, ((state == path_untracked) && (get_dtype(cdir.de, istate, path.buf, path.len) == DT_DIR) && ((dir->flags & DIR_SHOW_IGNORED_TOO) || - do_match_pathspec(istate, pathspec, path.buf, path.len, - baselen, NULL, DO_MATCH_LEADING_PATHSPEC) == MATCHED_RECURSIVELY_LEADING_PATHSPEC))) { + (pathspec && + do_match_pathspec(istate, pathspec, path.buf, path.len, + baselen, NULL, DO_MATCH_LEADING_PATHSPEC) == MATCHED_RECURSIVELY_LEADING_PATHSPEC)))) {struct untracked_cache_dir *ud; ud = lookup_untracked(dir->untracked, untracked, path.buf + baselen,@@ -1975,7 +1976,8 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,if (subdir_state > dir_state) dir_state = subdir_state; -if (!match_pathspec (istate, pathspec, path.buf, path.len, +if (pathspec && + !match_pathspec (istate, pathspec, path.buf, path.len,0 /* prefix */ ,NULL ,0 /* do NOT special case dirs */ )) state = path_none;t/t0050-filesystem.sh +2
-0@@ -131,4 +131,27 @@ $test_unicode 'merge (silent unicode normalization)' ' git merge topic ' +test_expect_success CASE_INSENSITIVE_FS 'checkout with no pathspec and a case insensitive fs' ' + git init repo &&
t/t0050-filesystem.sh $test_unicode 'merge (silent unicode normalization)' ' · line 145
Quoted 10 lines
t/t0050-filesystem.sh +10
-0+ ( + cd repo && + + >Gitweb && + git add Gitweb && + git commit -m "add Gitweb" && + + git checkout --orphan todo && + git reset --hard && + # the subdir is crucial, without it there is no segfault
We should either remove this comment or change the justification. A future reader may be confused at what particular segfault this refers to.
t/t0050-filesystem.sh $test_unicode 'merge (silent unicode normalization)' ' · line 149
Quoted 4 lines
t/t0050-filesystem.sh +4
-0+ mkdir -p gitweb/subdir && + >gitweb/subdir/file && + # it is not strictly necessary to add and commit the + # gitweb directory, its presence is sufficient
Same here, its presence is sufficient to... what?
Thanks,
Denton
Quoted 11 lines
t/t0050-filesystem.sh +7
-0+ git add gitweb && + git commit -m "add gitweb/subdir/file" && + + git checkout master + ) +' + test_doneSignature · 1 line
2.22.1.14.g885c22d24b
Re: [PATCH] dir: special case check for the possibility that pathspec is NULL
Quoted 98 lines · Denton Liu
Hi Elijah,
Quoted 35 lines · Elijah Newren
Commits
404ebceda01c("dir: also check directories for matching pathspecs", 2019-09-17) and89a1f4aaf765("dir: if our pathspec might match files under a dir, recurse into it", 2019-09-17) added calls tomatch_pathspec()anddo_match_pathspec()passing along their pathspec parameter. Bothmatch_pathspec()anddo_match_pathspec()assume the pathspec argument they are given is non-NULL. It turns out that unpack-tree.c'sverify_clean_subdirectory()callsread_directory()with pathspec == NULL, and it is possible on case insensitive filesystems for that NULL to make it to these new calls tomatch_pathspec()anddo_match_pathspec(). Add appropriate checks on the NULLness of pathspec to avoid a segfault.In case the negation throws anyone off (one of the calls was to
do_match_pathspec()while the other was to !match_pathspec(), yet no negation of the NULLness of pathspec is used), there are two ways to understand the differences:
* The code already handled the pathspec == NULL cases before this series, and this series only tried to change behavior when there was a pathspec, thus we only want to go into the if-block if pathspec is non-NULL.
* One of the calls is for whether to recurse into a subdirectory, the other is for after we've recursed into it for whether we want to remove the subdirectory itself (i.e. the subdirectory didn't match but something under it could have). That difference in situation leads to the slight differences in logic used (well, that and the slightly unusual fact that we don't want empty pathspecs to remove untracked directories by default).Helped-by Denton Liu <[email protected]>Helped-by SZEDER Gábor <[email protected]>Signed-off-by Elijah Newren <[email protected]>---
This patch applies on top of en/clean-nested-with-ignored, which is now in next.Quoted 5 lines
Denton found and analyzed one issue and provided the patch for the
match_pathspec()call, SZEDER figured out why the issue only reproduced for some folks and not others and provided the testcase, and I looked through the remainder of the series and noted thedo_match_pathspec()call that should have the same check.Thanks for catching what I missed.
Quoted 6 lines
So, I'm not sure who should be author and who should be helped-by; I feel like their contributions are possibly bigger than mine. While I tried to reproduce and debug, they ended up doing the work, and I just looked through the rest of the series for similar issues and wrote up a commit message. *shrug*
Eh, it doesn't really matter to me. GitHub appears to have de facto standardised the Co-authored-by: trailer to allow credit to be split amonst multiple authors so _maybe_ we could use that, but I'm pretty impartial.
Quoted 39 lines
dir.c | 8 +++++--- t/t0050-filesystem.sh | 23 +++++++++++++++++++++++ 2 files changed, 28 insertions(+), 3 deletions(-)dir.c +5
-3@@ -1962,8 +1962,9 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir, ((state == path_untracked) && (get_dtype (cdir.de, istate, path.buf, path.len) ==DT_DIR ) && ((dir->flags &DIR_SHOW_IGNORED_TOO ) || - do_match_pathspec(istate, pathspec, path.buf, path.len, - baselen, NULL, DO_MATCH_LEADING_PATHSPEC) == MATCHED_RECURSIVELY_LEADING_PATHSPEC))) { + (pathspec && +do_match_pathspec (istate, pathspec, path.buf, path.len, + baselen, NULL, DO_MATCH_LEADING_PATHSPEC) == MATCHED_RECURSIVELY_LEADING_PATHSPEC)))) { struct untracked_cache_dir *ud; ud = lookup_untracked(dir->untracked, untracked, path.buf + baselen,@@ -1975,7 +1976,8 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,if (subdir_state > dir_state) dir_state = subdir_state; -if (!match_pathspec (istate, pathspec, path.buf, path.len, +if (pathspec && + !match_pathspec (istate, pathspec, path.buf, path.len,0 /* prefix */ ,NULL ,0 /* do NOT special case dirs */ )) state = path_none;t/t0050-filesystem.sh +0
-0@@ -131,4 +131,27 @@ $test_unicode 'merge (silent unicode normalization)' ' git merge topic '
Quoted 16 lines
t/t0050-filesystem.sh $test_unicode 'merge (silent unicode normalization)' ' · line 145
Quoted 12 lines
t/t0050-filesystem.sh +12
-0+test_expect_success CASE_INSENSITIVE_FS 'checkout with no pathspec and a case insensitive fs' ' + git init repo && + ( + cd repo && + + >Gitweb && + git add Gitweb && + git commit -m "add Gitweb" && + + git checkout --orphan todo && + git reset --hard && + # the subdir is crucial, without it there is no segfaultWe should either remove this comment or change the justification. A future reader may be confused at what particular segfault this refers to.
Yep, good point, I'll just go ahead and remove it.
Quoted 6 lines
t/t0050-filesystem.sh $test_unicode 'merge (silent unicode normalization)' ' · line 149
Quoted 4 lines
t/t0050-filesystem.sh +4
-0+ mkdir -p gitweb/subdir && + >gitweb/subdir/file && + # it is not strictly necessary to add and commit the + # gitweb directory, its presence is sufficientSame here, its presence is sufficient to... what?
I will clean this one too and send a v2 tomorrow; it's getting late.
Thanks for all the digging you did on this bug to get it sorted out, Denton; I really appreciate it.
Elijah
PATCH v2 1/1 dir: special case check for the possibility that pathspec is NULL
In reply to Elijah NewrenCommits 404ebceda01c ("dir: also check directories for matching pathspecs", 2019-09-17) and 89a1f4aaf765 ("dir: if our pathspec might match files under a dir, recurse into it", 2019-09-17) added calls to match_pathspec() and do_match_pathspec() passing along their pathspec parameter. Both match_pathspec() and do_match_pathspec() assume the pathspec argument they are given is non-NULL. It turns out that unpack-tree.c's verify_clean_subdirectory() calls read_directory() with pathspec == NULL, and it is possible on case insensitive filesystems for that NULL to make it to these new calls to match_pathspec() and do_match_pathspec(). Add appropriate checks on the NULLness of pathspec to avoid a segfault.
In case the negation throws anyone off (one of the calls was to do_match_pathspec() while the other was to !match_pathspec(), yet no negation of the NULLness of pathspec is used), there are two ways to understand the differences:
* The code already handled the pathspec == NULL cases before this series, and this series only tried to change behavior when there was a pathspec, thus we only want to go into the if-block if pathspec is non-NULL.
* One of the calls is for whether to recurse into a subdirectory, the other is for after we've recursed into it for whether we want to remove the subdirectory itself (i.e. the subdirectory didn't match but something under it could have). That difference in situation leads to the slight differences in logic used (well, that and the slightly unusual fact that we don't want empty pathspecs to remove untracked directories by default).
Denton found and analyzed one issue and provided the patch for the match_pathspec() call, SZEDER figured out why the issue only reproduced for some folks and not others and provided the testcase, and I looked through the remainder of the series and noted the do_match_pathspec() call that should have the same check.
---
Note: Applies on top of en/clean-nested-with-ignored, in next.
As with v1, the authorship is really mixed, so I don't know if I should use Co-authored-by (highlighted as a possibility by Denton), or the far more common Helped-by (as suggested by Junio but based on a more limited summary of the different contributions), or if perhaps Denton or SZEDER should be marked as the author and I be marked as Helped-by or Co-authored-by. Since Denton commented on round 1, I used his suggestion for attribution in this round, but I'm open to changing it to whatever works best.
Changes since v1:
- Removed comments that made sense in context of the original thread but wouldn't be helpful to future readers.
- s/Helped-by/Co-authored-by/
Range-diff:
1: 885c22d24b ! 1: c495b9303c dir: special case check for the possibility that pathspec is NULL
@@ t/t0050-filesystem.sh: $test_unicode 'merge (silent unicode normalization)' '
+
+ git checkout --orphan todo &&
+ git reset --hard &&
-+ # the subdir is crucial, without it there is no segfault
+ mkdir -p gitweb/subdir &&
+ >gitweb/subdir/file &&
+ # it is not strictly necessary to add and commit the
-+ # gitweb directory, its presence is sufficient
+ git add gitweb &&
+ git commit -m "add gitweb/subdir/file" &&
+dir.c | 8 +++++--- t/t0050-filesystem.sh | 21 +++++++++++++++++++++ 2 files changed, 26 insertions(+), 3 deletions(-)
dir.c +5 -3
@@ -1962,8 +1962,9 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir, ((state == path_untracked) && (get_dtype(cdir.de, istate, path.buf, path.len) == DT_DIR) && ((dir->flags & DIR_SHOW_IGNORED_TOO) || - do_match_pathspec(istate, pathspec, path.buf, path.len, - baselen, NULL, DO_MATCH_LEADING_PATHSPEC) == MATCHED_RECURSIVELY_LEADING_PATHSPEC))) { + (pathspec && + do_match_pathspec(istate, pathspec, path.buf, path.len, + baselen, NULL, DO_MATCH_LEADING_PATHSPEC) == MATCHED_RECURSIVELY_LEADING_PATHSPEC)))) {struct untracked_cache_dir *ud; ud = lookup_untracked(dir->untracked, untracked, path.buf + baselen,
@@ -1975,7 +1976,8 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,if (subdir_state > dir_state) dir_state = subdir_state; -if (!match_pathspec (istate, pathspec, path.buf, path.len, +if (pathspec && + !match_pathspec (istate, pathspec, path.buf, path.len,0 /* prefix */ ,NULL ,0 /* do NOT special case dirs */ )) state = path_none;
t/t0050-filesystem.sh +21 -0
@@ -131,4 +131,25 @@ $test_unicode 'merge (silent unicode normalization)' ' git merge topic ' +test_expect_success CASE_INSENSITIVE_FS 'checkout with no pathspec and a case insensitive fs' ' + git init repo && + ( + cd repo && + + >Gitweb && + git add Gitweb && + git commit -m "add Gitweb" && + + git checkout --orphan todo && + git reset --hard && + mkdir -p gitweb/subdir && + >gitweb/subdir/file && + # it is not strictly necessary to add and commit the + git add gitweb && + git commit -m "add gitweb/subdir/file" && + + git checkout master + ) +' + test_done
Signature · 1 line
2.23.0.25.g3f4444bfd7.dirty
Re: [PATCH v2] dir: special case check for the possibility that pathspec is NULL
Hi Elijah,
On Tue, Oct 01, 2019 at 11:30:05AM -0700, Elijah Newren wrote:
[...]
Quoted 38 lines
dir.c +5
-3@@ -1962,8 +1962,9 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir, ((state == path_untracked) && (get_dtype(cdir.de, istate, path.buf, path.len) == DT_DIR) && ((dir->flags & DIR_SHOW_IGNORED_TOO) || - do_match_pathspec(istate, pathspec, path.buf, path.len, - baselen, NULL, DO_MATCH_LEADING_PATHSPEC) == MATCHED_RECURSIVELY_LEADING_PATHSPEC))) { + (pathspec && + do_match_pathspec(istate, pathspec, path.buf, path.len, + baselen, NULL, DO_MATCH_LEADING_PATHSPEC) == MATCHED_RECURSIVELY_LEADING_PATHSPEC)))) {struct untracked_cache_dir *ud; ud = lookup_untracked(dir->untracked, untracked, path.buf + baselen,@@ -1975,7 +1976,8 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,if (subdir_state > dir_state) dir_state = subdir_state; -if (!match_pathspec (istate, pathspec, path.buf, path.len, +if (pathspec && + !match_pathspec (istate, pathspec, path.buf, path.len,0 /* prefix */ ,NULL ,0 /* do NOT special case dirs */ )) state = path_none;t/t0050-filesystem.sh +4
-0@@ -131,4 +131,25 @@ $test_unicode 'merge (silent unicode normalization)' ' git merge topic ' +test_expect_success CASE_INSENSITIVE_FS 'checkout with no pathspec and a case insensitive fs' ' + git init repo && + ( + cd repo &&
t/t0050-filesystem.sh $test_unicode 'merge (silent unicode normalization)' ' · line 147
Quoted 10 lines
t/t0050-filesystem.sh +10
-0+ + >Gitweb && + git add Gitweb && + git commit -m "add Gitweb" && + + git checkout --orphan todo && + git reset --hard && + mkdir -p gitweb/subdir && + >gitweb/subdir/file && + # it is not strictly necessary to add and commit the
Probably not worth a reroll but we're missing "gitweb directory" at the end of the comment. Other than that, it looks good to me.
Thanks again for the prompt fix,
Denton
Quoted 11 lines
t/t0050-filesystem.sh +7
-0+ git add gitweb && + git commit -m "add gitweb/subdir/file" && + + git checkout master + ) +' + test_doneSignature · 1 line
2.23.0.25.g3f4444bfd7.dirty
Re: [PATCH v2] dir: special case check for the possibility that pathspec is NULL
Quoted 50 lines · Denton Liu
Hi Elijah,
On Tue, Oct 01, 2019 at 11:30:05AM -0700, Elijah Newren wrote:
[...]
Quoted 43 lines
dir.c +5
-3@@ -1962,8 +1962,9 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir, ((state == path_untracked) && (get_dtype (cdir.de, istate, path.buf, path.len) ==DT_DIR ) && ((dir->flags &DIR_SHOW_IGNORED_TOO ) || - do_match_pathspec(istate, pathspec, path.buf, path.len, - baselen, NULL, DO_MATCH_LEADING_PATHSPEC) == MATCHED_RECURSIVELY_LEADING_PATHSPEC))) { + (pathspec && +do_match_pathspec (istate, pathspec, path.buf, path.len, + baselen, NULL, DO_MATCH_LEADING_PATHSPEC) == MATCHED_RECURSIVELY_LEADING_PATHSPEC)))) { struct untracked_cache_dir *ud; ud = lookup_untracked(dir->untracked, untracked, path.buf + baselen,@@ -1975,7 +1976,8 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,if (subdir_state > dir_state) dir_state = subdir_state; -if (!match_pathspec (istate, pathspec, path.buf, path.len, +if (pathspec && + !match_pathspec (istate, pathspec, path.buf, path.len,0 /* prefix */ ,NULL ,0 /* do NOT special case dirs */ )) state = path_none;t/t0050-filesystem.sh +9
-0@@ -131,4 +131,25 @@ $test_unicode 'merge (silent unicode normalization)' ' git merge topic ' +test_expect_success CASE_INSENSITIVE_FS 'checkout with no pathspec and a case insensitive fs' ' + git init repo && + ( + cd repo && + + >Gitweb && + git add Gitweb && + git commit -m "add Gitweb" && +
Quoted 8 lines
t/t0050-filesystem.sh $test_unicode 'merge (silent unicode normalization)' ' · line 147
Quoted 5 lines
t/t0050-filesystem.sh +5
-0+ git checkout --orphan todo && + git reset --hard && + mkdir -p gitweb/subdir && + >gitweb/subdir/file && + # it is not strictly necessary to add and commit theProbably not worth a reroll but we're missing "gitweb directory" at the end of the comment. Other than that, it looks good to me.
Yuck, I accidentally only removed half the comment when I intended to remove it all? Whoops. I think it's worth a reroll; it's just a single patch. I'll send it out.
PATCH v3 1/1 dir: special case check for the possibility that pathspec is NULL
In reply to Denton LiuCommits 404ebceda01c ("dir: also check directories for matching pathspecs", 2019-09-17) and 89a1f4aaf765 ("dir: if our pathspec might match files under a dir, recurse into it", 2019-09-17) added calls to match_pathspec() and do_match_pathspec() passing along their pathspec parameter. Both match_pathspec() and do_match_pathspec() assume the pathspec argument they are given is non-NULL. It turns out that unpack-tree.c's verify_clean_subdirectory() calls read_directory() with pathspec == NULL, and it is possible on case insensitive filesystems for that NULL to make it to these new calls to match_pathspec() and do_match_pathspec(). Add appropriate checks on the NULLness of pathspec to avoid a segfault.
In case the negation throws anyone off (one of the calls was to do_match_pathspec() while the other was to !match_pathspec(), yet no negation of the NULLness of pathspec is used), there are two ways to understand the differences:
* The code already handled the pathspec == NULL cases before this series, and this series only tried to change behavior when there was a pathspec, thus we only want to go into the if-block if pathspec is non-NULL.
* One of the calls is for whether to recurse into a subdirectory, the other is for after we've recursed into it for whether we want to remove the subdirectory itself (i.e. the subdirectory didn't match but something under it could have). That difference in situation leads to the slight differences in logic used (well, that and the slightly unusual fact that we don't want empty pathspecs to remove untracked directories by default).
Denton found and analyzed one issue and provided the patch for the match_pathspec() call, SZEDER figured out why the issue only reproduced for some folks and not others and provided the testcase, and I looked through the remainder of the series and noted the do_match_pathspec() call that should have the same check.
---
Note: Applies on top of en/clean-nested-with-ignored, in next.
As with v1, the authorship is really mixed, so I don't know if I should use Co-authored-by (highlighted as a possibility by Denton), or the far more common Helped-by (as suggested by Junio but based on a more limited summary of the different contributions), or if perhaps Denton or SZEDER should be marked as the author and I be marked as Helped-by or Co-authored-by. Since Denton commented on round 1, I used his suggestion for attribution in this round, but I'm open to changing it to whatever works best.
Changes since v2:
- This time actually removed the entire unnecessary comment
Range-diff:
1: c495b9303c ! 1: 40392c6bba dir: special case check for the possibility that pathspec is NULL
@@ t/t0050-filesystem.sh: $test_unicode 'merge (silent unicode normalization)' '
+ git reset --hard &&
+ mkdir -p gitweb/subdir &&
+ >gitweb/subdir/file &&
-+ # it is not strictly necessary to add and commit the
+ git add gitweb &&
+ git commit -m "add gitweb/subdir/file" &&
+dir.c | 8 +++++--- t/t0050-filesystem.sh | 21 +++++++++++++++++++++ 2 files changed, 26 insertions(+), 3 deletions(-)
dir.c +5 -3
@@ -1962,8 +1962,9 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir, ((state == path_untracked) && (get_dtype(cdir.de, istate, path.buf, path.len) == DT_DIR) && ((dir->flags & DIR_SHOW_IGNORED_TOO) || - do_match_pathspec(istate, pathspec, path.buf, path.len, - baselen, NULL, DO_MATCH_LEADING_PATHSPEC) == MATCHED_RECURSIVELY_LEADING_PATHSPEC))) { + (pathspec && + do_match_pathspec(istate, pathspec, path.buf, path.len, + baselen, NULL, DO_MATCH_LEADING_PATHSPEC) == MATCHED_RECURSIVELY_LEADING_PATHSPEC)))) {struct untracked_cache_dir *ud; ud = lookup_untracked(dir->untracked, untracked, path.buf + baselen,
@@ -1975,7 +1976,8 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,if (subdir_state > dir_state) dir_state = subdir_state; -if (!match_pathspec (istate, pathspec, path.buf, path.len, +if (pathspec && + !match_pathspec (istate, pathspec, path.buf, path.len,0 /* prefix */ ,NULL ,0 /* do NOT special case dirs */ )) state = path_none;
t/t0050-filesystem.sh +20 -0
@@ -131,4 +131,25 @@ $test_unicode 'merge (silent unicode normalization)' ' git merge topic ' +test_expect_success CASE_INSENSITIVE_FS 'checkout with no pathspec and a case insensitive fs' ' + git init repo && + ( + cd repo && + + >Gitweb && + git add Gitweb && + git commit -m "add Gitweb" && + + git checkout --orphan todo && + git reset --hard && + mkdir -p gitweb/subdir && + >gitweb/subdir/file && + git add gitweb && + git commit -m "add gitweb/subdir/file" && + + git checkout master + ) +' + test_done
Signature · 1 line
2.23.0.25.g3f4444bfd7.dirty
Re: [PATCH v3] dir: special case check for the possibility that pathspec is NULL
Hi Elijah,
Sorry for dragging out this thread for so long...
On Tue, Oct 01, 2019 at 11:55:24AM -0700, Elijah Newren wrote:
[...]
t/t0050-filesystem.sh $test_unicode 'merge (silent unicode normalization)' '
Quoted 5 lines
t/t0050-filesystem.sh +0
-0@@ -131,4 +131,25 @@ $test_unicode 'merge (silent unicode normalization)' '
I had to change the 25 to a 24 for this to apply cleanly.
Quoted 14 lines
t/t0050-filesystem.sh +11
-0git merge topic ' +test_expect_success CASE_INSENSITIVE_FS 'checkout with no pathspec and a case insensitive fs' ' + git init repo && + ( + cd repo && + + >Gitweb && + git add Gitweb && + git commit -m "add Gitweb" && + + git checkout --orphan todo && + git reset --hard &&
t/t0050-filesystem.sh $test_unicode 'merge (silent unicode normalization)' ' · line 154
Quoted 10 lines
t/t0050-filesystem.sh +9
-0+ mkdir -p gitweb/subdir && + >gitweb/subdir/file && + git add gitweb && + git commit -m "add gitweb/subdir/file" && + + git checkout master + ) +' + test_done
Just wondering, how did you generate this patch? Did you manually edit the last patch and resend it or is this a bug in our diff machinery?
(Side note, I _hate_ how bad the feedback for git apply/am is. We should probably give more information than "error: corrupt patch at line 62" such as why patches are corrupt (unexpected characters, too many/few lines, something else?).)
Quoted 3 lines
Signature · 1 line
2.23.0.25.g3f4444bfd7.dirty
Re: [PATCH v3] dir: special case check for the possibility that pathspec is NULL
Quoted 36 lines · Denton Liu
Hi Elijah,
Sorry for dragging out this thread for so long...
On Tue, Oct 01, 2019 at 11:55:24AM -0700, Elijah Newren wrote:
[...]
t/t0050-filesystem.sh $test_unicode 'merge (silent unicode normalization)' '
Quoted 5 lines
t/t0050-filesystem.sh +0
-0@@ -131,4 +131,25 @@ $test_unicode 'merge (silent unicode normalization)' 'I had to change the 25 to a 24 for this to apply cleanly.
Quoted 19 lines
t/t0050-filesystem.sh +16
-0git merge topic ' +test_expect_success CASE_INSENSITIVE_FS 'checkout with no pathspec and a case insensitive fs' ' + git init repo && + ( + cd repo && + + >Gitweb && + git add Gitweb && + git commit -m "add Gitweb" && + + git checkout --orphan todo && + git reset --hard && + mkdir -p gitweb/subdir && + >gitweb/subdir/file && + git add gitweb && + git commit -m "add gitweb/subdir/file" && +
Quoted 8 lines
t/t0050-filesystem.sh $test_unicode 'merge (silent unicode normalization)' ' · line 154
Quoted 5 lines
t/t0050-filesystem.sh +4
-0+ git checkout master + ) +' + test_doneJust wondering, how did you generate this patch? Did you manually edit the last patch and resend it or is this a bug in our diff machinery?
I manually edited because it "was so simple" and of course just compounded the problem because I didn't fix the count, as you pointed out. Gah. Thanks for checking. Clearly, I'm bouncing between too many things this morning, and need to wait until I'm not so distracted and rushing so I don't mess things up. I'll sound out a v4 in a few hours when I've cleaned a few other things off my plate.
Re: [PATCH v3] dir: special case check for the possibility that pathspec is NULL
Quoted 47 lines · Elijah Newren
Quoted 36 lines · Denton Liu
Hi Elijah,
Sorry for dragging out this thread for so long...
On Tue, Oct 01, 2019 at 11:55:24AM -0700, Elijah Newren wrote:
[...]
t/t0050-filesystem.sh $test_unicode 'merge (silent unicode normalization)' '
Quoted 5 lines
t/t0050-filesystem.sh +0
-0@@ -131,4 +131,25 @@ $test_unicode 'merge (silent unicode normalization)' 'I had to change the 25 to a 24 for this to apply cleanly.
Quoted 19 lines
t/t0050-filesystem.sh +16
-0git merge topic ' +test_expect_success CASE_INSENSITIVE_FS 'checkout with no pathspec and a case insensitive fs' ' + git init repo && + ( + cd repo && + + >Gitweb && + git add Gitweb && + git commit -m "add Gitweb" && + + git checkout --orphan todo && + git reset --hard && + mkdir -p gitweb/subdir && + >gitweb/subdir/file && + git add gitweb && + git commit -m "add gitweb/subdir/file" && +Quoted 8 lines
t/t0050-filesystem.sh $test_unicode 'merge (silent unicode normalization)' ' · line 154
Quoted 5 lines
t/t0050-filesystem.sh +4
-0+ git checkout master + ) +' + test_doneJust wondering, how did you generate this patch? Did you manually edit the last patch and resend it or is this a bug in our diff machinery?
Quoted 6 lines
I manually edited because it "was so simple" and of course just compounded the problem because I didn't fix the count, as you pointed out. Gah. Thanks for checking. Clearly, I'm bouncing between too many things this morning, and need to wait until I'm not so distracted and rushing so I don't mess things up. I'll sound out a v4 in a few hours when I've cleaned a few other things off my plate.
I was going to send out a new version this morning, but it looks like Junio already picked up the patch and fixed it up (the tip of en/clean-nested-with-ignored already has what we want), so I won't resend after all. Thanks Denton, SZEDER, and Junio.
Re: [PATCH v3] dir: special case check for the possibility that pathspec is NULL
In reply to Elijah NewrenQuoted 110 lines · Elijah Newren
Commits
404ebceda01c("dir: also check directories for matching pathspecs", 2019-09-17) and89a1f4aaf765("dir: if our pathspec might match files under a dir, recurse into it", 2019-09-17) added calls tomatch_pathspec()anddo_match_pathspec()passing along their pathspec parameter. Bothmatch_pathspec()anddo_match_pathspec()assume the pathspec argument they are given is non-NULL. It turns out that unpack-tree.c'sverify_clean_subdirectory()callsread_directory()with pathspec == NULL, and it is possible on case insensitive filesystems for that NULL to make it to these new calls tomatch_pathspec()anddo_match_pathspec(). Add appropriate checks on the NULLness of pathspec to avoid a segfault.In case the negation throws anyone off (one of the calls was to
do_match_pathspec()while the other was to !match_pathspec(), yet no negation of the NULLness of pathspec is used), there are two ways to understand the differences:
* The code already handled the pathspec == NULL cases before this series, and this series only tried to change behavior when there was a pathspec, thus we only want to go into the if-block if pathspec is non-NULL.
* One of the calls is for whether to recurse into a subdirectory, the other is for after we've recursed into it for whether we want to remove the subdirectory itself (i.e. the subdirectory didn't match but something under it could have). That difference in situation leads to the slight differences in logic used (well, that and the slightly unusual fact that we don't want empty pathspecs to remove untracked directories by default).Denton found and analyzed one issue and provided the patch for the
match_pathspec()call, SZEDER figured out why the issue only reproduced for some folks and not others and provided the testcase, and I looked through the remainder of the series and noted thedo_match_pathspec()call that should have the same check.Co-authored-by Denton Liu <[email protected]>Co-authored-by SZEDER Gábor <[email protected]>Signed-off-by Elijah Newren <[email protected]>---
Note: Applies on top of en/clean-nested-with-ignored, in next.As with v1, the authorship is really mixed, so I don't know if I should use Co-authored-by (highlighted as a possibility by Denton), or the far more common Helped-by (as suggested by Junio but based on a more limited summary of the different contributions), or if perhaps Denton or SZEDER should be marked as the author and I be marked as Helped-by or Co-authored-by. Since Denton commented on round 1, I used his suggestion for attribution in this round, but I'm open to changing it to whatever works best.
Changes since v2:
- This time actually removed the entire unnecessary commentRange-diff:
1:c495b9303c! 1:40392c6bbadir: special case check for the possibility that pathspec is NULL@@ t/t0050-filesystem.sh: $test_unicode 'merge (silent unicode normalization)' ' + git reset --hard && + mkdir -p gitweb/subdir && + >gitweb/subdir/file && -+ # it is not strictly necessary to add and commit the + git add gitweb && + git commit -m "add gitweb/subdir/file" && +dir.c | 8 +++++--- t/t0050-filesystem.sh | 21 +++++++++++++++++++++ 2 files changed, 26 insertions(+), 3 deletions(-)dir.c +5
-3@@ -1962,8 +1962,9 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir, ((state == path_untracked) && (get_dtype(cdir.de, istate, path.buf, path.len) == DT_DIR) && ((dir->flags & DIR_SHOW_IGNORED_TOO) || - do_match_pathspec(istate, pathspec, path.buf, path.len, - baselen, NULL, DO_MATCH_LEADING_PATHSPEC) == MATCHED_RECURSIVELY_LEADING_PATHSPEC))) { + (pathspec && + do_match_pathspec(istate, pathspec, path.buf, path.len, + baselen, NULL, DO_MATCH_LEADING_PATHSPEC) == MATCHED_RECURSIVELY_LEADING_PATHSPEC)))) {struct untracked_cache_dir *ud; ud = lookup_untracked(dir->untracked, untracked, path.buf + baselen,@@ -1975,7 +1976,8 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,if (subdir_state > dir_state) dir_state = subdir_state; -if (!match_pathspec (istate, pathspec, path.buf, path.len, +if (pathspec && + !match_pathspec (istate, pathspec, path.buf, path.len,0 /* prefix */ ,NULL ,0 /* do NOT special case dirs */ )) state = path_none;t/t0050-filesystem.sh +9
-0@@ -131,4 +131,25 @@ $test_unicode 'merge (silent unicode normalization)' ' git merge topic ' +test_expect_success CASE_INSENSITIVE_FS 'checkout with no pathspec and a case insensitive fs' ' + git init repo && + ( + cd repo && + + >Gitweb && + git add Gitweb && + git commit -m "add Gitweb" && +
t/t0050-filesystem.sh $test_unicode 'merge (silent unicode normalization)' ' · line 152
Quoted 10 lines
t/t0050-filesystem.sh +10
-0+ git checkout --orphan todo && + git reset --hard && + mkdir -p gitweb/subdir && + >gitweb/subdir/file && + git add gitweb && + git commit -m "add gitweb/subdir/file" && + + git checkout master + ) +'
I don't like this test ;)
I only intended it as a "here is how to reliably reproduce the segfault without all the clutter of the full git.git repository" that I wrote way past my bedtime. But I think that:
- it shouldn't have the CASE_INSENSITIVE_FS prereq. Yes, that segfault could only be triggered on a case insensitive filesystem, but the given sequence of commands should succeed in a case sensitive file system just as well.
(Have no idea why I added that prereq in the first place; as I said above, it was way past my bedtime...)
- it's in the wrong test script; it would be better among other tests checking what 'git checkout' should or must not overwrite when switching branches, but not sure which test script that is.
(I think I added it to this test script, because it stood out a bit when grepping for case insensitive fs in the test suite; I play the "past my bedtime" card again :)
- it's already satisfied by 'git checkout master' not failing, but it doesn't check whether the resulting contents of the worktree are as expected.
- it still bothers me why that additional subdir was necessary to trigger the segfault. Did you look into it?
PATCH v4 5/12 dir: make the DO_MATCH_SUBMODULE code reusable for a non-submodule case
In reply to Elijah NewrenThe specific checks done in match_pathspec_item for the DO_MATCH_SUBMODULE case are useful for other cases which have nothing to do with submodules. Rename this constant; a subsequent commit will make use of this change.
--- dir.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
dir.c +3 -3
@@ -273,7 +273,7 @@ static int do_read_blob(const struct object_id *oid, struct oid_stat *oid_stat,#define DO_MATCH_EXCLUDE (1<<0)#define DO_MATCH_DIRECTORY (1<<1) -#define DO_MATCH_SUBMODULE (1<<2) +#define DO_MATCH_LEADING_PATHSPEC (1<<2)/* * Does 'match' match the given name?
@@ -354,7 +354,7 @@ static int match_pathspec_item(const struct index_state *istate,return MATCHED_FNMATCH ;/* Perform checks to see if "name" is a super set of the pathspec */ -if (flags &DO_MATCH_SUBMODULE ) { +if (flags &DO_MATCH_LEADING_PATHSPEC ) {/* name is a literal prefix of the pathspec */ int offset = name[namelen-1 ] =='/' ?1 :0 ; if ((namelen < matchlen) &&
@@ -498,7 +498,7 @@ int submodule_path_match(const struct index_state *istate,strlen (submodule_name),0 , seen,DO_MATCH_DIRECTORY | - DO_MATCH_SUBMODULE); + DO_MATCH_LEADING_PATHSPEC);return matched; }
Signature · 1 line
2.22.1.17.g6e632477f7.dirty
PATCH v4 6/12 dir: if our pathspec might match files under a dir, recurse into it
In reply to Elijah NewrenFor git clean, if a directory is entirely untracked and the user did not specify -d (corresponding to DIR_SHOW_IGNORED_TOO), then we usually do not want to remove that directory and thus do not recurse into it. However, if the user manually specified specific (or even globbed) paths somewhere under that directory to remove, then we need to recurse into the directory to make sure we remove the relevant paths under that directory as the user requested.
Note that this does not mean that the recursed-into directory will be added to dir->entries for later removal; as of a few commits earlier in this series, there is another more strict match check that is run after returning from a recursed-into directory before deciding to add it to the list of entries. Therefore, this will only result in files underneath the given directory which match one of the pathspecs being added to the entries list.
Two notes of potential interest to future readers:
* If we wanted to only recurse into a directory when it is specifically matched rather than matched-via-glob (e.g. '*.c'), then we could do so via making the final non-zero return in match_pathspec_item be MATCHED_RECURSIVELY instead of MATCHED_RECURSIVELY_LEADING_PATHSPEC. (Note that the relative order of MATCHED_RECURSIVELY_LEADING_PATHSPEC and MATCHED_RECURSIVELY are important for such a change.) I was leaving open that possibility while writing an RFC asking for the behavior we want, but even though we don't want it, that knowledge might help you understand the code flow better.
* There is a growing amount of logic in read_directory_recursive() for deciding whether to recurse into a subdirectory. However, there is a comment immediately preceding this logic that says to recurse if
instructed by treat_path(). It may be better for the logic in
read_directory_recursive() to ultimately be moved to treat_path() (or another function it calls, such as treat_directory()), but I have left that for someone else to tackle in the future.
--- dir.c | 10 ++++++---- dir.h | 5 +++-- t/t7300-clean.sh | 4 ++-- 3 files changed, 11 insertions(+), 8 deletions(-)
dir.c +6 -4
@@ -360,7 +360,7 @@ static int match_pathspec_item(const struct index_state *istate,if ((namelen < matchlen) && (match[namelen-offset] =='/' ) && !ps_strncmp (item, match, name, namelen)) -return MATCHED_RECURSIVELY ; +return MATCHED_RECURSIVELY_LEADING_PATHSPEC ;/* name" doesn't match up to the first wild character */ if (item->nowildcard_len < item->len &&
@@ -377,7 +377,7 @@ static int match_pathspec_item(const struct index_state *istate,* The submodules themselves will be able to perform more * accurate matching to determine if the pathspec matches. */ -return MATCHED_RECURSIVELY ; +return MATCHED_RECURSIVELY_LEADING_PATHSPEC ; }return 0 ;
@@ -1939,8 +1939,10 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,/* recurse into subdir if instructed by treat_path */ if ((state == path_recurse) || ((state == path_untracked) && - (dir->flags &DIR_SHOW_IGNORED_TOO ) && - (get_dtype (cdir.de, istate, path.buf, path.len) ==DT_DIR ))) { + (get_dtype (cdir.de, istate, path.buf, path.len) ==DT_DIR ) && + ((dir->flags &DIR_SHOW_IGNORED_TOO ) || +do_match_pathspec (istate, pathspec, path.buf, path.len, + baselen,NULL ,DO_MATCH_LEADING_PATHSPEC ) ==MATCHED_RECURSIVELY_LEADING_PATHSPEC ))) {struct untracked_cache_dir *ud; ud = lookup_untracked(dir->untracked, untracked, path.buf + baselen,
dir.h +3 -2
@@ -211,8 +211,9 @@ int count_slashes(const char *s);* when populating the seen[] array. */ #define MATCHED_RECURSIVELY 1 -#define MATCHED_FNMATCH 2 -#define MATCHED_EXACTLY 3 +#define MATCHED_RECURSIVELY_LEADING_PATHSPEC 2 +#define MATCHED_FNMATCH 3 +#define MATCHED_EXACTLY 4int simple_length (const char *match);int no_wildcard (const char *string);char *common_prefix (const struct pathspec *pathspec);
t/t7300-clean.sh +2 -2
@@ -691,7 +691,7 @@ test_expect_failure 'git clean -d skips nested repo containing ignored files' ' test_path_is_file nested-repo-with-ignored-file/file ' -test_expect_failure 'git clean handles being told what to clean' ' +test_expect_success 'git clean handles being told what to clean' ' mkdir -p d1 d2 && touch d1/ut d2/ut && git clean -f */ut &&
@@ -707,7 +707,7 @@ test_expect_success 'git clean handles being told what to clean, with -d' ' test_path_is_missing d2/ut ' -test_expect_failure 'git clean works if a glob is passed without -d' ' +test_expect_success 'git clean works if a glob is passed without -d' ' mkdir -p d1 d2 && touch d1/ut d2/ut && git clean -f "*ut" &&
Signature · 1 line
2.22.1.17.g6e632477f7.dirty
PATCH v4 7/12 dir: add commentary explaining match_pathspec_item's return value
In reply to Elijah NewrenThe way match_pathspec_item() handles names and pathspecs with trailing slash characters, in conjunction with special options like DO_MATCH_DIRECTORY and DO_MATCH_LEADING_PATHSPEC were non-obvious, and broken until this patch series. Add a table in a comment explaining the intent of how these work.
--- dir.c | 27 +++++++++++++++++++-------- 1 file changed, 19 insertions(+), 8 deletions(-)
dir.c +19 -8
@@ -276,16 +276,27 @@ static int do_read_blob(const struct object_id *oid, struct oid_stat *oid_stat,#define DO_MATCH_LEADING_PATHSPEC (1<<2)/* -* Does 'match' match the given name? -* A match is found if +* Does the given pathspec match the given name? A match is found if * -* (1) the 'match' string is leading directory of 'name', or -* (2) the 'match' string is a wildcard and matches 'name', or -* (3) the 'match' string is exactly the same as 'name'. +* (1) the pathspec string is leading directory of 'name' ("RECURSIVELY"), or +* (2) the pathspec string has a leading part matching 'name' ("LEADING"), or +* (3) the pathspec string is a wildcard and matches 'name' ("WILDCARD"), or +* (4) the pathspec string is exactly the same as 'name' ("EXACT"). * -* and the return value tells which case it was. +* Return value tells which case it was (1-4), or 0 when there is no match. * -* It returns 0 when there is no match. +* It may be instructive to look at a small table of concrete examples +* to understand the differences between 1, 2, and 4: +* +* Pathspecs +* | a/b | a/b/ | a/b/c +* ------+-----------+-----------+------------ +* a/b | EXACT | EXACT[1] | LEADING[2] +* Names a/b/ | RECURSIVE | EXACT | LEADING[2] +* a/b/c | RECURSIVE | RECURSIVE | EXACT +* +* [1] Only if DO_MATCH_DIRECTORY is passed; otherwise, this is NOT a match. +* [2] Only if DO_MATCH_LEADING_PATHSPEC is passed; otherwise, not a match. */ static int match_pathspec_item(const struct index_state *istate, const struct pathspec_item *item, int prefix,
@@ -353,7 +364,7 @@ static int match_pathspec_item(const struct index_state *istate, item->nowildcard_len - prefix))return MATCHED_FNMATCH ; -/* Perform checks to see if "name" is a super set of the pathspec */ +/* Perform checks to see if "name" is a leading string of the pathspec */ if (flags &DO_MATCH_LEADING_PATHSPEC ) {/* name is a literal prefix of the pathspec */ int offset = name[namelen-1 ] =='/' ?1 :0 ;
Signature · 1 line
2.22.1.17.g6e632477f7.dirty
PATCH v4 8/12 git-clean.txt: do not claim we will delete files with -n/--dry-run
In reply to Elijah NewrenIt appears that the wrong option got included in the list of what will cause git-clean to actually take action. Correct the list.
--- Documentation/git-clean.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Documentation/git-clean.txt +1 -1
@@ -35,7 +35,7 @@ OPTIONS --force:: If the Git configuration variable clean.requireForce is not set to false, 'git clean' will refuse to delete files or directories - unless given -f, -n or -i. Git will refuse to delete directories + unless given -f or -i. Git will refuse to delete directories with .git sub directory or file unless a second -f is given.
Signature · 1 line
2.22.1.17.g6e632477f7.dirty
PATCH v4 9/12 clean: disambiguate the definition of -d
In reply to Elijah NewrenThe -d flag pre-dated git-clean's ability to have paths specified. As such, the default for git-clean was to only remove untracked files in the current directory, and -d existed to allow it to recurse into subdirectories.
The interaction of paths and the -d option appears to not have been carefully considered, as evidenced by numerous bugs and a dearth of tests covering such pairings in the testsuite. The definition turns out to be important, so let's look at some of the various ways one could interpret the -d option:
A) Without -d, only look in subdirectories which contain tracked files under them; with -d, also look in subdirectories which are untracked for files to clean.
B) Without specified paths from the user for us to delete, we need to have some kind of default, so...without -d, only look in subdirectories which contain tracked files under them; with -d, also look in subdirectories which are untracked for files to clean.
The important distinction here is that choice B says that the presence or absence of '-d' is irrelevant if paths are specified. The logic behind option B is that if a user explicitly asked us to clean a specified pathspec, then we should clean anything that matches that pathspec. Some examples may clarify. Should
git clean -f untracked_dir/file
remove untracked_dir/file or not? It seems crazy not to, but a strict reading of option A says it shouldn't be removed. How about
git clean -f untracked_dir/file1 tracked_dir/file2
or
git clean -f untracked_dir_1/file1 untracked_dir_2/file2
? Should it remove either or both of these files? Should it require multiple runs to remove both the files listed? (If this sounds like a crazy question to even ask, see the commit message of "t7300: Add some testcases showing failure to clean specified pathspecs" added earlier in this patch series.) What if -ffd were used instead of -f -- should that allow these to be removed? Should it take multiple invocations with -ffd? What if a glob (such as '*tracked*') were used instead of spelling out the directory names? What if the filenames involved globs, such as
git clean -f '*.o'
or
git clean -f '*/*.o'
?
The current documentation actually suggests a definition that is slightly different than choice A, and the implementation prior to this series provided something radically different than either choices A or B. (The implementation, though, was clearly just buggy). There may be other choices as well. However, for almost any given choice of definition for -d that I can think of, some of the examples above will appear buggy to the user. The only case that doesn't have negative surprises is choice B: treat a user-specified path as a request to clean all untracked files which match that path specification, including recursing into any untracked directories.
Change the documentation and basic implementation to use this definition.
There were two regression tests that indirectly depended on the current implementation, but neither was about subdirectory handling. These two tests were introduced in commit 5b7570cfb41c ("git-clean: add tests for relative path", 2008-03-07) which was solely created to add coverage for the changes in commit fb328947c8e ("git-clean: correct printing relative path", 2008-03-07). Both tests specified a directory that happened to have an untracked subdirectory, but both were only checking that the resulting printout of a file that was removed was shown with a relative path. Update these tests appropriately.
--- Documentation/git-clean.txt | 10 ++++++---- builtin/clean.c | 8 ++++++++ t/t7300-clean.sh | 2 ++ 3 files changed, 16 insertions(+), 4 deletions(-)
Documentation/git-clean.txt +6 -4
@@ -26,10 +26,12 @@ are affected. OPTIONS ------- -d:: - Remove untracked directories in addition to untracked files. - If an untracked directory is managed by a different Git - repository, it is not removed by default. Use -f option twice - if you really want to remove such a directory. + Normally, when no <path> is specified, git clean will not + recurse into untracked directories to avoid removing too much. + Specify -d to have it recurse into such directories as well. + If any paths are specified, -d is irrelevant; all untracked + files matching the specified paths (with exceptions for nested + git directories mentioned under `--force`) will be removed. -f:: --force::
builtin/clean.c +8 -0
@@ -949,6 +949,14 @@ int cmd_clean(int argc, const char **argv, const char *prefix) dir.flags |=DIR_SHOW_OTHER_DIRECTORIES ; +if (argc) { +/* +* Remaining args implies pathspecs specified, and we should +* recurse within those. +*/ + remove_directories =1 ; + } +if (remove_directories) dir.flags |=DIR_SHOW_IGNORED_TOO |DIR_KEEP_UNTRACKED_CONTENTS ;
t/t7300-clean.sh +2 -0
@@ -117,6 +117,7 @@ test_expect_success C_LOCALE_OUTPUT 'git clean with relative prefix' ' would_clean=$( cd docs && git clean -n ../src | + grep part3 | sed -n -e "s|^Would remove ||p" ) && verbose test "$would_clean" = ../src/part3.c
@@ -129,6 +130,7 @@ test_expect_success C_LOCALE_OUTPUT 'git clean with absolute path' ' would_clean=$( cd docs && git clean -n "$(pwd)/../src" | + grep part3 | sed -n -e "s|^Would remove ||p" ) && verbose test "$would_clean" = ../src/part3.c
Signature · 1 line
2.22.1.17.g6e632477f7.dirty
PATCH v4 10/12 clean: avoid removing untracked files in a nested git repository
In reply to Elijah NewrenUsers expect files in a nested git repository to be left alone unless sufficiently forced (with two -f's). Unfortunately, in certain circumstances, git would delete both tracked (and possibly dirty) files and untracked files within a nested repository. To explain how this happens, let's contrast a couple cases. First, take the following example setup (which assumes we are already within a git repo):
git init nested
cd nested
Quoted 1 line
tracked
git add tracked
git commit -m init
Quoted 1 line
untracked
cd ..
In this setup, everything works as expected; running 'git clean -fd' will result in fill_directory() returning the following paths:
nested/
nested/tracked
nested/untracked
and then correct_untracked_entries() would notice this can be compressed to
nested/
and then since "nested/" is a directory, we would call remove_dirs("nested/", ...), which would
check is_nonbare_repository_dir() and then decide to skip it.
However, if someone also creates an ignored file:
Quoted 1 line
nested/ignored
then running 'git clean -fd' would result in fill_directory() returning the same paths:
nested/
nested/tracked
nested/untracked
but correct_untracked_entries() will notice that we had ignored entries under nested/ and thus simplify this list to
nested/tracked
nested/untracked
Since these are not directories, we do not call remove_dirs() which was the only place that had the is_nonbare_repository_dir() safety check -- resulting in us deleting both the untracked file and the tracked (and possibly dirty) file.
One possible fix for this issue would be walking the parent directories of each path and checking if they represent nonbare repositories, but that would be wasteful. Even if we added caching of some sort, it's still a waste because we should have been able to check that "nested/" represented a nonbare repository before even descending into it in the first place. Add a DIR_SKIP_NESTED_GIT flag to dir_struct.flags and use it to prevent fill_directory() and friends from descending into nested git repos.
With this change, we also modify two regression tests added in commit 91479b9c72f1 ("t7300: add tests to document behavior of clean and nested git", 2015-06-15). That commit, nor its series, nor the six previous iterations of that series on the mailing list discussed why those tests coded the expectation they did. In fact, it appears their purpose was simply to test _existing_ behavior to make sure that the performance changes didn't change the behavior. However, these two tests directly contradicted the manpage's claims that two -f's were required to delete files/directories under a nested git repository. While one could argue that the user gave an explicit path which matched files/directories that were within a nested repository, there's a slippery slope that becomes very difficult for users to understand once you go down that route (e.g. what if they specified "git clean -f -d '*.c'"?) It would also be hard to explain what the exact behavior was; avoid such problems by making it really simple.
Also, clean up some grammar errors describing this functionality in the git-clean manpage.
Finally, there are still a couple bugs with -ffd not cleaning out enough (e.g. missing the nested .git) and with -ffdX possibly cleaning out the wrong files (paying attention to outer .gitignore instead of inner). This patch does not address these cases at all (and does not change the behavior relative to those flags), it only fixes the handling when given a single -f. See
https://public-inbox.org/git/[email protected]/ for more discussion of the -ffd[X?] bugs.
--- Documentation/git-clean.txt | 6 +++--- builtin/clean.c | 2 ++ dir.c | 10 ++++++++++ dir.h | 3 ++- t/t7300-clean.sh | 10 +++++----- 5 files changed, 22 insertions(+), 9 deletions(-)
Documentation/git-clean.txt +3 -3
@@ -37,9 +37,9 @@ OPTIONS --force:: If the Git configuration variable clean.requireForce is not set to false, 'git clean' will refuse to delete files or directories - unless given -f or -i. Git will refuse to delete directories - with .git sub directory or file unless a second -f - is given. + unless given -f or -i. Git will refuse to modify untracked + nested git repositories (directories with a .git subdirectory) + unless a second -f is given. -i:: --interactive::
builtin/clean.c +2 -0
@@ -946,6 +946,8 @@ int cmd_clean(int argc, const char **argv, const char *prefix)if (force >1 ) rm_flags =0 ; +else + dir.flags |=DIR_SKIP_NESTED_GIT ; dir.flags |=DIR_SHOW_OTHER_DIRECTORIES ;
dir.c +10 -0
@@ -1451,6 +1451,16 @@ static enum path_treatment treat_directory(struct dir_struct *dir,return path_none;case index_nonexistent: +if (dir->flags &DIR_SKIP_NESTED_GIT ) { +int nested_repo; +struct strbuf sb =STRBUF_INIT ; +strbuf_addstr (&sb, dirname); + nested_repo =is_nonbare_repository_dir (&sb); +strbuf_release (&sb); +if (nested_repo) +return path_none; + } +if (dir->flags &DIR_SHOW_OTHER_DIRECTORIES )break ; if (exclude &&
dir.h +2 -1
@@ -156,7 +156,8 @@ struct dir_struct {DIR_SHOW_IGNORED_TOO =1 <<5 ,DIR_COLLECT_KILLED_ONLY =1 <<6 ,DIR_KEEP_UNTRACKED_CONTENTS =1 <<7 , - DIR_SHOW_IGNORED_TOO_MODE_MATCHING = 1<<8 +DIR_SHOW_IGNORED_TOO_MODE_MATCHING =1 <<8 , + DIR_SKIP_NESTED_GIT = 1<<9 } flags;struct dir_entry **entries;struct dir_entry **ignored;
t/t7300-clean.sh +5 -5
@@ -549,7 +549,7 @@ test_expect_failure 'nested (non-empty) bare repositories should be cleaned even test_path_is_missing strange_bare ' -test_expect_success 'giving path in nested git work tree will remove it' ' +test_expect_success 'giving path in nested git work tree will NOT remove it' ' rm -fr repo && mkdir repo && (
@@ -561,7 +561,7 @@ test_expect_success 'giving path in nested git work tree will remove it' ' git clean -f -d repo/bar/baz && test_path_is_file repo/.git/HEAD && test_path_is_dir repo/bar/ && - test_path_is_missing repo/bar/baz + test_path_is_file repo/bar/baz/hello.world ' test_expect_success 'giving path to nested .git will not remove it' '
@@ -579,7 +579,7 @@ test_expect_success 'giving path to nested .git will not remove it' ' test_path_is_dir untracked/ ' -test_expect_success 'giving path to nested .git/ will remove contents' ' +test_expect_success 'giving path to nested .git/ will NOT remove contents' ' rm -fr repo untracked && mkdir repo untracked && (
@@ -589,7 +589,7 @@ test_expect_success 'giving path to nested .git/ will remove contents' ' ) && git clean -f -d repo/.git/ && test_path_is_dir repo/.git && - test_dir_is_empty repo/.git && + test_path_is_file repo/.git/HEAD && test_path_is_dir untracked/ '
@@ -671,7 +671,7 @@ test_expect_success 'git clean -d skips untracked dirs containing ignored files' test_path_is_missing foo/b/bb ' -test_expect_failure 'git clean -d skips nested repo containing ignored files' ' +test_expect_success 'git clean -d skips nested repo containing ignored files' ' test_when_finished "rm -rf nested-repo-with-ignored-file" && git init nested-repo-with-ignored-file &&
Signature · 1 line
2.22.1.17.g6e632477f7.dirty
PATCH v4 11/12 clean: rewrap overly long line
In reply to Elijah Newren--- builtin/clean.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
builtin/clean.c +2 -1
@@ -158,7 +158,8 @@ static int remove_dirs(struct strbuf *path, const char *prefix, int force_flag, *dir_gone =1 ; -if ((force_flag &REMOVE_DIR_KEEP_NESTED_GIT ) &&is_nonbare_repository_dir (path)) { +if ((force_flag &REMOVE_DIR_KEEP_NESTED_GIT ) && +is_nonbare_repository_dir (path)) {if (!quiet) {quote_path_relative (path->buf, prefix, "ed); printf(dry_run ? _(msg_would_skip_git_dir) : _(msg_skip_git_dir),
Signature · 1 line
2.22.1.17.g6e632477f7.dirty
PATCH v4 12/12 clean: fix theoretical path corruption
In reply to Elijah Newrencmd_clean() had the following code structure:
struct strbuf abs_path = STRBUF_INIT;
for_each_string_list_item(item, &del_list) {
strbuf_addstr(&abs_path, prefix);
strbuf_addstr(&abs_path, item->string);
PROCESS(&abs_path);
strbuf_reset(&abs_path);
}where I've elided a bunch of unnecessary details and PROCESS(&abs_path) represents a big chunk of code rather than an actual function call. One piece of PROCESS was:
if (lstat(abs_path.buf, &st))
continue;which would cause the strbuf_reset() to be missed -- meaning that the next path to be handled would have two paths concatenated. This path used to use die_errno() instead of continue prior to commit 396049e5fb62 ("git-clean: refactor git-clean into two phases", 2013-06-25), but my understanding of how correct_untracked_entries() works is that it will prevent both dir/ and dir/file from being in the list to clean so this should be dead code and the die_errno() should be safe. But I hesitate to remove it since I am not certain.
However, we can fix both this bug and possible similar future bugs by simply moving the strbuf_reset(&abs_path) to the beginning of the loop. It'll result in N calls to strbuf_reset() instead of N-1, but that's a small price to pay to avoid sneaky bugs like this.
--- builtin/clean.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
builtin/clean.c +1 -1
@@ -1018,6 +1018,7 @@ int cmd_clean(int argc, const char **argv, const char *prefix)for_each_string_list_item (item, &del_list) {struct stat st; +strbuf_reset (&abs_path);if (prefix)strbuf_addstr (&abs_path, prefix);
@@ -1051,7 +1052,6 @@ int cmd_clean(int argc, const char **argv, const char *prefix)printf (dry_run ?_ (msg_would_remove) :_ (msg_remove), qname); } } -strbuf_reset (&abs_path); }strbuf_release (&abs_path);
Signature · 1 line
2.22.1.17.g6e632477f7.dirty