bug?: git grep HEAD with exclude in pathspec not taken into account

git init

echo foo > fileA
echo 'foo is false+' > fileB
git add fileA
git commit -m fileA

git add fileB
git commit -m fileB

git --no-pager grep foo HEAD -- ':!fileA'

git --no-pager grep foo HEAD -- ':!fileB'

git --no-pager grep foo -- ':!fileB'

git --no-pager grep foo e35df26 -- ':!fileB'

git archive --verbose master ':(top)'

/usr/local/bin/git archive --verbose master ':(top)' ':(exclude)fileA'

/usr/local/bin/git archive --verbose master ':(top)' ':(exclude)fileB'

Hi,

I observed an unexpected behavior while using git grep with both git 2.19.1 and 2.14.3. Here is how to reproduce it:

Quoted 1 line

git init

Initialized empty Git repository in /private/tmp/hello/.git/

Quoted 4 lines

echo foo > fileA
echo 'foo is false+' > fileB
git add fileA
git commit -m fileA

[master (root-commit) f2c83e7] fileA

 1 file changed, 1 insertion(+)
 create mode 100644 fileA

Quoted 2 lines

git add fileB
git commit -m fileB

[master e35df26] fileB

 1 file changed, 1 insertion(+)
 create mode 100644 fileB

Quoted 1 line

git --no-pager grep foo HEAD -- ':!fileA'

HEAD:fileB:foo is false+

Quoted 1 line

git --no-pager grep foo HEAD -- ':!fileB'

HEAD:fileA:foo
HEAD:fileB:foo is false+

In the last command, fileB appears in grep results but it should have been excluded.

If the HEAD parameter is removed, it works as expected:

git --no-pager grep foo HEAD -- ':!fileA' .

git --no-pager grep foo HEAD -- ':!fileB' .

Quoted 22 lines · Christophe Bliard

Hi,

I observed an unexpected behavior while using git grep with both git 2.19.1 and 2.14.3. Here is how to reproduce it:

Quoted 1 line

git init

Initialized empty Git repository in /private/tmp/hello/.git/

Quoted 4 lines

echo foo > fileA
echo 'foo is false+' > fileB
git add fileA
git commit -m fileA

[master (root-commit) f2c83e7] fileA

 1 file changed, 1 insertion(+)
 create mode 100644 fileA

Quoted 2 lines

git add fileB
git commit -m fileB

[master e35df26] fileB

 1 file changed, 1 insertion(+)
 create mode 100644 fileB

Quoted 1 line

git --no-pager grep foo HEAD -- ':!fileA'

HEAD:fileB:foo is false+

Quoted 8 lines

Quoted 1 line

git --no-pager grep foo HEAD -- ':!fileB'

HEAD:fileA:foo
HEAD:fileB:foo is false+

In the last command, fileB appears in grep results but it should have been excluded.

If the HEAD parameter is removed, it works as expected:

It's probably a bug. We have different code paths for matching things with or without HEAD, roughly speaking. I'll look into to this more on the weekend, unless somebody (is welcome to) beats me first.

Another thing that might also be a problem is, negative patterns are supposed to exclude stuff from "something" but you don't specify that "something" (i.e. positive patterns) in your grep commands above. If "grep foo HEAD -- :/ ':!fileB'" works, then you probably just run into some undefined corner case.
--
Duy

Hi,

I observed an unexpected behavior while using git grep with both git 2.19.1 and 2.14.3.

Rules 8 and 18 are now updated to be less eager. We conclude that the current entry is positively matched and included. But we say nothing about remaining entries. tree_entry_interesting() will be called again for those entries where we will determine entries individually.

Signed-off-by Nguyễn Thái Ngọc Duy <[email protected]>

---

t/t6132-pathspec-exclude.sh +9 -0

+               git commit -m twofiles &&
+               git grep -l file HEAD :^file2 >actual &&
+               echo HEAD:file1 >expected &&
+               test_cmp expected actual &&
+               git grep -l file HEAD :^file1 >actual &&
+               echo HEAD:file2 >expected &&
+               test_cmp expected actual
+       )
+'

t/t6132-pathspec-exclude.sh test_expect_success 'multiple exclusions' ' · line 203

Quoted 7 lines · Nguyễn Thái Ngọc Duy

t/t6132-pathspec-exclude.sh +7 -0

+test_expect_success 't_e_i() exclude case #8' '
+       git init case8 &&
+       (
+               cd case8 &&
+               echo file >file1 &&
+               echo file >file2 &&
+               git add . &&

Won't this loose git-add invocation end up adding all the junk files from earlier tests? One might have expected to see the more restricted invocation:

    git add file1 file2 &&

to make it easier to reason about the test and not worry about someone later inserting tests above this one which might interfere with it.

Quoted 9 lines · Eric Sunshine

t/t6132-pathspec-exclude.sh test_expect_success 'multiple exclusions' ' · line 203

Quoted 7 lines · Nguyễn Thái Ngọc Duy

t/t6132-pathspec-exclude.sh +7 -0

+test_expect_success 't_e_i() exclude case #8' '
+       git init case8 &&
+       (
+               cd case8 &&
+               echo file >file1 &&
+               echo file >file2 &&
+               git add . &&

Quoted 6 lines

Won't this loose git-add invocation end up adding all the junk files from earlier tests? One might have expected to see the more restricted invocation:

    git add file1 file2 &&

to make it easier to reason about the test and not worry about someone later inserting tests above this one which might interfere with it.

Upon reflection, there shouldn't be any junk files since this test is creating a new repository and "file1" and "file2" are the only files present. Apparently, I wasn't paying close enough attention when I made the earlier observation.

Anyhow, the more restricted git-add invocation you used in the re-roll is still preferable since it makes the intention obvious. Thanks.

tree_entry_interesting() is used for matching pathspec on a tree. The interesting thing about this function is that, because the tree entries are known to be sorted, this function can return more than just "yes, matched" and "no, not matched". It can also say "yes, this entry is matched and so is the remaining entries in the tree".

This is where I made a mistake when matching exclude pathspec. For exclude pathspec, we do matching twice, one with positive patterns and one with negative ones, then a rule table is applied to determine the final "include or exclude" result. Note that "matched" does not necessarily mean include. For negative patterns, "matched" means exclude.

This particular rule is too eager to include everything. Rule 8 says that "if all entries are positively matched" and the current entry is not negatively matched (i.e. not excluded), then all entries are positively matched and therefore included. But this is not true. If the _current_ entry is not negatively matched, it does not mean the next one will not be and we cannot conclude right away that all remaining entries are positively matched and can be included.

Rules 8 and 18 are now updated to be less eager. We conclude that the current entry is positively matched and included. But we say nothing about remaining entries. tree_entry_interesting() will be called again for those entries where we will determine entries individually.

Reported-by Christophe Bliard <[email protected]>

Signed-off-by Nguyễn Thái Ngọc Duy <[email protected]>

---
v2 fixes the too broad "git add ." in the test

 t/t6132-pathspec-exclude.sh | 17 +++++++++++++++++
 tree-walk.c                 | 11 ++++++++---
 2 files changed, 25 insertions(+), 3 deletions(-)

t/t6132-pathspec-exclude.sh +17 -0

index eb829fce97..2462b19ddd 100755
@@ -194,4 +194,21 @@ test_expect_success 'multiple exclusions' '
 	test_cmp expect actual
 '
 
+test_expect_success 't_e_i() exclude case #8' '
+	git init case8 &&
+	(
+		cd case8 &&
+		echo file >file1 &&
+		echo file >file2 &&
+		git add file1 file2 &&
+		git commit -m twofiles &&
+		git grep -l file HEAD :^file2 >actual &&
+		echo HEAD:file1 >expected &&
+		test_cmp expected actual &&
+		git grep -l file HEAD :^file1 >actual &&
+		echo HEAD:file2 >expected &&
+		test_cmp expected actual
+	)
+'
+
 test_done

tree-walk.c +8 -3

index 77b37f36fa..79bafbd1a2 100644
@@ -1107,7 +1107,7 @@ enum interesting tree_entry_interesting(const struct name_entry *entry,
 	 *   5  |  file |    1     |    1     |   0
 	 *   6  |  file |    1     |    2     |   0
 	 *   7  |  file |    2     |   -1     |   2
-	 *   8  |  file |    2     |    0     |   2
+	 *   8  |  file |    2     |    0     |   1
 	 *   9  |  file |    2     |    1     |   0
 	 *  10  |  file |    2     |    2     |  -1
 	 * -----+-------+----------+----------+-------

@@ -1118,7 +1118,7 @@ enum interesting tree_entry_interesting(const struct name_entry *entry,
 	 *  15  |  dir  |    1     |    1     |   1 (*)
 	 *  16  |  dir  |    1     |    2     |   0
 	 *  17  |  dir  |    2     |   -1     |   2
-	 *  18  |  dir  |    2     |    0     |   2
+	 *  18  |  dir  |    2     |    0     |   1
 	 *  19  |  dir  |    2     |    1     |   1 (*)
 	 *  20  |  dir  |    2     |    2     |  -1
 	 *

@@ -1134,7 +1134,12 @@ enum interesting tree_entry_interesting(const struct name_entry *entry,
 
 	negative = do_match(entry, base, base_offset, ps, 1);
 
-	/* #3, #4, #7, #8, #13, #14, #17, #18 */
+	/* #8, #18 */
+	if (positive == all_entries_interesting &&
+	    negative == entry_not_interesting)
+		return entry_interesting;
+
+	/* #3, #4, #7, #13, #14, #17 */
 	if (negative <= entry_not_interesting)
 		return positive;