Skip to contentSkip to conversation

fsmonitor: don't fill bitmap with entries to be removed

PATCH Superseded William Baker via GitGitGadget 5 participants 17 messages Oct 3, 2019 – Oct 15, 2019

lore
Share & tools
lore

William Baker via GitGitGadget

author
#0 lore raw

cover fsmonitor: don't fill bitmap with entries to be removed

This patch series fixes a segfault that I encountered while testing fsmonitor. Under some circumstances, the fsmonitor extension was being written with too many bits, and subsequent git commands would segfault when trying to apply the bits to the index.

As part of these changes I've added some BUG checks that would have helped catch this problem sooner. Special thanks to Dscho for pointing me in the right direction and suggesting a test that can reproduce the issue.

Thanks, William

William Baker (1):
fsmonitor: don't fill bitmap with entries to be removed

 fsmonitor.c                 | 29 ++++++++++++++++++++++++-----
 t/t7519-status-fsmonitor.sh | 12 ++++++++++++
 t/t7519/fsmonitor-env       | 24 ++++++++++++++++++++++++
 3 files changed, 60 insertions(+), 5 deletions(-)
 create mode 100755 t/t7519/fsmonitor-env

base-commit: 5fa0f5238b0cd46cfe7f6fa76c3f526ea98148d9
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-372%2Fwilbaker%2Ffix_git_fsmonitor_crash-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-372/wilbaker/fix_git_fsmonitor_crash-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/372

Signature · 1 line
gitgitgadget
author
#1 lore raw

PATCH 1/1 fsmonitor: don't fill bitmap with entries to be removed

From: William Baker <[email protected]>

While doing some testing with fsmonitor enabled I found that git commands would segfault after staging and unstaging an untracked file. Looking at the crash it appeared that fsmonitor_ewah_callback was attempting to adjust bits beyond the bounds of the index cache.

Digging into how this could happen it became clear that the fsmonitor extension must have been written with more bits than there were entries in the index. The root cause ended up being that fill_fsmonitor_bitmap was populating fsmonitor_dirty with bits for all entries in the index, even those that had been marked for removal.

To solve this problem fill_fsmonitor_bitmap has been updated to skip entries with the the CE_REMOVE flag set. With this change the bits written for the fsmonitor extension will be consistent with the index entries written by do_write_index. Additionally, BUG checks have been added to detect if the number of bits in fsmonitor_dirty should ever exceed the number of entries in the index again.

Another option that was considered was moving the call to fill_fsmonitor_bitmap closer to where the index is written (and where the fsmonitor extension itself is written). However, that did not work as the fsmonitor_dirty bitmap must be filled before the index is split during writing.

Signed-off-by William Baker <[email protected]>
---
 fsmonitor.c                 | 29 ++++++++++++++++++++++++-----
 t/t7519-status-fsmonitor.sh | 12 ++++++++++++
 t/t7519/fsmonitor-env       | 24 ++++++++++++++++++++++++
 3 files changed, 60 insertions(+), 5 deletions(-)
 create mode 100755 t/t7519/fsmonitor-env
fsmonitor.c +24 -5
index 231e83a94d..fa0b96d84e 100644
@@ -14,8 +14,13 @@ struct trace_key trace_fsmonitor = TRACE_KEY_INIT(FSMONITOR);
 static void fsmonitor_ewah_callback(size_t pos, void *is)
 {
 	struct index_state *istate = (struct index_state *)is;
-	struct cache_entry *ce = istate->cache[pos];
+	struct cache_entry *ce;
+	
+	if (pos >= istate->cache_nr)
+		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" >= %"PRIuMAX")",
+		    (uintmax_t)pos, (uintmax_t)istate->cache_nr);
 
+	ce = istate->cache[pos];
 	ce->ce_flags &= ~CE_FSMONITOR_VALID;
 }
 

@@ -50,17 +55,24 @@ int read_fsmonitor_extension(struct index_state *istate, const void *data,
 	}
 	istate->fsmonitor_dirty = fsmonitor_dirty;
 
+	if (istate->fsmonitor_dirty->bit_size > istate->cache_nr)
+		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %"PRIuMAX")",
+		    (uintmax_t)istate->fsmonitor_dirty->bit_size, (uintmax_t)istate->cache_nr);
+
 	trace_printf_key(&trace_fsmonitor, "read fsmonitor extension successful");
 	return 0;
 }
 
 void fill_fsmonitor_bitmap(struct index_state *istate)
 {
-	unsigned int i;
+	unsigned int i, skipped = 0;
 	istate->fsmonitor_dirty = ewah_new();
-	for (i = 0; i < istate->cache_nr; i++)
-		if (!(istate->cache[i]->ce_flags & CE_FSMONITOR_VALID))
-			ewah_set(istate->fsmonitor_dirty, i);
+	for (i = 0; i < istate->cache_nr; i++) {
+		if (istate->cache[i]->ce_flags & CE_REMOVE)
+			skipped++;
+		else if (!(istate->cache[i]->ce_flags & CE_FSMONITOR_VALID))
+			ewah_set(istate->fsmonitor_dirty, i - skipped);
+	}
 }
 
 void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)

@@ -71,6 +83,10 @@ void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)
 	uint32_t ewah_size = 0;
 	int fixup = 0;
 
+	if (istate->fsmonitor_dirty->bit_size > istate->cache_nr)
+		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %"PRIuMAX")",
+		    (uintmax_t)istate->fsmonitor_dirty->bit_size, (uintmax_t)istate->cache_nr);
+
 	put_be32(&hdr_version, INDEX_EXTENSION_VERSION);
 	strbuf_add(sb, &hdr_version, sizeof(uint32_t));
 

@@ -236,6 +252,9 @@ void tweak_fsmonitor(struct index_state *istate)
 			}
 
 			/* Mark all previously saved entries as dirty */
+			if (istate->fsmonitor_dirty->bit_size > istate->cache_nr)
+				BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %"PRIuMAX")",
+				    (uintmax_t)istate->fsmonitor_dirty->bit_size, (uintmax_t)istate->cache_nr);
 			ewah_each_bit(istate->fsmonitor_dirty, fsmonitor_ewah_callback, istate);
 
 			/* Now mark the untracked cache for fsmonitor usage */
t/t7519-status-fsmonitor.sh +12 -0
index 81a375fa0f..85df54f07c 100755
@@ -354,4 +354,16 @@ test_expect_success 'discard_index() also discards fsmonitor info' '
 	test_cmp expect actual
 '
 
+# Use test files that start with 'z' so that the entries being added
+# and removed appear at the end of the index.
+test_expect_success 'status succeeds after staging/unstaging ' '
+	test_commit initial &&
+	removed=$(test_seq 1 100 | sed "s/^/z/") &&
+	touch $removed &&
+	git add $removed &&
+	test_config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-env" &&
+	FSMONITOR_LIST="$removed" git restore -S $removed &&
+	FSMONITOR_LIST="$removed" git status
+'
+
 test_done
t/t7519/fsmonitor-env new file +24 -0
new file mode 100755
index 0000000000..8f1f7ab164
@@ -0,0 +1,24 @@
+#!/bin/sh
+#
+# An test hook script to integrate with git to test fsmonitor.
+#
+# The hook is passed a version (currently 1) and a time in nanoseconds
+# formatted as a string and outputs to stdout all files that have been
+# modified since the given time. Paths must be relative to the root of
+# the working tree and separated by a single NUL.
+#
+#echo "$0 $*" >&2
+
+if test "$#" -ne 2
+then
+	echo "$0: exactly 2 arguments expected" >&2
+	exit 2
+fi
+
+if test "$1" != 1
+then
+	echo "Unsupported core.fsmonitor hook version." >&2
+	exit 1
+fi
+
+printf '%s\n' $FSMONITOR_LIST
Signature · 1 line
gitgitgadget
#4 lore raw
In reply to William Baker via GitGitGadget

Hi all,

Quoted 10 lines · William Baker via GitGitGadget

This patch series fixes a segfault that I encountered while testing fsmonitor. Under some circumstances, the fsmonitor extension was being written with too many bits, and subsequent git commands would segfault when trying to apply the bits to the index.

As part of these changes I've added some BUG checks that would have helped catch this problem sooner. Special thanks to Dscho for pointing me in the right direction and suggesting a test that can reproduce the issue.

Thanks, William

Please note that I was involved with the development of this patch, reviewed a couple of iterations internally and am implictly okay with this first public iteration.

Ciao,
Dscho

Quoted 18 lines

William Baker (1):
fsmonitor: don't fill bitmap with entries to be removed

 fsmonitor.c                 | 29 ++++++++++++++++++++++++-----
 t/t7519-status-fsmonitor.sh | 12 ++++++++++++
 t/t7519/fsmonitor-env       | 24 ++++++++++++++++++++++++
 3 files changed, 60 insertions(+), 5 deletions(-)
 create mode 100755 t/t7519/fsmonitor-env

base-commit: 5fa0f5238b0cd46cfe7f6fa76c3f526ea98148d9
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-372%2Fwilbaker%2Ffix_git_fsmonitor_crash-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-372/wilbaker/fix_git_fsmonitor_crash-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/372
--
gitgitgadget

#5 lore raw
Quoted 13 lines · Johannes Schindelin

Hi all,

Quoted 9 lines · William Baker via GitGitGadget

This patch series fixes a segfault that I encountered while testing fsmonitor. Under some circumstances, the fsmonitor extension was being written with too many bits, and subsequent git commands would segfault when trying to apply the bits to the index.

As part of these changes I've added some BUG checks that would have helped catch this problem sooner. Special thanks to Dscho for pointing me in the right direction and suggesting a test that can reproduce the issue.

Quoted 5 lines
Quoted 1 line

Thanks, William

Please note that I was involved with the development of this patch, reviewed a couple of iterations internally and am implictly okay with this first public iteration.

IOW, "Reviewed-by: dscho". That's good to hear.

THanks.

#2 lore raw
In reply to William Baker via GitGitGadget

fsmonitor.c struct trace_key trace_fsmonitor = TRACE_KEY_INIT(FSMONITOR); · line 21

Quoted 5 lines · William Baker via GitGitGadget
 create mode 100755 t/t7519/fsmonitor-env

...

fsmonitor.c +3 -0
+	if (pos >= istate->cache_nr)
+		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" >= %"PRIuMAX")",
+		    (uintmax_t)pos, (uintmax_t)istate->cache_nr);

This is how we show size_t values without using "%z" that we avoid, but are "pos" and 'cache_nr" size_t or ssize_t? I thought they are plain boring unsigned, so shouldn't we use the plain boring "%u" without casting?

The same comment applies to other uses of uintmax_t cast in this patch.

Quoted 5 lines

void fill_fsmonitor_bitmap(struct index_state *istate) {

fsmonitor.c +1 -1
-	unsigned int i;
+	unsigned int i, skipped = 0;
 	istate->fsmonitor_dirty = ewah_new();

fsmonitor.c int read_fsmonitor_extension(struct index_state *istate, const void *data, · line 76

Quoted 10 lines
fsmonitor.c +6 -3
-	for (i = 0; i < istate->cache_nr; i++)
-		if (!(istate->cache[i]->ce_flags & CE_FSMONITOR_VALID))
-			ewah_set(istate->fsmonitor_dirty, i);
+	for (i = 0; i < istate->cache_nr; i++) {
+		if (istate->cache[i]->ce_flags & CE_REMOVE)
+			skipped++;
+		else if (!(istate->cache[i]->ce_flags & CE_FSMONITOR_VALID))
+			ewah_set(istate->fsmonitor_dirty, i - skipped);
+	}
 }

Matches the explanation in the proposed log message pretty well. Good job.

t/t7519-status-fsmonitor.sh test_expect_success 'discard_index() also discards fsmonitor info' ' · line 358

Quoted 6 lines
t/t7519-status-fsmonitor.sh +2 -0
@@ -354,4 +354,16 @@ test_expect_success 'discard_index() also discards fsmonitor info' '
 	test_cmp expect actual
 '
 
+# Use test files that start with 'z' so that the entries being added
+# and removed appear at the end of the index.

In other words, future developers are warned against adding entries to and leaving them in the index that sort later than z100 in new tests they add before this point. Is the above wording clear enough to tell them that, I wonder?

t/t7519-status-fsmonitor.sh test_expect_success 'discard_index() also discards fsmonitor info' ' · line 361

Quoted 3 lines
t/t7519-status-fsmonitor.sh +3 -0
+test_expect_success 'status succeeds after staging/unstaging ' '
+	test_commit initial &&
+	removed=$(test_seq 1 100 | sed "s/^/z/") &&

Thanks.

#3 lore raw
Quoted 9 lines · Junio C Hamano

fsmonitor.c struct trace_key trace_fsmonitor = TRACE_KEY_INIT(FSMONITOR); · line 21

Quoted 3 lines
fsmonitor.c +3 -0
+	if (pos >= istate->cache_nr)
+		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" >= %"PRIuMAX")",
+		    (uintmax_t)pos, (uintmax_t)istate->cache_nr);

This is how we show size_t values without using "%z" that we avoid, but are "pos" and 'cache_nr" size_t or ssize_t? I thought they are plain boring unsigned, so shouldn't we use the plain boring "%u" without casting?

Quoted 3 lines

The same comment applies to other uses of uintmax_t cast in this patch.

Thanks for catching this. I will update these BUGs in the next patch to avoid casting.

Quoted 8 lines

t/t7519-status-fsmonitor.sh test_expect_success 'discard_index() also discards fsmonitor info' ' · line 358

Quoted 2 lines
t/t7519-status-fsmonitor.sh +2 -0
+# Use test files that start with 'z' so that the entries being added
+# and removed appear at the end of the index.

In other words, future developers are warned against adding entries to and leaving them in the index that sort later than z100 in new tests they add before this point. Is the above wording clear enough to tell them that, I wonder?

You're understanding is correct, and I agree this comment could be clearer. I will fix this up in v2.

Thanks for the feedback!
William

author
#7 lore raw

PATCH v2 1/1 fsmonitor: don't fill bitmap with entries to be removed

In reply to William Baker via GitGitGadget

From: William Baker <[email protected]>

While doing some testing with fsmonitor enabled I found that git commands would segfault after staging and unstaging an untracked file. Looking at the crash it appeared that fsmonitor_ewah_callback was attempting to adjust bits beyond the bounds of the index cache.

Digging into how this could happen it became clear that the fsmonitor extension must have been written with more bits than there were entries in the index. The root cause ended up being that fill_fsmonitor_bitmap was populating fsmonitor_dirty with bits for all entries in the index, even those that had been marked for removal.

To solve this problem fill_fsmonitor_bitmap has been updated to skip entries with the the CE_REMOVE flag set. With this change the bits written for the fsmonitor extension will be consistent with the index entries written by do_write_index. Additionally, BUG checks have been added to detect if the number of bits in fsmonitor_dirty should ever exceed the number of entries in the index again.

Another option that was considered was moving the call to fill_fsmonitor_bitmap closer to where the index is written (and where the fsmonitor extension itself is written). However, that did not work as the fsmonitor_dirty bitmap must be filled before the index is split during writing.

Signed-off-by William Baker <[email protected]>
---
 fsmonitor.c                 | 29 ++++++++++++++++++++++++-----
 t/t7519-status-fsmonitor.sh | 13 +++++++++++++
 t/t7519/fsmonitor-env       | 24 ++++++++++++++++++++++++
 3 files changed, 61 insertions(+), 5 deletions(-)
 create mode 100755 t/t7519/fsmonitor-env
fsmonitor.c +24 -5
index 231e83a94d..7e27839278 100644
@@ -14,8 +14,13 @@ struct trace_key trace_fsmonitor = TRACE_KEY_INIT(FSMONITOR);
 static void fsmonitor_ewah_callback(size_t pos, void *is)
 {
 	struct index_state *istate = (struct index_state *)is;
-	struct cache_entry *ce = istate->cache[pos];
+	struct cache_entry *ce;
+	
+	if (pos >= istate->cache_nr)
+		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" >= %u)",
+		    (uintmax_t)pos, istate->cache_nr);
 
+	ce = istate->cache[pos];
 	ce->ce_flags &= ~CE_FSMONITOR_VALID;
 }
 

@@ -50,17 +55,24 @@ int read_fsmonitor_extension(struct index_state *istate, const void *data,
 	}
 	istate->fsmonitor_dirty = fsmonitor_dirty;
 
+	if (istate->fsmonitor_dirty->bit_size > istate->cache_nr)
+		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %u)",
+		    (uintmax_t)istate->fsmonitor_dirty->bit_size, istate->cache_nr);
+
 	trace_printf_key(&trace_fsmonitor, "read fsmonitor extension successful");
 	return 0;
 }
 
 void fill_fsmonitor_bitmap(struct index_state *istate)
 {
-	unsigned int i;
+	unsigned int i, skipped = 0;
 	istate->fsmonitor_dirty = ewah_new();
-	for (i = 0; i < istate->cache_nr; i++)
-		if (!(istate->cache[i]->ce_flags & CE_FSMONITOR_VALID))
-			ewah_set(istate->fsmonitor_dirty, i);
+	for (i = 0; i < istate->cache_nr; i++) {
+		if (istate->cache[i]->ce_flags & CE_REMOVE)
+			skipped++;
+		else if (!(istate->cache[i]->ce_flags & CE_FSMONITOR_VALID))
+			ewah_set(istate->fsmonitor_dirty, i - skipped);
+	}
 }
 
 void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)

@@ -71,6 +83,10 @@ void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)
 	uint32_t ewah_size = 0;
 	int fixup = 0;
 
+	if (istate->fsmonitor_dirty->bit_size > istate->cache_nr)
+		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %u)",
+		    (uintmax_t)istate->fsmonitor_dirty->bit_size, istate->cache_nr);
+
 	put_be32(&hdr_version, INDEX_EXTENSION_VERSION);
 	strbuf_add(sb, &hdr_version, sizeof(uint32_t));
 

@@ -236,6 +252,9 @@ void tweak_fsmonitor(struct index_state *istate)
 			}
 
 			/* Mark all previously saved entries as dirty */
+			if (istate->fsmonitor_dirty->bit_size > istate->cache_nr)
+				BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %u)",
+				    (uintmax_t)istate->fsmonitor_dirty->bit_size, istate->cache_nr);
 			ewah_each_bit(istate->fsmonitor_dirty, fsmonitor_ewah_callback, istate);
 
 			/* Now mark the untracked cache for fsmonitor usage */
t/t7519-status-fsmonitor.sh +13 -0
index 81a375fa0f..87042470ab 100755
@@ -354,4 +354,17 @@ test_expect_success 'discard_index() also discards fsmonitor info' '
 	test_cmp expect actual
 '
 
+# This test covers staging/unstaging files that appear at the end of the index.
+# Test files with names beginning with 'z' are used under the assumption that
+# earlier tests do not add/leave index entries that sort below them. 
+test_expect_success 'status succeeds after staging/unstaging ' '
+	test_commit initial &&
+	removed=$(test_seq 1 100 | sed "s/^/z/") &&
+	touch $removed &&
+	git add $removed &&
+	test_config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-env" &&
+	FSMONITOR_LIST="$removed" git restore -S $removed &&
+	FSMONITOR_LIST="$removed" git status
+'
+
 test_done
t/t7519/fsmonitor-env new file +24 -0
new file mode 100755
index 0000000000..8f1f7ab164
@@ -0,0 +1,24 @@
+#!/bin/sh
+#
+# An test hook script to integrate with git to test fsmonitor.
+#
+# The hook is passed a version (currently 1) and a time in nanoseconds
+# formatted as a string and outputs to stdout all files that have been
+# modified since the given time. Paths must be relative to the root of
+# the working tree and separated by a single NUL.
+#
+#echo "$0 $*" >&2
+
+if test "$#" -ne 2
+then
+	echo "$0: exactly 2 arguments expected" >&2
+	exit 2
+fi
+
+if test "$1" != 1
+then
+	echo "Unsupported core.fsmonitor hook version." >&2
+	exit 1
+fi
+
+printf '%s\n' $FSMONITOR_LIST
Signature · 1 line
gitgitgadget
author
#6 lore raw

v2 cover fsmonitor: don't fill bitmap with entries to be removed

In reply to William Baker via GitGitGadget

This is the second iteration of changes to fix the segfault that I encountered while testing fsmonitor. This iteration includes the following updates for feedback I received on v1:

* Use %u instead of %"PRIuMAX" for unsigned ints in BUG format strings
* Updated the new test's comment to be more descriptive

Thanks,

William

William Baker (1):
fsmonitor: don't fill bitmap with entries to be removed

 fsmonitor.c                 | 29 ++++++++++++++++++++++++-----
 t/t7519-status-fsmonitor.sh | 13 +++++++++++++
 t/t7519/fsmonitor-env       | 24 ++++++++++++++++++++++++
 3 files changed, 61 insertions(+), 5 deletions(-)
 create mode 100755 t/t7519/fsmonitor-env

base-commit: 5fa0f5238b0cd46cfe7f6fa76c3f526ea98148d9
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-372%2Fwilbaker%2Ffix_git_fsmonitor_crash-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-372/wilbaker/fix_git_fsmonitor_crash-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/372

Range-diff vs v1:

1: ce9bf4237e ! 1: 08741d986c fsmonitor: don't fill bitmap with entries to be removed

     @@ -44,8 +44,8 @@
      +	struct cache_entry *ce;
      +	
      +	if (pos >= istate->cache_nr)
     -+		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" >= %"PRIuMAX")",
     -+		    (uintmax_t)pos, (uintmax_t)istate->cache_nr);
     ++		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" >= %u)",
     ++		    (uintmax_t)pos, istate->cache_nr);
       
      +	ce = istate->cache[pos];
       	ce->ce_flags &= ~CE_FSMONITOR_VALID;
     @@ -56,8 +56,8 @@
       	istate->fsmonitor_dirty = fsmonitor_dirty;
       
      +	if (istate->fsmonitor_dirty->bit_size > istate->cache_nr)
     -+		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %"PRIuMAX")",
     -+		    (uintmax_t)istate->fsmonitor_dirty->bit_size, (uintmax_t)istate->cache_nr);
     ++		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %u)",
     ++		    (uintmax_t)istate->fsmonitor_dirty->bit_size, istate->cache_nr);
      +
       	trace_printf_key(&trace_fsmonitor, "read fsmonitor extension successful");
       	return 0;
     @@ -85,8 +85,8 @@
       	int fixup = 0;
       
      +	if (istate->fsmonitor_dirty->bit_size > istate->cache_nr)
     -+		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %"PRIuMAX")",
     -+		    (uintmax_t)istate->fsmonitor_dirty->bit_size, (uintmax_t)istate->cache_nr);
     ++		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %u)",
     ++		    (uintmax_t)istate->fsmonitor_dirty->bit_size, istate->cache_nr);
      +
       	put_be32(&hdr_version, INDEX_EXTENSION_VERSION);
       	strbuf_add(sb, &hdr_version, sizeof(uint32_t));
     @@ -96,8 +96,8 @@
       
       			/* Mark all previously saved entries as dirty */
      +			if (istate->fsmonitor_dirty->bit_size > istate->cache_nr)
     -+				BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %"PRIuMAX")",
     -+				    (uintmax_t)istate->fsmonitor_dirty->bit_size, (uintmax_t)istate->cache_nr);
     ++				BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %u)",
     ++				    (uintmax_t)istate->fsmonitor_dirty->bit_size, istate->cache_nr);
       			ewah_each_bit(istate->fsmonitor_dirty, fsmonitor_ewah_callback, istate);
       
       			/* Now mark the untracked cache for fsmonitor usage */
     @@ -109,8 +109,9 @@
       	test_cmp expect actual
       '
       
     -+# Use test files that start with 'z' so that the entries being added
     -+# and removed appear at the end of the index.
     ++# This test covers staging/unstaging files that appear at the end of the index.
     ++# Test files with names beginning with 'z' are used under the assumption that
     ++# earlier tests do not add/leave index entries that sort below them. 
      +test_expect_success 'status succeeds after staging/unstaging ' '
      +	test_commit initial &&
      +	removed=$(test_seq 1 100 | sed "s/^/z/") &&
Signature · 1 line
gitgitgadget
#11 lore raw
Quoted 7 lines · William Baker via GitGitGadget

This is the second iteration of changes to fix the segfault that I encountered while testing fsmonitor. This iteration includes the following updates for feedback I received on v1:

* Use %u instead of %"PRIuMAX" for unsigned ints in BUG format strings
* Updated the new test's comment to be more descriptive

Quoted 6 lines

Thanks,

William

William Baker (1):
fsmonitor: don't fill bitmap with entries to be removed

Thanks. Dscho, is this still Reviewed-by: you?

Quoted 72 lines
 fsmonitor.c                 | 29 ++++++++++++++++++++++++-----
 t/t7519-status-fsmonitor.sh | 13 +++++++++++++
 t/t7519/fsmonitor-env       | 24 ++++++++++++++++++++++++
 3 files changed, 61 insertions(+), 5 deletions(-)
 create mode 100755 t/t7519/fsmonitor-env

base-commit: 5fa0f5238b0cd46cfe7f6fa76c3f526ea98148d9
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-372%2Fwilbaker%2Ffix_git_fsmonitor_crash-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-372/wilbaker/fix_git_fsmonitor_crash-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/372

Range-diff vs v1:

1: ce9bf4237e ! 1: 08741d986c fsmonitor: don't fill bitmap with entries to be removed

     @@ -44,8 +44,8 @@
      +	struct cache_entry *ce;
      +	
      +	if (pos >= istate->cache_nr)
     -+		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" >= %"PRIuMAX")",
     -+		    (uintmax_t)pos, (uintmax_t)istate->cache_nr);
     ++		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" >= %u)",
     ++		    (uintmax_t)pos, istate->cache_nr);
       
      +	ce = istate->cache[pos];
       	ce->ce_flags &= ~CE_FSMONITOR_VALID;
     @@ -56,8 +56,8 @@
       	istate->fsmonitor_dirty = fsmonitor_dirty;
       
      +	if (istate->fsmonitor_dirty->bit_size > istate->cache_nr)
     -+		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %"PRIuMAX")",
     -+		    (uintmax_t)istate->fsmonitor_dirty->bit_size, (uintmax_t)istate->cache_nr);
     ++		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %u)",
     ++		    (uintmax_t)istate->fsmonitor_dirty->bit_size, istate->cache_nr);
      +
       	trace_printf_key(&trace_fsmonitor, "read fsmonitor extension successful");
       	return 0;
     @@ -85,8 +85,8 @@
       	int fixup = 0;
       
      +	if (istate->fsmonitor_dirty->bit_size > istate->cache_nr)
     -+		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %"PRIuMAX")",
     -+		    (uintmax_t)istate->fsmonitor_dirty->bit_size, (uintmax_t)istate->cache_nr);
     ++		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %u)",
     ++		    (uintmax_t)istate->fsmonitor_dirty->bit_size, istate->cache_nr);
      +
       	put_be32(&hdr_version, INDEX_EXTENSION_VERSION);
       	strbuf_add(sb, &hdr_version, sizeof(uint32_t));
     @@ -96,8 +96,8 @@
       
       			/* Mark all previously saved entries as dirty */
      +			if (istate->fsmonitor_dirty->bit_size > istate->cache_nr)
     -+				BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %"PRIuMAX")",
     -+				    (uintmax_t)istate->fsmonitor_dirty->bit_size, (uintmax_t)istate->cache_nr);
     ++				BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %u)",
     ++				    (uintmax_t)istate->fsmonitor_dirty->bit_size, istate->cache_nr);
       			ewah_each_bit(istate->fsmonitor_dirty, fsmonitor_ewah_callback, istate);
       
       			/* Now mark the untracked cache for fsmonitor usage */
     @@ -109,8 +109,9 @@
       	test_cmp expect actual
       '
       
     -+# Use test files that start with 'z' so that the entries being added
     -+# and removed appear at the end of the index.
     ++# This test covers staging/unstaging files that appear at the end of the index.
     ++# Test files with names beginning with 'z' are used under the assumption that
     ++# earlier tests do not add/leave index entries that sort below them. 
      +test_expect_success 'status succeeds after staging/unstaging ' '
      +	test_commit initial &&
      +	removed=$(test_seq 1 100 | sed "s/^/z/") &&
#8 lore raw
In reply to William Baker via GitGitGadget

t/t7519-status-fsmonitor.sh test_expect_success 'discard_index() also discards fsmonitor info' ' · line 361

Quoted 13 lines · William Baker via GitGitGadget
t/t7519-status-fsmonitor.sh +5 -0
index 81a375fa0f..87042470ab 100755
@@ -354,4 +354,17 @@ test_expect_success 'discard_index() also discards fsmonitor info' '
 	test_cmp expect actual
 '
 
+# This test covers staging/unstaging files that appear at the end of the index.
+# Test files with names beginning with 'z' are used under the assumption that
+# earlier tests do not add/leave index entries that sort below them. 
+test_expect_success 'status succeeds after staging/unstaging ' '
+	test_commit initial &&

This is confusing: this is the 29th test case in this script and it creates an "initial" commit?!

The first "setup" test case has already created an initial commit, so this should rather be called "second".

OTOH, none of the later commands in this test case seem to have anything to do with this second commit, and indeed the test case works even without it (i.e. 'git status' still segfaults without the fix and then succeeds with the fix applied), so instead of updating its message perhaps it could simply be removed.

Quoted 9 lines
t/t7519-status-fsmonitor.sh +8 -0
+	removed=$(test_seq 1 100 | sed "s/^/z/") &&
+	touch $removed &&
+	git add $removed &&
+	test_config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-env" &&
+	FSMONITOR_LIST="$removed" git restore -S $removed &&
+	FSMONITOR_LIST="$removed" git status
+'
+
 test_done
#9 lore raw
Quoted 9 lines · SZEDER Gábor
Quoted 8 lines · William Baker via GitGitGadget
t/t7519-status-fsmonitor.sh +0 -0
index 81a375fa0f..87042470ab 100755
@@ -354,4 +354,17 @@ test_expect_success 'discard_index() also discards fsmonitor info' '
 	test_cmp expect actual
 '
Quoted 3 lines

t/t7519-status-fsmonitor.sh test_expect_success 'discard_index() also discards fsmonitor info' ' · line 359

Quoted 3 lines
t/t7519-status-fsmonitor.sh +3 -0
+# This test covers staging/unstaging files that appear at the end of the index.
+# Test files with names beginning with 'z' are used under the assumption that
+# earlier tests do not add/leave index entries that sort below them.

I just read through Junio's comments on the first version of this patch, in particular his remarks about this comment.

If this new test case below were run in a dedicated repository, then this comment wouldn't be necessary, and all my comments below about that not-really-initial commit would be moot, too.

Quoted 25 lines

t/t7519-status-fsmonitor.sh test_expect_success 'discard_index() also discards fsmonitor info' ' · line 361

Quoted 2 lines
t/t7519-status-fsmonitor.sh +2 -0
+test_expect_success 'status succeeds after staging/unstaging ' '
+	test_commit initial &&

This is confusing: this is the 29th test case in this script and it creates an "initial" commit?!

The first "setup" test case has already created an initial commit, so this should rather be called "second".

OTOH, none of the later commands in this test case seem to have anything to do with this second commit, and indeed the test case works even without it (i.e. 'git status' still segfaults without the fix and then succeeds with the fix applied), so instead of updating its message perhaps it could simply be removed.

Quoted 9 lines
t/t7519-status-fsmonitor.sh +8 -0
+	removed=$(test_seq 1 100 | sed "s/^/z/") &&
+	touch $removed &&
+	git add $removed &&
+	test_config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-env" &&
+	FSMONITOR_LIST="$removed" git restore -S $removed &&
+	FSMONITOR_LIST="$removed" git status
+'
+
 test_done
#12 lore raw
In reply to Junio C Hamano

Hi Junio,

Quoted 9 lines · Junio C Hamano
Quoted 7 lines · William Baker via GitGitGadget

This is the second iteration of changes to fix the segfault that I encountered while testing fsmonitor. This iteration includes the following updates for feedback I received on v1:

* Use %u instead of %"PRIuMAX" for unsigned ints in BUG format strings
* Updated the new test's comment to be more descriptive

Quoted 8 lines
Quoted 6 lines

Thanks,

William

William Baker (1):
fsmonitor: don't fill bitmap with entries to be removed

Thanks. Dscho, is this still Reviewed-by: you?

Yes!

Thanks,
Dscho

P.S.: I guess we could even go and make this a Co-Authored-by: because the test case is actually by me, at least originally... ;-)

Quoted 74 lines
Quoted 73 lines

 fsmonitor.c                 | 29 ++++++++++++++++++++++++-----
 t/t7519-status-fsmonitor.sh | 13 +++++++++++++
 t/t7519/fsmonitor-env       | 24 ++++++++++++++++++++++++
 3 files changed, 61 insertions(+), 5 deletions(-)
 create mode 100755 t/t7519/fsmonitor-env

base-commit: 5fa0f5238b0cd46cfe7f6fa76c3f526ea98148d9
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-372%2Fwilbaker%2Ffix_git_fsmonitor_crash-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-372/wilbaker/fix_git_fsmonitor_crash-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/372

Range-diff vs v1:

1: ce9bf4237e ! 1: 08741d986c fsmonitor: don't fill bitmap with entries to be removed

     @@ -44,8 +44,8 @@
      +	struct cache_entry *ce;
      +
      +	if (pos >= istate->cache_nr)
     -+		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" >= %"PRIuMAX")",
     -+		    (uintmax_t)pos, (uintmax_t)istate->cache_nr);
     ++		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" >= %u)",
     ++		    (uintmax_t)pos, istate->cache_nr);

      +	ce = istate->cache[pos];
       	ce->ce_flags &= ~CE_FSMONITOR_VALID;
     @@ -56,8 +56,8 @@
       	istate->fsmonitor_dirty = fsmonitor_dirty;

      +	if (istate->fsmonitor_dirty->bit_size > istate->cache_nr)
     -+		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %"PRIuMAX")",
     -+		    (uintmax_t)istate->fsmonitor_dirty->bit_size, (uintmax_t)istate->cache_nr);
     ++		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %u)",
     ++		    (uintmax_t)istate->fsmonitor_dirty->bit_size, istate->cache_nr);
      +
       	trace_printf_key(&trace_fsmonitor, "read fsmonitor extension successful");
       	return 0;
     @@ -85,8 +85,8 @@
       	int fixup = 0;

      +	if (istate->fsmonitor_dirty->bit_size > istate->cache_nr)
     -+		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %"PRIuMAX")",
     -+		    (uintmax_t)istate->fsmonitor_dirty->bit_size, (uintmax_t)istate->cache_nr);
     ++		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %u)",
     ++		    (uintmax_t)istate->fsmonitor_dirty->bit_size, istate->cache_nr);
      +
       	put_be32(&hdr_version, INDEX_EXTENSION_VERSION);
       	strbuf_add(sb, &hdr_version, sizeof(uint32_t));
     @@ -96,8 +96,8 @@

       			/* Mark all previously saved entries as dirty */
      +			if (istate->fsmonitor_dirty->bit_size > istate->cache_nr)
     -+				BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %"PRIuMAX")",
     -+				    (uintmax_t)istate->fsmonitor_dirty->bit_size, (uintmax_t)istate->cache_nr);
     ++				BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %u)",
     ++				    (uintmax_t)istate->fsmonitor_dirty->bit_size, istate->cache_nr);
       			ewah_each_bit(istate->fsmonitor_dirty, fsmonitor_ewah_callback, istate);

       			/* Now mark the untracked cache for fsmonitor usage */
     @@ -109,8 +109,9 @@
       	test_cmp expect actual
       '

     -+# Use test files that start with 'z' so that the entries being added
     -+# and removed appear at the end of the index.
     ++# This test covers staging/unstaging files that appear at the end of the index.
     ++# Test files with names beginning with 'z' are used under the assumption that
     ++# earlier tests do not add/leave index entries that sort below them.
      +test_expect_success 'status succeeds after staging/unstaging ' '
      +	test_commit initial &&
      +	removed=$(test_seq 1 100 | sed "s/^/z/") &&
#10 lore raw
In reply to SZEDER Gábor
Quoted 11 lines · SZEDER Gábor
Quoted 3 lines

t/t7519-status-fsmonitor.sh test_expect_success 'discard_index() also discards fsmonitor info' ' · line 359

Quoted 3 lines
t/t7519-status-fsmonitor.sh +3 -0
+# This test covers staging/unstaging files that appear at the end of the index.
+# Test files with names beginning with 'z' are used under the assumption that
+# earlier tests do not add/leave index entries that sort below them.

I just read through Junio's comments on the first version of this patch, in particular his remarks about this comment.

If this new test case below were run in a dedicated repository, then this comment wouldn't be necessary, and all my comments below about that not-really-initial commit would be moot, too.

Thanks for this suggestion! I will submit a v3 version of the patch with an update to the test script.

- William

author
#14 lore raw

PATCH v3 1/1 fsmonitor: don't fill bitmap with entries to be removed

In reply to William Baker via GitGitGadget

From: William Baker <[email protected]>

While doing some testing with fsmonitor enabled I found that git commands would segfault after staging and unstaging an untracked file. Looking at the crash it appeared that fsmonitor_ewah_callback was attempting to adjust bits beyond the bounds of the index cache.

Digging into how this could happen it became clear that the fsmonitor extension must have been written with more bits than there were entries in the index. The root cause ended up being that fill_fsmonitor_bitmap was populating fsmonitor_dirty with bits for all entries in the index, even those that had been marked for removal.

To solve this problem fill_fsmonitor_bitmap has been updated to skip entries with the the CE_REMOVE flag set. With this change the bits written for the fsmonitor extension will be consistent with the index entries written by do_write_index. Additionally, BUG checks have been added to detect if the number of bits in fsmonitor_dirty should ever exceed the number of entries in the index again.

Another option that was considered was moving the call to fill_fsmonitor_bitmap closer to where the index is written (and where the fsmonitor extension itself is written). However, that did not work as the fsmonitor_dirty bitmap must be filled before the index is split during writing.

Signed-off-by William Baker <[email protected]>
---
 fsmonitor.c                 | 29 ++++++++++++++++++++++++-----
 t/t7519-status-fsmonitor.sh | 17 +++++++++++++++++
 t/t7519/fsmonitor-env       | 24 ++++++++++++++++++++++++
 3 files changed, 65 insertions(+), 5 deletions(-)
 create mode 100755 t/t7519/fsmonitor-env
fsmonitor.c +24 -5
index 231e83a94d..7e27839278 100644
@@ -14,8 +14,13 @@ struct trace_key trace_fsmonitor = TRACE_KEY_INIT(FSMONITOR);
 static void fsmonitor_ewah_callback(size_t pos, void *is)
 {
 	struct index_state *istate = (struct index_state *)is;
-	struct cache_entry *ce = istate->cache[pos];
+	struct cache_entry *ce;
+	
+	if (pos >= istate->cache_nr)
+		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" >= %u)",
+		    (uintmax_t)pos, istate->cache_nr);
 
+	ce = istate->cache[pos];
 	ce->ce_flags &= ~CE_FSMONITOR_VALID;
 }
 

@@ -50,17 +55,24 @@ int read_fsmonitor_extension(struct index_state *istate, const void *data,
 	}
 	istate->fsmonitor_dirty = fsmonitor_dirty;
 
+	if (istate->fsmonitor_dirty->bit_size > istate->cache_nr)
+		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %u)",
+		    (uintmax_t)istate->fsmonitor_dirty->bit_size, istate->cache_nr);
+
 	trace_printf_key(&trace_fsmonitor, "read fsmonitor extension successful");
 	return 0;
 }
 
 void fill_fsmonitor_bitmap(struct index_state *istate)
 {
-	unsigned int i;
+	unsigned int i, skipped = 0;
 	istate->fsmonitor_dirty = ewah_new();
-	for (i = 0; i < istate->cache_nr; i++)
-		if (!(istate->cache[i]->ce_flags & CE_FSMONITOR_VALID))
-			ewah_set(istate->fsmonitor_dirty, i);
+	for (i = 0; i < istate->cache_nr; i++) {
+		if (istate->cache[i]->ce_flags & CE_REMOVE)
+			skipped++;
+		else if (!(istate->cache[i]->ce_flags & CE_FSMONITOR_VALID))
+			ewah_set(istate->fsmonitor_dirty, i - skipped);
+	}
 }
 
 void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)

@@ -71,6 +83,10 @@ void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)
 	uint32_t ewah_size = 0;
 	int fixup = 0;
 
+	if (istate->fsmonitor_dirty->bit_size > istate->cache_nr)
+		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %u)",
+		    (uintmax_t)istate->fsmonitor_dirty->bit_size, istate->cache_nr);
+
 	put_be32(&hdr_version, INDEX_EXTENSION_VERSION);
 	strbuf_add(sb, &hdr_version, sizeof(uint32_t));
 

@@ -236,6 +252,9 @@ void tweak_fsmonitor(struct index_state *istate)
 			}
 
 			/* Mark all previously saved entries as dirty */
+			if (istate->fsmonitor_dirty->bit_size > istate->cache_nr)
+				BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %u)",
+				    (uintmax_t)istate->fsmonitor_dirty->bit_size, istate->cache_nr);
 			ewah_each_bit(istate->fsmonitor_dirty, fsmonitor_ewah_callback, istate);
 
 			/* Now mark the untracked cache for fsmonitor usage */
t/t7519-status-fsmonitor.sh +17 -0
index 81a375fa0f..af3f9951fa 100755
@@ -354,4 +354,21 @@ test_expect_success 'discard_index() also discards fsmonitor info' '
 	test_cmp expect actual
 '
 
+# Test staging/unstaging files that appear at the end of the index.  Test
+# file names begin with 'z' so that they are sorted to the end of the index. 
+test_expect_success 'status succeeds after staging/unstaging ' '
+	test_create_repo fsmonitor-stage-unstage &&
+	(
+		cd fsmonitor-stage-unstage &&
+		test_commit initial &&
+		git update-index --fsmonitor &&
+		removed=$(test_seq 1 100 | sed "s/^/z/") &&
+		touch $removed &&
+		git add $removed &&
+		git config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-env" &&
+		FSMONITOR_LIST="$removed" git restore -S $removed &&
+		FSMONITOR_LIST="$removed" git status
+	)
+'
+
 test_done
t/t7519/fsmonitor-env new file +24 -0
new file mode 100755
index 0000000000..8f1f7ab164
@@ -0,0 +1,24 @@
+#!/bin/sh
+#
+# An test hook script to integrate with git to test fsmonitor.
+#
+# The hook is passed a version (currently 1) and a time in nanoseconds
+# formatted as a string and outputs to stdout all files that have been
+# modified since the given time. Paths must be relative to the root of
+# the working tree and separated by a single NUL.
+#
+#echo "$0 $*" >&2
+
+if test "$#" -ne 2
+then
+	echo "$0: exactly 2 arguments expected" >&2
+	exit 2
+fi
+
+if test "$1" != 1
+then
+	echo "Unsupported core.fsmonitor hook version." >&2
+	exit 1
+fi
+
+printf '%s\n' $FSMONITOR_LIST
Signature · 1 line
gitgitgadget
author
#13 lore raw

v3 cover fsmonitor: don't fill bitmap with entries to be removed

In reply to William Baker via GitGitGadget

This is the third iteration of changes to fix the segfault that I encountered while testing fsmonitor. This iteration includes the following updates for feedback I received on v2:

* Update the new test case to use its own dedicated test repository

This latest v3 series has been reviewed by Dscho.

Thanks, William

William Baker (1):
fsmonitor: don't fill bitmap with entries to be removed

 fsmonitor.c                 | 29 ++++++++++++++++++++++++-----
 t/t7519-status-fsmonitor.sh | 17 +++++++++++++++++
 t/t7519/fsmonitor-env       | 24 ++++++++++++++++++++++++
 3 files changed, 65 insertions(+), 5 deletions(-)
 create mode 100755 t/t7519/fsmonitor-env

base-commit: 5fa0f5238b0cd46cfe7f6fa76c3f526ea98148d9
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-372%2Fwilbaker%2Ffix_git_fsmonitor_crash-v3
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-372/wilbaker/fix_git_fsmonitor_crash-v3
Pull-Request: https://github.com/gitgitgadget/git/pull/372

Range-diff vs v2:

1: 08741d986c ! 1: 840972e08b fsmonitor: don't fill bitmap with entries to be removed

     @@ -109,17 +109,21 @@
       	test_cmp expect actual
       '
       
     -+# This test covers staging/unstaging files that appear at the end of the index.
     -+# Test files with names beginning with 'z' are used under the assumption that
     -+# earlier tests do not add/leave index entries that sort below them. 
     ++# Test staging/unstaging files that appear at the end of the index.  Test
     ++# file names begin with 'z' so that they are sorted to the end of the index. 
      +test_expect_success 'status succeeds after staging/unstaging ' '
     -+	test_commit initial &&
     -+	removed=$(test_seq 1 100 | sed "s/^/z/") &&
     -+	touch $removed &&
     -+	git add $removed &&
     -+	test_config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-env" &&
     -+	FSMONITOR_LIST="$removed" git restore -S $removed &&
     -+	FSMONITOR_LIST="$removed" git status
     ++	test_create_repo fsmonitor-stage-unstage &&
     ++	(
     ++		cd fsmonitor-stage-unstage &&
     ++		test_commit initial &&
     ++		git update-index --fsmonitor &&
     ++		removed=$(test_seq 1 100 | sed "s/^/z/") &&
     ++		touch $removed &&
     ++		git add $removed &&
     ++		git config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-env" &&
     ++		FSMONITOR_LIST="$removed" git restore -S $removed &&
     ++		FSMONITOR_LIST="$removed" git status
     ++	)
      +'
      +
       test_done
Signature · 1 line
gitgitgadget
#15 lore raw
In reply to William Baker via GitGitGadget

t/t7519-status-fsmonitor.sh test_expect_success 'discard_index() also discards fsmonitor info' ' · line 358

Quoted 2 lines · William Baker via GitGitGadget
t/t7519-status-fsmonitor.sh +2 -0
+# Test staging/unstaging files that appear at the end of the index.  Test
+# file names begin with 'z' so that they are sorted to the end of the index.

Well, the test is now done in a freshly created repository, so the z* files are the only thing you have in here---technically they are at the end of the index, but so they are at the beginning, too.

Would it affect the effectiveness of the test that you do not have any other paths in the working tree or in the index, unlike the test in the previous rounds that did not use a newly created test repository?

This is not a rhetorical question, but purely asking. "no, this still tests what we want to test and shows breakage when the fix to the code in the patch gets reverted" is perfectly a good answer, but in that case, is "the end of" the most important trait of the condition this test is checking? Wouldn't the bug be exposed as long as we remove sufficiently large number of entries (like "removing more paths than the paths still in the index at the end" or something like that)?

Thanks.

Quoted 46 lines
t/t7519-status-fsmonitor.sh +15 -0
+test_expect_success 'status succeeds after staging/unstaging ' '
+	test_create_repo fsmonitor-stage-unstage &&
+	(
+		cd fsmonitor-stage-unstage &&
+		test_commit initial &&
+		git update-index --fsmonitor &&
+		removed=$(test_seq 1 100 | sed "s/^/z/") &&
+		touch $removed &&
+		git add $removed &&
+		git config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-env" &&
+		FSMONITOR_LIST="$removed" git restore -S $removed &&
+		FSMONITOR_LIST="$removed" git status
+	)
+'
+
 test_done
t/t7519/fsmonitor-env new file +24 -0
new file mode 100755
index 0000000000..8f1f7ab164
@@ -0,0 +1,24 @@
+#!/bin/sh
+#
+# An test hook script to integrate with git to test fsmonitor.
+#
+# The hook is passed a version (currently 1) and a time in nanoseconds
+# formatted as a string and outputs to stdout all files that have been
+# modified since the given time. Paths must be relative to the root of
+# the working tree and separated by a single NUL.
+#
+#echo "$0 $*" >&2
+
+if test "$#" -ne 2
+then
+	echo "$0: exactly 2 arguments expected" >&2
+	exit 2
+fi
+
+if test "$1" != 1
+then
+	echo "Unsupported core.fsmonitor hook version." >&2
+	exit 1
+fi
+
+printf '%s\n' $FSMONITOR_LIST
#16 lore raw
Quoted 9 lines · Junio C Hamano

t/t7519-status-fsmonitor.sh test_expect_success 'discard_index() also discards fsmonitor info' ' · line 358

Quoted 2 lines · William Baker via GitGitGadget
t/t7519-status-fsmonitor.sh +2 -0
+# Test staging/unstaging files that appear at the end of the index.  Test
+# file names begin with 'z' so that they are sorted to the end of the index.

Well, the test is now done in a freshly created repository, so the z* files are the only thing you have in here---technically they are at the end of the index, but so they are at the beginning, too.

There is one other file in the index created by 'test_commit', however, the point still stands that there are almost no other entries in the index now that the test is using its own repository.

Quoted 4 lines

Would it affect the effectiveness of the test that you do not have any other paths in the working tree or in the index, unlike the test in the previous rounds that did not use a newly created test repository?

The test still validates the scenario that we're concerned about, namely that the new index that's written has less entries than the index of the last entry in the old index that's is not flagged with CE_FSMONITOR_VALID but is flagged for removal (CE_REMOVE).

Quoted 8 lines

This is not a rhetorical question, but purely asking. "no, this still tests what we want to test and shows breakage when the fix to the code in the patch gets reverted" is perfectly a good answer, but in that case, is "the end of" the most important trait of the condition this test is checking? Wouldn't the bug be exposed as long as we remove sufficiently large number of entries (like "removing more paths than the paths still in the index at the end" or something like that)?

This is exactly right. The most important trait is that the last entry flagged with CE_REMOVE does not have CE_FSMONITOR_VALID set and has an index >= the number of entries in the new index being written.

I will send out a patch on top of 'wb/fsmonitor-bitmap-fix' with an update to the comment for this test.

Thanks,
William