fsmonitor: invalidate untracked cache after losing history

dir.c +18 -0

+static void invalidate_directory_recursive(struct untracked_cache *uc,
+					  struct untracked_cache_dir *ucd)
+{
+	invalidate_one_directory(uc, ucd);
+	for (size_t i = 0; i < ucd->dirs_nr; i++)
+		invalidate_directory_recursive(uc, ucd->dirs[i]);
+}
+
+void untracked_cache_invalidate_all(struct index_state *istate)
+{
+	if (!istate->untracked || !istate->untracked->root)
+		return;
+
+	invalidate_directory_recursive(istate->untracked,
+				       istate->untracked->root);
+	istate->cache_changed |= UNTRACKED_CHANGED;
+}
+
 /*
  * Normally when an entry is added or removed from a directory,
  * invalidating that directory is enough. No need to touch its

dir.h +1 -0

index 83e0f648a8..d2de762da8 100644
@@ -596,6 +596,7 @@ static inline int dir_path_match(struct index_state *istate,
 int cmp_dir_entry(const void *p1, const void *p2);
 int check_dir_entry_contains(const struct dir_entry *out, const struct dir_entry *in);
 
+void untracked_cache_invalidate_all(struct index_state *);
 void untracked_cache_invalidate_path(struct index_state *, const char *, int safe_path);
 /*
  * Invalidate the untracked-cache for this path, but first strip

fsmonitor.c +0 -0

index 107767527e..9c3079f64a 100644

From: Derrick Stolee <[email protected]>

When fsmonitor fails or returns a trivial response, refresh_fsmonitor() clears CE_FSMONITOR_VALID and disables fsmonitor use for the untracked cache. Unfortunately, this may leave untracked cache entries marked 'valid' so future commands may miss necessary updates.

A command such as 'git update-index --refresh' can save the replacement fsmonitor token without scanning untracked directories. If the next query returns no changes, fsmonitor use is enabled again and 'git status' trusts directory contents cached before the loss of event history. New untracked files can remain invisible.

This integration bug between fsmonitor and the untracked cache has existed since the fsmonitor feature was introduced in 883e248b8a0 (fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files., 2017-09-22).

To fix the problem, react more completely on a failed fsmonitor token request. In this case, invalidate every untracked cache directory and mark the untracked extension as changed. To guarantee that the index is updatede, set UNTRACKED_CHANGED so that the index is written even when no tracked entry changes. Keep the directory nodes intact to preserve pointers held by callers.
...
fsmonitor: invalidate untracked cache after losing history

A user of an internal monorepo reported this behavior, including an AI-assisted powershell script to prove how it worked. I used an agent to assist in porting that to a Git test script and a code fix, but then applied significant human editing and validation on top to get it up to my standards.

Thanks, -Stolee

Quoted 6 lines · Derrick Stolee via GitGitGadget

A user of an internal monorepo reported this behavior, including an AI-assisted powershell script to prove how it worked. I used an agent to assist in porting that to a Git test script and a code fix, but then applied significant human editing and validation on top to get it up to my standards.

Quoted 1 line

Thanks, -Stolee

Would this be a viable replacement for ik/fsmonitor-untracked-cache-trivial-response

https://lore.kernel.org/git/[email protected]/

that has been left unreviewed for a month?

Quoted 8 lines · Junio C Hamano

Quoted 6 lines · Derrick Stolee via GitGitGadget

A user of an internal monorepo reported this behavior, including an AI-assisted powershell script to prove how it worked. I used an agent to assist in porting that to a Git test script and a code fix, but then applied significant human editing and validation on top to get it up to my standards.

Quoted 7 lines

Quoted 1 line

Thanks, -Stolee

Would this be a viable replacement for ik/fsmonitor-untracked-cache-trivial-response

https://lore.kernel.org/git/[email protected]/

that has been left unreviewed for a month?

Thank you for pointing me at this. I didn't realize there was already a patch working on a fix for this issue.

Ilia's patch has the stronger approach of trying to preserve as much of the untracked cache as possible, which has some performance improvements over my change (in the "rare" case of the fsmonitor failing).

Let me first see if we can take that patch instead of mine. I'll start some review over there.

Let me first see if we can take that patch instead of mine. I'll start some review over there.