Skip to content

Fast git status via a file system watcher

Ben Peart git 3 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v8 Latest12 patches 15 repliesIn review
  2. v712 patches 24 repliesSuperseded
  3. v612 patches 31 repliesSuperseded

Patches in v8 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/12bswap: add 64 bit endianness helper get_be64compat/bswap.h+22 -04 replies
2/12preload-index: add override to enable testing preload-indexpreload-index.c+2 -00 replies
3/12update-index: add a new --force-write-index optionbuiltin/update-index.c+4 -10 replies
4/12fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.Makefile builtin/update-index.c cache.h +13 more+417 -190 replies
5/12fsmonitor: add documentation for the fsmonitor extension.Documentation/config.txt Documentation/git-ls-files.txt Documentation/git-update-index.txt +2 more+105 -10 replies
6/12ls-files: Add support in ls-files to display the fsmonitor valid bitbuiltin/ls-files.c+6 -20 replies
7/12update-index: add fsmonitor support to update-indexbuiltin/update-index.c+32 -10 replies
8/12fsmonitor: add a test tool to dump the index extensionMakefile t/helper/test-dump-fsmonitor.c+22 -03 replies
9/12split-index: disable the fsmonitor extension when running the split index testt/t1700-split-index.sh+1 -00 replies
10/12fsmonitor: add test cases for fsmonitor extensiont/t7519-status-fsmonitor.sh t/t7519/fsmonitor-all t/t7519/fsmonitor-none +1 more+490 -00 replies
11/12fsmonitor: add a sample integration script for Watchmantemplates/hooks--fsmonitor-watchman.sample+122 -00 replies
12/12fsmonitor: add a performance testMakefile t/helper/.gitignore t/helper/test-drop-caches.c +1 more+350 -00 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v8 yet.

Read the v8 thread

Cover letter [PATCH v8 00/12] Fast git status via a file system watcher

Thanks again to everyone who provided feedback. There are several spelling, documentation, and code comment changes.

The only behavioral change from V7 is the removal of unnecessary uses of CE_MATCH_IGNORE_FSMONITOR. With a better understanding of *why* the CE_MATCH_IGNORE_* flags are used, it is now clear they are not required in most cases where CE_MATCH_IGNORE_FSMONITOR was being passed out of an abundance of caution.

Since the fsmonitor data can be trusted and is kept in sync with the working directory, the only remaining valid uses are those locations where we don't want to trigger an unneeded refresh_fsmonitor() call.

One is where preload_index() is doing a fast precompute of state for the bulk of the index entries but is not required for correctness as refresh_cache_ent() will ensure any "missed" by preload_index() are up-to-date if/when they are needed.

The second is in is_staging_gitmodules_ok() where we don't want to trigger a complete refresh just to check the .gitignore file.

The net result of this change will be that there are more cases where we will be able to use the cached index state and avoid unnecessary lstat() calls.

Interdiff between V7 and V8:

Documentation/git-update-index.txt +1 -1
index 95231dbfcb..7c2f880a22 100644
@@ -476,7 +476,7 @@ inform it as to what files have been modified. This enables git to avoid
 having to lstat() every file to find modified files.
 
 When used in conjunction with the untracked cache, it can further improve
-performance by avoiding the cost of scaning the entire working directory
+performance by avoiding the cost of scanning the entire working directory
 looking for new files.
 
 If you want to enable (or disable) this feature, it is easier to use
apply.c +1 -1
index 9061cc5f15..71cbbd141c 100644
@@ -3399,7 +3399,7 @@ static int verify_index_match(const struct cache_entry *ce, struct stat *st)
 			return -1;
 		return 0;
 	}
-	return ce_match_stat(ce, st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_IGNORE_FSMONITOR);
+	return ce_match_stat(ce, st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE);
 }
 
 #define SUBMODULE_PATCH_WITHOUT_INDEX 1
cache.h +1 -1
index eccab968bd..f1c903e1b6 100644
@@ -682,7 +682,7 @@ extern void *read_blob_data_from_index(const struct index_state *, const char *,
 #define CE_MATCH_IGNORE_MISSING		0x08
 /* enable stat refresh */
 #define CE_MATCH_REFRESH		0x10
-/* do stat comparison even if CE_FSMONITOR_VALID is true */
+/* don't refresh_fsmonitor state or do stat comparison even if CE_FSMONITOR_VALID is true */
 #define CE_MATCH_IGNORE_FSMONITOR 0X20
 extern int ie_match_stat(struct index_state *, const struct cache_entry *, struct stat *, unsigned int);
 extern int ie_modified(struct index_state *, const struct cache_entry *, struct stat *, unsigned int);
entry.c +1 -1
index 5e6794f9fc..3a7b667373 100644
@@ -404,7 +404,7 @@ int checkout_entry(struct cache_entry *ce,
 
 	if (!check_path(path.buf, path.len, &st, state->base_dir_len)) {
 		const struct submodule *sub;
-		unsigned changed = ce_match_stat(ce, &st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_IGNORE_FSMONITOR);
+		unsigned changed = ce_match_stat(ce, &st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE);
 		/*
 		 * Needs to be checked before !changed returns early,
 		 * as the possibly empty directory was not changed
fsmonitor.c +1 -1
index b8b2d88fe1..7c1540c054 100644
@@ -176,7 +176,7 @@ void refresh_fsmonitor(struct index_state *istate)
 			core_fsmonitor, query_success ? "success" : "failure");
 	}
 
-	/* a fsmonitor process can return '*' to indicate all entries are invalid */
+	/* a fsmonitor process can return '/' to indicate all entries are invalid */
 	if (query_success && query_result.buf[0] != '/') {
 		/* Mark all entries returned by the monitor as dirty */
 		buf = query_result.buf;
fsmonitor.h +8 -3
index c2240b811a..8eb6163455 100644
@@ -35,7 +35,9 @@ extern void tweak_fsmonitor(struct index_state *istate);
 extern void refresh_fsmonitor(struct index_state *istate);
 
 /*
- * Set the given cache entries CE_FSMONITOR_VALID bit.
+ * Set the given cache entries CE_FSMONITOR_VALID bit. This should be
+ * called any time the cache entry has been updated to reflect the
+ * current state of the file on disk.
  */
 static inline void mark_fsmonitor_valid(struct cache_entry *ce)
 {

@@ -46,8 +48,11 @@ static inline void mark_fsmonitor_valid(struct cache_entry *ce)
 }
 
 /*
- * Clear the given cache entry's CE_FSMONITOR_VALID bit and invalidate any
- * corresponding untracked cache directory structures.
+ * Clear the given cache entry's CE_FSMONITOR_VALID bit and invalidate
+ * any corresponding untracked cache directory structures. This should
+ * be called any time git creates or modifies a file that should
+ * trigger an lstat() or invalidate the untracked cache for the
+ * corresponding directory
  */
 static inline void mark_fsmonitor_invalid(struct index_state *istate, struct cache_entry *ce)
 {
read-cache.c +2 -2
index 53093dbebf..05c0a33fdd 100644
@@ -641,7 +641,7 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,
 	int size, namelen, was_same;
 	mode_t st_mode = st->st_mode;
 	struct cache_entry *ce, *alias;
-	unsigned ce_option = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_RACY_IS_DIRTY|CE_MATCH_IGNORE_FSMONITOR;
+	unsigned ce_option = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_RACY_IS_DIRTY;
 	int verbose = flags & (ADD_CACHE_VERBOSE | ADD_CACHE_PRETEND);
 	int pretend = flags & ADD_CACHE_PRETEND;
 	int intent_only = flags & ADD_CACHE_INTENT;

@@ -1356,7 +1356,7 @@ int refresh_index(struct index_state *istate, unsigned int flags,
 	int first = 1;
 	int in_porcelain = (flags & REFRESH_IN_PORCELAIN);
 	unsigned int options = (CE_MATCH_REFRESH |
-				(really ? CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_FSMONITOR : 0) |
+				(really ? CE_MATCH_IGNORE_VALID : 0) |
 				(not_new ? CE_MATCH_IGNORE_MISSING : 0));
 	const char *modified_fmt;
 	const char *deleted_fmt;
t/helper/test-drop-caches.c +4 -2
index 4e5ca8f397..bd1a857d52 100644
@@ -82,6 +82,8 @@ static int cmd_dropcaches(void)
 	HANDLE hProcess = GetCurrentProcess();
 	HANDLE hToken;
 	HMODULE ntdll;
+	DWORD(WINAPI *NtSetSystemInformation)(INT, PVOID, ULONG);
+	SYSTEM_MEMORY_LIST_COMMAND command;
 	int status;
 
 	if (!OpenProcessToken(hProcess, TOKEN_QUERY | TOKEN_ADJUST_PRIVILEGES, &hToken))

@@ -96,12 +98,12 @@ static int cmd_dropcaches(void)
 	if (!ntdll)
 		return error("Can't load ntdll.dll, wrong Windows version?");
 
-	DWORD(WINAPI *NtSetSystemInformation)(INT, PVOID, ULONG) =
+	NtSetSystemInformation =
 		(DWORD(WINAPI *)(INT, PVOID, ULONG))GetProcAddress(ntdll, "NtSetSystemInformation");
 	if (!NtSetSystemInformation)
 		return error("Can't get function addresses, wrong Windows version?");
 
-	SYSTEM_MEMORY_LIST_COMMAND command = MemoryPurgeStandbyList;
+	command = MemoryPurgeStandbyList;
 	status = NtSetSystemInformation(
 		SystemMemoryListInformation,
 		&command,
unpack-trees.c +3 -3
index f724a61ac0..1f5d371636 100644
@@ -1456,7 +1456,7 @@ static int verify_uptodate_1(const struct cache_entry *ce,
 		return 0;
 
 	if (!lstat(ce->name, &st)) {
-		int flags = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_IGNORE_FSMONITOR;
+		int flags = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE;
 		unsigned changed = ie_match_stat(o->src_index, ce, &st, flags);
 
 		if (submodule_from_ce(ce)) {

@@ -1612,7 +1612,7 @@ static int icase_exists(struct unpack_trees_options *o, const char *name, int le
 	const struct cache_entry *src;
 
 	src = index_file_exists(o->src_index, name, len, 1);
-	return src && !ie_match_stat(o->src_index, src, st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_IGNORE_FSMONITOR);
+	return src && !ie_match_stat(o->src_index, src, st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE);
 }
 
 static int check_ok_to_remove(const char *name, int len, int dtype,

@@ -2136,7 +2136,7 @@ int oneway_merge(const struct cache_entry * const *src,
 		if (o->reset && o->update && !ce_uptodate(old) && !ce_skip_worktree(old)) {
 			struct stat st;
 			if (lstat(old->name, &st) ||
-			    ie_match_stat(o->src_index, old, &st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_IGNORE_FSMONITOR))
+			    ie_match_stat(o->src_index, old, &st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE))
 				update |= CE_UPDATE;
 		}
 		add_entry(o, old, update, 0);

Ben Peart (12):
bswap: add 64 bit endianness helper get_be64
preload-index: add override to enable testing preload-index
update-index: add a new --force-write-index option
fsmonitor: teach git to optionally utilize a file system monitor to
speed up detecting new or changed files.
fsmonitor: add documentation for the fsmonitor extension.
ls-files: Add support in ls-files to display the fsmonitor valid bit
update-index: add fsmonitor support to update-index
fsmonitor: add a test tool to dump the index extension
split-index: disable the fsmonitor extension when running the split
index test
fsmonitor: add test cases for fsmonitor extension
fsmonitor: add a sample integration script for Watchman
fsmonitor: add a performance test

 Documentation/config.txt                   |   7 +
 Documentation/git-ls-files.txt             |   7 +-
 Documentation/git-update-index.txt         |  45 +++++
 Documentation/githooks.txt                 |  28 +++
 Documentation/technical/index-format.txt   |  19 ++
 Makefile                                   |   3 +
 builtin/ls-files.c                         |   8 +-
 builtin/update-index.c                     |  38 +++-
 cache.h                                    |  10 +-
 compat/bswap.h                             |  22 +++
 config.c                                   |  14 ++
 config.h                                   |   1 +
 diff-lib.c                                 |   2 +
 dir.c                                      |  27 ++-
 dir.h                                      |   2 +
 entry.c                                    |   2 +
 environment.c                              |   1 +
 fsmonitor.c                                | 253 ++++++++++++++++++++++++
 fsmonitor.h                                |  66 +++++++
 preload-index.c                            |   8 +-
 read-cache.c                               |  45 ++++-
 submodule.c                                |   2 +-
 t/helper/.gitignore                        |   1 +
 t/helper/test-drop-caches.c                | 164 ++++++++++++++++
 t/helper/test-dump-fsmonitor.c             |  21 ++
 t/perf/p7519-fsmonitor.sh                  | 184 +++++++++++++++++
 t/t1700-split-index.sh                     |   1 +
 t/t7519-status-fsmonitor.sh                | 304 +++++++++++++++++++++++++++++
 t/t7519/fsmonitor-all                      |  24 +++
 t/t7519/fsmonitor-none                     |  22 +++
 t/t7519/fsmonitor-watchman                 | 140 +++++++++++++
 templates/hooks--fsmonitor-watchman.sample | 122 ++++++++++++
 unpack-trees.c                             |   2 +
 33 files changed, 1572 insertions(+), 23 deletions(-)
 create mode 100644 fsmonitor.c
 create mode 100644 fsmonitor.h
 create mode 100644 t/helper/test-drop-caches.c
 create mode 100644 t/helper/test-dump-fsmonitor.c
 create mode 100755 t/perf/p7519-fsmonitor.sh
 create mode 100755 t/t7519-status-fsmonitor.sh
 create mode 100755 t/t7519/fsmonitor-all
 create mode 100755 t/t7519/fsmonitor-none
 create mode 100755 t/t7519/fsmonitor-watchman
 create mode 100755 templates/hooks--fsmonitor-watchman.sample
Signature · 1 line
2.14.1.549.g6ff7ed0467

Read the cover letter in its thread