fsmonitor: implement filesystem change listener for Linux
Subsystems (3)
PATCH v11 7/12 fsmonitor: implement filesystem change listener for Linux
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
Implement the built-in fsmonitor daemon for Linux using the inotify API, bringing it to feature parity with the existing Windows and macOS implementations.
The implementation uses inotify rather than fanotify because fanotify requires either CAP_SYS_ADMIN or CAP_PERFMON capabilities, making it unsuitable for an unprivileged user-space daemon. While inotify has the limitation of requiring a separate watch on every directory (unlike macOS's FSEvents, which can monitor an entire directory tree with a single watch), it operates without elevated privileges and provides the per-file event granularity needed for fsmonitor.
The listener uses inotify_init1(O_NONBLOCK) with a poll loop that checks for events with a 50-millisecond timeout, keeping the inotify queue well-drained to minimize the risk of overflows. Bidirectional hashmaps map between watch descriptors and directory paths for efficient event resolution. Directory renames are tracked using inotify's cookie mechanism to correlate IN_MOVED_FROM and IN_MOVED_TO event pairs; a periodic check detects stale renames where the matching IN_MOVED_TO never arrived, forcing a resync.
New directory creation triggers recursive watch registration to ensure all subdirectories are monitored. The IN_MASK_CREATE flag is used where available to prevent modifying existing watches, with a fallback for older kernels. When IN_MASK_CREATE is available and inotify_add_watch returns EEXIST, it means another thread or recursive scan has already registered the watch, so it is safe to ignore.
Remote filesystem detection uses statfs() to identify network-mounted filesystems (NFS, CIFS, SMB, FUSE, etc.) via their magic numbers. Mount point information is read from /proc/mounts and matched against the statfs f_fsid to get accurate, human-readable filesystem type names for logging. When the .git directory is on a remote filesystem, the IPC socket falls back to $HOME or a user-configured directory via the fsmonitor.socketDir setting.
--- Documentation/config/fsmonitor--daemon.adoc | 4 +- Documentation/git-fsmonitor--daemon.adoc | 28 +- compat/fsmonitor/fsm-health-linux.c | 33 + compat/fsmonitor/fsm-listen-linux.c | 746 ++++++++++++++++++++ compat/fsmonitor/fsm-path-utils-linux.c | 217 ++++++ config.mak.uname | 10 + contrib/buildsystems/CMakeLists.txt | 8 +- meson.build | 4 + 8 files changed, 1042 insertions(+), 8 deletions(-) create mode 100644 compat/fsmonitor/fsm-health-linux.c create mode 100644 compat/fsmonitor/fsm-listen-linux.c create mode 100644 compat/fsmonitor/fsm-path-utils-linux.c
Documentation/config/fsmonitor--daemon.adoc +2 -2
@@ -4,8 +4,8 @@ fsmonitor.allowRemote:: behavior. Only respected when `core.fsmonitor` is set to `true`. fsmonitor.socketDir:: - This Mac OS-specific option, if set, specifies the directory in + This Mac OS and Linux-specific option, if set, specifies the directory in which to create the Unix domain socket used for communication between the fsmonitor daemon and various Git commands. The directory must - reside on a native Mac OS filesystem. Only respected when `core.fsmonitor` + reside on a native filesystem. Only respected when `core.fsmonitor` is set to `true`.
Documentation/git-fsmonitor--daemon.adoc +24 -4
@@ -76,9 +76,9 @@ repositories; this may be overridden by setting `fsmonitor.allowRemote` to correctly with all network-mounted repositories, so such use is considered experimental. -On Mac OS, the inter-process communication (IPC) between various Git +On Mac OS and Linux, the inter-process communication (IPC) between various Git commands and the fsmonitor daemon is done via a Unix domain socket (UDS) -- a -special type of file -- which is supported by native Mac OS filesystems, +special type of file -- which is supported by native Mac OS and Linux filesystems, but not on network-mounted filesystems, NTFS, or FAT32. Other filesystems may or may not have the needed support; the fsmonitor daemon is not guaranteed to work with these filesystems and such use is considered experimental.
@@ -87,13 +87,33 @@ By default, the socket is created in the `.git` directory. However, if the `.git` directory is on a network-mounted filesystem, it will instead be created at `$HOME/.git-fsmonitor-*` unless `$HOME` itself is on a network-mounted filesystem, in which case you must set the configuration -variable `fsmonitor.socketDir` to the path of a directory on a Mac OS native +variable `fsmonitor.socketDir` to the path of a directory on a native filesystem in which to create the socket file. If none of the above directories (`.git`, `$HOME`, or `fsmonitor.socketDir`) -is on a native Mac OS file filesystem the fsmonitor daemon will report an +is on a native filesystem the fsmonitor daemon will report an error that will cause the daemon and the currently running command to exit. +LINUX CAVEATS +~~~~~~~~~~~~~ + +On Linux, the fsmonitor daemon uses inotify to monitor filesystem events. +The inotify system has per-user limits on the number of watches that can +be created. The default limit is typically 8192 watches per user. + +For large repositories with many directories, you may need to increase +this limit. Check the current limit with: + + cat /proc/sys/fs/inotify/max_user_watches + +To temporarily increase the limit: + + sudo sysctl fs.inotify.max_user_watches=65536 + +To make the change permanent, add to `/etc/sysctl.conf`: + + fs.inotify.max_user_watches=65536 + CONFIGURATION -------------
compat/fsmonitor/fsm-health-linux.c new file +33 -0
@@ -0,0 +1,33 @@ +#include "git-compat-util.h" +#include "config.h" +#include "fsmonitor-ll.h" +#include "fsm-health.h" +#include "fsmonitor--daemon.h" + +/* +* The Linux fsmonitor implementation uses inotify which has its own +* mechanisms for detecting filesystem unmount and other events that +* would require the daemon to shutdown. Therefore, we don't need +* a separate health thread like Windows does. +* +* These stub functions satisfy the interface requirements. +*/ + +int fsm_health__ctor(struct fsmonitor_daemon_state *state UNUSED) +{ +return 0 ; +} + +void fsm_health__dtor(struct fsmonitor_daemon_state *state UNUSED) +{ +return ; +} + +void fsm_health__loop(struct fsmonitor_daemon_state *state UNUSED) +{ +return ; +} + +void fsm_health__stop_async(struct fsmonitor_daemon_state *state UNUSED) +{ +}
compat/fsmonitor/fsm-listen-linux.c new file +746 -0
746 lines not on this page · raw message
compat/fsmonitor/fsm-path-utils-linux.c new file +217 -0
217 lines not on this page · raw message
config.mak.uname +10 -0
@@ -68,6 +68,16 @@ ifeq ($(uname_S),Linux) BASIC_CFLAGS += -std=c99 endif LINK_FUZZ_PROGRAMS = YesPlease + + # The builtin FSMonitor on Linux builds upon Simple-IPC. Both require + # Unix domain sockets and PThreads. + ifndef NO_PTHREADS + ifndef NO_UNIX_SOCKETS + FSMONITOR_DAEMON_BACKEND = linux + FSMONITOR_OS_SETTINGS = unix + BASIC_CFLAGS += -DHAVE_LINUX_MAGIC_H + endif + endif endif ifeq ($(uname_S),GNU/kFreeBSD) HAVE_ALLOCA_H = YesPlease
contrib/buildsystems/CMakeLists.txt +6 -2
@@ -296,6 +296,10 @@ if(SUPPORTS_SIMPLE_IPC) elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") set(FSMONITOR_DAEMON_BACKEND "darwin") set(FSMONITOR_OS_SETTINGS "unix") + elseif(CMAKE_SYSTEM_NAME STREQUAL "Linux") + set(FSMONITOR_DAEMON_BACKEND "linux") + set(FSMONITOR_OS_SETTINGS "unix") + add_compile_definitions(HAVE_LINUX_MAGIC_H) endif() if(FSMONITOR_DAEMON_BACKEND)
@@ -1149,8 +1153,8 @@ endif() file(STRINGS ${CMAKE_SOURCE_DIR}/GIT-BUILD-OPTIONS.in git_build_options NEWLINE_CONSUME) string(REPLACE "@BROKEN_PATH_FIX@" "" git_build_options "${git_build_options}") string(REPLACE "@DIFF@" "'${DIFF}'" git_build_options "${git_build_options}") -string(REPLACE "@FSMONITOR_DAEMON_BACKEND@" "win32" git_build_options "${git_build_options}") -string(REPLACE "@FSMONITOR_OS_SETTINGS@" "win32" git_build_options "${git_build_options}") +string(REPLACE "@FSMONITOR_DAEMON_BACKEND@" "${FSMONITOR_DAEMON_BACKEND}" git_build_options "${git_build_options}") +string(REPLACE "@FSMONITOR_OS_SETTINGS@" "${FSMONITOR_OS_SETTINGS}" git_build_options "${git_build_options}") string(REPLACE "@GITWEBDIR@" "'${GITWEBDIR}'" git_build_options "${git_build_options}") string(REPLACE "@GIT_INTEROP_MAKE_OPTS@" "" git_build_options "${git_build_options}") string(REPLACE "@GIT_PERF_LARGE_REPO@" "" git_build_options "${git_build_options}")
meson.build +4 -0
@@ -1324,6 +1324,10 @@ fsmonitor_os = '' if host_machine.system() == 'windows' fsmonitor_backend = 'win32' fsmonitor_os = 'win32' +elif host_machine.system() == 'linux' and threads.found() and compiler.has_header('linux/magic.h') + fsmonitor_backend = 'linux' + fsmonitor_os = 'unix' + libgit_c_args += '-DHAVE_LINUX_MAGIC_H' elif host_machine.system() == 'darwin' fsmonitor_backend = 'darwin' fsmonitor_os = 'unix'
Signature · 1 line
gitgitgadget
PATCH v11 8/12 run-command: add close_fd_above_stderr option
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
Add a close_fd_above_stderr flag to struct child_process. When set, the child closes file descriptors 3 and above between fork and exec (skipping the child-notifier pipe), capped at sysconf(_SC_OPEN_MAX) or 4096, whichever is smaller. This prevents the child from inheriting pipe endpoints or other descriptors from the parent environment (e.g., the test harness).
--- run-command.c | 12 ++++++++++++ run-command.h | 9 +++++++++ 2 files changed, 21 insertions(+)
run-command.c +12 -0
@@ -546,6 +546,7 @@ static void atfork_parent(struct atfork_state *as) "restoring signal mask"); #endif } +#endif /* GIT_WINDOWS_NATIVE */ static inline void set_cloexec (int fd)
@@ -832,6 +833,17 @@ fail_pipe:child_close (cmd->out); } +if (cmd->close_fd_above_stderr) { +long max_fd =sysconf (_SC_OPEN_MAX); +int fd; +if (max_fd <0 || max_fd >4096 ) + max_fd =4096 ; +for (fd =3 ; fd < max_fd; fd++) { +if (fd != child_notifier) +close (fd); + } + } +if (cmd->dir &&chdir (cmd->dir))child_die (CHILD_ERR_CHDIR );
run-command.h +9 -0
@@ -141,6 +141,15 @@ struct child_process { unsigned stdout_to_stderr:1; unsigned clean_on_exit:1; unsigned wait_after_clean:1; + +/** +* Close file descriptors 3 and above in the child after forking +* but before exec. This prevents the child from inheriting +* pipe endpoints or other descriptors from the parent +* environment (e.g., the test harness). +*/ + unsigned close_fd_above_stderr:1; +void (*clean_on_exit_handler)(struct child_process *process); };
Signature · 1 line
gitgitgadget
PATCH v11 9/12 fsmonitor: close inherited file descriptors and detach in daemon
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
When the fsmonitor daemon is spawned as a background process, it may inherit file descriptors from its parent that it does not need. In particular, when the test harness or a CI system captures output through pipes, the daemon can inherit duplicated pipe endpoints. If the daemon holds these open, the parent process never sees EOF and may appear to hang.
Set close_fd_above_stderr on the child process at both daemon startup
paths: the explicit "fsmonitor--daemon start" command and the implicit spawn triggered by fsmonitor-ipc when a client finds no running daemon. Also suppress stdout and stderr on the implicit spawn path to prevent the background daemon from writing to the client's terminal.
Additionally, call setsid() when the daemon starts with --detach to create a new session and process group. This prevents the daemon from being part of the spawning shell's process group, which could cause the shell's "wait" to block until the daemon exits.
--- builtin/fsmonitor--daemon.c | 16 ++++++++++++++-- fsmonitor-ipc.c | 3 +++ 2 files changed, 17 insertions(+), 2 deletions(-)
builtin/fsmonitor--daemon.c +14 -2
@@ -1439,7 +1439,7 @@ done:return err; } -static int try_to_run_foreground_daemon(int detach_console MAYBE_UNUSED) +static int try_to_run_foreground_daemon (int detach_console) {/* * Technically, we don't need to probe for an existing daemon
@@ -1459,10 +1459,21 @@ static int try_to_run_foreground_daemon(int detach_console MAYBE_UNUSED)fflush (stderr); } +if (detach_console) {#ifdef GIT_WINDOWS_NATIVE -if (detach_console)FreeConsole (); +#else +/* +* Create a new session so that the daemon is detached +* from the parent's process group. This prevents +* shells with job control (e.g. bash with "set -m") +* from waiting on the daemon when they wait for a +* foreground command that implicitly spawned it. +*/ +if (setsid () ==-1 ) +warning_errno (_ ("setsid failed" ));#endif + }return !!fsmonitor_run_daemon (); }
@@ -1525,6 +1536,7 @@ static int try_to_start_background_daemon(void) cp.no_stdin =1 ; cp.no_stdout =1 ; cp.no_stderr =1 ; + cp.close_fd_above_stderr =1 ; sbgr =start_bg_command (&cp, bg_wait_cb,NULL , fsmonitor__start_timeout_sec);
fsmonitor-ipc.c +3 -0
@@ -61,6 +61,9 @@ static int spawn_daemon(void) cmd.git_cmd =1 ; cmd.no_stdin =1 ; + cmd.no_stdout =1 ; + cmd.no_stderr =1 ; + cmd.close_fd_above_stderr =1 ; cmd.trace2_child_class ="fsmonitor" ;strvec_pushl (&cmd.args,"fsmonitor--daemon" ,"start" ,NULL );
Signature · 1 line
gitgitgadget
PATCH v11 10/12 fsmonitor: add timeout to daemon stop command
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
The "fsmonitor--daemon stop" command polls in a loop waiting for the daemon to exit after sending a "quit" command over IPC. If the daemon fails to shut down (e.g. it is stuck or wedged), this loop spins forever.
Add a 30-second timeout so the stop command returns an error instead of blocking indefinitely.
--- builtin/fsmonitor--daemon.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
builtin/fsmonitor--daemon.c +11 -1
@@ -86,6 +86,8 @@ static int do_as_client__send_stop(void) {struct strbuf answer =STRBUF_INIT ;int ret; +int max_wait_ms =30000 ; +int elapsed_ms =0 ; ret =fsmonitor_ipc__send_command ("quit" , &answer);
@@ -96,8 +98,16 @@ static int do_as_client__send_stop(void)return ret;trace2_region_enter ("fsm_client" ,"polling-for-daemon-exit" ,NULL ); -while (fsmonitor_ipc__get_state () ==IPC_STATE__LISTENING ) +while (fsmonitor_ipc__get_state () ==IPC_STATE__LISTENING ) { +if (elapsed_ms >= max_wait_ms) { +trace2_region_leave ("fsm_client" , +"polling-for-daemon-exit" ,NULL ); +return error (_ ("daemon did not stop within %d seconds" ), + max_wait_ms /1000 ); + }sleep_millisec (50 ); + elapsed_ms +=50 ; + }trace2_region_leave ("fsm_client" ,"polling-for-daemon-exit" ,NULL );return 0 ;
Signature · 1 line
gitgitgadget
PATCH v11 11/12 fsmonitor: add tests for Linux
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
Add a smoke test that verifies the filesystem actually delivers inotify events to the daemon. On some configurations (e.g., overlayfs with older kernels), inotify watches succeed but events are never delivered. The daemon cookie wait will time out, but every subsequent test would fail. Skip the entire test file early when this is detected.
Add a test that exercises rapid nested directory creation to verify the daemon correctly handles the EEXIST race between recursive scan and queued inotify events. When IN_MASK_CREATE is available and a directory watch is added during recursive registration, the kernel may also deliver a queued IN_CREATE event for the same directory. The second inotify_add_watch() returns EEXIST, which must be treated as harmless. An earlier version of the listener crashed in this scenario.
Reduce --start-timeout from the default 60 seconds to 10 seconds so that tests fail promptly when the daemon cannot start.
Harden the test helpers to work in environments without procps (e.g., Fedora CI): fall back to reading /proc/$pid/stat for the process group ID when ps is unavailable, guard stop_git() against an empty pgid, and redirect stderr from kill to /dev/null to avoid noise when processes have already exited.
Use set -m to enable job control in the submodule-pull test so that the background git pull gets its own process group, preventing the shell wait from blocking on the daemon. setsid() in the previous commit detaches the daemon itself, but the intermediate git pull process still needs its own process group for the test shell to manage it correctly.
--- t/t7527-builtin-fsmonitor.sh | 89 +++++++++++++++++++++++++++++++++--- 1 file changed, 82 insertions(+), 7 deletions(-)
t/t7527-builtin-fsmonitor.sh +82 -7
@@ -10,9 +10,58 @@ then test_done fi +# Verify that the filesystem delivers events to the daemon. +# On some configurations (e.g., overlayfs with older kernels), +# inotify watches succeed but events are never delivered. The +# cookie wait will time out and the daemon logs a trace message. +# +# Use "timeout" (if available) to guard each step against hangs. +maybe_timeout () { + if type timeout >/dev/null 2>&1 + then + timeout "$@" + else + shift + "$@" + fi +} +verify_fsmonitor_works () { + git init test_fsmonitor_smoke || return 1 + + GIT_TRACE_FSMONITOR="$PWD/smoke.trace" && + export GIT_TRACE_FSMONITOR && + maybe_timeout 30 \ + git -C test_fsmonitor_smoke fsmonitor--daemon start \ + --start-timeout=10 + ret=$? + unset GIT_TRACE_FSMONITOR + if test $ret -ne 0 + then + rm -rf test_fsmonitor_smoke smoke.trace + return 1 + fi + + maybe_timeout 10 \ + test-tool -C test_fsmonitor_smoke fsmonitor-client query \ + --token 0 >/dev/null 2>&1 + maybe_timeout 5 \ + git -C test_fsmonitor_smoke fsmonitor--daemon stop 2>/dev/null + ! grep -q "cookie_wait timed out" "$PWD/smoke.trace" 2>/dev/null + ret=$? + rm -rf test_fsmonitor_smoke smoke.trace + return $ret +} + +if ! verify_fsmonitor_works +then + skip_all="filesystem does not deliver fsmonitor events (container/overlayfs?)" + test_done +fi + stop_daemon_delete_repo () { r=$1 && - test_might_fail git -C $r fsmonitor--daemon stop && + test_might_fail maybe_timeout 30 \ + git -C $r fsmonitor--daemon stop 2>/dev/null rm -rf $1 }
@@ -67,7 +116,7 @@ start_daemon () { export GIT_TEST_FSMONITOR_TOKEN fi && - git $r fsmonitor--daemon start && + git $r fsmonitor--daemon start --start-timeout=10 && git $r fsmonitor--daemon status ) }
@@ -520,6 +569,28 @@ test_expect_success 'directory changes to a file' ' grep "^event: dir1$" .git/trace ' +test_expect_success 'rapid nested directory creation' ' + test_when_finished "git fsmonitor--daemon stop; rm -rf rapid" && + + start_daemon --tf "$PWD/.git/trace" && + + # Rapidly create nested directories to exercise race conditions + # where directory watches may be added concurrently during + # event processing and recursive scanning. + for i in $(test_seq 1 20) + do + mkdir -p "rapid/nested/dir$i/subdir/deep" || return 1 + done && + + # Give the daemon time to process all events + sleep 1 && + + test-tool fsmonitor-client query --token 0 && + + # Verify daemon is still running (did not crash) + git fsmonitor--daemon status +' + # The next few test cases exercise the token-resync code. When filesystem # drops events (because of filesystem velocity or because the daemon isn't # polling fast enough), we need to discard the cached data (relative to the
@@ -910,7 +981,10 @@ test_expect_success "submodule absorbgitdirs implicitly starts daemon" ' start_git_in_background () { git "$@" & git_pid=$! - git_pgid=$(ps -o pgid= -p $git_pid) + git_pgid=$(ps -o pgid= -p $git_pid 2>/dev/null || + awk '{print $5}' /proc/$git_pid/stat 2>/dev/null) && + git_pgid="${git_pgid## }" && + git_pgid="${git_pgid%% }" nr_tries_left=10 while true do
@@ -921,15 +995,16 @@ start_git_in_background () { fi sleep 1 nr_tries_left=$(($nr_tries_left - 1)) - done >/dev/null 2>&1 & + done >/dev/null 2>&1 3>&- 4>&- 5>&- 6>&- 7>&- & watchdog_pid=$! wait $git_pid } stop_git () { - while kill -0 -- -$git_pgid + test -n "$git_pgid" || return 0 + while kill -0 -- -$git_pgid 2>/dev/null do - kill -- -$git_pgid + kill -- -$git_pgid 2>/dev/null sleep 1 done }
@@ -944,7 +1019,7 @@ stop_watchdog () { test_expect_success !MINGW "submodule implicitly starts daemon by pull" ' test_atexit "stop_watchdog" && - test_when_finished "stop_git; rm -rf cloned super sub" && + test_when_finished "set +m; stop_git; rm -rf cloned super sub" && create_super super && create_sub sub &&
Signature · 1 line
gitgitgadget
PATCH v11 12/12 fsmonitor: convert shown khash to strset in do_handle_client
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
Replace the khash-based string set used for deduplicating pathnames in do_handle_client() with a strset, which provides a cleaner interface for the same purpose.
Since the paths are interned strings from the batch data, use strdup_strings=0 to avoid unnecessary copies.
--- builtin/fsmonitor--daemon.c | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-)
builtin/fsmonitor--daemon.c +6 -11
@@ -16,7 +16,7 @@#include "fsmonitor--daemon.h" #include "simple-ipc.h" -#include "khash.h" +#include "strmap.h" #include "run-command.h" #include "trace.h" #include "trace2.h"
@@ -674,8 +674,6 @@ static int fsmonitor_parse_client_token(const char *buf_token,return 0 ; } -KHASH_INIT(str, const char *, int, 0, kh_str_hash_func, kh_str_hash_equal) - static int do_handle_client(struct fsmonitor_daemon_state *state, const char *command, ipc_server_reply_cb *reply,
@@ -692,8 +690,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,const struct fsmonitor_batch *batch;struct fsmonitor_batch *remainder =NULL ;intmax_t count =0 , duplicates =0 ; -kh_str_t *shown =NULL ; -int hash_ret; +struct strset shown =STRSET_INIT ;int do_trivial =0 ;int do_flush =0 ;int do_cookie =0 ;
@@ -882,14 +879,14 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,* so walk the batch list backwards from the current head back * to the batch (sequence number) they named. * -* We use khash to de-dup the list of pathnames. +* We use a strset to de-dup the list of pathnames. * * NEEDSWORK: each batch contains a list of interned strings, * so we only need to do pointer comparisons here to build the * hash table. Currently, we're still comparing the string * values. */ - shown =kh_init_str (); +strset_init_with_options (&shown,NULL ,0 );for (batch = batch_head; batch && batch->batch_seq_nr > requested_oldest_seq_nr; batch = batch->next) {
@@ -899,11 +896,9 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,const char *s = batch->interned_paths[k];size_t s_len; -if (kh_get_str (shown, s) !=kh_end (shown)) +if (!strset_add (&shown, s)) duplicates++;else { -kh_put_str (shown, s, &hash_ret); -trace_printf_key (&trace_fsmonitor,"send[%" PRIuMAX"]: %s" , count, s);
@@ -973,7 +968,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,trace2_data_intmax ("fsmonitor" , the_repository,"response/count/duplicates" , duplicates); cleanup: -kh_destroy_str (shown); +strset_clear (&shown);strbuf_release (&response_token);strbuf_release (&requested_token_id);strbuf_release (&payload);
Signature · 1 line
gitgitgadget
Quoted 4 lines · Paul Tarjan via GitGitGadget
Changes since v10:
* Reverted
pre_exec_cbcallback back to simpleclose_fd_above_stderrflag per Junio's clarification (same as v8)
One ask from my side: I would welcome it if you slowed down a bit with sending out new versions. Sending three different versions within a couple hours without any reviews in between makes me pause, as I have no idea whether the current version is good to be reviewed or whether I should expect another 5 rerolls.
So please take a bit more time to work feedback into your patch series before sending out the next version. It's totally fine to wait a couple days between iterations.
Also, could you please clarify whether the patch series has been written by AI and if so, which parts of it are?
Thanks!
Patrick
Quoted 5 lines · Patrick Steinhardt
One ask from my side: I would welcome it if you slowed down a bit with sending out new versions. Sending three different versions within a couple hours without any reviews in between makes me pause, as I have no idea whether the current version is good to be reviewed or whether I should expect another 5 rerolls.
Apologies for that. I thought integrating the feedback would be a quick fix since it was small and worked locally so I /submitted it without waiting for a Windows build. When it broke I fixed the Windows version, saw it passed and typed /submit. Then I got the feedback on the mailing list that it wasn't what Junio wanted so I undid that and noted in my front matter that (v11 == v8).
Quoted 3 lines
So please take a bit more time to work feedback into your patch series before sending out the next version. It's totally fine to wait a couple days between iterations.
Sorry for the quick submissions. I'm more used to the Github development flow where quick iteration on feedback is optimal to get a PR merged.
Quoted 2 lines
Also, could you please clarify whether the patch series has been written by AI and if so, which parts of it are?
I'm an Engineer at Anthropic so I've sadly completely lost my vim muscle that I built up over 20 years. All the file fwrite() operations were executed by my Claude wrapper but fully directed and iterated by me, with every line reviewed by me. That's just how I code nowadays. Claude is also an excellent code reviewer so it found multiple bugs in my code too. I'm aware of your policy and think I'm complying with it, but please let me know if you prefer something else:
https://git-scm.com/docs/SubmittingPatches#ai
These email replies are written by me but sent by Claude calling git commands as I've messed up formatting in the past trying to use Gmail and couldn't figure out the right git incantation to send to the mailing list, but Claude could.
Quoted 3 lines
Thanks!
Patrick
Quoted 16 lines · Patrick Steinhardt
Quoted 4 lines · Paul Tarjan via GitGitGadget
Changes since v10:
* Reverted
pre_exec_cbcallback back to simpleclose_fd_above_stderrflag per Junio's clarification (same as v8)One ask from my side: I would welcome it if you slowed down a bit with sending out new versions. Sending three different versions within a couple hours without any reviews in between makes me pause, as I have no idea whether the current version is good to be reviewed or whether I should expect another 5 rerolls.
So please take a bit more time to work feedback into your patch series before sending out the next version. It's totally fine to wait a couple days between iterations.
Quoted 6 lines
Also, could you please clarify whether the patch series has been written by AI and if so, which parts of it are?
Thanks!
Patrick
The topic went totally silent after this message; is this topic still viable?
Thanks.
Re: [PATCH v11 0/8] fsmonitor: add Linux support using inotify
Quoted 2 lines · Junio C Hamano
The topic went totally silent after this message; is this topic still viable?
Were you asking me, or Patrick? On my end: yes, very much. I've been using this across our fleet for months and would love to see it upstream.
I replied to Patrick on Mar 5 (slowed down, answered the AI question) and haven't sent a new version since. v11 is identical to v8.
Happy to do whatever's needed. Just let me know who the ball is with.
(Also a small process question: I keep getting gitgitgadget emails about the branch being integrated into "seen" every few days. Is that just informational, or does it mean something I should act on?)
Thanks
Paul
Quoted 21 lines · Junio C Hamano
Quoted 19 lines · Patrick Steinhardt
Quoted 4 lines · Paul Tarjan via GitGitGadget
Changes since v10:
* Reverted
pre_exec_cbcallback back to simpleclose_fd_above_stderrflag per Junio's clarification (same as v8)One ask from my side: I would welcome it if you slowed down a bit with sending out new versions. Sending three different versions within a couple hours without any reviews in between makes me pause, as I have no idea whether the current version is good to be reviewed or whether I should expect another 5 rerolls.
So please take a bit more time to work feedback into your patch series before sending out the next version. It's totally fine to wait a couple days between iterations.
Also, could you please clarify whether the patch series has been written by AI and if so, which parts of it are?
Quoted 6 lines
Quoted 3 lines
Thanks!
Patrick
The topic went totally silent after this message; is this topic still viable?
I didn't have the capacity recently to focus on this patch series.
Also, I think we need to be extra careful with merging this patch series as it's written mostly by AI as far as I understood. So I don't quite feel comfortable to ACK this series by myself, as I'm not sure to what degree the code is understood and reasoned about by anyone.
I'm not against merging it in general, but if we want to merge it I would like to have at least one more thorough review by somebody who understands this area before I feel comfortable committing to it and maintaining the code going forward.
Thanks!
Patrick
Quoted 21 lines · Patrick Steinhardt
Quoted 7 lines
Quoted 7 lines
So please take a bit more time to work feedback into your patch series before sending out the next version. It's totally fine to wait a couple days between iterations.
Also, could you please clarify whether the patch series has been written by AI and if so, which parts of it are?
Quoted 6 lines
Quoted 3 lines
Thanks!
Patrick
The topic went totally silent after this message; is this topic still viable?
I didn't have the capacity recently to focus on this patch series.
Also, I think we need to be extra careful with merging this patch series as it's written mostly by AI as far as I understood. So I don't quite feel comfortable to ACK this series by myself, as I'm not sure to what degree the code is understood and reasoned about by anyone.
Quoted 4 lines
I'm not against merging it in general, but if we want to merge it I would like to have at least one more thorough review by somebody who understands this area before I feel comfortable committing to it and maintaining the code going forward.
I share that exact feeling. After all, the section recently added to SubmittingPatches about AI-generated code was to address this kind of patches.
In any case, when this topic is merged to 'seen', it seem to break linux-TEST-vars CI job, which essentially runs the build and test with these settings:
export OPENSSL_SHA1_UNSAFE=YesPlease export GIT_TEST_SPLIT_INDEX=yes
export GIT_TEST_FULL_IN_PACK_ARRAY=true export GIT_TEST_OE_SIZE=10
export GIT_TEST_OE_DELTA_SIZE=5
export GIT_TEST_COMMIT_GRAPH=1
export GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1 export GIT_TEST_MULTI_PACK_INDEX=1
export GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=1 export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master export GIT_TEST_NO_WRITE_REV_INDEX=1 export GIT_TEST_CHECKOUT_WORKERS=2
export GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL=1
and t9210 fails with this topic merged (abfd972), and succeeds without (4b8355c6f8).
https://github.com/git/git/actions/runs/23775508256/job/69276399578#step:10:562
https://github.com/git/git/actions/runs/23777158230/job/69281498657
I also reproduced the breakage locally on a debian-testing derivative.
Help to diagnose and pinpoint the bad interactions with topics in flight is greatly appreciated.
! [abfd972] Merge branch 'pt/fsmonitor-linux' into seen
! [4b8355c6f8] Merge branch 'ps/receive-pack-updateinstead-in-worktree' into seen
--
- [abfd972] Merge branch 'pt/fsmonitor-linux' into seen + [abfd972^2] fsmonitor: convert shown khash to strset in do_handle_client + [abfd972^2^] fsmonitor: add tests for Linux + [abfd972^2~2] fsmonitor: add timeout to daemon stop command + [abfd972^2~3] fsmonitor: close inherited file descriptors and detach in daemon + [abfd972^2~4] run-command: add close_fd_above_stderr option + [abfd972^2~5] fsmonitor: implement filesystem change listener for Linux + [abfd972^2~6] fsmonitor: rename fsm-settings-darwin.c to fsm-settings-unix.c + [abfd972^2~7] fsmonitor: rename fsm-ipc-darwin.c to fsm-ipc-unix.c + [abfd972^2~8] fsmonitor: use pthread_cond_timedwait for cookie wait + [abfd972^2~9] compat/win32: add pthread_cond_timedwait + [abfd972^2~10] fsmonitor: fix hashmap memory leak in fsmonitor_run_daemon + [abfd972^2~11] fsmonitor: fix khash memory leak in do_handle_client
-- [4b8355c6f8] Merge branch 'ps/receive-pack-updateinstead-in-worktree' into seen
Re: [PATCH v11 0/8] fsmonitor: add Linux support using inotify
Quoted 9 lines · Junio C Hamano
In any case, when this topic is merged to 'seen', it seem to break linux-TEST-vars CI job, which essentially runs the build and test with these settings:
export GIT_TEST_SPLIT_INDEX=yes [...]and t9210 fails with this topic merged (abfd972), and succeeds without (
4b8355c6f8).
Thanks for the pointer. The failure is:
BUG: fsmonitor.c:27: fsmonitor_dirty has more entries than the index (2 > 0)
It's a pre-existing bug in the interaction between fsmonitor and split-index, now exposed because fsmonitor-linux enables the daemon on Linux for the first time (so scalar clone actually starts it). It needs the interaction with ps/setup-wo-the-repository in seen to trigger.
This is related to the issue that 05f28e4b3c ("scalar: use index.skipHash=true for performance", 2025-06-04) worked around by disabling GIT_TEST_SPLIT_INDEX in t9210, noting "the issue should be resolved in a series focused on the split index." My fix covers the fsmonitor bitmap side; the index.skipHash interaction remains.
The fsmonitor_dirty EWAH bitmap can reference positions from a previous index state. With split-index, cache_nr can be smaller than the bitmap expects. Two places hit this:
1. tweak_fsmonitor() calls assert_index_minimum() without the !istate->split_index guard that the read path (line 98) and write path (line 128) already have.
2. fsmonitor_ewah_callback() unconditionally asserts and then accesses istate->cache[pos], which is out of bounds when split-index hasn't merged all entries yet.
I reproduced this in a Fedora container by merging fsmonitor-linux into seen and running t9210 with GIT_TEST_SPLIT_INDEX=yes. Fails without the fix, passes with it.
Fix is pushed as a new commit at the end of the series:
fsmonitor_ewah_callback():
- assert_index_minimum(istate, pos + 1);
+ if (pos >= istate->cache_nr)
+ return;
tweak_fsmonitor():
- assert_index_minimum(istate, istate->fsmonitor_dirty->bit_size);
+ if (!istate->split_index)
+ assert_index_minimum(istate, istate->fsmonitor_dirty->bit_size);
Paul
Re: [PATCH v11 0/8] fsmonitor: add Linux support using inotify
Quoted 11 lines · Paul Tarjan
I reproduced this in a Fedora container by merging fsmonitor-linux into seen and running t9210 with GIT_TEST_SPLIT_INDEX=yes. Fails without the fix, passes with it.
Fix is pushed as a new commit at the end of the series:
fsmonitor_ewah_callback():
-assert_index_minimum(istate, pos + 1);
+ if (pos >=istate->cache_nr)
+ return;
Quoted 6 lines
tweak_fsmonitor():
-assert_index_minimum(istate, istate->fsmonitor_dirty->bit_size);
+ if (!istate->split_index)
+assert_index_minimum(istate, istate->fsmonitor_dirty->bit_size);Paul
Thanks for a quick turnaround.
If it is a preexisting breakage in the original and not what you introduced, wouldn't it make sense to apply the fix very early in the series of patches, instead of patching breakage on top? If we queue on top of what we have in 'seen', this topic will still be broken up to [8/9] in the same place, breaking bisectability, and fixed only when the last patch is applied.
v12 cover fsmonitor: implement filesystem change listener for Linux
In reply to Paul Tarjan via GitGitGadgetThis series implements the built-in fsmonitor daemon for Linux using the inotify API, bringing it to feature parity with the existing Windows and macOS implementations. It also fixes two memory leaks in the platform-independent daemon code and deduplicates the IPC and settings logic that is now shared between macOS and Linux.
The implementation uses inotify rather than fanotify because fanotify requires either CAP_SYS_ADMIN or CAP_PERFMON capabilities, making it unsuitable for an unprivileged user-space daemon. While inotify has the limitation of requiring a separate watch on every directory (unlike macOS FSEvents, which can monitor an entire directory tree with a single watch), it operates without elevated privileges and provides the per-file event granularity needed for fsmonitor.
The listener uses inotify_init1(O_NONBLOCK) with a poll loop that checks for events with a 50-millisecond timeout, keeping the inotify queue well-drained to minimize the risk of overflows. Bidirectional hashmaps map between watch descriptors and directory paths for efficient event resolution. Directory renames are tracked using inotify cookie mechanism to correlate IN_MOVED_FROM and IN_MOVED_TO event pairs; a periodic check detects stale renames where the matching IN_MOVED_TO never arrived, forcing a resync.
New directory creation triggers recursive watch registration to ensure all subdirectories are monitored. The IN_MASK_CREATE flag is used where available to prevent modifying existing watches, with a fallback for older kernels. When IN_MASK_CREATE is available and inotify_add_watch returns EEXIST, it means another thread or recursive scan has already registered the watch, so it is safe to ignore.
Remote filesystem detection uses statfs() to identify network-mounted filesystems (NFS, CIFS, SMB, FUSE, etc.) via their magic numbers. Mount point information is read from /proc/mounts and matched against the statfs f_fsid to get accurate, human-readable filesystem type names for logging. When the .git directory is on a remote filesystem, the IPC socket falls back to $HOME or a user-configured directory via the fsmonitor.socketDir setting.
This series builds on work from https://github.com/git/git/pull/1352 by Eric DeCosta and https://github.com/git/git/pull/1667 by Marziyeh Esipreh, updated to work with the current codebase and address all review feedback.
Changes since v11:
* Fix t9210 BUG assertion with GIT_TEST_SPLIT_INDEX=yes: guard fsmonitor_dirty bitmap access against split-index having fewer entries than the bitmap expects
Changes since v10:
* Reverted pre_exec_cb callback back to simple close_fd_above_stderr flag per Junio's clarification (same as v8)
Changes since v9:
* Fixed Windows build: close_fd_above_stderr() compiles as a no-op on Windows since there is no fork/exec
Changes since v8:
* Replaced close_fd_above_stderr flag with generic pre_exec_cb callback in struct child_process per Junio's review, with close_fd_above_stderr() as a ready-made callback
Changes since v7:
* Added patch 12: convert khash to strset in do_handle_client (Patrick's #leftoverbit suggestion)
* Fixed "Forcing shutdown" trace message to start with lowercase
* Fixed redundant statfs() call in find_mount() (caller already had the result)
* Fixed CMakeLists.txt GIT-BUILD-OPTIONS: was hardcoded to "win32" for FSMONITOR_DAEMON_BACKEND and FSMONITOR_OS_SETTINGS, now uses the CMake variables
* Fixed uninitialized strset on trivial response path (STRSET_INIT)
* Removed V9FS_MAGIC from get_fs_typename() to match is_remote_fs() (9p is local VM mounts)
* Split 30-second stop timeout into its own commit per review request
* Fixed misleading indentation on shutdown assignment in handle_events()
* Updated commit messages to describe all changes (test hardening, fsmonitor-ipc.c spawn changes)
* Updated Makefile comment for FSMONITOR_OS_SETTINGS to mention fsm-ipc
Changes since v6:
* Introduced FSMONITOR_OS_SETTINGS build variable (set to "unix" for macOS and Linux, "win32" for Windows) to eliminate if/else conditionals in Makefile, meson.build, and CMakeLists.txt per Junio's review
* Moved fsm-path-utils from FSMONITOR_OS_SETTINGS to FSMONITOR_DAEMON_BACKEND since path-utils files are platform-specific
* Removed V9FS_MAGIC from remote filesystem detection (9p is used for local VM/container host mounts where fsmonitor works fine)
* Removed redundant #include <libgen.h> (already provided by compat/posix.h)
* Fixed cookie wait comment wording ("see" to "observe")
* Rewrote commit messages for IPC and settings dedup patches
Changes since v5:
* Split monolithic commit into 10-patch series per Patrick's review
* Deduplicated fsm-ipc and fsm-settings into shared Unix implementations
* Rewrote commit message with prose paragraphs, explain inotify vs fanotify, removed "Issues addressed" sections, added Based-on-patch-by trailers
* Removed redundant includes already provided by compat/posix.h
* Fixed error/trace message capitalization per coding guidelines
* Fixed stale rename check interval from 1000 seconds to 1 second
* Changed poll timeout from 1ms to 50ms to reduce idle CPU wake-ups
* Replaced infinite pthread_cond_wait cookie loop with one-second pthread_cond_timedwait (prevents daemon hangs on overlay filesystems where events are never delivered)
* Added pthread_cond_timedwait to Windows pthread compatibility layer
* Separated test into its own commit with smoke test that skips when inotify events are not delivered (e.g., overlayfs with older kernels)
* Fixed test hang on Fedora CI: stop_git() looped forever when ps was unavailable because bash in POSIX/sh mode returns exit 0 from kill with an empty process group argument. Fixed by falling back to /proc/$pid/stat for process group ID and guarding stop_git against empty pgid.
* Redirect spawn_daemon() stdout/stderr to /dev/null and close inherited file descriptors to prevent the intermediate process from holding test pipe file descriptors
* Call setsid() on daemon detach to prevent shells with job control from waiting on the daemon process group
* Close inherited file descriptors 3-7 in the test watchdog subprocess
* Added 30-second timeout to "fsmonitor--daemon stop" to prevent indefinite blocking
* Added helpful error message when inotify watch limit (max_user_watches) is reached
* Initialize fd_inotify to -1 and use fd >= 0 check for correct fd 0 handling
* Use sysconf(_SC_OPEN_MAX) instead of hardcoded 1024 for fd close limit
* Check setsid() return value
Changes since v4:
* Added Meson build support
Changes since v3:
* Fix crash on rapid nested directory creation (EEXIST from inotify_add_watch with IN_MASK_CREATE)
* Extensive stress testing
Changes since v2:
* Fix khash memory leak in do_handle_client
Changes since v1:
* Fix hashmap memory leak in fsmonitor_run_daemon()
Paul Tarjan (13):
fsmonitor: fix khash memory leak in do_handle_client
fsmonitor: fix hashmap memory leak in fsmonitor_run_daemon compat/win32: add pthread_cond_timedwait
fsmonitor: use pthread_cond_timedwait for cookie wait
fsmonitor: rename fsm-ipc-darwin.c to fsm-ipc-unix.c
fsmonitor: rename fsm-settings-darwin.c to fsm-settings-unix.c
fsmonitor: implement filesystem change listener for Linux
run-command: add close_fd_above_stderr option
fsmonitor: close inherited file descriptors and detach in daemon
fsmonitor: add timeout to daemon stop command
fsmonitor: add tests for Linux
fsmonitor: convert shown khash to strset in do_handle_client
fsmonitor: fix split-index bitmap bounds in tweak_fsmonitor()
Documentation/config/fsmonitor--daemon.adoc | 4 +-
Documentation/git-fsmonitor--daemon.adoc | 28 +-
Makefile | 6 +-
builtin/fsmonitor--daemon.c | 92 ++-
compat/fsmonitor/fsm-health-linux.c | 33 +
.../{fsm-ipc-darwin.c => fsm-ipc-unix.c} | 0
compat/fsmonitor/fsm-listen-linux.c | 746 ++++++++++++++++++
compat/fsmonitor/fsm-path-utils-linux.c | 217 +++++
...-settings-darwin.c => fsm-settings-unix.c} | 0
compat/win32/pthread.c | 26 +
compat/win32/pthread.h | 2 +
config.mak.uname | 12 +-
contrib/buildsystems/CMakeLists.txt | 33 +-
fsmonitor-ipc.c | 3 +
fsmonitor.c | 6 +-
meson.build | 13 +-
run-command.c | 12 +
run-command.h | 9 +
t/t7527-builtin-fsmonitor.sh | 89 ++-
19 files changed, 1266 insertions(+), 65 deletions(-)
create mode 100644 compat/fsmonitor/fsm-health-linux.c
rename compat/fsmonitor/{fsm-ipc-darwin.c => fsm-ipc-unix.c} (100%)
create mode 100644 compat/fsmonitor/fsm-listen-linux.c
create mode 100644 compat/fsmonitor/fsm-path-utils-linux.c
rename compat/fsmonitor/{fsm-settings-darwin.c => fsm-settings-unix.c} (100%)base-commit: 3e0db84c88c57e70ac8be8c196dfa92c5d656fbc
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-2147%2Fptarjan%2Fclaude%2Fupdate-pr-1352-current-85Gk8-v12
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-2147/ptarjan/claude/update-pr-1352-current-85Gk8-v12
Pull-Request: https://github.com/git/git/pull/2147
Range-diff vs v11:
1: 4d4dec8fa1 = 1: 4d4dec8fa1 fsmonitor: fix khash memory leak in do_handle_client 2: cb270120f0 = 2: cb270120f0 fsmonitor: fix hashmap memory leak in fsmonitor_run_daemon 3: 44a063074d = 3: 44a063074d compat/win32: add pthread_cond_timedwait 4: b1081d1e13 = 4: b1081d1e13 fsmonitor: use pthread_cond_timedwait for cookie wait 5: dec0fb144f = 5: dec0fb144f fsmonitor: rename fsm-ipc-darwin.c to fsm-ipc-unix.c 6: b2aaadb4ae = 6: b2aaadb4ae fsmonitor: rename fsm-settings-darwin.c to fsm-settings-unix.c 7: 03cf12d01b = 7: 03cf12d01b fsmonitor: implement filesystem change listener for Linux 8: 50f5b4676e = 8: 50f5b4676e run-command: add close_fd_above_stderr option 9: 057b3098bc = 9: 057b3098bc fsmonitor: close inherited file descriptors and detach in daemon 10: e6bc3bfcb2 = 10: e6bc3bfcb2 fsmonitor: add timeout to daemon stop command 11: 81f8cd1599 = 11: 81f8cd1599 fsmonitor: add tests for Linux 12: 8fa6a74e0d = 12: 8fa6a74e0d fsmonitor: convert shown khash to strset in do_handle_client -: ---------- > 13: 84ddbb30bb fsmonitor: fix split-index bitmap bounds in tweak_fsmonitor()
Signature · 1 line
gitgitgadget
PATCH v12 1/13 fsmonitor: fix khash memory leak in do_handle_client
From: Paul Tarjan <[email protected]>
The shown kh_str_t was freed with kh_release_str() at a point in the code only reachable in the non-trivial response path. When the client receives a trivial response, the code jumps to the cleanup label, skipping the kh_release_str() call entirely and leaking the hash table.
Fix this by initializing shown to NULL and moving the cleanup to the cleanup label using kh_destroy_str(), which is safe to call on NULL. This ensures the hash table is freed regardless of which code path is taken.
--- builtin/fsmonitor--daemon.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
builtin/fsmonitor--daemon.c +2 -3
@@ -671,7 +671,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,const struct fsmonitor_batch *batch;struct fsmonitor_batch *remainder =NULL ;intmax_t count =0 , duplicates =0 ; -kh_str_t *shown; +kh_str_t *shown =NULL ;int hash_ret;int do_trivial =0 ;int do_flush =0 ;
@@ -909,8 +909,6 @@ static int do_handle_client(struct fsmonitor_daemon_state *state, total_response_len += payload.len; } -kh_release_str (shown); -pthread_mutex_lock (&state->main_lock);if (token_data->client_ref_count >0 )
@@ -954,6 +952,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,trace2_data_intmax ("fsmonitor" , the_repository,"response/count/duplicates" , duplicates); cleanup: +kh_destroy_str (shown);strbuf_release (&response_token);strbuf_release (&requested_token_id);strbuf_release (&payload);
Signature · 1 line
gitgitgadget
PATCH v12 2/13 fsmonitor: fix hashmap memory leak in fsmonitor_run_daemon
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
The state.cookies hashmap is initialized during daemon startup but never freed during cleanup in the done: label of fsmonitor_run_daemon(). The cookie entries also have names allocated via strbuf_detach() that must be freed individually.
Iterate the hashmap to free each cookie name, then call hashmap_clear_and_free() to release the entries and table.
--- builtin/fsmonitor--daemon.c | 9 +++++++++ 1 file changed, 9 insertions(+)
builtin/fsmonitor--daemon.c +9 -0
@@ -1404,6 +1404,15 @@ static int fsmonitor_run_daemon(void) done:pthread_cond_destroy (&state.cookies_cond);pthread_mutex_destroy (&state.main_lock); + { +struct hashmap_iter iter; +struct fsmonitor_cookie_item *cookie; + +hashmap_for_each_entry (&state.cookies, &iter, cookie, entry) +free (cookie->name); +hashmap_clear_and_free (&state.cookies, + struct fsmonitor_cookie_item, entry); + }fsm_listen__dtor (&state);fsm_health__dtor (&state);
Signature · 1 line
gitgitgadget
PATCH v12 3/13 compat/win32: add pthread_cond_timedwait
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
Add a pthread_cond_timedwait() implementation to the Windows pthread compatibility layer using SleepConditionVariableCS() with a millisecond timeout computed from the absolute deadline.
--- compat/win32/pthread.c | 26 ++++++++++++++++++++++++++ compat/win32/pthread.h | 2 ++ 2 files changed, 28 insertions(+)
compat/win32/pthread.c +26 -0
@@ -66,3 +66,29 @@ int pthread_cond_wait(pthread_cond_t *cond, pthread_mutex_t *mutex)return err_win_to_posix (GetLastError ());return 0 ; } + +int pthread_cond_timedwait (pthread_cond_t *cond,pthread_mutex_t *mutex, +const struct timespec *abstime) +{ +struct timeval now; +long long now_ms, deadline_ms; +DWORD timeout_ms; + +gettimeofday (&now,NULL ); + now_ms = (long long )now.tv_sec *1000 + now.tv_usec /1000 ; + deadline_ms = (long long )abstime->tv_sec *1000 + + abstime->tv_nsec /1000000 ; + +if (deadline_ms <= now_ms) +return ETIMEDOUT ; +else + timeout_ms = (DWORD )(deadline_ms - now_ms); + +if (SleepConditionVariableCS (cond, mutex, timeout_ms) ==0 ) { +DWORD err =GetLastError (); +if (err ==ERROR_TIMEOUT ) +return ETIMEDOUT ; +return err_win_to_posix (err); + } +return 0 ; +}
compat/win32/pthread.h +2 -0
@@ -64,6 +64,8 @@ int win32_pthread_join(pthread_t *thread, void **value_ptr);pthread_t pthread_self (void );int pthread_cond_wait (pthread_cond_t *cond,pthread_mutex_t *mutex); +int pthread_cond_timedwait (pthread_cond_t *cond,pthread_mutex_t *mutex, +const struct timespec *abstime); static inline void NORETURN pthread_exit(void *ret) {
Signature · 1 line
gitgitgadget
PATCH v12 4/13 fsmonitor: use pthread_cond_timedwait for cookie wait
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
The cookie wait in with_lock__wait_for_cookie() uses an infinite pthread_cond_wait() loop. The existing comment notes the desire to switch to pthread_cond_timedwait(), but the routine was not available in git thread-utils.
On certain container or overlay filesystems, inotify watches may succeed but events are never delivered. In this case the daemon would hang indefinitely waiting for the cookie event, which in turn causes the client to hang.
Replace the infinite wait with a one-second timeout using pthread_cond_timedwait(). If the timeout fires, report an error and let the client proceed with a trivial (full-scan) response rather than blocking forever.
--- builtin/fsmonitor--daemon.c | 37 ++++++++++++++++++++++++------------- 1 file changed, 24 insertions(+), 13 deletions(-)
builtin/fsmonitor--daemon.c +24 -13
@@ -197,20 +197,31 @@ static enum fsmonitor_cookie_item_result with_lock__wait_for_cookie(unlink (cookie_pathname.buf);/* -* Technically, this is an infinite wait (well, unless another -* thread sends us an abort). I'd like to change this to -* use `pthread_cond_timedwait()` and return an error/timeout -* and let the caller do the trivial response thing, but we -* don't have that routine in our thread-utils. -* -* After extensive beta testing I'm not really worried about -* this. Also note that the above open() and unlink() calls -* will cause at least two FS events on that path, so the odds -* of getting stuck are pretty slim. +* Wait for the listener thread to observe the cookie file. +* Time out after a short interval so that the client +* does not hang forever if the filesystem does not deliver +* events (e.g., on certain container/overlay filesystems +* where inotify watches succeed but events never arrive). */ -while (cookie->result ==FCIR_INIT ) -pthread_cond_wait (&state->cookies_cond, - &state->main_lock); + { +struct timeval now; +struct timespec ts; +int err =0 ; + +gettimeofday (&now,NULL ); + ts.tv_sec = now.tv_sec +1 ; + ts.tv_nsec = now.tv_usec *1000 ; + +while (cookie->result ==FCIR_INIT && !err) + err =pthread_cond_timedwait (&state->cookies_cond, + &state->main_lock, + &ts); +if (err ==ETIMEDOUT && cookie->result ==FCIR_INIT ) { +trace_printf_key (&trace_fsmonitor, +"cookie_wait timed out" ); + cookie->result =FCIR_ERROR ; + } + } done:hashmap_remove (&state->cookies, &cookie->entry,NULL );
Signature · 1 line
gitgitgadget
PATCH v12 5/13 fsmonitor: rename fsm-ipc-darwin.c to fsm-ipc-unix.c
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
The fsmonitor IPC path logic in fsm-ipc-darwin.c is not Darwin-specific and will be reused by the upcoming Linux implementation. Rename it to fsm-ipc-unix.c to reflect that it is shared by all Unix platforms.
Introduce FSMONITOR_OS_SETTINGS (set to "unix" for non-Windows, "win32" for Windows) as a separate variable from FSMONITOR_DAEMON_BACKEND so that the build files can distinguish between platform-specific files (listen, health, path-utils) and shared Unix files (ipc, settings).
Move fsm-ipc to the FSMONITOR_OS_SETTINGS section in the Makefile, and switch fsm-path-utils to use FSMONITOR_DAEMON_BACKEND since path-utils is platform-specific (there will be separate darwin and linux versions).
---
Makefile | 6 ++---
.../{fsm-ipc-darwin.c => fsm-ipc-unix.c} | 0
config.mak.uname | 2 +-
contrib/buildsystems/CMakeLists.txt | 25 +++++++++----------
meson.build | 7 ++++--
5 files changed, 21 insertions(+), 19 deletions(-)
rename compat/fsmonitor/{fsm-ipc-darwin.c => fsm-ipc-unix.c} (100%)Makefile +3 -3
@@ -417,7 +417,7 @@ include shared.mak # If your platform has OS-specific ways to tell if a repo is incompatible with # fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS # to the "<name>" of the corresponding `compat/fsmonitor/fsm-settings-<name>.c` -# that implements the `fsm_os_settings__*()` routines. +# and `compat/fsmonitor/fsm-ipc-<name>.c` files. # # Define LINK_FUZZ_PROGRAMS if you want `make all` to also build the fuzz test # programs in oss-fuzz/.
@@ -2365,13 +2365,13 @@ ifdef FSMONITOR_DAEMON_BACKEND COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o - COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_BACKEND).o endif ifdef FSMONITOR_OS_SETTINGS COMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS + COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_OS_SETTINGS).o COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o - COMPAT_OBJS += compat/fsmonitor/fsm-path-utils-$(FSMONITOR_OS_SETTINGS).o + COMPAT_OBJS += compat/fsmonitor/fsm-path-utils-$(FSMONITOR_DAEMON_BACKEND).o endif ifdef WITH_BREAKING_CHANGES
compat/fsmonitor/fsm-ipc-unix.c renamed from compat/fsmonitor/fsm-ipc-darwin.c +0 -0
config.mak.uname +1 -1
@@ -165,7 +165,7 @@ ifeq ($(uname_S),Darwin) ifndef NO_PTHREADS ifndef NO_UNIX_SOCKETS FSMONITOR_DAEMON_BACKEND = darwin - FSMONITOR_OS_SETTINGS = darwin + FSMONITOR_OS_SETTINGS = unix endif endif
contrib/buildsystems/CMakeLists.txt +12 -13
@@ -291,23 +291,22 @@ endif() if(SUPPORTS_SIMPLE_IPC) if(CMAKE_SYSTEM_NAME STREQUAL "Windows") - add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-win32.c) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-win32.c) - - add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c) + set(FSMONITOR_DAEMON_BACKEND "win32") + set(FSMONITOR_OS_SETTINGS "win32") elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") + set(FSMONITOR_DAEMON_BACKEND "darwin") + set(FSMONITOR_OS_SETTINGS "unix") + endif() + + if(FSMONITOR_DAEMON_BACKEND) add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-darwin.c) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-darwin.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-${FSMONITOR_DAEMON_BACKEND}.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-${FSMONITOR_DAEMON_BACKEND}.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-${FSMONITOR_OS_SETTINGS}.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-${FSMONITOR_DAEMON_BACKEND}.c) add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-${FSMONITOR_DAEMON_BACKEND}.c) endif() endif()
meson.build +5 -2
@@ -1320,10 +1320,13 @@ else endif fsmonitor_backend = '' +fsmonitor_os = '' if host_machine.system() == 'windows' fsmonitor_backend = 'win32' + fsmonitor_os = 'win32' elif host_machine.system() == 'darwin' fsmonitor_backend = 'darwin' + fsmonitor_os = 'unix' libgit_dependencies += dependency('CoreServices') endif if fsmonitor_backend != ''
@@ -1332,14 +1335,14 @@ if fsmonitor_backend != '' libgit_sources += [ 'compat/fsmonitor/fsm-health-' + fsmonitor_backend + '.c', - 'compat/fsmonitor/fsm-ipc-' + fsmonitor_backend + '.c', + 'compat/fsmonitor/fsm-ipc-' + fsmonitor_os + '.c', 'compat/fsmonitor/fsm-listen-' + fsmonitor_backend + '.c', 'compat/fsmonitor/fsm-path-utils-' + fsmonitor_backend + '.c', 'compat/fsmonitor/fsm-settings-' + fsmonitor_backend + '.c', ] endif build_options_config.set_quoted('FSMONITOR_DAEMON_BACKEND', fsmonitor_backend) -build_options_config.set_quoted('FSMONITOR_OS_SETTINGS', fsmonitor_backend) +build_options_config.set_quoted('FSMONITOR_OS_SETTINGS', fsmonitor_os) if not get_option('b_sanitize').contains('address') and get_option('regex').allowed() and compiler.has_header('regex.h') and compiler.get_define('REG_STARTEND', prefix: '#include <regex.h>') != '' build_options_config.set('NO_REGEX', '')
Signature · 1 line
gitgitgadget
PATCH v12 6/13 fsmonitor: rename fsm-settings-darwin.c to fsm-settings-unix.c
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
The fsmonitor settings logic in fsm-settings-darwin.c is not Darwin-specific and will be reused by the upcoming Linux implementation. Rename it to fsm-settings-unix.c to reflect that it is shared by all Unix platforms.
Update the build files (meson.build and CMakeLists.txt) to use FSMONITOR_OS_SETTINGS for fsm-settings, matching the approach already used for fsm-ipc.
---
compat/fsmonitor/{fsm-settings-darwin.c => fsm-settings-unix.c} | 0
contrib/buildsystems/CMakeLists.txt | 2 +-
meson.build | 2 +-
3 files changed, 2 insertions(+), 2 deletions(-)
rename compat/fsmonitor/{fsm-settings-darwin.c => fsm-settings-unix.c} (100%)compat/fsmonitor/fsm-settings-unix.c renamed from compat/fsmonitor/fsm-settings-darwin.c +0 -0
contrib/buildsystems/CMakeLists.txt +1 -1
@@ -306,7 +306,7 @@ if(SUPPORTS_SIMPLE_IPC) list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-${FSMONITOR_DAEMON_BACKEND}.c) add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-${FSMONITOR_DAEMON_BACKEND}.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-${FSMONITOR_OS_SETTINGS}.c) endif() endif()
meson.build +1 -1
@@ -1338,7 +1338,7 @@ if fsmonitor_backend != '' 'compat/fsmonitor/fsm-ipc-' + fsmonitor_os + '.c', 'compat/fsmonitor/fsm-listen-' + fsmonitor_backend + '.c', 'compat/fsmonitor/fsm-path-utils-' + fsmonitor_backend + '.c', - 'compat/fsmonitor/fsm-settings-' + fsmonitor_backend + '.c', + 'compat/fsmonitor/fsm-settings-' + fsmonitor_os + '.c', ] endif build_options_config.set_quoted('FSMONITOR_DAEMON_BACKEND', fsmonitor_backend)
Signature · 1 line
gitgitgadget
PATCH v12 7/13 fsmonitor: implement filesystem change listener for Linux
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
Implement the built-in fsmonitor daemon for Linux using the inotify API, bringing it to feature parity with the existing Windows and macOS implementations.
The implementation uses inotify rather than fanotify because fanotify requires either CAP_SYS_ADMIN or CAP_PERFMON capabilities, making it unsuitable for an unprivileged user-space daemon. While inotify has the limitation of requiring a separate watch on every directory (unlike macOS's FSEvents, which can monitor an entire directory tree with a single watch), it operates without elevated privileges and provides the per-file event granularity needed for fsmonitor.
The listener uses inotify_init1(O_NONBLOCK) with a poll loop that checks for events with a 50-millisecond timeout, keeping the inotify queue well-drained to minimize the risk of overflows. Bidirectional hashmaps map between watch descriptors and directory paths for efficient event resolution. Directory renames are tracked using inotify's cookie mechanism to correlate IN_MOVED_FROM and IN_MOVED_TO event pairs; a periodic check detects stale renames where the matching IN_MOVED_TO never arrived, forcing a resync.
New directory creation triggers recursive watch registration to ensure all subdirectories are monitored. The IN_MASK_CREATE flag is used where available to prevent modifying existing watches, with a fallback for older kernels. When IN_MASK_CREATE is available and inotify_add_watch returns EEXIST, it means another thread or recursive scan has already registered the watch, so it is safe to ignore.
Remote filesystem detection uses statfs() to identify network-mounted filesystems (NFS, CIFS, SMB, FUSE, etc.) via their magic numbers. Mount point information is read from /proc/mounts and matched against the statfs f_fsid to get accurate, human-readable filesystem type names for logging. When the .git directory is on a remote filesystem, the IPC socket falls back to $HOME or a user-configured directory via the fsmonitor.socketDir setting.
--- Documentation/config/fsmonitor--daemon.adoc | 4 +- Documentation/git-fsmonitor--daemon.adoc | 28 +- compat/fsmonitor/fsm-health-linux.c | 33 + compat/fsmonitor/fsm-listen-linux.c | 746 ++++++++++++++++++++ compat/fsmonitor/fsm-path-utils-linux.c | 217 ++++++ config.mak.uname | 10 + contrib/buildsystems/CMakeLists.txt | 8 +- meson.build | 4 + 8 files changed, 1042 insertions(+), 8 deletions(-) create mode 100644 compat/fsmonitor/fsm-health-linux.c create mode 100644 compat/fsmonitor/fsm-listen-linux.c create mode 100644 compat/fsmonitor/fsm-path-utils-linux.c
Documentation/config/fsmonitor--daemon.adoc +2 -2
@@ -4,8 +4,8 @@ fsmonitor.allowRemote:: behavior. Only respected when `core.fsmonitor` is set to `true`. fsmonitor.socketDir:: - This Mac OS-specific option, if set, specifies the directory in + This Mac OS and Linux-specific option, if set, specifies the directory in which to create the Unix domain socket used for communication between the fsmonitor daemon and various Git commands. The directory must - reside on a native Mac OS filesystem. Only respected when `core.fsmonitor` + reside on a native filesystem. Only respected when `core.fsmonitor` is set to `true`.
Documentation/git-fsmonitor--daemon.adoc +24 -4
@@ -76,9 +76,9 @@ repositories; this may be overridden by setting `fsmonitor.allowRemote` to correctly with all network-mounted repositories, so such use is considered experimental. -On Mac OS, the inter-process communication (IPC) between various Git +On Mac OS and Linux, the inter-process communication (IPC) between various Git commands and the fsmonitor daemon is done via a Unix domain socket (UDS) -- a -special type of file -- which is supported by native Mac OS filesystems, +special type of file -- which is supported by native Mac OS and Linux filesystems, but not on network-mounted filesystems, NTFS, or FAT32. Other filesystems may or may not have the needed support; the fsmonitor daemon is not guaranteed to work with these filesystems and such use is considered experimental.
@@ -87,13 +87,33 @@ By default, the socket is created in the `.git` directory. However, if the `.git` directory is on a network-mounted filesystem, it will instead be created at `$HOME/.git-fsmonitor-*` unless `$HOME` itself is on a network-mounted filesystem, in which case you must set the configuration -variable `fsmonitor.socketDir` to the path of a directory on a Mac OS native +variable `fsmonitor.socketDir` to the path of a directory on a native filesystem in which to create the socket file. If none of the above directories (`.git`, `$HOME`, or `fsmonitor.socketDir`) -is on a native Mac OS file filesystem the fsmonitor daemon will report an +is on a native filesystem the fsmonitor daemon will report an error that will cause the daemon and the currently running command to exit. +LINUX CAVEATS +~~~~~~~~~~~~~ + +On Linux, the fsmonitor daemon uses inotify to monitor filesystem events. +The inotify system has per-user limits on the number of watches that can +be created. The default limit is typically 8192 watches per user. + +For large repositories with many directories, you may need to increase +this limit. Check the current limit with: + + cat /proc/sys/fs/inotify/max_user_watches + +To temporarily increase the limit: + + sudo sysctl fs.inotify.max_user_watches=65536 + +To make the change permanent, add to `/etc/sysctl.conf`: + + fs.inotify.max_user_watches=65536 + CONFIGURATION -------------
compat/fsmonitor/fsm-health-linux.c new file +33 -0
@@ -0,0 +1,33 @@ +#include "git-compat-util.h" +#include "config.h" +#include "fsmonitor-ll.h" +#include "fsm-health.h" +#include "fsmonitor--daemon.h" + +/* +* The Linux fsmonitor implementation uses inotify which has its own +* mechanisms for detecting filesystem unmount and other events that +* would require the daemon to shutdown. Therefore, we don't need +* a separate health thread like Windows does. +* +* These stub functions satisfy the interface requirements. +*/ + +int fsm_health__ctor(struct fsmonitor_daemon_state *state UNUSED) +{ +return 0 ; +} + +void fsm_health__dtor(struct fsmonitor_daemon_state *state UNUSED) +{ +return ; +} + +void fsm_health__loop(struct fsmonitor_daemon_state *state UNUSED) +{ +return ; +} + +void fsm_health__stop_async(struct fsmonitor_daemon_state *state UNUSED) +{ +}
compat/fsmonitor/fsm-listen-linux.c new file +746 -0
746 lines not on this page · raw message
compat/fsmonitor/fsm-path-utils-linux.c new file +217 -0
217 lines not on this page · raw message
config.mak.uname +10 -0
@@ -68,6 +68,16 @@ ifeq ($(uname_S),Linux) BASIC_CFLAGS += -std=c99 endif LINK_FUZZ_PROGRAMS = YesPlease + + # The builtin FSMonitor on Linux builds upon Simple-IPC. Both require + # Unix domain sockets and PThreads. + ifndef NO_PTHREADS + ifndef NO_UNIX_SOCKETS + FSMONITOR_DAEMON_BACKEND = linux + FSMONITOR_OS_SETTINGS = unix + BASIC_CFLAGS += -DHAVE_LINUX_MAGIC_H + endif + endif endif ifeq ($(uname_S),GNU/kFreeBSD) HAVE_ALLOCA_H = YesPlease
contrib/buildsystems/CMakeLists.txt +6 -2
@@ -296,6 +296,10 @@ if(SUPPORTS_SIMPLE_IPC) elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") set(FSMONITOR_DAEMON_BACKEND "darwin") set(FSMONITOR_OS_SETTINGS "unix") + elseif(CMAKE_SYSTEM_NAME STREQUAL "Linux") + set(FSMONITOR_DAEMON_BACKEND "linux") + set(FSMONITOR_OS_SETTINGS "unix") + add_compile_definitions(HAVE_LINUX_MAGIC_H) endif() if(FSMONITOR_DAEMON_BACKEND)
@@ -1149,8 +1153,8 @@ endif() file(STRINGS ${CMAKE_SOURCE_DIR}/GIT-BUILD-OPTIONS.in git_build_options NEWLINE_CONSUME) string(REPLACE "@BROKEN_PATH_FIX@" "" git_build_options "${git_build_options}") string(REPLACE "@DIFF@" "'${DIFF}'" git_build_options "${git_build_options}") -string(REPLACE "@FSMONITOR_DAEMON_BACKEND@" "win32" git_build_options "${git_build_options}") -string(REPLACE "@FSMONITOR_OS_SETTINGS@" "win32" git_build_options "${git_build_options}") +string(REPLACE "@FSMONITOR_DAEMON_BACKEND@" "${FSMONITOR_DAEMON_BACKEND}" git_build_options "${git_build_options}") +string(REPLACE "@FSMONITOR_OS_SETTINGS@" "${FSMONITOR_OS_SETTINGS}" git_build_options "${git_build_options}") string(REPLACE "@GITWEBDIR@" "'${GITWEBDIR}'" git_build_options "${git_build_options}") string(REPLACE "@GIT_INTEROP_MAKE_OPTS@" "" git_build_options "${git_build_options}") string(REPLACE "@GIT_PERF_LARGE_REPO@" "" git_build_options "${git_build_options}")
meson.build +4 -0
@@ -1324,6 +1324,10 @@ fsmonitor_os = '' if host_machine.system() == 'windows' fsmonitor_backend = 'win32' fsmonitor_os = 'win32' +elif host_machine.system() == 'linux' and threads.found() and compiler.has_header('linux/magic.h') + fsmonitor_backend = 'linux' + fsmonitor_os = 'unix' + libgit_c_args += '-DHAVE_LINUX_MAGIC_H' elif host_machine.system() == 'darwin' fsmonitor_backend = 'darwin' fsmonitor_os = 'unix'
Signature · 1 line
gitgitgadget
PATCH v12 8/13 run-command: add close_fd_above_stderr option
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
Add a close_fd_above_stderr flag to struct child_process. When set, the child closes file descriptors 3 and above between fork and exec (skipping the child-notifier pipe), capped at sysconf(_SC_OPEN_MAX) or 4096, whichever is smaller. This prevents the child from inheriting pipe endpoints or other descriptors from the parent environment (e.g., the test harness).
--- run-command.c | 12 ++++++++++++ run-command.h | 9 +++++++++ 2 files changed, 21 insertions(+)
run-command.c +12 -0
@@ -546,6 +546,7 @@ static void atfork_parent(struct atfork_state *as) "restoring signal mask"); #endif } +#endif /* GIT_WINDOWS_NATIVE */ static inline void set_cloexec (int fd)
@@ -832,6 +833,17 @@ fail_pipe:child_close (cmd->out); } +if (cmd->close_fd_above_stderr) { +long max_fd =sysconf (_SC_OPEN_MAX); +int fd; +if (max_fd <0 || max_fd >4096 ) + max_fd =4096 ; +for (fd =3 ; fd < max_fd; fd++) { +if (fd != child_notifier) +close (fd); + } + } +if (cmd->dir &&chdir (cmd->dir))child_die (CHILD_ERR_CHDIR );
run-command.h +9 -0
@@ -141,6 +141,15 @@ struct child_process { unsigned stdout_to_stderr:1; unsigned clean_on_exit:1; unsigned wait_after_clean:1; + +/** +* Close file descriptors 3 and above in the child after forking +* but before exec. This prevents the child from inheriting +* pipe endpoints or other descriptors from the parent +* environment (e.g., the test harness). +*/ + unsigned close_fd_above_stderr:1; +void (*clean_on_exit_handler)(struct child_process *process); };
Signature · 1 line
gitgitgadget
PATCH v12 9/13 fsmonitor: close inherited file descriptors and detach in daemon
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
When the fsmonitor daemon is spawned as a background process, it may inherit file descriptors from its parent that it does not need. In particular, when the test harness or a CI system captures output through pipes, the daemon can inherit duplicated pipe endpoints. If the daemon holds these open, the parent process never sees EOF and may appear to hang.
Set close_fd_above_stderr on the child process at both daemon startup
paths: the explicit "fsmonitor--daemon start" command and the implicit spawn triggered by fsmonitor-ipc when a client finds no running daemon. Also suppress stdout and stderr on the implicit spawn path to prevent the background daemon from writing to the client's terminal.
Additionally, call setsid() when the daemon starts with --detach to create a new session and process group. This prevents the daemon from being part of the spawning shell's process group, which could cause the shell's "wait" to block until the daemon exits.
--- builtin/fsmonitor--daemon.c | 16 ++++++++++++++-- fsmonitor-ipc.c | 3 +++ 2 files changed, 17 insertions(+), 2 deletions(-)
builtin/fsmonitor--daemon.c +14 -2
@@ -1439,7 +1439,7 @@ done:return err; } -static int try_to_run_foreground_daemon(int detach_console MAYBE_UNUSED) +static int try_to_run_foreground_daemon (int detach_console) {/* * Technically, we don't need to probe for an existing daemon
@@ -1459,10 +1459,21 @@ static int try_to_run_foreground_daemon(int detach_console MAYBE_UNUSED)fflush (stderr); } +if (detach_console) {#ifdef GIT_WINDOWS_NATIVE -if (detach_console)FreeConsole (); +#else +/* +* Create a new session so that the daemon is detached +* from the parent's process group. This prevents +* shells with job control (e.g. bash with "set -m") +* from waiting on the daemon when they wait for a +* foreground command that implicitly spawned it. +*/ +if (setsid () ==-1 ) +warning_errno (_ ("setsid failed" ));#endif + }return !!fsmonitor_run_daemon (); }
@@ -1525,6 +1536,7 @@ static int try_to_start_background_daemon(void) cp.no_stdin =1 ; cp.no_stdout =1 ; cp.no_stderr =1 ; + cp.close_fd_above_stderr =1 ; sbgr =start_bg_command (&cp, bg_wait_cb,NULL , fsmonitor__start_timeout_sec);
fsmonitor-ipc.c +3 -0
@@ -61,6 +61,9 @@ static int spawn_daemon(void) cmd.git_cmd =1 ; cmd.no_stdin =1 ; + cmd.no_stdout =1 ; + cmd.no_stderr =1 ; + cmd.close_fd_above_stderr =1 ; cmd.trace2_child_class ="fsmonitor" ;strvec_pushl (&cmd.args,"fsmonitor--daemon" ,"start" ,NULL );
Signature · 1 line
gitgitgadget
PATCH v12 10/13 fsmonitor: add timeout to daemon stop command
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
The "fsmonitor--daemon stop" command polls in a loop waiting for the daemon to exit after sending a "quit" command over IPC. If the daemon fails to shut down (e.g. it is stuck or wedged), this loop spins forever.
Add a 30-second timeout so the stop command returns an error instead of blocking indefinitely.
--- builtin/fsmonitor--daemon.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
builtin/fsmonitor--daemon.c +11 -1
@@ -86,6 +86,8 @@ static int do_as_client__send_stop(void) {struct strbuf answer =STRBUF_INIT ;int ret; +int max_wait_ms =30000 ; +int elapsed_ms =0 ; ret =fsmonitor_ipc__send_command ("quit" , &answer);
@@ -96,8 +98,16 @@ static int do_as_client__send_stop(void)return ret;trace2_region_enter ("fsm_client" ,"polling-for-daemon-exit" ,NULL ); -while (fsmonitor_ipc__get_state () ==IPC_STATE__LISTENING ) +while (fsmonitor_ipc__get_state () ==IPC_STATE__LISTENING ) { +if (elapsed_ms >= max_wait_ms) { +trace2_region_leave ("fsm_client" , +"polling-for-daemon-exit" ,NULL ); +return error (_ ("daemon did not stop within %d seconds" ), + max_wait_ms /1000 ); + }sleep_millisec (50 ); + elapsed_ms +=50 ; + }trace2_region_leave ("fsm_client" ,"polling-for-daemon-exit" ,NULL );return 0 ;
Signature · 1 line
gitgitgadget
PATCH v12 11/13 fsmonitor: add tests for Linux
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
Add a smoke test that verifies the filesystem actually delivers inotify events to the daemon. On some configurations (e.g., overlayfs with older kernels), inotify watches succeed but events are never delivered. The daemon cookie wait will time out, but every subsequent test would fail. Skip the entire test file early when this is detected.
Add a test that exercises rapid nested directory creation to verify the daemon correctly handles the EEXIST race between recursive scan and queued inotify events. When IN_MASK_CREATE is available and a directory watch is added during recursive registration, the kernel may also deliver a queued IN_CREATE event for the same directory. The second inotify_add_watch() returns EEXIST, which must be treated as harmless. An earlier version of the listener crashed in this scenario.
Reduce --start-timeout from the default 60 seconds to 10 seconds so that tests fail promptly when the daemon cannot start.
Harden the test helpers to work in environments without procps (e.g., Fedora CI): fall back to reading /proc/$pid/stat for the process group ID when ps is unavailable, guard stop_git() against an empty pgid, and redirect stderr from kill to /dev/null to avoid noise when processes have already exited.
Use set -m to enable job control in the submodule-pull test so that the background git pull gets its own process group, preventing the shell wait from blocking on the daemon. setsid() in the previous commit detaches the daemon itself, but the intermediate git pull process still needs its own process group for the test shell to manage it correctly.
--- t/t7527-builtin-fsmonitor.sh | 89 +++++++++++++++++++++++++++++++++--- 1 file changed, 82 insertions(+), 7 deletions(-)
t/t7527-builtin-fsmonitor.sh +82 -7
@@ -10,9 +10,58 @@ then test_done fi +# Verify that the filesystem delivers events to the daemon. +# On some configurations (e.g., overlayfs with older kernels), +# inotify watches succeed but events are never delivered. The +# cookie wait will time out and the daemon logs a trace message. +# +# Use "timeout" (if available) to guard each step against hangs. +maybe_timeout () { + if type timeout >/dev/null 2>&1 + then + timeout "$@" + else + shift + "$@" + fi +} +verify_fsmonitor_works () { + git init test_fsmonitor_smoke || return 1 + + GIT_TRACE_FSMONITOR="$PWD/smoke.trace" && + export GIT_TRACE_FSMONITOR && + maybe_timeout 30 \ + git -C test_fsmonitor_smoke fsmonitor--daemon start \ + --start-timeout=10 + ret=$? + unset GIT_TRACE_FSMONITOR + if test $ret -ne 0 + then + rm -rf test_fsmonitor_smoke smoke.trace + return 1 + fi + + maybe_timeout 10 \ + test-tool -C test_fsmonitor_smoke fsmonitor-client query \ + --token 0 >/dev/null 2>&1 + maybe_timeout 5 \ + git -C test_fsmonitor_smoke fsmonitor--daemon stop 2>/dev/null + ! grep -q "cookie_wait timed out" "$PWD/smoke.trace" 2>/dev/null + ret=$? + rm -rf test_fsmonitor_smoke smoke.trace + return $ret +} + +if ! verify_fsmonitor_works +then + skip_all="filesystem does not deliver fsmonitor events (container/overlayfs?)" + test_done +fi + stop_daemon_delete_repo () { r=$1 && - test_might_fail git -C $r fsmonitor--daemon stop && + test_might_fail maybe_timeout 30 \ + git -C $r fsmonitor--daemon stop 2>/dev/null rm -rf $1 }
@@ -67,7 +116,7 @@ start_daemon () { export GIT_TEST_FSMONITOR_TOKEN fi && - git $r fsmonitor--daemon start && + git $r fsmonitor--daemon start --start-timeout=10 && git $r fsmonitor--daemon status ) }
@@ -520,6 +569,28 @@ test_expect_success 'directory changes to a file' ' grep "^event: dir1$" .git/trace ' +test_expect_success 'rapid nested directory creation' ' + test_when_finished "git fsmonitor--daemon stop; rm -rf rapid" && + + start_daemon --tf "$PWD/.git/trace" && + + # Rapidly create nested directories to exercise race conditions + # where directory watches may be added concurrently during + # event processing and recursive scanning. + for i in $(test_seq 1 20) + do + mkdir -p "rapid/nested/dir$i/subdir/deep" || return 1 + done && + + # Give the daemon time to process all events + sleep 1 && + + test-tool fsmonitor-client query --token 0 && + + # Verify daemon is still running (did not crash) + git fsmonitor--daemon status +' + # The next few test cases exercise the token-resync code. When filesystem # drops events (because of filesystem velocity or because the daemon isn't # polling fast enough), we need to discard the cached data (relative to the
@@ -910,7 +981,10 @@ test_expect_success "submodule absorbgitdirs implicitly starts daemon" ' start_git_in_background () { git "$@" & git_pid=$! - git_pgid=$(ps -o pgid= -p $git_pid) + git_pgid=$(ps -o pgid= -p $git_pid 2>/dev/null || + awk '{print $5}' /proc/$git_pid/stat 2>/dev/null) && + git_pgid="${git_pgid## }" && + git_pgid="${git_pgid%% }" nr_tries_left=10 while true do
@@ -921,15 +995,16 @@ start_git_in_background () { fi sleep 1 nr_tries_left=$(($nr_tries_left - 1)) - done >/dev/null 2>&1 & + done >/dev/null 2>&1 3>&- 4>&- 5>&- 6>&- 7>&- & watchdog_pid=$! wait $git_pid } stop_git () { - while kill -0 -- -$git_pgid + test -n "$git_pgid" || return 0 + while kill -0 -- -$git_pgid 2>/dev/null do - kill -- -$git_pgid + kill -- -$git_pgid 2>/dev/null sleep 1 done }
@@ -944,7 +1019,7 @@ stop_watchdog () { test_expect_success !MINGW "submodule implicitly starts daemon by pull" ' test_atexit "stop_watchdog" && - test_when_finished "stop_git; rm -rf cloned super sub" && + test_when_finished "set +m; stop_git; rm -rf cloned super sub" && create_super super && create_sub sub &&
Signature · 1 line
gitgitgadget
PATCH v12 12/13 fsmonitor: convert shown khash to strset in do_handle_client
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
Replace the khash-based string set used for deduplicating pathnames in do_handle_client() with a strset, which provides a cleaner interface for the same purpose.
Since the paths are interned strings from the batch data, use strdup_strings=0 to avoid unnecessary copies.
--- builtin/fsmonitor--daemon.c | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-)
builtin/fsmonitor--daemon.c +6 -11
@@ -16,7 +16,7 @@#include "fsmonitor--daemon.h" #include "simple-ipc.h" -#include "khash.h" +#include "strmap.h" #include "run-command.h" #include "trace.h" #include "trace2.h"
@@ -674,8 +674,6 @@ static int fsmonitor_parse_client_token(const char *buf_token,return 0 ; } -KHASH_INIT(str, const char *, int, 0, kh_str_hash_func, kh_str_hash_equal) - static int do_handle_client(struct fsmonitor_daemon_state *state, const char *command, ipc_server_reply_cb *reply,
@@ -692,8 +690,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,const struct fsmonitor_batch *batch;struct fsmonitor_batch *remainder =NULL ;intmax_t count =0 , duplicates =0 ; -kh_str_t *shown =NULL ; -int hash_ret; +struct strset shown =STRSET_INIT ;int do_trivial =0 ;int do_flush =0 ;int do_cookie =0 ;
@@ -882,14 +879,14 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,* so walk the batch list backwards from the current head back * to the batch (sequence number) they named. * -* We use khash to de-dup the list of pathnames. +* We use a strset to de-dup the list of pathnames. * * NEEDSWORK: each batch contains a list of interned strings, * so we only need to do pointer comparisons here to build the * hash table. Currently, we're still comparing the string * values. */ - shown =kh_init_str (); +strset_init_with_options (&shown,NULL ,0 );for (batch = batch_head; batch && batch->batch_seq_nr > requested_oldest_seq_nr; batch = batch->next) {
@@ -899,11 +896,9 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,const char *s = batch->interned_paths[k];size_t s_len; -if (kh_get_str (shown, s) !=kh_end (shown)) +if (!strset_add (&shown, s)) duplicates++;else { -kh_put_str (shown, s, &hash_ret); -trace_printf_key (&trace_fsmonitor,"send[%" PRIuMAX"]: %s" , count, s);
@@ -973,7 +968,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,trace2_data_intmax ("fsmonitor" , the_repository,"response/count/duplicates" , duplicates); cleanup: -kh_destroy_str (shown); +strset_clear (&shown);strbuf_release (&response_token);strbuf_release (&requested_token_id);strbuf_release (&payload);
Signature · 1 line
gitgitgadget
PATCH v12 13/13 fsmonitor: fix split-index bitmap bounds in tweak_fsmonitor()
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
When GIT_TEST_SPLIT_INDEX=yes is set and the fsmonitor daemon is active, tweak_fsmonitor() can hit a BUG() assertion:
BUG: fsmonitor.c:27: fsmonitor_dirty has more entries than the index (2 > 0)
The fsmonitor_dirty EWAH bitmap may reference positions from a previous index state. With split-index, cache_nr can be smaller than the bitmap expects because entries have not been merged yet.
This is related to the issue that 05f28e4b3c (scalar: use index.skipHash=true for performance, 2025-06-04) worked around by disabling GIT_TEST_SPLIT_INDEX in t9210, noting "the issue should be resolved in a series focused on the split index." This fixes the fsmonitor bitmap side; the index.skipHash interaction remains.
Two places hit this:
- tweak_fsmonitor() calls assert_index_minimum() without the !istate->split_index guard that the read path (line 98) and write path (line 128) already have. Add the same guard.
- fsmonitor_ewah_callback() unconditionally asserts and then accesses istate->cache[pos], which is out of bounds with split-index. Replace the assertion with a bounds check that silently skips positions beyond cache_nr.
--- fsmonitor.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
fsmonitor.c +4 -2
@@ -33,7 +33,8 @@ static void fsmonitor_ewah_callback(size_t pos, void *is)struct index_state *istate = (struct index_state *)is;struct cache_entry *ce; -assert_index_minimum (istate, pos +1 ); +if (pos >= istate->cache_nr) +return ; ce = istate->cache[pos]; ce->ce_flags &= ~CE_FSMONITOR_VALID ;
@@ -805,7 +806,8 @@ void tweak_fsmonitor(struct index_state *istate) }/* Mark all previously saved entries as dirty */ -assert_index_minimum (istate, istate->fsmonitor_dirty->bit_size); +if (!istate->split_index) +assert_index_minimum (istate, istate->fsmonitor_dirty->bit_size);ewah_each_bit (istate->fsmonitor_dirty, fsmonitor_ewah_callback, istate);refresh_fsmonitor (istate);
Signature · 1 line
gitgitgadget
Re: [PATCH v12 13/13] fsmonitor: fix split-index bitmap bounds in tweak_fsmonitor()
Quoted 11 lines · Paul Tarjan via GitGitGadget
From: Paul Tarjan <[email protected]>
When GIT_TEST_SPLIT_INDEX=yes is set and the fsmonitor daemon is active,
tweak_fsmonitor()can hit aBUG()assertion:BUG: fsmonitor.c:27: fsmonitor_dirty has more entries than the index (2 > 0)The
fsmonitor_dirtyEWAH bitmap may reference positions from a previous index state. With split-index,cache_nrcan be smaller than the bitmap expects because entries have not been merged yet.
Quoted 5 lines
This is related to the issue that
05f28e4b3c(scalar: use index.skipHash=true for performance, 2025-06-04) worked around by disablingGIT_TEST_SPLIT_INDEXin t9210, noting "the issue should be resolved in a series focused on the split index." This fixes the fsmonitor bitmap side; the index.skipHash interaction remains.
Let's ask for input from the author of that scalar commit 05f28e4b3c, which hints that the fix should come with the focus on split-index.
I also have to wonder if we should bury the issue like this patch does, which is not limited to the split case (i.e., the assert should trigger if the requirement is not met while split-index is not active, but the patch castrates it), instead of mimicking what Derrick did to work it around on the test side, leaving the production code still broken, which will give us better feel on the urgency of the split-index case in the real world. I dunno.
Quoted 40 lines
Two places hit this:
-
tweak_fsmonitor()callsassert_index_minimum()without the !istate->split_index guard that the read path (line 98) and write path (line 128) already have. Add the same guard.-
fsmonitor_ewah_callback()unconditionally asserts and then accesses istate->cache[pos], which is out of bounds with split-index. Replace the assertion with a bounds check that silently skips positions beyondcache_nr.Signed-off-by Paul Tarjan <[email protected]>--- fsmonitor.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)fsmonitor.c +4
-2@@ -33,7 +33,8 @@ static void fsmonitor_ewah_callback(size_t pos, void *is)struct index_state *istate = (struct index_state *)is;struct cache_entry *ce; -assert_index_minimum (istate, pos +1 ); +if (pos >= istate->cache_nr) +return ; ce = istate->cache[pos]; ce->ce_flags &= ~CE_FSMONITOR_VALID ;@@ -805,7 +806,8 @@ void tweak_fsmonitor(struct index_state *istate) }/* Mark all previously saved entries as dirty */ -assert_index_minimum (istate, istate->fsmonitor_dirty->bit_size); +if (!istate->split_index) +assert_index_minimum (istate, istate->fsmonitor_dirty->bit_size);ewah_each_bit (istate->fsmonitor_dirty, fsmonitor_ewah_callback, istate);refresh_fsmonitor (istate);
Re: [PATCH v12 13/13] fsmonitor: fix split-index bitmap bounds in tweak_fsmonitor()
Quoted 23 lines · Junio C Hamano
I also have to wonder if we should bury the issue like this patch does, which is not limited to the split case (i.e., the assert should trigger if the requirement is not met while split-index is not active, but the patch castrates it), instead of mimicking what Derrick did to work it around on the test side, leaving the production code still broken, which will give us better feel on the urgency of the split-index case in the real world. I dunno.
Quoted 15 lines
Two places hit this:
-
tweak_fsmonitor()callsassert_index_minimum()without the !istate->split_index guard that the read path (line 98) and write path (line 128) already have. Add the same guard.-
fsmonitor_ewah_callback()unconditionally asserts and then accesses istate->cache[pos], which is out of bounds with split-index. Replace the assertion with a bounds check that silently skips positions beyondcache_nr.Signed-off-by Paul Tarjan <[email protected]>--- fsmonitor.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
Let me ask the question differently.
Quoted 11 lines
fsmonitor.c static void fsmonitor_ewah_callback(size_t pos, void *is) · line 37
Quoted 11 lines
fsmonitor.c +2
-1@@ -33,7 +33,8 @@ static void fsmonitor_ewah_callback(size_t pos, void *is)struct index_state *istate = (struct index_state *)is;struct cache_entry *ce; -assert_index_minimum (istate, pos +1 ); +if (pos >= istate->cache_nr) +return ;
While this may avoid this "assert" stopping the process, what is the implication of returning from here? Who are the callers, what did the callers want us to do to the cache entry at the "pos", and what happens to their expectations if we simply returned to them without doing anything they asked us to do?
Quoted 7 lines
fsmonitor.c void tweak_fsmonitor(struct index_state *istate) · line 810
Quoted 7 lines
fsmonitor.c +2
-1@@ -805,7 +806,8 @@ void tweak_fsmonitor(struct index_state *istate) }/* Mark all previously saved entries as dirty */ -assert_index_minimum (istate, istate->fsmonitor_dirty->bit_size); +if (!istate->split_index) +assert_index_minimum (istate, istate->fsmonitor_dirty->bit_size);
This part does make sense; the minimum may be relevant only when split-index is in effect, so we limit the check to non split case.
I do have to wonder if there are some invariants we can/want to enforce here in the split case, but I do not mind punting it to a fix that is more focused on the split-index theme, as Derrick hinted in the earlier commit you found.
Quoted 3 lines
fsmonitor.c void tweak_fsmonitor(struct index_state *istate) · line 813
Quoted 3 lines
fsmonitor.c +0
-0ewah_each_bit (istate->fsmonitor_dirty, fsmonitor_ewah_callback, istate);refresh_fsmonitor (istate);
Thanks.
Re: [PATCH v12 13/13] fsmonitor: fix split-index bitmap bounds in tweak_fsmonitor()
Quoted 5 lines · Junio C Hamano
While this may avoid this "assert" stopping the process, what is the implication of returning from here? Who are the callers, what did the callers want us to do to the cache entry at the "pos", and what happens to their expectations if we simply returned to them without doing anything they asked us to do?
The callback's job is to clear CE_FSMONITOR_VALID on the entry at pos (marking it dirty). The only caller is tweak_fsmonitor() via ewah_each_bit().
In practice this guard shouldn't fire during normal operation. By the time tweak_fsmonitor() runs, merge_base_index() has already merged the split-index entries, so cache_nr is the full merged size and the bitmap positions are all in range. The call order in read_index_from() is:
do_read_index() -- reads split index, sets fsmonitor_dirty do_read_index(base) -- reads shared index merge_base_index() -- merges base into istate, cache_nr grows post_read_index_from() -- calls tweak_split_index, then tweak_fsmonitor
The guard only fires if the bitmap references positions that no longer exist (e.g., entries were removed between write and read). In that case returning is correct -- there's no entry to mark dirty.
And even if we somehow missed marking something, refresh_fsmonitor() runs right after (line 813) and re-queries the daemon by pathname, so it catches any changes independently of the bitmap.
That said, if you'd rather keep the BUG() assertion as a canary and handle this on the test side like Derrick did, I'm fine with dropping the callback change and just keeping the !istate->split_index guard in tweak_fsmonitor().
Quoted 2 lines
This part does make sense; the minimum may be relevant only when split-index is in effect, so we limit the check to non split case.
Right.
Quoted 4 lines
I do have to wonder if there are some invariants we can/want to enforce here in the split case, but I do not mind punting it to a fix that is more focused on the split-index theme, as Derrick hinted in the earlier commit you found.
Agreed. The bitmap is written against the full merged index and read back before the merge, so during the read phase the positions can legitimately exceed the split cache_nr. After the merge they should all be in range. The read and write paths already punt on this with the !istate->split_index guard; enforcing anything tighter probably belongs in a split-index focused series.
I've also moved this commit to 1/13 per your earlier feedback about bisectability.
Paul
Re: [PATCH v13 01/13] t9210: disable GIT_TEST_SPLIT_INDEX for scalar clone tests
In reply to Junio C HamanoQuoted 1 line · Junio C Hamano
Not just "suspect", but it does break exactly at t9211.
Thanks. I've added sane_unset GIT_TEST_SPLIT_INDEX at the top of t9211 (every test there does scalar clone) and verified both t9210 and t9211 pass with GIT_TEST_SPLIT_INDEX=yes on Fedora against seen.
Paul
Re: [PATCH v12 13/13] fsmonitor: fix split-index bitmap bounds in tweak_fsmonitor()
In reply to Paul Tarjan via GitGitGadgetHi Paul,
Quoted 19 lines · Paul Tarjan via GitGitGadget
From: Paul Tarjan <[email protected]>
When GIT_TEST_SPLIT_INDEX=yes is set and the fsmonitor daemon is active,
tweak_fsmonitor()can hit aBUG()assertion:BUG: fsmonitor.c:27: fsmonitor_dirty has more entries than the index (2 > 0)The
fsmonitor_dirtyEWAH bitmap may reference positions from a previous index state. With split-index,cache_nrcan be smaller than the bitmap expects because entries have not been merged yet.This is related to the issue that
05f28e4b3c(scalar: use index.skipHash=true for performance, 2025-06-04) worked around by disablingGIT_TEST_SPLIT_INDEXin t9210, noting "the issue should be resolved in a series focused on the split index." This fixes the fsmonitor bitmap side; the index.skipHash interaction remains.Two places hit this:
Quoted 8 lines
-
tweak_fsmonitor()callsassert_index_minimum()without the !istate->split_index guard that the read path (line 98) and write path (line 128) already have. Add the same guard.-
fsmonitor_ewah_callback()unconditionally asserts and then accesses istate->cache[pos], which is out of bounds with split-index. Replace the assertion with a bounds check that silently skips positions beyondcache_nr.
This issue (and in particular the BUG assertion reported by the CI failures Junio pointed at in
https://lore.kernel.org/git/[email protected]/) sounded quite familiar to me, so I consulted my notes. I had fixed a similar symptom after a seriously tedious debugging session via 3b7a4475b091 (split-index; stop abusing the base_oid to strip the "link" extension, 2023-03-26). But alas, this reference did not help, I found no similar bug that could be the culprit here.
So I bisected the CI failure and it turns out that, as I suspected, this patch addresses a symptom, not the root cause. The bug has nothing to do with Linux FSMonitor support and this patch should be dropped from the series. See below for a suggestion what to do instead.
The failing CI job is linux-TEST-vars, which is the _only_ job that sets GIT_TEST_SPLIT_INDEX=yes. Before this series, fsmonitor_ipc__is_supported() returned 0 on Linux, so Scalar never set core.fsmonitor=true, and therefore there was never a FSMonitor bitmap in the index, and the assertion in tweak_fsmonitor() could never fire. Now that Linux has FSMonitor support, Scalar enables it, and the bitmap sanity check exposes a _pre-existing_ corruption in the index.skipHash + split-index interaction.
The interacting topic is hn/git-checkout-m-with-stash (specifically ced477a20fa4, "checkout: -m (--merge) uses autostash when switching branches"). That commit adds an unconditional discard_index() + repo_read_index() cycle at the end of switch_branches(), even when no autostash was created. (That is probably undesirable behavior, it should be guarded.)
Here is the chain of events:
1. Scalar sets index.skipHash=true (this is Scalar's default since 05f28e4b3cc1 (scalar: use index.skipHash=true for performance, 2025-12-12)).
2. With GIT_TEST_SPLIT_INDEX=yes, write_locked_index() writes a split index. It calls write_shared_index(), which calls do_write_index() for the shared index. Because skipHash=true, the trailing hash is left as all-zeros, so si->base->oid is the null OID.
3. write_shared_index() then does
(https://gitlab.com/git-scm/git/-/blob/v2.53.0/read-cache.c#L3283):
oidcpy(&si->base_oid, &si->base->oid);
...copying the null OID into base_oid. The shared index file is renamed to sharedindex.0000000000000000000000000000000000000000.
4. The hn/git-checkout-m-with-stash topic's unconditional discard_index() + repo_read_index() re-reads the index from disk.
5. In read_index_from() (read-cache.c:2375), the condition
if (!split_index || is_null_oid(&split_index->base_oid))
is true, so the shared index is never loaded, merge_base_index() is never called, and we jump straight to post_read_index_from().
6. This leaves cache_nr=0 (all entries live in the never-loaded shared index), but the FSMonitor extension's fsmonitor_dirty bitmap was written against the full merged index (2 entries).
7. tweak_fsmonitor() (called from post_read_index_from()) hits:
assert_index_minimum(istate, istate->fsmonitor_dirty->bit_size);
and fires the BUG: bit_size=2 > cache_nr=0.
The same bug would reproduce on macOS or Windows if those platforms had a TEST-vars CI job with GIT_TEST_SPLIT_INDEX=yes.
Now, about the patch itself (reordering the hunks for clarity):
Quoted 10 lines
Signed-off-by Paul Tarjan <[email protected]>--- fsmonitor.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)fsmonitor.c +0
-0
fsmonitor.c void tweak_fsmonitor(struct index_state *istate) · line 813
Quoted 9 lines
fsmonitor.c +2
-1@@ -805,7 +806,8 @@ void tweak_fsmonitor(struct index_state *istate) }/* Mark all previously saved entries as dirty */ -assert_index_minimum (istate, istate->fsmonitor_dirty->bit_size); +if (!istate->split_index) +assert_index_minimum (istate, istate->fsmonitor_dirty->bit_size);ewah_each_bit (istate->fsmonitor_dirty, fsmonitor_ewah_callback, istate);refresh_fsmonitor (istate);
This guard in tweak_fsmonitor() is not analogous to the guards in read_fsmonitor_extension() and write_fsmonitor_extension(). Those guards exist because those functions run before merge_base_index() has been called. But tweak_fsmonitor() runs after merge_base_index() in the normal read path (via post_read_index_from()). The fact that cache_nr=0 when tweak_fsmonitor() runs means the shared index was never loaded at all, which is the real bug.
For historical context: ba1b9caca699 ("fsmonitor: delay updating state until after split index is merged", 2017-10-27) intentionally arranged for tweak_fsmonitor() to run post-merge, and 392d797e2e60 (fsmonitor: do not check fsmonitor_dirty bits against cache entries on split index, 2019-10-31) and 7621a6f43428 (fsmonitor: do not compare bitmap size with size of split index, 2019-11-21) added the !istate->split_index guard to the read/write paths only, deliberately leaving tweak_fsmonitor() unguarded because at that point the merge is expected to have happened. cae70acf2431 (fsmonitor: de-duplicate BUG()s around dirty bits, 2021-01-23) refactored the BUG()s into assert_index_minimum(), again preserving this pattern. Suppressing the assertion here would hide the fact that the index is in a broken state.
fsmonitor.c static void fsmonitor_ewah_callback(size_t pos, void *is) · line 40
Quoted 10 lines
fsmonitor.c +2
-1@@ -33,7 +33,8 @@ static void fsmonitor_ewah_callback(size_t pos, void *is)struct index_state *istate = (struct index_state *)is;struct cache_entry *ce; -assert_index_minimum (istate, pos +1 ); +if (pos >= istate->cache_nr) +return ; ce = istate->cache[pos]; ce->ce_flags &= ~CE_FSMONITOR_VALID ;
This change to fsmonitor_ewah_callback() is particularly dangerous:
silently skipping dirty entries means files that the FSMonitor daemon reported as changed could be treated as unchanged. That is a correctness bug.
The proper fix belongs in one of two places:
1. write_shared_index() should not allow base_oid to become the null OID. When index.skipHash=true, the shared index needs a real identifier. This is the root cause.
2. Alternatively, the hn/git-checkout-m-with-stash topic should guard its discard_index() + repo_read_index() with something like if (created_autostash), since the index reload is only needed when an autostash was actually applied.
But that would probably only buy some time until another code path introduces a similar pattern, and maybe shouldn't even be considered a real fix.
Alternatively, I'd suggest:
3. The same work-around as was introduced in 05f28e4b3cc1 (scalar: use index.skipHash=true for performance, 2025-12-12), namely to simply force-disable split index in the offending test case (or move the existing sane_unset to the top of the file, since index.skipHash=true is a Scalar thing):
-- snip -- diff --git a/t/t9210-scalar.sh b/t/t9210-scalar.sh index 009437a5f316..a4e38ad644e4 100755 --- a/t/t9210-scalar.sh +++ b/t/t9210-scalar.sh @@ -152,6 +152,11 @@ test_expect_success 'set up repository to clone' ' '
test_expect_success 'scalar clone' '
+ # The split index refers to the base index via OID; Scalar sets
+ # index.skipHash, though, and therefore that OID is always bogus;
+ # Scalar/index.skipHash are simply incompatible with split-index
+ sane_unset GIT_TEST_SPLIT_INDEX &&
+
second=$(git rev-parse --verify second:second.t) && scalar clone "file://$(pwd)" cloned --single-branch && (
-- snap --
Either way, this 13/13 patch should be dropped from the fsmonitor-linux series, I think.
Ciao,
Johannes
Re: [PATCH v12 13/13] fsmonitor: fix split-index bitmap bounds in tweak_fsmonitor()
Quoted 3 lines · Johannes Schindelin
So the actual bug fix would be to ensure that
write_shared_index()produces a hash even ifindex.skipHash=true, because that hash is needed to identify the shared index.
You're right, and I verified this. I reverted my fsmonitor.c patch from seen, applied the fix below in write_shared_index(), and ran t9210 with GIT_TEST_SPLIT_INDEX=yes on Fedora:
Without either fix: not ok 12, not ok 13 (BUG assertion)
With only skipHash fix: passed all 22 test(s)
The fsmonitor.c patch is not needed. I'll drop it from the next version of the series.
Happy to submit this as a separate patch or include it in my series if that's helpful. Or if someone is already working on the skipHash + split-index interaction, I'll stay out of the way.
--- >8 ---
read-cache.c +0 -0
@@ write_shared_indexmove_cache_to_base_index (istate);convert_to_sparse (istate,0 );
- trace2_region_enter_printf("index", "shared/do_write_index",
- the_repository, "%s", get_tempfile_path(*temp));
- ret = do_write_index(si->base, *temp, WRITE_NO_EXTENSION, flags);
- trace2_region_leave_printf("index", "shared/do_write_index",
- the_repository, "%s", get_tempfile_path(*temp));
+ /*
+ * The shared index is identified by the hash of its contents
+ * (sharedindex.<oid>). If index.skipHash is set, do_write_index()
+ * would produce an all-zero hash and the shared index would not
+ * be found on re-read (is_null_oid() check in read_index_from()).
+ * Temporarily force hashing for the shared index write.
+ */
+ {
+ struct repository *r = the_repository;
+ int save_skip_hash;
+
+ prepare_repo_settings(r);
+ save_skip_hash = r->settings.index_skip_hash;
+ r->settings.index_skip_hash = 0;
+
+ trace2_region_enter_printf("index", "shared/do_write_index",
+ the_repository, "%s", get_tempfile_path(*temp));
+ ret = do_write_index(si->base, *temp, WRITE_NO_EXTENSION, flags);
+ trace2_region_leave_printf("index", "shared/do_write_index",
+ the_repository, "%s", get_tempfile_path(*temp));
+
+ r->settings.index_skip_hash = save_skip_hash;
+ }
if (was_full)
ensure_full_index(istate);
Paul
Re: [PATCH v12 13/13] fsmonitor: fix split-index bitmap bounds in tweak_fsmonitor()
Hi Paul,
Quoted 10 lines · Paul Tarjan
Quoted 3 lines · Johannes Schindelin
So the actual bug fix would be to ensure that
write_shared_index()produces a hash even ifindex.skipHash=true, because that hash is needed to identify the shared index.You're right, and I verified this. I reverted my
fsmonitor.cpatch from seen, applied the fix below inwrite_shared_index(), and ran t9210 with GIT_TEST_SPLIT_INDEX=yes on Fedora:
Quoted 5 lines
Without either fix: not ok 12, not ok 13 (BUG assertion)
With only skipHash fix: passed all 22 test(s)The
fsmonitor.cpatch is not needed. I'll drop it from the next version of the series.
Thank you for confirming.
Quoted 34 lines
Happy to submit this as a separate patch or include it in my series if that's helpful. Or if someone is already working on the skipHash + split-index interaction, I'll stay out of the way.
--- >8 ---
read-cache.c +0
-0@@
write_shared_indexread-cache.c +16
-5move_cache_to_base_index (istate);convert_to_sparse (istate,0 ); -trace2_region_enter_printf ("index" ,"shared/do_write_index" , - the_repository,"%s" ,get_tempfile_path (*temp)); - ret =do_write_index (si->base, *temp,WRITE_NO_EXTENSION , flags); -trace2_region_leave_printf ("index" ,"shared/do_write_index" , - the_repository,"%s" ,get_tempfile_path (*temp)); +/* +* The shared index is identified by the hash of its contents +* (sharedindex.<oid>). If index.skipHash is set, do_write_index() +* would produce an all-zero hash and the shared index would not +* be found on re-read (is_null_oid() check in read_index_from()). +* Temporarily force hashing for the shared index write. +*/ + { +struct repository *r = the_repository; +int save_skip_hash; + +prepare_repo_settings (r); + save_skip_hash = r->settings.index_skip_hash; + r->settings.index_skip_hash =0 ; + + trace2_region_enter_printf("index", "shared/do_write_index",
read-cache.c
Quoted 10 lines
read-cache.c +7
-0+ the_repository, "%s", get_tempfile_path(*temp)); + ret =do_write_index (si->base, *temp,WRITE_NO_EXTENSION , flags); +trace2_region_leave_printf ("index" ,"shared/do_write_index" , + the_repository,"%s" ,get_tempfile_path (*temp)); + + r->settings.index_skip_hash = save_skip_hash; + }if (was_full)ensure_full_index (istate);
This patch essentially tells Git to disobey the index.skipHash config, at least under some circumstances. While that _looks_ like it works around the observed problem, it is unlikely to be desirable in the long run because such an inconsistency is prone to cause problems down the line.
The fundamental problem at hand is that the split-index _requires_ the index' hash to be calculated, while the index.skipHash config specifically _skips_ it. In other words, those two features are fundamentally incompatible with one another.
Mind you, I could imagine a sort of hacky way to make them work with each
other: rely on the fact that padding the mtime's raw ytes with enough NULs to fill the OID array would result in a sort of fake OID that is as unlikely to clash with a real SHA as any other real SHA. In other words, under index.skipHash, instead of accepting a base_oid that consists of all NULs, fake one that won't fall into the is_null_oid() trap.
This approach is still fraught with challenges. For one, a written .git/index file will not indicate whether it was written with or without index.skipHash (unlike the split-index, which results in a link extension to be included), unless you count that all-NUL OID as a hint. Therefore, just changing the OID that is written under index.skipHash won't work, you'd have to introduce some sort of flag into the index file format _and_ then you'd run into compatibility issues with other Git implementations (or older Git versions) trying to read the index file written with that flag.
So the safest approach I can think of really is what I suggested, to force the GIT_TEST_SPLIT_INDEX variable to be unset in t9210-scalar.sh.
It would probably also make sense to contribute a separate patch that lets Git error out if it is asked to read or write a split-index under index.skipHash, but that might very well lead down another rabbit hole, therefore I won't recommend it.
Ciao,
Johannes
I briefly considered skipping t9210 altogether if that variable is set, but then all of those tests would be skipped in linux-TEST-vars, which would be counter-productive.
Ciao,
Johannes
Re: [PATCH v12 13/13] fsmonitor: fix split-index bitmap bounds in tweak_fsmonitor()
Quoted 3 lines · Johannes Schindelin
So the safest approach I can think of really is what I suggested, to force the
GIT_TEST_SPLIT_INDEXvariable to be unset int9210-scalar.sh.
Thanks for the thorough analysis. I've dropped both the fsmonitor.c workaround and the read-cache.c skipHash fix, and added sane_unset GIT_TEST_SPLIT_INDEX to the scalar clone tests that hit this (tests 12, 13, and 22), matching the existing workaround in test 16.
Verified on Fedora: seen + fsmonitor-linux with
GIT_TEST_SPLIT_INDEX=yes passes all 22 tests.
--- >8 ---
t/t9210-scalar.sh +6 -0
@@ -152,6 +152,10 @@ test_expect_success 'set up repository to clone' ' ' test_expect_success 'scalar clone' ' + # index.skipHash (Scalar default) and GIT_TEST_SPLIT_INDEX are + # incompatible: the shared index gets a null OID and fails to + # load on re-read. + sane_unset GIT_TEST_SPLIT_INDEX && second=$(git rev-parse --verify second:second.t) && scalar clone "file://$(pwd)" cloned --single-branch && (
@@ -182,6 +186,7 @@ test_expect_success 'scalar clone' ' ' test_expect_success 'scalar clone --no-... opts' ' + sane_unset GIT_TEST_SPLIT_INDEX && # Note: redirect stderr always to avoid having a verbose test # run result in a difference in the --[no-]progress option. GIT_TRACE2_EVENT="$(pwd)/no-opt-trace" scalar clone \
@@ -307,6 +312,7 @@ test_expect_success '`scalar [...] <dir>` errors out when dir is missing' ' SQ="'" test_expect_success UNZIP 'scalar diagnose' ' + sane_unset GIT_TEST_SPLIT_INDEX && scalar clone "file://$(pwd)" cloned --single-branch && git repack &&
Paul
Re: [PATCH v12 13/13] fsmonitor: fix split-index bitmap bounds in tweak_fsmonitor()
In reply to Johannes SchindelinQuoted 9 lines · Johannes Schindelin
This patch essentially tells Git to disobey the
index.skipHashconfig, at least under some circumstances. While that _looks_ like it works around the observed problem, it is unlikely to be desirable in the long run because such an inconsistency is prone to cause problems down the line.The fundamental problem at hand is that the split-index _requires_ the index' hash to be calculated, while the
index.skipHashconfig specifically _skips_ it. In other words, those two features are fundamentally incompatible with one another.
Thanks for clearly spelling out the core problem ...
Quoted 2 lines
So the safest approach I can think of really is what I suggested, to force the
GIT_TEST_SPLIT_INDEXvariable to be unset int9210-scalar.sh.
... and a clear recommendation. Very much appreciated.
v13 cover fsmonitor: implement filesystem change listener for Linux
In reply to Paul Tarjan via GitGitGadgetThis series implements the built-in fsmonitor daemon for Linux using the inotify API, bringing it to feature parity with the existing Windows and macOS implementations. It also fixes two memory leaks in the platform-independent daemon code and deduplicates the IPC and settings logic that is now shared between macOS and Linux.
The implementation uses inotify rather than fanotify because fanotify requires either CAP_SYS_ADMIN or CAP_PERFMON capabilities, making it unsuitable for an unprivileged user-space daemon. While inotify has the limitation of requiring a separate watch on every directory (unlike macOS FSEvents, which can monitor an entire directory tree with a single watch), it operates without elevated privileges and provides the per-file event granularity needed for fsmonitor.
The listener uses inotify_init1(O_NONBLOCK) with a poll loop that checks for events with a 50-millisecond timeout, keeping the inotify queue well-drained to minimize the risk of overflows. Bidirectional hashmaps map between watch descriptors and directory paths for efficient event resolution. Directory renames are tracked using inotify cookie mechanism to correlate IN_MOVED_FROM and IN_MOVED_TO event pairs; a periodic check detects stale renames where the matching IN_MOVED_TO never arrived, forcing a resync.
New directory creation triggers recursive watch registration to ensure all subdirectories are monitored. The IN_MASK_CREATE flag is used where available to prevent modifying existing watches, with a fallback for older kernels. When IN_MASK_CREATE is available and inotify_add_watch returns EEXIST, it means another thread or recursive scan has already registered the watch, so it is safe to ignore.
Remote filesystem detection uses statfs() to identify network-mounted filesystems (NFS, CIFS, SMB, FUSE, etc.) via their magic numbers. Mount point information is read from /proc/mounts and matched against the statfs f_fsid to get accurate, human-readable filesystem type names for logging. When the .git directory is on a remote filesystem, the IPC socket falls back to $HOME or a user-configured directory via the fsmonitor.socketDir setting.
This series builds on work from https://github.com/git/git/pull/1352 by Eric DeCosta and https://github.com/git/git/pull/1667 by Marziyeh Esipreh, updated to work with the current codebase and address all review feedback.
Changes since v12:
* Dropped both the fsmonitor.c workaround and the read-cache.c skipHash fix per Dscho's review: split-index and index.skipHash are fundamentally incompatible
* Added sane_unset GIT_TEST_SPLIT_INDEX to scalar clone tests in t9210 (patch 1/13), matching the existing workaround in test 16
Changes since v11:
* Fix t9210 BUG assertion with GIT_TEST_SPLIT_INDEX=yes: guard fsmonitor_dirty bitmap access against split-index having fewer entries than the bitmap expects
Changes since v10:
* Reverted pre_exec_cb callback back to simple close_fd_above_stderr flag per Junio's clarification (same as v8)
Changes since v9:
* Fixed Windows build: close_fd_above_stderr() compiles as a no-op on Windows since there is no fork/exec
Changes since v8:
* Replaced close_fd_above_stderr flag with generic pre_exec_cb callback in struct child_process per Junio's review, with close_fd_above_stderr() as a ready-made callback
Changes since v7:
* Added patch 12: convert khash to strset in do_handle_client (Patrick's #leftoverbit suggestion)
* Fixed "Forcing shutdown" trace message to start with lowercase
* Fixed redundant statfs() call in find_mount() (caller already had the result)
* Fixed CMakeLists.txt GIT-BUILD-OPTIONS: was hardcoded to "win32" for FSMONITOR_DAEMON_BACKEND and FSMONITOR_OS_SETTINGS, now uses the CMake variables
* Fixed uninitialized strset on trivial response path (STRSET_INIT)
* Removed V9FS_MAGIC from get_fs_typename() to match is_remote_fs() (9p is local VM mounts)
* Split 30-second stop timeout into its own commit per review request
* Fixed misleading indentation on shutdown assignment in handle_events()
* Updated commit messages to describe all changes (test hardening, fsmonitor-ipc.c spawn changes)
* Updated Makefile comment for FSMONITOR_OS_SETTINGS to mention fsm-ipc
Changes since v6:
* Introduced FSMONITOR_OS_SETTINGS build variable (set to "unix" for macOS and Linux, "win32" for Windows) to eliminate if/else conditionals in Makefile, meson.build, and CMakeLists.txt per Junio's review
* Moved fsm-path-utils from FSMONITOR_OS_SETTINGS to FSMONITOR_DAEMON_BACKEND since path-utils files are platform-specific
* Removed V9FS_MAGIC from remote filesystem detection (9p is used for local VM/container host mounts where fsmonitor works fine)
* Removed redundant #include <libgen.h> (already provided by compat/posix.h)
* Fixed cookie wait comment wording ("see" to "observe")
* Rewrote commit messages for IPC and settings dedup patches
Changes since v5:
* Split monolithic commit into 10-patch series per Patrick's review
* Deduplicated fsm-ipc and fsm-settings into shared Unix implementations
* Rewrote commit message with prose paragraphs, explain inotify vs fanotify, removed "Issues addressed" sections, added Based-on-patch-by trailers
* Removed redundant includes already provided by compat/posix.h
* Fixed error/trace message capitalization per coding guidelines
* Fixed stale rename check interval from 1000 seconds to 1 second
* Changed poll timeout from 1ms to 50ms to reduce idle CPU wake-ups
* Replaced infinite pthread_cond_wait cookie loop with one-second pthread_cond_timedwait (prevents daemon hangs on overlay filesystems where events are never delivered)
* Added pthread_cond_timedwait to Windows pthread compatibility layer
* Separated test into its own commit with smoke test that skips when inotify events are not delivered (e.g., overlayfs with older kernels)
* Fixed test hang on Fedora CI: stop_git() looped forever when ps was unavailable because bash in POSIX/sh mode returns exit 0 from kill with an empty process group argument. Fixed by falling back to /proc/$pid/stat for process group ID and guarding stop_git against empty pgid.
* Redirect spawn_daemon() stdout/stderr to /dev/null and close inherited file descriptors to prevent the intermediate process from holding test pipe file descriptors
* Call setsid() on daemon detach to prevent shells with job control from waiting on the daemon process group
* Close inherited file descriptors 3-7 in the test watchdog subprocess
* Added 30-second timeout to "fsmonitor--daemon stop" to prevent indefinite blocking
* Added helpful error message when inotify watch limit (max_user_watches) is reached
* Initialize fd_inotify to -1 and use fd >= 0 check for correct fd 0 handling
* Use sysconf(_SC_OPEN_MAX) instead of hardcoded 1024 for fd close limit
* Check setsid() return value
Changes since v4:
* Added Meson build support
Changes since v3:
* Fix crash on rapid nested directory creation (EEXIST from inotify_add_watch with IN_MASK_CREATE)
* Extensive stress testing
Changes since v2:
* Fix khash memory leak in do_handle_client
Changes since v1:
* Fix hashmap memory leak in fsmonitor_run_daemon()
Paul Tarjan (13):
t9210: disable GIT_TEST_SPLIT_INDEX for scalar clone tests
fsmonitor: fix khash memory leak in do_handle_client
fsmonitor: fix hashmap memory leak in fsmonitor_run_daemon compat/win32: add pthread_cond_timedwait
fsmonitor: use pthread_cond_timedwait for cookie wait
fsmonitor: rename fsm-ipc-darwin.c to fsm-ipc-unix.c
fsmonitor: rename fsm-settings-darwin.c to fsm-settings-unix.c
fsmonitor: implement filesystem change listener for Linux
run-command: add close_fd_above_stderr option
fsmonitor: close inherited file descriptors and detach in daemon
fsmonitor: add timeout to daemon stop command
fsmonitor: add tests for Linux
fsmonitor: convert shown khash to strset in do_handle_client
Documentation/config/fsmonitor--daemon.adoc | 4 +-
Documentation/git-fsmonitor--daemon.adoc | 28 +-
Makefile | 6 +-
builtin/fsmonitor--daemon.c | 92 ++-
compat/fsmonitor/fsm-health-linux.c | 33 +
.../{fsm-ipc-darwin.c => fsm-ipc-unix.c} | 0
compat/fsmonitor/fsm-listen-linux.c | 746 ++++++++++++++++++
compat/fsmonitor/fsm-path-utils-linux.c | 217 +++++
...-settings-darwin.c => fsm-settings-unix.c} | 0
compat/win32/pthread.c | 26 +
compat/win32/pthread.h | 2 +
config.mak.uname | 12 +-
contrib/buildsystems/CMakeLists.txt | 33 +-
fsmonitor-ipc.c | 3 +
meson.build | 13 +-
run-command.c | 12 +
run-command.h | 9 +
t/t7527-builtin-fsmonitor.sh | 89 ++-
t/t9210-scalar.sh | 6 +
19 files changed, 1268 insertions(+), 63 deletions(-)
create mode 100644 compat/fsmonitor/fsm-health-linux.c
rename compat/fsmonitor/{fsm-ipc-darwin.c => fsm-ipc-unix.c} (100%)
create mode 100644 compat/fsmonitor/fsm-listen-linux.c
create mode 100644 compat/fsmonitor/fsm-path-utils-linux.c
rename compat/fsmonitor/{fsm-settings-darwin.c => fsm-settings-unix.c} (100%)base-commit: 3e0db84c88c57e70ac8be8c196dfa92c5d656fbc
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-2147%2Fptarjan%2Fclaude%2Fupdate-pr-1352-current-85Gk8-v13
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-2147/ptarjan/claude/update-pr-1352-current-85Gk8-v13
Pull-Request: https://github.com/git/git/pull/2147
Range-diff vs v12:
-: ---------- > 1: 28c5aca413 t9210: disable GIT_TEST_SPLIT_INDEX for scalar clone tests 1: 4d4dec8fa1 = 2: 9f666beea7 fsmonitor: fix khash memory leak in do_handle_client 2: cb270120f0 = 3: a4a65a6dfa fsmonitor: fix hashmap memory leak in fsmonitor_run_daemon 3: 44a063074d = 4: 427073fc38 compat/win32: add pthread_cond_timedwait 4: b1081d1e13 = 5: 71effc4d47 fsmonitor: use pthread_cond_timedwait for cookie wait 5: dec0fb144f = 6: e897193b0e fsmonitor: rename fsm-ipc-darwin.c to fsm-ipc-unix.c 6: b2aaadb4ae = 7: e13d938ddb fsmonitor: rename fsm-settings-darwin.c to fsm-settings-unix.c 7: 03cf12d01b = 8: 3431eae60f fsmonitor: implement filesystem change listener for Linux 8: 50f5b4676e = 9: 7ce0ab87fb run-command: add close_fd_above_stderr option 9: 057b3098bc = 10: 2bf134a041 fsmonitor: close inherited file descriptors and detach in daemon 10: e6bc3bfcb2 = 11: cb4d511f21 fsmonitor: add timeout to daemon stop command 11: 81f8cd1599 = 12: 9a8647884e fsmonitor: add tests for Linux 12: 8fa6a74e0d = 13: a5fc9ad415 fsmonitor: convert shown khash to strset in do_handle_client 13: 84ddbb30bb < -: ---------- fsmonitor: fix split-index bitmap bounds in tweak_fsmonitor()
Signature · 1 line
gitgitgadget
PATCH v13 1/13 t9210: disable GIT_TEST_SPLIT_INDEX for scalar clone tests
From: Paul Tarjan <[email protected]>
index.skipHash (Scalar default) and split-index are incompatible:
the shared index gets a null OID when skipHash skips computing the hash, and the null OID causes the shared index to not be loaded on re-read. This triggers a BUG assertion in fsmonitor when the fsmonitor_dirty bitmap references more entries than the (now empty) index has.
Disable GIT_TEST_SPLIT_INDEX in the scalar clone tests that hit this, matching the existing workaround in test 16.
--- t/t9210-scalar.sh | 6 ++++++ 1 file changed, 6 insertions(+)
t/t9210-scalar.sh +6 -0
@@ -152,6 +152,10 @@ test_expect_success 'set up repository to clone' ' ' test_expect_success 'scalar clone' ' + # index.skipHash (Scalar default) and GIT_TEST_SPLIT_INDEX are + # incompatible: the shared index gets a null OID and fails to + # load on re-read. + sane_unset GIT_TEST_SPLIT_INDEX && second=$(git rev-parse --verify second:second.t) && scalar clone "file://$(pwd)" cloned --single-branch && (
@@ -182,6 +186,7 @@ test_expect_success 'scalar clone' ' ' test_expect_success 'scalar clone --no-... opts' ' + sane_unset GIT_TEST_SPLIT_INDEX && # Note: redirect stderr always to avoid having a verbose test # run result in a difference in the --[no-]progress option. GIT_TRACE2_EVENT="$(pwd)/no-opt-trace" scalar clone \
@@ -307,6 +312,7 @@ test_expect_success '`scalar [...] <dir>` errors out when dir is missing' ' SQ="'" test_expect_success UNZIP 'scalar diagnose' ' + sane_unset GIT_TEST_SPLIT_INDEX && scalar clone "file://$(pwd)" cloned --single-branch && git repack && echo "$(pwd)/.git/objects/" >>cloned/src/.git/objects/info/alternates &&
Signature · 1 line
gitgitgadget
Re: [PATCH v13 01/13] t9210: disable GIT_TEST_SPLIT_INDEX for scalar clone tests
Quoted 9 lines · Paul Tarjan via GitGitGadget
From: Paul Tarjan <[email protected]>
index.skipHash (Scalar default) and split-index are incompatible:
the shared index gets a null OID when skipHash skips computing the hash, and the null OID causes the shared index to not be loaded on re-read. This triggers a BUG assertion in fsmonitor when thefsmonitor_dirtybitmap references more entries than the (now empty) index has.
Quoted 7 lines
Disable
GIT_TEST_SPLIT_INDEXin the scalar clone tests that hit this, matching the existing workaround in test 16.Signed-off-by Paul Tarjan <[email protected]>--- t/t9210-scalar.sh | 6 ++++++ 1 file changed, 6 insertions(+)
The same treatment is needed for t9211, I suspect.
--- >8 ---
#!/bin/sh
export OPENSSL_SHA1_UNSAFE=YesPlease
export GIT_TEST_SPLIT_INDEX=yes
export GIT_TEST_FULL_IN_PACK_ARRAY=true export GIT_TEST_OE_SIZE=10
export GIT_TEST_OE_DELTA_SIZE=5
export GIT_TEST_COMMIT_GRAPH=1
export GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1 export GIT_TEST_MULTI_PACK_INDEX=1
export GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=1 export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master export GIT_TEST_NO_WRITE_REV_INDEX=1
export GIT_TEST_CHECKOUT_WORKERS=2
export GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL=1
make && cd t && sh t9211-*.sh
--- 8< ---
Re: [PATCH v13 01/13] t9210: disable GIT_TEST_SPLIT_INDEX for scalar clone tests
Quoted 14 lines · Junio C Hamano
Quoted 12 lines · Paul Tarjan via GitGitGadget
From: Paul Tarjan <[email protected]>
index.skipHash (Scalar default) and split-index are incompatible:
the shared index gets a null OID when skipHash skips computing the hash, and the null OID causes the shared index to not be loaded on re-read. This triggers a BUG assertion in fsmonitor when thefsmonitor_dirtybitmap references more entries than the (now empty) index has.Disable
GIT_TEST_SPLIT_INDEXin the scalar clone tests that hit this, matching the existing workaround in test 16.
Quoted 6 lines
Quoted 4 lines
Signed-off-by Paul Tarjan <[email protected]>--- t/t9210-scalar.sh | 6 ++++++ 1 file changed, 6 insertions(+)The same treatment is needed for t9211, I suspect.
Not just "suspect", but it does break exactly at t9211.
https://github.com/git/git/actions/runs/24060447371/job/70175464363
Quoted 20 lines
--- >8 ---
#!/bin/shexport OPENSSL_SHA1_UNSAFE=YesPlease
export GIT_TEST_SPLIT_INDEX=yes
export GIT_TEST_FULL_IN_PACK_ARRAY=true export GIT_TEST_OE_SIZE=10
export GIT_TEST_OE_DELTA_SIZE=5
export GIT_TEST_COMMIT_GRAPH=1
export GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1 export GIT_TEST_MULTI_PACK_INDEX=1
export GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=1 export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master export GIT_TEST_NO_WRITE_REV_INDEX=1
export GIT_TEST_CHECKOUT_WORKERS=2
export GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL=1make && cd t && sh t9211-*.sh
--- 8< ---
PATCH v13 2/13 fsmonitor: fix khash memory leak in do_handle_client
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
The shown kh_str_t was freed with kh_release_str() at a point in the code only reachable in the non-trivial response path. When the client receives a trivial response, the code jumps to the cleanup label, skipping the kh_release_str() call entirely and leaking the hash table.
Fix this by initializing shown to NULL and moving the cleanup to the cleanup label using kh_destroy_str(), which is safe to call on NULL. This ensures the hash table is freed regardless of which code path is taken.
--- builtin/fsmonitor--daemon.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
builtin/fsmonitor--daemon.c +2 -3
@@ -671,7 +671,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,const struct fsmonitor_batch *batch;struct fsmonitor_batch *remainder =NULL ;intmax_t count =0 , duplicates =0 ; -kh_str_t *shown; +kh_str_t *shown =NULL ;int hash_ret;int do_trivial =0 ;int do_flush =0 ;
@@ -909,8 +909,6 @@ static int do_handle_client(struct fsmonitor_daemon_state *state, total_response_len += payload.len; } -kh_release_str (shown); -pthread_mutex_lock (&state->main_lock);if (token_data->client_ref_count >0 )
@@ -954,6 +952,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,trace2_data_intmax ("fsmonitor" , the_repository,"response/count/duplicates" , duplicates); cleanup: +kh_destroy_str (shown);strbuf_release (&response_token);strbuf_release (&requested_token_id);strbuf_release (&payload);
Signature · 1 line
gitgitgadget
PATCH v13 3/13 fsmonitor: fix hashmap memory leak in fsmonitor_run_daemon
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
The state.cookies hashmap is initialized during daemon startup but never freed during cleanup in the done: label of fsmonitor_run_daemon(). The cookie entries also have names allocated via strbuf_detach() that must be freed individually.
Iterate the hashmap to free each cookie name, then call hashmap_clear_and_free() to release the entries and table.
--- builtin/fsmonitor--daemon.c | 9 +++++++++ 1 file changed, 9 insertions(+)
builtin/fsmonitor--daemon.c +9 -0
@@ -1404,6 +1404,15 @@ static int fsmonitor_run_daemon(void) done:pthread_cond_destroy (&state.cookies_cond);pthread_mutex_destroy (&state.main_lock); + { +struct hashmap_iter iter; +struct fsmonitor_cookie_item *cookie; + +hashmap_for_each_entry (&state.cookies, &iter, cookie, entry) +free (cookie->name); +hashmap_clear_and_free (&state.cookies, + struct fsmonitor_cookie_item, entry); + }fsm_listen__dtor (&state);fsm_health__dtor (&state);
Signature · 1 line
gitgitgadget
PATCH v13 4/13 compat/win32: add pthread_cond_timedwait
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
Add a pthread_cond_timedwait() implementation to the Windows pthread compatibility layer using SleepConditionVariableCS() with a millisecond timeout computed from the absolute deadline.
--- compat/win32/pthread.c | 26 ++++++++++++++++++++++++++ compat/win32/pthread.h | 2 ++ 2 files changed, 28 insertions(+)
compat/win32/pthread.c +26 -0
@@ -66,3 +66,29 @@ int pthread_cond_wait(pthread_cond_t *cond, pthread_mutex_t *mutex)return err_win_to_posix (GetLastError ());return 0 ; } + +int pthread_cond_timedwait (pthread_cond_t *cond,pthread_mutex_t *mutex, +const struct timespec *abstime) +{ +struct timeval now; +long long now_ms, deadline_ms; +DWORD timeout_ms; + +gettimeofday (&now,NULL ); + now_ms = (long long )now.tv_sec *1000 + now.tv_usec /1000 ; + deadline_ms = (long long )abstime->tv_sec *1000 + + abstime->tv_nsec /1000000 ; + +if (deadline_ms <= now_ms) +return ETIMEDOUT ; +else + timeout_ms = (DWORD )(deadline_ms - now_ms); + +if (SleepConditionVariableCS (cond, mutex, timeout_ms) ==0 ) { +DWORD err =GetLastError (); +if (err ==ERROR_TIMEOUT ) +return ETIMEDOUT ; +return err_win_to_posix (err); + } +return 0 ; +}
compat/win32/pthread.h +2 -0
@@ -64,6 +64,8 @@ int win32_pthread_join(pthread_t *thread, void **value_ptr);pthread_t pthread_self (void );int pthread_cond_wait (pthread_cond_t *cond,pthread_mutex_t *mutex); +int pthread_cond_timedwait (pthread_cond_t *cond,pthread_mutex_t *mutex, +const struct timespec *abstime); static inline void NORETURN pthread_exit(void *ret) {
Signature · 1 line
gitgitgadget
PATCH v13 5/13 fsmonitor: use pthread_cond_timedwait for cookie wait
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
The cookie wait in with_lock__wait_for_cookie() uses an infinite pthread_cond_wait() loop. The existing comment notes the desire to switch to pthread_cond_timedwait(), but the routine was not available in git thread-utils.
On certain container or overlay filesystems, inotify watches may succeed but events are never delivered. In this case the daemon would hang indefinitely waiting for the cookie event, which in turn causes the client to hang.
Replace the infinite wait with a one-second timeout using pthread_cond_timedwait(). If the timeout fires, report an error and let the client proceed with a trivial (full-scan) response rather than blocking forever.
--- builtin/fsmonitor--daemon.c | 37 ++++++++++++++++++++++++------------- 1 file changed, 24 insertions(+), 13 deletions(-)
builtin/fsmonitor--daemon.c +24 -13
@@ -197,20 +197,31 @@ static enum fsmonitor_cookie_item_result with_lock__wait_for_cookie(unlink (cookie_pathname.buf);/* -* Technically, this is an infinite wait (well, unless another -* thread sends us an abort). I'd like to change this to -* use `pthread_cond_timedwait()` and return an error/timeout -* and let the caller do the trivial response thing, but we -* don't have that routine in our thread-utils. -* -* After extensive beta testing I'm not really worried about -* this. Also note that the above open() and unlink() calls -* will cause at least two FS events on that path, so the odds -* of getting stuck are pretty slim. +* Wait for the listener thread to observe the cookie file. +* Time out after a short interval so that the client +* does not hang forever if the filesystem does not deliver +* events (e.g., on certain container/overlay filesystems +* where inotify watches succeed but events never arrive). */ -while (cookie->result ==FCIR_INIT ) -pthread_cond_wait (&state->cookies_cond, - &state->main_lock); + { +struct timeval now; +struct timespec ts; +int err =0 ; + +gettimeofday (&now,NULL ); + ts.tv_sec = now.tv_sec +1 ; + ts.tv_nsec = now.tv_usec *1000 ; + +while (cookie->result ==FCIR_INIT && !err) + err =pthread_cond_timedwait (&state->cookies_cond, + &state->main_lock, + &ts); +if (err ==ETIMEDOUT && cookie->result ==FCIR_INIT ) { +trace_printf_key (&trace_fsmonitor, +"cookie_wait timed out" ); + cookie->result =FCIR_ERROR ; + } + } done:hashmap_remove (&state->cookies, &cookie->entry,NULL );
Signature · 1 line
gitgitgadget
PATCH v13 6/13 fsmonitor: rename fsm-ipc-darwin.c to fsm-ipc-unix.c
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
The fsmonitor IPC path logic in fsm-ipc-darwin.c is not Darwin-specific and will be reused by the upcoming Linux implementation. Rename it to fsm-ipc-unix.c to reflect that it is shared by all Unix platforms.
Introduce FSMONITOR_OS_SETTINGS (set to "unix" for non-Windows, "win32" for Windows) as a separate variable from FSMONITOR_DAEMON_BACKEND so that the build files can distinguish between platform-specific files (listen, health, path-utils) and shared Unix files (ipc, settings).
Move fsm-ipc to the FSMONITOR_OS_SETTINGS section in the Makefile, and switch fsm-path-utils to use FSMONITOR_DAEMON_BACKEND since path-utils is platform-specific (there will be separate darwin and linux versions).
---
Makefile | 6 ++---
.../{fsm-ipc-darwin.c => fsm-ipc-unix.c} | 0
config.mak.uname | 2 +-
contrib/buildsystems/CMakeLists.txt | 25 +++++++++----------
meson.build | 7 ++++--
5 files changed, 21 insertions(+), 19 deletions(-)
rename compat/fsmonitor/{fsm-ipc-darwin.c => fsm-ipc-unix.c} (100%)Makefile +3 -3
@@ -417,7 +417,7 @@ include shared.mak # If your platform has OS-specific ways to tell if a repo is incompatible with # fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS # to the "<name>" of the corresponding `compat/fsmonitor/fsm-settings-<name>.c` -# that implements the `fsm_os_settings__*()` routines. +# and `compat/fsmonitor/fsm-ipc-<name>.c` files. # # Define LINK_FUZZ_PROGRAMS if you want `make all` to also build the fuzz test # programs in oss-fuzz/.
@@ -2365,13 +2365,13 @@ ifdef FSMONITOR_DAEMON_BACKEND COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o - COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_BACKEND).o endif ifdef FSMONITOR_OS_SETTINGS COMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS + COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_OS_SETTINGS).o COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o - COMPAT_OBJS += compat/fsmonitor/fsm-path-utils-$(FSMONITOR_OS_SETTINGS).o + COMPAT_OBJS += compat/fsmonitor/fsm-path-utils-$(FSMONITOR_DAEMON_BACKEND).o endif ifdef WITH_BREAKING_CHANGES
compat/fsmonitor/fsm-ipc-unix.c renamed from compat/fsmonitor/fsm-ipc-darwin.c +0 -0
config.mak.uname +1 -1
@@ -165,7 +165,7 @@ ifeq ($(uname_S),Darwin) ifndef NO_PTHREADS ifndef NO_UNIX_SOCKETS FSMONITOR_DAEMON_BACKEND = darwin - FSMONITOR_OS_SETTINGS = darwin + FSMONITOR_OS_SETTINGS = unix endif endif
contrib/buildsystems/CMakeLists.txt +12 -13
@@ -291,23 +291,22 @@ endif() if(SUPPORTS_SIMPLE_IPC) if(CMAKE_SYSTEM_NAME STREQUAL "Windows") - add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-win32.c) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-win32.c) - - add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c) + set(FSMONITOR_DAEMON_BACKEND "win32") + set(FSMONITOR_OS_SETTINGS "win32") elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") + set(FSMONITOR_DAEMON_BACKEND "darwin") + set(FSMONITOR_OS_SETTINGS "unix") + endif() + + if(FSMONITOR_DAEMON_BACKEND) add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-darwin.c) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-darwin.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-${FSMONITOR_DAEMON_BACKEND}.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-${FSMONITOR_DAEMON_BACKEND}.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-${FSMONITOR_OS_SETTINGS}.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-${FSMONITOR_DAEMON_BACKEND}.c) add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-${FSMONITOR_DAEMON_BACKEND}.c) endif() endif()
meson.build +5 -2
@@ -1320,10 +1320,13 @@ else endif fsmonitor_backend = '' +fsmonitor_os = '' if host_machine.system() == 'windows' fsmonitor_backend = 'win32' + fsmonitor_os = 'win32' elif host_machine.system() == 'darwin' fsmonitor_backend = 'darwin' + fsmonitor_os = 'unix' libgit_dependencies += dependency('CoreServices') endif if fsmonitor_backend != ''
@@ -1332,14 +1335,14 @@ if fsmonitor_backend != '' libgit_sources += [ 'compat/fsmonitor/fsm-health-' + fsmonitor_backend + '.c', - 'compat/fsmonitor/fsm-ipc-' + fsmonitor_backend + '.c', + 'compat/fsmonitor/fsm-ipc-' + fsmonitor_os + '.c', 'compat/fsmonitor/fsm-listen-' + fsmonitor_backend + '.c', 'compat/fsmonitor/fsm-path-utils-' + fsmonitor_backend + '.c', 'compat/fsmonitor/fsm-settings-' + fsmonitor_backend + '.c', ] endif build_options_config.set_quoted('FSMONITOR_DAEMON_BACKEND', fsmonitor_backend) -build_options_config.set_quoted('FSMONITOR_OS_SETTINGS', fsmonitor_backend) +build_options_config.set_quoted('FSMONITOR_OS_SETTINGS', fsmonitor_os) if not get_option('b_sanitize').contains('address') and get_option('regex').allowed() and compiler.has_header('regex.h') and compiler.get_define('REG_STARTEND', prefix: '#include <regex.h>') != '' build_options_config.set('NO_REGEX', '')
Signature · 1 line
gitgitgadget
PATCH v13 7/13 fsmonitor: rename fsm-settings-darwin.c to fsm-settings-unix.c
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
The fsmonitor settings logic in fsm-settings-darwin.c is not Darwin-specific and will be reused by the upcoming Linux implementation. Rename it to fsm-settings-unix.c to reflect that it is shared by all Unix platforms.
Update the build files (meson.build and CMakeLists.txt) to use FSMONITOR_OS_SETTINGS for fsm-settings, matching the approach already used for fsm-ipc.
---
compat/fsmonitor/{fsm-settings-darwin.c => fsm-settings-unix.c} | 0
contrib/buildsystems/CMakeLists.txt | 2 +-
meson.build | 2 +-
3 files changed, 2 insertions(+), 2 deletions(-)
rename compat/fsmonitor/{fsm-settings-darwin.c => fsm-settings-unix.c} (100%)compat/fsmonitor/fsm-settings-unix.c renamed from compat/fsmonitor/fsm-settings-darwin.c +0 -0
contrib/buildsystems/CMakeLists.txt +1 -1
@@ -306,7 +306,7 @@ if(SUPPORTS_SIMPLE_IPC) list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-${FSMONITOR_DAEMON_BACKEND}.c) add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-${FSMONITOR_DAEMON_BACKEND}.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-${FSMONITOR_OS_SETTINGS}.c) endif() endif()
meson.build +1 -1
@@ -1338,7 +1338,7 @@ if fsmonitor_backend != '' 'compat/fsmonitor/fsm-ipc-' + fsmonitor_os + '.c', 'compat/fsmonitor/fsm-listen-' + fsmonitor_backend + '.c', 'compat/fsmonitor/fsm-path-utils-' + fsmonitor_backend + '.c', - 'compat/fsmonitor/fsm-settings-' + fsmonitor_backend + '.c', + 'compat/fsmonitor/fsm-settings-' + fsmonitor_os + '.c', ] endif build_options_config.set_quoted('FSMONITOR_DAEMON_BACKEND', fsmonitor_backend)
Signature · 1 line
gitgitgadget
PATCH v13 8/13 fsmonitor: implement filesystem change listener for Linux
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
Implement the built-in fsmonitor daemon for Linux using the inotify API, bringing it to feature parity with the existing Windows and macOS implementations.
The implementation uses inotify rather than fanotify because fanotify requires either CAP_SYS_ADMIN or CAP_PERFMON capabilities, making it unsuitable for an unprivileged user-space daemon. While inotify has the limitation of requiring a separate watch on every directory (unlike macOS's FSEvents, which can monitor an entire directory tree with a single watch), it operates without elevated privileges and provides the per-file event granularity needed for fsmonitor.
The listener uses inotify_init1(O_NONBLOCK) with a poll loop that checks for events with a 50-millisecond timeout, keeping the inotify queue well-drained to minimize the risk of overflows. Bidirectional hashmaps map between watch descriptors and directory paths for efficient event resolution. Directory renames are tracked using inotify's cookie mechanism to correlate IN_MOVED_FROM and IN_MOVED_TO event pairs; a periodic check detects stale renames where the matching IN_MOVED_TO never arrived, forcing a resync.
New directory creation triggers recursive watch registration to ensure all subdirectories are monitored. The IN_MASK_CREATE flag is used where available to prevent modifying existing watches, with a fallback for older kernels. When IN_MASK_CREATE is available and inotify_add_watch returns EEXIST, it means another thread or recursive scan has already registered the watch, so it is safe to ignore.
Remote filesystem detection uses statfs() to identify network-mounted filesystems (NFS, CIFS, SMB, FUSE, etc.) via their magic numbers. Mount point information is read from /proc/mounts and matched against the statfs f_fsid to get accurate, human-readable filesystem type names for logging. When the .git directory is on a remote filesystem, the IPC socket falls back to $HOME or a user-configured directory via the fsmonitor.socketDir setting.
--- Documentation/config/fsmonitor--daemon.adoc | 4 +- Documentation/git-fsmonitor--daemon.adoc | 28 +- compat/fsmonitor/fsm-health-linux.c | 33 + compat/fsmonitor/fsm-listen-linux.c | 746 ++++++++++++++++++++ compat/fsmonitor/fsm-path-utils-linux.c | 217 ++++++ config.mak.uname | 10 + contrib/buildsystems/CMakeLists.txt | 8 +- meson.build | 4 + 8 files changed, 1042 insertions(+), 8 deletions(-) create mode 100644 compat/fsmonitor/fsm-health-linux.c create mode 100644 compat/fsmonitor/fsm-listen-linux.c create mode 100644 compat/fsmonitor/fsm-path-utils-linux.c
Documentation/config/fsmonitor--daemon.adoc +2 -2
@@ -4,8 +4,8 @@ fsmonitor.allowRemote:: behavior. Only respected when `core.fsmonitor` is set to `true`. fsmonitor.socketDir:: - This Mac OS-specific option, if set, specifies the directory in + This Mac OS and Linux-specific option, if set, specifies the directory in which to create the Unix domain socket used for communication between the fsmonitor daemon and various Git commands. The directory must - reside on a native Mac OS filesystem. Only respected when `core.fsmonitor` + reside on a native filesystem. Only respected when `core.fsmonitor` is set to `true`.
Documentation/git-fsmonitor--daemon.adoc +24 -4
@@ -76,9 +76,9 @@ repositories; this may be overridden by setting `fsmonitor.allowRemote` to correctly with all network-mounted repositories, so such use is considered experimental. -On Mac OS, the inter-process communication (IPC) between various Git +On Mac OS and Linux, the inter-process communication (IPC) between various Git commands and the fsmonitor daemon is done via a Unix domain socket (UDS) -- a -special type of file -- which is supported by native Mac OS filesystems, +special type of file -- which is supported by native Mac OS and Linux filesystems, but not on network-mounted filesystems, NTFS, or FAT32. Other filesystems may or may not have the needed support; the fsmonitor daemon is not guaranteed to work with these filesystems and such use is considered experimental.
@@ -87,13 +87,33 @@ By default, the socket is created in the `.git` directory. However, if the `.git` directory is on a network-mounted filesystem, it will instead be created at `$HOME/.git-fsmonitor-*` unless `$HOME` itself is on a network-mounted filesystem, in which case you must set the configuration -variable `fsmonitor.socketDir` to the path of a directory on a Mac OS native +variable `fsmonitor.socketDir` to the path of a directory on a native filesystem in which to create the socket file. If none of the above directories (`.git`, `$HOME`, or `fsmonitor.socketDir`) -is on a native Mac OS file filesystem the fsmonitor daemon will report an +is on a native filesystem the fsmonitor daemon will report an error that will cause the daemon and the currently running command to exit. +LINUX CAVEATS +~~~~~~~~~~~~~ + +On Linux, the fsmonitor daemon uses inotify to monitor filesystem events. +The inotify system has per-user limits on the number of watches that can +be created. The default limit is typically 8192 watches per user. + +For large repositories with many directories, you may need to increase +this limit. Check the current limit with: + + cat /proc/sys/fs/inotify/max_user_watches + +To temporarily increase the limit: + + sudo sysctl fs.inotify.max_user_watches=65536 + +To make the change permanent, add to `/etc/sysctl.conf`: + + fs.inotify.max_user_watches=65536 + CONFIGURATION -------------
compat/fsmonitor/fsm-health-linux.c new file +33 -0
@@ -0,0 +1,33 @@ +#include "git-compat-util.h" +#include "config.h" +#include "fsmonitor-ll.h" +#include "fsm-health.h" +#include "fsmonitor--daemon.h" + +/* +* The Linux fsmonitor implementation uses inotify which has its own +* mechanisms for detecting filesystem unmount and other events that +* would require the daemon to shutdown. Therefore, we don't need +* a separate health thread like Windows does. +* +* These stub functions satisfy the interface requirements. +*/ + +int fsm_health__ctor(struct fsmonitor_daemon_state *state UNUSED) +{ +return 0 ; +} + +void fsm_health__dtor(struct fsmonitor_daemon_state *state UNUSED) +{ +return ; +} + +void fsm_health__loop(struct fsmonitor_daemon_state *state UNUSED) +{ +return ; +} + +void fsm_health__stop_async(struct fsmonitor_daemon_state *state UNUSED) +{ +}
compat/fsmonitor/fsm-listen-linux.c new file +746 -0
746 lines not on this page · raw message
compat/fsmonitor/fsm-path-utils-linux.c new file +217 -0
217 lines not on this page · raw message
config.mak.uname +10 -0
@@ -68,6 +68,16 @@ ifeq ($(uname_S),Linux) BASIC_CFLAGS += -std=c99 endif LINK_FUZZ_PROGRAMS = YesPlease + + # The builtin FSMonitor on Linux builds upon Simple-IPC. Both require + # Unix domain sockets and PThreads. + ifndef NO_PTHREADS + ifndef NO_UNIX_SOCKETS + FSMONITOR_DAEMON_BACKEND = linux + FSMONITOR_OS_SETTINGS = unix + BASIC_CFLAGS += -DHAVE_LINUX_MAGIC_H + endif + endif endif ifeq ($(uname_S),GNU/kFreeBSD) HAVE_ALLOCA_H = YesPlease
contrib/buildsystems/CMakeLists.txt +6 -2
@@ -296,6 +296,10 @@ if(SUPPORTS_SIMPLE_IPC) elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") set(FSMONITOR_DAEMON_BACKEND "darwin") set(FSMONITOR_OS_SETTINGS "unix") + elseif(CMAKE_SYSTEM_NAME STREQUAL "Linux") + set(FSMONITOR_DAEMON_BACKEND "linux") + set(FSMONITOR_OS_SETTINGS "unix") + add_compile_definitions(HAVE_LINUX_MAGIC_H) endif() if(FSMONITOR_DAEMON_BACKEND)
@@ -1149,8 +1153,8 @@ endif() file(STRINGS ${CMAKE_SOURCE_DIR}/GIT-BUILD-OPTIONS.in git_build_options NEWLINE_CONSUME) string(REPLACE "@BROKEN_PATH_FIX@" "" git_build_options "${git_build_options}") string(REPLACE "@DIFF@" "'${DIFF}'" git_build_options "${git_build_options}") -string(REPLACE "@FSMONITOR_DAEMON_BACKEND@" "win32" git_build_options "${git_build_options}") -string(REPLACE "@FSMONITOR_OS_SETTINGS@" "win32" git_build_options "${git_build_options}") +string(REPLACE "@FSMONITOR_DAEMON_BACKEND@" "${FSMONITOR_DAEMON_BACKEND}" git_build_options "${git_build_options}") +string(REPLACE "@FSMONITOR_OS_SETTINGS@" "${FSMONITOR_OS_SETTINGS}" git_build_options "${git_build_options}") string(REPLACE "@GITWEBDIR@" "'${GITWEBDIR}'" git_build_options "${git_build_options}") string(REPLACE "@GIT_INTEROP_MAKE_OPTS@" "" git_build_options "${git_build_options}") string(REPLACE "@GIT_PERF_LARGE_REPO@" "" git_build_options "${git_build_options}")
meson.build +4 -0
@@ -1324,6 +1324,10 @@ fsmonitor_os = '' if host_machine.system() == 'windows' fsmonitor_backend = 'win32' fsmonitor_os = 'win32' +elif host_machine.system() == 'linux' and threads.found() and compiler.has_header('linux/magic.h') + fsmonitor_backend = 'linux' + fsmonitor_os = 'unix' + libgit_c_args += '-DHAVE_LINUX_MAGIC_H' elif host_machine.system() == 'darwin' fsmonitor_backend = 'darwin' fsmonitor_os = 'unix'
Signature · 1 line
gitgitgadget
PATCH v13 9/13 run-command: add close_fd_above_stderr option
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
Add a close_fd_above_stderr flag to struct child_process. When set, the child closes file descriptors 3 and above between fork and exec (skipping the child-notifier pipe), capped at sysconf(_SC_OPEN_MAX) or 4096, whichever is smaller. This prevents the child from inheriting pipe endpoints or other descriptors from the parent environment (e.g., the test harness).
--- run-command.c | 12 ++++++++++++ run-command.h | 9 +++++++++ 2 files changed, 21 insertions(+)
run-command.c +12 -0
@@ -546,6 +546,7 @@ static void atfork_parent(struct atfork_state *as) "restoring signal mask"); #endif } +#endif /* GIT_WINDOWS_NATIVE */ static inline void set_cloexec (int fd)
@@ -832,6 +833,17 @@ fail_pipe:child_close (cmd->out); } +if (cmd->close_fd_above_stderr) { +long max_fd =sysconf (_SC_OPEN_MAX); +int fd; +if (max_fd <0 || max_fd >4096 ) + max_fd =4096 ; +for (fd =3 ; fd < max_fd; fd++) { +if (fd != child_notifier) +close (fd); + } + } +if (cmd->dir &&chdir (cmd->dir))child_die (CHILD_ERR_CHDIR );
run-command.h +9 -0
@@ -141,6 +141,15 @@ struct child_process { unsigned stdout_to_stderr:1; unsigned clean_on_exit:1; unsigned wait_after_clean:1; + +/** +* Close file descriptors 3 and above in the child after forking +* but before exec. This prevents the child from inheriting +* pipe endpoints or other descriptors from the parent +* environment (e.g., the test harness). +*/ + unsigned close_fd_above_stderr:1; +void (*clean_on_exit_handler)(struct child_process *process); };
Signature · 1 line
gitgitgadget
PATCH v13 10/13 fsmonitor: close inherited file descriptors and detach in daemon
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
When the fsmonitor daemon is spawned as a background process, it may inherit file descriptors from its parent that it does not need. In particular, when the test harness or a CI system captures output through pipes, the daemon can inherit duplicated pipe endpoints. If the daemon holds these open, the parent process never sees EOF and may appear to hang.
Set close_fd_above_stderr on the child process at both daemon startup
paths: the explicit "fsmonitor--daemon start" command and the implicit spawn triggered by fsmonitor-ipc when a client finds no running daemon. Also suppress stdout and stderr on the implicit spawn path to prevent the background daemon from writing to the client's terminal.
Additionally, call setsid() when the daemon starts with --detach to create a new session and process group. This prevents the daemon from being part of the spawning shell's process group, which could cause the shell's "wait" to block until the daemon exits.
--- builtin/fsmonitor--daemon.c | 16 ++++++++++++++-- fsmonitor-ipc.c | 3 +++ 2 files changed, 17 insertions(+), 2 deletions(-)
builtin/fsmonitor--daemon.c +14 -2
@@ -1439,7 +1439,7 @@ done:return err; } -static int try_to_run_foreground_daemon(int detach_console MAYBE_UNUSED) +static int try_to_run_foreground_daemon (int detach_console) {/* * Technically, we don't need to probe for an existing daemon
@@ -1459,10 +1459,21 @@ static int try_to_run_foreground_daemon(int detach_console MAYBE_UNUSED)fflush (stderr); } +if (detach_console) {#ifdef GIT_WINDOWS_NATIVE -if (detach_console)FreeConsole (); +#else +/* +* Create a new session so that the daemon is detached +* from the parent's process group. This prevents +* shells with job control (e.g. bash with "set -m") +* from waiting on the daemon when they wait for a +* foreground command that implicitly spawned it. +*/ +if (setsid () ==-1 ) +warning_errno (_ ("setsid failed" ));#endif + }return !!fsmonitor_run_daemon (); }
@@ -1525,6 +1536,7 @@ static int try_to_start_background_daemon(void) cp.no_stdin =1 ; cp.no_stdout =1 ; cp.no_stderr =1 ; + cp.close_fd_above_stderr =1 ; sbgr =start_bg_command (&cp, bg_wait_cb,NULL , fsmonitor__start_timeout_sec);
fsmonitor-ipc.c +3 -0
@@ -61,6 +61,9 @@ static int spawn_daemon(void) cmd.git_cmd =1 ; cmd.no_stdin =1 ; + cmd.no_stdout =1 ; + cmd.no_stderr =1 ; + cmd.close_fd_above_stderr =1 ; cmd.trace2_child_class ="fsmonitor" ;strvec_pushl (&cmd.args,"fsmonitor--daemon" ,"start" ,NULL );
Signature · 1 line
gitgitgadget
PATCH v13 11/13 fsmonitor: add timeout to daemon stop command
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
The "fsmonitor--daemon stop" command polls in a loop waiting for the daemon to exit after sending a "quit" command over IPC. If the daemon fails to shut down (e.g. it is stuck or wedged), this loop spins forever.
Add a 30-second timeout so the stop command returns an error instead of blocking indefinitely.
--- builtin/fsmonitor--daemon.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
builtin/fsmonitor--daemon.c +11 -1
@@ -86,6 +86,8 @@ static int do_as_client__send_stop(void) {struct strbuf answer =STRBUF_INIT ;int ret; +int max_wait_ms =30000 ; +int elapsed_ms =0 ; ret =fsmonitor_ipc__send_command ("quit" , &answer);
@@ -96,8 +98,16 @@ static int do_as_client__send_stop(void)return ret;trace2_region_enter ("fsm_client" ,"polling-for-daemon-exit" ,NULL ); -while (fsmonitor_ipc__get_state () ==IPC_STATE__LISTENING ) +while (fsmonitor_ipc__get_state () ==IPC_STATE__LISTENING ) { +if (elapsed_ms >= max_wait_ms) { +trace2_region_leave ("fsm_client" , +"polling-for-daemon-exit" ,NULL ); +return error (_ ("daemon did not stop within %d seconds" ), + max_wait_ms /1000 ); + }sleep_millisec (50 ); + elapsed_ms +=50 ; + }trace2_region_leave ("fsm_client" ,"polling-for-daemon-exit" ,NULL );return 0 ;
Signature · 1 line
gitgitgadget
PATCH v13 12/13 fsmonitor: add tests for Linux
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
Add a smoke test that verifies the filesystem actually delivers inotify events to the daemon. On some configurations (e.g., overlayfs with older kernels), inotify watches succeed but events are never delivered. The daemon cookie wait will time out, but every subsequent test would fail. Skip the entire test file early when this is detected.
Add a test that exercises rapid nested directory creation to verify the daemon correctly handles the EEXIST race between recursive scan and queued inotify events. When IN_MASK_CREATE is available and a directory watch is added during recursive registration, the kernel may also deliver a queued IN_CREATE event for the same directory. The second inotify_add_watch() returns EEXIST, which must be treated as harmless. An earlier version of the listener crashed in this scenario.
Reduce --start-timeout from the default 60 seconds to 10 seconds so that tests fail promptly when the daemon cannot start.
Harden the test helpers to work in environments without procps (e.g., Fedora CI): fall back to reading /proc/$pid/stat for the process group ID when ps is unavailable, guard stop_git() against an empty pgid, and redirect stderr from kill to /dev/null to avoid noise when processes have already exited.
Use set -m to enable job control in the submodule-pull test so that the background git pull gets its own process group, preventing the shell wait from blocking on the daemon. setsid() in the previous commit detaches the daemon itself, but the intermediate git pull process still needs its own process group for the test shell to manage it correctly.
--- t/t7527-builtin-fsmonitor.sh | 89 +++++++++++++++++++++++++++++++++--- 1 file changed, 82 insertions(+), 7 deletions(-)
t/t7527-builtin-fsmonitor.sh +82 -7
@@ -10,9 +10,58 @@ then test_done fi +# Verify that the filesystem delivers events to the daemon. +# On some configurations (e.g., overlayfs with older kernels), +# inotify watches succeed but events are never delivered. The +# cookie wait will time out and the daemon logs a trace message. +# +# Use "timeout" (if available) to guard each step against hangs. +maybe_timeout () { + if type timeout >/dev/null 2>&1 + then + timeout "$@" + else + shift + "$@" + fi +} +verify_fsmonitor_works () { + git init test_fsmonitor_smoke || return 1 + + GIT_TRACE_FSMONITOR="$PWD/smoke.trace" && + export GIT_TRACE_FSMONITOR && + maybe_timeout 30 \ + git -C test_fsmonitor_smoke fsmonitor--daemon start \ + --start-timeout=10 + ret=$? + unset GIT_TRACE_FSMONITOR + if test $ret -ne 0 + then + rm -rf test_fsmonitor_smoke smoke.trace + return 1 + fi + + maybe_timeout 10 \ + test-tool -C test_fsmonitor_smoke fsmonitor-client query \ + --token 0 >/dev/null 2>&1 + maybe_timeout 5 \ + git -C test_fsmonitor_smoke fsmonitor--daemon stop 2>/dev/null + ! grep -q "cookie_wait timed out" "$PWD/smoke.trace" 2>/dev/null + ret=$? + rm -rf test_fsmonitor_smoke smoke.trace + return $ret +} + +if ! verify_fsmonitor_works +then + skip_all="filesystem does not deliver fsmonitor events (container/overlayfs?)" + test_done +fi + stop_daemon_delete_repo () { r=$1 && - test_might_fail git -C $r fsmonitor--daemon stop && + test_might_fail maybe_timeout 30 \ + git -C $r fsmonitor--daemon stop 2>/dev/null rm -rf $1 }
@@ -67,7 +116,7 @@ start_daemon () { export GIT_TEST_FSMONITOR_TOKEN fi && - git $r fsmonitor--daemon start && + git $r fsmonitor--daemon start --start-timeout=10 && git $r fsmonitor--daemon status ) }
@@ -520,6 +569,28 @@ test_expect_success 'directory changes to a file' ' grep "^event: dir1$" .git/trace ' +test_expect_success 'rapid nested directory creation' ' + test_when_finished "git fsmonitor--daemon stop; rm -rf rapid" && + + start_daemon --tf "$PWD/.git/trace" && + + # Rapidly create nested directories to exercise race conditions + # where directory watches may be added concurrently during + # event processing and recursive scanning. + for i in $(test_seq 1 20) + do + mkdir -p "rapid/nested/dir$i/subdir/deep" || return 1 + done && + + # Give the daemon time to process all events + sleep 1 && + + test-tool fsmonitor-client query --token 0 && + + # Verify daemon is still running (did not crash) + git fsmonitor--daemon status +' + # The next few test cases exercise the token-resync code. When filesystem # drops events (because of filesystem velocity or because the daemon isn't # polling fast enough), we need to discard the cached data (relative to the
@@ -910,7 +981,10 @@ test_expect_success "submodule absorbgitdirs implicitly starts daemon" ' start_git_in_background () { git "$@" & git_pid=$! - git_pgid=$(ps -o pgid= -p $git_pid) + git_pgid=$(ps -o pgid= -p $git_pid 2>/dev/null || + awk '{print $5}' /proc/$git_pid/stat 2>/dev/null) && + git_pgid="${git_pgid## }" && + git_pgid="${git_pgid%% }" nr_tries_left=10 while true do
@@ -921,15 +995,16 @@ start_git_in_background () { fi sleep 1 nr_tries_left=$(($nr_tries_left - 1)) - done >/dev/null 2>&1 & + done >/dev/null 2>&1 3>&- 4>&- 5>&- 6>&- 7>&- & watchdog_pid=$! wait $git_pid } stop_git () { - while kill -0 -- -$git_pgid + test -n "$git_pgid" || return 0 + while kill -0 -- -$git_pgid 2>/dev/null do - kill -- -$git_pgid + kill -- -$git_pgid 2>/dev/null sleep 1 done }
@@ -944,7 +1019,7 @@ stop_watchdog () { test_expect_success !MINGW "submodule implicitly starts daemon by pull" ' test_atexit "stop_watchdog" && - test_when_finished "stop_git; rm -rf cloned super sub" && + test_when_finished "set +m; stop_git; rm -rf cloned super sub" && create_super super && create_sub sub &&
Signature · 1 line
gitgitgadget
PATCH v13 13/13 fsmonitor: convert shown khash to strset in do_handle_client
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
Replace the khash-based string set used for deduplicating pathnames in do_handle_client() with a strset, which provides a cleaner interface for the same purpose.
Since the paths are interned strings from the batch data, use strdup_strings=0 to avoid unnecessary copies.
--- builtin/fsmonitor--daemon.c | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-)
builtin/fsmonitor--daemon.c +6 -11
@@ -16,7 +16,7 @@#include "fsmonitor--daemon.h" #include "simple-ipc.h" -#include "khash.h" +#include "strmap.h" #include "run-command.h" #include "trace.h" #include "trace2.h"
@@ -674,8 +674,6 @@ static int fsmonitor_parse_client_token(const char *buf_token,return 0 ; } -KHASH_INIT(str, const char *, int, 0, kh_str_hash_func, kh_str_hash_equal) - static int do_handle_client(struct fsmonitor_daemon_state *state, const char *command, ipc_server_reply_cb *reply,
@@ -692,8 +690,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,const struct fsmonitor_batch *batch;struct fsmonitor_batch *remainder =NULL ;intmax_t count =0 , duplicates =0 ; -kh_str_t *shown =NULL ; -int hash_ret; +struct strset shown =STRSET_INIT ;int do_trivial =0 ;int do_flush =0 ;int do_cookie =0 ;
@@ -882,14 +879,14 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,* so walk the batch list backwards from the current head back * to the batch (sequence number) they named. * -* We use khash to de-dup the list of pathnames. +* We use a strset to de-dup the list of pathnames. * * NEEDSWORK: each batch contains a list of interned strings, * so we only need to do pointer comparisons here to build the * hash table. Currently, we're still comparing the string * values. */ - shown =kh_init_str (); +strset_init_with_options (&shown,NULL ,0 );for (batch = batch_head; batch && batch->batch_seq_nr > requested_oldest_seq_nr; batch = batch->next) {
@@ -899,11 +896,9 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,const char *s = batch->interned_paths[k];size_t s_len; -if (kh_get_str (shown, s) !=kh_end (shown)) +if (!strset_add (&shown, s)) duplicates++;else { -kh_put_str (shown, s, &hash_ret); -trace_printf_key (&trace_fsmonitor,"send[%" PRIuMAX"]: %s" , count, s);
@@ -973,7 +968,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,trace2_data_intmax ("fsmonitor" , the_repository,"response/count/duplicates" , duplicates); cleanup: -kh_destroy_str (shown); +strset_clear (&shown);strbuf_release (&response_token);strbuf_release (&requested_token_id);strbuf_release (&payload);
Signature · 1 line
gitgitgadget
v14 cover fsmonitor: implement filesystem change listener for Linux
In reply to Paul Tarjan via GitGitGadgetThis series implements the built-in fsmonitor daemon for Linux using the inotify API, bringing it to feature parity with the existing Windows and macOS implementations. It also fixes two memory leaks in the platform-independent daemon code and deduplicates the IPC and settings logic that is now shared between macOS and Linux.
The implementation uses inotify rather than fanotify because fanotify requires either CAP_SYS_ADMIN or CAP_PERFMON capabilities, making it unsuitable for an unprivileged user-space daemon. While inotify has the limitation of requiring a separate watch on every directory (unlike macOS FSEvents, which can monitor an entire directory tree with a single watch), it operates without elevated privileges and provides the per-file event granularity needed for fsmonitor.
The listener uses inotify_init1(O_NONBLOCK) with a poll loop that checks for events with a 50-millisecond timeout, keeping the inotify queue well-drained to minimize the risk of overflows. Bidirectional hashmaps map between watch descriptors and directory paths for efficient event resolution. Directory renames are tracked using inotify cookie mechanism to correlate IN_MOVED_FROM and IN_MOVED_TO event pairs; a periodic check detects stale renames where the matching IN_MOVED_TO never arrived, forcing a resync.
New directory creation triggers recursive watch registration to ensure all subdirectories are monitored. The IN_MASK_CREATE flag is used where available to prevent modifying existing watches, with a fallback for older kernels. When IN_MASK_CREATE is available and inotify_add_watch returns EEXIST, it means another thread or recursive scan has already registered the watch, so it is safe to ignore.
Remote filesystem detection uses statfs() to identify network-mounted filesystems (NFS, CIFS, SMB, FUSE, etc.) via their magic numbers. Mount point information is read from /proc/mounts and matched against the statfs f_fsid to get accurate, human-readable filesystem type names for logging. When the .git directory is on a remote filesystem, the IPC socket falls back to $HOME or a user-configured directory via the fsmonitor.socketDir setting.
This series builds on work from https://github.com/git/git/pull/1352 by Eric DeCosta and https://github.com/git/git/pull/1667 by Marziyeh Esipreh, updated to work with the current codebase and address all review feedback.
Changes since v13:
* Also disable GIT_TEST_SPLIT_INDEX in t9211 per Junio's feedback (every test does scalar clone)
Changes since v12:
* Dropped both the fsmonitor.c workaround and the read-cache.c skipHash fix per Dscho's review: split-index and index.skipHash are fundamentally incompatible
* Added sane_unset GIT_TEST_SPLIT_INDEX to scalar clone tests in t9210 (patch 1/13)
Changes since v11:
* Fix t9210 BUG assertion with GIT_TEST_SPLIT_INDEX=yes: guard fsmonitor_dirty bitmap access against split-index having fewer entries than the bitmap expects
Changes since v10:
* Reverted pre_exec_cb callback back to simple close_fd_above_stderr flag per Junio's clarification (same as v8)
Changes since v9:
* Fixed Windows build: close_fd_above_stderr() compiles as a no-op on Windows since there is no fork/exec
Changes since v8:
* Replaced close_fd_above_stderr flag with generic pre_exec_cb callback in struct child_process per Junio's review, with close_fd_above_stderr() as a ready-made callback
Changes since v7:
* Added patch 12: convert khash to strset in do_handle_client (Patrick's #leftoverbit suggestion)
* Fixed "Forcing shutdown" trace message to start with lowercase
* Fixed redundant statfs() call in find_mount() (caller already had the result)
* Fixed CMakeLists.txt GIT-BUILD-OPTIONS: was hardcoded to "win32" for FSMONITOR_DAEMON_BACKEND and FSMONITOR_OS_SETTINGS, now uses the CMake variables
* Fixed uninitialized strset on trivial response path (STRSET_INIT)
* Removed V9FS_MAGIC from get_fs_typename() to match is_remote_fs() (9p is local VM mounts)
* Split 30-second stop timeout into its own commit per review request
* Fixed misleading indentation on shutdown assignment in handle_events()
* Updated commit messages to describe all changes (test hardening, fsmonitor-ipc.c spawn changes)
* Updated Makefile comment for FSMONITOR_OS_SETTINGS to mention fsm-ipc
Changes since v6:
* Introduced FSMONITOR_OS_SETTINGS build variable (set to "unix" for macOS and Linux, "win32" for Windows) to eliminate if/else conditionals in Makefile, meson.build, and CMakeLists.txt per Junio's review
* Moved fsm-path-utils from FSMONITOR_OS_SETTINGS to FSMONITOR_DAEMON_BACKEND since path-utils files are platform-specific
* Removed V9FS_MAGIC from remote filesystem detection (9p is used for local VM/container host mounts where fsmonitor works fine)
* Removed redundant #include <libgen.h> (already provided by compat/posix.h)
* Fixed cookie wait comment wording ("see" to "observe")
* Rewrote commit messages for IPC and settings dedup patches
Changes since v5:
* Split monolithic commit into 10-patch series per Patrick's review
* Deduplicated fsm-ipc and fsm-settings into shared Unix implementations
* Rewrote commit message with prose paragraphs, explain inotify vs fanotify, removed "Issues addressed" sections, added Based-on-patch-by trailers
* Removed redundant includes already provided by compat/posix.h
* Fixed error/trace message capitalization per coding guidelines
* Fixed stale rename check interval from 1000 seconds to 1 second
* Changed poll timeout from 1ms to 50ms to reduce idle CPU wake-ups
* Replaced infinite pthread_cond_wait cookie loop with one-second pthread_cond_timedwait (prevents daemon hangs on overlay filesystems where events are never delivered)
* Added pthread_cond_timedwait to Windows pthread compatibility layer
* Separated test into its own commit with smoke test that skips when inotify events are not delivered (e.g., overlayfs with older kernels)
* Fixed test hang on Fedora CI: stop_git() looped forever when ps was unavailable because bash in POSIX/sh mode returns exit 0 from kill with an empty process group argument. Fixed by falling back to /proc/$pid/stat for process group ID and guarding stop_git against empty pgid.
* Redirect spawn_daemon() stdout/stderr to /dev/null and close inherited file descriptors to prevent the intermediate process from holding test pipe file descriptors
* Call setsid() on daemon detach to prevent shells with job control from waiting on the daemon process group
* Close inherited file descriptors 3-7 in the test watchdog subprocess
* Added 30-second timeout to "fsmonitor--daemon stop" to prevent indefinite blocking
* Added helpful error message when inotify watch limit (max_user_watches) is reached
* Initialize fd_inotify to -1 and use fd >= 0 check for correct fd 0 handling
* Use sysconf(_SC_OPEN_MAX) instead of hardcoded 1024 for fd close limit
* Check setsid() return value
Changes since v4:
* Added Meson build support
Changes since v3:
* Fix crash on rapid nested directory creation (EEXIST from inotify_add_watch with IN_MASK_CREATE)
* Extensive stress testing
Changes since v2:
* Fix khash memory leak in do_handle_client
Changes since v1:
* Fix hashmap memory leak in fsmonitor_run_daemon()
Paul Tarjan (13):
t9210, t9211: disable GIT_TEST_SPLIT_INDEX for scalar clone tests
fsmonitor: fix khash memory leak in do_handle_client
fsmonitor: fix hashmap memory leak in fsmonitor_run_daemon compat/win32: add pthread_cond_timedwait
fsmonitor: use pthread_cond_timedwait for cookie wait
fsmonitor: rename fsm-ipc-darwin.c to fsm-ipc-unix.c
fsmonitor: rename fsm-settings-darwin.c to fsm-settings-unix.c
fsmonitor: implement filesystem change listener for Linux
run-command: add close_fd_above_stderr option
fsmonitor: close inherited file descriptors and detach in daemon
fsmonitor: add timeout to daemon stop command
fsmonitor: add tests for Linux
fsmonitor: convert shown khash to strset in do_handle_client
Documentation/config/fsmonitor--daemon.adoc | 4 +-
Documentation/git-fsmonitor--daemon.adoc | 28 +-
Makefile | 6 +-
builtin/fsmonitor--daemon.c | 92 ++-
compat/fsmonitor/fsm-health-linux.c | 33 +
.../{fsm-ipc-darwin.c => fsm-ipc-unix.c} | 0
compat/fsmonitor/fsm-listen-linux.c | 746 ++++++++++++++++++
compat/fsmonitor/fsm-path-utils-linux.c | 217 +++++
...-settings-darwin.c => fsm-settings-unix.c} | 0
compat/win32/pthread.c | 26 +
compat/win32/pthread.h | 2 +
config.mak.uname | 12 +-
contrib/buildsystems/CMakeLists.txt | 33 +-
fsmonitor-ipc.c | 3 +
meson.build | 13 +-
run-command.c | 12 +
run-command.h | 9 +
t/t7527-builtin-fsmonitor.sh | 89 ++-
t/t9210-scalar.sh | 6 +
t/t9211-scalar-clone.sh | 5 +
20 files changed, 1273 insertions(+), 63 deletions(-)
create mode 100644 compat/fsmonitor/fsm-health-linux.c
rename compat/fsmonitor/{fsm-ipc-darwin.c => fsm-ipc-unix.c} (100%)
create mode 100644 compat/fsmonitor/fsm-listen-linux.c
create mode 100644 compat/fsmonitor/fsm-path-utils-linux.c
rename compat/fsmonitor/{fsm-settings-darwin.c => fsm-settings-unix.c} (100%)base-commit: 3e0db84c88c57e70ac8be8c196dfa92c5d656fbc
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-2147%2Fptarjan%2Fclaude%2Fupdate-pr-1352-current-85Gk8-v14
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-2147/ptarjan/claude/update-pr-1352-current-85Gk8-v14
Pull-Request: https://github.com/git/git/pull/2147
Range-diff vs v13:
1: 28c5aca413 ! 1: 721a951423 t9210: disable GIT_TEST_SPLIT_INDEX for scalar clone tests
@@ Metadata
Author: Paul Tarjan <[email protected]>
## Commit message ##
- t9210: disable GIT_TEST_SPLIT_INDEX for scalar clone tests
+ t9210, t9211: disable GIT_TEST_SPLIT_INDEX for scalar clone tests
index.skipHash (Scalar default) and split-index are incompatible:
the shared index gets a null OID when skipHash skips computing the
@@ Commit message
index has.
Disable GIT_TEST_SPLIT_INDEX in the scalar clone tests that hit
- this, matching the existing workaround in test 16.
+ this: tests 12, 13, and 22 in t9210 (matching the existing
+ workaround in test 16), and all of t9211 (every test does scalar
+ clone).
@@ t/t9210-scalar.sh: test_expect_success 'scalar [...] <dir> errors out when dir
scalar clone "file://$(pwd)" cloned --single-branch &&
git repack &&
echo "$(pwd)/.git/objects/" >>cloned/src/.git/objects/info/alternates &&
+
+ ## t/t9211-scalar-clone.sh ##
+@@ t/t9211-scalar-clone.sh: test_description='test the scalar clone subcommand'
+ GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt,launchctl:true,schtasks:true"
+ export GIT_TEST_MAINT_SCHEDULER
+
++# index.skipHash (Scalar default) and GIT_TEST_SPLIT_INDEX are
++# incompatible: the shared index gets a null OID and fails to
++# load on re-read. Every test here uses scalar clone.
++sane_unset GIT_TEST_SPLIT_INDEX
++
+ test_expect_success 'set up repository to clone' '
+ rm -rf .git &&
+ git init to-clone &&
2: 9f666beea7 = 2: 1283d25968 fsmonitor: fix khash memory leak in do_handle_client 3: a4a65a6dfa = 3: 11ba6ca9ac fsmonitor: fix hashmap memory leak in fsmonitor_run_daemon 4: 427073fc38 = 4: a0d430c2f4 compat/win32: add pthread_cond_timedwait 5: 71effc4d47 = 5: 7b62c95c44 fsmonitor: use pthread_cond_timedwait for cookie wait 6: e897193b0e = 6: 7086cd4530 fsmonitor: rename fsm-ipc-darwin.c to fsm-ipc-unix.c 7: e13d938ddb = 7: 46e8c2b74f fsmonitor: rename fsm-settings-darwin.c to fsm-settings-unix.c 8: 3431eae60f = 8: b3f40a497b fsmonitor: implement filesystem change listener for Linux 9: 7ce0ab87fb = 9: 5791edbef2 run-command: add close_fd_above_stderr option 10: 2bf134a041 = 10: 22d425ebeb fsmonitor: close inherited file descriptors and detach in daemon 11: cb4d511f21 = 11: fd6bdc8c55 fsmonitor: add timeout to daemon stop command 12: 9a8647884e = 12: f85983ca93 fsmonitor: add tests for Linux 13: a5fc9ad415 = 13: 2085b21e23 fsmonitor: convert shown khash to strset in do_handle_client
Signature · 1 line
gitgitgadget
PATCH v14 1/13 t9210, t9211: disable GIT_TEST_SPLIT_INDEX for scalar clone tests
From: Paul Tarjan <[email protected]>
index.skipHash (Scalar default) and split-index are incompatible:
the shared index gets a null OID when skipHash skips computing the hash, and the null OID causes the shared index to not be loaded on re-read. This triggers a BUG assertion in fsmonitor when the fsmonitor_dirty bitmap references more entries than the (now empty) index has.
Disable GIT_TEST_SPLIT_INDEX in the scalar clone tests that hit
this: tests 12, 13, and 22 in t9210 (matching the existing workaround in test 16), and all of t9211 (every test does scalar clone).
--- t/t9210-scalar.sh | 6 ++++++ t/t9211-scalar-clone.sh | 5 +++++ 2 files changed, 11 insertions(+)
t/t9210-scalar.sh +6 -0
@@ -152,6 +152,10 @@ test_expect_success 'set up repository to clone' ' ' test_expect_success 'scalar clone' ' + # index.skipHash (Scalar default) and GIT_TEST_SPLIT_INDEX are + # incompatible: the shared index gets a null OID and fails to + # load on re-read. + sane_unset GIT_TEST_SPLIT_INDEX && second=$(git rev-parse --verify second:second.t) && scalar clone "file://$(pwd)" cloned --single-branch && (
@@ -182,6 +186,7 @@ test_expect_success 'scalar clone' ' ' test_expect_success 'scalar clone --no-... opts' ' + sane_unset GIT_TEST_SPLIT_INDEX && # Note: redirect stderr always to avoid having a verbose test # run result in a difference in the --[no-]progress option. GIT_TRACE2_EVENT="$(pwd)/no-opt-trace" scalar clone \
@@ -307,6 +312,7 @@ test_expect_success '`scalar [...] <dir>` errors out when dir is missing' ' SQ="'" test_expect_success UNZIP 'scalar diagnose' ' + sane_unset GIT_TEST_SPLIT_INDEX && scalar clone "file://$(pwd)" cloned --single-branch && git repack && echo "$(pwd)/.git/objects/" >>cloned/src/.git/objects/info/alternates &&
t/t9211-scalar-clone.sh +5 -0
@@ -8,6 +8,11 @@ test_description='test the `scalar clone` subcommand' GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt,launchctl:true,schtasks:true" export GIT_TEST_MAINT_SCHEDULER +# index.skipHash (Scalar default) and GIT_TEST_SPLIT_INDEX are +# incompatible: the shared index gets a null OID and fails to +# load on re-read. Every test here uses scalar clone. +sane_unset GIT_TEST_SPLIT_INDEX + test_expect_success 'set up repository to clone' ' rm -rf .git && git init to-clone &&
Signature · 1 line
gitgitgadget
PATCH v14 2/13 fsmonitor: fix khash memory leak in do_handle_client
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
The shown kh_str_t was freed with kh_release_str() at a point in the code only reachable in the non-trivial response path. When the client receives a trivial response, the code jumps to the cleanup label, skipping the kh_release_str() call entirely and leaking the hash table.
Fix this by initializing shown to NULL and moving the cleanup to the cleanup label using kh_destroy_str(), which is safe to call on NULL. This ensures the hash table is freed regardless of which code path is taken.
--- builtin/fsmonitor--daemon.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
builtin/fsmonitor--daemon.c +2 -3
@@ -671,7 +671,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,const struct fsmonitor_batch *batch;struct fsmonitor_batch *remainder =NULL ;intmax_t count =0 , duplicates =0 ; -kh_str_t *shown; +kh_str_t *shown =NULL ;int hash_ret;int do_trivial =0 ;int do_flush =0 ;
@@ -909,8 +909,6 @@ static int do_handle_client(struct fsmonitor_daemon_state *state, total_response_len += payload.len; } -kh_release_str (shown); -pthread_mutex_lock (&state->main_lock);if (token_data->client_ref_count >0 )
@@ -954,6 +952,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,trace2_data_intmax ("fsmonitor" , the_repository,"response/count/duplicates" , duplicates); cleanup: +kh_destroy_str (shown);strbuf_release (&response_token);strbuf_release (&requested_token_id);strbuf_release (&payload);
Signature · 1 line
gitgitgadget
PATCH v14 3/13 fsmonitor: fix hashmap memory leak in fsmonitor_run_daemon
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
The state.cookies hashmap is initialized during daemon startup but never freed during cleanup in the done: label of fsmonitor_run_daemon(). The cookie entries also have names allocated via strbuf_detach() that must be freed individually.
Iterate the hashmap to free each cookie name, then call hashmap_clear_and_free() to release the entries and table.
--- builtin/fsmonitor--daemon.c | 9 +++++++++ 1 file changed, 9 insertions(+)
builtin/fsmonitor--daemon.c +9 -0
@@ -1404,6 +1404,15 @@ static int fsmonitor_run_daemon(void) done:pthread_cond_destroy (&state.cookies_cond);pthread_mutex_destroy (&state.main_lock); + { +struct hashmap_iter iter; +struct fsmonitor_cookie_item *cookie; + +hashmap_for_each_entry (&state.cookies, &iter, cookie, entry) +free (cookie->name); +hashmap_clear_and_free (&state.cookies, + struct fsmonitor_cookie_item, entry); + }fsm_listen__dtor (&state);fsm_health__dtor (&state);
Signature · 1 line
gitgitgadget
PATCH v14 4/13 compat/win32: add pthread_cond_timedwait
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
Add a pthread_cond_timedwait() implementation to the Windows pthread compatibility layer using SleepConditionVariableCS() with a millisecond timeout computed from the absolute deadline.
--- compat/win32/pthread.c | 26 ++++++++++++++++++++++++++ compat/win32/pthread.h | 2 ++ 2 files changed, 28 insertions(+)
compat/win32/pthread.c +26 -0
@@ -66,3 +66,29 @@ int pthread_cond_wait(pthread_cond_t *cond, pthread_mutex_t *mutex)return err_win_to_posix (GetLastError ());return 0 ; } + +int pthread_cond_timedwait (pthread_cond_t *cond,pthread_mutex_t *mutex, +const struct timespec *abstime) +{ +struct timeval now; +long long now_ms, deadline_ms; +DWORD timeout_ms; + +gettimeofday (&now,NULL ); + now_ms = (long long )now.tv_sec *1000 + now.tv_usec /1000 ; + deadline_ms = (long long )abstime->tv_sec *1000 + + abstime->tv_nsec /1000000 ; + +if (deadline_ms <= now_ms) +return ETIMEDOUT ; +else + timeout_ms = (DWORD )(deadline_ms - now_ms); + +if (SleepConditionVariableCS (cond, mutex, timeout_ms) ==0 ) { +DWORD err =GetLastError (); +if (err ==ERROR_TIMEOUT ) +return ETIMEDOUT ; +return err_win_to_posix (err); + } +return 0 ; +}
compat/win32/pthread.h +2 -0
@@ -64,6 +64,8 @@ int win32_pthread_join(pthread_t *thread, void **value_ptr);pthread_t pthread_self (void );int pthread_cond_wait (pthread_cond_t *cond,pthread_mutex_t *mutex); +int pthread_cond_timedwait (pthread_cond_t *cond,pthread_mutex_t *mutex, +const struct timespec *abstime); static inline void NORETURN pthread_exit(void *ret) {
Signature · 1 line
gitgitgadget
PATCH v14 5/13 fsmonitor: use pthread_cond_timedwait for cookie wait
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
The cookie wait in with_lock__wait_for_cookie() uses an infinite pthread_cond_wait() loop. The existing comment notes the desire to switch to pthread_cond_timedwait(), but the routine was not available in git thread-utils.
On certain container or overlay filesystems, inotify watches may succeed but events are never delivered. In this case the daemon would hang indefinitely waiting for the cookie event, which in turn causes the client to hang.
Replace the infinite wait with a one-second timeout using pthread_cond_timedwait(). If the timeout fires, report an error and let the client proceed with a trivial (full-scan) response rather than blocking forever.
--- builtin/fsmonitor--daemon.c | 37 ++++++++++++++++++++++++------------- 1 file changed, 24 insertions(+), 13 deletions(-)
builtin/fsmonitor--daemon.c +24 -13
@@ -197,20 +197,31 @@ static enum fsmonitor_cookie_item_result with_lock__wait_for_cookie(unlink (cookie_pathname.buf);/* -* Technically, this is an infinite wait (well, unless another -* thread sends us an abort). I'd like to change this to -* use `pthread_cond_timedwait()` and return an error/timeout -* and let the caller do the trivial response thing, but we -* don't have that routine in our thread-utils. -* -* After extensive beta testing I'm not really worried about -* this. Also note that the above open() and unlink() calls -* will cause at least two FS events on that path, so the odds -* of getting stuck are pretty slim. +* Wait for the listener thread to observe the cookie file. +* Time out after a short interval so that the client +* does not hang forever if the filesystem does not deliver +* events (e.g., on certain container/overlay filesystems +* where inotify watches succeed but events never arrive). */ -while (cookie->result ==FCIR_INIT ) -pthread_cond_wait (&state->cookies_cond, - &state->main_lock); + { +struct timeval now; +struct timespec ts; +int err =0 ; + +gettimeofday (&now,NULL ); + ts.tv_sec = now.tv_sec +1 ; + ts.tv_nsec = now.tv_usec *1000 ; + +while (cookie->result ==FCIR_INIT && !err) + err =pthread_cond_timedwait (&state->cookies_cond, + &state->main_lock, + &ts); +if (err ==ETIMEDOUT && cookie->result ==FCIR_INIT ) { +trace_printf_key (&trace_fsmonitor, +"cookie_wait timed out" ); + cookie->result =FCIR_ERROR ; + } + } done:hashmap_remove (&state->cookies, &cookie->entry,NULL );
Signature · 1 line
gitgitgadget
PATCH v14 6/13 fsmonitor: rename fsm-ipc-darwin.c to fsm-ipc-unix.c
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
The fsmonitor IPC path logic in fsm-ipc-darwin.c is not Darwin-specific and will be reused by the upcoming Linux implementation. Rename it to fsm-ipc-unix.c to reflect that it is shared by all Unix platforms.
Introduce FSMONITOR_OS_SETTINGS (set to "unix" for non-Windows, "win32" for Windows) as a separate variable from FSMONITOR_DAEMON_BACKEND so that the build files can distinguish between platform-specific files (listen, health, path-utils) and shared Unix files (ipc, settings).
Move fsm-ipc to the FSMONITOR_OS_SETTINGS section in the Makefile, and switch fsm-path-utils to use FSMONITOR_DAEMON_BACKEND since path-utils is platform-specific (there will be separate darwin and linux versions).
---
Makefile | 6 ++---
.../{fsm-ipc-darwin.c => fsm-ipc-unix.c} | 0
config.mak.uname | 2 +-
contrib/buildsystems/CMakeLists.txt | 25 +++++++++----------
meson.build | 7 ++++--
5 files changed, 21 insertions(+), 19 deletions(-)
rename compat/fsmonitor/{fsm-ipc-darwin.c => fsm-ipc-unix.c} (100%)Makefile +3 -3
@@ -417,7 +417,7 @@ include shared.mak # If your platform has OS-specific ways to tell if a repo is incompatible with # fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS # to the "<name>" of the corresponding `compat/fsmonitor/fsm-settings-<name>.c` -# that implements the `fsm_os_settings__*()` routines. +# and `compat/fsmonitor/fsm-ipc-<name>.c` files. # # Define LINK_FUZZ_PROGRAMS if you want `make all` to also build the fuzz test # programs in oss-fuzz/.
@@ -2365,13 +2365,13 @@ ifdef FSMONITOR_DAEMON_BACKEND COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o - COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_BACKEND).o endif ifdef FSMONITOR_OS_SETTINGS COMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS + COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_OS_SETTINGS).o COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o - COMPAT_OBJS += compat/fsmonitor/fsm-path-utils-$(FSMONITOR_OS_SETTINGS).o + COMPAT_OBJS += compat/fsmonitor/fsm-path-utils-$(FSMONITOR_DAEMON_BACKEND).o endif ifdef WITH_BREAKING_CHANGES
compat/fsmonitor/fsm-ipc-unix.c renamed from compat/fsmonitor/fsm-ipc-darwin.c +0 -0
config.mak.uname +1 -1
@@ -165,7 +165,7 @@ ifeq ($(uname_S),Darwin) ifndef NO_PTHREADS ifndef NO_UNIX_SOCKETS FSMONITOR_DAEMON_BACKEND = darwin - FSMONITOR_OS_SETTINGS = darwin + FSMONITOR_OS_SETTINGS = unix endif endif
contrib/buildsystems/CMakeLists.txt +12 -13
@@ -291,23 +291,22 @@ endif() if(SUPPORTS_SIMPLE_IPC) if(CMAKE_SYSTEM_NAME STREQUAL "Windows") - add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-win32.c) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-win32.c) - - add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c) + set(FSMONITOR_DAEMON_BACKEND "win32") + set(FSMONITOR_OS_SETTINGS "win32") elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") + set(FSMONITOR_DAEMON_BACKEND "darwin") + set(FSMONITOR_OS_SETTINGS "unix") + endif() + + if(FSMONITOR_DAEMON_BACKEND) add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-darwin.c) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-darwin.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-${FSMONITOR_DAEMON_BACKEND}.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-${FSMONITOR_DAEMON_BACKEND}.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-${FSMONITOR_OS_SETTINGS}.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-${FSMONITOR_DAEMON_BACKEND}.c) add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-${FSMONITOR_DAEMON_BACKEND}.c) endif() endif()
meson.build +5 -2
@@ -1320,10 +1320,13 @@ else endif fsmonitor_backend = '' +fsmonitor_os = '' if host_machine.system() == 'windows' fsmonitor_backend = 'win32' + fsmonitor_os = 'win32' elif host_machine.system() == 'darwin' fsmonitor_backend = 'darwin' + fsmonitor_os = 'unix' libgit_dependencies += dependency('CoreServices') endif if fsmonitor_backend != ''
@@ -1332,14 +1335,14 @@ if fsmonitor_backend != '' libgit_sources += [ 'compat/fsmonitor/fsm-health-' + fsmonitor_backend + '.c', - 'compat/fsmonitor/fsm-ipc-' + fsmonitor_backend + '.c', + 'compat/fsmonitor/fsm-ipc-' + fsmonitor_os + '.c', 'compat/fsmonitor/fsm-listen-' + fsmonitor_backend + '.c', 'compat/fsmonitor/fsm-path-utils-' + fsmonitor_backend + '.c', 'compat/fsmonitor/fsm-settings-' + fsmonitor_backend + '.c', ] endif build_options_config.set_quoted('FSMONITOR_DAEMON_BACKEND', fsmonitor_backend) -build_options_config.set_quoted('FSMONITOR_OS_SETTINGS', fsmonitor_backend) +build_options_config.set_quoted('FSMONITOR_OS_SETTINGS', fsmonitor_os) if not get_option('b_sanitize').contains('address') and get_option('regex').allowed() and compiler.has_header('regex.h') and compiler.get_define('REG_STARTEND', prefix: '#include <regex.h>') != '' build_options_config.set('NO_REGEX', '')
Signature · 1 line
gitgitgadget
PATCH v14 7/13 fsmonitor: rename fsm-settings-darwin.c to fsm-settings-unix.c
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
The fsmonitor settings logic in fsm-settings-darwin.c is not Darwin-specific and will be reused by the upcoming Linux implementation. Rename it to fsm-settings-unix.c to reflect that it is shared by all Unix platforms.
Update the build files (meson.build and CMakeLists.txt) to use FSMONITOR_OS_SETTINGS for fsm-settings, matching the approach already used for fsm-ipc.
---
compat/fsmonitor/{fsm-settings-darwin.c => fsm-settings-unix.c} | 0
contrib/buildsystems/CMakeLists.txt | 2 +-
meson.build | 2 +-
3 files changed, 2 insertions(+), 2 deletions(-)
rename compat/fsmonitor/{fsm-settings-darwin.c => fsm-settings-unix.c} (100%)compat/fsmonitor/fsm-settings-unix.c renamed from compat/fsmonitor/fsm-settings-darwin.c +0 -0
contrib/buildsystems/CMakeLists.txt +1 -1
@@ -306,7 +306,7 @@ if(SUPPORTS_SIMPLE_IPC) list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-${FSMONITOR_DAEMON_BACKEND}.c) add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-${FSMONITOR_DAEMON_BACKEND}.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-${FSMONITOR_OS_SETTINGS}.c) endif() endif()
meson.build +1 -1
@@ -1338,7 +1338,7 @@ if fsmonitor_backend != '' 'compat/fsmonitor/fsm-ipc-' + fsmonitor_os + '.c', 'compat/fsmonitor/fsm-listen-' + fsmonitor_backend + '.c', 'compat/fsmonitor/fsm-path-utils-' + fsmonitor_backend + '.c', - 'compat/fsmonitor/fsm-settings-' + fsmonitor_backend + '.c', + 'compat/fsmonitor/fsm-settings-' + fsmonitor_os + '.c', ] endif build_options_config.set_quoted('FSMONITOR_DAEMON_BACKEND', fsmonitor_backend)
Signature · 1 line
gitgitgadget
PATCH v14 8/13 fsmonitor: implement filesystem change listener for Linux
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
Implement the built-in fsmonitor daemon for Linux using the inotify API, bringing it to feature parity with the existing Windows and macOS implementations.
The implementation uses inotify rather than fanotify because fanotify requires either CAP_SYS_ADMIN or CAP_PERFMON capabilities, making it unsuitable for an unprivileged user-space daemon. While inotify has the limitation of requiring a separate watch on every directory (unlike macOS's FSEvents, which can monitor an entire directory tree with a single watch), it operates without elevated privileges and provides the per-file event granularity needed for fsmonitor.
The listener uses inotify_init1(O_NONBLOCK) with a poll loop that checks for events with a 50-millisecond timeout, keeping the inotify queue well-drained to minimize the risk of overflows. Bidirectional hashmaps map between watch descriptors and directory paths for efficient event resolution. Directory renames are tracked using inotify's cookie mechanism to correlate IN_MOVED_FROM and IN_MOVED_TO event pairs; a periodic check detects stale renames where the matching IN_MOVED_TO never arrived, forcing a resync.
New directory creation triggers recursive watch registration to ensure all subdirectories are monitored. The IN_MASK_CREATE flag is used where available to prevent modifying existing watches, with a fallback for older kernels. When IN_MASK_CREATE is available and inotify_add_watch returns EEXIST, it means another thread or recursive scan has already registered the watch, so it is safe to ignore.
Remote filesystem detection uses statfs() to identify network-mounted filesystems (NFS, CIFS, SMB, FUSE, etc.) via their magic numbers. Mount point information is read from /proc/mounts and matched against the statfs f_fsid to get accurate, human-readable filesystem type names for logging. When the .git directory is on a remote filesystem, the IPC socket falls back to $HOME or a user-configured directory via the fsmonitor.socketDir setting.
--- Documentation/config/fsmonitor--daemon.adoc | 4 +- Documentation/git-fsmonitor--daemon.adoc | 28 +- compat/fsmonitor/fsm-health-linux.c | 33 + compat/fsmonitor/fsm-listen-linux.c | 746 ++++++++++++++++++++ compat/fsmonitor/fsm-path-utils-linux.c | 217 ++++++ config.mak.uname | 10 + contrib/buildsystems/CMakeLists.txt | 8 +- meson.build | 4 + 8 files changed, 1042 insertions(+), 8 deletions(-) create mode 100644 compat/fsmonitor/fsm-health-linux.c create mode 100644 compat/fsmonitor/fsm-listen-linux.c create mode 100644 compat/fsmonitor/fsm-path-utils-linux.c
Documentation/config/fsmonitor--daemon.adoc +2 -2
@@ -4,8 +4,8 @@ fsmonitor.allowRemote:: behavior. Only respected when `core.fsmonitor` is set to `true`. fsmonitor.socketDir:: - This Mac OS-specific option, if set, specifies the directory in + This Mac OS and Linux-specific option, if set, specifies the directory in which to create the Unix domain socket used for communication between the fsmonitor daemon and various Git commands. The directory must - reside on a native Mac OS filesystem. Only respected when `core.fsmonitor` + reside on a native filesystem. Only respected when `core.fsmonitor` is set to `true`.
Documentation/git-fsmonitor--daemon.adoc +24 -4
@@ -76,9 +76,9 @@ repositories; this may be overridden by setting `fsmonitor.allowRemote` to correctly with all network-mounted repositories, so such use is considered experimental. -On Mac OS, the inter-process communication (IPC) between various Git +On Mac OS and Linux, the inter-process communication (IPC) between various Git commands and the fsmonitor daemon is done via a Unix domain socket (UDS) -- a -special type of file -- which is supported by native Mac OS filesystems, +special type of file -- which is supported by native Mac OS and Linux filesystems, but not on network-mounted filesystems, NTFS, or FAT32. Other filesystems may or may not have the needed support; the fsmonitor daemon is not guaranteed to work with these filesystems and such use is considered experimental.
@@ -87,13 +87,33 @@ By default, the socket is created in the `.git` directory. However, if the `.git` directory is on a network-mounted filesystem, it will instead be created at `$HOME/.git-fsmonitor-*` unless `$HOME` itself is on a network-mounted filesystem, in which case you must set the configuration -variable `fsmonitor.socketDir` to the path of a directory on a Mac OS native +variable `fsmonitor.socketDir` to the path of a directory on a native filesystem in which to create the socket file. If none of the above directories (`.git`, `$HOME`, or `fsmonitor.socketDir`) -is on a native Mac OS file filesystem the fsmonitor daemon will report an +is on a native filesystem the fsmonitor daemon will report an error that will cause the daemon and the currently running command to exit. +LINUX CAVEATS +~~~~~~~~~~~~~ + +On Linux, the fsmonitor daemon uses inotify to monitor filesystem events. +The inotify system has per-user limits on the number of watches that can +be created. The default limit is typically 8192 watches per user. + +For large repositories with many directories, you may need to increase +this limit. Check the current limit with: + + cat /proc/sys/fs/inotify/max_user_watches + +To temporarily increase the limit: + + sudo sysctl fs.inotify.max_user_watches=65536 + +To make the change permanent, add to `/etc/sysctl.conf`: + + fs.inotify.max_user_watches=65536 + CONFIGURATION -------------
compat/fsmonitor/fsm-health-linux.c new file +33 -0
@@ -0,0 +1,33 @@ +#include "git-compat-util.h" +#include "config.h" +#include "fsmonitor-ll.h" +#include "fsm-health.h" +#include "fsmonitor--daemon.h" + +/* +* The Linux fsmonitor implementation uses inotify which has its own +* mechanisms for detecting filesystem unmount and other events that +* would require the daemon to shutdown. Therefore, we don't need +* a separate health thread like Windows does. +* +* These stub functions satisfy the interface requirements. +*/ + +int fsm_health__ctor(struct fsmonitor_daemon_state *state UNUSED) +{ +return 0 ; +} + +void fsm_health__dtor(struct fsmonitor_daemon_state *state UNUSED) +{ +return ; +} + +void fsm_health__loop(struct fsmonitor_daemon_state *state UNUSED) +{ +return ; +} + +void fsm_health__stop_async(struct fsmonitor_daemon_state *state UNUSED) +{ +}
compat/fsmonitor/fsm-listen-linux.c new file +746 -0
746 lines not on this page · raw message
compat/fsmonitor/fsm-path-utils-linux.c new file +217 -0
217 lines not on this page · raw message
config.mak.uname +10 -0
@@ -68,6 +68,16 @@ ifeq ($(uname_S),Linux) BASIC_CFLAGS += -std=c99 endif LINK_FUZZ_PROGRAMS = YesPlease + + # The builtin FSMonitor on Linux builds upon Simple-IPC. Both require + # Unix domain sockets and PThreads. + ifndef NO_PTHREADS + ifndef NO_UNIX_SOCKETS + FSMONITOR_DAEMON_BACKEND = linux + FSMONITOR_OS_SETTINGS = unix + BASIC_CFLAGS += -DHAVE_LINUX_MAGIC_H + endif + endif endif ifeq ($(uname_S),GNU/kFreeBSD) HAVE_ALLOCA_H = YesPlease
contrib/buildsystems/CMakeLists.txt +6 -2
@@ -296,6 +296,10 @@ if(SUPPORTS_SIMPLE_IPC) elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") set(FSMONITOR_DAEMON_BACKEND "darwin") set(FSMONITOR_OS_SETTINGS "unix") + elseif(CMAKE_SYSTEM_NAME STREQUAL "Linux") + set(FSMONITOR_DAEMON_BACKEND "linux") + set(FSMONITOR_OS_SETTINGS "unix") + add_compile_definitions(HAVE_LINUX_MAGIC_H) endif() if(FSMONITOR_DAEMON_BACKEND)
@@ -1149,8 +1153,8 @@ endif() file(STRINGS ${CMAKE_SOURCE_DIR}/GIT-BUILD-OPTIONS.in git_build_options NEWLINE_CONSUME) string(REPLACE "@BROKEN_PATH_FIX@" "" git_build_options "${git_build_options}") string(REPLACE "@DIFF@" "'${DIFF}'" git_build_options "${git_build_options}") -string(REPLACE "@FSMONITOR_DAEMON_BACKEND@" "win32" git_build_options "${git_build_options}") -string(REPLACE "@FSMONITOR_OS_SETTINGS@" "win32" git_build_options "${git_build_options}") +string(REPLACE "@FSMONITOR_DAEMON_BACKEND@" "${FSMONITOR_DAEMON_BACKEND}" git_build_options "${git_build_options}") +string(REPLACE "@FSMONITOR_OS_SETTINGS@" "${FSMONITOR_OS_SETTINGS}" git_build_options "${git_build_options}") string(REPLACE "@GITWEBDIR@" "'${GITWEBDIR}'" git_build_options "${git_build_options}") string(REPLACE "@GIT_INTEROP_MAKE_OPTS@" "" git_build_options "${git_build_options}") string(REPLACE "@GIT_PERF_LARGE_REPO@" "" git_build_options "${git_build_options}")
meson.build +4 -0
@@ -1324,6 +1324,10 @@ fsmonitor_os = '' if host_machine.system() == 'windows' fsmonitor_backend = 'win32' fsmonitor_os = 'win32' +elif host_machine.system() == 'linux' and threads.found() and compiler.has_header('linux/magic.h') + fsmonitor_backend = 'linux' + fsmonitor_os = 'unix' + libgit_c_args += '-DHAVE_LINUX_MAGIC_H' elif host_machine.system() == 'darwin' fsmonitor_backend = 'darwin' fsmonitor_os = 'unix'
Signature · 1 line
gitgitgadget
PATCH v14 9/13 run-command: add close_fd_above_stderr option
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
Add a close_fd_above_stderr flag to struct child_process. When set, the child closes file descriptors 3 and above between fork and exec (skipping the child-notifier pipe), capped at sysconf(_SC_OPEN_MAX) or 4096, whichever is smaller. This prevents the child from inheriting pipe endpoints or other descriptors from the parent environment (e.g., the test harness).
--- run-command.c | 12 ++++++++++++ run-command.h | 9 +++++++++ 2 files changed, 21 insertions(+)
run-command.c +12 -0
@@ -546,6 +546,7 @@ static void atfork_parent(struct atfork_state *as) "restoring signal mask"); #endif } +#endif /* GIT_WINDOWS_NATIVE */ static inline void set_cloexec (int fd)
@@ -832,6 +833,17 @@ fail_pipe:child_close (cmd->out); } +if (cmd->close_fd_above_stderr) { +long max_fd =sysconf (_SC_OPEN_MAX); +int fd; +if (max_fd <0 || max_fd >4096 ) + max_fd =4096 ; +for (fd =3 ; fd < max_fd; fd++) { +if (fd != child_notifier) +close (fd); + } + } +if (cmd->dir &&chdir (cmd->dir))child_die (CHILD_ERR_CHDIR );
run-command.h +9 -0
@@ -141,6 +141,15 @@ struct child_process { unsigned stdout_to_stderr:1; unsigned clean_on_exit:1; unsigned wait_after_clean:1; + +/** +* Close file descriptors 3 and above in the child after forking +* but before exec. This prevents the child from inheriting +* pipe endpoints or other descriptors from the parent +* environment (e.g., the test harness). +*/ + unsigned close_fd_above_stderr:1; +void (*clean_on_exit_handler)(struct child_process *process); };
Signature · 1 line
gitgitgadget
PATCH v14 10/13 fsmonitor: close inherited file descriptors and detach in daemon
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
When the fsmonitor daemon is spawned as a background process, it may inherit file descriptors from its parent that it does not need. In particular, when the test harness or a CI system captures output through pipes, the daemon can inherit duplicated pipe endpoints. If the daemon holds these open, the parent process never sees EOF and may appear to hang.
Set close_fd_above_stderr on the child process at both daemon startup
paths: the explicit "fsmonitor--daemon start" command and the implicit spawn triggered by fsmonitor-ipc when a client finds no running daemon. Also suppress stdout and stderr on the implicit spawn path to prevent the background daemon from writing to the client's terminal.
Additionally, call setsid() when the daemon starts with --detach to create a new session and process group. This prevents the daemon from being part of the spawning shell's process group, which could cause the shell's "wait" to block until the daemon exits.
--- builtin/fsmonitor--daemon.c | 16 ++++++++++++++-- fsmonitor-ipc.c | 3 +++ 2 files changed, 17 insertions(+), 2 deletions(-)
builtin/fsmonitor--daemon.c +14 -2
@@ -1439,7 +1439,7 @@ done:return err; } -static int try_to_run_foreground_daemon(int detach_console MAYBE_UNUSED) +static int try_to_run_foreground_daemon (int detach_console) {/* * Technically, we don't need to probe for an existing daemon
@@ -1459,10 +1459,21 @@ static int try_to_run_foreground_daemon(int detach_console MAYBE_UNUSED)fflush (stderr); } +if (detach_console) {#ifdef GIT_WINDOWS_NATIVE -if (detach_console)FreeConsole (); +#else +/* +* Create a new session so that the daemon is detached +* from the parent's process group. This prevents +* shells with job control (e.g. bash with "set -m") +* from waiting on the daemon when they wait for a +* foreground command that implicitly spawned it. +*/ +if (setsid () ==-1 ) +warning_errno (_ ("setsid failed" ));#endif + }return !!fsmonitor_run_daemon (); }
@@ -1525,6 +1536,7 @@ static int try_to_start_background_daemon(void) cp.no_stdin =1 ; cp.no_stdout =1 ; cp.no_stderr =1 ; + cp.close_fd_above_stderr =1 ; sbgr =start_bg_command (&cp, bg_wait_cb,NULL , fsmonitor__start_timeout_sec);
fsmonitor-ipc.c +3 -0
@@ -61,6 +61,9 @@ static int spawn_daemon(void) cmd.git_cmd =1 ; cmd.no_stdin =1 ; + cmd.no_stdout =1 ; + cmd.no_stderr =1 ; + cmd.close_fd_above_stderr =1 ; cmd.trace2_child_class ="fsmonitor" ;strvec_pushl (&cmd.args,"fsmonitor--daemon" ,"start" ,NULL );
Signature · 1 line
gitgitgadget
PATCH v14 11/13 fsmonitor: add timeout to daemon stop command
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
The "fsmonitor--daemon stop" command polls in a loop waiting for the daemon to exit after sending a "quit" command over IPC. If the daemon fails to shut down (e.g. it is stuck or wedged), this loop spins forever.
Add a 30-second timeout so the stop command returns an error instead of blocking indefinitely.
--- builtin/fsmonitor--daemon.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
builtin/fsmonitor--daemon.c +11 -1
@@ -86,6 +86,8 @@ static int do_as_client__send_stop(void) {struct strbuf answer =STRBUF_INIT ;int ret; +int max_wait_ms =30000 ; +int elapsed_ms =0 ; ret =fsmonitor_ipc__send_command ("quit" , &answer);
@@ -96,8 +98,16 @@ static int do_as_client__send_stop(void)return ret;trace2_region_enter ("fsm_client" ,"polling-for-daemon-exit" ,NULL ); -while (fsmonitor_ipc__get_state () ==IPC_STATE__LISTENING ) +while (fsmonitor_ipc__get_state () ==IPC_STATE__LISTENING ) { +if (elapsed_ms >= max_wait_ms) { +trace2_region_leave ("fsm_client" , +"polling-for-daemon-exit" ,NULL ); +return error (_ ("daemon did not stop within %d seconds" ), + max_wait_ms /1000 ); + }sleep_millisec (50 ); + elapsed_ms +=50 ; + }trace2_region_leave ("fsm_client" ,"polling-for-daemon-exit" ,NULL );return 0 ;
Signature · 1 line
gitgitgadget
PATCH v14 12/13 fsmonitor: add tests for Linux
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
Add a smoke test that verifies the filesystem actually delivers inotify events to the daemon. On some configurations (e.g., overlayfs with older kernels), inotify watches succeed but events are never delivered. The daemon cookie wait will time out, but every subsequent test would fail. Skip the entire test file early when this is detected.
Add a test that exercises rapid nested directory creation to verify the daemon correctly handles the EEXIST race between recursive scan and queued inotify events. When IN_MASK_CREATE is available and a directory watch is added during recursive registration, the kernel may also deliver a queued IN_CREATE event for the same directory. The second inotify_add_watch() returns EEXIST, which must be treated as harmless. An earlier version of the listener crashed in this scenario.
Reduce --start-timeout from the default 60 seconds to 10 seconds so that tests fail promptly when the daemon cannot start.
Harden the test helpers to work in environments without procps (e.g., Fedora CI): fall back to reading /proc/$pid/stat for the process group ID when ps is unavailable, guard stop_git() against an empty pgid, and redirect stderr from kill to /dev/null to avoid noise when processes have already exited.
Use set -m to enable job control in the submodule-pull test so that the background git pull gets its own process group, preventing the shell wait from blocking on the daemon. setsid() in the previous commit detaches the daemon itself, but the intermediate git pull process still needs its own process group for the test shell to manage it correctly.
--- t/t7527-builtin-fsmonitor.sh | 89 +++++++++++++++++++++++++++++++++--- 1 file changed, 82 insertions(+), 7 deletions(-)
t/t7527-builtin-fsmonitor.sh +82 -7
@@ -10,9 +10,58 @@ then test_done fi +# Verify that the filesystem delivers events to the daemon. +# On some configurations (e.g., overlayfs with older kernels), +# inotify watches succeed but events are never delivered. The +# cookie wait will time out and the daemon logs a trace message. +# +# Use "timeout" (if available) to guard each step against hangs. +maybe_timeout () { + if type timeout >/dev/null 2>&1 + then + timeout "$@" + else + shift + "$@" + fi +} +verify_fsmonitor_works () { + git init test_fsmonitor_smoke || return 1 + + GIT_TRACE_FSMONITOR="$PWD/smoke.trace" && + export GIT_TRACE_FSMONITOR && + maybe_timeout 30 \ + git -C test_fsmonitor_smoke fsmonitor--daemon start \ + --start-timeout=10 + ret=$? + unset GIT_TRACE_FSMONITOR + if test $ret -ne 0 + then + rm -rf test_fsmonitor_smoke smoke.trace + return 1 + fi + + maybe_timeout 10 \ + test-tool -C test_fsmonitor_smoke fsmonitor-client query \ + --token 0 >/dev/null 2>&1 + maybe_timeout 5 \ + git -C test_fsmonitor_smoke fsmonitor--daemon stop 2>/dev/null + ! grep -q "cookie_wait timed out" "$PWD/smoke.trace" 2>/dev/null + ret=$? + rm -rf test_fsmonitor_smoke smoke.trace + return $ret +} + +if ! verify_fsmonitor_works +then + skip_all="filesystem does not deliver fsmonitor events (container/overlayfs?)" + test_done +fi + stop_daemon_delete_repo () { r=$1 && - test_might_fail git -C $r fsmonitor--daemon stop && + test_might_fail maybe_timeout 30 \ + git -C $r fsmonitor--daemon stop 2>/dev/null rm -rf $1 }
@@ -67,7 +116,7 @@ start_daemon () { export GIT_TEST_FSMONITOR_TOKEN fi && - git $r fsmonitor--daemon start && + git $r fsmonitor--daemon start --start-timeout=10 && git $r fsmonitor--daemon status ) }
@@ -520,6 +569,28 @@ test_expect_success 'directory changes to a file' ' grep "^event: dir1$" .git/trace ' +test_expect_success 'rapid nested directory creation' ' + test_when_finished "git fsmonitor--daemon stop; rm -rf rapid" && + + start_daemon --tf "$PWD/.git/trace" && + + # Rapidly create nested directories to exercise race conditions + # where directory watches may be added concurrently during + # event processing and recursive scanning. + for i in $(test_seq 1 20) + do + mkdir -p "rapid/nested/dir$i/subdir/deep" || return 1 + done && + + # Give the daemon time to process all events + sleep 1 && + + test-tool fsmonitor-client query --token 0 && + + # Verify daemon is still running (did not crash) + git fsmonitor--daemon status +' + # The next few test cases exercise the token-resync code. When filesystem # drops events (because of filesystem velocity or because the daemon isn't # polling fast enough), we need to discard the cached data (relative to the
@@ -910,7 +981,10 @@ test_expect_success "submodule absorbgitdirs implicitly starts daemon" ' start_git_in_background () { git "$@" & git_pid=$! - git_pgid=$(ps -o pgid= -p $git_pid) + git_pgid=$(ps -o pgid= -p $git_pid 2>/dev/null || + awk '{print $5}' /proc/$git_pid/stat 2>/dev/null) && + git_pgid="${git_pgid## }" && + git_pgid="${git_pgid%% }" nr_tries_left=10 while true do
@@ -921,15 +995,16 @@ start_git_in_background () { fi sleep 1 nr_tries_left=$(($nr_tries_left - 1)) - done >/dev/null 2>&1 & + done >/dev/null 2>&1 3>&- 4>&- 5>&- 6>&- 7>&- & watchdog_pid=$! wait $git_pid } stop_git () { - while kill -0 -- -$git_pgid + test -n "$git_pgid" || return 0 + while kill -0 -- -$git_pgid 2>/dev/null do - kill -- -$git_pgid + kill -- -$git_pgid 2>/dev/null sleep 1 done }
@@ -944,7 +1019,7 @@ stop_watchdog () { test_expect_success !MINGW "submodule implicitly starts daemon by pull" ' test_atexit "stop_watchdog" && - test_when_finished "stop_git; rm -rf cloned super sub" && + test_when_finished "set +m; stop_git; rm -rf cloned super sub" && create_super super && create_sub sub &&
Signature · 1 line
gitgitgadget
Re: [PATCH v14 12/13] fsmonitor: add tests for Linux
Quoted 90 lines · Paul Tarjan via GitGitGadget
From: Paul Tarjan <[email protected]>
Add a smoke test that verifies the filesystem actually delivers inotify events to the daemon. On some configurations (e.g., overlayfs with older kernels), inotify watches succeed but events are never delivered. The daemon cookie wait will time out, but every subsequent test would fail. Skip the entire test file early when this is detected.
Add a test that exercises rapid nested directory creation to verify the daemon correctly handles the EEXIST race between recursive scan and queued inotify events. When
IN_MASK_CREATEis available and a directory watch is added during recursive registration, the kernel may also deliver a queuedIN_CREATEevent for the same directory. The secondinotify_add_watch()returns EEXIST, which must be treated as harmless. An earlier version of the listener crashed in this scenario.Reduce --start-timeout from the default 60 seconds to 10 seconds so that tests fail promptly when the daemon cannot start.
Harden the test helpers to work in environments without procps (e.g., Fedora CI): fall back to reading
/proc/$pid/statfor the process group ID when ps is unavailable, guardstop_git()against an empty pgid, and redirect stderr from kill to/dev/nullto avoid noise when processes have already exited.Use set -m to enable job control in the submodule-pull test so that the background git pull gets its own process group, preventing the shell wait from blocking on the daemon.
setsid()in the previous commit detaches the daemon itself, but the intermediate git pull process still needs its own process group for the test shell to manage it correctly.Signed-off-by Paul Tarjan <[email protected]>--- t/t7527-builtin-fsmonitor.sh | 89 +++++++++++++++++++++++++++++++++--- 1 file changed, 82 insertions(+), 7 deletions(-)t/t7527-builtin-fsmonitor.sh +43
-0@@ -10,9 +10,58 @@ then test_done fi +# Verify that the filesystem delivers events to the daemon. +# On some configurations (e.g., overlayfs with older kernels), +# inotify watches succeed but events are never delivered. The +# cookie wait will time out and the daemon logs a trace message. +# +# Use "timeout" (if available) to guard each step against hangs. +maybe_timeout () { + if type timeout >/dev/null 2>&1 + then + timeout "$@" + else + shift + "$@" + fi +} +verify_fsmonitor_works () { + git init test_fsmonitor_smoke || return 1 + + GIT_TRACE_FSMONITOR="$PWD/smoke.trace" && + export GIT_TRACE_FSMONITOR && + maybe_timeout 30 \ + git -C test_fsmonitor_smoke fsmonitor--daemon start \ + --start-timeout=10 + ret=$? + unset GIT_TRACE_FSMONITOR + if test $ret -ne 0 + then + rm -rf test_fsmonitor_smoke smoke.trace + return 1 + fi + + maybe_timeout 10 \ + test-tool -C test_fsmonitor_smoke fsmonitor-client query \ + --token 0 >/dev/null 2>&1 + maybe_timeout 5 \ + git -C test_fsmonitor_smoke fsmonitor--daemon stop 2>/dev/null + ! grep -q "cookie_wait timed out" "$PWD/smoke.trace" 2>/dev/null + ret=$? + rm -rf test_fsmonitor_smoke smoke.trace + return $ret +} + +if ! verify_fsmonitor_works
t/t7527-builtin-fsmonitor.sh then · line 64
Quoted 10 lines
t/t7527-builtin-fsmonitor.sh +7
-1+then + skip_all="filesystem does not deliver fsmonitor events (container/overlayfs?)" + test_done +fi + stop_daemon_delete_repo () { r=$1 && - test_might_fail git -C $r fsmonitor--daemon stop && + test_might_fail maybe_timeout 30 \ + git -C $r fsmonitor--daemon stop 2>/dev/null
"test_might_fail" only allows a few select commands and functions, and the "maybe_timeout" helper function introduced in this patch is, of course, not one of them, so it returns with error and without running the given command. Consequently, after this test script is finished I have still two fsmonitor daemon processes running in the background.
Alas, this went unnoticed, because this patch broke the &&-chain and redirected "test_might_fail"'s
test_must_fail: only 'git' is allowed: maybe_timeout 30 git -C test_explicit fsmonitor--daemon stop
error messages to /dev/null. With the &&-chain restored over 40 test cases fail because of this.
Quoted 85 lines
t/t7527-builtin-fsmonitor.sh +32
-6rm -rf $1 }@@ -67,7 +116,7 @@ start_daemon () { export GIT_TEST_FSMONITOR_TOKEN fi && - git $r fsmonitor--daemon start && + git $r fsmonitor--daemon start --start-timeout=10 && git $r fsmonitor--daemon status ) }@@ -520,6 +569,28 @@ test_expect_success 'directory changes to a file' ' grep "^event: dir1$" .git/trace ' +test_expect_success 'rapid nested directory creation' ' + test_when_finished "git fsmonitor--daemon stop; rm -rf rapid" && + + start_daemon --tf "$PWD/.git/trace" && + + # Rapidly create nested directories to exercise race conditions + # where directory watches may be added concurrently during + # event processing and recursive scanning. + for i in $(test_seq 1 20) + do + mkdir -p "rapid/nested/dir$i/subdir/deep" || return 1 + done && + + # Give the daemon time to process all events + sleep 1 && + + test-tool fsmonitor-client query --token 0 && + + # Verify daemon is still running (did not crash) + git fsmonitor--daemon status +' + # The next few test cases exercise the token-resync code. When filesystem # drops events (because of filesystem velocity or because the daemon isn't # polling fast enough), we need to discard the cached data (relative to the@@ -910,7 +981,10 @@ test_expect_success "submodule absorbgitdirs implicitly starts daemon" ' start_git_in_background () { git "$@" & git_pid=$! - git_pgid=$(ps -o pgid= -p $git_pid) + git_pgid=$(ps -o pgid= -p $git_pid 2>/dev/null || + awk '{print $5}' /proc/$git_pid/stat 2>/dev/null) && + git_pgid="${git_pgid## }" && + git_pgid="${git_pgid%% }" nr_tries_left=10 while true do@@ -921,15 +995,16 @@ start_git_in_background () { fi sleep 1 nr_tries_left=$(($nr_tries_left - 1)) - done >/dev/null 2>&1 & + done >/dev/null 2>&1 3>&- 4>&- 5>&- 6>&- 7>&- & watchdog_pid=$! wait $git_pid } stop_git () { - while kill -0 -- -$git_pgid + test -n "$git_pgid" || return 0 + while kill -0 -- -$git_pgid 2>/dev/null do - kill -- -$git_pgid + kill -- -$git_pgid 2>/dev/null sleep 1 done }@@ -944,7 +1019,7 @@ stop_watchdog () { test_expect_success !MINGW "submodule implicitly starts daemon by pull" ' test_atexit "stop_watchdog" && - test_when_finished "stop_git; rm -rf cloned super sub" && + test_when_finished "set +m; stop_git; rm -rf cloned super sub" && create_super super && create_sub sub &&Signature · 1 line
gitgitgadget
Re: [PATCH v14 12/13] fsmonitor: add tests for Linux
Quoted 12 lines · SZEDER Gábor
Review
Quoted 5 lines
stop_daemon_delete_repo() {(quoted diff) +2
-1r=$1 && - test_might_fail git -C $r fsmonitor--daemon stop && + test_might_fail maybe_timeout 30 \ + git -C $r fsmonitor--daemon stop 2>/dev/null"
test_might_fail" only allows a few select commands and functions, and the "maybe_timeout" helper function introduced in this patch is, of course, not one of them, so it returns with error and without running the given command. Consequently, after this test script is finished I have still two fsmonitor daemon processes running in the background.
Quoted 10 lines
Alas, this went unnoticed, because this patch broke the &&-chain and redirected "test_might_fail"'s
test_must_fail: only 'git' is allowed:maybe_timeout30 git -Ctest_explicitfsmonitor--daemon stoperror messages to
/dev/null. With the &&-chain restored over 40 test cases fail because of this.Quoted 2 lines
rm -rf $1
}
Yikes. Would it help to apply a patch like this, then?
t/t7527-builtin-fsmonitor.sh | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
t/t7527-builtin-fsmonitor.sh +1 -2
@@ -60,8 +60,7 @@ fi stop_daemon_delete_repo () { r=$1 && - test_might_fail maybe_timeout 30 \ - git -C $r fsmonitor--daemon stop 2>/dev/null + maybe_timeout 30 git -C $r fsmonitor--daemon stop || : && rm -rf $1 }
Re: [PATCH v14 12/13] fsmonitor: add tests for Linux
t/t7527-builtin-fsmonitor.sh fi · line 65
Quoted 13 lines · Junio C Hamano
t/t7527-builtin-fsmonitor.sh +1
-2@@ -60,8 +60,7 @@ fi stop_daemon_delete_repo () { r=$1 && - test_might_fail maybe_timeout 30 \ - git -C $r fsmonitor--daemon stop 2>/dev/null + maybe_timeout 30 git -C $r fsmonitor--daemon stop || : && rm -rf $1 }
Do we need to put it in curly braces to avoid interfering with the && chain? Otherwise a failure of anything before the maybe_timeout will hit the "||".
I guess in this case it is just "r=$1", which will never fail, but it feels like we should model best practice.
-Peff
PATCH v14 13/13 fsmonitor: convert shown khash to strset in do_handle_client
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
Replace the khash-based string set used for deduplicating pathnames in do_handle_client() with a strset, which provides a cleaner interface for the same purpose.
Since the paths are interned strings from the batch data, use strdup_strings=0 to avoid unnecessary copies.
--- builtin/fsmonitor--daemon.c | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-)
builtin/fsmonitor--daemon.c +6 -11
@@ -16,7 +16,7 @@#include "fsmonitor--daemon.h" #include "simple-ipc.h" -#include "khash.h" +#include "strmap.h" #include "run-command.h" #include "trace.h" #include "trace2.h"
@@ -674,8 +674,6 @@ static int fsmonitor_parse_client_token(const char *buf_token,return 0 ; } -KHASH_INIT(str, const char *, int, 0, kh_str_hash_func, kh_str_hash_equal) - static int do_handle_client(struct fsmonitor_daemon_state *state, const char *command, ipc_server_reply_cb *reply,
@@ -692,8 +690,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,const struct fsmonitor_batch *batch;struct fsmonitor_batch *remainder =NULL ;intmax_t count =0 , duplicates =0 ; -kh_str_t *shown =NULL ; -int hash_ret; +struct strset shown =STRSET_INIT ;int do_trivial =0 ;int do_flush =0 ;int do_cookie =0 ;
@@ -882,14 +879,14 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,* so walk the batch list backwards from the current head back * to the batch (sequence number) they named. * -* We use khash to de-dup the list of pathnames. +* We use a strset to de-dup the list of pathnames. * * NEEDSWORK: each batch contains a list of interned strings, * so we only need to do pointer comparisons here to build the * hash table. Currently, we're still comparing the string * values. */ - shown =kh_init_str (); +strset_init_with_options (&shown,NULL ,0 );for (batch = batch_head; batch && batch->batch_seq_nr > requested_oldest_seq_nr; batch = batch->next) {
@@ -899,11 +896,9 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,const char *s = batch->interned_paths[k];size_t s_len; -if (kh_get_str (shown, s) !=kh_end (shown)) +if (!strset_add (&shown, s)) duplicates++;else { -kh_put_str (shown, s, &hash_ret); -trace_printf_key (&trace_fsmonitor,"send[%" PRIuMAX"]: %s" , count, s);
@@ -973,7 +968,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,trace2_data_intmax ("fsmonitor" , the_repository,"response/count/duplicates" , duplicates); cleanup: -kh_destroy_str (shown); +strset_clear (&shown);strbuf_release (&response_token);strbuf_release (&requested_token_id);strbuf_release (&payload);
Signature · 1 line
gitgitgadget
Quoted 8 lines · Paul Tarjan via GitGitGadget
This series builds on work from https://github.com/git/git/pull/1352 by Eric DeCosta and https://github.com/git/git/pull/1667 by Marziyeh Esipreh, updated to work with the current codebase and address all review feedback.
Changes since v13:
* Also disable
GIT_TEST_SPLIT_INDEXin t9211 per Junio's feedback (every test does scalar clone)
Thanks. The range-diff since the last iteration looks good.
Will queue.
v15 cover fsmonitor: implement filesystem change listener for Linux
In reply to Paul Tarjan via GitGitGadgetThis series implements the built-in fsmonitor daemon for Linux using the inotify API, bringing it to feature parity with the existing Windows and macOS implementations. It also fixes two memory leaks in the platform-independent daemon code and deduplicates the IPC and settings logic that is now shared between macOS and Linux.
The implementation uses inotify rather than fanotify because fanotify requires either CAP_SYS_ADMIN or CAP_PERFMON capabilities, making it unsuitable for an unprivileged user-space daemon. While inotify has the limitation of requiring a separate watch on every directory (unlike macOS FSEvents, which can monitor an entire directory tree with a single watch), it operates without elevated privileges and provides the per-file event granularity needed for fsmonitor.
The listener uses inotify_init1(O_NONBLOCK) with a poll loop that checks for events with a 50-millisecond timeout, keeping the inotify queue well-drained to minimize the risk of overflows. Bidirectional hashmaps map between watch descriptors and directory paths for efficient event resolution. Directory renames are tracked using inotify cookie mechanism to correlate IN_MOVED_FROM and IN_MOVED_TO event pairs; a periodic check detects stale renames where the matching IN_MOVED_TO never arrived, forcing a resync.
New directory creation triggers recursive watch registration to ensure all subdirectories are monitored. The IN_MASK_CREATE flag is used where available to prevent modifying existing watches, with a fallback for older kernels. When IN_MASK_CREATE is available and inotify_add_watch returns EEXIST, it means another thread or recursive scan has already registered the watch, so it is safe to ignore.
Remote filesystem detection uses statfs() to identify network-mounted filesystems (NFS, CIFS, SMB, FUSE, etc.) via their magic numbers. Mount point information is read from /proc/mounts and matched against the statfs f_fsid to get accurate, human-readable filesystem type names for logging. When the .git directory is on a remote filesystem, the IPC socket falls back to $HOME or a user-configured directory via the fsmonitor.socketDir setting.
This series builds on work from https://github.com/git/git/pull/1352 by Eric DeCosta and https://github.com/git/git/pull/1667 by Marziyeh Esipreh, updated to work with the current codebase and address all review feedback.
Changes since v14:
* Fix stop_daemon_delete_repo: test_might_fail doesn't allow maybe_timeout, replaced with { ... || :; } per SZEDER/Peff review, restoring broken &&-chain
Changes since v13:
* Also disable GIT_TEST_SPLIT_INDEX in t9211 per Junio's feedback (every test does scalar clone)
Changes since v12:
* Dropped both the fsmonitor.c workaround and the read-cache.c skipHash fix per Dscho's review: split-index and index.skipHash are fundamentally incompatible
* Added sane_unset GIT_TEST_SPLIT_INDEX to scalar clone tests in t9210 (patch 1/13)
Changes since v11:
* Fix t9210 BUG assertion with GIT_TEST_SPLIT_INDEX=yes: guard fsmonitor_dirty bitmap access against split-index having fewer entries than the bitmap expects
Changes since v10:
* Reverted pre_exec_cb callback back to simple close_fd_above_stderr flag per Junio's clarification (same as v8)
Changes since v9:
* Fixed Windows build: close_fd_above_stderr() compiles as a no-op on Windows since there is no fork/exec
Changes since v8:
* Replaced close_fd_above_stderr flag with generic pre_exec_cb callback in struct child_process per Junio's review, with close_fd_above_stderr() as a ready-made callback
Changes since v7:
* Added patch 12: convert khash to strset in do_handle_client (Patrick's #leftoverbit suggestion)
* Fixed "Forcing shutdown" trace message to start with lowercase
* Fixed redundant statfs() call in find_mount() (caller already had the result)
* Fixed CMakeLists.txt GIT-BUILD-OPTIONS: was hardcoded to "win32" for FSMONITOR_DAEMON_BACKEND and FSMONITOR_OS_SETTINGS, now uses the CMake variables
* Fixed uninitialized strset on trivial response path (STRSET_INIT)
* Removed V9FS_MAGIC from get_fs_typename() to match is_remote_fs() (9p is local VM mounts)
* Split 30-second stop timeout into its own commit per review request
* Fixed misleading indentation on shutdown assignment in handle_events()
* Updated commit messages to describe all changes (test hardening, fsmonitor-ipc.c spawn changes)
* Updated Makefile comment for FSMONITOR_OS_SETTINGS to mention fsm-ipc
Changes since v6:
* Introduced FSMONITOR_OS_SETTINGS build variable (set to "unix" for macOS and Linux, "win32" for Windows) to eliminate if/else conditionals in Makefile, meson.build, and CMakeLists.txt per Junio's review
* Moved fsm-path-utils from FSMONITOR_OS_SETTINGS to FSMONITOR_DAEMON_BACKEND since path-utils files are platform-specific
* Removed V9FS_MAGIC from remote filesystem detection (9p is used for local VM/container host mounts where fsmonitor works fine)
* Removed redundant #include <libgen.h> (already provided by compat/posix.h)
* Fixed cookie wait comment wording ("see" to "observe")
* Rewrote commit messages for IPC and settings dedup patches
Changes since v5:
* Split monolithic commit into 10-patch series per Patrick's review
* Deduplicated fsm-ipc and fsm-settings into shared Unix implementations
* Rewrote commit message with prose paragraphs, explain inotify vs fanotify, removed "Issues addressed" sections, added Based-on-patch-by trailers
* Removed redundant includes already provided by compat/posix.h
* Fixed error/trace message capitalization per coding guidelines
* Fixed stale rename check interval from 1000 seconds to 1 second
* Changed poll timeout from 1ms to 50ms to reduce idle CPU wake-ups
* Replaced infinite pthread_cond_wait cookie loop with one-second pthread_cond_timedwait (prevents daemon hangs on overlay filesystems where events are never delivered)
* Added pthread_cond_timedwait to Windows pthread compatibility layer
* Separated test into its own commit with smoke test that skips when inotify events are not delivered (e.g., overlayfs with older kernels)
* Fixed test hang on Fedora CI: stop_git() looped forever when ps was unavailable because bash in POSIX/sh mode returns exit 0 from kill with an empty process group argument. Fixed by falling back to /proc/$pid/stat for process group ID and guarding stop_git against empty pgid.
* Redirect spawn_daemon() stdout/stderr to /dev/null and close inherited file descriptors to prevent the intermediate process from holding test pipe file descriptors
* Call setsid() on daemon detach to prevent shells with job control from waiting on the daemon process group
* Close inherited file descriptors 3-7 in the test watchdog subprocess
* Added 30-second timeout to "fsmonitor--daemon stop" to prevent indefinite blocking
* Added helpful error message when inotify watch limit (max_user_watches) is reached
* Initialize fd_inotify to -1 and use fd >= 0 check for correct fd 0 handling
* Use sysconf(_SC_OPEN_MAX) instead of hardcoded 1024 for fd close limit
* Check setsid() return value
Changes since v4:
* Added Meson build support
Changes since v3:
* Fix crash on rapid nested directory creation (EEXIST from inotify_add_watch with IN_MASK_CREATE)
* Extensive stress testing
Changes since v2:
* Fix khash memory leak in do_handle_client
Changes since v1:
* Fix hashmap memory leak in fsmonitor_run_daemon()
Paul Tarjan (13):
t9210, t9211: disable GIT_TEST_SPLIT_INDEX for scalar clone tests
fsmonitor: fix khash memory leak in do_handle_client
fsmonitor: fix hashmap memory leak in fsmonitor_run_daemon compat/win32: add pthread_cond_timedwait
fsmonitor: use pthread_cond_timedwait for cookie wait
fsmonitor: rename fsm-ipc-darwin.c to fsm-ipc-unix.c
fsmonitor: rename fsm-settings-darwin.c to fsm-settings-unix.c
fsmonitor: implement filesystem change listener for Linux
run-command: add close_fd_above_stderr option
fsmonitor: close inherited file descriptors and detach in daemon
fsmonitor: add timeout to daemon stop command
fsmonitor: add tests for Linux
fsmonitor: convert shown khash to strset in do_handle_client
Documentation/config/fsmonitor--daemon.adoc | 4 +-
Documentation/git-fsmonitor--daemon.adoc | 28 +-
Makefile | 6 +-
builtin/fsmonitor--daemon.c | 92 ++-
compat/fsmonitor/fsm-health-linux.c | 33 +
.../{fsm-ipc-darwin.c => fsm-ipc-unix.c} | 0
compat/fsmonitor/fsm-listen-linux.c | 746 ++++++++++++++++++
compat/fsmonitor/fsm-path-utils-linux.c | 217 +++++
...-settings-darwin.c => fsm-settings-unix.c} | 0
compat/win32/pthread.c | 26 +
compat/win32/pthread.h | 2 +
config.mak.uname | 12 +-
contrib/buildsystems/CMakeLists.txt | 33 +-
fsmonitor-ipc.c | 3 +
meson.build | 13 +-
run-command.c | 12 +
run-command.h | 9 +
t/t7527-builtin-fsmonitor.sh | 88 ++-
t/t9210-scalar.sh | 6 +
t/t9211-scalar-clone.sh | 5 +
20 files changed, 1272 insertions(+), 63 deletions(-)
create mode 100644 compat/fsmonitor/fsm-health-linux.c
rename compat/fsmonitor/{fsm-ipc-darwin.c => fsm-ipc-unix.c} (100%)
create mode 100644 compat/fsmonitor/fsm-listen-linux.c
create mode 100644 compat/fsmonitor/fsm-path-utils-linux.c
rename compat/fsmonitor/{fsm-settings-darwin.c => fsm-settings-unix.c} (100%)base-commit: 3e0db84c88c57e70ac8be8c196dfa92c5d656fbc
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-2147%2Fptarjan%2Fclaude%2Fupdate-pr-1352-current-85Gk8-v15
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-2147/ptarjan/claude/update-pr-1352-current-85Gk8-v15
Pull-Request: https://github.com/git/git/pull/2147
Range-diff vs v14:
1: 721a951423 = 1: 721a951423 t9210, t9211: disable GIT_TEST_SPLIT_INDEX for scalar clone tests
2: 1283d25968 = 2: 1283d25968 fsmonitor: fix khash memory leak in do_handle_client
3: 11ba6ca9ac = 3: 11ba6ca9ac fsmonitor: fix hashmap memory leak in fsmonitor_run_daemon
4: a0d430c2f4 = 4: a0d430c2f4 compat/win32: add pthread_cond_timedwait
5: 7b62c95c44 = 5: 7b62c95c44 fsmonitor: use pthread_cond_timedwait for cookie wait
6: 7086cd4530 = 6: 7086cd4530 fsmonitor: rename fsm-ipc-darwin.c to fsm-ipc-unix.c
7: 46e8c2b74f = 7: 46e8c2b74f fsmonitor: rename fsm-settings-darwin.c to fsm-settings-unix.c
8: b3f40a497b = 8: b3f40a497b fsmonitor: implement filesystem change listener for Linux
9: 5791edbef2 = 9: 5791edbef2 run-command: add close_fd_above_stderr option
10: 22d425ebeb = 10: 22d425ebeb fsmonitor: close inherited file descriptors and detach in daemon
11: fd6bdc8c55 = 11: fd6bdc8c55 fsmonitor: add timeout to daemon stop command
12: f85983ca93 ! 12: ffffd64a4b fsmonitor: add tests for Linux
@@ t/t7527-builtin-fsmonitor.sh: then
stop_daemon_delete_repo () {
r=$1 &&
- test_might_fail git -C $r fsmonitor--daemon stop &&
-+ test_might_fail maybe_timeout 30 \
-+ git -C $r fsmonitor--daemon stop 2>/dev/null
++ { maybe_timeout 30 git -C $r fsmonitor--daemon stop 2>/dev/null || :; } &&
rm -rf $1
} 13: 2085b21e23 = 13: ada3f1888b fsmonitor: convert shown khash to strset in do_handle_client
Signature · 1 line
gitgitgadget
PATCH v15 1/13 t9210, t9211: disable GIT_TEST_SPLIT_INDEX for scalar clone tests
From: Paul Tarjan <[email protected]>
index.skipHash (Scalar default) and split-index are incompatible:
the shared index gets a null OID when skipHash skips computing the hash, and the null OID causes the shared index to not be loaded on re-read. This triggers a BUG assertion in fsmonitor when the fsmonitor_dirty bitmap references more entries than the (now empty) index has.
Disable GIT_TEST_SPLIT_INDEX in the scalar clone tests that hit
this: tests 12, 13, and 22 in t9210 (matching the existing workaround in test 16), and all of t9211 (every test does scalar clone).
--- t/t9210-scalar.sh | 6 ++++++ t/t9211-scalar-clone.sh | 5 +++++ 2 files changed, 11 insertions(+)
t/t9210-scalar.sh +6 -0
@@ -152,6 +152,10 @@ test_expect_success 'set up repository to clone' ' ' test_expect_success 'scalar clone' ' + # index.skipHash (Scalar default) and GIT_TEST_SPLIT_INDEX are + # incompatible: the shared index gets a null OID and fails to + # load on re-read. + sane_unset GIT_TEST_SPLIT_INDEX && second=$(git rev-parse --verify second:second.t) && scalar clone "file://$(pwd)" cloned --single-branch && (
@@ -182,6 +186,7 @@ test_expect_success 'scalar clone' ' ' test_expect_success 'scalar clone --no-... opts' ' + sane_unset GIT_TEST_SPLIT_INDEX && # Note: redirect stderr always to avoid having a verbose test # run result in a difference in the --[no-]progress option. GIT_TRACE2_EVENT="$(pwd)/no-opt-trace" scalar clone \
@@ -307,6 +312,7 @@ test_expect_success '`scalar [...] <dir>` errors out when dir is missing' ' SQ="'" test_expect_success UNZIP 'scalar diagnose' ' + sane_unset GIT_TEST_SPLIT_INDEX && scalar clone "file://$(pwd)" cloned --single-branch && git repack && echo "$(pwd)/.git/objects/" >>cloned/src/.git/objects/info/alternates &&
t/t9211-scalar-clone.sh +5 -0
@@ -8,6 +8,11 @@ test_description='test the `scalar clone` subcommand' GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt,launchctl:true,schtasks:true" export GIT_TEST_MAINT_SCHEDULER +# index.skipHash (Scalar default) and GIT_TEST_SPLIT_INDEX are +# incompatible: the shared index gets a null OID and fails to +# load on re-read. Every test here uses scalar clone. +sane_unset GIT_TEST_SPLIT_INDEX + test_expect_success 'set up repository to clone' ' rm -rf .git && git init to-clone &&
Signature · 1 line
gitgitgadget
PATCH v15 2/13 fsmonitor: fix khash memory leak in do_handle_client
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
The shown kh_str_t was freed with kh_release_str() at a point in the code only reachable in the non-trivial response path. When the client receives a trivial response, the code jumps to the cleanup label, skipping the kh_release_str() call entirely and leaking the hash table.
Fix this by initializing shown to NULL and moving the cleanup to the cleanup label using kh_destroy_str(), which is safe to call on NULL. This ensures the hash table is freed regardless of which code path is taken.
--- builtin/fsmonitor--daemon.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
builtin/fsmonitor--daemon.c +2 -3
@@ -671,7 +671,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,const struct fsmonitor_batch *batch;struct fsmonitor_batch *remainder =NULL ;intmax_t count =0 , duplicates =0 ; -kh_str_t *shown; +kh_str_t *shown =NULL ;int hash_ret;int do_trivial =0 ;int do_flush =0 ;
@@ -909,8 +909,6 @@ static int do_handle_client(struct fsmonitor_daemon_state *state, total_response_len += payload.len; } -kh_release_str (shown); -pthread_mutex_lock (&state->main_lock);if (token_data->client_ref_count >0 )
@@ -954,6 +952,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,trace2_data_intmax ("fsmonitor" , the_repository,"response/count/duplicates" , duplicates); cleanup: +kh_destroy_str (shown);strbuf_release (&response_token);strbuf_release (&requested_token_id);strbuf_release (&payload);
Signature · 1 line
gitgitgadget
PATCH v15 3/13 fsmonitor: fix hashmap memory leak in fsmonitor_run_daemon
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
The state.cookies hashmap is initialized during daemon startup but never freed during cleanup in the done: label of fsmonitor_run_daemon(). The cookie entries also have names allocated via strbuf_detach() that must be freed individually.
Iterate the hashmap to free each cookie name, then call hashmap_clear_and_free() to release the entries and table.
--- builtin/fsmonitor--daemon.c | 9 +++++++++ 1 file changed, 9 insertions(+)
builtin/fsmonitor--daemon.c +9 -0
@@ -1404,6 +1404,15 @@ static int fsmonitor_run_daemon(void) done:pthread_cond_destroy (&state.cookies_cond);pthread_mutex_destroy (&state.main_lock); + { +struct hashmap_iter iter; +struct fsmonitor_cookie_item *cookie; + +hashmap_for_each_entry (&state.cookies, &iter, cookie, entry) +free (cookie->name); +hashmap_clear_and_free (&state.cookies, + struct fsmonitor_cookie_item, entry); + }fsm_listen__dtor (&state);fsm_health__dtor (&state);
Signature · 1 line
gitgitgadget
PATCH v15 4/13 compat/win32: add pthread_cond_timedwait
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
Add a pthread_cond_timedwait() implementation to the Windows pthread compatibility layer using SleepConditionVariableCS() with a millisecond timeout computed from the absolute deadline.
--- compat/win32/pthread.c | 26 ++++++++++++++++++++++++++ compat/win32/pthread.h | 2 ++ 2 files changed, 28 insertions(+)
compat/win32/pthread.c +26 -0
@@ -66,3 +66,29 @@ int pthread_cond_wait(pthread_cond_t *cond, pthread_mutex_t *mutex)return err_win_to_posix (GetLastError ());return 0 ; } + +int pthread_cond_timedwait (pthread_cond_t *cond,pthread_mutex_t *mutex, +const struct timespec *abstime) +{ +struct timeval now; +long long now_ms, deadline_ms; +DWORD timeout_ms; + +gettimeofday (&now,NULL ); + now_ms = (long long )now.tv_sec *1000 + now.tv_usec /1000 ; + deadline_ms = (long long )abstime->tv_sec *1000 + + abstime->tv_nsec /1000000 ; + +if (deadline_ms <= now_ms) +return ETIMEDOUT ; +else + timeout_ms = (DWORD )(deadline_ms - now_ms); + +if (SleepConditionVariableCS (cond, mutex, timeout_ms) ==0 ) { +DWORD err =GetLastError (); +if (err ==ERROR_TIMEOUT ) +return ETIMEDOUT ; +return err_win_to_posix (err); + } +return 0 ; +}
compat/win32/pthread.h +2 -0
@@ -64,6 +64,8 @@ int win32_pthread_join(pthread_t *thread, void **value_ptr);pthread_t pthread_self (void );int pthread_cond_wait (pthread_cond_t *cond,pthread_mutex_t *mutex); +int pthread_cond_timedwait (pthread_cond_t *cond,pthread_mutex_t *mutex, +const struct timespec *abstime); static inline void NORETURN pthread_exit(void *ret) {
Signature · 1 line
gitgitgadget
PATCH v15 5/13 fsmonitor: use pthread_cond_timedwait for cookie wait
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
The cookie wait in with_lock__wait_for_cookie() uses an infinite pthread_cond_wait() loop. The existing comment notes the desire to switch to pthread_cond_timedwait(), but the routine was not available in git thread-utils.
On certain container or overlay filesystems, inotify watches may succeed but events are never delivered. In this case the daemon would hang indefinitely waiting for the cookie event, which in turn causes the client to hang.
Replace the infinite wait with a one-second timeout using pthread_cond_timedwait(). If the timeout fires, report an error and let the client proceed with a trivial (full-scan) response rather than blocking forever.
--- builtin/fsmonitor--daemon.c | 37 ++++++++++++++++++++++++------------- 1 file changed, 24 insertions(+), 13 deletions(-)
builtin/fsmonitor--daemon.c +24 -13
@@ -197,20 +197,31 @@ static enum fsmonitor_cookie_item_result with_lock__wait_for_cookie(unlink (cookie_pathname.buf);/* -* Technically, this is an infinite wait (well, unless another -* thread sends us an abort). I'd like to change this to -* use `pthread_cond_timedwait()` and return an error/timeout -* and let the caller do the trivial response thing, but we -* don't have that routine in our thread-utils. -* -* After extensive beta testing I'm not really worried about -* this. Also note that the above open() and unlink() calls -* will cause at least two FS events on that path, so the odds -* of getting stuck are pretty slim. +* Wait for the listener thread to observe the cookie file. +* Time out after a short interval so that the client +* does not hang forever if the filesystem does not deliver +* events (e.g., on certain container/overlay filesystems +* where inotify watches succeed but events never arrive). */ -while (cookie->result ==FCIR_INIT ) -pthread_cond_wait (&state->cookies_cond, - &state->main_lock); + { +struct timeval now; +struct timespec ts; +int err =0 ; + +gettimeofday (&now,NULL ); + ts.tv_sec = now.tv_sec +1 ; + ts.tv_nsec = now.tv_usec *1000 ; + +while (cookie->result ==FCIR_INIT && !err) + err =pthread_cond_timedwait (&state->cookies_cond, + &state->main_lock, + &ts); +if (err ==ETIMEDOUT && cookie->result ==FCIR_INIT ) { +trace_printf_key (&trace_fsmonitor, +"cookie_wait timed out" ); + cookie->result =FCIR_ERROR ; + } + } done:hashmap_remove (&state->cookies, &cookie->entry,NULL );
Signature · 1 line
gitgitgadget
PATCH v15 6/13 fsmonitor: rename fsm-ipc-darwin.c to fsm-ipc-unix.c
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
The fsmonitor IPC path logic in fsm-ipc-darwin.c is not Darwin-specific and will be reused by the upcoming Linux implementation. Rename it to fsm-ipc-unix.c to reflect that it is shared by all Unix platforms.
Introduce FSMONITOR_OS_SETTINGS (set to "unix" for non-Windows, "win32" for Windows) as a separate variable from FSMONITOR_DAEMON_BACKEND so that the build files can distinguish between platform-specific files (listen, health, path-utils) and shared Unix files (ipc, settings).
Move fsm-ipc to the FSMONITOR_OS_SETTINGS section in the Makefile, and switch fsm-path-utils to use FSMONITOR_DAEMON_BACKEND since path-utils is platform-specific (there will be separate darwin and linux versions).
---
Makefile | 6 ++---
.../{fsm-ipc-darwin.c => fsm-ipc-unix.c} | 0
config.mak.uname | 2 +-
contrib/buildsystems/CMakeLists.txt | 25 +++++++++----------
meson.build | 7 ++++--
5 files changed, 21 insertions(+), 19 deletions(-)
rename compat/fsmonitor/{fsm-ipc-darwin.c => fsm-ipc-unix.c} (100%)Makefile +3 -3
@@ -417,7 +417,7 @@ include shared.mak # If your platform has OS-specific ways to tell if a repo is incompatible with # fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS # to the "<name>" of the corresponding `compat/fsmonitor/fsm-settings-<name>.c` -# that implements the `fsm_os_settings__*()` routines. +# and `compat/fsmonitor/fsm-ipc-<name>.c` files. # # Define LINK_FUZZ_PROGRAMS if you want `make all` to also build the fuzz test # programs in oss-fuzz/.
@@ -2365,13 +2365,13 @@ ifdef FSMONITOR_DAEMON_BACKEND COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o - COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_BACKEND).o endif ifdef FSMONITOR_OS_SETTINGS COMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS + COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_OS_SETTINGS).o COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o - COMPAT_OBJS += compat/fsmonitor/fsm-path-utils-$(FSMONITOR_OS_SETTINGS).o + COMPAT_OBJS += compat/fsmonitor/fsm-path-utils-$(FSMONITOR_DAEMON_BACKEND).o endif ifdef WITH_BREAKING_CHANGES
compat/fsmonitor/fsm-ipc-unix.c renamed from compat/fsmonitor/fsm-ipc-darwin.c +0 -0
config.mak.uname +1 -1
@@ -165,7 +165,7 @@ ifeq ($(uname_S),Darwin) ifndef NO_PTHREADS ifndef NO_UNIX_SOCKETS FSMONITOR_DAEMON_BACKEND = darwin - FSMONITOR_OS_SETTINGS = darwin + FSMONITOR_OS_SETTINGS = unix endif endif
contrib/buildsystems/CMakeLists.txt +12 -13
@@ -291,23 +291,22 @@ endif() if(SUPPORTS_SIMPLE_IPC) if(CMAKE_SYSTEM_NAME STREQUAL "Windows") - add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-win32.c) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-win32.c) - - add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c) + set(FSMONITOR_DAEMON_BACKEND "win32") + set(FSMONITOR_OS_SETTINGS "win32") elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") + set(FSMONITOR_DAEMON_BACKEND "darwin") + set(FSMONITOR_OS_SETTINGS "unix") + endif() + + if(FSMONITOR_DAEMON_BACKEND) add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-darwin.c) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-darwin.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-${FSMONITOR_DAEMON_BACKEND}.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-${FSMONITOR_DAEMON_BACKEND}.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-${FSMONITOR_OS_SETTINGS}.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-${FSMONITOR_DAEMON_BACKEND}.c) add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-${FSMONITOR_DAEMON_BACKEND}.c) endif() endif()
meson.build +5 -2
@@ -1320,10 +1320,13 @@ else endif fsmonitor_backend = '' +fsmonitor_os = '' if host_machine.system() == 'windows' fsmonitor_backend = 'win32' + fsmonitor_os = 'win32' elif host_machine.system() == 'darwin' fsmonitor_backend = 'darwin' + fsmonitor_os = 'unix' libgit_dependencies += dependency('CoreServices') endif if fsmonitor_backend != ''
@@ -1332,14 +1335,14 @@ if fsmonitor_backend != '' libgit_sources += [ 'compat/fsmonitor/fsm-health-' + fsmonitor_backend + '.c', - 'compat/fsmonitor/fsm-ipc-' + fsmonitor_backend + '.c', + 'compat/fsmonitor/fsm-ipc-' + fsmonitor_os + '.c', 'compat/fsmonitor/fsm-listen-' + fsmonitor_backend + '.c', 'compat/fsmonitor/fsm-path-utils-' + fsmonitor_backend + '.c', 'compat/fsmonitor/fsm-settings-' + fsmonitor_backend + '.c', ] endif build_options_config.set_quoted('FSMONITOR_DAEMON_BACKEND', fsmonitor_backend) -build_options_config.set_quoted('FSMONITOR_OS_SETTINGS', fsmonitor_backend) +build_options_config.set_quoted('FSMONITOR_OS_SETTINGS', fsmonitor_os) if not get_option('b_sanitize').contains('address') and get_option('regex').allowed() and compiler.has_header('regex.h') and compiler.get_define('REG_STARTEND', prefix: '#include <regex.h>') != '' build_options_config.set('NO_REGEX', '')
Signature · 1 line
gitgitgadget
PATCH v15 7/13 fsmonitor: rename fsm-settings-darwin.c to fsm-settings-unix.c
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
The fsmonitor settings logic in fsm-settings-darwin.c is not Darwin-specific and will be reused by the upcoming Linux implementation. Rename it to fsm-settings-unix.c to reflect that it is shared by all Unix platforms.
Update the build files (meson.build and CMakeLists.txt) to use FSMONITOR_OS_SETTINGS for fsm-settings, matching the approach already used for fsm-ipc.
---
compat/fsmonitor/{fsm-settings-darwin.c => fsm-settings-unix.c} | 0
contrib/buildsystems/CMakeLists.txt | 2 +-
meson.build | 2 +-
3 files changed, 2 insertions(+), 2 deletions(-)
rename compat/fsmonitor/{fsm-settings-darwin.c => fsm-settings-unix.c} (100%)compat/fsmonitor/fsm-settings-unix.c renamed from compat/fsmonitor/fsm-settings-darwin.c +0 -0
contrib/buildsystems/CMakeLists.txt +1 -1
@@ -306,7 +306,7 @@ if(SUPPORTS_SIMPLE_IPC) list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-${FSMONITOR_DAEMON_BACKEND}.c) add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-${FSMONITOR_DAEMON_BACKEND}.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-${FSMONITOR_OS_SETTINGS}.c) endif() endif()
meson.build +1 -1
@@ -1338,7 +1338,7 @@ if fsmonitor_backend != '' 'compat/fsmonitor/fsm-ipc-' + fsmonitor_os + '.c', 'compat/fsmonitor/fsm-listen-' + fsmonitor_backend + '.c', 'compat/fsmonitor/fsm-path-utils-' + fsmonitor_backend + '.c', - 'compat/fsmonitor/fsm-settings-' + fsmonitor_backend + '.c', + 'compat/fsmonitor/fsm-settings-' + fsmonitor_os + '.c', ] endif build_options_config.set_quoted('FSMONITOR_DAEMON_BACKEND', fsmonitor_backend)
Signature · 1 line
gitgitgadget
PATCH v15 8/13 fsmonitor: implement filesystem change listener for Linux
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
Implement the built-in fsmonitor daemon for Linux using the inotify API, bringing it to feature parity with the existing Windows and macOS implementations.
The implementation uses inotify rather than fanotify because fanotify requires either CAP_SYS_ADMIN or CAP_PERFMON capabilities, making it unsuitable for an unprivileged user-space daemon. While inotify has the limitation of requiring a separate watch on every directory (unlike macOS's FSEvents, which can monitor an entire directory tree with a single watch), it operates without elevated privileges and provides the per-file event granularity needed for fsmonitor.
The listener uses inotify_init1(O_NONBLOCK) with a poll loop that checks for events with a 50-millisecond timeout, keeping the inotify queue well-drained to minimize the risk of overflows. Bidirectional hashmaps map between watch descriptors and directory paths for efficient event resolution. Directory renames are tracked using inotify's cookie mechanism to correlate IN_MOVED_FROM and IN_MOVED_TO event pairs; a periodic check detects stale renames where the matching IN_MOVED_TO never arrived, forcing a resync.
New directory creation triggers recursive watch registration to ensure all subdirectories are monitored. The IN_MASK_CREATE flag is used where available to prevent modifying existing watches, with a fallback for older kernels. When IN_MASK_CREATE is available and inotify_add_watch returns EEXIST, it means another thread or recursive scan has already registered the watch, so it is safe to ignore.
Remote filesystem detection uses statfs() to identify network-mounted filesystems (NFS, CIFS, SMB, FUSE, etc.) via their magic numbers. Mount point information is read from /proc/mounts and matched against the statfs f_fsid to get accurate, human-readable filesystem type names for logging. When the .git directory is on a remote filesystem, the IPC socket falls back to $HOME or a user-configured directory via the fsmonitor.socketDir setting.
--- Documentation/config/fsmonitor--daemon.adoc | 4 +- Documentation/git-fsmonitor--daemon.adoc | 28 +- compat/fsmonitor/fsm-health-linux.c | 33 + compat/fsmonitor/fsm-listen-linux.c | 746 ++++++++++++++++++++ compat/fsmonitor/fsm-path-utils-linux.c | 217 ++++++ config.mak.uname | 10 + contrib/buildsystems/CMakeLists.txt | 8 +- meson.build | 4 + 8 files changed, 1042 insertions(+), 8 deletions(-) create mode 100644 compat/fsmonitor/fsm-health-linux.c create mode 100644 compat/fsmonitor/fsm-listen-linux.c create mode 100644 compat/fsmonitor/fsm-path-utils-linux.c
Documentation/config/fsmonitor--daemon.adoc +2 -2
@@ -4,8 +4,8 @@ fsmonitor.allowRemote:: behavior. Only respected when `core.fsmonitor` is set to `true`. fsmonitor.socketDir:: - This Mac OS-specific option, if set, specifies the directory in + This Mac OS and Linux-specific option, if set, specifies the directory in which to create the Unix domain socket used for communication between the fsmonitor daemon and various Git commands. The directory must - reside on a native Mac OS filesystem. Only respected when `core.fsmonitor` + reside on a native filesystem. Only respected when `core.fsmonitor` is set to `true`.
Documentation/git-fsmonitor--daemon.adoc +24 -4
@@ -76,9 +76,9 @@ repositories; this may be overridden by setting `fsmonitor.allowRemote` to correctly with all network-mounted repositories, so such use is considered experimental. -On Mac OS, the inter-process communication (IPC) between various Git +On Mac OS and Linux, the inter-process communication (IPC) between various Git commands and the fsmonitor daemon is done via a Unix domain socket (UDS) -- a -special type of file -- which is supported by native Mac OS filesystems, +special type of file -- which is supported by native Mac OS and Linux filesystems, but not on network-mounted filesystems, NTFS, or FAT32. Other filesystems may or may not have the needed support; the fsmonitor daemon is not guaranteed to work with these filesystems and such use is considered experimental.
@@ -87,13 +87,33 @@ By default, the socket is created in the `.git` directory. However, if the `.git` directory is on a network-mounted filesystem, it will instead be created at `$HOME/.git-fsmonitor-*` unless `$HOME` itself is on a network-mounted filesystem, in which case you must set the configuration -variable `fsmonitor.socketDir` to the path of a directory on a Mac OS native +variable `fsmonitor.socketDir` to the path of a directory on a native filesystem in which to create the socket file. If none of the above directories (`.git`, `$HOME`, or `fsmonitor.socketDir`) -is on a native Mac OS file filesystem the fsmonitor daemon will report an +is on a native filesystem the fsmonitor daemon will report an error that will cause the daemon and the currently running command to exit. +LINUX CAVEATS +~~~~~~~~~~~~~ + +On Linux, the fsmonitor daemon uses inotify to monitor filesystem events. +The inotify system has per-user limits on the number of watches that can +be created. The default limit is typically 8192 watches per user. + +For large repositories with many directories, you may need to increase +this limit. Check the current limit with: + + cat /proc/sys/fs/inotify/max_user_watches + +To temporarily increase the limit: + + sudo sysctl fs.inotify.max_user_watches=65536 + +To make the change permanent, add to `/etc/sysctl.conf`: + + fs.inotify.max_user_watches=65536 + CONFIGURATION -------------
compat/fsmonitor/fsm-health-linux.c new file +33 -0
@@ -0,0 +1,33 @@ +#include "git-compat-util.h" +#include "config.h" +#include "fsmonitor-ll.h" +#include "fsm-health.h" +#include "fsmonitor--daemon.h" + +/* +* The Linux fsmonitor implementation uses inotify which has its own +* mechanisms for detecting filesystem unmount and other events that +* would require the daemon to shutdown. Therefore, we don't need +* a separate health thread like Windows does. +* +* These stub functions satisfy the interface requirements. +*/ + +int fsm_health__ctor(struct fsmonitor_daemon_state *state UNUSED) +{ +return 0 ; +} + +void fsm_health__dtor(struct fsmonitor_daemon_state *state UNUSED) +{ +return ; +} + +void fsm_health__loop(struct fsmonitor_daemon_state *state UNUSED) +{ +return ; +} + +void fsm_health__stop_async(struct fsmonitor_daemon_state *state UNUSED) +{ +}
compat/fsmonitor/fsm-listen-linux.c new file +746 -0
746 lines not on this page · raw message
compat/fsmonitor/fsm-path-utils-linux.c new file +217 -0
217 lines not on this page · raw message
config.mak.uname +10 -0
@@ -68,6 +68,16 @@ ifeq ($(uname_S),Linux) BASIC_CFLAGS += -std=c99 endif LINK_FUZZ_PROGRAMS = YesPlease + + # The builtin FSMonitor on Linux builds upon Simple-IPC. Both require + # Unix domain sockets and PThreads. + ifndef NO_PTHREADS + ifndef NO_UNIX_SOCKETS + FSMONITOR_DAEMON_BACKEND = linux + FSMONITOR_OS_SETTINGS = unix + BASIC_CFLAGS += -DHAVE_LINUX_MAGIC_H + endif + endif endif ifeq ($(uname_S),GNU/kFreeBSD) HAVE_ALLOCA_H = YesPlease
contrib/buildsystems/CMakeLists.txt +6 -2
@@ -296,6 +296,10 @@ if(SUPPORTS_SIMPLE_IPC) elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") set(FSMONITOR_DAEMON_BACKEND "darwin") set(FSMONITOR_OS_SETTINGS "unix") + elseif(CMAKE_SYSTEM_NAME STREQUAL "Linux") + set(FSMONITOR_DAEMON_BACKEND "linux") + set(FSMONITOR_OS_SETTINGS "unix") + add_compile_definitions(HAVE_LINUX_MAGIC_H) endif() if(FSMONITOR_DAEMON_BACKEND)
@@ -1149,8 +1153,8 @@ endif() file(STRINGS ${CMAKE_SOURCE_DIR}/GIT-BUILD-OPTIONS.in git_build_options NEWLINE_CONSUME) string(REPLACE "@BROKEN_PATH_FIX@" "" git_build_options "${git_build_options}") string(REPLACE "@DIFF@" "'${DIFF}'" git_build_options "${git_build_options}") -string(REPLACE "@FSMONITOR_DAEMON_BACKEND@" "win32" git_build_options "${git_build_options}") -string(REPLACE "@FSMONITOR_OS_SETTINGS@" "win32" git_build_options "${git_build_options}") +string(REPLACE "@FSMONITOR_DAEMON_BACKEND@" "${FSMONITOR_DAEMON_BACKEND}" git_build_options "${git_build_options}") +string(REPLACE "@FSMONITOR_OS_SETTINGS@" "${FSMONITOR_OS_SETTINGS}" git_build_options "${git_build_options}") string(REPLACE "@GITWEBDIR@" "'${GITWEBDIR}'" git_build_options "${git_build_options}") string(REPLACE "@GIT_INTEROP_MAKE_OPTS@" "" git_build_options "${git_build_options}") string(REPLACE "@GIT_PERF_LARGE_REPO@" "" git_build_options "${git_build_options}")
meson.build +4 -0
@@ -1324,6 +1324,10 @@ fsmonitor_os = '' if host_machine.system() == 'windows' fsmonitor_backend = 'win32' fsmonitor_os = 'win32' +elif host_machine.system() == 'linux' and threads.found() and compiler.has_header('linux/magic.h') + fsmonitor_backend = 'linux' + fsmonitor_os = 'unix' + libgit_c_args += '-DHAVE_LINUX_MAGIC_H' elif host_machine.system() == 'darwin' fsmonitor_backend = 'darwin' fsmonitor_os = 'unix'
Signature · 1 line
gitgitgadget
PATCH v15 9/13 run-command: add close_fd_above_stderr option
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
Add a close_fd_above_stderr flag to struct child_process. When set, the child closes file descriptors 3 and above between fork and exec (skipping the child-notifier pipe), capped at sysconf(_SC_OPEN_MAX) or 4096, whichever is smaller. This prevents the child from inheriting pipe endpoints or other descriptors from the parent environment (e.g., the test harness).
--- run-command.c | 12 ++++++++++++ run-command.h | 9 +++++++++ 2 files changed, 21 insertions(+)
run-command.c +12 -0
@@ -546,6 +546,7 @@ static void atfork_parent(struct atfork_state *as) "restoring signal mask"); #endif } +#endif /* GIT_WINDOWS_NATIVE */ static inline void set_cloexec (int fd)
@@ -832,6 +833,17 @@ fail_pipe:child_close (cmd->out); } +if (cmd->close_fd_above_stderr) { +long max_fd =sysconf (_SC_OPEN_MAX); +int fd; +if (max_fd <0 || max_fd >4096 ) + max_fd =4096 ; +for (fd =3 ; fd < max_fd; fd++) { +if (fd != child_notifier) +close (fd); + } + } +if (cmd->dir &&chdir (cmd->dir))child_die (CHILD_ERR_CHDIR );
run-command.h +9 -0
@@ -141,6 +141,15 @@ struct child_process { unsigned stdout_to_stderr:1; unsigned clean_on_exit:1; unsigned wait_after_clean:1; + +/** +* Close file descriptors 3 and above in the child after forking +* but before exec. This prevents the child from inheriting +* pipe endpoints or other descriptors from the parent +* environment (e.g., the test harness). +*/ + unsigned close_fd_above_stderr:1; +void (*clean_on_exit_handler)(struct child_process *process); };
Signature · 1 line
gitgitgadget
PATCH v15 10/13 fsmonitor: close inherited file descriptors and detach in daemon
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
When the fsmonitor daemon is spawned as a background process, it may inherit file descriptors from its parent that it does not need. In particular, when the test harness or a CI system captures output through pipes, the daemon can inherit duplicated pipe endpoints. If the daemon holds these open, the parent process never sees EOF and may appear to hang.
Set close_fd_above_stderr on the child process at both daemon startup
paths: the explicit "fsmonitor--daemon start" command and the implicit spawn triggered by fsmonitor-ipc when a client finds no running daemon. Also suppress stdout and stderr on the implicit spawn path to prevent the background daemon from writing to the client's terminal.
Additionally, call setsid() when the daemon starts with --detach to create a new session and process group. This prevents the daemon from being part of the spawning shell's process group, which could cause the shell's "wait" to block until the daemon exits.
--- builtin/fsmonitor--daemon.c | 16 ++++++++++++++-- fsmonitor-ipc.c | 3 +++ 2 files changed, 17 insertions(+), 2 deletions(-)
builtin/fsmonitor--daemon.c +14 -2
@@ -1439,7 +1439,7 @@ done:return err; } -static int try_to_run_foreground_daemon(int detach_console MAYBE_UNUSED) +static int try_to_run_foreground_daemon (int detach_console) {/* * Technically, we don't need to probe for an existing daemon
@@ -1459,10 +1459,21 @@ static int try_to_run_foreground_daemon(int detach_console MAYBE_UNUSED)fflush (stderr); } +if (detach_console) {#ifdef GIT_WINDOWS_NATIVE -if (detach_console)FreeConsole (); +#else +/* +* Create a new session so that the daemon is detached +* from the parent's process group. This prevents +* shells with job control (e.g. bash with "set -m") +* from waiting on the daemon when they wait for a +* foreground command that implicitly spawned it. +*/ +if (setsid () ==-1 ) +warning_errno (_ ("setsid failed" ));#endif + }return !!fsmonitor_run_daemon (); }
@@ -1525,6 +1536,7 @@ static int try_to_start_background_daemon(void) cp.no_stdin =1 ; cp.no_stdout =1 ; cp.no_stderr =1 ; + cp.close_fd_above_stderr =1 ; sbgr =start_bg_command (&cp, bg_wait_cb,NULL , fsmonitor__start_timeout_sec);
fsmonitor-ipc.c +3 -0
@@ -61,6 +61,9 @@ static int spawn_daemon(void) cmd.git_cmd =1 ; cmd.no_stdin =1 ; + cmd.no_stdout =1 ; + cmd.no_stderr =1 ; + cmd.close_fd_above_stderr =1 ; cmd.trace2_child_class ="fsmonitor" ;strvec_pushl (&cmd.args,"fsmonitor--daemon" ,"start" ,NULL );
Signature · 1 line
gitgitgadget
PATCH v15 11/13 fsmonitor: add timeout to daemon stop command
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
The "fsmonitor--daemon stop" command polls in a loop waiting for the daemon to exit after sending a "quit" command over IPC. If the daemon fails to shut down (e.g. it is stuck or wedged), this loop spins forever.
Add a 30-second timeout so the stop command returns an error instead of blocking indefinitely.
--- builtin/fsmonitor--daemon.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
builtin/fsmonitor--daemon.c +11 -1
@@ -86,6 +86,8 @@ static int do_as_client__send_stop(void) {struct strbuf answer =STRBUF_INIT ;int ret; +int max_wait_ms =30000 ; +int elapsed_ms =0 ; ret =fsmonitor_ipc__send_command ("quit" , &answer);
@@ -96,8 +98,16 @@ static int do_as_client__send_stop(void)return ret;trace2_region_enter ("fsm_client" ,"polling-for-daemon-exit" ,NULL ); -while (fsmonitor_ipc__get_state () ==IPC_STATE__LISTENING ) +while (fsmonitor_ipc__get_state () ==IPC_STATE__LISTENING ) { +if (elapsed_ms >= max_wait_ms) { +trace2_region_leave ("fsm_client" , +"polling-for-daemon-exit" ,NULL ); +return error (_ ("daemon did not stop within %d seconds" ), + max_wait_ms /1000 ); + }sleep_millisec (50 ); + elapsed_ms +=50 ; + }trace2_region_leave ("fsm_client" ,"polling-for-daemon-exit" ,NULL );return 0 ;
Signature · 1 line
gitgitgadget
PATCH v15 12/13 fsmonitor: add tests for Linux
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
Add a smoke test that verifies the filesystem actually delivers inotify events to the daemon. On some configurations (e.g., overlayfs with older kernels), inotify watches succeed but events are never delivered. The daemon cookie wait will time out, but every subsequent test would fail. Skip the entire test file early when this is detected.
Add a test that exercises rapid nested directory creation to verify the daemon correctly handles the EEXIST race between recursive scan and queued inotify events. When IN_MASK_CREATE is available and a directory watch is added during recursive registration, the kernel may also deliver a queued IN_CREATE event for the same directory. The second inotify_add_watch() returns EEXIST, which must be treated as harmless. An earlier version of the listener crashed in this scenario.
Reduce --start-timeout from the default 60 seconds to 10 seconds so that tests fail promptly when the daemon cannot start.
Harden the test helpers to work in environments without procps (e.g., Fedora CI): fall back to reading /proc/$pid/stat for the process group ID when ps is unavailable, guard stop_git() against an empty pgid, and redirect stderr from kill to /dev/null to avoid noise when processes have already exited.
Use set -m to enable job control in the submodule-pull test so that the background git pull gets its own process group, preventing the shell wait from blocking on the daemon. setsid() in the previous commit detaches the daemon itself, but the intermediate git pull process still needs its own process group for the test shell to manage it correctly.
--- t/t7527-builtin-fsmonitor.sh | 88 +++++++++++++++++++++++++++++++++--- 1 file changed, 81 insertions(+), 7 deletions(-)
t/t7527-builtin-fsmonitor.sh +81 -7
@@ -10,9 +10,57 @@ then test_done fi +# Verify that the filesystem delivers events to the daemon. +# On some configurations (e.g., overlayfs with older kernels), +# inotify watches succeed but events are never delivered. The +# cookie wait will time out and the daemon logs a trace message. +# +# Use "timeout" (if available) to guard each step against hangs. +maybe_timeout () { + if type timeout >/dev/null 2>&1 + then + timeout "$@" + else + shift + "$@" + fi +} +verify_fsmonitor_works () { + git init test_fsmonitor_smoke || return 1 + + GIT_TRACE_FSMONITOR="$PWD/smoke.trace" && + export GIT_TRACE_FSMONITOR && + maybe_timeout 30 \ + git -C test_fsmonitor_smoke fsmonitor--daemon start \ + --start-timeout=10 + ret=$? + unset GIT_TRACE_FSMONITOR + if test $ret -ne 0 + then + rm -rf test_fsmonitor_smoke smoke.trace + return 1 + fi + + maybe_timeout 10 \ + test-tool -C test_fsmonitor_smoke fsmonitor-client query \ + --token 0 >/dev/null 2>&1 + maybe_timeout 5 \ + git -C test_fsmonitor_smoke fsmonitor--daemon stop 2>/dev/null + ! grep -q "cookie_wait timed out" "$PWD/smoke.trace" 2>/dev/null + ret=$? + rm -rf test_fsmonitor_smoke smoke.trace + return $ret +} + +if ! verify_fsmonitor_works +then + skip_all="filesystem does not deliver fsmonitor events (container/overlayfs?)" + test_done +fi + stop_daemon_delete_repo () { r=$1 && - test_might_fail git -C $r fsmonitor--daemon stop && + { maybe_timeout 30 git -C $r fsmonitor--daemon stop 2>/dev/null || :; } && rm -rf $1 }
@@ -67,7 +115,7 @@ start_daemon () { export GIT_TEST_FSMONITOR_TOKEN fi && - git $r fsmonitor--daemon start && + git $r fsmonitor--daemon start --start-timeout=10 && git $r fsmonitor--daemon status ) }
@@ -520,6 +568,28 @@ test_expect_success 'directory changes to a file' ' grep "^event: dir1$" .git/trace ' +test_expect_success 'rapid nested directory creation' ' + test_when_finished "git fsmonitor--daemon stop; rm -rf rapid" && + + start_daemon --tf "$PWD/.git/trace" && + + # Rapidly create nested directories to exercise race conditions + # where directory watches may be added concurrently during + # event processing and recursive scanning. + for i in $(test_seq 1 20) + do + mkdir -p "rapid/nested/dir$i/subdir/deep" || return 1 + done && + + # Give the daemon time to process all events + sleep 1 && + + test-tool fsmonitor-client query --token 0 && + + # Verify daemon is still running (did not crash) + git fsmonitor--daemon status +' + # The next few test cases exercise the token-resync code. When filesystem # drops events (because of filesystem velocity or because the daemon isn't # polling fast enough), we need to discard the cached data (relative to the
@@ -910,7 +980,10 @@ test_expect_success "submodule absorbgitdirs implicitly starts daemon" ' start_git_in_background () { git "$@" & git_pid=$! - git_pgid=$(ps -o pgid= -p $git_pid) + git_pgid=$(ps -o pgid= -p $git_pid 2>/dev/null || + awk '{print $5}' /proc/$git_pid/stat 2>/dev/null) && + git_pgid="${git_pgid## }" && + git_pgid="${git_pgid%% }" nr_tries_left=10 while true do
@@ -921,15 +994,16 @@ start_git_in_background () { fi sleep 1 nr_tries_left=$(($nr_tries_left - 1)) - done >/dev/null 2>&1 & + done >/dev/null 2>&1 3>&- 4>&- 5>&- 6>&- 7>&- & watchdog_pid=$! wait $git_pid } stop_git () { - while kill -0 -- -$git_pgid + test -n "$git_pgid" || return 0 + while kill -0 -- -$git_pgid 2>/dev/null do - kill -- -$git_pgid + kill -- -$git_pgid 2>/dev/null sleep 1 done }
@@ -944,7 +1018,7 @@ stop_watchdog () { test_expect_success !MINGW "submodule implicitly starts daemon by pull" ' test_atexit "stop_watchdog" && - test_when_finished "stop_git; rm -rf cloned super sub" && + test_when_finished "set +m; stop_git; rm -rf cloned super sub" && create_super super && create_sub sub &&
Signature · 1 line
gitgitgadget
PATCH v15 13/13 fsmonitor: convert shown khash to strset in do_handle_client
In reply to Paul Tarjan via GitGitGadgetFrom: Paul Tarjan <[email protected]>
Replace the khash-based string set used for deduplicating pathnames in do_handle_client() with a strset, which provides a cleaner interface for the same purpose.
Since the paths are interned strings from the batch data, use strdup_strings=0 to avoid unnecessary copies.
--- builtin/fsmonitor--daemon.c | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-)
builtin/fsmonitor--daemon.c +6 -11
@@ -16,7 +16,7 @@#include "fsmonitor--daemon.h" #include "simple-ipc.h" -#include "khash.h" +#include "strmap.h" #include "run-command.h" #include "trace.h" #include "trace2.h"
@@ -674,8 +674,6 @@ static int fsmonitor_parse_client_token(const char *buf_token,return 0 ; } -KHASH_INIT(str, const char *, int, 0, kh_str_hash_func, kh_str_hash_equal) - static int do_handle_client(struct fsmonitor_daemon_state *state, const char *command, ipc_server_reply_cb *reply,
@@ -692,8 +690,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,const struct fsmonitor_batch *batch;struct fsmonitor_batch *remainder =NULL ;intmax_t count =0 , duplicates =0 ; -kh_str_t *shown =NULL ; -int hash_ret; +struct strset shown =STRSET_INIT ;int do_trivial =0 ;int do_flush =0 ;int do_cookie =0 ;
@@ -882,14 +879,14 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,* so walk the batch list backwards from the current head back * to the batch (sequence number) they named. * -* We use khash to de-dup the list of pathnames. +* We use a strset to de-dup the list of pathnames. * * NEEDSWORK: each batch contains a list of interned strings, * so we only need to do pointer comparisons here to build the * hash table. Currently, we're still comparing the string * values. */ - shown =kh_init_str (); +strset_init_with_options (&shown,NULL ,0 );for (batch = batch_head; batch && batch->batch_seq_nr > requested_oldest_seq_nr; batch = batch->next) {
@@ -899,11 +896,9 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,const char *s = batch->interned_paths[k];size_t s_len; -if (kh_get_str (shown, s) !=kh_end (shown)) +if (!strset_add (&shown, s)) duplicates++;else { -kh_put_str (shown, s, &hash_ret); -trace_printf_key (&trace_fsmonitor,"send[%" PRIuMAX"]: %s" , count, s);
@@ -973,7 +968,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,trace2_data_intmax ("fsmonitor" , the_repository,"response/count/duplicates" , duplicates); cleanup: -kh_destroy_str (shown); +strset_clear (&shown);strbuf_release (&response_token);strbuf_release (&requested_token_id);strbuf_release (&payload);
Signature · 1 line
gitgitgadget
Quoted 8 lines · Paul Tarjan via GitGitGadget
This series implements the built-in fsmonitor daemon for Linux using the inotify API, bringing it to feature parity with the existing Windows and macOS implementations. It also fixes two memory leaks in the platform-independent daemon code and deduplicates the IPC and settings logic that is now shared between macOS and Linux.
Troubleshooting a Gentoo build failure of next has me pretty convinced this topic is in there already. Junio should probably check my math, but I think that means we want to see fixes on top of that base now (unless we are reverting this topic from next and queuing a new version?).
(The failure is a Gentoo-ism; we carry a patch that stops applying with this series. Not anything this project needs to worry about.)
Quoted 8 lines · Ben Knoble
Quoted 7 lines · Paul Tarjan via GitGitGadget
This series implements the built-in fsmonitor daemon for Linux using the inotify API, bringing it to feature parity with the existing Windows and macOS implementations. It also fixes two memory leaks in the platform-independent daemon code and deduplicates the IPC and settings logic that is now shared between macOS and Linux.
Quoted 8 lines
Troubleshooting a Gentoo build failure of next has me pretty convinced this topic is in there already. Junio should probably check my math, but I think that means we want to see fixes on top of that base now (unless we are reverting this topic from next and queuing a new version?).
(The failure is a Gentoo-ism; we carry a patch that stops applying with this series. Not anything this project needs to worry about.)
I reverted it yesterday out of 'next'. I do not recall I had a chance to push it out, but my intention is keep this round in 'seen' but out of 'next' until people are happy with it (which means folks who test diligently like you do would need to peek into 'seen' as well as 'next').
Quoted 20 lines · Junio C Hamano
Quoted 14 lines · Ben Knoble
Quoted 7 lines
Quoted 1 line · Paul Tarjan via GitGitGadget
This series implements the built-in fsmonitor daemon for Linux using the inotify API, bringing it to feature parity with the existing Windows and macOS implementations. It also fixes two memory leaks in the platform-independent daemon code and deduplicates the IPC and settings logic that is now shared between macOS and Linux.
Troubleshooting a Gentoo build failure of next has me pretty convinced this topic is in there already. Junio should probably check my math, but I think that means we want to see fixes on top of that base now (unless we are reverting this topic from next and queuing a new version?).
Quoted 8 lines
Quoted 2 lines
(The failure is a Gentoo-ism; we carry a patch that stops applying with this series. Not anything this project needs to worry about.)
I reverted it yesterday out of 'next'. I do not recall I had a chance to push it out, but my intention is keep this round in 'seen' but out of 'next' until people are happy with it (which means folks who test diligently like you do would need to peek into 'seen' as well as 'next').
Thanks. Hopefully I’ll get to both seen and next soon.
Quoted 8 lines · Ben Knoble
Quoted 7 lines · Paul Tarjan via GitGitGadget
This series implements the built-in fsmonitor daemon for Linux using the inotify API, bringing it to feature parity with the existing Windows and macOS implementations. It also fixes two memory leaks in the platform-independent daemon code and deduplicates the IPC and settings logic that is now shared between macOS and Linux.
Quoted 8 lines
Troubleshooting a Gentoo build failure of next has me pretty convinced this topic is in there already. Junio should probably check my math, but I think that means we want to see fixes on top of that base now (unless we are reverting this topic from next and queuing a new version?).
(The failure is a Gentoo-ism; we carry a patch that stops applying with this series. Not anything this project needs to worry about.)
So is there a verdict already, which this project may not need to worry about? This has been kept out of 'next' after getting reverted but if the breakage was due to Gentoo-ism whose workaround does not need to get upstreamed, and if there are no other reasons to block the topic, I am inclined to mark the topic for 'next'.
Thanks.
Quoted 20 lines · Junio C Hamano
Quoted 8 lines · Ben Knoble
Quoted 7 lines · Paul Tarjan via GitGitGadget
This series implements the built-in fsmonitor daemon for Linux using the inotify API, bringing it to feature parity with the existing Windows and macOS implementations. It also fixes two memory leaks in the platform-independent daemon code and deduplicates the IPC and settings logic that is now shared between macOS and Linux.
Quoted 8 lines
Troubleshooting a Gentoo build failure of next has me pretty convinced this topic is in there already. Junio should probably check my math, but I think that means we want to see fixes on top of that base now (unless we are reverting this topic from next and queuing a new version?).
(The failure is a Gentoo-ism; we carry a patch that stops applying with this series. Not anything this project needs to worry about.)
Quoted 7 lines
So is there a verdict already, which this project may not need to worry about? This has been kept out of 'next' after getting reverted but if the breakage was due to Gentoo-ism whose workaround does not need to get upstreamed, and if there are no other reasons to block the topic, I am inclined to mark the topic for 'next'.
Thanks.
I think Gentoo folks know what needs done (which I will try to spend some time on soon) to adjust the patches we carry. (That will not need to get upstreamed, no.)
I would certainly not keep this out of the tree only on behalf of Gentoo, unless there are other issues.
Signature · 1 line
D. Ben Knoble
Quoted 5 lines · Paul Tarjan via GitGitGadget
Changes since v14:
* Fix
stop_daemon_delete_repo:test_might_faildoesn't allowmaybe_timeout, replaced with { ... || :; } per SZEDER/Peff review, restoring broken &&-chain
I verified that this is the only change since the version this iteration replaces. Will queue.
Thanks.
Thanks for a quick turnaround, but it would be more efficient if you hunted all the leaks yourself, instead of getting a report for one issue and updating the patch to fix that one issue.
Here is what I am getting these:
$ make SANITIZE=address CC=clang && cd t && sh t7527-*.sh -i -v
Note that "-i" is to say "stop at the first one".
expecting success of 7527.12 'create some files':
test_when_finished clean_up_repo_and_stop_daemon &&
start_daemon --tf "$PWD/.git/trace" &&
create_files &&
test-tool fsmonitor-client query --token 0 &&
grep "^event: dir1/new$" .git/trace &&
grep "^event: dir2/new$" .git/trace &&
grep "^event: new$" .git/tracefsmonitor-daemon is watching '/home/gitster/w/git.git/t/trash directory.t7527-builtin-fsmonitor' builtin:0.1039108.20251230T123036.129805Z:0/event: dir1/new
event: dir1/new
event: dir2/new
event: dir2/new
event: new
event: new
HEAD is now at 1d1edcb initial
Removing dir1/new
Removing dir2/new
Removing new
not ok 12 - create some files
#
# test_when_finished clean_up_repo_and_stop_daemon &&
#
# start_daemon --tf "$PWD/.git/trace" &&
#
# create_files &&
#
# test-tool fsmonitor-client query --token 0 &&
#
# grep "^event: dir1/new$" .git/trace && # grep "^event: dir2/new$" .git/trace && # grep "^event: new$" .git/trace
#
1..12
=================================================================
==git==1039073==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 40 byte(s) in 1 object(s) allocated from:
#0 0x55c18d8d4042 in calloc (git+0x8c042) (BuildId: 4097db008a82663ae0b3398128a7ab4e09bbdd21)
#1 0x55c18dc10f14 in xcalloc wrapper.c:154:8
#2 0x55c18d945f72 in kh_init_str builtin/fsmonitor--daemon.c:656:1
#3 0x55c18d945828 in do_handle_client builtin/fsmonitor--daemon.c:871:10
#4 0x55c18d945191 in handle_client builtin/fsmonitor--daemon.c:987:11
#5 0x55c18dc283e2 in worker_thread__do_io compat/simple-ipc/ipc-unix-socket.c:532:9
#6 0x55c18dc27a7f in worker_thread_proc compat/simple-ipc/ipc-unix-socket.c:606:9
#7 0x55c18d8d64f4 in void* ThreadStartFunc<false>(void*) lsan_interceptors.cpp.o
#8 0x7fe358257b7a in start_thread nptl/pthread_create.c:448:8
#9 0x7fe3582d57b7 in __GI___clone3 misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:78