fsmonitor: Implement fsmonitor for Linux
Subsystems (2)
cover fsmonitor: Implement fsmonitor for Linux
Goal is to deliver fsmonitor for Linux that is on par with fsmonitor for Windows and Mac OS.
This patch set builds upon previous work for done for Windows and Mac OS (first 6 patches) to implement a fsmonitor back-end for Linux based on the Linux inotify API. inotify differs significantly from the equivalent Windows and Mac OS APIs in that a watch must be registered for every directory of interest (rather than a singular watch at the root of the directory tree) and special care must be taken to handle directory renames correctly.
More information about inotify:
https://man7.org/linux/man-pages/man7/inotify.7.html
Eric DeCosta (12):
fsmonitor: refactor filesystem checks to common interface
fsmonitor: relocate socket file if .git directory is remote
fsmonitor: avoid socket location check if using hook
fsmonitor: deal with synthetic firmlinks on macOS
fsmonitor: check for compatability before communicating with fsmonitor
fsmonitor: add documentation for allowRemote and socketDir options
fsmonitor: prepare to share code between Mac OS and Linux
fsmonitor: determine if filesystem is local or remote
fsmonitor: implement filesystem change listener for Linux
fsmonitor: enable fsmonitor for Linux
fsmonitor: test updates
fsmonitor: update doc for Linux
Documentation/config.txt | 2 +
Documentation/config/fsmonitor--daemon.txt | 11 +
Documentation/git-fsmonitor--daemon.txt | 45 +-
Makefile | 6 +-
builtin/fsmonitor--daemon.c | 11 +-
...{fsm-health-darwin.c => fsm-health-unix.c} | 0
compat/fsmonitor/fsm-ipc-unix.c | 52 ++
compat/fsmonitor/fsm-ipc-win32.c | 9 +
compat/fsmonitor/fsm-listen-darwin.c | 14 +-
compat/fsmonitor/fsm-listen-linux.c | 664 ++++++++++++++++++
compat/fsmonitor/fsm-path-utils-darwin.c | 135 ++++
compat/fsmonitor/fsm-path-utils-linux.c | 163 +++++
compat/fsmonitor/fsm-path-utils-win32.c | 145 ++++
compat/fsmonitor/fsm-settings-darwin.c | 89 ---
compat/fsmonitor/fsm-settings-unix.c | 62 ++
compat/fsmonitor/fsm-settings-win32.c | 174 +----
config.mak.uname | 13 +
contrib/buildsystems/CMakeLists.txt | 19 +-
fsmonitor--daemon.h | 3 +
fsmonitor-ipc.c | 18 +-
fsmonitor-ipc.h | 4 +-
fsmonitor-path-utils.h | 60 ++
fsmonitor-settings.c | 68 +-
fsmonitor-settings.h | 4 +-
fsmonitor.c | 7 +
t/t7527-builtin-fsmonitor.sh | 58 +-
26 files changed, 1531 insertions(+), 305 deletions(-)
create mode 100644 Documentation/config/fsmonitor--daemon.txt
rename compat/fsmonitor/{fsm-health-darwin.c => fsm-health-unix.c} (100%)
create mode 100644 compat/fsmonitor/fsm-ipc-unix.c
create mode 100644 compat/fsmonitor/fsm-ipc-win32.c
create mode 100644 compat/fsmonitor/fsm-listen-linux.c
create mode 100644 compat/fsmonitor/fsm-path-utils-darwin.c
create mode 100644 compat/fsmonitor/fsm-path-utils-linux.c
create mode 100644 compat/fsmonitor/fsm-path-utils-win32.c
delete mode 100644 compat/fsmonitor/fsm-settings-darwin.c
create mode 100644 compat/fsmonitor/fsm-settings-unix.c
create mode 100644 fsmonitor-path-utils.hbase-commit: 3dcec76d9df911ed8321007b1d197c1a206dc164
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1352%2Fedecosta-mw%2Ffsmonitor_linux-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1352/edecosta-mw/fsmonitor_linux-v1
Pull-Request: https://github.com/git/git/pull/1352
Signature · 1 line
gitgitgadget
PATCH 2/12 fsmonitor: relocate socket file if .git directory is remote
From: Eric DeCosta <[email protected]>
If the .git directory is on a remote filesystem, create the socket file in 'fsmonitor.socketDir' if it is defined, else create it in $HOME.
--- Makefile | 1 + builtin/fsmonitor--daemon.c | 3 +- compat/fsmonitor/fsm-ipc-darwin.c | 52 ++++++++++++++++++++++++++ compat/fsmonitor/fsm-ipc-win32.c | 9 +++++ compat/fsmonitor/fsm-settings-darwin.c | 2 +- contrib/buildsystems/CMakeLists.txt | 2 + fsmonitor-ipc.c | 18 ++++----- fsmonitor-ipc.h | 4 +- 8 files changed, 78 insertions(+), 13 deletions(-) create mode 100644 compat/fsmonitor/fsm-ipc-darwin.c create mode 100644 compat/fsmonitor/fsm-ipc-win32.c
Makefile +1 -0
@@ -2039,6 +2039,7 @@ ifdef FSMONITOR_DAEMON_BACKEND COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o + COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_BACKEND).o endif ifdef FSMONITOR_OS_SETTINGS
builtin/fsmonitor--daemon.c +2 -1
@@ -1343,7 +1343,8 @@ static int fsmonitor_run_daemon(void)* directory.) */ strbuf_init (&state.path_ipc,0 ); -strbuf_addstr (&state.path_ipc,absolute_path (fsmonitor_ipc__get_path ())); +strbuf_addstr (&state.path_ipc, +absolute_path (fsmonitor_ipc__get_path (the_repository)));/* * Confirm that we can create platform-specific resources for the
compat/fsmonitor/fsm-ipc-darwin.c new file +52 -0
52 lines not on this page · raw message
compat/fsmonitor/fsm-ipc-win32.c new file +9 -0
@@ -0,0 +1,9 @@ +#include "config.h" +#include "fsmonitor-ipc.h" + +const char *fsmonitor_ipc__get_path (struct repository *r) { +static char *ret; +if (!ret) + ret =git_pathdup ("fsmonitor--daemon.ipc" ); +return ret; +}
compat/fsmonitor/fsm-settings-darwin.c +1 -1
@@ -26,7 +26,7 @@static enum fsmonitor_reason check_uds_volume (struct repository *r) {struct fs_info fs; -const char *ipc_path =fsmonitor_ipc__get_path (); +const char *ipc_path =fsmonitor_ipc__get_path (r);struct strbuf path =STRBUF_INIT ;strbuf_add (&path, ipc_path,strlen (ipc_path));
contrib/buildsystems/CMakeLists.txt +2 -0
@@ -308,6 +308,7 @@ if(SUPPORTS_SIMPLE_IPC) add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c) list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-win32.c) list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-win32.c) add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
@@ -316,6 +317,7 @@ if(SUPPORTS_SIMPLE_IPC) add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c) list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-darwin.c) list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-darwin.c) add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
fsmonitor-ipc.c +8 -10
@@ -18,7 +18,7 @@ int fsmonitor_ipc__is_supported(void)return 0 ; } -const char *fsmonitor_ipc__get_path (void ) +const char *fsmonitor_ipc__get_path (struct repository *r) {return NULL ; }
@@ -47,11 +47,9 @@ int fsmonitor_ipc__is_supported(void)return 1 ; } -GIT_PATH_FUNC(fsmonitor_ipc__get_path, "fsmonitor--daemon.ipc") -enum ipc_active_state fsmonitor_ipc__get_state (void ) { -return ipc_get_active_state (fsmonitor_ipc__get_path ()); +return ipc_get_active_state (fsmonitor_ipc__get_path (the_repository)); }static int spawn_daemon (void )
@@ -81,8 +79,8 @@ int fsmonitor_ipc__send_query(const char *since_token,trace2_data_string ("fsm_client" ,NULL ,"query/command" , tok); try_again: - state =ipc_client_try_connect (fsmonitor_ipc__get_path (), &options, - &connection); + state =ipc_client_try_connect (fsmonitor_ipc__get_path (the_repository), + &options, &connection);switch (state) {case IPC_STATE__LISTENING :
@@ -117,13 +115,13 @@ try_again:case IPC_STATE__INVALID_PATH : ret =error (_ ("fsmonitor_ipc__send_query: invalid path '%s'" ), -fsmonitor_ipc__get_path ()); +fsmonitor_ipc__get_path (the_repository)); goto done;case IPC_STATE__OTHER_ERROR :default : ret =error (_ ("fsmonitor_ipc__send_query: unspecified error on '%s'" ), -fsmonitor_ipc__get_path ()); +fsmonitor_ipc__get_path (the_repository)); goto done; }
@@ -149,8 +147,8 @@ int fsmonitor_ipc__send_command(const char *command, options.wait_if_busy =1 ; options.wait_if_not_found =0 ; - state =ipc_client_try_connect (fsmonitor_ipc__get_path (), &options, - &connection); + state =ipc_client_try_connect (fsmonitor_ipc__get_path (the_repository), + &options, &connection);if (state !=IPC_STATE__LISTENING ) {die (_ ("fsmonitor--daemon is not running" ));return -1 ;
fsmonitor-ipc.h +3 -1
@@ -3,6 +3,8 @@#include "simple-ipc.h" +struct repository ; +/* * Returns true if built-in file system monitor daemon is defined * for this platform.
@@ -16,7 +18,7 @@ int fsmonitor_ipc__is_supported(void);* * Returns NULL if the daemon is not supported on this platform. */ -const char *fsmonitor_ipc__get_path (void ); +const char *fsmonitor_ipc__get_path (struct repository *r);/* * Try to determine whether there is a `git-fsmonitor--daemon` process
Signature · 1 line
gitgitgadget
PATCH 4/12 fsmonitor: deal with synthetic firmlinks on macOS
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
Starting with macOS 10.15 (Catalina), Apple introduced a new feature called 'firmlinks' in order to separate the boot volume into two volumes, one read-only and one writable but still present them to the user as a single volume. Along with this change, Apple removed the ability to create symlinks in the root directory and replaced them with 'synthetic firmlinks'. See 'man synthetic.conf'
When FSEevents reports the path of changed files, if the path involves a synthetic firmlink, the path is reported from the point of the synthetic firmlink and not the real path. For example:
Real path:
/System/Volumes/Data/network/working/directory/foo.txt
Synthetic firmlink:
/network -> /System/Volumes/Data/network
FSEvents path:
/network/working/directory/foo.txt
This causes the FSEvents path to not match against the worktree directory.
There are several ways in which synthetic firmlinks can be created:
they can be defined in /etc/synthetic.conf, the automounter can create them, and there may be other means. Simply reading /etc/synthetic.conf is insufficient. No matter what process creates synthetic firmlinks, they all get created in the root directory.
Therefore, in order to deal with synthetic firmlinks, the root directory is scanned and the first possible synthetic firmink that, when resolved, is a prefix of the worktree is used to map FSEvents paths to worktree paths.
--- builtin/fsmonitor--daemon.c | 8 +++ compat/fsmonitor/fsm-listen-darwin.c | 14 +++- compat/fsmonitor/fsm-path-utils-darwin.c | 92 ++++++++++++++++++++++++ compat/fsmonitor/fsm-path-utils-win32.c | 17 +++++ fsmonitor--daemon.h | 3 + fsmonitor-path-utils.h | 36 +++++++++- 6 files changed, 167 insertions(+), 3 deletions(-)
builtin/fsmonitor--daemon.c +8 -0
@@ -3,6 +3,7 @@#include "parse-options.h" #include "fsmonitor.h" #include "fsmonitor-ipc.h" +#include "fsmonitor-path-utils.h" #include "compat/fsmonitor/fsm-health.h" #include "compat/fsmonitor/fsm-listen.h" #include "fsmonitor--daemon.h"
@@ -1282,6 +1283,11 @@ static int fsmonitor_run_daemon(void)strbuf_addstr (&state.path_worktree_watch,absolute_path (get_git_work_tree ())); state.nr_paths_watching =1 ; +strbuf_init (&state.alias.alias,0 ); +strbuf_init (&state.alias.points_to,0 ); +if ((err =fsmonitor__get_alias (state.path_worktree_watch.buf, &state.alias))) + goto done; +/* * We create and delete cookie files somewhere inside the .git * directory to help us keep sync with the file system. If
@@ -1391,6 +1397,8 @@ done:strbuf_release (&state.path_gitdir_watch);strbuf_release (&state.path_cookie_prefix);strbuf_release (&state.path_ipc); +strbuf_release (&state.alias.alias); +strbuf_release (&state.alias.points_to);return err; }
compat/fsmonitor/fsm-listen-darwin.c +12 -2
@@ -26,6 +26,7 @@#include "fsmonitor.h" #include "fsm-listen.h" #include "fsmonitor--daemon.h" +#include "fsmonitor-path-utils.h" struct fsm_listen_data {
@@ -198,8 +199,9 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,struct string_list cookie_list =STRING_LIST_INIT_DUP ;const char *path_k;const char *slash; -int k; +char *resolved =NULL ;struct strbuf tmp =STRBUF_INIT ; +int k;/* * Build a list of all filesystem changes into a private/local
@@ -209,7 +211,12 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,/* * On Mac, we receive an array of absolute paths. */ - path_k = paths[k]; +free (resolved); + resolved =fsmonitor__resolve_alias (paths[k], &state->alias); +if (resolved) + path_k = resolved; +else + path_k = paths[k];/* * If you want to debug FSEvents, log them to GIT_TRACE_FSMONITOR.
@@ -238,6 +245,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,fsmonitor_force_resync (state);fsmonitor_batch__free_list (batch);string_list_clear (&cookie_list,0 ); + batch =NULL ;/* * We assume that any events that we received
@@ -360,12 +368,14 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef, } } +free (resolved);fsmonitor_publish (state, batch, &cookie_list);string_list_clear (&cookie_list,0 );strbuf_release (&tmp);return ; force_shutdown: +free (resolved);fsmonitor_batch__free_list (batch);string_list_clear (&cookie_list,0 );
compat/fsmonitor/fsm-path-utils-darwin.c +92 -0
100 lines not on this page · raw message
compat/fsmonitor/fsm-path-utils-win32.c +17 -0
@@ -126,3 +126,20 @@ int fsmonitor__is_fs_remote(const char *path)return -1 ;return fs.is_remote; } + +/* +* No-op for now. +*/ +int fsmonitor__get_alias (const char *path,struct alias_info *info) +{ +return 0 ; +} + +/* +* No-op for now. +*/ +char *fsmonitor__resolve_alias (const char *path, +const struct alias_info *info) +{ +return NULL ; +}
fsmonitor--daemon.h +3 -0
@@ -8,6 +8,7 @@#include "run-command.h" #include "simple-ipc.h" #include "thread-utils.h" +#include "fsmonitor-path-utils.h" struct fsmonitor_batch ;struct fsmonitor_token_data ;
@@ -43,6 +44,7 @@ struct fsmonitor_daemon_state {struct strbuf path_worktree_watch;struct strbuf path_gitdir_watch; +struct alias_info alias;int nr_paths_watching;struct fsmonitor_token_data *current_token_data;
@@ -59,6 +61,7 @@ struct fsmonitor_daemon_state {struct ipc_server_data *ipc_server_data;struct strbuf path_ipc; + };/*
fsmonitor-path-utils.h +35 -1
@@ -1,13 +1,21 @@#ifndef FSM_PATH_UTILS_H #define FSM_PATH_UTILS_H +#include "strbuf.h" + +struct alias_info +{ +struct strbuf alias; +struct strbuf points_to; +}; +struct fs_info {int is_remote;char *typename; };/* -* Get some basic filesystem informtion for the given path +* Get some basic filesystem information for the given path * * The caller owns the storage that is occupied by fs_info and * is responsible for releasing it.
@@ -23,4 +31,30 @@ int fsmonitor__get_fs_info(const char *path, struct fs_info *fs_info);*/ int fsmonitor__is_fs_remote (const char *path); +/* +* Get the alias in given path, if any. +* +* Sets alias to the first alias that matches any part of the path. +* +* If an alias is found, info.alias and info.points_to are set to the +* found mapping. +* +* Returns -1 on error, 0 otherwise. +* +* The caller owns the storage that is occupied by info.alias and +* info.points_to and is responsible for releasing it. +*/ +int fsmonitor__get_alias (const char *path,struct alias_info *info); + +/* +* Resolve the path against the given alias. +* +* Returns the resolved path if there is one, NULL otherwise. +* +* The caller owns the storage that the returned string occupies and +* is responsible for releasing it. +*/ +char *fsmonitor__resolve_alias (const char *path, +const struct alias_info *info); +#endif
Signature · 1 line
gitgitgadget
PATCH 3/12 fsmonitor: avoid socket location check if using hook
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
If monitoring is done via fsmonitor hook rather than IPC there is no need to check if the location of the Unix Domain socket (UDS) file is on a remote filesystem.
--- compat/fsmonitor/fsm-settings-darwin.c | 10 ++++++---- compat/fsmonitor/fsm-settings-win32.c | 2 +- fsmonitor-settings.c | 8 ++++---- fsmonitor-settings.h | 2 +- 4 files changed, 12 insertions(+), 10 deletions(-)
compat/fsmonitor/fsm-settings-darwin.c +6 -4
@@ -48,13 +48,15 @@ static enum fsmonitor_reason check_uds_volume(struct repository *r)return FSMONITOR_REASON_OK ; } -enum fsmonitor_reason fsm_os__incompatible (struct repository *r) +enum fsmonitor_reason fsm_os__incompatible (struct repository *r,int ipc) {enum fsmonitor_reason reason; - reason =check_uds_volume (r); -if (reason !=FSMONITOR_REASON_OK ) -return reason; +if (ipc) { + reason =check_uds_volume (r); +if (reason !=FSMONITOR_REASON_OK ) +return reason; + }return FSMONITOR_REASON_OK ; }
compat/fsmonitor/fsm-settings-win32.c +1 -1
@@ -25,7 +25,7 @@ static enum fsmonitor_reason check_vfs4git(struct repository *r)return FSMONITOR_REASON_OK ; } -enum fsmonitor_reason fsm_os__incompatible (struct repository *r) +enum fsmonitor_reason fsm_os__incompatible (struct repository *r,int ipc) {enum fsmonitor_reason reason;
fsmonitor-settings.c +4 -4
@@ -60,7 +60,7 @@ static enum fsmonitor_reason check_remote(struct repository *r) }#endif -static enum fsmonitor_reason check_for_incompatible (struct repository *r) +static enum fsmonitor_reason check_for_incompatible (struct repository *r,int ipc) {if (!r->worktree) {/*
@@ -77,7 +77,7 @@ static enum fsmonitor_reason check_for_incompatible(struct repository *r) reason =check_remote (r);if (reason !=FSMONITOR_REASON_OK )return reason; - reason =fsm_os__incompatible (r); + reason =fsm_os__incompatible (r, ipc);if (reason !=FSMONITOR_REASON_OK )return reason; }
@@ -162,7 +162,7 @@ const char *fsm_settings__get_hook_path(struct repository *r)void fsm_settings__set_ipc (struct repository *r) { -enum fsmonitor_reason reason =check_for_incompatible (r); +enum fsmonitor_reason reason =check_for_incompatible (r,1 );if (reason !=FSMONITOR_REASON_OK ) {fsm_settings__set_incompatible (r, reason);
@@ -185,7 +185,7 @@ void fsm_settings__set_ipc(struct repository *r)void fsm_settings__set_hook (struct repository *r,const char *path) { -enum fsmonitor_reason reason =check_for_incompatible (r); +enum fsmonitor_reason reason =check_for_incompatible (r,0 );if (reason !=FSMONITOR_REASON_OK ) {fsm_settings__set_incompatible (r, reason);
fsmonitor-settings.h +1 -1
@@ -48,7 +48,7 @@ struct fsmonitor_settings;* fsm_os__* routines should considered private to fsm_settings__ * routines. */ -enum fsmonitor_reason fsm_os__incompatible (struct repository *r); +enum fsmonitor_reason fsm_os__incompatible (struct repository *r,int ipc);#endif /* HAVE_FSMONITOR_OS_SETTINGS */ #endif /* FSMONITOR_SETTINGS_H */
Signature · 1 line
gitgitgadget
PATCH 5/12 fsmonitor: check for compatability before communicating with fsmonitor
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
If fsmonitor is not in a compatible state, warn with an appropriate message.
--- fsmonitor-settings.c | 10 +++++++--- fsmonitor-settings.h | 2 +- fsmonitor.c | 7 +++++++ 3 files changed, 15 insertions(+), 4 deletions(-)
fsmonitor-settings.c +7 -3
@@ -1,6 +1,7 @@#include "cache.h" #include "config.h" #include "repository.h" +#include "fsmonitor-ipc.h" #include "fsmonitor-settings.h" #include "fsmonitor-path-utils.h"
@@ -242,10 +243,11 @@ enum fsmonitor_reason fsm_settings__get_reason(struct repository *r)return r->settings.fsmonitor->reason; } -char *fsm_settings__get_incompatible_msg (const struct repository *r, +char *fsm_settings__get_incompatible_msg (struct repository *r,enum fsmonitor_reason reason) {struct strbuf msg =STRBUF_INIT ; +const char *socket_dir;switch (reason) {case FSMONITOR_REASON_UNTESTED :
@@ -281,9 +283,11 @@ char *fsm_settings__get_incompatible_msg(const struct repository *r, goto done;case FSMONITOR_REASON_NOSOCKETS : + socket_dir =dirname ((char *)fsmonitor_ipc__get_path (r));strbuf_addf (&msg, -_ ("repository '%s' is incompatible with fsmonitor due to lack of Unix sockets" ), - r->worktree); +_ ("socket directory '%s' is incompatible with fsmonitor due" +" to lack of Unix sockets support" ), + socket_dir); goto done; }
fsmonitor-settings.h +1 -1
@@ -33,7 +33,7 @@ enum fsmonitor_mode fsm_settings__get_mode(struct repository *r);const char *fsm_settings__get_hook_path (struct repository *r);enum fsmonitor_reason fsm_settings__get_reason (struct repository *r); -char *fsm_settings__get_incompatible_msg (const struct repository *r, +char *fsm_settings__get_incompatible_msg (struct repository *r,enum fsmonitor_reason reason);struct fsmonitor_settings ;
fsmonitor.c +7 -0
@@ -295,6 +295,7 @@ static int fsmonitor_force_update_threshold = 100;void refresh_fsmonitor (struct index_state *istate) { +static int warn_once =0 ;struct strbuf query_result =STRBUF_INIT ;int query_success =0 , hook_version =-1 ;size_t bol =0 ;/* beginning of line */
@@ -305,6 +306,12 @@ void refresh_fsmonitor(struct index_state *istate)int is_trivial =0 ;struct repository *r = istate->repo ? istate->repo : the_repository;enum fsmonitor_mode fsm_mode =fsm_settings__get_mode (r); +enum fsmonitor_reason reason =fsm_settings__get_reason (r); + +if (!warn_once && reason >FSMONITOR_REASON_OK ) { + warn_once =1 ; +warning ("%s" ,fsm_settings__get_incompatible_msg (r, reason)); + }if (fsm_mode <=FSMONITOR_MODE_DISABLED || istate->fsmonitor_has_run_once)
Signature · 1 line
gitgitgadget
PATCH 1/12 fsmonitor: refactor filesystem checks to common interface
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
Provide a common interface for getting basic filesystem information including filesystem type and whether the filesystem is remote.
Refactor existing code for getting basic filesystem info and detecting remote file systems to the new interface.
Refactor filesystem checks to leverage new interface. For macOS, error-out if the Unix Domain socket (UDS) file is on a remote filesystem.
--- Makefile | 1 + compat/fsmonitor/fsm-path-utils-darwin.c | 43 ++++++ compat/fsmonitor/fsm-path-utils-win32.c | 128 +++++++++++++++++ compat/fsmonitor/fsm-settings-darwin.c | 69 +++------ compat/fsmonitor/fsm-settings-win32.c | 172 +---------------------- contrib/buildsystems/CMakeLists.txt | 2 + fsmonitor-path-utils.h | 26 ++++ fsmonitor-settings.c | 50 +++++++ 8 files changed, 272 insertions(+), 219 deletions(-) create mode 100644 compat/fsmonitor/fsm-path-utils-darwin.c create mode 100644 compat/fsmonitor/fsm-path-utils-win32.c create mode 100644 fsmonitor-path-utils.h
Makefile +1 -0
@@ -2044,6 +2044,7 @@ endif ifdef FSMONITOR_OS_SETTINGS COMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o + COMPAT_OBJS += compat/fsmonitor/fsm-path-utils-$(FSMONITOR_OS_SETTINGS).o endif ifeq ($(TCLTK_PATH),)
compat/fsmonitor/fsm-path-utils-darwin.c new file +43 -0
43 lines not on this page · raw message
compat/fsmonitor/fsm-path-utils-win32.c new file +128 -0
128 lines not on this page · raw message
compat/fsmonitor/fsm-settings-darwin.c +20 -49
101 lines not on this page · raw message
compat/fsmonitor/fsm-settings-win32.c +2 -170
190 lines not on this page · raw message
contrib/buildsystems/CMakeLists.txt +2 -0
@@ -308,6 +308,7 @@ if(SUPPORTS_SIMPLE_IPC) add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c) list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-win32.c) add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c)
@@ -315,6 +316,7 @@ if(SUPPORTS_SIMPLE_IPC) add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c) list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-darwin.c) add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)
fsmonitor-path-utils.h new file +26 -0
@@ -0,0 +1,26 @@ +#ifndef FSM_PATH_UTILS_H +#define FSM_PATH_UTILS_H + +struct fs_info { +int is_remote; +char *typename; +}; + +/* +* Get some basic filesystem informtion for the given path +* +* The caller owns the storage that is occupied by fs_info and +* is responsible for releasing it. +* +* Returns -1 on error, zero otherwise. +*/ +int fsmonitor__get_fs_info (const char *path,struct fs_info *fs_info); + +/* +* Determines if the filesystem that path resides on is remote. +* +* Returns -1 on error, 0 if not remote, 1 if remote. +*/ +int fsmonitor__is_fs_remote (const char *path); + +#endif
fsmonitor-settings.c +50 -0
68 lines not on this page · raw message
Signature · 1 line
gitgitgadget
PATCH 6/12 fsmonitor: add documentation for allowRemote and socketDir options
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
Add documentation for 'fsmonitor.allowRemote' and 'fsmonitor.socketDir'. Call-out experimental nature of 'fsmonitor.allowRemote' and limited filesystem support for 'fsmonitor.socketDir'.
--- Documentation/config.txt | 2 ++ Documentation/config/fsmonitor--daemon.txt | 11 +++++++ Documentation/git-fsmonitor--daemon.txt | 37 ++++++++++++++++++++-- 3 files changed, 47 insertions(+), 3 deletions(-) create mode 100644 Documentation/config/fsmonitor--daemon.txt
Documentation/config.txt +2 -0
@@ -423,6 +423,8 @@ include::config/filter.txt[] include::config/fsck.txt[] +include::config/fsmonitor--daemon.txt[] + include::config/gc.txt[] include::config/gitcvs.txt[]
Documentation/config/fsmonitor--daemon.txt new file +11 -0
@@ -0,0 +1,11 @@ +fsmonitor.allowRemote:: + By default, the fsmonitor daemon refuses to work against network-mounted + repositories. Setting `fsmonitor.allowRemote` to `true` overrides this + behavior. Only respected when `core.fsmonitor` is set to `true`. + +fsmonitor.socketDir:: + This Mac OS-specific option, if set, specifies the directory in + which to create the Unix domain socket used for communication + between the fsmonitor daemon and various Git commands. The directory must + reside on a native Mac OS filesystem. Only respected when `core.fsmonitor` + is set to `true`.
Documentation/git-fsmonitor--daemon.txt +34 -3
@@ -3,7 +3,7 @@ git-fsmonitor{litdd}daemon(1) NAME ---- -git-fsmonitor--daemon - A Built-in File System Monitor +git-fsmonitor--daemon - A Built-in Filesystem Monitor SYNOPSIS --------
@@ -17,7 +17,7 @@ DESCRIPTION ----------- A daemon to watch the working directory for file and directory -changes using platform-specific file system notification facilities. +changes using platform-specific filesystem notification facilities. This daemon communicates directly with commands like `git status` using the link:technical/api-simple-ipc.html[simple IPC] interface
@@ -63,13 +63,44 @@ CAVEATS ------- The fsmonitor daemon does not currently know about submodules and does -not know to filter out file system events that happen within a +not know to filter out filesystem events that happen within a submodule. If fsmonitor daemon is watching a super repo and a file is modified within the working directory of a submodule, it will report the change (as happening against the super repo). However, the client will properly ignore these extra events, so performance may be affected but it will not cause an incorrect result. +By default, the fsmonitor daemon refuses to work against network-mounted +repositories; this may be overridden by setting `fsmonitor.allowRemote` to +`true`. Note, however, that the fsmonitor daemon is not guaranteed to work +correctly with all network-mounted repositories and such use is considered +experimental. + +On Mac OS, the inter-process communication (IPC) between various Git +commands and the fsmonitor daemon is done via a Unix domain socket (UDS) -- a +special type of file -- which is supported by native Mac OS filesystems, +but not on network-mounted filesystems, NTFS, or FAT32. Other filesystems +may or may not have the needed support; the fsmonitor daemon is not guaranteed +to work with these filesystems and such use is considered experimental. + +By default, the socket is created in the `.git` directory, however, if the +`.git` directory is on a network-mounted filesystem, it will be instead be +created at `$HOME/.git-fsmonitor-*` unless `$HOME` itself is on a +network-mounted filesystem in which case you must set the configuration +variable `fsmonitor.socketDir` to the path of a directory on a Mac OS native +filesystem in which to create the socket file. + +If none of the above directories (`.git`, `$HOME`, or `fsmonitor.socketDir`) +is on a native Mac OS file filesystem the fsmonitor daemon will report an +error that will cause the daemon and the currently running command to exit. + +CONFIGURATION +------------- + +include::includes/cmd-config-section-all.txt[] + +include::config/fsmonitor--daemon.txt[] + GIT --- Part of the linkgit:git[1] suite
Signature · 1 line
gitgitgadget
PATCH 7/12 fsmonitor: prepare to share code between Mac OS and Linux
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
Linux and Mac OS can share some of the code originally developed for Mac OS. Rename the shared code from compat/fsmonitor/fsm-*-dawrin.c to compat/fsmonitor/fsm-*-unix.c
Update the build to enable sharing of the fsm-*-unix.c files.
Minor update to compat/fsmonitor/fsm-ipc-unix.c to make it cross-platform.
---
Makefile | 6 +++---
.../fsmonitor/{fsm-health-darwin.c => fsm-health-unix.c} | 0
compat/fsmonitor/{fsm-ipc-darwin.c => fsm-ipc-unix.c} | 8 ++++----
.../{fsm-settings-darwin.c => fsm-settings-unix.c} | 0
config.mak.uname | 4 ++++
contrib/buildsystems/CMakeLists.txt | 6 +++---
6 files changed, 14 insertions(+), 10 deletions(-)
rename compat/fsmonitor/{fsm-health-darwin.c => fsm-health-unix.c} (100%)
rename compat/fsmonitor/{fsm-ipc-darwin.c => fsm-ipc-unix.c} (89%)
rename compat/fsmonitor/{fsm-settings-darwin.c => fsm-settings-unix.c} (100%)Makefile +3 -3
@@ -2038,13 +2038,13 @@ endif ifdef FSMONITOR_DAEMON_BACKEND COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o - COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o - COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_BACKEND).o + COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_COMMON).o + COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_COMMON).o endif ifdef FSMONITOR_OS_SETTINGS COMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS - COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o + COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_DAEMON_COMMON).o COMPAT_OBJS += compat/fsmonitor/fsm-path-utils-$(FSMONITOR_OS_SETTINGS).o endif
compat/fsmonitor/fsm-health-unix.c renamed from compat/fsmonitor/fsm-health-darwin.c +0 -0
compat/fsmonitor/fsm-ipc-unix.c renamed from compat/fsmonitor/fsm-ipc-darwin.c +4 -4
@@ -10,7 +10,7 @@ static GIT_PATH_FUNC(fsmonitor_ipc__get_default_path, "fsmonitor--daemon.ipc")const char *fsmonitor_ipc__get_path (struct repository *r) {static const char *ipc_path =NULL ; -SHA_CTX sha1ctx; +git_SHA_CTX sha1ctx;char *sock_dir =NULL ;struct strbuf ipc_file =STRBUF_INIT ;unsigned char hash[SHA_DIGEST_LENGTH ];
@@ -28,9 +28,9 @@ const char *fsmonitor_ipc__get_path(struct repository *r)return ipc_path; } -SHA1_Init (&sha1ctx); -SHA1_Update (&sha1ctx, r->worktree,strlen (r->worktree)); -SHA1_Final (hash, &sha1ctx); +git_SHA1_Init (&sha1ctx); +git_SHA1_Update (&sha1ctx, r->worktree,strlen (r->worktree)); +git_SHA1_Final (hash, &sha1ctx);repo_config_get_string (r,"fsmonitor.socketdir" , &sock_dir);
compat/fsmonitor/fsm-settings-unix.c renamed from compat/fsmonitor/fsm-settings-darwin.c +0 -0
config.mak.uname +4 -0
@@ -68,6 +68,7 @@ ifeq ($(uname_S),Linux) ifneq ($(findstring .el7.,$(uname_R)),) BASIC_CFLAGS += -std=c99 endif + endif ifeq ($(uname_S),GNU/kFreeBSD) HAVE_ALLOCA_H = YesPlease
@@ -165,6 +166,7 @@ ifeq ($(uname_S),Darwin) ifndef NO_UNIX_SOCKETS FSMONITOR_DAEMON_BACKEND = darwin FSMONITOR_OS_SETTINGS = darwin + FSMONITOR_DAEMON_COMMON = unix endif endif
@@ -453,6 +455,7 @@ ifeq ($(uname_S),Windows) # support it. FSMONITOR_DAEMON_BACKEND = win32 FSMONITOR_OS_SETTINGS = win32 + FSMONITOR_DAEMON_COMMON = win32 NO_SVN_TESTS = YesPlease RUNTIME_PREFIX = YesPlease
@@ -645,6 +648,7 @@ ifeq ($(uname_S),MINGW) # support it. FSMONITOR_DAEMON_BACKEND = win32 FSMONITOR_OS_SETTINGS = win32 + FSMONITOR_DAEMON_COMMON = win32 RUNTIME_PREFIX = YesPlease HAVE_WPGMPTR = YesWeDo
contrib/buildsystems/CMakeLists.txt +3 -3
@@ -315,13 +315,13 @@ if(SUPPORTS_SIMPLE_IPC) list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c) elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-unix.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-unix.c) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-darwin.c) list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-darwin.c) add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-unix.c) endif() endif()
Signature · 1 line
gitgitgadget
PATCH 8/12 fsmonitor: determine if filesystem is local or remote
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
Compare the given path to the mounted filesystems. Find the mount that is the longest prefix of the path (if any) and determine if that mount is on a local or remote filesystem.
--- compat/fsmonitor/fsm-path-utils-linux.c | 163 ++++++++++++++++++++++++ 1 file changed, 163 insertions(+) create mode 100644 compat/fsmonitor/fsm-path-utils-linux.c
compat/fsmonitor/fsm-path-utils-linux.c new file +163 -0
@@ -0,0 +1,163 @@ +#include "fsmonitor.h" +#include "fsmonitor-path-utils.h" +#include <errno.h> +#include <mntent.h> +#include <sys/mount.h> +#include <sys/statvfs.h> + +/* +* https://github.com/coreutils/gnulib/blob/master/lib/mountlist.c +*/ +#ifndef ME_REMOTE +/* A file system is "remote" if its Fs_name contains a ':' +or if (it is of type (smbfs or cifs) and its Fs_name starts with '//') +or if it is of any other of the listed types +or Fs_name is equal to "-hosts" (used by autofs to mount remote fs). +"VM" file systems like prl_fs or vboxsf are not considered remote here. */ +# define ME_REMOTE (Fs_name, Fs_type) \ + (strchr (Fs_name, ':') != NULL \ + || ((Fs_name)[0] == '/' \ + && (Fs_name)[1] == '/' \ + && (strcmp (Fs_type, "smbfs") == 0 \ + || strcmp (Fs_type, "smb3") == 0 \ + || strcmp (Fs_type, "cifs") == 0)) \ + || strcmp (Fs_type, "acfs") == 0 \ + || strcmp (Fs_type, "afs") == 0 \ + || strcmp (Fs_type, "coda") == 0 \ + || strcmp (Fs_type, "auristorfs") == 0 \ + || strcmp (Fs_type, "fhgfs") == 0 \ + || strcmp (Fs_type, "gpfs") == 0 \ + || strcmp (Fs_type, "ibrix") == 0 \ + || strcmp (Fs_type, "ocfs2") == 0 \ + || strcmp (Fs_type, "vxfs") == 0 \ + || strcmp ("-hosts", Fs_name) == 0) +#endif + +static struct mntent *find_mount (const char *path,const struct statvfs *fs) +{ +const char *const mounts ="/proc/mounts" ; +const char *rp =real_pathdup (path,1 ); +struct mntent *ment =NULL ; +struct mntent *found =NULL ; +struct statvfs mntfs; +FILE *fp; +int dlen, plen, flen =0 ; + + fp =setmntent (mounts,"r" ); +if (!fp) { +error_errno (_ ("setmntent('%s') failed" ), mounts); +return NULL ; + } + + plen =strlen (rp); + +/* read all the mount information and compare to path */ +while ((ment =getmntent (fp)) !=NULL ) { +if (statvfs (ment->mnt_dir, &mntfs)) { +switch (errno) { +case EPERM : +case ESRCH : +case EACCES : +continue ; +default : +error_errno (_ ("statvfs('%s') failed" ), ment->mnt_dir); +endmntent (fp); +return NULL ; + } + } + +/* is mount on the same filesystem and is a prefix of the path */ +if ((fs->f_fsid == mntfs.f_fsid) && + !strncmp (ment->mnt_dir, rp,strlen (ment->mnt_dir))) { + dlen =strlen (ment->mnt_dir); +if (dlen > plen) +continue ; +/* +* root is always a potential match; otherwise look for +* directory prefix +*/ +if ((dlen ==1 && ment->mnt_dir[0 ] =='/' ) || + (dlen > flen && (!rp[dlen] || rp[dlen] =='/' ))) { + flen = dlen; +/* +* https://man7.org/linux/man-pages/man3/getmntent.3.html +* +* The pointer points to a static area of memory which is +* overwritten by subsequent calls to getmntent(). +*/ +if (!found) +CALLOC_ARRAY (found,1 ); +free (found->mnt_dir); +free (found->mnt_type); +free (found->mnt_fsname); + found->mnt_dir =xmemdupz (ment->mnt_dir,strlen (ment->mnt_dir)); + found->mnt_type =xmemdupz (ment->mnt_type,strlen (ment->mnt_type)); + found->mnt_fsname =xmemdupz (ment->mnt_fsname,strlen (ment->mnt_fsname)); + } + } + } +endmntent (fp); + +return found; +} + +int fsmonitor__get_fs_info (const char *path,struct fs_info *fs_info) +{ +struct mntent *ment; +struct statvfs fs; + +if (statvfs (path, &fs)) +return error_errno (_ ("statvfs('%s') failed" ), path); + + ment =find_mount (path, &fs); +if (!ment) +return -1 ; + +trace_printf_key (&trace_fsmonitor, +"statvfs('%s') [flags 0x%08lx] '%s' '%s'" , + path, fs.f_flag, ment->mnt_type, ment->mnt_fsname); + +if (ME_REMOTE (ment->mnt_fsname, ment->mnt_type)) + fs_info->is_remote =1 ; +else + fs_info->is_remote =0 ; + + fs_info->typename = ment->mnt_fsname; +free (ment->mnt_dir); +free (ment->mnt_type); +free (ment); + +trace_printf_key (&trace_fsmonitor, +"'%s' is_remote: %d" , + path, fs_info->is_remote); +return 0 ; +} + +int fsmonitor__is_fs_remote (const char *path) +{ +struct fs_info fs; + +if (fsmonitor__get_fs_info (path, &fs)) +return -1 ; + +free (fs.typename); + +return fs.is_remote; +} + +/* +* No-op for now. +*/ +int fsmonitor__get_alias (const char *path,struct alias_info *info) +{ +return 0 ; +} + +/* +* No-op for now. +*/ +char *fsmonitor__resolve_alias (const char *path, +const struct alias_info *info) +{ +return NULL ; +}
Signature · 1 line
gitgitgadget
PATCH 9/12 fsmonitor: implement filesystem change listener for Linux
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
Implement a filesystem change listener for Linux based on the inotify API:
https://man7.org/linux/man-pages/man7/inotify.7.html
inotify requires registering a watch on every directory in the worktree and special handling of moves/renames.
--- compat/fsmonitor/fsm-listen-linux.c | 664 ++++++++++++++++++++++++++++ 1 file changed, 664 insertions(+) create mode 100644 compat/fsmonitor/fsm-listen-linux.c
compat/fsmonitor/fsm-listen-linux.c new file +664 -0
@@ -0,0 +1,664 @@ +#include "cache.h" +#include "fsmonitor.h" +#include "fsm-listen.h" +#include "fsmonitor--daemon.h" +#include <dirent.h> +#include <fcntl.h> +#include <sys/inotify.h> +#include <sys/stat.h> + +/* +* Safe value to bitwise OR with rest of mask for +* kernels that do not support IN_MASK_CREATE +*/ +#ifndef IN_MASK_CREATE +#define IN_MASK_CREATE 0x00000000 +#endif + +enum shutdown_reason { +SHUTDOWN_CONTINUE =0 , +SHUTDOWN_STOP , +SHUTDOWN_ERROR , +SHUTDOWN_FORCE +}; + +struct watch_entry { +struct hashmap_entry ent; +int wd; +uint32_t cookie; +const char *dir; +}; + +struct rename_entry { +struct hashmap_entry ent; +time_t whence; +uint32_t cookie; +const char *dir; +}; + +struct fsm_listen_data { +int fd_inotify; +enum shutdown_reason shutdown; +struct hashmap watches; +struct hashmap renames; +struct hashmap revwatches; +}; + +static int watch_entry_cmp (const void *cmp_data, +const struct hashmap_entry *eptr, +const struct hashmap_entry *entry_or_key, +const void *keydata) +{ +const struct watch_entry *e1, *e2; + + e1 = container_of(eptr, const struct watch_entry, ent); + e2 = container_of(eptr, const struct watch_entry, ent); +return e1->wd != e2->wd; +} + +static int revwatches_entry_cmp (const void *cmp_data, +const struct hashmap_entry *eptr, +const struct hashmap_entry *entry_or_key, +const void *keydata) +{ +const struct watch_entry *e1, *e2; + + e1 = container_of(eptr, const struct watch_entry, ent); + e2 = container_of(eptr, const struct watch_entry, ent); +return strcmp (e1->dir, e2->dir); +} + +static int rename_entry_cmp (const void *cmp_data, +const struct hashmap_entry *eptr, +const struct hashmap_entry *entry_or_key, +const void *keydata) +{ +const struct rename_entry *e1, *e2; + + e1 = container_of(eptr, const struct rename_entry, ent); + e2 = container_of(eptr, const struct rename_entry, ent); +return e1->cookie != e2->cookie; +} + +/* +* Register an inotify watch, add watch descriptor to path mapping +* and the reverse mapping. +*/ +static int add_watch (const char *path,struct fsm_listen_data *data) +{ +const char *interned =strintern (path); +struct watch_entry *w1, *w2; + +/* add the inotify watch, don't allow watches to be modified */ +int wd =inotify_add_watch (data->fd_inotify, interned, + (IN_ALL_EVENTS |IN_ONLYDIR |IN_MASK_CREATE ) + ^IN_ACCESS ^IN_CLOSE ^IN_OPEN ); +if (wd <0 ) +return error_errno ("inotify_add_watch('%s') failed" , interned); + +/* add watch descriptor -> directory mapping */ +CALLOC_ARRAY (w1,1 ); + w1->wd = wd; + w1->dir = interned; +hashmap_entry_init (&w1->ent,memhash (&w1->wd,sizeof (int ))); +hashmap_add (&data->watches, &w1->ent); + +/* add directory -> watch descriptor mapping */ +CALLOC_ARRAY (w2,1 ); + w2->wd = wd; + w2->dir = interned; +hashmap_entry_init (&w2->ent,memhash (w2->dir,strlen (w2->dir))); +hashmap_add (&data->revwatches, &w2->ent); + +return 0 ; +} + +/* +* Remove the inotify watch, the watch descriptor to path mapping +* and the reverse mapping. +*/ +static void remove_watch (struct watch_entry *w, +struct fsm_listen_data *data) +{ +struct watch_entry k1, k2, *w1, *w2; + +/* remove watch, ignore error if kernel already did it */ +if (inotify_rm_watch (data->fd_inotify, w->wd) && errno !=EINVAL ) +error_errno ("inotify_rm_watch() failed" ); + +hashmap_entry_init (&k1.ent,memhash (&w->wd,sizeof (int ))); + w1 =hashmap_remove_entry (&data->watches, &k1, ent,NULL ); +if (!w1) +BUG ("Double remove of watch for '%s'" , w->dir); + +if (w1->cookie) +BUG ("Removing watch for '%s' which has a pending rename" , w1->dir); + +hashmap_entry_init (&k2.ent,memhash (w->dir,strlen (w->dir))); + w2 =hashmap_remove_entry (&data->revwatches, &k2, ent,NULL ); +if (!w2) +BUG ("Double remove of reverse watch for '%s'" , w->dir); + +/* w1->dir and w2->dir are interned strings, we don't own them */ +free (w1); +free (w2); +} + +/* +* Check for stale directory renames. +* +* https://man7.org/linux/man-pages/man7/inotify.7.html +* +* Allow for some small timeout to account for the fact that insertion of the +* IN_MOVED_FROM+IN_MOVED_TO event pair is not atomic, and the possibility that +* there may not be any IN_MOVED_TO event. +* +* If the IN_MOVED_TO event is not received within the timeout then events have +* been missed and the monitor is in an inconsistent state with respect to the +* filesystem. +*/ +static int check_stale_dir_renames (struct hashmap *renames,time_t max_age) +{ +struct rename_entry *re; +struct hashmap_iter iter; + +hashmap_for_each_entry (renames, &iter, re, ent) { +if (re->whence <= max_age) +return -1 ; + } +return 0 ; +} + +/* +* Track pending renames. +* +* Tracking is done via a event cookie to watch descriptor mapping. +* +* A rename is not complete until matching a IN_MOVED_TO event is received +* for a corresponding IN_MOVED_FROM event. +*/ +static void add_dir_rename (uint32_t cookie,const char *path, +struct fsm_listen_data *data) +{ +struct watch_entry k, *w; +struct rename_entry *re; + +/* lookup the watch descriptor for the given path */ +hashmap_entry_init (&k.ent,memhash (path,strlen (path))); + w =hashmap_get_entry (&data->revwatches, &k, ent,NULL ); +if (!w)/* should never happen */ +BUG ("No watch for '%s'" , path); + w->cookie = cookie; + +/* add the pending rename to match against later */ +CALLOC_ARRAY (re,1 ); + re->dir = w->dir; + re->cookie = w->cookie; + re->whence =time (NULL ); +hashmap_entry_init (&re->ent,memhash (&re->cookie,sizeof (uint32_t ))); +hashmap_add (&data->renames, &re->ent); +} + +/* +* Handle directory renames +* +* Once a IN_MOVED_TO event is received, lookup the rename tracking information +* via the event cookie and use this information to update the watch. +*/ +static void rename_dir (uint32_t cookie,const char *path, +struct fsm_listen_data *data) +{ +struct rename_entry rek, *re; +struct watch_entry k, *w; + +/* lookup a pending rename to match */ + rek.cookie = cookie; +hashmap_entry_init (&rek.ent,memhash (&rek.cookie,sizeof (uint32_t ))); + re =hashmap_get_entry (&data->renames, &rek, ent,NULL ); +if (re) { + k.dir = re->dir; +hashmap_entry_init (&k.ent,memhash (k.dir,strlen (k.dir))); + w =hashmap_get_entry (&data->revwatches, &k, ent,NULL ); +if (w) { + w->cookie =0 ;/* rename handled */ +remove_watch (w, data); +add_watch (path, data); + }else { +BUG ("No matching watch" ); + } + }else { +BUG ("No matching cookie" ); + } +} + +/* +* Recursively add watches to every directory under path +*/ +static int register_inotify (const char *path,struct fsm_listen_data *data) +{ +DIR *dir; +struct strbuf current =STRBUF_INIT ; +struct dirent *de; +struct stat fs; +int ret =-1 ; + + dir =opendir (path); +if (!dir) +return error_errno ("opendir('%s') failed" , path); + +while ((de =readdir_skip_dot_and_dotdot (dir)) !=NULL ) { +strbuf_reset (¤t); +strbuf_addf (¤t,"%s/%s" , path, de->d_name); +if (lstat (current.buf, &fs)) { +error_errno ("lstat('%s') failed" , current.buf); + goto failed; + } + +/* recurse into directory */ +if (S_ISDIR (fs.st_mode)) { +if (add_watch (current.buf, data)) + goto failed; +if (register_inotify (current.buf, data)) + goto failed; + } + } + ret =0 ; + +failed: +strbuf_release (¤t); +if (closedir (dir) <0 ) +return error_errno ("closedir('%s') failed" , path); +return ret; +} + +static int em_rename_dir_from (u_int32_t mask) +{ +return ((mask &IN_ISDIR ) && (mask &IN_MOVED_FROM )); +} + +static int em_rename_dir_to (u_int32_t mask) +{ +return ((mask &IN_ISDIR ) && (mask &IN_MOVED_TO )); +} + +static int em_remove_watch (u_int32_t mask) +{ +return (mask &IN_DELETE_SELF ); +} + +static int em_dir_renamed (u_int32_t mask) +{ +return ((mask &IN_ISDIR ) && (mask &IN_MOVE )); +} + +static int em_dir_created (u_int32_t mask) +{ +return ((mask &IN_ISDIR ) && (mask &IN_CREATE )); +} + +static int em_dir_deleted (uint32_t mask) +{ +return ((mask &IN_ISDIR ) && (mask &IN_DELETE )); +} + +static int em_force_shutdown (u_int32_t mask) +{ +return (mask &IN_UNMOUNT ) || (mask &IN_Q_OVERFLOW ); +} + +static int em_ignore (u_int32_t mask) +{ +return (mask &IN_IGNORED ) || (mask &IN_MOVE_SELF ); +} + +static void log_mask_set (const char *path,u_int32_t mask) +{ +struct strbuf msg =STRBUF_INIT ; + +if (mask &IN_ACCESS ) +strbuf_addstr (&msg,"IN_ACCESS|" ); +if (mask &IN_MODIFY ) +strbuf_addstr (&msg,"IN_MODIFY|" ); +if (mask &IN_ATTRIB ) +strbuf_addstr (&msg,"IN_ATTRIB|" ); +if (mask &IN_CLOSE_WRITE ) +strbuf_addstr (&msg,"IN_CLOSE_WRITE|" ); +if (mask &IN_CLOSE_NOWRITE ) +strbuf_addstr (&msg,"IN_CLOSE_NOWRITE|" ); +if (mask &IN_OPEN ) +strbuf_addstr (&msg,"IN_OPEN|" ); +if (mask &IN_MOVED_FROM ) +strbuf_addstr (&msg,"IN_MOVED_FROM|" ); +if (mask &IN_MOVED_TO ) +strbuf_addstr (&msg,"IN_MOVED_TO|" ); +if (mask &IN_CREATE ) +strbuf_addstr (&msg,"IN_CREATE|" ); +if (mask &IN_DELETE ) +strbuf_addstr (&msg,"IN_DELETE|" ); +if (mask &IN_DELETE_SELF ) +strbuf_addstr (&msg,"IN_DELETE_SELF|" ); +if (mask &IN_MOVE_SELF ) +strbuf_addstr (&msg,"IN_MOVE_SELF|" ); +if (mask &IN_UNMOUNT ) +strbuf_addstr (&msg,"IN_UNMOUNT|" ); +if (mask &IN_Q_OVERFLOW ) +strbuf_addstr (&msg,"IN_Q_OVERFLOW|" ); +if (mask &IN_IGNORED ) +strbuf_addstr (&msg,"IN_IGNORED|" ); +if (mask &IN_ISDIR ) +strbuf_addstr (&msg,"IN_ISDIR|" ); + +trace_printf_key (&trace_fsmonitor,"inotify_event: '%s', mask=%#8.8x %s" , + path, mask, msg.buf); + +strbuf_release (&msg); +} + +int fsm_listen__ctor (struct fsmonitor_daemon_state *state) +{ +int fd; +int ret =0 ; +struct fsm_listen_data *data; + +CALLOC_ARRAY (data,1 ); + state->listen_data = data; + state->listen_error_code =-1 ; + data->shutdown =SHUTDOWN_ERROR ; + + fd =inotify_init1 (O_NONBLOCK ); +if (fd <0 ) +return error_errno ("inotify_init1() failed" ); + + data->fd_inotify = fd; + +hashmap_init (&data->watches, watch_entry_cmp,NULL ,0 ); +hashmap_init (&data->renames, rename_entry_cmp,NULL ,0 ); +hashmap_init (&data->revwatches, revwatches_entry_cmp,NULL ,0 ); + +if (add_watch (state->path_worktree_watch.buf, data)) + ret =-1 ; +else if (register_inotify (state->path_worktree_watch.buf, data)) + ret =-1 ; +else if (state->nr_paths_watching >1 ) { +if (add_watch (state->path_gitdir_watch.buf, data)) + ret =-1 ; +else if (register_inotify (state->path_gitdir_watch.buf, data)) + ret =-1 ; + } + +if (!ret) { + state->listen_error_code =0 ; + data->shutdown =SHUTDOWN_CONTINUE ; + } + +return ret; +} + +void fsm_listen__dtor (struct fsmonitor_daemon_state *state) +{ +struct fsm_listen_data *data; +struct hashmap_iter iter; +struct watch_entry *w; +int fd; + +if (!state || !state->listen_data) +return ; + + data = state->listen_data; + fd = data->fd_inotify; + +hashmap_for_each_entry (&data->watches, &iter, w, ent) { + w->cookie =0 ;/* ignore any pending renames */ +remove_watch (w, data); + } +hashmap_clear (&data->watches); + +hashmap_clear (&data->revwatches);/* remove_watch freed the entries */ + + hashmap_clear_and_free(&data->renames, struct rename_entry, ent); + +FREE_AND_NULL (state->listen_data); + +if (fd && (close (fd) <0 )) +error_errno (_ ("closing inotify file descriptor failed" )); +} + +void fsm_listen__stop_async (struct fsmonitor_daemon_state *state) +{ +if (!state->listen_data->shutdown) + state->listen_data->shutdown =SHUTDOWN_STOP ; +} + +/* +* Process a single inotify event and queue for publication. +*/ +static int process_event (const char *path, +const struct inotify_event *event, +struct fsmonitor_batch *batch, +struct string_list *cookie_list, +struct fsmonitor_daemon_state *state) +{ +const char *rel; +const char *last_sep; + +switch (fsmonitor_classify_path_absolute (state, path)) { +case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX : +case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX : +/* Use just the filename of the cookie file. */ + last_sep =find_last_dir_sep (path); +string_list_append (cookie_list, + last_sep ? last_sep +1 : path); +break ; +case IS_INSIDE_DOT_GIT : +case IS_INSIDE_GITDIR : +break ; +case IS_DOT_GIT : +case IS_GITDIR : +/* +* If .git directory is deleted or renamed away, +* we have to quit. +*/ +if (em_dir_deleted (event->mask)) { +trace_printf_key (&trace_fsmonitor, +"event: gitdir removed" ); + state->listen_data->shutdown =SHUTDOWN_FORCE ; + goto done; + } + +if (em_dir_renamed (event->mask)) { +trace_printf_key (&trace_fsmonitor, +"event: gitdir renamed" ); + state->listen_data->shutdown =SHUTDOWN_FORCE ; + goto done; + } +break ; +case IS_WORKDIR_PATH : +/* normal events in the working directory */ +if (trace_pass_fl (&trace_fsmonitor)) +log_mask_set (path, event->mask); + + rel = path + state->path_worktree_watch.len +1 ; +fsmonitor_batch__add_path (batch, rel); + +if (em_dir_deleted (event->mask)) +break ; + +/* received IN_MOVE_FROM, add tracking for expected IN_MOVE_TO */ +if (em_rename_dir_from (event->mask)) +add_dir_rename (event->cookie, path, state->listen_data); + +/* received IN_MOVE_TO, update watch to reflect new path */ +if (em_rename_dir_to (event->mask)) +rename_dir (event->cookie, path, state->listen_data); + +if (em_dir_created (event->mask)) { +if (add_watch (path, state->listen_data)) { + state->listen_data->shutdown =SHUTDOWN_ERROR ; + goto done; + } +if (register_inotify (path, state->listen_data)) { + state->listen_data->shutdown =SHUTDOWN_ERROR ; + goto done; + } + } +break ; +case IS_OUTSIDE_CONE : +default : +trace_printf_key (&trace_fsmonitor, +"ignoring '%s'" , path); +break ; + } +return 0 ; +done: +return -1 ; +} + +/* +* Read the inotify event stream and pre-process events before further +* processing and eventual publishing. +*/ +static void handle_events (struct fsmonitor_daemon_state *state) +{ +/* See https://man7.org/linux/man-pages/man7/inotify.7.html */ + char buf[4096] + __attribute__ ((aligned(__alignof__(struct inotify_event)))); + +struct hashmap watches = state->listen_data->watches; +struct fsmonitor_batch *batch =NULL ; +struct string_list cookie_list =STRING_LIST_INIT_DUP ; +struct watch_entry k, *w; +struct strbuf path; +const struct inotify_event *event; +int fd = state->listen_data->fd_inotify; +ssize_t len; +char *ptr, *p; + +strbuf_init (&path,PATH_MAX ); + +for (;;) { + len =read (fd, buf,sizeof (buf)); +if (len ==-1 && errno !=EAGAIN ) { +error_errno (_ ("reading inotify message stream failed" )); + state->listen_data->shutdown =SHUTDOWN_ERROR ; + goto done; + } + +/* nothing to read */ +if (len <=0 ) + goto done; + +/* Loop over all events in the buffer. */ +for (ptr = buf; ptr < buf + len; + ptr +=sizeof (struct inotify_event ) + event->len) { + + event = (const struct inotify_event *) ptr; + +if (em_ignore (event->mask)) +continue ; + +/* File system was unmounted or event queue overflowed */ +if (em_force_shutdown (event->mask)) { +if (trace_pass_fl (&trace_fsmonitor)) +log_mask_set ("Forcing shutdown" , event->mask); + state->listen_data->shutdown =SHUTDOWN_FORCE ; + goto done; + } + +hashmap_entry_init (&k.ent,memhash (&event->wd,sizeof (int ))); + k.wd = event->wd; + + w =hashmap_get_entry (&watches, &k, ent,NULL ); +if (!w)/* should never happen */ +BUG ("No watch for '%s'" , event->name); + +/* directory watch was removed */ +if (em_remove_watch (event->mask)) { +remove_watch (w, state->listen_data); +continue ; + } + +strbuf_reset (&path); +strbuf_add (&path, w->dir,strlen (w->dir)); +strbuf_addch (&path,'/' ); +strbuf_add (&path, event->name,strlen (event->name)); + + p =fsmonitor__resolve_alias (path.buf, &state->alias); +if (!p) + p =strbuf_detach (&path,NULL ); + +if (!batch) + batch =fsmonitor_batch__new (); + +if (process_event (p, event, batch, &cookie_list, state)) { +free (p); + goto done; + } +free (p); + } +strbuf_reset (&path); +fsmonitor_publish (state, batch, &cookie_list); +string_list_clear (&cookie_list,0 ); + batch =NULL ; + } +done: +strbuf_release (&path); +fsmonitor_batch__free_list (batch); +string_list_clear (&cookie_list,0 ); +} + +/* +* Non-blocking read of the inotify events stream. The inotify fd is polled +* every few millisec to help minimize the number of queue overflows. +*/ +void fsm_listen__loop (struct fsmonitor_daemon_state *state) +{ +int poll_num; +const int interval =1000 ; +time_t checked =time (NULL ); +struct pollfd fds[1 ]; + fds[0 ].fd = state->listen_data->fd_inotify; + fds[0 ].events =POLLIN ; + +for (;;) { +switch (state->listen_data->shutdown) { +case SHUTDOWN_CONTINUE : + poll_num =poll (fds,1 ,10 ); +if (poll_num ==-1 ) { +if (errno ==EINTR ) +continue ; +error_errno (_ ("polling inotify message stream failed" )); + state->listen_data->shutdown =SHUTDOWN_ERROR ; +continue ; + } + +if ((time (NULL ) - checked) >= interval) { + checked =time (NULL ); +if (check_stale_dir_renames (&state->listen_data->renames, + checked - interval)) { +trace_printf_key (&trace_fsmonitor, +"Missed IN_MOVED_TO events, forcing shutdown" ); + state->listen_data->shutdown =SHUTDOWN_FORCE ; +continue ; + } + } + +if (poll_num >0 && (fds[0 ].revents &POLLIN )) +handle_events (state); + +continue ; +case SHUTDOWN_ERROR : + state->listen_error_code =-1 ; +ipc_server_stop_async (state->ipc_server_data); +break ; +case SHUTDOWN_FORCE : + state->listen_error_code =0 ; +ipc_server_stop_async (state->ipc_server_data); +break ; +case SHUTDOWN_STOP : +default : + state->listen_error_code =0 ; +break ; + } +return ; + } +}
Signature · 1 line
gitgitgadget
PATCH 10/12 fsmonitor: enable fsmonitor for Linux
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
Uodate build to enable fsmonitor for Linux.
--- config.mak.uname | 9 +++++++++ contrib/buildsystems/CMakeLists.txt | 11 ++++++++++- 2 files changed, 19 insertions(+), 1 deletion(-)
config.mak.uname +9 -0
@@ -69,6 +69,15 @@ ifeq ($(uname_S),Linux) BASIC_CFLAGS += -std=c99 endif + # The builtin FSMonitor on Linux builds upon Simple-IPC. Both require + # Unix domain sockets and PThreads. + ifndef NO_PTHREADS + ifndef NO_UNIX_SOCKETS + FSMONITOR_DAEMON_BACKEND = linux + FSMONITOR_OS_SETTINGS = linux + FSMONITOR_DAEMON_COMMON = unix + endif + endif endif ifeq ($(uname_S),GNU/kFreeBSD) HAVE_ALLOCA_H = YesPlease
contrib/buildsystems/CMakeLists.txt +10 -1
@@ -304,7 +304,16 @@ else() endif() if(SUPPORTS_SIMPLE_IPC) - if(CMAKE_SYSTEM_NAME STREQUAL "Windows") + if(CMAKE_SYSTEM_NAME STREQUAL "Linux") + add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-unix.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-unix.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-linux.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-linux.c) + + add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-unix.c) + elseif(CMAKE_SYSTEM_NAME STREQUAL "Windows") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c) list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c)
Signature · 1 line
gitgitgadget
PATCH 11/12 fsmonitor: test updates
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
t7527-builtin-fsmonitor was leaking fsmonitor--daemon processes in some cases.
Accomodate slight difference in the number of events generated on Linux.
On lower-powered systems, spin a little to give the daemon time to respond to and log filesystem events.
--- t/t7527-builtin-fsmonitor.sh | 58 +++++++++++++++++++++++++++--------- 1 file changed, 44 insertions(+), 14 deletions(-)
t/t7527-builtin-fsmonitor.sh +44 -14
@@ -13,7 +13,7 @@ fi stop_daemon_delete_repo () { r=$1 && test_might_fail git -C $r fsmonitor--daemon stop && - rm -rf $1 + rm -rf $r } start_daemon () {
@@ -72,6 +72,32 @@ start_daemon () { ) } +IMPLICIT_TIMEOUT=5 + +wait_for_update () { + func=$1 && + file=$2 && + sz=$(wc -c < "$file") && + last=0 && + $func && + k=0 && + while test "$k" -lt $IMPLICIT_TIMEOUT + do + nsz=$(wc -c < "$file") + if test "$nsz" -gt "$sz" + then + if test "$last" -eq "$nsz" + then + return 0 + fi + last=$nsz + fi + sleep 1 + k=$(( $k + 1 )) + done && + return 0 +} + # Is a Trace2 data event present with the given catetory and key? # We do not care what the value is. #
@@ -137,7 +163,6 @@ test_expect_success 'implicit daemon start' ' # machines (where it might take a moment to wake and reschedule the # daemon process) to avoid false alarms during test runs.) # -IMPLICIT_TIMEOUT=5 verify_implicit_shutdown () { r=$1 &&
@@ -373,6 +398,10 @@ create_files () { echo 3 >dir2/new } +rename_directory () { + mv dirtorename dirrenamed +} + rename_files () { mv rename renamed && mv dir1/rename dir1/renamed &&
@@ -427,7 +456,7 @@ test_expect_success 'edit some files' ' start_daemon --tf "$PWD/.git/trace" && - edit_files && + wait_for_update edit_files "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 &&
@@ -442,7 +471,7 @@ test_expect_success 'create some files' ' start_daemon --tf "$PWD/.git/trace" && - create_files && + wait_for_update create_files "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 &&
@@ -456,7 +485,7 @@ test_expect_success 'delete some files' ' start_daemon --tf "$PWD/.git/trace" && - delete_files && + wait_for_update delete_files "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 &&
@@ -470,7 +499,7 @@ test_expect_success 'rename some files' ' start_daemon --tf "$PWD/.git/trace" && - rename_files && + wait_for_update rename_files "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 &&
@@ -487,7 +516,7 @@ test_expect_success 'rename directory' ' start_daemon --tf "$PWD/.git/trace" && - mv dirtorename dirrenamed && + wait_for_update rename_directory "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 &&
@@ -500,7 +529,7 @@ test_expect_success 'file changes to directory' ' start_daemon --tf "$PWD/.git/trace" && - file_to_directory && + wait_for_update file_to_directory "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 &&
@@ -513,7 +542,7 @@ test_expect_success 'directory changes to a file' ' start_daemon --tf "$PWD/.git/trace" && - directory_to_file && + wait_for_update directory_to_file "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 &&
@@ -561,7 +590,7 @@ test_expect_success 'flush cached data' ' test-tool -C test_flush fsmonitor-client query --token "builtin:test_00000002:0" >actual_2 && nul_to_q <actual_2 >actual_q2 && - grep "^builtin:test_00000002:0Q$" actual_q2 && + grep "^builtin:test_00000002:[0-1]Q$" actual_q2 && >test_flush/file_3 &&
@@ -732,7 +761,8 @@ u_values="$u1 $u2" for u in $u_values do test_expect_success "unicode in repo root path: $u" ' - test_when_finished "stop_daemon_delete_repo $u" && + test_when_finished \ + "stop_daemon_delete_repo `echo "$u" | sed 's:x:\\\\\\\\\\\\\\x:g'`" && git init "$u" && echo 1 >"$u"/file1 &&
@@ -814,8 +844,7 @@ my_match_and_clean () { } test_expect_success 'submodule always visited' ' - test_when_finished "git -C super fsmonitor--daemon stop; \ - rm -rf super; \ + test_when_finished "rm -rf super; \ rm -rf sub" && create_super super &&
@@ -883,7 +912,8 @@ have_t2_error_event () { } test_expect_success "stray submodule super-prefix warning" ' - test_when_finished "rm -rf super; \ + test_when_finished "git -C super/dir_1/dir_2/sub fsmonitor--daemon stop; \ + rm -rf super; \ rm -rf sub; \ rm super-sub.trace" &&
Signature · 1 line
gitgitgadget
PATCH 12/12 fsmonitor: update doc for Linux
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
Update the documentation for Linux.
--- Documentation/config/fsmonitor--daemon.txt | 4 ++-- Documentation/git-fsmonitor--daemon.txt | 24 ++++++++++++++-------- 2 files changed, 18 insertions(+), 10 deletions(-)
Documentation/config/fsmonitor--daemon.txt +2 -2
@@ -4,8 +4,8 @@ fsmonitor.allowRemote:: behavior. Only respected when `core.fsmonitor` is set to `true`. fsmonitor.socketDir:: - This Mac OS-specific option, if set, specifies the directory in + Mac OS and Linux-specific option. If set, specifies the directory in which to create the Unix domain socket used for communication between the fsmonitor daemon and various Git commands. The directory must - reside on a native Mac OS filesystem. Only respected when `core.fsmonitor` + reside on a native filesystem. Only respected when `core.fsmonitor` is set to `true`.
Documentation/git-fsmonitor--daemon.txt +16 -8
@@ -76,23 +76,31 @@ repositories; this may be overridden by setting `fsmonitor.allowRemote` to correctly with all network-mounted repositories and such use is considered experimental. -On Mac OS, the inter-process communication (IPC) between various Git +On Linux and Mac OS, the inter-process communication (IPC) between various Git commands and the fsmonitor daemon is done via a Unix domain socket (UDS) -- a -special type of file -- which is supported by native Mac OS filesystems, -but not on network-mounted filesystems, NTFS, or FAT32. Other filesystems -may or may not have the needed support; the fsmonitor daemon is not guaranteed -to work with these filesystems and such use is considered experimental. +special type of file -- which is supported by many native Linux and Mac OS +filesystems, but not on network-mounted filesystems, NTFS, or FAT32. Other +filesystems may or may not have the needed support; the fsmonitor daemon is not +guaranteed to work with these filesystems and such use is considered +experimental. By default, the socket is created in the `.git` directory, however, if the `.git` directory is on a network-mounted filesystem, it will be instead be created at `$HOME/.git-fsmonitor-*` unless `$HOME` itself is on a network-mounted filesystem in which case you must set the configuration -variable `fsmonitor.socketDir` to the path of a directory on a Mac OS native +variable `fsmonitor.socketDir` to the path of a directory on a native filesystem in which to create the socket file. If none of the above directories (`.git`, `$HOME`, or `fsmonitor.socketDir`) -is on a native Mac OS file filesystem the fsmonitor daemon will report an -error that will cause the daemon and the currently running command to exit. +is on a native Linux or Mac OS filesystem the fsmonitor daemon will report +an error that will cause the daemon to exit and the currently running command +to issue a warning. + +On Linux, the fsmonitor daemon registers a watch for each directory in the +repository. The default per-user limit for the number of watches on most Linux +systems is 8192. This may not be sufficient for large repositories or if +multiple instances of the fsmonitor daemon are running. +See https://watchexec.github.io/docs/inotify-limits.html[Linux inotify limits] for more information. CONFIGURATION -------------
Signature · 1 line
gitgitgadget
Re: [PATCH 07/12] fsmonitor: prepare to share code between Mac OS and Linux
In reply to Eric DeCosta via GitGitGadgetQuoted 10 lines · Eric DeCosta via GitGitGadget
Makefile +0
-2@@ -2038,13 +2038,13 @@ endif ifdef FSMONITOR_DAEMON_BACKEND COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o - COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o - COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_BACKEND).o
Makefile endif · line 2049
Quoted 10 lines
Makefile +3
-1+ COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_COMMON).o + COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_COMMON).o endif ifdef FSMONITOR_OS_SETTINGS COMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS - COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o + COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_DAEMON_COMMON).o COMPAT_OBJS += compat/fsmonitor/fsm-path-utils-$(FSMONITOR_OS_SETTINGS).o endif
These two look sloppier than the existing "if we have DAEMON_BACKEND defined, then add $(DAEMON_BACKEND).o to the set of objects used". $(DAEMON_COMMON).o should be used after the same kind of check, i.e.
+ifdef FSMONITOR_DAEMON_COMMON
+ COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_COMMON).o
+ COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_COMMON).o
+ COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_DAEMON_COMMON).o
+endif
as a separate if/endif block, without touching the above two, perhaps?
Quoted 13 lines
compat/fsmonitor/fsm-ipc-unix.c renamed from compat/fsmonitor/fsm-ipc-darwin.c +1
-1@@ -10,7 +10,7 @@ static GIT_PATH_FUNC(fsmonitor_ipc__get_default_path, "fsmonitor--daemon.ipc")const char *fsmonitor_ipc__get_path (struct repository *r) {static const char *ipc_path =NULL ; -SHA_CTX sha1ctx; +git_SHA_CTX sha1ctx;char *sock_dir =NULL ;struct strbuf ipc_file =STRBUF_INIT ;unsigned char hash[SHA_DIGEST_LENGTH ];@@ -28,9 +28,9 @@ const char *fsmonitor_ipc__get_path(struct repository *r)return ipc_path;
compat/fsmonitor/fsm-ipc-unix.c const char *fsmonitor_ipc__get_path(struct repository *r) · line 35
Quoted 10 lines
compat/fsmonitor/fsm-ipc-unix.c renamed from compat/fsmonitor/fsm-ipc-darwin.c +3
-3} -SHA1_Init (&sha1ctx); -SHA1_Update (&sha1ctx, r->worktree,strlen (r->worktree)); -SHA1_Final (hash, &sha1ctx); +git_SHA1_Init (&sha1ctx); +git_SHA1_Update (&sha1ctx, r->worktree,strlen (r->worktree)); +git_SHA1_Final (hash, &sha1ctx);repo_config_get_string (r,"fsmonitor.socketdir" , &sock_dir);
Interesting. The result of course is good, but I wonder how we have been happy to queue the original code that lack git_ prefix in the first place X-<.
config.mak.uname ifeq ($(uname_S),Linux) · line 74
Quoted 12 lines
config.mak.uname +1
-0@@ -68,6 +68,7 @@ ifeq ($(uname_S),Linux) ifneq ($(findstring .el7.,$(uname_R)),) BASIC_CFLAGS += -std=c99 endif + endif ifeq ($(uname_S),GNU/kFreeBSD) HAVE_ALLOCA_H = YesPlease
That's a new blank line in an unexpected place. Did you mean to add it after the outer endif?
Quoted 5 lines · Eric DeCosta via GitGitGadget
Goal is to deliver fsmonitor for Linux that is on par with fsmonitor for Windows and Mac OS.
This patch set builds upon previous work for done for Windows and Mac OS (first 6 patches) ...
For those who are watching from sidelines...
The Windows part is already in Git 2.38; the changes needed for macOS are already in 'next' and the first 6 patches in this 12-patch series are identical to them. The patches 7-12 are new.
Quoted 5 lines
to implement a fsmonitor back-end for Linux based on the
Linux inotify API. inotify differs significantly from the equivalent Windows and Mac OS APIs in that a watch must be registered for every directory of interest (rather than a singular watch at the root of the directory tree) and special care must be taken to handle directory renames correctly.
Quoted 4 lines · Junio C Hamano
Quoted 3 lines · Eric DeCosta via GitGitGadget
Goal is to deliver fsmonitor for Linux that is on par with fsmonitor for Windows and Mac OS.
Quoted 8 lines
Quoted 2 lines
This patch set builds upon previous work for done for Windows and Mac OS (first 6 patches) ...
For those who are watching from sidelines...
The Windows part is already in Git 2.38; the changes needed for macOS are already in 'next' and the first 6 patches in this 12-patch series are identical to them. The patches 7-12 are new.
Thanks for clarifying. I found it confusing that there were a number of patches in this series which I had already seen despite the cover letter's claim that this series builds upon "previous work". Thus, it wasn't clear whether this series was a reroll (refining some already-seen patches) with additional patches for Linux, or if it was purely new work with the original patches included by accident[1].
[1]: In the few times I've used GitGitGadget, I've had to jump through hoops to make it send just the "new" patches when I've built a series atop some other series only in 'next' or 'seen', so I can understand the inclusion of the first six patches being accidental. (Regarding the hoop-jumping, it may be that I just don't understand how to "work" GitGitGadget or GitHub pull-requests.)
Re: [PATCH 08/12] fsmonitor: determine if filesystem is local or remote
In reply to Eric DeCosta via GitGitGadgetQuoted 42 lines · Eric DeCosta via GitGitGadget
From: Eric DeCosta <[email protected]>
Compare the given path to the mounted filesystems. Find the mount that is the longest prefix of the path (if any) and determine if that mount is on a local or remote filesystem.
Signed-off-by Eric DeCosta <[email protected]>--- compat/fsmonitor/fsm-path-utils-linux.c | 163 ++++++++++++++++++++++++ 1 file changed, 163 insertions(+) create mode 100644 compat/fsmonitor/fsm-path-utils-linux.ccompat/fsmonitor/fsm-path-utils-linux.c new file +24
-0@@ -0,0 +1,163 @@ +#include "fsmonitor.h" +#include "fsmonitor-path-utils.h" +#include <errno.h> +#include <mntent.h> +#include <sys/mount.h> +#include <sys/statvfs.h> + +/* +* https://github.com/coreutils/gnulib/blob/master/lib/mountlist.c +*/ +#ifndef ME_REMOTE +/* A file system is "remote" if its Fs_name contains a ':' +or if (it is of type (smbfs or cifs) and its Fs_name starts with '//') +or if it is of any other of the listed types +or Fs_name is equal to "-hosts" (used by autofs to mount remote fs). +"VM" file systems like prl_fs or vboxsf are not considered remote here. */ +# define ME_REMOTE (Fs_name, Fs_type) \ + (strchr (Fs_name, ':') != NULL \ + || ((Fs_name)[0] == '/' \ + && (Fs_name)[1] == '/' \ + && (strcmp (Fs_type, "smbfs") == 0 \ + || strcmp (Fs_type, "smb3") == 0 \ + || strcmp (Fs_type, "cifs") == 0)) \ + || strcmp (Fs_type, "acfs") == 0 \
compat/fsmonitor/fsm-path-utils-linux.c line 34
Quoted 10 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +10
-0+ || strcmp (Fs_type, "afs") == 0 \ + || strcmp (Fs_type, "coda") == 0 \ + || strcmp (Fs_type, "auristorfs") == 0 \ + || strcmp (Fs_type, "fhgfs") == 0 \ + || strcmp (Fs_type, "gpfs") == 0 \ + || strcmp (Fs_type, "ibrix") == 0 \ + || strcmp (Fs_type, "ocfs2") == 0 \ + || strcmp (Fs_type, "vxfs") == 0 \ + || strcmp ("-hosts", Fs_name) == 0) +#endif
So, this is just copy/pasted GPLv3 code into our GPLv2-only codebase?:
https://github.com/coreutils/gnulib/blob/cd1fdabe8b66c102124b6a5b0c97dded20246b46/lib/mountlist.c#L230-L247
Quoted 4 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +4
-0+static struct mntent *find_mount (const char *path,const struct statvfs *fs) +{ +const char *const mounts ="/proc/mounts" ; +const char *rp =real_pathdup (path,1 );
compat/fsmonitor/fsm-path-utils-linux.c line 49
Quoted 10 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +10
-0+struct mntent *ment =NULL ; +struct mntent *found =NULL ; +struct statvfs mntfs; +FILE *fp; +int dlen, plen, flen =0 ; + + fp =setmntent (mounts,"r" ); +if (!fp) { +error_errno (_ ("setmntent('%s') failed" ), mounts); +return NULL ;
This code would probably be nicer if you returned int, and passed a pointer to a struct to populate as a paremeter. Then you could just "return error..." for this and the below cases.
Quoted 30 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +30
-0+ } + + plen =strlen (rp); + +/* read all the mount information and compare to path */ +while ((ment =getmntent (fp)) !=NULL ) { +if (statvfs (ment->mnt_dir, &mntfs)) { +switch (errno) { +case EPERM : +case ESRCH : +case EACCES : +continue ; +default : +error_errno (_ ("statvfs('%s') failed" ), ment->mnt_dir); +endmntent (fp);is + return NULL; + } + } + +/* is mount on the same filesystem and is a prefix of the path */ +if ((fs->f_fsid == mntfs.f_fsid) && + !strncmp (ment->mnt_dir, rp,strlen (ment->mnt_dir))) { + dlen =strlen (ment->mnt_dir); +if (dlen > plen) +continue ; +/* +* root is always a potential match; otherwise look for +* directory prefix +*/ + if ((dlen == 1 && ment->mnt_dir[0] == '/') ||
compat/fsmonitor/fsm-path-utils-linux.c line 89
Quoted 10 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +10
-0+ (dlen > flen && (!rp[dlen] || rp[dlen] == '/'))) { + flen = dlen; +/* +* https://man7.org/linux/man-pages/man3/getmntent.3.html +* +* The pointer points to a static area of memory which is +* overwritten by subsequent calls to getmntent(). +*/ +if (!found) +CALLOC_ARRAY (found,1 );
It seems we never populate >1 of these, so don't you just want xmalloc(). Or actually...
compat/fsmonitor/fsm-path-utils-linux.c line 95
Quoted 6 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +6
-0+free (found->mnt_dir); +free (found->mnt_type); +free (found->mnt_fsname); + found->mnt_dir =xmemdupz (ment->mnt_dir,strlen (ment->mnt_dir)); + found->mnt_type =xmemdupz (ment->mnt_type,strlen (ment->mnt_type)); + found->mnt_fsname =xmemdupz (ment->mnt_fsname,strlen (ment->mnt_fsname));
Don't mix mem*() and str*(). In this case we need the string to be '\0' delimited, so use xstrndup().
And once we do that, we might wonder why we're explicitly finding the '\0', just to pass it to the xstrn*() function, when we can just do:
found->mnt_dir = xstrdup(ment->mnt_dir); ...
Which would AFAICT be equivalent to what you're doing here.
compat/fsmonitor/fsm-path-utils-linux.c line 106
Quoted 11 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +11
-0+ } + } + } +endmntent (fp); + +return found; +} + +int fsmonitor__get_fs_info (const char *path,struct fs_info *fs_info) +{ +struct mntent *ment;
...make this (or above) a "struct mntent ment", then pass &ment down, so we can fill that struct? Dunno...
Quoted 7 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +7
-0+struct statvfs fs; + +if (statvfs (path, &fs)) +return error_errno (_ ("statvfs('%s') failed" ), path); + + ment =find_mount (path, &fs); +if (!ment)
compat/fsmonitor/fsm-path-utils-linux.c line 123
Quoted 10 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +10
-0+return -1 ; + +trace_printf_key (&trace_fsmonitor, +"statvfs('%s') [flags 0x%08lx] '%s' '%s'" , + path, fs.f_flag, ment->mnt_type, ment->mnt_fsname); + +if (ME_REMOTE (ment->mnt_fsname, ment->mnt_type)) + fs_info->is_remote =1 ; +else + fs_info->is_remote =0 ;
Shorter:
fs_info->is_remote = ME_REMOTE(ment->mnt_fsname, ment->mnt_type);
Re: [PATCH 07/12] fsmonitor: prepare to share code between Mac OS and Linux
In reply to Eric DeCosta via GitGitGadgetQuoted 9 lines · Eric DeCosta via GitGitGadget
From: Eric DeCosta <[email protected]>
Linux and Mac OS can share some of the code originally developed for Mac OS. Rename the shared code from
compat/fsmonitor/fsm-*-dawrin.ctocompat/fsmonitor/fsm-*-unix.cUpdate the build to enable sharing of the fsm-*-unix.c files.
Minor update to
compat/fsmonitor/fsm-ipc-unix.cto make it cross-platform.
I fear that it may be too early to claim that the platform-specific code can truly be shared (at least without some amount of future ifdef'ing). Yes, there are some commonalities, such as our use of Unix domain sockets on Mac and *nix, but there may be other things that are not.
So maybe do the rename for the IPC code, but not for the health, for example. Or, keep the platform-named files and add new -unix-common.c files that can be called from the other (or vice versa).
Quoted 22 lines
Signed-off-by Eric DeCosta <[email protected]>--- Makefile | 6 +++--- .../fsmonitor/{fsm-health-darwin.c => fsm-health-unix.c} | 0 compat/fsmonitor/{fsm-ipc-darwin.c => fsm-ipc-unix.c} | 8 ++++---- .../{fsm-settings-darwin.c => fsm-settings-unix.c} | 0 config.mak.uname | 4 ++++ contrib/buildsystems/CMakeLists.txt | 6 +++--- 6 files changed, 14 insertions(+), 10 deletions(-)rename
compat/fsmonitor/{fsm-health-darwin.c=>fsm-health-unix.c} (100%) renamecompat/fsmonitor/{fsm-ipc-darwin.c=>fsm-ipc-unix.c} (89%) renamecompat/fsmonitor/{fsm-settings-darwin.c=>fsm-settings-unix.c} (100%)Makefile +0
-0@@ -2038,13 +2038,13 @@ endif ifdef FSMONITOR_DAEMON_BACKEND COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o
Makefile endif · line 2047
Quoted 10 lines
Makefile +3
-3- COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o - COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_BACKEND).o + COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_COMMON).o + COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_COMMON).o endif ifdef FSMONITOR_OS_SETTINGS COMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS - COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o + COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_DAEMON_COMMON).o
This is wrong. you're mixing the usage of the FSMONITOR_OS_SETTINGS definition and the usage of your new _COMMON definition.
And I agree with Junio's comment about grouping the _COMMON ones in their own block and leaving the original COMPAT_OBJS lines as they were.
But really, I think the use of that "common" definition is going to cause us problems later on. I mean, you're using it to smash together the ipc, health, and settings code -- because they cluster around the unix sockets and how on unix to tell if something is remote. That grouping may break if a platform needs something else, such as how to behave when running under inetd vs launchctl vs <whatever>, for example.
Quoted 15 lines
Makefile +0
-0COMPAT_OBJS += compat/fsmonitor/fsm-path-utils-$(FSMONITOR_OS_SETTINGS).o endifcompat/fsmonitor/fsm-health-unix.c renamed from compat/fsmonitor/fsm-health-darwin.c +0
-0compat/fsmonitor/fsm-ipc-unix.c renamed from compat/fsmonitor/fsm-ipc-darwin.c +0
-0
compat/fsmonitor/fsm-ipc-unix.c static GIT_PATH_FUNC(fsmonitor_ipc__get_default_path, "fsmonitor--daemon.ipc") · line 13
Quoted 5 lines
compat/fsmonitor/fsm-ipc-unix.c renamed from compat/fsmonitor/fsm-ipc-darwin.c +1
-1@@ -10,7 +10,7 @@ static GIT_PATH_FUNC(fsmonitor_ipc__get_default_path, "fsmonitor--daemon.ipc")const char *fsmonitor_ipc__get_path (struct repository *r) {static const char *ipc_path =NULL ; -SHA_CTX sha1ctx; +git_SHA_CTX sha1ctx;
Could we put these in a separate commit. They're not related to the file renaming focus of this commit.
Quoted 78 lines
compat/fsmonitor/fsm-ipc-unix.c renamed from compat/fsmonitor/fsm-ipc-darwin.c +3
-3char *sock_dir =NULL ;struct strbuf ipc_file =STRBUF_INIT ;unsigned char hash[SHA_DIGEST_LENGTH ];@@ -28,9 +28,9 @@ const char *fsmonitor_ipc__get_path(struct repository *r)return ipc_path; } -SHA1_Init (&sha1ctx); -SHA1_Update (&sha1ctx, r->worktree,strlen (r->worktree)); -SHA1_Final (hash, &sha1ctx); +git_SHA1_Init (&sha1ctx); +git_SHA1_Update (&sha1ctx, r->worktree,strlen (r->worktree)); +git_SHA1_Final (hash, &sha1ctx);repo_config_get_string (r,"fsmonitor.socketdir" , &sock_dir);compat/fsmonitor/fsm-settings-unix.c renamed from compat/fsmonitor/fsm-settings-darwin.c +0
-0config.mak.uname +4
-0@@ -68,6 +68,7 @@ ifeq ($(uname_S),Linux) ifneq ($(findstring .el7.,$(uname_R)),) BASIC_CFLAGS += -std=c99 endif + endif ifeq ($(uname_S),GNU/kFreeBSD) HAVE_ALLOCA_H = YesPlease@@ -165,6 +166,7 @@ ifeq ($(uname_S),Darwin) ifndef NO_UNIX_SOCKETS FSMONITOR_DAEMON_BACKEND = darwin FSMONITOR_OS_SETTINGS = darwin + FSMONITOR_DAEMON_COMMON = unix endif endif@@ -453,6 +455,7 @@ ifeq ($(uname_S),Windows) # support it. FSMONITOR_DAEMON_BACKEND = win32 FSMONITOR_OS_SETTINGS = win32 + FSMONITOR_DAEMON_COMMON = win32 NO_SVN_TESTS = YesPlease RUNTIME_PREFIX = YesPlease@@ -645,6 +648,7 @@ ifeq ($(uname_S),MINGW) # support it. FSMONITOR_DAEMON_BACKEND = win32 FSMONITOR_OS_SETTINGS = win32 + FSMONITOR_DAEMON_COMMON = win32 RUNTIME_PREFIX = YesPlease HAVE_WPGMPTR = YesWeDocontrib/buildsystems/CMakeLists.txt +3
-3@@ -315,13 +315,13 @@ if(SUPPORTS_SIMPLE_IPC) list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c) elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-unix.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-unix.c) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-darwin.c) list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-darwin.c) add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-unix.c) endif() endif()
Quoted 9 lines · Eric DeCosta via GitGitGadget
Goal is to deliver fsmonitor for Linux that is on par with fsmonitor for Windows and Mac OS.
This patch set builds upon previous work for done for Windows and Mac OS (first 6 patches) to implement a fsmonitor back-end for Linux based on the Linux inotify API. inotify differs significantly from the equivalent Windows and Mac OS APIs in that a watch must be registered for every directory of interest (rather than a singular watch at the root of the directory tree) and special care must be taken to handle directory renames correctly.
I am seeing occasional breakages of t7527.16 that does not seem to reproduce reliably.
RE: [PATCH 08/12] fsmonitor: determine if filesystem is local or remote
In reply to Ævar Arnfjörð BjarmasonQuoted 62 lines
-----Original Message-----
From: Ævar Arnfjörð Bjarmason <[email protected]>
Sent: Monday, October 10, 2022 6:04 AM
To: Eric DeCosta via GitGitGadget <[email protected]>
Cc: [email protected]; Eric DeCosta <[email protected]>
Subject: Re: [PATCH 08/12] fsmonitor: determine if filesystem is local or remoteQuoted 9 lines · Eric DeCosta via GitGitGadget
From: Eric DeCosta <[email protected]>
Compare the given path to the mounted filesystems. Find the mount that is the longest prefix of the path (if any) and determine if that mount is on a local or remote filesystem.
Signed-off-by Eric DeCosta <[email protected]>---
compat/fsmonitor/fsm-path-utils-linux.c| 163++++++++++++++++++++++++
Quoted 10 lines
1 file changed, 163 insertions(+)create mode 100644
compat/fsmonitor/fsm-path-utils-linux.ccompat/fsmonitor/fsm-path-utils-linux.c new file +0
-0compat/fsmonitor/fsm-path-utils-linux.c line 10
Quoted 10 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +10
-0@@ -0,0 +1,163 @@ +#include "fsmonitor.h" +#include "fsmonitor-path-utils.h" +#include <errno.h> +#include <mntent.h> +#include <sys/mount.h> +#include <sys/statvfs.h> + +/* +* https://github.com/coreutils/gnulib/blob/master/lib/mountlist.c +<https://protect- us.mimecast.com/s/wKJsC9rj7lc3WKmYUOq9E9?domain=gith
Quoted 18 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +18
-0+ub.com> +*/ +#ifndef ME_REMOTE +/* A file system is "remote" if its Fs_name contains a ':' +or if (it is of type (smbfs or cifs) and its Fs_name starts with +'//') or if it is of any other of the listed types or Fs_name is +equal to "-hosts" (used by autofs to mount remote fs). +"VM" file systems like prl_fs or vboxsf are not considered remote +here. */ # define ME_REMOTE (Fs_name, Fs_type) \ (strchr (Fs_name, +':') != NULL \ + || ((Fs_name)[0 ] =='/' \ + && (Fs_name)[1 ] =='/' \ + && (strcmp (Fs_type,"smbfs" ) ==0 \ + ||strcmp (Fs_type,"smb3" ) ==0 \ + || strcmp (Fs_type, "cifs") == 0)) \ strcmp (Fs_type, "acfs") == 0 \ + || strcmp (Fs_type, "afs") == 0 \ strcmp (Fs_type, "coda") == 0 \ + || strcmp (Fs_type, "auristorfs") == 0 \ strcmp (Fs_type, "fhgfs") + || == 0 \ strcmp (Fs_type, "gpfs") == 0 \ strcmp (Fs_type, "ibrix")
Quoted 9 lines
compat/fsmonitor/fsm-path-utils-linux.c line 31
Quoted 3 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +3
-0+ || == 0 \ strcmp (Fs_type, "ocfs2") == 0 \ strcmp (Fs_type, "vxfs") + || == 0 \ strcmp ("-hosts", Fs_name) == 0) +#endif So, this is just copy/pasted GPLv3 code into our GPLv2-only codebase?:
https://github.com/coreutils/gnulib/blob/cd1fdabe8b66c102124b6a5b0c97d
ded20246b46/lib/mountlist.c#L230-L247 <https://protect-
us.mimecast.com/s/zxPbC0RMy1hoXW2rCOSXJD?domain=github.com>
Yes. I was hoping for some guidance as to what to do with ME_REMOTE.
I also found it, verbatim, here in midnight commander:
https://github.com/MidnightCommander/mc/blob/e48cd98ac13a9b4366bd88287f632413766b967f/src/filemanager/mountlist.c#L258-L281
But that's just another GPLv3 code base.
Quoted 107 lines
compat/fsmonitor/fsm-path-utils-linux.c line 40
Quoted 9 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +9
-0+static struct mntent *find_mount (const char *path,const struct +statvfs *fs) {const char *const mounts ="/proc/mounts" ;const +char *rp =real_pathdup (path,1 );struct mntent *ment =NULL ; +struct mntent *found =NULL ;struct statvfs mntfs;FILE *fp;int +dlen, plen, flen =0 ; + + fp =setmntent (mounts,"r" ); +if (!fp) { +error_errno (_ ("setmntent('%s') failed" ), mounts);return NULL ;This code would probably be nicer if you returned int, and passed a pointer to a struct to populate as a paremeter. Then you could just "return error..." for this and the below cases.
Quoted 17 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +17
-0+ } + + plen =strlen (rp); + +/* read all the mount information and compare to path */ while + ((ment =getmntent (fp)) !=NULL ) {if (statvfs (ment->mnt_dir, + &mntfs)) {switch (errno) {case EPERM : +case ESRCH : +case EACCES : +continue ; +default : +error_errno (_ ("statvfs('%s') failed" ), ment->mnt_dir); + endmntent(fp);is return NULL; } } + +/* is mount on the same filesystem and is a prefix of the path */ if + ((fs->f_fsid == mntfs.f_fsid) && !strncmp (ment->mnt_dir, rp, + strlen(ment->mnt_dir))) { dlen = strlen(ment->mnt_dir); if (dlen >compat/fsmonitor/fsm-path-utils-linux.c
Quoted 10 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +10
-0+ plen) continue; +/* +* root is always a potential match; otherwise look for +* directory prefix +*/ +if ((dlen ==1 && ment->mnt_dir[0 ] =='/' ) || (dlen > flen && + (!rp[dlen] || rp[dlen] =='/' ))) { flen = dlen; +/* +* https://man7.org/linux/man-pages/man3/getmntent.3.html +<https://protect- us.mimecast.com/s/aOmSCgJyVrT01WlYc76tSR?domain=man
compat/fsmonitor/fsm-path-utils-linux.c line 89
Quoted 7 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +7
-0+7.org> +* +* The pointer points to a static area of memory which is +* overwritten by subsequent calls to getmntent(). +*/ +if (!found) +CALLOC_ARRAY (found,1 );It seems we never populate >1 of these, so don't you just want
xmalloc(). Or actually...Quoted 8 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +5
-0+free (found->mnt_dir); +free (found->mnt_type); +free (found->mnt_fsname); + found->mnt_dir =xmemdupz (ment->mnt_dir,strlen (ment->mnt_dir)); + found->mnt_type = xmemdupz(ment->mnt_type, strlen(ment-
mnt_type));
+found->mnt_fsname= xmemdupz(ment->mnt_fsname,
+found->strlen(ment->mnt_fsname));Don't mix mem*() and str*(). In this case we need the string to be '\0' delimited, so use
xstrndup().And once we do that, we might wonder why we're explicitly finding the '\0', just to pass it to the xstrn*() function, when we can just do:
found->mnt_dir=xstrdup(ment->mnt_dir);
...Which would AFAICT be equivalent to what you're doing here.
compat/fsmonitor/fsm-path-utils-linux.c
Quoted 10 lines
+ }
+ }
+ }
+ endmntent(fp);compat/fsmonitor/fsm-path-utils-linux.c new file +6
-0+ +return found; +} + +int fsmonitor__get_fs_info (const char *path,struct fs_info *fs_info) +{struct mntent *ment;...make this (or above) a "struct mntent ment", then pass &ment down, so we can fill that struct? Dunno...
Quoted 14 lines
+ struct statvfs fs;
compat/fsmonitor/fsm-path-utils-linux.c new file +13
-0+ +if (statvfs (path, &fs)) +return error_errno (_ ("statvfs('%s') failed" ), path); + + ment =find_mount (path, &fs); +if (!ment) +return -1 ; + +trace_printf_key (&trace_fsmonitor, +"statvfs('%s') [flags 0x%08lx] '%s' '%s'" , path, fs.f_flag, + ment->mnt_type, ment->mnt_fsname); + + if (ME_REMOTE(ment->mnt_fsname, ment->mnt_type)) fs_info-Quoted 2 lines
is_remote
+ = 1; elsefs_info->is_remote= 0;Shorter:
fs_info->is_remote=ME_REMOTE(ment->mnt_fsname, ment->mnt_type);
PATCH v2 1/12 fsmonitor: refactor filesystem checks to common interface
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
Provide a common interface for getting basic filesystem information including filesystem type and whether the filesystem is remote.
Refactor existing code for getting basic filesystem info and detecting remote file systems to the new interface.
Refactor filesystem checks to leverage new interface. For macOS, error-out if the Unix Domain socket (UDS) file is on a remote filesystem.
--- Makefile | 1 + compat/fsmonitor/fsm-path-utils-darwin.c | 43 ++++++ compat/fsmonitor/fsm-path-utils-win32.c | 128 +++++++++++++++++ compat/fsmonitor/fsm-settings-darwin.c | 69 +++------ compat/fsmonitor/fsm-settings-win32.c | 172 +---------------------- contrib/buildsystems/CMakeLists.txt | 2 + fsmonitor-path-utils.h | 26 ++++ fsmonitor-settings.c | 50 +++++++ 8 files changed, 272 insertions(+), 219 deletions(-) create mode 100644 compat/fsmonitor/fsm-path-utils-darwin.c create mode 100644 compat/fsmonitor/fsm-path-utils-win32.c create mode 100644 fsmonitor-path-utils.h
Makefile +1 -0
@@ -2043,6 +2043,7 @@ endif ifdef FSMONITOR_OS_SETTINGS COMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o + COMPAT_OBJS += compat/fsmonitor/fsm-path-utils-$(FSMONITOR_OS_SETTINGS).o endif ifeq ($(TCLTK_PATH),)
compat/fsmonitor/fsm-path-utils-darwin.c new file +43 -0
43 lines not on this page · raw message
compat/fsmonitor/fsm-path-utils-win32.c new file +128 -0
128 lines not on this page · raw message
compat/fsmonitor/fsm-settings-darwin.c +20 -49
101 lines not on this page · raw message
compat/fsmonitor/fsm-settings-win32.c +2 -170
190 lines not on this page · raw message
contrib/buildsystems/CMakeLists.txt +2 -0
@@ -308,6 +308,7 @@ if(SUPPORTS_SIMPLE_IPC) add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c) list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-win32.c) add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c)
@@ -315,6 +316,7 @@ if(SUPPORTS_SIMPLE_IPC) add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c) list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-darwin.c) add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)
fsmonitor-path-utils.h new file +26 -0
@@ -0,0 +1,26 @@ +#ifndef FSM_PATH_UTILS_H +#define FSM_PATH_UTILS_H + +struct fs_info { +int is_remote; +char *typename; +}; + +/* +* Get some basic filesystem informtion for the given path +* +* The caller owns the storage that is occupied by fs_info and +* is responsible for releasing it. +* +* Returns -1 on error, zero otherwise. +*/ +int fsmonitor__get_fs_info (const char *path,struct fs_info *fs_info); + +/* +* Determines if the filesystem that path resides on is remote. +* +* Returns -1 on error, 0 if not remote, 1 if remote. +*/ +int fsmonitor__is_fs_remote (const char *path); + +#endif
fsmonitor-settings.c +50 -0
68 lines not on this page · raw message
Signature · 1 line
gitgitgadget
PATCH v2 2/12 fsmonitor: relocate socket file if .git directory is remote
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
If the .git directory is on a remote filesystem, create the socket file in 'fsmonitor.socketDir' if it is defined, else create it in $HOME.
--- Makefile | 1 + builtin/fsmonitor--daemon.c | 3 +- compat/fsmonitor/fsm-ipc-darwin.c | 52 ++++++++++++++++++++++++++ compat/fsmonitor/fsm-ipc-win32.c | 9 +++++ compat/fsmonitor/fsm-settings-darwin.c | 2 +- contrib/buildsystems/CMakeLists.txt | 2 + fsmonitor-ipc.c | 18 ++++----- fsmonitor-ipc.h | 4 +- 8 files changed, 78 insertions(+), 13 deletions(-) create mode 100644 compat/fsmonitor/fsm-ipc-darwin.c create mode 100644 compat/fsmonitor/fsm-ipc-win32.c
Makefile +1 -0
@@ -2038,6 +2038,7 @@ ifdef FSMONITOR_DAEMON_BACKEND COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o + COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_BACKEND).o endif ifdef FSMONITOR_OS_SETTINGS
builtin/fsmonitor--daemon.c +2 -1
@@ -1343,7 +1343,8 @@ static int fsmonitor_run_daemon(void)* directory.) */ strbuf_init (&state.path_ipc,0 ); -strbuf_addstr (&state.path_ipc,absolute_path (fsmonitor_ipc__get_path ())); +strbuf_addstr (&state.path_ipc, +absolute_path (fsmonitor_ipc__get_path (the_repository)));/* * Confirm that we can create platform-specific resources for the
compat/fsmonitor/fsm-ipc-darwin.c new file +52 -0
52 lines not on this page · raw message
compat/fsmonitor/fsm-ipc-win32.c new file +9 -0
@@ -0,0 +1,9 @@ +#include "config.h" +#include "fsmonitor-ipc.h" + +const char *fsmonitor_ipc__get_path (struct repository *r) { +static char *ret; +if (!ret) + ret =git_pathdup ("fsmonitor--daemon.ipc" ); +return ret; +}
compat/fsmonitor/fsm-settings-darwin.c +1 -1
@@ -26,7 +26,7 @@static enum fsmonitor_reason check_uds_volume (struct repository *r) {struct fs_info fs; -const char *ipc_path =fsmonitor_ipc__get_path (); +const char *ipc_path =fsmonitor_ipc__get_path (r);struct strbuf path =STRBUF_INIT ;strbuf_add (&path, ipc_path,strlen (ipc_path));
contrib/buildsystems/CMakeLists.txt +2 -0
@@ -308,6 +308,7 @@ if(SUPPORTS_SIMPLE_IPC) add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c) list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-win32.c) list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-win32.c) add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
@@ -316,6 +317,7 @@ if(SUPPORTS_SIMPLE_IPC) add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c) list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-darwin.c) list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-darwin.c) add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
fsmonitor-ipc.c +8 -10
@@ -18,7 +18,7 @@ int fsmonitor_ipc__is_supported(void)return 0 ; } -const char *fsmonitor_ipc__get_path (void ) +const char *fsmonitor_ipc__get_path (struct repository *r) {return NULL ; }
@@ -47,11 +47,9 @@ int fsmonitor_ipc__is_supported(void)return 1 ; } -GIT_PATH_FUNC(fsmonitor_ipc__get_path, "fsmonitor--daemon.ipc") -enum ipc_active_state fsmonitor_ipc__get_state (void ) { -return ipc_get_active_state (fsmonitor_ipc__get_path ()); +return ipc_get_active_state (fsmonitor_ipc__get_path (the_repository)); }static int spawn_daemon (void )
@@ -81,8 +79,8 @@ int fsmonitor_ipc__send_query(const char *since_token,trace2_data_string ("fsm_client" ,NULL ,"query/command" , tok); try_again: - state =ipc_client_try_connect (fsmonitor_ipc__get_path (), &options, - &connection); + state =ipc_client_try_connect (fsmonitor_ipc__get_path (the_repository), + &options, &connection);switch (state) {case IPC_STATE__LISTENING :
@@ -117,13 +115,13 @@ try_again:case IPC_STATE__INVALID_PATH : ret =error (_ ("fsmonitor_ipc__send_query: invalid path '%s'" ), -fsmonitor_ipc__get_path ()); +fsmonitor_ipc__get_path (the_repository)); goto done;case IPC_STATE__OTHER_ERROR :default : ret =error (_ ("fsmonitor_ipc__send_query: unspecified error on '%s'" ), -fsmonitor_ipc__get_path ()); +fsmonitor_ipc__get_path (the_repository)); goto done; }
@@ -149,8 +147,8 @@ int fsmonitor_ipc__send_command(const char *command, options.wait_if_busy =1 ; options.wait_if_not_found =0 ; - state =ipc_client_try_connect (fsmonitor_ipc__get_path (), &options, - &connection); + state =ipc_client_try_connect (fsmonitor_ipc__get_path (the_repository), + &options, &connection);if (state !=IPC_STATE__LISTENING ) {die (_ ("fsmonitor--daemon is not running" ));return -1 ;
fsmonitor-ipc.h +3 -1
@@ -3,6 +3,8 @@#include "simple-ipc.h" +struct repository ; +/* * Returns true if built-in file system monitor daemon is defined * for this platform.
@@ -16,7 +18,7 @@ int fsmonitor_ipc__is_supported(void);* * Returns NULL if the daemon is not supported on this platform. */ -const char *fsmonitor_ipc__get_path (void ); +const char *fsmonitor_ipc__get_path (struct repository *r);/* * Try to determine whether there is a `git-fsmonitor--daemon` process
Signature · 1 line
gitgitgadget
v2 cover fsmonitor: Implement fsmonitor for Linux
In reply to Eric DeCosta via GitGitGadgetGoal is to deliver fsmonitor for Linux that is on par with fsmonitor for Windows and Mac OS.
This patch set builds upon previous work for done for Windows and Mac OS (first 6 patches) to implement a fsmonitor back-end for Linux based on the Linux inotify API. inotify differs significantly from the equivalent Windows and Mac OS APIs in that a watch must be registered for every directory of interest (rather than a singular watch at the root of the directory tree) and special care must be taken to handle directory renames correctly.
More information about inotify:
https://man7.org/linux/man-pages/man7/inotify.7.html
v1 differs from v0:
* Code review feedback
* Update how and which code can be shared between Mac OS and Linux
* Increase polling frequency to every 1ms (matches Mac OS)
* Updates to t7527 to improve test stability
Eric DeCosta (12):
fsmonitor: refactor filesystem checks to common interface
fsmonitor: relocate socket file if .git directory is remote
fsmonitor: avoid socket location check if using hook
fsmonitor: deal with synthetic firmlinks on macOS
fsmonitor: check for compatability before communicating with fsmonitor
fsmonitor: add documentation for allowRemote and socketDir options
fsmonitor: prepare to share code between Mac OS and Linux
fsmonitor: determine if filesystem is local or remote
fsmonitor: implement filesystem change listener for Linux
fsmonitor: enable fsmonitor for Linux
fsmonitor: test updates
fsmonitor: update doc for Linux
Documentation/config.txt | 2 + Documentation/config/fsmonitor--daemon.txt | 11 + Documentation/git-fsmonitor--daemon.txt | 45 +- Makefile | 9 + builtin/fsmonitor--daemon.c | 11 +- compat/fsmonitor/fsm-health-linux.c | 24 + compat/fsmonitor/fsm-ipc-unix.c | 52 ++ compat/fsmonitor/fsm-ipc-win32.c | 9 + compat/fsmonitor/fsm-listen-darwin.c | 14 +- compat/fsmonitor/fsm-listen-linux.c | 664 +++++++++++++++++++++ compat/fsmonitor/fsm-path-utils-darwin.c | 135 +++++ compat/fsmonitor/fsm-path-utils-linux.c | 169 ++++++ compat/fsmonitor/fsm-path-utils-win32.c | 145 +++++ compat/fsmonitor/fsm-settings-darwin.c | 90 +-- compat/fsmonitor/fsm-settings-linux.c | 11 + compat/fsmonitor/fsm-settings-unix.c | 62 ++ compat/fsmonitor/fsm-settings-unix.h | 11 + compat/fsmonitor/fsm-settings-win32.c | 174 +----- config.mak.uname | 10 + contrib/buildsystems/CMakeLists.txt | 19 +- fsmonitor--daemon.h | 3 + fsmonitor-ipc.c | 18 +- fsmonitor-ipc.h | 4 +- fsmonitor-path-utils.h | 60 ++ fsmonitor-settings.c | 68 ++- fsmonitor-settings.h | 4 +- fsmonitor.c | 7 + t/t7527-builtin-fsmonitor.sh | 72 ++- 28 files changed, 1606 insertions(+), 297 deletions(-) create mode 100644 Documentation/config/fsmonitor--daemon.txt create mode 100644 compat/fsmonitor/fsm-health-linux.c create mode 100644 compat/fsmonitor/fsm-ipc-unix.c create mode 100644 compat/fsmonitor/fsm-ipc-win32.c create mode 100644 compat/fsmonitor/fsm-listen-linux.c create mode 100644 compat/fsmonitor/fsm-path-utils-darwin.c create mode 100644 compat/fsmonitor/fsm-path-utils-linux.c create mode 100644 compat/fsmonitor/fsm-path-utils-win32.c create mode 100644 compat/fsmonitor/fsm-settings-linux.c create mode 100644 compat/fsmonitor/fsm-settings-unix.c create mode 100644 compat/fsmonitor/fsm-settings-unix.h create mode 100644 fsmonitor-path-utils.h
base-commit: d420dda0576340909c3faff364cfbd1485f70376
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1352%2Fedecosta-mw%2Ffsmonitor_linux-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1352/edecosta-mw/fsmonitor_linux-v2
Pull-Request: https://github.com/git/git/pull/1352
Range-diff vs v1:
1: ec49a74086d = 1: cd46bed37c3 fsmonitor: refactor filesystem checks to common interface
2: 7bf1cdfe3b2 = 2: 21d114bda4b fsmonitor: relocate socket file if .git directory is remote
3: c5e8b6cfe5d = 3: 664259ed57a fsmonitor: avoid socket location check if using hook
4: 863063aefee = 4: d8f20032d6b fsmonitor: deal with synthetic firmlinks on macOS
5: fa974bfd5ef = 5: ab1e0ab967c fsmonitor: check for compatability before communicating with fsmonitor
6: af7309745f7 = 6: 9c552239b57 fsmonitor: add documentation for allowRemote and socketDir options
7: c085fc15b31 ! 7: 295beb89ab1 fsmonitor: prepare to share code between Mac OS and Linux
@@ Commit message
fsmonitor: prepare to share code between Mac OS and Linux
Linux and Mac OS can share some of the code originally developed for Mac OS.
- Rename the shared code from compat/fsmonitor/fsm-*-dawrin.c to
- compat/fsmonitor/fsm-*-unix.c
-
- Update the build to enable sharing of the fsm-*-unix.c files.
Minor update to compat/fsmonitor/fsm-ipc-unix.c to make it cross-platform.
+ Mac OS and Linux can share fsm-ipc-unix.c
+
+ Both platforms can also share compat/fsmonitor/fsm-settings-unix.c but we
+ will leave room for future, platform-specific checks by having the platform-
+ specific implementations call into fsm-settings-unix.
Signed-off-by: Eric DeCosta <[email protected]>
## Makefile ##
@@ Makefile: endif
+
ifdef FSMONITOR_DAEMON_BACKEND
COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND
++ ifdef FSMONITOR_DAEMON_COMMON
++ COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_COMMON).o
++ else
++ COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_BACKEND).o
++ endif
COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o
-- COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o
+ COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o
- COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_BACKEND).o
-+ COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_COMMON).o
-+ COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_COMMON).o
endif
ifdef FSMONITOR_OS_SETTINGS
COMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS
-- COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o
-+ COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_DAEMON_COMMON).o
++ ifdef FSMONITOR_DAEMON_COMMON
++ COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_DAEMON_COMMON).o
++ endif
+ COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o
COMPAT_OBJS += compat/fsmonitor/fsm-path-utils-$(FSMONITOR_OS_SETTINGS).o
endif
-
- ## compat/fsmonitor/fsm-health-darwin.c => compat/fsmonitor/fsm-health-unix.c ##
+ ## compat/fsmonitor/fsm-health-linux.c (new) ##
+@@
++#include "cache.h"
++#include "config.h"
++#include "fsmonitor.h"
++#include "fsm-health.h"
++#include "fsmonitor--daemon.h"
++
++int fsm_health__ctor(struct fsmonitor_daemon_state *state)
++{
++ return 0;
++}
++
++void fsm_health__dtor(struct fsmonitor_daemon_state *state)
++{
++ return;
++}
++
++void fsm_health__loop(struct fsmonitor_daemon_state *state)
++{
++ return;
++}
++
++void fsm_health__stop_async(struct fsmonitor_daemon_state *state)
++{
++}
## compat/fsmonitor/fsm-ipc-darwin.c => compat/fsmonitor/fsm-ipc-unix.c ##
@@ compat/fsmonitor/fsm-ipc-unix.c: static GIT_PATH_FUNC(fsmonitor_ipc__get_default_path, "fsmonitor--daemon.ipc")
@@ compat/fsmonitor/fsm-ipc-unix.c: const char *fsmonitor_ipc__get_path(struct repo
repo_config_get_string(r, "fsmonitor.socketdir", &sock_dir);
- ## compat/fsmonitor/fsm-settings-darwin.c => compat/fsmonitor/fsm-settings-unix.c ##
+ ## compat/fsmonitor/fsm-settings-darwin.c ##
+@@
+ #include "config.h"
+ #include "fsmonitor.h"
+ #include "fsmonitor-ipc.h"
+-#include "fsmonitor-settings.h"
+ #include "fsmonitor-path-utils.h"
+-
+- /*
+- * For the builtin FSMonitor, we create the Unix domain socket for the
+- * IPC in the .git directory. If the working directory is remote,
+- * then the socket will be created on the remote file system. This
+- * can fail if the remote file system does not support UDS file types
+- * (e.g. smbfs to a Windows server) or if the remote kernel does not
+- * allow a non-local process to bind() the socket. (These problems
+- * could be fixed by moving the UDS out of the .git directory and to a
+- * well-known local directory on the client machine, but care should
+- * be taken to ensure that $HOME is actually local and not a managed
+- * file share.)
+- *
+- * FAT32 and NTFS working directories are problematic too.
+- *
+- * The builtin FSMonitor uses a Unix domain socket in the .git
+- * directory for IPC. These Windows drive formats do not support
+- * Unix domain sockets, so mark them as incompatible for the daemon.
+- *
+- */
+-static enum fsmonitor_reason check_uds_volume(struct repository *r)
+-{
+- struct fs_info fs;
+- const char *ipc_path = fsmonitor_ipc__get_path(r);
+- struct strbuf path = STRBUF_INIT;
+- strbuf_add(&path, ipc_path, strlen(ipc_path));
+-
+- if (fsmonitor__get_fs_info(dirname(path.buf), &fs) == -1) {
+- strbuf_release(&path);
+- return FSMONITOR_REASON_ERROR;
+- }
+-
+- strbuf_release(&path);
+-
+- if (fs.is_remote ||
+- !strcmp(fs.typename, "msdos") ||
+- !strcmp(fs.typename, "ntfs")) {
+- free(fs.typename);
+- return FSMONITOR_REASON_NOSOCKETS;
+- }
+-
+- free(fs.typename);
+- return FSMONITOR_REASON_OK;
+-}
++#include "fsmonitor-settings.h"
++#include "fsm-settings-unix.h"
+
+ enum fsmonitor_reason fsm_os__incompatible(struct repository *r, int ipc)
+ {
+- enum fsmonitor_reason reason;
+-
+- if (ipc) {
+- reason = check_uds_volume(r);
+- if (reason != FSMONITOR_REASON_OK)
+- return reason;
+- }
+-
+- return FSMONITOR_REASON_OK;
++ return fsm_os__incompatible_unix(r, ipc);
+ }
+
+ ## compat/fsmonitor/fsm-settings-linux.c (new) ##
+@@
++#include "config.h"
++#include "fsmonitor.h"
++#include "fsmonitor-ipc.h"
++#include "fsmonitor-path-utils.h"
++#include "fsmonitor-settings.h"
++#include "fsm-settings-unix.h"
++
++enum fsmonitor_reason fsm_os__incompatible(struct repository *r, int ipc)
++{
++ return fsm_os__incompatible_unix(r, ipc);
++}
- ## config.mak.uname ##
-@@ config.mak.uname: ifeq ($(uname_S),Linux)
- ifneq ($(findstring .el7.,$(uname_R)),)
- BASIC_CFLAGS += -std=c99
- endif
+ ## compat/fsmonitor/fsm-settings-unix.c (new) ##
+@@
++#include "config.h"
++#include "fsmonitor.h"
++#include "fsmonitor-ipc.h"
++#include "fsmonitor-path-utils.h"
++#include "fsm-settings-unix.h"
+
- endif
- ifeq ($(uname_S),GNU/kFreeBSD)
- HAVE_ALLOCA_H = YesPlease
++ /*
++ * For the builtin FSMonitor, we create the Unix domain socket for the
++ * IPC in the .git directory. If the working directory is remote,
++ * then the socket will be created on the remote file system. This
++ * can fail if the remote file system does not support UDS file types
++ * (e.g. smbfs to a Windows server) or if the remote kernel does not
++ * allow a non-local process to bind() the socket. (These problems
++ * could be fixed by moving the UDS out of the .git directory and to a
++ * well-known local directory on the client machine, but care should
++ * be taken to ensure that $HOME is actually local and not a managed
++ * file share.)
++ *
++ * FAT32 and NTFS working directories are problematic too.
++ *
++ * The builtin FSMonitor uses a Unix domain socket in the .git
++ * directory for IPC. These Windows drive formats do not support
++ * Unix domain sockets, so mark them as incompatible for the daemon.
++ *
++ */
++static enum fsmonitor_reason check_uds_volume(struct repository *r)
++{
++ struct fs_info fs;
++ const char *ipc_path = fsmonitor_ipc__get_path(r);
++ struct strbuf path = STRBUF_INIT;
++ strbuf_add(&path, ipc_path, strlen(ipc_path));
++
++ if (fsmonitor__get_fs_info(dirname(path.buf), &fs) == -1) {
++ strbuf_release(&path);
++ return FSMONITOR_REASON_ERROR;
++ }
++
++ strbuf_release(&path);
++
++ if (fs.is_remote ||
++ !strcmp(fs.typename, "msdos") ||
++ !strcmp(fs.typename, "ntfs")) {
++ free(fs.typename);
++ return FSMONITOR_REASON_NOSOCKETS;
++ }
++
++ free(fs.typename);
++ return FSMONITOR_REASON_OK;
++}
++
++enum fsmonitor_reason fsm_os__incompatible_unix(struct repository *r, int ipc)
++{
++ enum fsmonitor_reason reason;
++
++ if (ipc) {
++ reason = check_uds_volume(r);
++ if (reason != FSMONITOR_REASON_OK)
++ return reason;
++ }
++
++ return FSMONITOR_REASON_OK;
++}
+
+ ## compat/fsmonitor/fsm-settings-unix.h (new) ##
+@@
++#ifndef FSM_SETTINGS_UNIX_H
++#define FSM_SETTINGS_UNIX_H
++
++#ifdef HAVE_FSMONITOR_OS_SETTINGS
++/*
++ * Check for compatibility on unix-like systems (e.g. Darwin and Linux)
++ */
++enum fsmonitor_reason fsm_os__incompatible_unix(struct repository *r, int ipc);
++#endif /* HAVE_FSMONITOR_OS_SETTINGS */
++
++#endif /* FSM_SETTINGS_UNIX_H */
+
+ ## config.mak.uname ##
@@ config.mak.uname: ifeq ($(uname_S),Darwin)
ifndef NO_UNIX_SOCKETS
FSMONITOR_DAEMON_BACKEND = darwin
@@ config.mak.uname: ifeq ($(uname_S),Darwin)
endif
endif
-@@ config.mak.uname: ifeq ($(uname_S),Windows)
- # support it.
- FSMONITOR_DAEMON_BACKEND = win32
- FSMONITOR_OS_SETTINGS = win32
-+ FSMONITOR_DAEMON_COMMON = win32
-
- NO_SVN_TESTS = YesPlease
- RUNTIME_PREFIX = YesPlease
-@@ config.mak.uname: ifeq ($(uname_S),MINGW)
- # support it.
- FSMONITOR_DAEMON_BACKEND = win32
- FSMONITOR_OS_SETTINGS = win32
-+ FSMONITOR_DAEMON_COMMON = win32
-
- RUNTIME_PREFIX = YesPlease
- HAVE_WPGMPTR = YesWeDo
## contrib/buildsystems/CMakeLists.txt ##
+@@ contrib/buildsystems/CMakeLists.txt: else()
+ endif()
+
+ if(SUPPORTS_SIMPLE_IPC)
+- if(CMAKE_SYSTEM_NAME STREQUAL "Windows")
++ if(CMAKE_SYSTEM_NAME STREQUAL "Linux")
++ add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
++ list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-unix.c)
++ list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-linux.c)
++ list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-linux.c)
++ list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-linux.c)
++
++ add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
++ list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-unix.c)
++ list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-linux.c)
++ elseif(CMAKE_SYSTEM_NAME STREQUAL "Windows")
+ add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c)
@@ contrib/buildsystems/CMakeLists.txt: if(SUPPORTS_SIMPLE_IPC)
list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c)
elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin")
add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
+- list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-unix.c)
-+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-unix.c)
- list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)
-- list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c)
- list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-darwin.c)
++ list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)
list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-darwin.c)
add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
-- list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-unix.c)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)
endif()
endif()
-
8: 5ecbb3082f1 ! 8: 7d7ef78728f fsmonitor: determine if filesystem is local or remote
@@ compat/fsmonitor/fsm-path-utils-linux.c (new)
+ || strcmp ("-hosts", Fs_name) == 0)
+#endif
+
-+static struct mntent *find_mount(const char *path, const struct statvfs *fs)
++static int find_mount(const char *path, const struct statvfs *fs,
++ struct mntent *ent)
+{
+ const char *const mounts = "/proc/mounts";
+ const char *rp = real_pathdup(path, 1);
+ struct mntent *ment = NULL;
-+ struct mntent *found = NULL;
+ struct statvfs mntfs;
+ FILE *fp;
++ int found = 0;
+ int dlen, plen, flen = 0;
+
++ ent->mnt_fsname = NULL;
++ ent->mnt_dir = NULL;
++ ent->mnt_type = NULL;
++
+ fp = setmntent(mounts, "r");
+ if (!fp) {
+ error_errno(_("setmntent('%s') failed"), mounts);
-+ return NULL;
++ return -1;
+ }
+
+ plen = strlen(rp);
@@ compat/fsmonitor/fsm-path-utils-linux.c (new)
+ default:
+ error_errno(_("statvfs('%s') failed"), ment->mnt_dir);
+ endmntent(fp);
-+ return NULL;
++ return -1;
+ }
+ }
+
@@ compat/fsmonitor/fsm-path-utils-linux.c (new)
+ * The pointer points to a static area of memory which is
+ * overwritten by subsequent calls to getmntent().
+ */
-+ if (!found)
-+ CALLOC_ARRAY(found, 1);
-+ free(found->mnt_dir);
-+ free(found->mnt_type);
-+ free(found->mnt_fsname);
-+ found->mnt_dir = xmemdupz(ment->mnt_dir, strlen(ment->mnt_dir));
-+ found->mnt_type = xmemdupz(ment->mnt_type, strlen(ment->mnt_type));
-+ found->mnt_fsname = xmemdupz(ment->mnt_fsname, strlen(ment->mnt_fsname));
++ found = 1;
++ free(ent->mnt_fsname);
++ free(ent->mnt_dir);
++ free(ent->mnt_type);
++ ent->mnt_fsname = xstrdup(ment->mnt_fsname);
++ ent->mnt_dir = xstrdup(ment->mnt_dir);
++ ent->mnt_type = xstrdup(ment->mnt_type);
+ }
+ }
+ }
+ endmntent(fp);
+
-+ return found;
++ if (!found)
++ return -1;
++
++ return 0;
+}
+
+int fsmonitor__get_fs_info(const char *path, struct fs_info *fs_info)
+{
-+ struct mntent *ment;
++ struct mntent ment;
+ struct statvfs fs;
+
+ if (statvfs(path, &fs))
+ return error_errno(_("statvfs('%s') failed"), path);
+
-+ ment = find_mount(path, &fs);
-+ if (!ment)
++
++ if (find_mount(path, &fs, &ment) < 0) {
++ free(ment.mnt_fsname);
++ free(ment.mnt_dir);
++ free(ment.mnt_type);
+ return -1;
++ }
+
+ trace_printf_key(&trace_fsmonitor,
+ "statvfs('%s') [flags 0x%08lx] '%s' '%s'",
-+ path, fs.f_flag, ment->mnt_type, ment->mnt_fsname);
-+
-+ if (ME_REMOTE(ment->mnt_fsname, ment->mnt_type))
-+ fs_info->is_remote = 1;
-+ else
-+ fs_info->is_remote = 0;
++ path, fs.f_flag, ment.mnt_type, ment.mnt_fsname);
+
-+ fs_info->typename = ment->mnt_fsname;
-+ free(ment->mnt_dir);
-+ free(ment->mnt_type);
-+ free(ment);
++ fs_info->is_remote = ME_REMOTE(ment.mnt_fsname, ment.mnt_type);
++ fs_info->typename = ment.mnt_fsname;
++ free(ment.mnt_dir);
++ free(ment.mnt_type);
+
+ trace_printf_key(&trace_fsmonitor,
+ "'%s' is_remote: %d",
9: 7465fe7a8b4 ! 9: 4f9c5358475 fsmonitor: implement filesystem change listener for Linux
@@ compat/fsmonitor/fsm-listen-linux.c (new)
+
+/*
+ * Non-blocking read of the inotify events stream. The inotify fd is polled
-+ * every few millisec to help minimize the number of queue overflows.
++ * frequently to help minimize the number of queue overflows.
+ */
+void fsm_listen__loop(struct fsmonitor_daemon_state *state)
+{
@@ compat/fsmonitor/fsm-listen-linux.c (new)
+ for(;;) {
+ switch (state->listen_data->shutdown) {
+ case SHUTDOWN_CONTINUE:
-+ poll_num = poll(fds, 1, 10);
++ poll_num = poll(fds, 1, 1);
+ if (poll_num == -1) {
+ if (errno == EINTR)
+ continue; 10: 3a2db9aa076 ! 10: 07650ecd27b fsmonitor: enable fsmonitor for Linux
@@ Metadata
## Commit message ##
fsmonitor: enable fsmonitor for Linux
- Uodate build to enable fsmonitor for Linux.
+ Update build to enable fsmonitor for Linux.
## config.mak.uname ##
@@ config.mak.uname: ifeq ($(uname_S),Linux)
+ ifneq ($(findstring .el7.,$(uname_R)),)
BASIC_CFLAGS += -std=c99
endif
-
+ # The builtin FSMonitor on Linux builds upon Simple-IPC. Both require
+ # Unix domain sockets and PThreads.
+ ifndef NO_PTHREADS
@@ config.mak.uname: ifeq ($(uname_S),Linux)
endif
ifeq ($(uname_S),GNU/kFreeBSD)
HAVE_ALLOCA_H = YesPlease
-
- ## contrib/buildsystems/CMakeLists.txt ##
-@@ contrib/buildsystems/CMakeLists.txt: else()
- endif()
-
- if(SUPPORTS_SIMPLE_IPC)
-- if(CMAKE_SYSTEM_NAME STREQUAL "Windows")
-+ if(CMAKE_SYSTEM_NAME STREQUAL "Linux")
-+ add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
-+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-unix.c)
-+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-unix.c)
-+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-linux.c)
-+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-linux.c)
-+
-+ add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
-+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-unix.c)
-+ elseif(CMAKE_SYSTEM_NAME STREQUAL "Windows")
- add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
- list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)
- list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c)
11: 743bdacded5 ! 11: 6682938fff8 fsmonitor: test updates
@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'edit some files' '
test-tool fsmonitor-client query --token 0 &&
++ test_might_fail git fsmonitor--daemon stop &&
++
+ grep "^event: dir1/modified$" .git/trace &&
+ grep "^event: dir2/modified$" .git/trace &&
+ grep "^event: modified$" .git/trace &&
@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'create some files' '
start_daemon --tf "$PWD/.git/trace" &&
@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'create some files' '
test-tool fsmonitor-client query --token 0 &&
++ test_might_fail git fsmonitor--daemon stop &&
++
+ grep "^event: dir1/new$" .git/trace &&
+ grep "^event: dir2/new$" .git/trace &&
+ grep "^event: new$" .git/trace
@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'delete some files' '
start_daemon --tf "$PWD/.git/trace" &&
@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'delete some files' '
test-tool fsmonitor-client query --token 0 &&
++ test_might_fail git fsmonitor--daemon stop &&
++
+ grep "^event: dir1/delete$" .git/trace &&
+ grep "^event: dir2/delete$" .git/trace &&
+ grep "^event: delete$" .git/trace
@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'rename some files' '
start_daemon --tf "$PWD/.git/trace" &&
@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'rename some files' '
test-tool fsmonitor-client query --token 0 &&
++ test_might_fail git fsmonitor--daemon stop &&
++
+ grep "^event: dir1/rename$" .git/trace &&
+ grep "^event: dir2/rename$" .git/trace &&
+ grep "^event: rename$" .git/trace &&
@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'rename directory' '
start_daemon --tf "$PWD/.git/trace" &&
@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'rename directory' '
test-tool fsmonitor-client query --token 0 &&
++ test_might_fail git fsmonitor--daemon stop &&
++
+ grep "^event: dirtorename/*$" .git/trace &&
+ grep "^event: dirrenamed/*$" .git/trace
+ '
@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'file changes to directory' '
start_daemon --tf "$PWD/.git/trace" &&
@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'file changes to directory' '
test-tool fsmonitor-client query --token 0 &&
++ test_might_fail git fsmonitor--daemon stop &&
++
+ grep "^event: delete$" .git/trace &&
+ grep "^event: delete/new$" .git/trace
+ '
@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'directory changes to a file' '
start_daemon --tf "$PWD/.git/trace" &&
@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'directory changes to a file'
test-tool fsmonitor-client query --token 0 &&
++ test_might_fail git fsmonitor--daemon stop &&
++
+ grep "^event: dir1$" .git/trace
+ '
+
@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'flush cached data' '
test-tool -C test_flush fsmonitor-client query --token "builtin:test_00000002:0" >actual_2 && nul_to_q <actual_2 >actual_q2 &&
12: 77ed35b3b80 = 12: dd73e126810 fsmonitor: update doc for Linux
Signature · 1 line
gitgitgadget
PATCH v2 5/12 fsmonitor: check for compatability before communicating with fsmonitor
From: Eric DeCosta <[email protected]>
If fsmonitor is not in a compatible state, warn with an appropriate message.
--- fsmonitor-settings.c | 10 +++++++--- fsmonitor-settings.h | 2 +- fsmonitor.c | 7 +++++++ 3 files changed, 15 insertions(+), 4 deletions(-)
fsmonitor-settings.c +7 -3
@@ -1,6 +1,7 @@#include "cache.h" #include "config.h" #include "repository.h" +#include "fsmonitor-ipc.h" #include "fsmonitor-settings.h" #include "fsmonitor-path-utils.h"
@@ -242,10 +243,11 @@ enum fsmonitor_reason fsm_settings__get_reason(struct repository *r)return r->settings.fsmonitor->reason; } -char *fsm_settings__get_incompatible_msg (const struct repository *r, +char *fsm_settings__get_incompatible_msg (struct repository *r,enum fsmonitor_reason reason) {struct strbuf msg =STRBUF_INIT ; +const char *socket_dir;switch (reason) {case FSMONITOR_REASON_UNTESTED :
@@ -281,9 +283,11 @@ char *fsm_settings__get_incompatible_msg(const struct repository *r, goto done;case FSMONITOR_REASON_NOSOCKETS : + socket_dir =dirname ((char *)fsmonitor_ipc__get_path (r));strbuf_addf (&msg, -_ ("repository '%s' is incompatible with fsmonitor due to lack of Unix sockets" ), - r->worktree); +_ ("socket directory '%s' is incompatible with fsmonitor due" +" to lack of Unix sockets support" ), + socket_dir); goto done; }
fsmonitor-settings.h +1 -1
@@ -33,7 +33,7 @@ enum fsmonitor_mode fsm_settings__get_mode(struct repository *r);const char *fsm_settings__get_hook_path (struct repository *r);enum fsmonitor_reason fsm_settings__get_reason (struct repository *r); -char *fsm_settings__get_incompatible_msg (const struct repository *r, +char *fsm_settings__get_incompatible_msg (struct repository *r,enum fsmonitor_reason reason);struct fsmonitor_settings ;
fsmonitor.c +7 -0
@@ -295,6 +295,7 @@ static int fsmonitor_force_update_threshold = 100;void refresh_fsmonitor (struct index_state *istate) { +static int warn_once =0 ;struct strbuf query_result =STRBUF_INIT ;int query_success =0 , hook_version =-1 ;size_t bol =0 ;/* beginning of line */
@@ -305,6 +306,12 @@ void refresh_fsmonitor(struct index_state *istate)int is_trivial =0 ;struct repository *r = istate->repo ? istate->repo : the_repository;enum fsmonitor_mode fsm_mode =fsm_settings__get_mode (r); +enum fsmonitor_reason reason =fsm_settings__get_reason (r); + +if (!warn_once && reason >FSMONITOR_REASON_OK ) { + warn_once =1 ; +warning ("%s" ,fsm_settings__get_incompatible_msg (r, reason)); + }if (fsm_mode <=FSMONITOR_MODE_DISABLED || istate->fsmonitor_has_run_once)
Signature · 1 line
gitgitgadget
PATCH v2 3/12 fsmonitor: avoid socket location check if using hook
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
If monitoring is done via fsmonitor hook rather than IPC there is no need to check if the location of the Unix Domain socket (UDS) file is on a remote filesystem.
--- compat/fsmonitor/fsm-settings-darwin.c | 10 ++++++---- compat/fsmonitor/fsm-settings-win32.c | 2 +- fsmonitor-settings.c | 8 ++++---- fsmonitor-settings.h | 2 +- 4 files changed, 12 insertions(+), 10 deletions(-)
compat/fsmonitor/fsm-settings-darwin.c +6 -4
@@ -48,13 +48,15 @@ static enum fsmonitor_reason check_uds_volume(struct repository *r)return FSMONITOR_REASON_OK ; } -enum fsmonitor_reason fsm_os__incompatible (struct repository *r) +enum fsmonitor_reason fsm_os__incompatible (struct repository *r,int ipc) {enum fsmonitor_reason reason; - reason =check_uds_volume (r); -if (reason !=FSMONITOR_REASON_OK ) -return reason; +if (ipc) { + reason =check_uds_volume (r); +if (reason !=FSMONITOR_REASON_OK ) +return reason; + }return FSMONITOR_REASON_OK ; }
compat/fsmonitor/fsm-settings-win32.c +1 -1
@@ -25,7 +25,7 @@ static enum fsmonitor_reason check_vfs4git(struct repository *r)return FSMONITOR_REASON_OK ; } -enum fsmonitor_reason fsm_os__incompatible (struct repository *r) +enum fsmonitor_reason fsm_os__incompatible (struct repository *r,int ipc) {enum fsmonitor_reason reason;
fsmonitor-settings.c +4 -4
@@ -60,7 +60,7 @@ static enum fsmonitor_reason check_remote(struct repository *r) }#endif -static enum fsmonitor_reason check_for_incompatible (struct repository *r) +static enum fsmonitor_reason check_for_incompatible (struct repository *r,int ipc) {if (!r->worktree) {/*
@@ -77,7 +77,7 @@ static enum fsmonitor_reason check_for_incompatible(struct repository *r) reason =check_remote (r);if (reason !=FSMONITOR_REASON_OK )return reason; - reason =fsm_os__incompatible (r); + reason =fsm_os__incompatible (r, ipc);if (reason !=FSMONITOR_REASON_OK )return reason; }
@@ -162,7 +162,7 @@ const char *fsm_settings__get_hook_path(struct repository *r)void fsm_settings__set_ipc (struct repository *r) { -enum fsmonitor_reason reason =check_for_incompatible (r); +enum fsmonitor_reason reason =check_for_incompatible (r,1 );if (reason !=FSMONITOR_REASON_OK ) {fsm_settings__set_incompatible (r, reason);
@@ -185,7 +185,7 @@ void fsm_settings__set_ipc(struct repository *r)void fsm_settings__set_hook (struct repository *r,const char *path) { -enum fsmonitor_reason reason =check_for_incompatible (r); +enum fsmonitor_reason reason =check_for_incompatible (r,0 );if (reason !=FSMONITOR_REASON_OK ) {fsm_settings__set_incompatible (r, reason);
fsmonitor-settings.h +1 -1
@@ -48,7 +48,7 @@ struct fsmonitor_settings;* fsm_os__* routines should considered private to fsm_settings__ * routines. */ -enum fsmonitor_reason fsm_os__incompatible (struct repository *r); +enum fsmonitor_reason fsm_os__incompatible (struct repository *r,int ipc);#endif /* HAVE_FSMONITOR_OS_SETTINGS */ #endif /* FSMONITOR_SETTINGS_H */
Signature · 1 line
gitgitgadget
PATCH v2 4/12 fsmonitor: deal with synthetic firmlinks on macOS
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
Starting with macOS 10.15 (Catalina), Apple introduced a new feature called 'firmlinks' in order to separate the boot volume into two volumes, one read-only and one writable but still present them to the user as a single volume. Along with this change, Apple removed the ability to create symlinks in the root directory and replaced them with 'synthetic firmlinks'. See 'man synthetic.conf'
When FSEevents reports the path of changed files, if the path involves a synthetic firmlink, the path is reported from the point of the synthetic firmlink and not the real path. For example:
Real path:
/System/Volumes/Data/network/working/directory/foo.txt
Synthetic firmlink:
/network -> /System/Volumes/Data/network
FSEvents path:
/network/working/directory/foo.txt
This causes the FSEvents path to not match against the worktree directory.
There are several ways in which synthetic firmlinks can be created:
they can be defined in /etc/synthetic.conf, the automounter can create them, and there may be other means. Simply reading /etc/synthetic.conf is insufficient. No matter what process creates synthetic firmlinks, they all get created in the root directory.
Therefore, in order to deal with synthetic firmlinks, the root directory is scanned and the first possible synthetic firmink that, when resolved, is a prefix of the worktree is used to map FSEvents paths to worktree paths.
--- builtin/fsmonitor--daemon.c | 8 +++ compat/fsmonitor/fsm-listen-darwin.c | 14 +++- compat/fsmonitor/fsm-path-utils-darwin.c | 92 ++++++++++++++++++++++++ compat/fsmonitor/fsm-path-utils-win32.c | 17 +++++ fsmonitor--daemon.h | 3 + fsmonitor-path-utils.h | 36 +++++++++- 6 files changed, 167 insertions(+), 3 deletions(-)
builtin/fsmonitor--daemon.c +8 -0
@@ -3,6 +3,7 @@#include "parse-options.h" #include "fsmonitor.h" #include "fsmonitor-ipc.h" +#include "fsmonitor-path-utils.h" #include "compat/fsmonitor/fsm-health.h" #include "compat/fsmonitor/fsm-listen.h" #include "fsmonitor--daemon.h"
@@ -1282,6 +1283,11 @@ static int fsmonitor_run_daemon(void)strbuf_addstr (&state.path_worktree_watch,absolute_path (get_git_work_tree ())); state.nr_paths_watching =1 ; +strbuf_init (&state.alias.alias,0 ); +strbuf_init (&state.alias.points_to,0 ); +if ((err =fsmonitor__get_alias (state.path_worktree_watch.buf, &state.alias))) + goto done; +/* * We create and delete cookie files somewhere inside the .git * directory to help us keep sync with the file system. If
@@ -1391,6 +1397,8 @@ done:strbuf_release (&state.path_gitdir_watch);strbuf_release (&state.path_cookie_prefix);strbuf_release (&state.path_ipc); +strbuf_release (&state.alias.alias); +strbuf_release (&state.alias.points_to);return err; }
compat/fsmonitor/fsm-listen-darwin.c +12 -2
@@ -26,6 +26,7 @@#include "fsmonitor.h" #include "fsm-listen.h" #include "fsmonitor--daemon.h" +#include "fsmonitor-path-utils.h" struct fsm_listen_data {
@@ -198,8 +199,9 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,struct string_list cookie_list =STRING_LIST_INIT_DUP ;const char *path_k;const char *slash; -int k; +char *resolved =NULL ;struct strbuf tmp =STRBUF_INIT ; +int k;/* * Build a list of all filesystem changes into a private/local
@@ -209,7 +211,12 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,/* * On Mac, we receive an array of absolute paths. */ - path_k = paths[k]; +free (resolved); + resolved =fsmonitor__resolve_alias (paths[k], &state->alias); +if (resolved) + path_k = resolved; +else + path_k = paths[k];/* * If you want to debug FSEvents, log them to GIT_TRACE_FSMONITOR.
@@ -238,6 +245,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,fsmonitor_force_resync (state);fsmonitor_batch__free_list (batch);string_list_clear (&cookie_list,0 ); + batch =NULL ;/* * We assume that any events that we received
@@ -360,12 +368,14 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef, } } +free (resolved);fsmonitor_publish (state, batch, &cookie_list);string_list_clear (&cookie_list,0 );strbuf_release (&tmp);return ; force_shutdown: +free (resolved);fsmonitor_batch__free_list (batch);string_list_clear (&cookie_list,0 );
compat/fsmonitor/fsm-path-utils-darwin.c +92 -0
100 lines not on this page · raw message
compat/fsmonitor/fsm-path-utils-win32.c +17 -0
@@ -126,3 +126,20 @@ int fsmonitor__is_fs_remote(const char *path)return -1 ;return fs.is_remote; } + +/* +* No-op for now. +*/ +int fsmonitor__get_alias (const char *path,struct alias_info *info) +{ +return 0 ; +} + +/* +* No-op for now. +*/ +char *fsmonitor__resolve_alias (const char *path, +const struct alias_info *info) +{ +return NULL ; +}
fsmonitor--daemon.h +3 -0
@@ -8,6 +8,7 @@#include "run-command.h" #include "simple-ipc.h" #include "thread-utils.h" +#include "fsmonitor-path-utils.h" struct fsmonitor_batch ;struct fsmonitor_token_data ;
@@ -43,6 +44,7 @@ struct fsmonitor_daemon_state {struct strbuf path_worktree_watch;struct strbuf path_gitdir_watch; +struct alias_info alias;int nr_paths_watching;struct fsmonitor_token_data *current_token_data;
@@ -59,6 +61,7 @@ struct fsmonitor_daemon_state {struct ipc_server_data *ipc_server_data;struct strbuf path_ipc; + };/*
fsmonitor-path-utils.h +35 -1
@@ -1,13 +1,21 @@#ifndef FSM_PATH_UTILS_H #define FSM_PATH_UTILS_H +#include "strbuf.h" + +struct alias_info +{ +struct strbuf alias; +struct strbuf points_to; +}; +struct fs_info {int is_remote;char *typename; };/* -* Get some basic filesystem informtion for the given path +* Get some basic filesystem information for the given path * * The caller owns the storage that is occupied by fs_info and * is responsible for releasing it.
@@ -23,4 +31,30 @@ int fsmonitor__get_fs_info(const char *path, struct fs_info *fs_info);*/ int fsmonitor__is_fs_remote (const char *path); +/* +* Get the alias in given path, if any. +* +* Sets alias to the first alias that matches any part of the path. +* +* If an alias is found, info.alias and info.points_to are set to the +* found mapping. +* +* Returns -1 on error, 0 otherwise. +* +* The caller owns the storage that is occupied by info.alias and +* info.points_to and is responsible for releasing it. +*/ +int fsmonitor__get_alias (const char *path,struct alias_info *info); + +/* +* Resolve the path against the given alias. +* +* Returns the resolved path if there is one, NULL otherwise. +* +* The caller owns the storage that the returned string occupies and +* is responsible for releasing it. +*/ +char *fsmonitor__resolve_alias (const char *path, +const struct alias_info *info); +#endif
Signature · 1 line
gitgitgadget
PATCH v2 6/12 fsmonitor: add documentation for allowRemote and socketDir options
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
Add documentation for 'fsmonitor.allowRemote' and 'fsmonitor.socketDir'. Call-out experimental nature of 'fsmonitor.allowRemote' and limited filesystem support for 'fsmonitor.socketDir'.
--- Documentation/config.txt | 2 ++ Documentation/config/fsmonitor--daemon.txt | 11 +++++++ Documentation/git-fsmonitor--daemon.txt | 37 ++++++++++++++++++++-- 3 files changed, 47 insertions(+), 3 deletions(-) create mode 100644 Documentation/config/fsmonitor--daemon.txt
Documentation/config.txt +2 -0
@@ -423,6 +423,8 @@ include::config/filter.txt[] include::config/fsck.txt[] +include::config/fsmonitor--daemon.txt[] + include::config/gc.txt[] include::config/gitcvs.txt[]
Documentation/config/fsmonitor--daemon.txt new file +11 -0
@@ -0,0 +1,11 @@ +fsmonitor.allowRemote:: + By default, the fsmonitor daemon refuses to work against network-mounted + repositories. Setting `fsmonitor.allowRemote` to `true` overrides this + behavior. Only respected when `core.fsmonitor` is set to `true`. + +fsmonitor.socketDir:: + This Mac OS-specific option, if set, specifies the directory in + which to create the Unix domain socket used for communication + between the fsmonitor daemon and various Git commands. The directory must + reside on a native Mac OS filesystem. Only respected when `core.fsmonitor` + is set to `true`.
Documentation/git-fsmonitor--daemon.txt +34 -3
@@ -3,7 +3,7 @@ git-fsmonitor{litdd}daemon(1) NAME ---- -git-fsmonitor--daemon - A Built-in File System Monitor +git-fsmonitor--daemon - A Built-in Filesystem Monitor SYNOPSIS --------
@@ -17,7 +17,7 @@ DESCRIPTION ----------- A daemon to watch the working directory for file and directory -changes using platform-specific file system notification facilities. +changes using platform-specific filesystem notification facilities. This daemon communicates directly with commands like `git status` using the link:technical/api-simple-ipc.html[simple IPC] interface
@@ -63,13 +63,44 @@ CAVEATS ------- The fsmonitor daemon does not currently know about submodules and does -not know to filter out file system events that happen within a +not know to filter out filesystem events that happen within a submodule. If fsmonitor daemon is watching a super repo and a file is modified within the working directory of a submodule, it will report the change (as happening against the super repo). However, the client will properly ignore these extra events, so performance may be affected but it will not cause an incorrect result. +By default, the fsmonitor daemon refuses to work against network-mounted +repositories; this may be overridden by setting `fsmonitor.allowRemote` to +`true`. Note, however, that the fsmonitor daemon is not guaranteed to work +correctly with all network-mounted repositories and such use is considered +experimental. + +On Mac OS, the inter-process communication (IPC) between various Git +commands and the fsmonitor daemon is done via a Unix domain socket (UDS) -- a +special type of file -- which is supported by native Mac OS filesystems, +but not on network-mounted filesystems, NTFS, or FAT32. Other filesystems +may or may not have the needed support; the fsmonitor daemon is not guaranteed +to work with these filesystems and such use is considered experimental. + +By default, the socket is created in the `.git` directory, however, if the +`.git` directory is on a network-mounted filesystem, it will be instead be +created at `$HOME/.git-fsmonitor-*` unless `$HOME` itself is on a +network-mounted filesystem in which case you must set the configuration +variable `fsmonitor.socketDir` to the path of a directory on a Mac OS native +filesystem in which to create the socket file. + +If none of the above directories (`.git`, `$HOME`, or `fsmonitor.socketDir`) +is on a native Mac OS file filesystem the fsmonitor daemon will report an +error that will cause the daemon and the currently running command to exit. + +CONFIGURATION +------------- + +include::includes/cmd-config-section-all.txt[] + +include::config/fsmonitor--daemon.txt[] + GIT --- Part of the linkgit:git[1] suite
Signature · 1 line
gitgitgadget
PATCH v2 7/12 fsmonitor: prepare to share code between Mac OS and Linux
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
Linux and Mac OS can share some of the code originally developed for Mac OS.
Minor update to compat/fsmonitor/fsm-ipc-unix.c to make it cross-platform. Mac OS and Linux can share fsm-ipc-unix.c
Both platforms can also share compat/fsmonitor/fsm-settings-unix.c but we will leave room for future, platform-specific checks by having the platform- specific implementations call into fsm-settings-unix.
---
Makefile | 9 ++-
compat/fsmonitor/fsm-health-linux.c | 24 +++++++
.../{fsm-ipc-darwin.c => fsm-ipc-unix.c} | 8 +--
compat/fsmonitor/fsm-settings-darwin.c | 57 +----------------
compat/fsmonitor/fsm-settings-linux.c | 11 ++++
compat/fsmonitor/fsm-settings-unix.c | 62 +++++++++++++++++++
compat/fsmonitor/fsm-settings-unix.h | 11 ++++
config.mak.uname | 1 +
contrib/buildsystems/CMakeLists.txt | 17 ++++-
9 files changed, 138 insertions(+), 62 deletions(-)
create mode 100644 compat/fsmonitor/fsm-health-linux.c
rename compat/fsmonitor/{fsm-ipc-darwin.c => fsm-ipc-unix.c} (89%)
create mode 100644 compat/fsmonitor/fsm-settings-linux.c
create mode 100644 compat/fsmonitor/fsm-settings-unix.c
create mode 100644 compat/fsmonitor/fsm-settings-unix.hMakefile +8 -1
@@ -2036,13 +2036,20 @@ endif ifdef FSMONITOR_DAEMON_BACKEND COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND + ifdef FSMONITOR_DAEMON_COMMON + COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_COMMON).o + else + COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_BACKEND).o + endif COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o - COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_BACKEND).o endif ifdef FSMONITOR_OS_SETTINGS COMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS + ifdef FSMONITOR_DAEMON_COMMON + COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_DAEMON_COMMON).o + endif COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o COMPAT_OBJS += compat/fsmonitor/fsm-path-utils-$(FSMONITOR_OS_SETTINGS).o endif
compat/fsmonitor/fsm-health-linux.c new file +24 -0
@@ -0,0 +1,24 @@ +#include "cache.h" +#include "config.h" +#include "fsmonitor.h" +#include "fsm-health.h" +#include "fsmonitor--daemon.h" + +int fsm_health__ctor (struct fsmonitor_daemon_state *state) +{ +return 0 ; +} + +void fsm_health__dtor (struct fsmonitor_daemon_state *state) +{ +return ; +} + +void fsm_health__loop (struct fsmonitor_daemon_state *state) +{ +return ; +} + +void fsm_health__stop_async (struct fsmonitor_daemon_state *state) +{ +}
compat/fsmonitor/fsm-ipc-unix.c renamed from compat/fsmonitor/fsm-ipc-darwin.c +4 -4
@@ -10,7 +10,7 @@ static GIT_PATH_FUNC(fsmonitor_ipc__get_default_path, "fsmonitor--daemon.ipc")const char *fsmonitor_ipc__get_path (struct repository *r) {static const char *ipc_path =NULL ; -SHA_CTX sha1ctx; +git_SHA_CTX sha1ctx;char *sock_dir =NULL ;struct strbuf ipc_file =STRBUF_INIT ;unsigned char hash[SHA_DIGEST_LENGTH ];
@@ -28,9 +28,9 @@ const char *fsmonitor_ipc__get_path(struct repository *r)return ipc_path; } -SHA1_Init (&sha1ctx); -SHA1_Update (&sha1ctx, r->worktree,strlen (r->worktree)); -SHA1_Final (hash, &sha1ctx); +git_SHA1_Init (&sha1ctx); +git_SHA1_Update (&sha1ctx, r->worktree,strlen (r->worktree)); +git_SHA1_Final (hash, &sha1ctx);repo_config_get_string (r,"fsmonitor.socketdir" , &sock_dir);
compat/fsmonitor/fsm-settings-darwin.c +3 -54
65 lines not on this page · raw message
compat/fsmonitor/fsm-settings-linux.c new file +11 -0
@@ -0,0 +1,11 @@ +#include "config.h" +#include "fsmonitor.h" +#include "fsmonitor-ipc.h" +#include "fsmonitor-path-utils.h" +#include "fsmonitor-settings.h" +#include "fsm-settings-unix.h" + +enum fsmonitor_reason fsm_os__incompatible (struct repository *r,int ipc) +{ +return fsm_os__incompatible_unix (r, ipc); +}
compat/fsmonitor/fsm-settings-unix.c new file +62 -0
62 lines not on this page · raw message
compat/fsmonitor/fsm-settings-unix.h new file +11 -0
@@ -0,0 +1,11 @@ +#ifndef FSM_SETTINGS_UNIX_H +#define FSM_SETTINGS_UNIX_H + +#ifdef HAVE_FSMONITOR_OS_SETTINGS +/* +* Check for compatibility on unix-like systems (e.g. Darwin and Linux) +*/ +enum fsmonitor_reason fsm_os__incompatible_unix (struct repository *r,int ipc); +#endif /* HAVE_FSMONITOR_OS_SETTINGS */ + +#endif /* FSM_SETTINGS_UNIX_H */
config.mak.uname +1 -0
@@ -165,6 +165,7 @@ ifeq ($(uname_S),Darwin) ifndef NO_UNIX_SOCKETS FSMONITOR_DAEMON_BACKEND = darwin FSMONITOR_OS_SETTINGS = darwin + FSMONITOR_DAEMON_COMMON = unix endif endif
contrib/buildsystems/CMakeLists.txt +14 -3
@@ -304,7 +304,17 @@ else() endif() if(SUPPORTS_SIMPLE_IPC) - if(CMAKE_SYSTEM_NAME STREQUAL "Windows") + if(CMAKE_SYSTEM_NAME STREQUAL "Linux") + add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-unix.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-linux.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-linux.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-linux.c) + + add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-unix.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-linux.c) + elseif(CMAKE_SYSTEM_NAME STREQUAL "Windows") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c) list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c)
@@ -315,12 +325,13 @@ if(SUPPORTS_SIMPLE_IPC) list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c) elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-unix.c) list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c) - list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-darwin.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c) list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-darwin.c) add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-unix.c) list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c) endif() endif()
Signature · 1 line
gitgitgadget
PATCH v2 10/12 fsmonitor: enable fsmonitor for Linux
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
Update build to enable fsmonitor for Linux.
--- config.mak.uname | 9 +++++++++ 1 file changed, 9 insertions(+)
config.mak.uname +9 -0
@@ -68,6 +68,15 @@ ifeq ($(uname_S),Linux) ifneq ($(findstring .el7.,$(uname_R)),) BASIC_CFLAGS += -std=c99 endif + # The builtin FSMonitor on Linux builds upon Simple-IPC. Both require + # Unix domain sockets and PThreads. + ifndef NO_PTHREADS + ifndef NO_UNIX_SOCKETS + FSMONITOR_DAEMON_BACKEND = linux + FSMONITOR_OS_SETTINGS = linux + FSMONITOR_DAEMON_COMMON = unix + endif + endif endif ifeq ($(uname_S),GNU/kFreeBSD) HAVE_ALLOCA_H = YesPlease
Signature · 1 line
gitgitgadget
PATCH v2 11/12 fsmonitor: test updates
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
t7527-builtin-fsmonitor was leaking fsmonitor--daemon processes in some cases.
Accomodate slight difference in the number of events generated on Linux.
On lower-powered systems, spin a little to give the daemon time to respond to and log filesystem events.
--- t/t7527-builtin-fsmonitor.sh | 72 +++++++++++++++++++++++++++++------- 1 file changed, 58 insertions(+), 14 deletions(-)
t/t7527-builtin-fsmonitor.sh +58 -14
@@ -13,7 +13,7 @@ fi stop_daemon_delete_repo () { r=$1 && test_might_fail git -C $r fsmonitor--daemon stop && - rm -rf $1 + rm -rf $r } start_daemon () {
@@ -72,6 +72,32 @@ start_daemon () { ) } +IMPLICIT_TIMEOUT=5 + +wait_for_update () { + func=$1 && + file=$2 && + sz=$(wc -c < "$file") && + last=0 && + $func && + k=0 && + while test "$k" -lt $IMPLICIT_TIMEOUT + do + nsz=$(wc -c < "$file") + if test "$nsz" -gt "$sz" + then + if test "$last" -eq "$nsz" + then + return 0 + fi + last=$nsz + fi + sleep 1 + k=$(( $k + 1 )) + done && + return 0 +} + # Is a Trace2 data event present with the given catetory and key? # We do not care what the value is. #
@@ -137,7 +163,6 @@ test_expect_success 'implicit daemon start' ' # machines (where it might take a moment to wake and reschedule the # daemon process) to avoid false alarms during test runs.) # -IMPLICIT_TIMEOUT=5 verify_implicit_shutdown () { r=$1 &&
@@ -373,6 +398,10 @@ create_files () { echo 3 >dir2/new } +rename_directory () { + mv dirtorename dirrenamed +} + rename_files () { mv rename renamed && mv dir1/rename dir1/renamed &&
@@ -427,10 +456,12 @@ test_expect_success 'edit some files' ' start_daemon --tf "$PWD/.git/trace" && - edit_files && + wait_for_update edit_files "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 && + test_might_fail git fsmonitor--daemon stop && + grep "^event: dir1/modified$" .git/trace && grep "^event: dir2/modified$" .git/trace && grep "^event: modified$" .git/trace &&
@@ -442,10 +473,12 @@ test_expect_success 'create some files' ' start_daemon --tf "$PWD/.git/trace" && - create_files && + wait_for_update create_files "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 && + test_might_fail git fsmonitor--daemon stop && + grep "^event: dir1/new$" .git/trace && grep "^event: dir2/new$" .git/trace && grep "^event: new$" .git/trace
@@ -456,10 +489,12 @@ test_expect_success 'delete some files' ' start_daemon --tf "$PWD/.git/trace" && - delete_files && + wait_for_update delete_files "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 && + test_might_fail git fsmonitor--daemon stop && + grep "^event: dir1/delete$" .git/trace && grep "^event: dir2/delete$" .git/trace && grep "^event: delete$" .git/trace
@@ -470,10 +505,12 @@ test_expect_success 'rename some files' ' start_daemon --tf "$PWD/.git/trace" && - rename_files && + wait_for_update rename_files "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 && + test_might_fail git fsmonitor--daemon stop && + grep "^event: dir1/rename$" .git/trace && grep "^event: dir2/rename$" .git/trace && grep "^event: rename$" .git/trace &&
@@ -487,10 +524,12 @@ test_expect_success 'rename directory' ' start_daemon --tf "$PWD/.git/trace" && - mv dirtorename dirrenamed && + wait_for_update rename_directory "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 && + test_might_fail git fsmonitor--daemon stop && + grep "^event: dirtorename/*$" .git/trace && grep "^event: dirrenamed/*$" .git/trace '
@@ -500,10 +539,12 @@ test_expect_success 'file changes to directory' ' start_daemon --tf "$PWD/.git/trace" && - file_to_directory && + wait_for_update file_to_directory "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 && + test_might_fail git fsmonitor--daemon stop && + grep "^event: delete$" .git/trace && grep "^event: delete/new$" .git/trace '
@@ -513,10 +554,12 @@ test_expect_success 'directory changes to a file' ' start_daemon --tf "$PWD/.git/trace" && - directory_to_file && + wait_for_update directory_to_file "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 && + test_might_fail git fsmonitor--daemon stop && + grep "^event: dir1$" .git/trace '
@@ -561,7 +604,7 @@ test_expect_success 'flush cached data' ' test-tool -C test_flush fsmonitor-client query --token "builtin:test_00000002:0" >actual_2 && nul_to_q <actual_2 >actual_q2 && - grep "^builtin:test_00000002:0Q$" actual_q2 && + grep "^builtin:test_00000002:[0-1]Q$" actual_q2 && >test_flush/file_3 &&
@@ -732,7 +775,8 @@ u_values="$u1 $u2" for u in $u_values do test_expect_success "unicode in repo root path: $u" ' - test_when_finished "stop_daemon_delete_repo $u" && + test_when_finished \ + "stop_daemon_delete_repo `echo "$u" | sed 's:x:\\\\\\\\\\\\\\x:g'`" && git init "$u" && echo 1 >"$u"/file1 &&
@@ -814,8 +858,7 @@ my_match_and_clean () { } test_expect_success 'submodule always visited' ' - test_when_finished "git -C super fsmonitor--daemon stop; \ - rm -rf super; \ + test_when_finished "rm -rf super; \ rm -rf sub" && create_super super &&
@@ -883,7 +926,8 @@ have_t2_error_event () { } test_expect_success "stray submodule super-prefix warning" ' - test_when_finished "rm -rf super; \ + test_when_finished "git -C super/dir_1/dir_2/sub fsmonitor--daemon stop; \ + rm -rf super; \ rm -rf sub; \ rm super-sub.trace" &&
Signature · 1 line
gitgitgadget
PATCH v2 9/12 fsmonitor: implement filesystem change listener for Linux
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
Implement a filesystem change listener for Linux based on the inotify API:
https://man7.org/linux/man-pages/man7/inotify.7.html
inotify requires registering a watch on every directory in the worktree and special handling of moves/renames.
--- compat/fsmonitor/fsm-listen-linux.c | 664 ++++++++++++++++++++++++++++ 1 file changed, 664 insertions(+) create mode 100644 compat/fsmonitor/fsm-listen-linux.c
compat/fsmonitor/fsm-listen-linux.c new file +664 -0
@@ -0,0 +1,664 @@ +#include "cache.h" +#include "fsmonitor.h" +#include "fsm-listen.h" +#include "fsmonitor--daemon.h" +#include <dirent.h> +#include <fcntl.h> +#include <sys/inotify.h> +#include <sys/stat.h> + +/* +* Safe value to bitwise OR with rest of mask for +* kernels that do not support IN_MASK_CREATE +*/ +#ifndef IN_MASK_CREATE +#define IN_MASK_CREATE 0x00000000 +#endif + +enum shutdown_reason { +SHUTDOWN_CONTINUE =0 , +SHUTDOWN_STOP , +SHUTDOWN_ERROR , +SHUTDOWN_FORCE +}; + +struct watch_entry { +struct hashmap_entry ent; +int wd; +uint32_t cookie; +const char *dir; +}; + +struct rename_entry { +struct hashmap_entry ent; +time_t whence; +uint32_t cookie; +const char *dir; +}; + +struct fsm_listen_data { +int fd_inotify; +enum shutdown_reason shutdown; +struct hashmap watches; +struct hashmap renames; +struct hashmap revwatches; +}; + +static int watch_entry_cmp (const void *cmp_data, +const struct hashmap_entry *eptr, +const struct hashmap_entry *entry_or_key, +const void *keydata) +{ +const struct watch_entry *e1, *e2; + + e1 = container_of(eptr, const struct watch_entry, ent); + e2 = container_of(eptr, const struct watch_entry, ent); +return e1->wd != e2->wd; +} + +static int revwatches_entry_cmp (const void *cmp_data, +const struct hashmap_entry *eptr, +const struct hashmap_entry *entry_or_key, +const void *keydata) +{ +const struct watch_entry *e1, *e2; + + e1 = container_of(eptr, const struct watch_entry, ent); + e2 = container_of(eptr, const struct watch_entry, ent); +return strcmp (e1->dir, e2->dir); +} + +static int rename_entry_cmp (const void *cmp_data, +const struct hashmap_entry *eptr, +const struct hashmap_entry *entry_or_key, +const void *keydata) +{ +const struct rename_entry *e1, *e2; + + e1 = container_of(eptr, const struct rename_entry, ent); + e2 = container_of(eptr, const struct rename_entry, ent); +return e1->cookie != e2->cookie; +} + +/* +* Register an inotify watch, add watch descriptor to path mapping +* and the reverse mapping. +*/ +static int add_watch (const char *path,struct fsm_listen_data *data) +{ +const char *interned =strintern (path); +struct watch_entry *w1, *w2; + +/* add the inotify watch, don't allow watches to be modified */ +int wd =inotify_add_watch (data->fd_inotify, interned, + (IN_ALL_EVENTS |IN_ONLYDIR |IN_MASK_CREATE ) + ^IN_ACCESS ^IN_CLOSE ^IN_OPEN ); +if (wd <0 ) +return error_errno ("inotify_add_watch('%s') failed" , interned); + +/* add watch descriptor -> directory mapping */ +CALLOC_ARRAY (w1,1 ); + w1->wd = wd; + w1->dir = interned; +hashmap_entry_init (&w1->ent,memhash (&w1->wd,sizeof (int ))); +hashmap_add (&data->watches, &w1->ent); + +/* add directory -> watch descriptor mapping */ +CALLOC_ARRAY (w2,1 ); + w2->wd = wd; + w2->dir = interned; +hashmap_entry_init (&w2->ent,memhash (w2->dir,strlen (w2->dir))); +hashmap_add (&data->revwatches, &w2->ent); + +return 0 ; +} + +/* +* Remove the inotify watch, the watch descriptor to path mapping +* and the reverse mapping. +*/ +static void remove_watch (struct watch_entry *w, +struct fsm_listen_data *data) +{ +struct watch_entry k1, k2, *w1, *w2; + +/* remove watch, ignore error if kernel already did it */ +if (inotify_rm_watch (data->fd_inotify, w->wd) && errno !=EINVAL ) +error_errno ("inotify_rm_watch() failed" ); + +hashmap_entry_init (&k1.ent,memhash (&w->wd,sizeof (int ))); + w1 =hashmap_remove_entry (&data->watches, &k1, ent,NULL ); +if (!w1) +BUG ("Double remove of watch for '%s'" , w->dir); + +if (w1->cookie) +BUG ("Removing watch for '%s' which has a pending rename" , w1->dir); + +hashmap_entry_init (&k2.ent,memhash (w->dir,strlen (w->dir))); + w2 =hashmap_remove_entry (&data->revwatches, &k2, ent,NULL ); +if (!w2) +BUG ("Double remove of reverse watch for '%s'" , w->dir); + +/* w1->dir and w2->dir are interned strings, we don't own them */ +free (w1); +free (w2); +} + +/* +* Check for stale directory renames. +* +* https://man7.org/linux/man-pages/man7/inotify.7.html +* +* Allow for some small timeout to account for the fact that insertion of the +* IN_MOVED_FROM+IN_MOVED_TO event pair is not atomic, and the possibility that +* there may not be any IN_MOVED_TO event. +* +* If the IN_MOVED_TO event is not received within the timeout then events have +* been missed and the monitor is in an inconsistent state with respect to the +* filesystem. +*/ +static int check_stale_dir_renames (struct hashmap *renames,time_t max_age) +{ +struct rename_entry *re; +struct hashmap_iter iter; + +hashmap_for_each_entry (renames, &iter, re, ent) { +if (re->whence <= max_age) +return -1 ; + } +return 0 ; +} + +/* +* Track pending renames. +* +* Tracking is done via a event cookie to watch descriptor mapping. +* +* A rename is not complete until matching a IN_MOVED_TO event is received +* for a corresponding IN_MOVED_FROM event. +*/ +static void add_dir_rename (uint32_t cookie,const char *path, +struct fsm_listen_data *data) +{ +struct watch_entry k, *w; +struct rename_entry *re; + +/* lookup the watch descriptor for the given path */ +hashmap_entry_init (&k.ent,memhash (path,strlen (path))); + w =hashmap_get_entry (&data->revwatches, &k, ent,NULL ); +if (!w)/* should never happen */ +BUG ("No watch for '%s'" , path); + w->cookie = cookie; + +/* add the pending rename to match against later */ +CALLOC_ARRAY (re,1 ); + re->dir = w->dir; + re->cookie = w->cookie; + re->whence =time (NULL ); +hashmap_entry_init (&re->ent,memhash (&re->cookie,sizeof (uint32_t ))); +hashmap_add (&data->renames, &re->ent); +} + +/* +* Handle directory renames +* +* Once a IN_MOVED_TO event is received, lookup the rename tracking information +* via the event cookie and use this information to update the watch. +*/ +static void rename_dir (uint32_t cookie,const char *path, +struct fsm_listen_data *data) +{ +struct rename_entry rek, *re; +struct watch_entry k, *w; + +/* lookup a pending rename to match */ + rek.cookie = cookie; +hashmap_entry_init (&rek.ent,memhash (&rek.cookie,sizeof (uint32_t ))); + re =hashmap_get_entry (&data->renames, &rek, ent,NULL ); +if (re) { + k.dir = re->dir; +hashmap_entry_init (&k.ent,memhash (k.dir,strlen (k.dir))); + w =hashmap_get_entry (&data->revwatches, &k, ent,NULL ); +if (w) { + w->cookie =0 ;/* rename handled */ +remove_watch (w, data); +add_watch (path, data); + }else { +BUG ("No matching watch" ); + } + }else { +BUG ("No matching cookie" ); + } +} + +/* +* Recursively add watches to every directory under path +*/ +static int register_inotify (const char *path,struct fsm_listen_data *data) +{ +DIR *dir; +struct strbuf current =STRBUF_INIT ; +struct dirent *de; +struct stat fs; +int ret =-1 ; + + dir =opendir (path); +if (!dir) +return error_errno ("opendir('%s') failed" , path); + +while ((de =readdir_skip_dot_and_dotdot (dir)) !=NULL ) { +strbuf_reset (¤t); +strbuf_addf (¤t,"%s/%s" , path, de->d_name); +if (lstat (current.buf, &fs)) { +error_errno ("lstat('%s') failed" , current.buf); + goto failed; + } + +/* recurse into directory */ +if (S_ISDIR (fs.st_mode)) { +if (add_watch (current.buf, data)) + goto failed; +if (register_inotify (current.buf, data)) + goto failed; + } + } + ret =0 ; + +failed: +strbuf_release (¤t); +if (closedir (dir) <0 ) +return error_errno ("closedir('%s') failed" , path); +return ret; +} + +static int em_rename_dir_from (u_int32_t mask) +{ +return ((mask &IN_ISDIR ) && (mask &IN_MOVED_FROM )); +} + +static int em_rename_dir_to (u_int32_t mask) +{ +return ((mask &IN_ISDIR ) && (mask &IN_MOVED_TO )); +} + +static int em_remove_watch (u_int32_t mask) +{ +return (mask &IN_DELETE_SELF ); +} + +static int em_dir_renamed (u_int32_t mask) +{ +return ((mask &IN_ISDIR ) && (mask &IN_MOVE )); +} + +static int em_dir_created (u_int32_t mask) +{ +return ((mask &IN_ISDIR ) && (mask &IN_CREATE )); +} + +static int em_dir_deleted (uint32_t mask) +{ +return ((mask &IN_ISDIR ) && (mask &IN_DELETE )); +} + +static int em_force_shutdown (u_int32_t mask) +{ +return (mask &IN_UNMOUNT ) || (mask &IN_Q_OVERFLOW ); +} + +static int em_ignore (u_int32_t mask) +{ +return (mask &IN_IGNORED ) || (mask &IN_MOVE_SELF ); +} + +static void log_mask_set (const char *path,u_int32_t mask) +{ +struct strbuf msg =STRBUF_INIT ; + +if (mask &IN_ACCESS ) +strbuf_addstr (&msg,"IN_ACCESS|" ); +if (mask &IN_MODIFY ) +strbuf_addstr (&msg,"IN_MODIFY|" ); +if (mask &IN_ATTRIB ) +strbuf_addstr (&msg,"IN_ATTRIB|" ); +if (mask &IN_CLOSE_WRITE ) +strbuf_addstr (&msg,"IN_CLOSE_WRITE|" ); +if (mask &IN_CLOSE_NOWRITE ) +strbuf_addstr (&msg,"IN_CLOSE_NOWRITE|" ); +if (mask &IN_OPEN ) +strbuf_addstr (&msg,"IN_OPEN|" ); +if (mask &IN_MOVED_FROM ) +strbuf_addstr (&msg,"IN_MOVED_FROM|" ); +if (mask &IN_MOVED_TO ) +strbuf_addstr (&msg,"IN_MOVED_TO|" ); +if (mask &IN_CREATE ) +strbuf_addstr (&msg,"IN_CREATE|" ); +if (mask &IN_DELETE ) +strbuf_addstr (&msg,"IN_DELETE|" ); +if (mask &IN_DELETE_SELF ) +strbuf_addstr (&msg,"IN_DELETE_SELF|" ); +if (mask &IN_MOVE_SELF ) +strbuf_addstr (&msg,"IN_MOVE_SELF|" ); +if (mask &IN_UNMOUNT ) +strbuf_addstr (&msg,"IN_UNMOUNT|" ); +if (mask &IN_Q_OVERFLOW ) +strbuf_addstr (&msg,"IN_Q_OVERFLOW|" ); +if (mask &IN_IGNORED ) +strbuf_addstr (&msg,"IN_IGNORED|" ); +if (mask &IN_ISDIR ) +strbuf_addstr (&msg,"IN_ISDIR|" ); + +trace_printf_key (&trace_fsmonitor,"inotify_event: '%s', mask=%#8.8x %s" , + path, mask, msg.buf); + +strbuf_release (&msg); +} + +int fsm_listen__ctor (struct fsmonitor_daemon_state *state) +{ +int fd; +int ret =0 ; +struct fsm_listen_data *data; + +CALLOC_ARRAY (data,1 ); + state->listen_data = data; + state->listen_error_code =-1 ; + data->shutdown =SHUTDOWN_ERROR ; + + fd =inotify_init1 (O_NONBLOCK ); +if (fd <0 ) +return error_errno ("inotify_init1() failed" ); + + data->fd_inotify = fd; + +hashmap_init (&data->watches, watch_entry_cmp,NULL ,0 ); +hashmap_init (&data->renames, rename_entry_cmp,NULL ,0 ); +hashmap_init (&data->revwatches, revwatches_entry_cmp,NULL ,0 ); + +if (add_watch (state->path_worktree_watch.buf, data)) + ret =-1 ; +else if (register_inotify (state->path_worktree_watch.buf, data)) + ret =-1 ; +else if (state->nr_paths_watching >1 ) { +if (add_watch (state->path_gitdir_watch.buf, data)) + ret =-1 ; +else if (register_inotify (state->path_gitdir_watch.buf, data)) + ret =-1 ; + } + +if (!ret) { + state->listen_error_code =0 ; + data->shutdown =SHUTDOWN_CONTINUE ; + } + +return ret; +} + +void fsm_listen__dtor (struct fsmonitor_daemon_state *state) +{ +struct fsm_listen_data *data; +struct hashmap_iter iter; +struct watch_entry *w; +int fd; + +if (!state || !state->listen_data) +return ; + + data = state->listen_data; + fd = data->fd_inotify; + +hashmap_for_each_entry (&data->watches, &iter, w, ent) { + w->cookie =0 ;/* ignore any pending renames */ +remove_watch (w, data); + } +hashmap_clear (&data->watches); + +hashmap_clear (&data->revwatches);/* remove_watch freed the entries */ + + hashmap_clear_and_free(&data->renames, struct rename_entry, ent); + +FREE_AND_NULL (state->listen_data); + +if (fd && (close (fd) <0 )) +error_errno (_ ("closing inotify file descriptor failed" )); +} + +void fsm_listen__stop_async (struct fsmonitor_daemon_state *state) +{ +if (!state->listen_data->shutdown) + state->listen_data->shutdown =SHUTDOWN_STOP ; +} + +/* +* Process a single inotify event and queue for publication. +*/ +static int process_event (const char *path, +const struct inotify_event *event, +struct fsmonitor_batch *batch, +struct string_list *cookie_list, +struct fsmonitor_daemon_state *state) +{ +const char *rel; +const char *last_sep; + +switch (fsmonitor_classify_path_absolute (state, path)) { +case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX : +case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX : +/* Use just the filename of the cookie file. */ + last_sep =find_last_dir_sep (path); +string_list_append (cookie_list, + last_sep ? last_sep +1 : path); +break ; +case IS_INSIDE_DOT_GIT : +case IS_INSIDE_GITDIR : +break ; +case IS_DOT_GIT : +case IS_GITDIR : +/* +* If .git directory is deleted or renamed away, +* we have to quit. +*/ +if (em_dir_deleted (event->mask)) { +trace_printf_key (&trace_fsmonitor, +"event: gitdir removed" ); + state->listen_data->shutdown =SHUTDOWN_FORCE ; + goto done; + } + +if (em_dir_renamed (event->mask)) { +trace_printf_key (&trace_fsmonitor, +"event: gitdir renamed" ); + state->listen_data->shutdown =SHUTDOWN_FORCE ; + goto done; + } +break ; +case IS_WORKDIR_PATH : +/* normal events in the working directory */ +if (trace_pass_fl (&trace_fsmonitor)) +log_mask_set (path, event->mask); + + rel = path + state->path_worktree_watch.len +1 ; +fsmonitor_batch__add_path (batch, rel); + +if (em_dir_deleted (event->mask)) +break ; + +/* received IN_MOVE_FROM, add tracking for expected IN_MOVE_TO */ +if (em_rename_dir_from (event->mask)) +add_dir_rename (event->cookie, path, state->listen_data); + +/* received IN_MOVE_TO, update watch to reflect new path */ +if (em_rename_dir_to (event->mask)) +rename_dir (event->cookie, path, state->listen_data); + +if (em_dir_created (event->mask)) { +if (add_watch (path, state->listen_data)) { + state->listen_data->shutdown =SHUTDOWN_ERROR ; + goto done; + } +if (register_inotify (path, state->listen_data)) { + state->listen_data->shutdown =SHUTDOWN_ERROR ; + goto done; + } + } +break ; +case IS_OUTSIDE_CONE : +default : +trace_printf_key (&trace_fsmonitor, +"ignoring '%s'" , path); +break ; + } +return 0 ; +done: +return -1 ; +} + +/* +* Read the inotify event stream and pre-process events before further +* processing and eventual publishing. +*/ +static void handle_events (struct fsmonitor_daemon_state *state) +{ +/* See https://man7.org/linux/man-pages/man7/inotify.7.html */ + char buf[4096] + __attribute__ ((aligned(__alignof__(struct inotify_event)))); + +struct hashmap watches = state->listen_data->watches; +struct fsmonitor_batch *batch =NULL ; +struct string_list cookie_list =STRING_LIST_INIT_DUP ; +struct watch_entry k, *w; +struct strbuf path; +const struct inotify_event *event; +int fd = state->listen_data->fd_inotify; +ssize_t len; +char *ptr, *p; + +strbuf_init (&path,PATH_MAX ); + +for (;;) { + len =read (fd, buf,sizeof (buf)); +if (len ==-1 && errno !=EAGAIN ) { +error_errno (_ ("reading inotify message stream failed" )); + state->listen_data->shutdown =SHUTDOWN_ERROR ; + goto done; + } + +/* nothing to read */ +if (len <=0 ) + goto done; + +/* Loop over all events in the buffer. */ +for (ptr = buf; ptr < buf + len; + ptr +=sizeof (struct inotify_event ) + event->len) { + + event = (const struct inotify_event *) ptr; + +if (em_ignore (event->mask)) +continue ; + +/* File system was unmounted or event queue overflowed */ +if (em_force_shutdown (event->mask)) { +if (trace_pass_fl (&trace_fsmonitor)) +log_mask_set ("Forcing shutdown" , event->mask); + state->listen_data->shutdown =SHUTDOWN_FORCE ; + goto done; + } + +hashmap_entry_init (&k.ent,memhash (&event->wd,sizeof (int ))); + k.wd = event->wd; + + w =hashmap_get_entry (&watches, &k, ent,NULL ); +if (!w)/* should never happen */ +BUG ("No watch for '%s'" , event->name); + +/* directory watch was removed */ +if (em_remove_watch (event->mask)) { +remove_watch (w, state->listen_data); +continue ; + } + +strbuf_reset (&path); +strbuf_add (&path, w->dir,strlen (w->dir)); +strbuf_addch (&path,'/' ); +strbuf_add (&path, event->name,strlen (event->name)); + + p =fsmonitor__resolve_alias (path.buf, &state->alias); +if (!p) + p =strbuf_detach (&path,NULL ); + +if (!batch) + batch =fsmonitor_batch__new (); + +if (process_event (p, event, batch, &cookie_list, state)) { +free (p); + goto done; + } +free (p); + } +strbuf_reset (&path); +fsmonitor_publish (state, batch, &cookie_list); +string_list_clear (&cookie_list,0 ); + batch =NULL ; + } +done: +strbuf_release (&path); +fsmonitor_batch__free_list (batch); +string_list_clear (&cookie_list,0 ); +} + +/* +* Non-blocking read of the inotify events stream. The inotify fd is polled +* frequently to help minimize the number of queue overflows. +*/ +void fsm_listen__loop (struct fsmonitor_daemon_state *state) +{ +int poll_num; +const int interval =1000 ; +time_t checked =time (NULL ); +struct pollfd fds[1 ]; + fds[0 ].fd = state->listen_data->fd_inotify; + fds[0 ].events =POLLIN ; + +for (;;) { +switch (state->listen_data->shutdown) { +case SHUTDOWN_CONTINUE : + poll_num =poll (fds,1 ,1 ); +if (poll_num ==-1 ) { +if (errno ==EINTR ) +continue ; +error_errno (_ ("polling inotify message stream failed" )); + state->listen_data->shutdown =SHUTDOWN_ERROR ; +continue ; + } + +if ((time (NULL ) - checked) >= interval) { + checked =time (NULL ); +if (check_stale_dir_renames (&state->listen_data->renames, + checked - interval)) { +trace_printf_key (&trace_fsmonitor, +"Missed IN_MOVED_TO events, forcing shutdown" ); + state->listen_data->shutdown =SHUTDOWN_FORCE ; +continue ; + } + } + +if (poll_num >0 && (fds[0 ].revents &POLLIN )) +handle_events (state); + +continue ; +case SHUTDOWN_ERROR : + state->listen_error_code =-1 ; +ipc_server_stop_async (state->ipc_server_data); +break ; +case SHUTDOWN_FORCE : + state->listen_error_code =0 ; +ipc_server_stop_async (state->ipc_server_data); +break ; +case SHUTDOWN_STOP : +default : + state->listen_error_code =0 ; +break ; + } +return ; + } +}
Signature · 1 line
gitgitgadget
PATCH v2 8/12 fsmonitor: determine if filesystem is local or remote
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
Compare the given path to the mounted filesystems. Find the mount that is the longest prefix of the path (if any) and determine if that mount is on a local or remote filesystem.
--- compat/fsmonitor/fsm-path-utils-linux.c | 169 ++++++++++++++++++++++++ 1 file changed, 169 insertions(+) create mode 100644 compat/fsmonitor/fsm-path-utils-linux.c
compat/fsmonitor/fsm-path-utils-linux.c new file +169 -0
@@ -0,0 +1,169 @@ +#include "fsmonitor.h" +#include "fsmonitor-path-utils.h" +#include <errno.h> +#include <mntent.h> +#include <sys/mount.h> +#include <sys/statvfs.h> + +/* +* https://github.com/coreutils/gnulib/blob/master/lib/mountlist.c +*/ +#ifndef ME_REMOTE +/* A file system is "remote" if its Fs_name contains a ':' +or if (it is of type (smbfs or cifs) and its Fs_name starts with '//') +or if it is of any other of the listed types +or Fs_name is equal to "-hosts" (used by autofs to mount remote fs). +"VM" file systems like prl_fs or vboxsf are not considered remote here. */ +# define ME_REMOTE (Fs_name, Fs_type) \ + (strchr (Fs_name, ':') != NULL \ + || ((Fs_name)[0] == '/' \ + && (Fs_name)[1] == '/' \ + && (strcmp (Fs_type, "smbfs") == 0 \ + || strcmp (Fs_type, "smb3") == 0 \ + || strcmp (Fs_type, "cifs") == 0)) \ + || strcmp (Fs_type, "acfs") == 0 \ + || strcmp (Fs_type, "afs") == 0 \ + || strcmp (Fs_type, "coda") == 0 \ + || strcmp (Fs_type, "auristorfs") == 0 \ + || strcmp (Fs_type, "fhgfs") == 0 \ + || strcmp (Fs_type, "gpfs") == 0 \ + || strcmp (Fs_type, "ibrix") == 0 \ + || strcmp (Fs_type, "ocfs2") == 0 \ + || strcmp (Fs_type, "vxfs") == 0 \ + || strcmp ("-hosts", Fs_name) == 0) +#endif + +static int find_mount (const char *path,const struct statvfs *fs, +struct mntent *ent) +{ +const char *const mounts ="/proc/mounts" ; +const char *rp =real_pathdup (path,1 ); +struct mntent *ment =NULL ; +struct statvfs mntfs; +FILE *fp; +int found =0 ; +int dlen, plen, flen =0 ; + + ent->mnt_fsname =NULL ; + ent->mnt_dir =NULL ; + ent->mnt_type =NULL ; + + fp =setmntent (mounts,"r" ); +if (!fp) { +error_errno (_ ("setmntent('%s') failed" ), mounts); +return -1 ; + } + + plen =strlen (rp); + +/* read all the mount information and compare to path */ +while ((ment =getmntent (fp)) !=NULL ) { +if (statvfs (ment->mnt_dir, &mntfs)) { +switch (errno) { +case EPERM : +case ESRCH : +case EACCES : +continue ; +default : +error_errno (_ ("statvfs('%s') failed" ), ment->mnt_dir); +endmntent (fp); +return -1 ; + } + } + +/* is mount on the same filesystem and is a prefix of the path */ +if ((fs->f_fsid == mntfs.f_fsid) && + !strncmp (ment->mnt_dir, rp,strlen (ment->mnt_dir))) { + dlen =strlen (ment->mnt_dir); +if (dlen > plen) +continue ; +/* +* root is always a potential match; otherwise look for +* directory prefix +*/ +if ((dlen ==1 && ment->mnt_dir[0 ] =='/' ) || + (dlen > flen && (!rp[dlen] || rp[dlen] =='/' ))) { + flen = dlen; +/* +* https://man7.org/linux/man-pages/man3/getmntent.3.html +* +* The pointer points to a static area of memory which is +* overwritten by subsequent calls to getmntent(). +*/ + found =1 ; +free (ent->mnt_fsname); +free (ent->mnt_dir); +free (ent->mnt_type); + ent->mnt_fsname =xstrdup (ment->mnt_fsname); + ent->mnt_dir =xstrdup (ment->mnt_dir); + ent->mnt_type =xstrdup (ment->mnt_type); + } + } + } +endmntent (fp); + +if (!found) +return -1 ; + +return 0 ; +} + +int fsmonitor__get_fs_info (const char *path,struct fs_info *fs_info) +{ +struct mntent ment; +struct statvfs fs; + +if (statvfs (path, &fs)) +return error_errno (_ ("statvfs('%s') failed" ), path); + + +if (find_mount (path, &fs, &ment) <0 ) { +free (ment.mnt_fsname); +free (ment.mnt_dir); +free (ment.mnt_type); +return -1 ; + } + +trace_printf_key (&trace_fsmonitor, +"statvfs('%s') [flags 0x%08lx] '%s' '%s'" , + path, fs.f_flag, ment.mnt_type, ment.mnt_fsname); + + fs_info->is_remote =ME_REMOTE (ment.mnt_fsname, ment.mnt_type); + fs_info->typename = ment.mnt_fsname; +free (ment.mnt_dir); +free (ment.mnt_type); + +trace_printf_key (&trace_fsmonitor, +"'%s' is_remote: %d" , + path, fs_info->is_remote); +return 0 ; +} + +int fsmonitor__is_fs_remote (const char *path) +{ +struct fs_info fs; + +if (fsmonitor__get_fs_info (path, &fs)) +return -1 ; + +free (fs.typename); + +return fs.is_remote; +} + +/* +* No-op for now. +*/ +int fsmonitor__get_alias (const char *path,struct alias_info *info) +{ +return 0 ; +} + +/* +* No-op for now. +*/ +char *fsmonitor__resolve_alias (const char *path, +const struct alias_info *info) +{ +return NULL ; +}
Signature · 1 line
gitgitgadget
PATCH v2 12/12 fsmonitor: update doc for Linux
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
Update the documentation for Linux.
--- Documentation/config/fsmonitor--daemon.txt | 4 ++-- Documentation/git-fsmonitor--daemon.txt | 24 ++++++++++++++-------- 2 files changed, 18 insertions(+), 10 deletions(-)
Documentation/config/fsmonitor--daemon.txt +2 -2
@@ -4,8 +4,8 @@ fsmonitor.allowRemote:: behavior. Only respected when `core.fsmonitor` is set to `true`. fsmonitor.socketDir:: - This Mac OS-specific option, if set, specifies the directory in + Mac OS and Linux-specific option. If set, specifies the directory in which to create the Unix domain socket used for communication between the fsmonitor daemon and various Git commands. The directory must - reside on a native Mac OS filesystem. Only respected when `core.fsmonitor` + reside on a native filesystem. Only respected when `core.fsmonitor` is set to `true`.
Documentation/git-fsmonitor--daemon.txt +16 -8
@@ -76,23 +76,31 @@ repositories; this may be overridden by setting `fsmonitor.allowRemote` to correctly with all network-mounted repositories and such use is considered experimental. -On Mac OS, the inter-process communication (IPC) between various Git +On Linux and Mac OS, the inter-process communication (IPC) between various Git commands and the fsmonitor daemon is done via a Unix domain socket (UDS) -- a -special type of file -- which is supported by native Mac OS filesystems, -but not on network-mounted filesystems, NTFS, or FAT32. Other filesystems -may or may not have the needed support; the fsmonitor daemon is not guaranteed -to work with these filesystems and such use is considered experimental. +special type of file -- which is supported by many native Linux and Mac OS +filesystems, but not on network-mounted filesystems, NTFS, or FAT32. Other +filesystems may or may not have the needed support; the fsmonitor daemon is not +guaranteed to work with these filesystems and such use is considered +experimental. By default, the socket is created in the `.git` directory, however, if the `.git` directory is on a network-mounted filesystem, it will be instead be created at `$HOME/.git-fsmonitor-*` unless `$HOME` itself is on a network-mounted filesystem in which case you must set the configuration -variable `fsmonitor.socketDir` to the path of a directory on a Mac OS native +variable `fsmonitor.socketDir` to the path of a directory on a native filesystem in which to create the socket file. If none of the above directories (`.git`, `$HOME`, or `fsmonitor.socketDir`) -is on a native Mac OS file filesystem the fsmonitor daemon will report an -error that will cause the daemon and the currently running command to exit. +is on a native Linux or Mac OS filesystem the fsmonitor daemon will report +an error that will cause the daemon to exit and the currently running command +to issue a warning. + +On Linux, the fsmonitor daemon registers a watch for each directory in the +repository. The default per-user limit for the number of watches on most Linux +systems is 8192. This may not be sufficient for large repositories or if +multiple instances of the fsmonitor daemon are running. +See https://watchexec.github.io/docs/inotify-limits.html[Linux inotify limits] for more information. CONFIGURATION -------------
Signature · 1 line
gitgitgadget
Quoted 6 lines · Eric DeCosta via GitGitGadget
Goal is to deliver fsmonitor for Linux that is on par with fsmonitor for Windows and Mac OS.
This patch set builds upon previous work for done for Windows and Mac OS (first 6 patches) to implement a fsmonitor back-end for Linux based on the Linux inotify API.
Again, the first six patches are a part of what is queued as ed/fsmonitor-on-networked-macos that is now in 'next' but lacks a fix-up commit from Jeff King.
I understand that it might not be easy/possible (e.g. perhaps it is a limitation of GGG?), but I really prefer not to see them re-posted as part of this series, as I have to apply them and make sure there are no changes from the last one before discarding them.
Anyway, thanks for an update. Will requeue.
Re: [PATCH v2 07/12] fsmonitor: prepare to share code between Mac OS and Linux
In reply to Eric DeCosta via GitGitGadgetQuoted 10 lines · Eric DeCosta via GitGitGadget
ifdef
FSMONITOR_DAEMON_BACKENDMakefile +5
-1COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND + ifdef FSMONITOR_DAEMON_COMMON + COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_COMMON).o + else + COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_BACKEND).o + endif COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o - COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_BACKEND).o
Makefile endif · line 2055
Quoted 10 lines
Makefile +3
-0endif ifdef FSMONITOR_OS_SETTINGS COMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS + ifdef FSMONITOR_DAEMON_COMMON + COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_DAEMON_COMMON).o + endif COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o COMPAT_OBJS += compat/fsmonitor/fsm-path-utils-$(FSMONITOR_OS_SETTINGS).o endif
Ugly.
One overrides backend with common, while the other one doesn't. That asymmetry alone should stop us and wonder if there is something fishy in the approach that can be improved. It makes it look like the word "common" means something quite different between the -ipc and the -settings world.
I suspect that in both, you should not expose "unix" to this part of the Makefile. Linux and macOS occasionally being similar in some places does not have to be exposed here. INstead you can use backend "linux" and "macos", whose C sources may include from a separate C source file whose name may contain "unix". That would allow you to get rid of FSMONITOR_DAEMON_COMMON in a cleaner way.
RE: [PATCH v2 07/12] fsmonitor: prepare to share code between Mac OS and Linux
Quoted 51 lines
-----Original Message-----
From: Junio C Hamano <[email protected]>
Sent: Friday, October 14, 2022 7:46 PM
To: Eric DeCosta via GitGitGadget <[email protected]>
Cc: [email protected]; Eric Sunshine <[email protected]>; Ævar Arnfjörð Bjarmason <[email protected]>; Eric DeCosta <[email protected]>
Subject: Re: [PATCH v2 07/12] fsmonitor: prepare to share code between Mac OS and LinuxMakefile endif
Quoted 4 lines · Eric DeCosta via GitGitGadget
ifdef
FSMONITOR_DAEMON_BACKENDMakefile +2
-0COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND + ifdef FSMONITOR_DAEMON_COMMON + COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_COMMON).o
Review
Quoted 2 lines
(quoted diff) +2
-0+ else + COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_BACKEND).o
Review
Quoted 2 lines
(quoted diff) +1
-0+ endif COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o
Quoted 1 line
COMPAT_OBJS+= compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o
Review
Quoted 1 line
(quoted diff) +0
-1- COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_BACKEND).o
Makefile endif
Quoted 6 lines
endif
ifdef
FSMONITOR_OS_SETTINGSMakefile +2
-0COMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS + ifdef FSMONITOR_DAEMON_COMMON + COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_DAEMON_COMMON).o
Review
Quoted 2 lines
(quoted diff) +1
-0+ endif COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o
Quoted 3 lines
COMPAT_OBJS+=
compat/fsmonitor/fsm-path-utils-$(FSMONITOR_OS_SETTINGS).o
endifUgly.
One overrides backend with common, while the other one doesn't. That asymmetry alone should stop us and wonder if there is something fishy in the approach that can be improved. It makes it look like the word "common" means something quite different between the -ipc and the - settings world.
Quoted 6 lines
I suspect that in both, you should not expose "unix" to this part of the
Makefile. Linux and macOS occasionally being similar in some places does not have to be exposed here. INstead you can use backend "linux" and "macos", whose C sources may include from a separate C source file whose name may contain "unix". That would allow you to get rid ofFSMONITOR_DAEMON_COMMONin a cleaner way.
Let me see if I am understanding you correctly. Are you suggesting something like:
ifdef FSMONITOR_DAEMON_BACKEND_LINUX
COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND COMPAT_OBJS += compat/fsmonitor/fsm-ipc-unix.o COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_BACKEND_LINUX).o COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND_LINUX).o COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND_LINUX).o
endif
ifdef FSMONITOR_DAEMON_BACKEND_DARWIN
COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND COMPAT_OBJS += compat/fsmonitor/fsm-ipc-unix.o COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_BACKEND_DARWIN).o COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND_DARWIN).o COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND_DARWIN).o
endif
And similarly for settings:
ifdef FSMONITOR_DAEMON_SETTINGS_LINUX
COMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS COMPAT_OBJS += compat/fsmonitor/fsm-settings-unix.o COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_DAEMON_SETTINGS_LINUX).o COMPAT_OBJS += compat/fsmonitor/fsm-path-utils-$(FSMONITOR_DAEMON_SETTINGS_LINUX).o
endif
ifdef FSMONITOR_DAEMON_SETTINGS_DARWIN
COMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS COMPAT_OBJS += compat/fsmonitor/fsm-settings-unix.o COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_DAEMON_SETTINGS_DARWIN).o COMPAT_OBJS += compat/fsmonitor/fsm-path-utils-$(FSMONITOR_DAEMON_SETTINGS_DARWIN).o
endif
And the rest of the platforms remain as was, with FSMONITOR_DAEMON_BACKEND and FSMONITOR_OS_SETTINGS
-Eric
Quoted 24 lines
-----Original Message-----
From: Junio C Hamano <[email protected]>
Sent: Friday, October 14, 2022 7:33 PM
To: Eric DeCosta via GitGitGadget <[email protected]>
Cc: [email protected]; Eric Sunshine <[email protected]>; Ævar Arnfjörð Bjarmason <[email protected]>; Eric DeCosta <[email protected]>
Subject: Re: [PATCH v2 00/12] fsmonitor: Implement fsmonitor for LinuxQuoted 6 lines · Eric DeCosta via GitGitGadget
Goal is to deliver fsmonitor for Linux that is on par with fsmonitor for Windows and Mac OS.
This patch set builds upon previous work for done for Windows and Mac OS (first 6 patches) to implement a fsmonitor back-end for Linux based on the Linux inotify API.
Again, the first six patches are a part of what is queued as ed/fsmonitor-on- networked-macos that is now in 'next' but lacks a fix-up commit from Jeff King.
Quoted 6 lines
I understand that it might not be easy/possible (e.g. perhaps it is a limitation of GGG?), but I really prefer not to see them re-posted as part of this series, as I have to apply them and make sure there are no changes from the last one before discarding them.
Anyway, thanks for an update. Will requeue.
I looks like I can use GGG with the next branch, but I will have to open a new PR (and close the existing one).
-Eric
At $DAYJOB, we observed that this topic breaks MacOS builds with sha1dc:
$ make NO_APPLE_COMMON_CRYPTO=1 DC_SHA1=1 NO_OPENSSL=1 compat/fsmonitor/fsm-ipc-darwin.o
CC compat/fsmonitor/fsm-ipc-darwin.o
compat/fsmonitor/fsm-ipc-darwin.c:13:2: error: unknown type name 'SHA_CTX'; did you mean 'SHA1_CTX'?
SHA_CTX sha1ctx;
^~~~~~~
SHA1_CTX
./sha1dc/sha1.h:55:3: note: 'SHA1_CTX' declared here
} SHA1_CTX;
^
compat/fsmonitor/fsm-ipc-darwin.c:16:21: error: use of undeclared identifier 'SHA_DIGEST_LENGTH'
unsigned char hash[SHA_DIGEST_LENGTH];
^
compat/fsmonitor/fsm-ipc-darwin.c:31:2: error: implicit declaration of function 'SHA1_Init' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
SHA1_Init(&sha1ctx);
^
compat/fsmonitor/fsm-ipc-darwin.c:31:2: note: did you mean 'SHA1DCInit'?
./sha1dc/sha1.h:58:6: note: 'SHA1DCInit' declared here
void SHA1DCInit(SHA1_CTX*);
^
compat/fsmonitor/fsm-ipc-darwin.c:32:2: error: implicit declaration of function 'SHA1_Update' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
SHA1_Update(&sha1ctx, r->worktree, strlen(r->worktree));
^
compat/fsmonitor/fsm-ipc-darwin.c:32:2: note: did you mean 'SHA1DCUpdate'?
./sha1dc/sha1.h:96:6: note: 'SHA1DCUpdate' declared here
void SHA1DCUpdate(SHA1_CTX*, const char*, size_t);
^
compat/fsmonitor/fsm-ipc-darwin.c:33:2: error: implicit declaration of function 'SHA1_Final' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
SHA1_Final(hash, &sha1ctx);
^
compat/fsmonitor/fsm-ipc-darwin.c:33:2: note: did you mean 'SHA1DCFinal'?
./sha1dc/sha1.h:100:6: note: 'SHA1DCFinal' declared here
int SHA1DCFinal(unsigned char[20], SHA1_CTX*);
^
5 errors generated.
make: *** [compat/fsmonitor/fsm-ipc-darwin.o] Error 1Without NO_OPENSSL, this still fails, but with slightly different error messages.
$ make NO_APPLE_COMMON_CRYPTO=1 DC_SHA1=1 compat/fsmonitor/fsm-ipc-darwin.o
CC compat/fsmonitor/fsm-ipc-darwin.o
compat/fsmonitor/fsm-ipc-darwin.c:31:2: error: 'SHA1_Init' is deprecated [-Werror,-Wdeprecated-declarations]
SHA1_Init(&sha1ctx);
^
/opt/local/include/openssl/sha.h:49:1: note: 'SHA1_Init' has been explicitly marked deprecated here OSSL_DEPRECATEDIN_3_0 int SHA1_Init(SHA_CTX *c);
^
/opt/local/include/openssl/macros.h:182:49: note: expanded from macro 'OSSL_DEPRECATEDIN_3_0'
# define OSSL_DEPRECATEDIN_3_0 OSSL_DEPRECATED(3.0)
^
/opt/local/include/openssl/macros.h:62:52: note: expanded from macro 'OSSL_DEPRECATED'
# define OSSL_DEPRECATED(since) __attribute__((deprecated))
^
compat/fsmonitor/fsm-ipc-darwin.c:32:2: error: 'SHA1_Update' is deprecated [-Werror,-Wdeprecated-declarations]
SHA1_Update(&sha1ctx, r->worktree, strlen(r->worktree));
^
/opt/local/include/openssl/sha.h:50:1: note: 'SHA1_Update' has been explicitly marked deprecated here OSSL_DEPRECATEDIN_3_0 int SHA1_Update(SHA_CTX *c, const void *data, size_t len);
^
/opt/local/include/openssl/macros.h:182:49: note: expanded from macro 'OSSL_DEPRECATEDIN_3_0'
# define OSSL_DEPRECATEDIN_3_0 OSSL_DEPRECATED(3.0)
^
/opt/local/include/openssl/macros.h:62:52: note: expanded from macro 'OSSL_DEPRECATED'
# define OSSL_DEPRECATED(since) __attribute__((deprecated))
^
compat/fsmonitor/fsm-ipc-darwin.c:33:2: error: 'SHA1_Final' is deprecated [-Werror,-Wdeprecated-declarations]
SHA1_Final(hash, &sha1ctx);
^
/opt/local/include/openssl/sha.h:51:1: note: 'SHA1_Final' has been explicitly marked deprecated here OSSL_DEPRECATEDIN_3_0 int SHA1_Final(unsigned char *md, SHA_CTX *c);
^
/opt/local/include/openssl/macros.h:182:49: note: expanded from macro 'OSSL_DEPRECATEDIN_3_0'
# define OSSL_DEPRECATEDIN_3_0 OSSL_DEPRECATED(3.0)
^
/opt/local/include/openssl/macros.h:62:52: note: expanded from macro 'OSSL_DEPRECATED'
# define OSSL_DEPRECATED(since) __attribute__((deprecated))
^
3 errors generated.
make: *** [compat/fsmonitor/fsm-ipc-darwin.o] Error 1
Quoted 20 lines · Eric DeCosta via GitGitGadget
Goal is to deliver fsmonitor for Linux that is on par with fsmonitor for Windows and Mac OS.
This patch set builds upon previous work for done for Windows and Mac OS (first 6 patches) to implement a fsmonitor back-end for Linux based on the Linux inotify API. inotify differs significantly from the equivalent Windows and Mac OS APIs in that a watch must be registered for every directory of interest (rather than a singular watch at the root of the directory tree) and special care must be taken to handle directory renames correctly.
More information about inotify:
https://man7.org/linux/man-pages/man7/inotify.7.htmlv1 differs from v0:
* Code review feedback
* Update how and which code can be shared between Mac OS and Linux
* Increase polling frequency to every 1ms (matches Mac OS)
* Updates to t7527 to improve test stability
Quoted 9 lines · Eric DeCosta
Quoted 6 lines
I understand that it might not be easy/possible (e.g. perhaps it is a limitation of GGG?), but I really prefer not to see them re-posted as part of this series, as I have to apply them and make sure there are no changes from the last one before discarding them.
Anyway, thanks for an update. Will requeue.
I looks like I can use GGG with the next branch, but I will have to open a new PR (and close the existing one).
Please don't.
The macOS topic (thanks for working on it, by the way) is about to graduate to 'master', and the situation would probably "improve" anyway when it happens, I hope.
Thanks.
Quoted 1 line · Glen Choo
At $DAYJOB, we observed that this topic breaks MacOS builds with sha1dc:
Thanks for a report.
How dissapointing. The thing is that the topic has been in 'next' since the 11th (i.e. early last week), and I know that you guys rely on the tip of 'next' in working order to cut your internal releases, but we did not hear about this until now. What makes it taste even worse is that nobody else caught this, even though we seem to have a couple of macOS jobs at GitHub Actions CI, there we didn't see any breakage related to this.
Possible action items:
* See what configurations our two macOS jobs are using. If neither is using sha1dc, I would say that is criminal [*] and at least one of them should be updated to do so right away.
* The "two macOS CI jobs sharing too many configuration knobs" may not be limited to just SHA-1 implementation, but unlike Linux builds and tests, we may have similar "monoculture" issue in our macOS CI builds. Those users, who depend on macOS port being healthy, should help identify unnecessary overlaps between the two, and more importantly, make sure we have CI builds with configuration similar to what they actually use.
* Adding a few build-only-without-tests CI jobs also might help.
* Those who depend on working macOS port, especially those with corporate backing who choose to use configurations that are different from what we have CI builds for, are requested to arrange a more frequent build test to catch a problem like this much earlier.
Anything else I forgot that we can do to improve the situation? I personally do not use macOS, I am not interested in using one, but I do value those who choose to use macOS have happy git working on their platform, so the stakeholders need to chip in.
Thanks.
[Footnote]
* Until the world migrates over to SHA-256, the collision detecting SHA-1 implementation is what we must use unless there is a strong reason not to. If we are not testing something that ought to be the default, we are not doing a very good job.
Re: [PATCH v2 07/12] fsmonitor: prepare to share code between Mac OS and Linux
In reply to Eric DeCostaQuoted 8 lines · Eric DeCosta
Quoted 8 lines
Ugly.
One overrides backend with common, while the other one doesn't. That asymmetry alone should stop us and wonder if there is something fishy in the approach that can be improved. It makes it look like the word "common" means something quite different between the -ipc and the - settings world.
Quoted 8 lines
Quoted 6 lines
I suspect that in both, you should not expose "unix" to this part of the
Makefile. Linux and macOS occasionally being similar in some places does not have to be exposed here. INstead you can use backend "linux" and "macos", whose C sources may include from a separate C source file whose name may contain "unix". That would allow you to get rid ofFSMONITOR_DAEMON_COMMONin a cleaner way.Let me see if I am understanding you correctly. Are you suggesting something like:
Not really.
What I had in mind was an arrangement more like
$ for variant in linux macos; do grep include fsm-ipc-$variant.c; done
#include "fsm-ipc-common-unix.cinclude"
#include "fsm-ipc-common-unix.cinclude"and COMPAT_OBJS knowning only about fsm-ipc-linux.o or fsm-ipc-macos.o, depending on which platform you are building.
Re: [PATCH v2 09/12] fsmonitor: implement filesystem change listener for Linux
In reply to Eric DeCosta via GitGitGadgetcompat/fsmonitor/fsm-listen-linux.c line 583
Quoted 5 lines · Eric DeCosta via GitGitGadget
From: Eric DeCosta <[email protected]>
[...]compat/fsmonitor/fsm-listen-linux.c new file +3
-0+strbuf_add (&path, w->dir,strlen (w->dir)); +strbuf_addch (&path,'/' ); +strbuf_add (&path, event->name,strlen (event->name));
Don't do strbuf_add(&buf, x, strlen(x), just use strbuf_addstr(&buf, x) instead.
The same goes for a couple of existing occurances that hit "master" already in the just-merged fsmonitor topic, but in this case we can change it in-flight still. Thanks!
Re: [PATCH v2 02/12] fsmonitor: relocate socket file if .git directory is remote
In reply to Eric DeCosta via GitGitGadgetcompat/fsmonitor/fsm-ipc-darwin.c line 12
Quoted 5 lines · Eric DeCosta via GitGitGadget
From: Eric DeCosta <[email protected]>
[..]compat/fsmonitor/fsm-ipc-darwin.c new file +3
-0+const char *fsmonitor_ipc__get_path (struct repository *r) +{ +static const char *ipc_path =NULL ;
Urm, I see some of this interface made it into master already with 6beb2688d33 (fsmonitor: relocate socket file if .git directory is remote, 2022-10-04), but this is just weird. We already have "struct fsmonitor_settings" which "lazily loads" these various settings.
So e.g. in refresh_fsmonitor() we might get the hook path, which we "lazy load" and cache there.
And here we have an extra layer of another type of lazy loading, and this will e.g. be called from fsmonitor-settings.c.
Is there really a good reason for why the relevant codepaths can't either not-lazy-load this (e.g. when we get the socket), or why we can't just use fsmonitor-settings.c's existing caching?
Just wondering...
compat/fsmonitor/fsm-ipc-darwin.c line 44
Quoted 10 lines
compat/fsmonitor/fsm-ipc-darwin.c new file +10
-0+repo_config_get_string (r,"fsmonitor.socketdir" , &sock_dir); + +/* Create the socket file in either socketDir or $HOME */ +if (sock_dir && *sock_dir) { +strbuf_addf (&ipc_file,"%s/.git-fsmonitor-%s" , + sock_dir,hash_to_hex (hash)); + }else { +strbuf_addf (&ipc_file,"~/.git-fsmonitor-%s" ,hash_to_hex (hash)); + } +free (sock_dir);
Instead of this init sock_dir to NULL & then check if it's non-NULL etc. Just do:
char *sock_dir; if (!repo_config_get_string_tmp(r, "fsmonitor.socketdir", &sock_dir) && *sock_dir) ... else ...
Note also the *_tmp(), then you won't have to free() it.
compat/fsmonitor/fsm-ipc-darwin.c line 51
Quoted 7 lines
compat/fsmonitor/fsm-ipc-darwin.c new file +7
-0+ + ipc_path =interpolate_path (ipc_file.buf,1 ); +if (!ipc_path) +die (_ ("Invalid path: %s" ), ipc_file.buf); + +strbuf_release (&ipc_file); +return ipc_path;
Why can't we just do something more simpler like this in the *nix case?
I think because you're creating this socket for a NFS-mounted repo, so of course we might have multiple clients.
But you also seem to be making it "unique" by hashing r->worktree, which is the path to the repo work tree. At least in my use-cases for NFS mounted repos (which have been limited) all of those paths would be the same for me across my systems, because I'd have them at /home/avar/g/some-git-tree/...
In builtin/gc.c we create a "unique for system" lock file using xgethostname(), isn't that a lot simpler & closer to what you actually want?
In any case, some details about motivations, why etc. are sorely missing in the commit message. We can see *what* you're doing, but not *why*, or what other simpler solutions etc. might have been tried & dismissed...
Re: [PATCH 01/12] fsmonitor: refactor filesystem checks to common interface
In reply to Eric DeCosta via GitGitGadgetQuoted 11 lines · Eric DeCosta via GitGitGadget
From: Eric DeCosta <[email protected]>
[...]compat/fsmonitor/fsm-path-utils-darwin.c new file +9
-0+int fsmonitor__get_fs_info (const char *path,struct fs_info *fs_info) +{ +struct statfs fs; +if (statfs (path, &fs) ==-1 ) { +int saved_errno = errno; +trace_printf_key (&trace_fsmonitor,"statfs('%s') failed: %s" , + path,strerror (saved_errno)); + errno = saved_errno; +return -1 ;
compat/fsmonitor/fsm-path-utils-darwin.c line 24
Quoted 10 lines
compat/fsmonitor/fsm-path-utils-darwin.c new file +10
-0+ } + +trace_printf_key (&trace_fsmonitor, +"statfs('%s') [type 0x%08x][flags 0x%08x] '%s'" , + path, fs.f_type, fs.f_flags, fs.f_fstypename); + +if (!(fs.f_flags &MNT_LOCAL )) + fs_info->is_remote =1 ; +else + fs_info->is_remote =0 ;
Odd to go through the trouble of inverting the flag check like this just to assign to it manually. I'd think:
if (flags & LOCAL) is_remote = 0; else is_remote = 1;
Would make more sense, or actually if you do invert it:
fs_info->is_remote = !(fs.f_flags & MNT_LOCAL);
No?
compat/fsmonitor/fsm-path-utils-darwin.c line 26
Quoted 1 line
compat/fsmonitor/fsm-path-utils-darwin.c new file +1
-0+ fs_info->typename =xstrdup (fs.f_fstypename);
Instead of xstrdup()-ing this...
Quoted 4 lines
compat/fsmonitor/fsm-path-utils-darwin.c new file +4
-0+ +trace_printf_key (&trace_fsmonitor, +"'%s' is_remote: %d" , + path, fs_info->is_remote);
compat/fsmonitor/fsm-path-utils-darwin.c line 40
Quoted 10 lines
compat/fsmonitor/fsm-path-utils-darwin.c new file +10
-0+return 0 ; +} + +int fsmonitor__is_fs_remote (const char *path) +{ +struct fs_info fs; +if (fsmonitor__get_fs_info (path, &fs)) +return -1 ; + +free (fs.typename);
...just to have some callers free() it again, maybe it makes sense to say that return a const char *, and if you need it past the call to fsmonitor__get_fs_info() you should do the xstrdup() yourself.
Skimming the end-state the OSX one gives you a copy of your own, as it's stuck into the struct you provided.
For Linux we don't get a copy, but it's filled per getmntent() call.
Anyway, small potatoes...
compat/fsmonitor/fsm-path-utils-win32.c line 22
Quoted 5 lines
compat/fsmonitor/fsm-path-utils-win32.c new file +5
-0+if (h ==INVALID_HANDLE_VALUE ) { +error (_ ("[GLE %ld] unable to open for read '%ls'" ), +GetLastError (), wpath); +return -1 ; + }
Do away with the braces and just "return error(..."
compat/fsmonitor/fsm-path-utils-win32.c line 32
Quoted 10 lines
compat/fsmonitor/fsm-path-utils-win32.c new file +10
-0+ +if (!GetFileInformationByHandleEx (h, FileRemoteProtocolInfo, + &proto_info,sizeof (proto_info))) { +error (_ ("[GLE %ld] unable to get protocol information for '%ls'" ), +GetLastError (), wpath); +CloseHandle (h); +return -1 ; + } + +CloseHandle (h);
And this duplication you can avoid with an earlier:
int ret = 0;
Then
ret = error(...) goto cleanup;
and then at the end:
cleanup:
CloseHandle(h) return ret;
compat/fsmonitor/fsm-path-utils-win32.c line 81
Quoted 7 lines
compat/fsmonitor/fsm-path-utils-win32.c new file +7
-0+/* +* Do everything in wide chars because the drive letter might be +* a multi-byte sequence. See win32_has_dos_drive_prefix(). +*/ +if (xutftowcs_path (wpath, path) <0 ) { +return -1 ; + }
Don't need the braces.
Quoted 11 lines
compat/fsmonitor/fsm-path-utils-win32.c new file +11
-0+ +/* +* GetDriveTypeW() requires a final slash. We assume that the +* worktree pathname points to an actual directory. +*/ + wlen =wcslen (wpath); +if (wpath[wlen -1 ] !=L'\\' && wpath[wlen -1 ] !=L'/' ) { + wpath[wlen++] =L'\\' ; + wpath[wlen] =0 ; + } +
compat/fsmonitor/fsm-path-utils-win32.c line 102
Quoted 10 lines
compat/fsmonitor/fsm-path-utils-win32.c new file +10
-0+/* +* Normalize the path. If nothing else, this converts forward +* slashes to backslashes. This is essential to get GetDriveTypeW() +* correctly handle some UNC "\\server\share\..." paths. +*/ +if (!GetFullPathNameW (wpath,MAX_PATH , wfullpath,NULL )) { +return -1 ; + } + + driveType =GetDriveTypeW (wfullpath);
Can't you just do away with this driveType variable and assign directly to fs_info->is_remote if it's == DRIVE_REMOTE?...
compat/fsmonitor/fsm-path-utils-win32.c line 105
Quoted 3 lines
compat/fsmonitor/fsm-path-utils-win32.c new file +3
-0+trace_printf_key (&trace_fsmonitor, +"DriveType '%s' L'%ls' (%u)" , + path, wfullpath, driveType);
...well, not due to this, but do we really need to log the raw value & is that meaningful? A user of the trace2 logs would need to reverse map that, doesn't the API provide some "what is it then?" function?
compat/fsmonitor/fsm-path-utils-win32.c line 108
Quoted 3 lines
compat/fsmonitor/fsm-path-utils-win32.c new file +3
-0+ +if (driveType ==DRIVE_REMOTE ) { + fs_info->is_remote =1 ;
Here we do that assignment...
compat/fsmonitor/fsm-path-utils-win32.c line 117
Quoted 9 lines
compat/fsmonitor/fsm-path-utils-win32.c new file +9
-0+if (check_remote_protocol (wfullpath) <0 ) +return -1 ; + }else { + fs_info->is_remote =0 ; + } + +trace_printf_key (&trace_fsmonitor, +"'%s' is_remote: %d" , + path, fs_info->is_remote);
ditto just "is remote?"
Re: [PATCH 11/12] fsmonitor: test updates
In reply to Eric DeCosta via GitGitGadgett/t7527-builtin-fsmonitor.sh start_daemon () { · line 82
Quoted 8 lines · Eric DeCosta via GitGitGadget
From: Eric DeCosta <[email protected]>
t/t7527-builtin-fsmonitor.sh +6
-0+wait_for_update () { + func=$1 && + file=$2 && + sz=$(wc -c < "$file") && + last=0 && + $func &&
Odd not to quote "$func" here, but anyway...
t/t7527-builtin-fsmonitor.sh create_files () { · line 403
Quoted 3 lines
t/t7527-builtin-fsmonitor.sh +3
-0+rename_directory () { + mv dirtorename dirrenamed +}
...uh, we need to wrap "mv"? Why?
t/t7527-builtin-fsmonitor.sh test_expect_success 'rename directory' ' · line 519
Quoted 2 lines
t/t7527-builtin-fsmonitor.sh +1
-1- mv dirtorename dirrenamed && + wait_for_update rename_directory "$PWD/.git/trace" &&
Just to get around some quoting issue here?
Re: [PATCH 12/12] fsmonitor: update doc for Linux
In reply to Eric DeCosta via GitGitGadgetQuoted 15 lines · Eric DeCosta via GitGitGadget
From: Eric DeCosta <[email protected]>
Update the documentation for Linux.
Signed-off-by Eric DeCosta <[email protected]>--- Documentation/config/fsmonitor--daemon.txt | 4 ++-- Documentation/git-fsmonitor--daemon.txt | 24 ++++++++++++++-------- 2 files changed, 18 insertions(+), 10 deletions(-)Documentation/config/fsmonitor--daemon.txt +0
-0
Documentation/config/fsmonitor--daemon.txt fsmonitor.allowRemote:: · line 11
Quoted 10 lines
Documentation/config/fsmonitor--daemon.txt +2
-2@@ -4,8 +4,8 @@ fsmonitor.allowRemote:: behavior. Only respected when `core.fsmonitor` is set to `true`. fsmonitor.socketDir:: - This Mac OS-specific option, if set, specifies the directory in + Mac OS and Linux-specific option. If set, specifies the directory in which to create the Unix domain socket used for communication between the fsmonitor daemon and various Git commands. The directory must - reside on a native Mac OS filesystem. Only respected when `core.fsmonitor` + reside on a native filesystem. Only respected when `core.fsmonitor` is set to `true`.
I think this should be squashed into the relevant commit(s) when we start supporting this fsmonitor.socketDir for that platform.
But on the content: Shouldn't we just reword this to say something like "*nix"-specific, we only have one platform which is likely to not have "sockets", i.e. Windows, no?
So rather than having an ever growing list of OS's that'll inevitably grow to something like:
Mac OS, Linux, FreeBSD, NetBSD, OpenBSD, GNU/Hurd-specific option, if set...
Let's find some way to refer to the platforms we do support, and instead mention the only one we don't support sockets on?
Documentation/git-fsmonitor--daemon.txt repositories; this may be overridden by setting `fsmonitor.allowRemote` to · line 79
Quoted 10 lines
Documentation/git-fsmonitor--daemon.txt +1
-1@@ -76,23 +76,31 @@ repositories; this may be overridden by setting `fsmonitor.allowRemote` to correctly with all network-mounted repositories and such use is considered experimental. -On Mac OS, the inter-process communication (IPC) between various Git +On Linux and Mac OS, the inter-process communication (IPC) between various Git
Ditto.
Quoted 2 lines · Eric DeCosta via GitGitGadget
Goal is to deliver fsmonitor for Linux that is on par with fsmonitor for Windows and Mac OS.
I applaud the effort, I gave this some light reviewing, I hope it helps.
Quoted 9 lines
This patch set builds upon previous work for done for Windows and Mac OS (first 6 patches) to implement a fsmonitor back-end for Linux based on the Linux inotify API. inotify differs significantly from the equivalent Windows and Mac OS APIs in that a watch must be registered for every directory of interest (rather than a singular watch at the root of the directory tree) and special care must be taken to handle directory renames correctly.
More information about inotify:
https://man7.org/linux/man-pages/man7/inotify.7.html
You haven't said why/if you considered using fanotify() instead of inotify(), which seems like a more natural target to me. It's unlikely that anyone who cares to use this isn't also using a new enough kernel.
See previous on-list discussions at:
https://lore.kernel.org/git/?q=fanotify
I think it would address some of the rac econditions you're mentioning here.
Hi Eric,
Quoted 18 lines · Eric DeCosta
Quoted 18 lines
-----Original Message-----
From: Junio C Hamano <[email protected]>
Sent: Friday, October 14, 2022 7:33 PM
To: Eric DeCosta via GitGitGadget <[email protected]>
Cc: [email protected]; Eric Sunshine <[email protected]>; Ævar Arnfjörð Bjarmason <[email protected]>; Eric DeCosta <[email protected]>
Subject: Re: [PATCH v2 00/12] fsmonitor: Implement fsmonitor for LinuxQuoted 6 lines · Eric DeCosta via GitGitGadget
Goal is to deliver fsmonitor for Linux that is on par with fsmonitor for Windows and Mac OS.
This patch set builds upon previous work for done for Windows and Mac OS (first 6 patches) to implement a fsmonitor back-end for Linux based on the Linux inotify API.
Quoted 8 lines
Quoted 8 lines
Again, the first six patches are a part of what is queued as ed/fsmonitor-on- networked-macos that is now in 'next' but lacks a fix-up commit from Jeff King.
I understand that it might not be easy/possible (e.g. perhaps it is a limitation of GGG?), but I really prefer not to see them re-posted as part of this series, as I have to apply them and make sure there are no changes from the last one before discarding them.
GitGitGadget mirrors all branches from gitster/git to gitgitgadget/git, so if you open a PR in the latter repository, you can use all of those branches as targets.
But this PR is in git/git, which does not offer that.
Quoted 3 lines
Quoted 1 line
Anyway, thanks for an update. Will requeue.
I looks like I can use GGG with the next branch, but I will have to open a new PR (and close the existing one).
No, you do not have to open a new PR for that.
You just hit the Edit button on the top (as if you wanted to change the title) and then select the target ("base") branch in the drop-down box under the title.
Ciao,
Dscho
Cc-ed Johannes, who would know a lot more about CI than I do.
Quoted 10 lines · Junio C Hamano
Quoted 1 line · Glen Choo
At $DAYJOB, we observed that this topic breaks MacOS builds with sha1dc:
Thanks for a report.
How dissapointing. The thing is that the topic has been in 'next' since the 11th (i.e. early last week), and I know that you guys rely on the tip of 'next' in working order to cut your internal releases, but we did not hear about this until now.
Yes. Unfortunately, we (Google's Git team) release on a weekly cadence; we merge on Fridays and build on Mondays (PST), which makes this timing extremely unfortunate.
Quoted 5 lines
Possible action items:
* See what configurations our two macOS jobs are using. If neither is using sha1dc, I would say that is criminal [*] and at least one of them should be updated to do so right away.
I'm not too familiar with the CI, but I took a quick peek at ci/lib.sh and noticed that none of the jobs build with sha1dc, not even the Linux or Windows ones, so..
Quoted 7 lines
* The "two macOS CI jobs sharing too many configuration knobs" may not be limited to just SHA-1 implementation, but unlike Linux builds and tests, we may have similar "monoculture" issue in our macOS CI builds. Those users, who depend on macOS port being healthy, should help identify unnecessary overlaps between the two, and more importantly, make sure we have CI builds with configuration similar to what they actually use.
I don't think this is a macOS-specific issue; our CI just doesn't do a good job with testing various build configurations.
Quoted 1 line
* Adding a few build-only-without-tests CI jobs also might help.
This sounds like the way to go IMO. It might be too expensive to run the full test suite on every build configuration, but build-without-test might be ok.
Quoted 5 lines
* Those who depend on working macOS port, especially those with corporate backing who choose to use configurations that are different from what we have CI builds for, are requested to arrange a more frequent build test to catch a problem like this much earlier.
I wished we had caught it sooner too. The folks here generally agree that our weekly release cycle is not ideal for reasons such as this. Hopefully this is good motivation to move that work forward, though I can't promise anything right now.
Quoted 4 lines
Anything else I forgot that we can do to improve the situation? I personally do not use macOS, I am not interested in using one, but I do value those who choose to use macOS have happy git working on their platform, so the stakeholders need to chip in.
There's nothing else I can think of at the moment. Thanks for your patience and for moving the conversation along.
Quoted 10 lines
Thanks.
[Footnote]
* Until the world migrates over to SHA-256, the collision detecting SHA-1 implementation is what we must use unless there is a strong reason not to. If we are not testing something that ought to be the default, we are not doing a very good job.
Quoted 4 lines · Glen Choo
I wished we had caught it sooner too. The folks here generally agree that our weekly release cycle is not ideal for reasons such as this. Hopefully this is good motivation to move that work forward, though I can't promise anything right now.
It is perfectly OK to have an automated trial build job that runs more frequently than your weekly release cycle, though. It should usually yield only a single bit of usable information (e.g. "there is no 'does not even build from the source' issue in upstream") that may give you assurance (e.g. "if we maintain the course, the next real build for release would hopefully go smoothly"), but when it breaks, you have more time to react.
Thanks.
Quoted 6 lines · Junio C Hamano
Quoted 1 line · Glen Choo
At $DAYJOB, we observed that this topic breaks MacOS builds with sha1dc:
Thanks for a report.
Quoted 7 lines
How dissapointing. The thing is that the topic has been in 'next' since the 11th (i.e. early last week), and I know that you guys rely on the tip of 'next' in working order to cut your internal releases, but we did not hear about this until now. What makes it taste even worse is that nobody else caught this, even though we seem to have a couple of macOS jobs at GitHub Actions CI, there we didn't see any breakage related to this.
FWIW I see you caught it on the 9th in
https://lore.kernel.org/git/[email protected]/, but then the base topic was merged down on the 17th.
Quoted 5 lines
Possible action items:
* See what configurations our two macOS jobs are using. If neither is using sha1dc, I would say that is criminal [*] and at least one of them should be updated to do so right away.
I submitted a v2 of my series to finally make OSX use SHA1DC by default:
https://lore.kernel.org/git/[email protected]/
As part of that we'll CI that & the current "apple common crypto" implementations, currently we just do the latter, which is why we didn't catch this.
We could follow-up and CI the OpenSSL one too, but that was a larger change, so I punted on it.
In any case,
https://lore.kernel.org/git/[email protected]/
in that series un-breaks master, maybe you're interested in peeling it off & fast-tracking it? I didn't submit it separately because I don't know how much of a rush we're in there (given that CI isn't broken).
Quoted 21 lines · Glen Choo
Cc-ed Johannes, who would know a lot more about CI than I do.
Quoted 10 lines · Junio C Hamano
Quoted 1 line · Glen Choo
At $DAYJOB, we observed that this topic breaks MacOS builds with sha1dc:
Thanks for a report.
How dissapointing. The thing is that the topic has been in 'next' since the 11th (i.e. early last week), and I know that you guys rely on the tip of 'next' in working order to cut your internal releases, but we did not hear about this until now.
Yes. Unfortunately, we (Google's Git team) release on a weekly cadence; we merge on Fridays and build on Mondays (PST), which makes this timing extremely unfortunate.
Quoted 2 lines
Possible action items:
Quoted 7 lines
Quoted 3 lines
* See what configurations our two macOS jobs are using. If neither is using sha1dc, I would say that is criminal [*] and at least one of them should be updated to do so right away.
I'm not too familiar with the CI, but I took a quick peek at
ci/lib.shand noticed that none of the jobs build with sha1dc, not even the Linux or Windows ones, so..
All of our jobs except the OSX one build with SHA1_DC, because it's the default.
Per my just-sent
https://lore.kernel.org/git/[email protected]/
the blind spot has been lack fo SHA1_DC on OSX, for others it's the reverse, we don't test e.g. BLK_SHA1.
In practice we've been catching SHA-implementation specific code early because the OSX implementation was different, but in this case it's OSX-only code, so it only supported the Apple Common Crypto backend.
Quoted 10 lines
Quoted 7 lines
* The "two macOS CI jobs sharing too many configuration knobs" may not be limited to just SHA-1 implementation, but unlike Linux builds and tests, we may have similar "monoculture" issue in our macOS CI builds. Those users, who depend on macOS port being healthy, should help identify unnecessary overlaps between the two, and more importantly, make sure we have CI builds with configuration similar to what they actually use.
I don't think this is a macOS-specific issue; our CI just doesn't do a good job with testing various build configurations.
Yes, definitely.
Quoted 5 lines
Quoted 1 line
* Adding a few build-only-without-tests CI jobs also might help.
This sounds like the way to go IMO. It might be too expensive to run the full test suite on every build configuration, but build-without-test might be ok.
Yes, there's not much point in running the full tests for some of these variants.
A lot of it we can also get for free, e.g. we run some linux jobs with clang, some with gcc etc., we could also run one with BLK_SHA1, one with OPENSSL_SHA1 etc.
Quoted 8 lines · Ævar Arnfjörð Bjarmason
Quoted 3 lines · Glen Choo
I'm not too familiar with the CI, but I took a quick peek at
ci/lib.shand noticed that none of the jobs build with sha1dc, not even the Linux or Windows ones, so..All of our jobs except the OSX one build with
SHA1_DC, because it's the default.
Quoted 8 lines
Per my just-sent
https://lore.kernel.org/git/[email protected]/
the blind spot has been lack foSHA1_DCon OSX, for others it's the reverse, we don't test e.g.BLK_SHA1.In practice we've been catching SHA-implementation specific code early because the OSX implementation was different, but in this case it's OSX-only code, so it only supported the Apple Common Crypto backend.
I don't know how germane it is to the current thread, but previous discussions[1,2,3,4] favored dropping use of Apple's Common Crypto altogether since it doesn't seem to buy us much (or anything) and is incomplete; it doesn't support all of the OpenSSL API Git uses.
[1]: https://lore.kernel.org/git/CAPig+cTfMx_kwUAxBRHp6kNSOtXsdsv=odUQSRYVpV21DnRuvA@mail.gmail.com/
[2]: https://lore.kernel.org/git/CAMYxyaVQyVRQb-b0nVv412tMZ3rEnOfUPRakg2dEREg5_Ba5Ag@mail.gmail.com/T/
[3]: https://lore.kernel.org/git/[email protected]/
[4]: https://lore.kernel.org/git/CAPig+cQ5kKAt2_RQnqT7Rn=uGmHV9VvxpQ+UgDPOj=D=pq6arg@mail.gmail.com/
Quoted 16 lines · Ævar Arnfjörð Bjarmason
Quoted 6 lines · Junio C Hamano
Quoted 1 line · Glen Choo
At $DAYJOB, we observed that this topic breaks MacOS builds with sha1dc:
Thanks for a report.
Quoted 7 lines
How dissapointing. The thing is that the topic has been in 'next' since the 11th (i.e. early last week), and I know that you guys rely on the tip of 'next' in working order to cut your internal releases, but we did not hear about this until now. What makes it taste even worse is that nobody else caught this, even though we seem to have a couple of macOS jobs at GitHub Actions CI, there we didn't see any breakage related to this.
Quoted 3 lines
FWIW I see you caught it on the 9th in
https://lore.kernel.org/git/[email protected]/, but then the base topic was merged down on the 17th.
Heh, I already forgot I made that comment myself ;-) Thanks for reminding, and thanks for picking the single step to fast-track the fix.
Quoted 4 lines · Eric Sunshine
I don't know how germane it is to the current thread, but previous discussions[1,2,3,4] favored dropping use of Apple's Common Crypto altogether since it doesn't seem to buy us much (or anything) and is incomplete; it doesn't support all of the OpenSSL API Git uses.
Yeah, that matches my recollection. Unless the situation has much changed, dropping it may not be a bad thing to do.
But
* Fixing the fsmonitor code so that it can also be used with things other than Common Crypto is the most urgent. The topic gave us a grave regression (those who used to successfully build Git can no longer build with their favourite configuration).
* Updating the build procedure so that sha1dc is used by default everywhere is a good idea, but that is less urgent and should be done separately, preferrably long after the dust settles from the above.
* Removing Common Crypto support may not be a bad idea, but that is even less urgent, unless the support burden is slowing us down or forcing us to settle on common set of features that is too limiting.
Thanks.
Quoted 24 lines · Eric Sunshine
Quoted 8 lines · Ævar Arnfjörð Bjarmason
Quoted 3 lines · Glen Choo
I'm not too familiar with the CI, but I took a quick peek at
ci/lib.shand noticed that none of the jobs build with sha1dc, not even the Linux or Windows ones, so..All of our jobs except the OSX one build with
SHA1_DC, because it's the default.Quoted 8 lines
Per my just-sent
https://lore.kernel.org/git/[email protected]/
the blind spot has been lack foSHA1_DCon OSX, for others it's the reverse, we don't test e.g.BLK_SHA1.In practice we've been catching SHA-implementation specific code early because the OSX implementation was different, but in this case it's OSX-only code, so it only supported the Apple Common Crypto backend.
I don't know how germane it is to the current thread, but previous discussions[1,2,3,4] favored dropping use of Apple's Common Crypto altogether since it doesn't seem to buy us much (or anything) and is incomplete; it doesn't support all of the OpenSSL API Git uses.
Quoted 4 lines
[1]: https://lore.kernel.org/git/CAPig+cTfMx_kwUAxBRHp6kNSOtXsdsv=odUQSRYVpV21DnRuvA@mail.gmail.com/
[2]: https://lore.kernel.org/git/CAMYxyaVQyVRQb-b0nVv412tMZ3rEnOfUPRakg2dEREg5_Ba5Ag@mail.gmail.com/T/
[3]: https://lore.kernel.org/git/[email protected]/
[4]: https://lore.kernel.org/git/CAPig+cQ5kKAt2_RQnqT7Rn=uGmHV9VvxpQ+UgDPOj=D=pq6arg@mail.gmail.com/
Yeah, maybe. I'd be 100% OK with that happening, but I don't use OSX outside fo CI really, so I wanted to avoid scope creep to "non-SHA stuff we use OpenSSL/AppleCommonCrypto/whatever" for, in the cases where we can/do use OpenSSL/AppleCommonCrypto/whatever for SHA also.
I.e. that's basically a question about what https://, git-imap-send etc. should be using if not vanilla OpenSSL & the like.
But if you/someone can come up with a patch that confidently asserts that it's useless & drops it I'd be happy to either integrate it & submit it as part of this series if it makes things simpler, or alternatively re-roll on top of it.
I'm just not confident in making that case myself, since I hardly use the platform, am not too familiar with the various concerns (aside from skimming the links above), and don't really have interest in pursuing that.
OTOH if you do want to make the case for dropping Apple Common Crypto that would also presumably be much easier after this series, once we're past justifying the hurdle of "wait, we're switching the thing we use for SHA-1 by default, which we build practically everything in git on?" :)
1. https://lore.kernel.org/git/CAPig+cQ5kKAt2_RQnqT7Rn=uGmHV9VvxpQ+UgDPOj=D=pq6arg@mail.gmail.com/
Quoted 12 lines · Ævar Arnfjörð Bjarmason
Quoted 10 lines · Eric Sunshine
Quoted 3 lines · Ævar Arnfjörð Bjarmason
In practice we've been catching SHA-implementation specific code early because the OSX implementation was different, but in this case it's OSX-only code, so it only supported the Apple Common Crypto backend.
I don't know how germane it is to the current thread, but previous discussions[1,2,3,4] favored dropping use of Apple's Common Crypto altogether since it doesn't seem to buy us much (or anything) and is incomplete; it doesn't support all of the OpenSSL API Git uses.
Quoted 4 lines
Yeah, maybe. I'd be 100% OK with that happening, but I don't use OSX outside fo CI really, so I wanted to avoid scope creep to "non-SHA stuff we use OpenSSL/AppleCommonCrypto/whatever" for, in the cases where we can/do use OpenSSL/AppleCommonCrypto/whatever for SHA also.
Indeed, I was not suggesting that retirement of AppleCommonCrypto be tackled by you or by the series you posted to fix the build failure; doing so is well outside the scope of the immediate problem. I brought up those old discussions only as reference and as a pointer that whatever fix is eventually adopted for the immediate build problem need not necessarily bend over backward for AppleCommonCrypto support if a long-term goal is to drop AppleCommonCrypto (i.e. do as little as necessary to keep AppleCommonCrypto in a working state, but don't go overboard trying to give it first-class support since it will never be a complete replacement for OpenSSL).
Quoted 9 lines
But if you/someone can come up with a patch that confidently asserts that it's useless & drops it I'd be happy to either integrate it & submit it as part of this series if it makes things simpler, or alternatively re-roll on top of it.
I'm just not confident in making that case myself, since I hardly use the platform, am not too familiar with the various concerns (aside from skimming the links above), and don't really have interest in pursuing that.
I was not suggesting holding up your series or integrating retirement of AppleCommonCrypto with it; they are separate concerns.
Quoted 4 lines
OTOH if you do want to make the case for dropping Apple Common Crypto that would also presumably be much easier after this series, once we're past justifying the hurdle of "wait, we're switching the thing we use for SHA-1 by default, which we build practically everything in git on?"
My "don't know how germane it is to the current thread" observation was in response to the fsmonitor thread, before I ever saw the series you posted for fixing the build failure, so it wasn't given as any sort of feedback on your series, and wasn't asking you to incorporate retirement of AppleCommonCrypto into your series. That said, it does appear that dropping AppleCommonCrypto should become easier after your changes land if someone opts to tackle the task.
Quoted 3 lines · Glen Choo
At $DAYJOB, we observed that this topic breaks MacOS builds with sha1dc:
$ make NO_APPLE_COMMON_CRYPTO=1 DC_SHA1=1 NO_OPENSSL=1 compat/fsmonitor/fsm-ipc-darwin.o
Glen, Ævar has cherry-picked the SHA_CTX -> git_SHA_CTX build fix and I have merged it to 'next'. Can you test-build to see if that change is sufficient to fix "does not even build from the source" issue for you guys? You do not have to go through any official full qualification cycle or release procedure---just checking that you would be OK when you need to do the build the next time before it has to happen is sufficient.
Thanks.
Quoted 5 lines · Junio C Hamano
Possible action items:
* See what configurations our two macOS jobs are using. If neither is using sha1dc, I would say that is criminal [*] and at least one of them should be updated to do so right away.
So here is my "panda-see-panda-do" attempt.
----- >8 --------- >8 --------- >8 --------- >8 --------- >8 -----
Subject: [PATCH] ci: use DC_SHA1=1 on osx-clang job for CI
All other jobs were using the default DC_SHA1 (which is a recommended practice), but the default for macOS jobs being Apple's common crypt, we didn't catch recent breakage soon enough.
We may want to give similar diversity for Linux jobs so that some of them build with other implementations of SHA-1, but let's start small and fill only the immediate need.
--- ci/lib.sh | 2 ++ 1 file changed, 2 insertions(+)
ci/lib.sh +2 -0
@@ -259,6 +259,8 @@ macos-latest) MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=$(which python3)" else MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=$(which python2)" + MAKEFLAGS="$MAKEFLAGS NO_APPLE_COMMON_CRYPTO=NoThanks" + MAKEFLAGS="$MAKEFLAGS DC_SHA1=1 NO_OPENSSL=1" fi ;; esac
Signature · 1 line
2.38.1-267-g82836053dd
Quoted 8 lines · Junio C Hamano
So here is my "panda-see-panda-do" attempt.
----- >8 --------- >8 --------- >8 --------- >8 --------- >8 -----
Subject: [PATCH] ci: use DC_SHA1=1 on osx-clang job for CIAll other jobs were using the default
DC_SHA1(which is a recommended practice), but the default for macOS jobs being Apple's common crypt, we didn't catch recent breakage soon enough.
"recent breakage" is quite vague and probably won't help future readers understand what this is actually fixing. Possible
improvements: (1) a prose description of the breakage; (2) the actual compiler error message; (3) a pointer[1] to the email reporting the problem. One or more of the above improvements to the commit message would help future readers.
[1]: https://lore.kernel.org/git/[email protected]/
Quoted 19 lines
We may want to give similar diversity for Linux jobs so that some of them build with other implementations of SHA-1, but let's start small and fill only the immediate need.
Signed-off-by Junio C Hamano <[email protected]>---
ci/lib.sh +2
-0@@ -259,6 +259,8 @@ macos-latest) MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=$(which python3)" else MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=$(which python2)" + MAKEFLAGS="$MAKEFLAGS NO_APPLE_COMMON_CRYPTO=NoThanks" + MAKEFLAGS="$MAKEFLAGS DC_SHA1=1 NO_OPENSSL=1" fi ;; esac
Quoted 10 lines · Eric Sunshine
Quoted 8 lines · Junio C Hamano
So here is my "panda-see-panda-do" attempt.
----- >8 --------- >8 --------- >8 --------- >8 --------- >8 -----
Subject: [PATCH] ci: use DC_SHA1=1 on osx-clang job for CIAll other jobs were using the default
DC_SHA1(which is a recommended practice), but the default for macOS jobs being Apple's common crypt, we didn't catch recent breakage soon enough.
Quoted 6 lines
"recent breakage" is quite vague and probably won't help future readers understand what this is actually fixing. Possible
improvements: (1) a prose description of the breakage; (2) the actual compiler error message; (3) a pointer[1] to the email reporting the problem. One or more of the above improvements to the commit message would help future readers.
I do not think (2) or (3) would help all that much. A finger that points at the exact commit that broke the build (with the condition under which the build breaks) would probably be the most useful to those who read "git log" output.
Thanks.
----- >8 --------- >8 --------- >8 --------- >8 --------- >8 -----
Subject: [PATCH] ci: use DC_SHA1=YesPlease on osx-clang job for CI
7b8cfe34 (Merge branch 'ed/fsmonitor-on-networked-macos', 2022-10-17) broke the build on macOS with sha1dc by bypassing our hash abstraction (git_SHA_CTX etc.), but it wasn't caught before the problematic topic was merged down to the 'master' branch. Nobody was even compile testing with DC_SHA1 set, although it is the recommended choice in these days for folks when they use SHA-1.
This was because the default for macOS uses Apple Common Crypto, and both of the two CI jobs did not override the default. Tweak one of them to use DC_SHA1 to improve the coverage.
We may want to give similar diversity for Linux jobs so that some of them build with other implementations of SHA-1; they currently all build and test with DC_SHA1 as that is the default on everywhere other than macOS.
But let's start small to fill only the immediate need.
--- ci/lib.sh | 2 ++ 1 file changed, 2 insertions(+)
ci/lib.sh +2 -0
@@ -259,6 +259,8 @@ macos-latest) MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=$(which python3)" else MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=$(which python2)" + MAKEFLAGS="$MAKEFLAGS NO_APPLE_COMMON_CRYPTO=NoThanks" + MAKEFLAGS="$MAKEFLAGS DC_SHA1=YesPlease NO_OPENSSL=NoThanks" fi ;; esac
Signature · 1 line
2.38.1-271-ge78cd6c470
Quoted 28 lines · Junio C Hamano
Quoted 11 lines · Eric Sunshine
Quoted 3 lines · Junio C Hamano
All other jobs were using the default
DC_SHA1(which is a recommended practice), but the default for macOS jobs being Apple's common crypt, we didn't catch recent breakage soon enough."recent breakage" is quite vague and probably won't help future readers understand what this is actually fixing. Possible
improvements: (1) a prose description of the breakage; (2) the actual compiler error message; (3) a pointer[1] to the email reporting the problem. One or more of the above improvements to the commit message would help future readers.I do not think (2) or (3) would help all that much. A finger that points at the exact commit that broke the build (with the condition under which the build breaks) would probably be the most useful to those who read "git log" output.
----- >8 --------- >8 --------- >8 --------- >8 --------- >8 -----
Subject: [PATCH] ci: use DC_SHA1=YesPlease on osx-clang job for CI
7b8cfe34(Merge branch 'ed/fsmonitor-on-networked-macos', 2022-10-17) broke the build on macOS with sha1dc by bypassing our hash abstraction (git_SHA_CTXetc.), but it wasn't caught before the problematic topic was merged down to the 'master' branch. Nobody was even compile testing withDC_SHA1set, although it is the recommended choice in these days for folks when they use SHA-1.
Quoted 3 lines
This was because the default for macOS uses Apple Common Crypto, and both of the two CI jobs did not override the default. Tweak one of them to use
DC_SHA1to improve the coverage.
Thanks. This revised commit message does a much better job of explaining the problem the patch is addressing.
Quoted 14 lines · Junio C Hamano
Quoted 3 lines · Glen Choo
At $DAYJOB, we observed that this topic breaks MacOS builds with sha1dc:
$ make NO_APPLE_COMMON_CRYPTO=1 DC_SHA1=1 NO_OPENSSL=1 compat/fsmonitor/fsm-ipc-darwin.oGlen, Ævar has cherry-picked the
SHA_CTX->git_SHA_CTXbuild fix and I have merged it to 'next'. Can you test-build to see if that change is sufficient to fix "does not even build from the source" issue for you guys? You do not have to go through any official full qualification cycle or release procedure---just checking that you would be OK when you need to do the build the next time before it has to happen is sufficient.
Quoted 1 line
Thanks.
FYI, https://github.com/git/git/actions/runs/3292710187 is a successful (wow, what's a rare event these days) run of CI on 'seen' that includes
* Peff's -O0 fix for unconfusing LSan to prevent it from giving false positives
* Eric DeCosta's change, cherry-picked by Ævar, to let fsmonitor stuff compile with sha1dc as well as Apple's Common Crypto.
* A tweak to make one of the macOS CI jobs to build with sha1dc.
among all the other topics in flight. I plan to merge all of them to 'next' (I think the first two are already in 'next') and fast track them down to 'master'.
v3 cover fsmonitor: Implement fsmonitor for Linux
In reply to Eric DeCosta via GitGitGadgetGoal is to deliver fsmonitor for Linux that is on par with fsmonitor for Windows and Mac OS.
This patch set builds upon previous work for done for Windows and Mac OS to implement a fsmonitor back-end for Linux based on the Linux inotify API. inotify differs significantly from the equivalent Windows and Mac OS APIs in that a watch must be registered for every directory of interest (rather than a singular watch at the root of the directory tree) and special care must be taken to handle directory renames correctly.
More information about inotify:
https://man7.org/linux/man-pages/man7/inotify.7.html
v2 differs from v1:
* Prior work for Windows and Mac OS has been merged to master, reducing the patch set from 12 to 6 patches
* Code review feedback
* Identified and resolved race condition revealed by CI test system, see "Limitations and caveats" regarding monitoring of directory trees from the man page, above
* Apologies for being away from this for so long, but my attention was needed elsewhere
v1 differs from v0:
* Code review feedback
* Update how and which code can be shared between Mac OS and Linux
* Increase polling frequency to every 1ms (matches Mac OS)
* Updates to t7527 to improve test stability
Eric DeCosta (6):
fsmonitor: prepare to share code between Mac OS and Linux
fsmonitor: determine if filesystem is local or remote
fsmonitor: implement filesystem change listener for Linux
fsmonitor: enable fsmonitor for Linux
fsmonitor: test updates
fsmonitor: update doc for Linux
Documentation/config/fsmonitor--daemon.txt | 4 +- Documentation/git-fsmonitor--daemon.txt | 24 +- compat/fsmonitor/fsm-health-linux.c | 24 + compat/fsmonitor/fsm-ipc-darwin.c | 53 +- compat/fsmonitor/fsm-ipc-linux.c | 1 + compat/fsmonitor/fsm-ipc-unix.c | 52 ++ compat/fsmonitor/fsm-listen-linux.c | 676 +++++++++++++++++++++ compat/fsmonitor/fsm-path-utils-linux.c | 169 ++++++ compat/fsmonitor/fsm-settings-darwin.c | 63 +- compat/fsmonitor/fsm-settings-linux.c | 1 + compat/fsmonitor/fsm-settings-unix.c | 61 ++ config.mak.uname | 8 + contrib/buildsystems/CMakeLists.txt | 11 +- t/t7527-builtin-fsmonitor.sh | 94 ++- 14 files changed, 1102 insertions(+), 139 deletions(-) create mode 100644 compat/fsmonitor/fsm-health-linux.c create mode 100644 compat/fsmonitor/fsm-ipc-linux.c create mode 100644 compat/fsmonitor/fsm-ipc-unix.c create mode 100644 compat/fsmonitor/fsm-listen-linux.c create mode 100644 compat/fsmonitor/fsm-path-utils-linux.c create mode 100644 compat/fsmonitor/fsm-settings-linux.c create mode 100644 compat/fsmonitor/fsm-settings-unix.c
base-commit: 319605f8f00e402f3ea758a02c63534ff800a711
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1352%2Fedecosta-mw%2Ffsmonitor_linux-v3
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1352/edecosta-mw/fsmonitor_linux-v3
Pull-Request: https://github.com/git/git/pull/1352
Range-diff vs v2:
1: cd46bed37c3 < -: ----------- fsmonitor: refactor filesystem checks to common interface
2: 21d114bda4b < -: ----------- fsmonitor: relocate socket file if .git directory is remote
3: 664259ed57a < -: ----------- fsmonitor: avoid socket location check if using hook
4: d8f20032d6b < -: ----------- fsmonitor: deal with synthetic firmlinks on macOS
5: ab1e0ab967c < -: ----------- fsmonitor: check for compatability before communicating with fsmonitor
6: 9c552239b57 < -: ----------- fsmonitor: add documentation for allowRemote and socketDir options
7: 295beb89ab1 ! 1: 99d684c7bdf fsmonitor: prepare to share code between Mac OS and Linux
@@ Commit message
Linux and Mac OS can share some of the code originally developed for Mac OS.
- Minor update to compat/fsmonitor/fsm-ipc-unix.c to make it cross-platform.
- Mac OS and Linux can share fsm-ipc-unix.c
-
- Both platforms can also share compat/fsmonitor/fsm-settings-unix.c but we
- will leave room for future, platform-specific checks by having the platform-
- specific implementations call into fsm-settings-unix.
+ Mac OS and Linux can share fsm-ipc-unix.c and fsm-settings-unix.c
Signed-off-by: Eric DeCosta <[email protected]>
- ## Makefile ##
-@@ Makefile: endif
-
- ifdef FSMONITOR_DAEMON_BACKEND
- COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND
-+ ifdef FSMONITOR_DAEMON_COMMON
-+ COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_COMMON).o
-+ else
-+ COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_BACKEND).o
-+ endif
- COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o
- COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o
-- COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_BACKEND).o
- endif
-
- ifdef FSMONITOR_OS_SETTINGS
- COMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS
-+ ifdef FSMONITOR_DAEMON_COMMON
-+ COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_DAEMON_COMMON).o
-+ endif
- COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o
- COMPAT_OBJS += compat/fsmonitor/fsm-path-utils-$(FSMONITOR_OS_SETTINGS).o
- endif
-
## compat/fsmonitor/fsm-health-linux.c (new) ##
@@
+#include "cache.h"
@@ compat/fsmonitor/fsm-health-linux.c (new)
+{
+}
- ## compat/fsmonitor/fsm-ipc-darwin.c => compat/fsmonitor/fsm-ipc-unix.c ##
-@@ compat/fsmonitor/fsm-ipc-unix.c: static GIT_PATH_FUNC(fsmonitor_ipc__get_default_path, "fsmonitor--daemon.ipc")
- const char *fsmonitor_ipc__get_path(struct repository *r)
- {
- static const char *ipc_path = NULL;
-- SHA_CTX sha1ctx;
+ ## compat/fsmonitor/fsm-ipc-darwin.c ##
+@@
+-#include "cache.h"
+-#include "config.h"
+-#include "strbuf.h"
+-#include "fsmonitor.h"
+-#include "fsmonitor-ipc.h"
+-#include "fsmonitor-path-utils.h"
+-
+-static GIT_PATH_FUNC(fsmonitor_ipc__get_default_path, "fsmonitor--daemon.ipc")
+-
+-const char *fsmonitor_ipc__get_path(struct repository *r)
+-{
+- static const char *ipc_path = NULL;
+- git_SHA_CTX sha1ctx;
+- char *sock_dir = NULL;
+- struct strbuf ipc_file = STRBUF_INIT;
+- unsigned char hash[GIT_MAX_RAWSZ];
+-
+- if (!r)
+- BUG("No repository passed into fsmonitor_ipc__get_path");
+-
+- if (ipc_path)
+- return ipc_path;
+-
+-
+- /* By default the socket file is created in the .git directory */
+- if (fsmonitor__is_fs_remote(r->gitdir) < 1) {
+- ipc_path = fsmonitor_ipc__get_default_path();
+- return ipc_path;
+- }
+-
+- git_SHA1_Init(&sha1ctx);
+- git_SHA1_Update(&sha1ctx, r->worktree, strlen(r->worktree));
+- git_SHA1_Final(hash, &sha1ctx);
+-
+- repo_config_get_string(r, "fsmonitor.socketdir", &sock_dir);
+-
+- /* Create the socket file in either socketDir or $HOME */
+- if (sock_dir && *sock_dir) {
+- strbuf_addf(&ipc_file, "%s/.git-fsmonitor-%s",
+- sock_dir, hash_to_hex(hash));
+- } else {
+- strbuf_addf(&ipc_file, "~/.git-fsmonitor-%s", hash_to_hex(hash));
+- }
+- free(sock_dir);
+-
+- ipc_path = interpolate_path(ipc_file.buf, 1);
+- if (!ipc_path)
+- die(_("Invalid path: %s"), ipc_file.buf);
+-
+- strbuf_release(&ipc_file);
+- return ipc_path;
+-}
++#include "fsm-ipc-unix.c"
+
+ ## compat/fsmonitor/fsm-ipc-linux.c (new) ##
+@@
++#include "fsm-ipc-unix.c"
+
+ ## compat/fsmonitor/fsm-ipc-unix.c (new) ##
+@@
++#include "cache.h"
++#include "config.h"
++#include "strbuf.h"
++#include "fsmonitor.h"
++#include "fsmonitor-ipc.h"
++#include "fsmonitor-path-utils.h"
++
++static GIT_PATH_FUNC(fsmonitor_ipc__get_default_path, "fsmonitor--daemon.ipc")
++
++const char *fsmonitor_ipc__get_path(struct repository *r)
++{
++ static const char *ipc_path = NULL;
+ git_SHA_CTX sha1ctx;
- char *sock_dir = NULL;
- struct strbuf ipc_file = STRBUF_INIT;
- unsigned char hash[SHA_DIGEST_LENGTH];
-@@ compat/fsmonitor/fsm-ipc-unix.c: const char *fsmonitor_ipc__get_path(struct repository *r)
- return ipc_path;
- }
-
-- SHA1_Init(&sha1ctx);
-- SHA1_Update(&sha1ctx, r->worktree, strlen(r->worktree));
-- SHA1_Final(hash, &sha1ctx);
++ char *sock_dir = NULL;
++ struct strbuf ipc_file = STRBUF_INIT;
++ unsigned char hash[GIT_MAX_RAWSZ];
++
++ if (!r)
++ BUG("No repository passed into fsmonitor_ipc__get_path");
++
++ if (ipc_path)
++ return ipc_path;
++
++
++ /* By default the socket file is created in the .git directory */
++ if (fsmonitor__is_fs_remote(r->gitdir) < 1) {
++ ipc_path = fsmonitor_ipc__get_default_path();
++ return ipc_path;
++ }
++
+ git_SHA1_Init(&sha1ctx);
+ git_SHA1_Update(&sha1ctx, r->worktree, strlen(r->worktree));
+ git_SHA1_Final(hash, &sha1ctx);
-
- repo_config_get_string(r, "fsmonitor.socketdir", &sock_dir);
-
++
++ repo_config_get_string(r, "fsmonitor.socketdir", &sock_dir);
++
++ /* Create the socket file in either socketDir or $HOME */
++ if (sock_dir && *sock_dir) {
++ strbuf_addf(&ipc_file, "%s/.git-fsmonitor-%s",
++ sock_dir, hash_to_hex(hash));
++ } else {
++ strbuf_addf(&ipc_file, "~/.git-fsmonitor-%s", hash_to_hex(hash));
++ }
++ free(sock_dir);
++
++ ipc_path = interpolate_path(ipc_file.buf, 1);
++ if (!ipc_path)
++ die(_("Invalid path: %s"), ipc_file.buf);
++
++ strbuf_release(&ipc_file);
++ return ipc_path;
++}
## compat/fsmonitor/fsm-settings-darwin.c ##
@@
- #include "config.h"
- #include "fsmonitor.h"
- #include "fsmonitor-ipc.h"
+-#include "config.h"
+-#include "fsmonitor.h"
+-#include "fsmonitor-ipc.h"
-#include "fsmonitor-settings.h"
- #include "fsmonitor-path-utils.h"
+-#include "fsmonitor-path-utils.h"
-
- /*
- * For the builtin FSMonitor, we create the Unix domain socket for the
@@ compat/fsmonitor/fsm-settings-darwin.c
- free(fs.typename);
- return FSMONITOR_REASON_OK;
-}
-+#include "fsmonitor-settings.h"
-+#include "fsm-settings-unix.h"
-
- enum fsmonitor_reason fsm_os__incompatible(struct repository *r, int ipc)
- {
+-
+-enum fsmonitor_reason fsm_os__incompatible(struct repository *r, int ipc)
+-{
- enum fsmonitor_reason reason;
-
- if (ipc) {
@@ compat/fsmonitor/fsm-settings-darwin.c
- }
-
- return FSMONITOR_REASON_OK;
-+ return fsm_os__incompatible_unix(r, ipc);
- }
+-}
++#include "fsm-settings-unix.c"
## compat/fsmonitor/fsm-settings-linux.c (new) ##
@@
-+#include "config.h"
-+#include "fsmonitor.h"
-+#include "fsmonitor-ipc.h"
-+#include "fsmonitor-path-utils.h"
-+#include "fsmonitor-settings.h"
-+#include "fsm-settings-unix.h"
-+
-+enum fsmonitor_reason fsm_os__incompatible(struct repository *r, int ipc)
-+{
-+ return fsm_os__incompatible_unix(r, ipc);
-+}
++#include "fsm-settings-unix.c"
## compat/fsmonitor/fsm-settings-unix.c (new) ##
@@
@@ compat/fsmonitor/fsm-settings-unix.c (new)
+#include "fsmonitor.h"
+#include "fsmonitor-ipc.h"
+#include "fsmonitor-path-utils.h"
-+#include "fsm-settings-unix.h"
+
+ /*
+ * For the builtin FSMonitor, we create the Unix domain socket for the
@@ compat/fsmonitor/fsm-settings-unix.c (new)
+ struct fs_info fs;
+ const char *ipc_path = fsmonitor_ipc__get_path(r);
+ struct strbuf path = STRBUF_INIT;
-+ strbuf_add(&path, ipc_path, strlen(ipc_path));
++ strbuf_addstr(&path, ipc_path);
+
+ if (fsmonitor__get_fs_info(dirname(path.buf), &fs) == -1) {
+ strbuf_release(&path);
@@ compat/fsmonitor/fsm-settings-unix.c (new)
+ return FSMONITOR_REASON_OK;
+}
+
-+enum fsmonitor_reason fsm_os__incompatible_unix(struct repository *r, int ipc)
++enum fsmonitor_reason fsm_os__incompatible(struct repository *r, int ipc)
+{
+ enum fsmonitor_reason reason;
+
@@ compat/fsmonitor/fsm-settings-unix.c (new)
+
+ return FSMONITOR_REASON_OK;
+}
-
- ## compat/fsmonitor/fsm-settings-unix.h (new) ##
-@@
-+#ifndef FSM_SETTINGS_UNIX_H
-+#define FSM_SETTINGS_UNIX_H
-+
-+#ifdef HAVE_FSMONITOR_OS_SETTINGS
-+/*
-+ * Check for compatibility on unix-like systems (e.g. Darwin and Linux)
-+ */
-+enum fsmonitor_reason fsm_os__incompatible_unix(struct repository *r, int ipc);
-+#endif /* HAVE_FSMONITOR_OS_SETTINGS */
-+
-+#endif /* FSM_SETTINGS_UNIX_H */
-
- ## config.mak.uname ##
-@@ config.mak.uname: ifeq ($(uname_S),Darwin)
- ifndef NO_UNIX_SOCKETS
- FSMONITOR_DAEMON_BACKEND = darwin
- FSMONITOR_OS_SETTINGS = darwin
-+ FSMONITOR_DAEMON_COMMON = unix
- endif
- endif
-
-
- ## contrib/buildsystems/CMakeLists.txt ##
-@@ contrib/buildsystems/CMakeLists.txt: else()
- endif()
-
- if(SUPPORTS_SIMPLE_IPC)
-- if(CMAKE_SYSTEM_NAME STREQUAL "Windows")
-+ if(CMAKE_SYSTEM_NAME STREQUAL "Linux")
-+ add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
-+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-unix.c)
-+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-linux.c)
-+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-linux.c)
-+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-linux.c)
-+
-+ add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
-+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-unix.c)
-+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-linux.c)
-+ elseif(CMAKE_SYSTEM_NAME STREQUAL "Windows")
- add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
- list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)
- list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c)
-@@ contrib/buildsystems/CMakeLists.txt: if(SUPPORTS_SIMPLE_IPC)
- list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c)
- elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin")
- add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
-- list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)
-+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-unix.c)
- list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c)
-- list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-darwin.c)
-+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)
- list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-darwin.c)
-
- add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
-+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-unix.c)
- list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)
- endif()
- endif()
8: 7d7ef78728f = 2: aa405379891 fsmonitor: determine if filesystem is local or remote
9: 4f9c5358475 ! 3: c2e5a7201aa fsmonitor: implement filesystem change listener for Linux
@@ compat/fsmonitor/fsm-listen-linux.c (new)
+/*
+ * Recursively add watches to every directory under path
+ */
-+static int register_inotify(const char *path, struct fsm_listen_data *data)
++static int register_inotify(const char *path,
++ struct fsmonitor_daemon_state *state,
++ struct fsmonitor_batch *batch)
+{
+ DIR *dir;
++ const char *rel;
+ struct strbuf current = STRBUF_INIT;
+ struct dirent *de;
+ struct stat fs;
@@ compat/fsmonitor/fsm-listen-linux.c (new)
+
+ /* recurse into directory */
+ if (S_ISDIR(fs.st_mode)) {
-+ if (add_watch(current.buf, data))
++ if (add_watch(current.buf, state->listen_data))
+ goto failed;
-+ if (register_inotify(current.buf, data))
++ if (register_inotify(current.buf, state, batch))
+ goto failed;
++ } else if (batch) {
++ rel = current.buf + state->path_worktree_watch.len + 1;
++ trace_printf_key(&trace_fsmonitor, "explicitly adding '%s'", rel);
++ fsmonitor_batch__add_path(batch, rel);
+ }
+ }
+ ret = 0;
@@ compat/fsmonitor/fsm-listen-linux.c (new)
+
+ if (add_watch(state->path_worktree_watch.buf, data))
+ ret = -1;
-+ else if (register_inotify(state->path_worktree_watch.buf, data))
++ else if (register_inotify(state->path_worktree_watch.buf, state, NULL))
+ ret = -1;
+ else if (state->nr_paths_watching > 1) {
+ if (add_watch(state->path_gitdir_watch.buf, data))
+ ret = -1;
-+ else if (register_inotify(state->path_gitdir_watch.buf, data))
++ else if (register_inotify(state->path_gitdir_watch.buf, state, NULL))
+ ret = -1;
+ }
+
@@ compat/fsmonitor/fsm-listen-linux.c (new)
+ add_dir_rename(event->cookie, path, state->listen_data);
+
+ /* received IN_MOVE_TO, update watch to reflect new path */
-+ if (em_rename_dir_to(event->mask))
++ if (em_rename_dir_to(event->mask)) {
+ rename_dir(event->cookie, path, state->listen_data);
++ if (register_inotify(path, state, batch)) {
++ state->listen_data->shutdown = SHUTDOWN_ERROR;
++ goto done;
++ }
++ }
+
+ if (em_dir_created(event->mask)) {
+ if (add_watch(path, state->listen_data)) {
+ state->listen_data->shutdown = SHUTDOWN_ERROR;
+ goto done;
+ }
-+ if (register_inotify(path, state->listen_data)) {
++ if (register_inotify(path, state, batch)) {
+ state->listen_data->shutdown = SHUTDOWN_ERROR;
+ goto done;
+ }
@@ compat/fsmonitor/fsm-listen-linux.c (new)
+ strbuf_reset(&path);
+ strbuf_add(&path, w->dir, strlen(w->dir));
+ strbuf_addch(&path, '/');
-+ strbuf_add(&path, event->name, strlen(event->name));
++ strbuf_addstr(&path, event->name);
+
+ p = fsmonitor__resolve_alias(path.buf, &state->alias);
+ if (!p)
10: 07650ecd27b ! 4: 05f5b2dd4fb fsmonitor: enable fsmonitor for Linux
@@ config.mak.uname: ifeq ($(uname_S),Linux)
+ ifndef NO_UNIX_SOCKETS
+ FSMONITOR_DAEMON_BACKEND = linux
+ FSMONITOR_OS_SETTINGS = linux
-+ FSMONITOR_DAEMON_COMMON = unix
+ endif
+ endif
endif
ifeq ($(uname_S),GNU/kFreeBSD)
HAVE_ALLOCA_H = YesPlease
+
+ ## contrib/buildsystems/CMakeLists.txt ##
+@@ contrib/buildsystems/CMakeLists.txt: else()
+ endif()
+
+ if(SUPPORTS_SIMPLE_IPC)
+- if(CMAKE_SYSTEM_NAME STREQUAL "Windows")
++ if(CMAKE_SYSTEM_NAME STREQUAL "Linux")
++ add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
++ list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-linux.c)
++ list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-linux.c)
++ list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-linux.c)
++ list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-linux.c)
++
++ add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
++ list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-linux.c)
++ elseif(CMAKE_SYSTEM_NAME STREQUAL "Windows")
+ add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c)
11: 6682938fff8 ! 5: 754355ca44f fsmonitor: test updates
@@ t/t7527-builtin-fsmonitor.sh: start_daemon () {
+ then
+ if test "$last" -eq "$nsz"
+ then
++ cat "$file" &&
+ return 0
+ fi
+ last=$nsz
@@ t/t7527-builtin-fsmonitor.sh: start_daemon () {
+ sleep 1
+ k=$(( $k + 1 ))
+ done &&
++ cat "$file" &&
+ return 0
+}
+
@@ t/t7527-builtin-fsmonitor.sh: create_files () {
+rename_directory () {
+ mv dirtorename dirrenamed
+}
++
++rename_directory_file () {
++ mv dirtorename dirrenamed &&
++ echo 1 > dirrenamed/new
++}
+
rename_files () {
mv rename renamed &&
@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'rename directory' '
grep "^event: dirtorename/*$" .git/trace &&
grep "^event: dirrenamed/*$" .git/trace
'
-@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'file changes to directory' '
+
++test_expect_success 'rename directory file' '
++ test_when_finished clean_up_repo_and_stop_daemon &&
++
++ start_daemon --tf "$PWD/.git/trace" &&
++
++ wait_for_update rename_directory_file "$PWD/.git/trace" &&
++
++ test-tool fsmonitor-client query --token 0 &&
++
++ test_might_fail git fsmonitor--daemon stop &&
++
++ grep "^event: dirtorename/*$" .git/trace &&
++ grep "^event: dirrenamed/*$" .git/trace &&
++ grep "^event: dirrenamed/new$" .git/trace
++'
+ test_expect_success 'file changes to directory' '
+ test_when_finished clean_up_repo_and_stop_daemon &&
start_daemon --tf "$PWD/.git/trace" &&
@@ t/t7527-builtin-fsmonitor.sh: u_values="$u1 $u2"
git init "$u" &&
echo 1 >"$u"/file1 &&
-@@ t/t7527-builtin-fsmonitor.sh: my_match_and_clean () {
- }
+@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'submodule setup' '
+ '
test_expect_success 'submodule always visited' '
- test_when_finished "git -C super fsmonitor--daemon stop; \ 12: dd73e126810 = 6: f56175e097a fsmonitor: update doc for Linux
Signature · 1 line
gitgitgadget
PATCH v3 1/6 fsmonitor: prepare to share code between Mac OS and Linux
From: Eric DeCosta <[email protected]>
Linux and Mac OS can share some of the code originally developed for Mac OS.
Mac OS and Linux can share fsm-ipc-unix.c and fsm-settings-unix.c
--- compat/fsmonitor/fsm-health-linux.c | 24 ++++++++++ compat/fsmonitor/fsm-ipc-darwin.c | 53 +--------------------- compat/fsmonitor/fsm-ipc-linux.c | 1 + compat/fsmonitor/fsm-ipc-unix.c | 52 +++++++++++++++++++++ compat/fsmonitor/fsm-settings-darwin.c | 63 +------------------------- compat/fsmonitor/fsm-settings-linux.c | 1 + compat/fsmonitor/fsm-settings-unix.c | 61 +++++++++++++++++++++++++ 7 files changed, 141 insertions(+), 114 deletions(-) create mode 100644 compat/fsmonitor/fsm-health-linux.c create mode 100644 compat/fsmonitor/fsm-ipc-linux.c create mode 100644 compat/fsmonitor/fsm-ipc-unix.c create mode 100644 compat/fsmonitor/fsm-settings-linux.c create mode 100644 compat/fsmonitor/fsm-settings-unix.c
compat/fsmonitor/fsm-health-linux.c new file +24 -0
@@ -0,0 +1,24 @@ +#include "cache.h" +#include "config.h" +#include "fsmonitor.h" +#include "fsm-health.h" +#include "fsmonitor--daemon.h" + +int fsm_health__ctor (struct fsmonitor_daemon_state *state) +{ +return 0 ; +} + +void fsm_health__dtor (struct fsmonitor_daemon_state *state) +{ +return ; +} + +void fsm_health__loop (struct fsmonitor_daemon_state *state) +{ +return ; +} + +void fsm_health__stop_async (struct fsmonitor_daemon_state *state) +{ +}
compat/fsmonitor/fsm-ipc-darwin.c +1 -52
53 lines not on this page · raw message
compat/fsmonitor/fsm-ipc-linux.c new file +1 -0
@@ -0,0 +1 @@ +#include "fsm-ipc-unix.c"
compat/fsmonitor/fsm-ipc-unix.c new file +52 -0
52 lines not on this page · raw message
compat/fsmonitor/fsm-settings-darwin.c +1 -62
63 lines not on this page · raw message
compat/fsmonitor/fsm-settings-linux.c new file +1 -0
@@ -0,0 +1 @@ +#include "fsm-settings-unix.c"
compat/fsmonitor/fsm-settings-unix.c new file +61 -0
61 lines not on this page · raw message
Signature · 1 line
gitgitgadget
PATCH v3 2/6 fsmonitor: determine if filesystem is local or remote
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
Compare the given path to the mounted filesystems. Find the mount that is the longest prefix of the path (if any) and determine if that mount is on a local or remote filesystem.
--- compat/fsmonitor/fsm-path-utils-linux.c | 169 ++++++++++++++++++++++++ 1 file changed, 169 insertions(+) create mode 100644 compat/fsmonitor/fsm-path-utils-linux.c
compat/fsmonitor/fsm-path-utils-linux.c new file +169 -0
@@ -0,0 +1,169 @@ +#include "fsmonitor.h" +#include "fsmonitor-path-utils.h" +#include <errno.h> +#include <mntent.h> +#include <sys/mount.h> +#include <sys/statvfs.h> + +/* +* https://github.com/coreutils/gnulib/blob/master/lib/mountlist.c +*/ +#ifndef ME_REMOTE +/* A file system is "remote" if its Fs_name contains a ':' +or if (it is of type (smbfs or cifs) and its Fs_name starts with '//') +or if it is of any other of the listed types +or Fs_name is equal to "-hosts" (used by autofs to mount remote fs). +"VM" file systems like prl_fs or vboxsf are not considered remote here. */ +# define ME_REMOTE (Fs_name, Fs_type) \ + (strchr (Fs_name, ':') != NULL \ + || ((Fs_name)[0] == '/' \ + && (Fs_name)[1] == '/' \ + && (strcmp (Fs_type, "smbfs") == 0 \ + || strcmp (Fs_type, "smb3") == 0 \ + || strcmp (Fs_type, "cifs") == 0)) \ + || strcmp (Fs_type, "acfs") == 0 \ + || strcmp (Fs_type, "afs") == 0 \ + || strcmp (Fs_type, "coda") == 0 \ + || strcmp (Fs_type, "auristorfs") == 0 \ + || strcmp (Fs_type, "fhgfs") == 0 \ + || strcmp (Fs_type, "gpfs") == 0 \ + || strcmp (Fs_type, "ibrix") == 0 \ + || strcmp (Fs_type, "ocfs2") == 0 \ + || strcmp (Fs_type, "vxfs") == 0 \ + || strcmp ("-hosts", Fs_name) == 0) +#endif + +static int find_mount (const char *path,const struct statvfs *fs, +struct mntent *ent) +{ +const char *const mounts ="/proc/mounts" ; +const char *rp =real_pathdup (path,1 ); +struct mntent *ment =NULL ; +struct statvfs mntfs; +FILE *fp; +int found =0 ; +int dlen, plen, flen =0 ; + + ent->mnt_fsname =NULL ; + ent->mnt_dir =NULL ; + ent->mnt_type =NULL ; + + fp =setmntent (mounts,"r" ); +if (!fp) { +error_errno (_ ("setmntent('%s') failed" ), mounts); +return -1 ; + } + + plen =strlen (rp); + +/* read all the mount information and compare to path */ +while ((ment =getmntent (fp)) !=NULL ) { +if (statvfs (ment->mnt_dir, &mntfs)) { +switch (errno) { +case EPERM : +case ESRCH : +case EACCES : +continue ; +default : +error_errno (_ ("statvfs('%s') failed" ), ment->mnt_dir); +endmntent (fp); +return -1 ; + } + } + +/* is mount on the same filesystem and is a prefix of the path */ +if ((fs->f_fsid == mntfs.f_fsid) && + !strncmp (ment->mnt_dir, rp,strlen (ment->mnt_dir))) { + dlen =strlen (ment->mnt_dir); +if (dlen > plen) +continue ; +/* +* root is always a potential match; otherwise look for +* directory prefix +*/ +if ((dlen ==1 && ment->mnt_dir[0 ] =='/' ) || + (dlen > flen && (!rp[dlen] || rp[dlen] =='/' ))) { + flen = dlen; +/* +* https://man7.org/linux/man-pages/man3/getmntent.3.html +* +* The pointer points to a static area of memory which is +* overwritten by subsequent calls to getmntent(). +*/ + found =1 ; +free (ent->mnt_fsname); +free (ent->mnt_dir); +free (ent->mnt_type); + ent->mnt_fsname =xstrdup (ment->mnt_fsname); + ent->mnt_dir =xstrdup (ment->mnt_dir); + ent->mnt_type =xstrdup (ment->mnt_type); + } + } + } +endmntent (fp); + +if (!found) +return -1 ; + +return 0 ; +} + +int fsmonitor__get_fs_info (const char *path,struct fs_info *fs_info) +{ +struct mntent ment; +struct statvfs fs; + +if (statvfs (path, &fs)) +return error_errno (_ ("statvfs('%s') failed" ), path); + + +if (find_mount (path, &fs, &ment) <0 ) { +free (ment.mnt_fsname); +free (ment.mnt_dir); +free (ment.mnt_type); +return -1 ; + } + +trace_printf_key (&trace_fsmonitor, +"statvfs('%s') [flags 0x%08lx] '%s' '%s'" , + path, fs.f_flag, ment.mnt_type, ment.mnt_fsname); + + fs_info->is_remote =ME_REMOTE (ment.mnt_fsname, ment.mnt_type); + fs_info->typename = ment.mnt_fsname; +free (ment.mnt_dir); +free (ment.mnt_type); + +trace_printf_key (&trace_fsmonitor, +"'%s' is_remote: %d" , + path, fs_info->is_remote); +return 0 ; +} + +int fsmonitor__is_fs_remote (const char *path) +{ +struct fs_info fs; + +if (fsmonitor__get_fs_info (path, &fs)) +return -1 ; + +free (fs.typename); + +return fs.is_remote; +} + +/* +* No-op for now. +*/ +int fsmonitor__get_alias (const char *path,struct alias_info *info) +{ +return 0 ; +} + +/* +* No-op for now. +*/ +char *fsmonitor__resolve_alias (const char *path, +const struct alias_info *info) +{ +return NULL ; +}
Signature · 1 line
gitgitgadget
PATCH v3 3/6 fsmonitor: implement filesystem change listener for Linux
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
Implement a filesystem change listener for Linux based on the inotify API:
https://man7.org/linux/man-pages/man7/inotify.7.html
inotify requires registering a watch on every directory in the worktree and special handling of moves/renames.
--- compat/fsmonitor/fsm-listen-linux.c | 676 ++++++++++++++++++++++++++++ 1 file changed, 676 insertions(+) create mode 100644 compat/fsmonitor/fsm-listen-linux.c
compat/fsmonitor/fsm-listen-linux.c new file +676 -0
@@ -0,0 +1,676 @@ +#include "cache.h" +#include "fsmonitor.h" +#include "fsm-listen.h" +#include "fsmonitor--daemon.h" +#include <dirent.h> +#include <fcntl.h> +#include <sys/inotify.h> +#include <sys/stat.h> + +/* +* Safe value to bitwise OR with rest of mask for +* kernels that do not support IN_MASK_CREATE +*/ +#ifndef IN_MASK_CREATE +#define IN_MASK_CREATE 0x00000000 +#endif + +enum shutdown_reason { +SHUTDOWN_CONTINUE =0 , +SHUTDOWN_STOP , +SHUTDOWN_ERROR , +SHUTDOWN_FORCE +}; + +struct watch_entry { +struct hashmap_entry ent; +int wd; +uint32_t cookie; +const char *dir; +}; + +struct rename_entry { +struct hashmap_entry ent; +time_t whence; +uint32_t cookie; +const char *dir; +}; + +struct fsm_listen_data { +int fd_inotify; +enum shutdown_reason shutdown; +struct hashmap watches; +struct hashmap renames; +struct hashmap revwatches; +}; + +static int watch_entry_cmp (const void *cmp_data, +const struct hashmap_entry *eptr, +const struct hashmap_entry *entry_or_key, +const void *keydata) +{ +const struct watch_entry *e1, *e2; + + e1 = container_of(eptr, const struct watch_entry, ent); + e2 = container_of(eptr, const struct watch_entry, ent); +return e1->wd != e2->wd; +} + +static int revwatches_entry_cmp (const void *cmp_data, +const struct hashmap_entry *eptr, +const struct hashmap_entry *entry_or_key, +const void *keydata) +{ +const struct watch_entry *e1, *e2; + + e1 = container_of(eptr, const struct watch_entry, ent); + e2 = container_of(eptr, const struct watch_entry, ent); +return strcmp (e1->dir, e2->dir); +} + +static int rename_entry_cmp (const void *cmp_data, +const struct hashmap_entry *eptr, +const struct hashmap_entry *entry_or_key, +const void *keydata) +{ +const struct rename_entry *e1, *e2; + + e1 = container_of(eptr, const struct rename_entry, ent); + e2 = container_of(eptr, const struct rename_entry, ent); +return e1->cookie != e2->cookie; +} + +/* +* Register an inotify watch, add watch descriptor to path mapping +* and the reverse mapping. +*/ +static int add_watch (const char *path,struct fsm_listen_data *data) +{ +const char *interned =strintern (path); +struct watch_entry *w1, *w2; + +/* add the inotify watch, don't allow watches to be modified */ +int wd =inotify_add_watch (data->fd_inotify, interned, + (IN_ALL_EVENTS |IN_ONLYDIR |IN_MASK_CREATE ) + ^IN_ACCESS ^IN_CLOSE ^IN_OPEN ); +if (wd <0 ) +return error_errno ("inotify_add_watch('%s') failed" , interned); + +/* add watch descriptor -> directory mapping */ +CALLOC_ARRAY (w1,1 ); + w1->wd = wd; + w1->dir = interned; +hashmap_entry_init (&w1->ent,memhash (&w1->wd,sizeof (int ))); +hashmap_add (&data->watches, &w1->ent); + +/* add directory -> watch descriptor mapping */ +CALLOC_ARRAY (w2,1 ); + w2->wd = wd; + w2->dir = interned; +hashmap_entry_init (&w2->ent,memhash (w2->dir,strlen (w2->dir))); +hashmap_add (&data->revwatches, &w2->ent); + +return 0 ; +} + +/* +* Remove the inotify watch, the watch descriptor to path mapping +* and the reverse mapping. +*/ +static void remove_watch (struct watch_entry *w, +struct fsm_listen_data *data) +{ +struct watch_entry k1, k2, *w1, *w2; + +/* remove watch, ignore error if kernel already did it */ +if (inotify_rm_watch (data->fd_inotify, w->wd) && errno !=EINVAL ) +error_errno ("inotify_rm_watch() failed" ); + +hashmap_entry_init (&k1.ent,memhash (&w->wd,sizeof (int ))); + w1 =hashmap_remove_entry (&data->watches, &k1, ent,NULL ); +if (!w1) +BUG ("Double remove of watch for '%s'" , w->dir); + +if (w1->cookie) +BUG ("Removing watch for '%s' which has a pending rename" , w1->dir); + +hashmap_entry_init (&k2.ent,memhash (w->dir,strlen (w->dir))); + w2 =hashmap_remove_entry (&data->revwatches, &k2, ent,NULL ); +if (!w2) +BUG ("Double remove of reverse watch for '%s'" , w->dir); + +/* w1->dir and w2->dir are interned strings, we don't own them */ +free (w1); +free (w2); +} + +/* +* Check for stale directory renames. +* +* https://man7.org/linux/man-pages/man7/inotify.7.html +* +* Allow for some small timeout to account for the fact that insertion of the +* IN_MOVED_FROM+IN_MOVED_TO event pair is not atomic, and the possibility that +* there may not be any IN_MOVED_TO event. +* +* If the IN_MOVED_TO event is not received within the timeout then events have +* been missed and the monitor is in an inconsistent state with respect to the +* filesystem. +*/ +static int check_stale_dir_renames (struct hashmap *renames,time_t max_age) +{ +struct rename_entry *re; +struct hashmap_iter iter; + +hashmap_for_each_entry (renames, &iter, re, ent) { +if (re->whence <= max_age) +return -1 ; + } +return 0 ; +} + +/* +* Track pending renames. +* +* Tracking is done via a event cookie to watch descriptor mapping. +* +* A rename is not complete until matching a IN_MOVED_TO event is received +* for a corresponding IN_MOVED_FROM event. +*/ +static void add_dir_rename (uint32_t cookie,const char *path, +struct fsm_listen_data *data) +{ +struct watch_entry k, *w; +struct rename_entry *re; + +/* lookup the watch descriptor for the given path */ +hashmap_entry_init (&k.ent,memhash (path,strlen (path))); + w =hashmap_get_entry (&data->revwatches, &k, ent,NULL ); +if (!w)/* should never happen */ +BUG ("No watch for '%s'" , path); + w->cookie = cookie; + +/* add the pending rename to match against later */ +CALLOC_ARRAY (re,1 ); + re->dir = w->dir; + re->cookie = w->cookie; + re->whence =time (NULL ); +hashmap_entry_init (&re->ent,memhash (&re->cookie,sizeof (uint32_t ))); +hashmap_add (&data->renames, &re->ent); +} + +/* +* Handle directory renames +* +* Once a IN_MOVED_TO event is received, lookup the rename tracking information +* via the event cookie and use this information to update the watch. +*/ +static void rename_dir (uint32_t cookie,const char *path, +struct fsm_listen_data *data) +{ +struct rename_entry rek, *re; +struct watch_entry k, *w; + +/* lookup a pending rename to match */ + rek.cookie = cookie; +hashmap_entry_init (&rek.ent,memhash (&rek.cookie,sizeof (uint32_t ))); + re =hashmap_get_entry (&data->renames, &rek, ent,NULL ); +if (re) { + k.dir = re->dir; +hashmap_entry_init (&k.ent,memhash (k.dir,strlen (k.dir))); + w =hashmap_get_entry (&data->revwatches, &k, ent,NULL ); +if (w) { + w->cookie =0 ;/* rename handled */ +remove_watch (w, data); +add_watch (path, data); + }else { +BUG ("No matching watch" ); + } + }else { +BUG ("No matching cookie" ); + } +} + +/* +* Recursively add watches to every directory under path +*/ +static int register_inotify (const char *path, +struct fsmonitor_daemon_state *state, +struct fsmonitor_batch *batch) +{ +DIR *dir; +const char *rel; +struct strbuf current =STRBUF_INIT ; +struct dirent *de; +struct stat fs; +int ret =-1 ; + + dir =opendir (path); +if (!dir) +return error_errno ("opendir('%s') failed" , path); + +while ((de =readdir_skip_dot_and_dotdot (dir)) !=NULL ) { +strbuf_reset (¤t); +strbuf_addf (¤t,"%s/%s" , path, de->d_name); +if (lstat (current.buf, &fs)) { +error_errno ("lstat('%s') failed" , current.buf); + goto failed; + } + +/* recurse into directory */ +if (S_ISDIR (fs.st_mode)) { +if (add_watch (current.buf, state->listen_data)) + goto failed; +if (register_inotify (current.buf, state, batch)) + goto failed; + }else if (batch) { + rel = current.buf + state->path_worktree_watch.len +1 ; +trace_printf_key (&trace_fsmonitor,"explicitly adding '%s'" , rel); +fsmonitor_batch__add_path (batch, rel); + } + } + ret =0 ; + +failed: +strbuf_release (¤t); +if (closedir (dir) <0 ) +return error_errno ("closedir('%s') failed" , path); +return ret; +} + +static int em_rename_dir_from (u_int32_t mask) +{ +return ((mask &IN_ISDIR ) && (mask &IN_MOVED_FROM )); +} + +static int em_rename_dir_to (u_int32_t mask) +{ +return ((mask &IN_ISDIR ) && (mask &IN_MOVED_TO )); +} + +static int em_remove_watch (u_int32_t mask) +{ +return (mask &IN_DELETE_SELF ); +} + +static int em_dir_renamed (u_int32_t mask) +{ +return ((mask &IN_ISDIR ) && (mask &IN_MOVE )); +} + +static int em_dir_created (u_int32_t mask) +{ +return ((mask &IN_ISDIR ) && (mask &IN_CREATE )); +} + +static int em_dir_deleted (uint32_t mask) +{ +return ((mask &IN_ISDIR ) && (mask &IN_DELETE )); +} + +static int em_force_shutdown (u_int32_t mask) +{ +return (mask &IN_UNMOUNT ) || (mask &IN_Q_OVERFLOW ); +} + +static int em_ignore (u_int32_t mask) +{ +return (mask &IN_IGNORED ) || (mask &IN_MOVE_SELF ); +} + +static void log_mask_set (const char *path,u_int32_t mask) +{ +struct strbuf msg =STRBUF_INIT ; + +if (mask &IN_ACCESS ) +strbuf_addstr (&msg,"IN_ACCESS|" ); +if (mask &IN_MODIFY ) +strbuf_addstr (&msg,"IN_MODIFY|" ); +if (mask &IN_ATTRIB ) +strbuf_addstr (&msg,"IN_ATTRIB|" ); +if (mask &IN_CLOSE_WRITE ) +strbuf_addstr (&msg,"IN_CLOSE_WRITE|" ); +if (mask &IN_CLOSE_NOWRITE ) +strbuf_addstr (&msg,"IN_CLOSE_NOWRITE|" ); +if (mask &IN_OPEN ) +strbuf_addstr (&msg,"IN_OPEN|" ); +if (mask &IN_MOVED_FROM ) +strbuf_addstr (&msg,"IN_MOVED_FROM|" ); +if (mask &IN_MOVED_TO ) +strbuf_addstr (&msg,"IN_MOVED_TO|" ); +if (mask &IN_CREATE ) +strbuf_addstr (&msg,"IN_CREATE|" ); +if (mask &IN_DELETE ) +strbuf_addstr (&msg,"IN_DELETE|" ); +if (mask &IN_DELETE_SELF ) +strbuf_addstr (&msg,"IN_DELETE_SELF|" ); +if (mask &IN_MOVE_SELF ) +strbuf_addstr (&msg,"IN_MOVE_SELF|" ); +if (mask &IN_UNMOUNT ) +strbuf_addstr (&msg,"IN_UNMOUNT|" ); +if (mask &IN_Q_OVERFLOW ) +strbuf_addstr (&msg,"IN_Q_OVERFLOW|" ); +if (mask &IN_IGNORED ) +strbuf_addstr (&msg,"IN_IGNORED|" ); +if (mask &IN_ISDIR ) +strbuf_addstr (&msg,"IN_ISDIR|" ); + +trace_printf_key (&trace_fsmonitor,"inotify_event: '%s', mask=%#8.8x %s" , + path, mask, msg.buf); + +strbuf_release (&msg); +} + +int fsm_listen__ctor (struct fsmonitor_daemon_state *state) +{ +int fd; +int ret =0 ; +struct fsm_listen_data *data; + +CALLOC_ARRAY (data,1 ); + state->listen_data = data; + state->listen_error_code =-1 ; + data->shutdown =SHUTDOWN_ERROR ; + + fd =inotify_init1 (O_NONBLOCK ); +if (fd <0 ) +return error_errno ("inotify_init1() failed" ); + + data->fd_inotify = fd; + +hashmap_init (&data->watches, watch_entry_cmp,NULL ,0 ); +hashmap_init (&data->renames, rename_entry_cmp,NULL ,0 ); +hashmap_init (&data->revwatches, revwatches_entry_cmp,NULL ,0 ); + +if (add_watch (state->path_worktree_watch.buf, data)) + ret =-1 ; +else if (register_inotify (state->path_worktree_watch.buf, state,NULL )) + ret =-1 ; +else if (state->nr_paths_watching >1 ) { +if (add_watch (state->path_gitdir_watch.buf, data)) + ret =-1 ; +else if (register_inotify (state->path_gitdir_watch.buf, state,NULL )) + ret =-1 ; + } + +if (!ret) { + state->listen_error_code =0 ; + data->shutdown =SHUTDOWN_CONTINUE ; + } + +return ret; +} + +void fsm_listen__dtor (struct fsmonitor_daemon_state *state) +{ +struct fsm_listen_data *data; +struct hashmap_iter iter; +struct watch_entry *w; +int fd; + +if (!state || !state->listen_data) +return ; + + data = state->listen_data; + fd = data->fd_inotify; + +hashmap_for_each_entry (&data->watches, &iter, w, ent) { + w->cookie =0 ;/* ignore any pending renames */ +remove_watch (w, data); + } +hashmap_clear (&data->watches); + +hashmap_clear (&data->revwatches);/* remove_watch freed the entries */ + + hashmap_clear_and_free(&data->renames, struct rename_entry, ent); + +FREE_AND_NULL (state->listen_data); + +if (fd && (close (fd) <0 )) +error_errno (_ ("closing inotify file descriptor failed" )); +} + +void fsm_listen__stop_async (struct fsmonitor_daemon_state *state) +{ +if (!state->listen_data->shutdown) + state->listen_data->shutdown =SHUTDOWN_STOP ; +} + +/* +* Process a single inotify event and queue for publication. +*/ +static int process_event (const char *path, +const struct inotify_event *event, +struct fsmonitor_batch *batch, +struct string_list *cookie_list, +struct fsmonitor_daemon_state *state) +{ +const char *rel; +const char *last_sep; + +switch (fsmonitor_classify_path_absolute (state, path)) { +case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX : +case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX : +/* Use just the filename of the cookie file. */ + last_sep =find_last_dir_sep (path); +string_list_append (cookie_list, + last_sep ? last_sep +1 : path); +break ; +case IS_INSIDE_DOT_GIT : +case IS_INSIDE_GITDIR : +break ; +case IS_DOT_GIT : +case IS_GITDIR : +/* +* If .git directory is deleted or renamed away, +* we have to quit. +*/ +if (em_dir_deleted (event->mask)) { +trace_printf_key (&trace_fsmonitor, +"event: gitdir removed" ); + state->listen_data->shutdown =SHUTDOWN_FORCE ; + goto done; + } + +if (em_dir_renamed (event->mask)) { +trace_printf_key (&trace_fsmonitor, +"event: gitdir renamed" ); + state->listen_data->shutdown =SHUTDOWN_FORCE ; + goto done; + } +break ; +case IS_WORKDIR_PATH : +/* normal events in the working directory */ +if (trace_pass_fl (&trace_fsmonitor)) +log_mask_set (path, event->mask); + + rel = path + state->path_worktree_watch.len +1 ; +fsmonitor_batch__add_path (batch, rel); + +if (em_dir_deleted (event->mask)) +break ; + +/* received IN_MOVE_FROM, add tracking for expected IN_MOVE_TO */ +if (em_rename_dir_from (event->mask)) +add_dir_rename (event->cookie, path, state->listen_data); + +/* received IN_MOVE_TO, update watch to reflect new path */ +if (em_rename_dir_to (event->mask)) { +rename_dir (event->cookie, path, state->listen_data); +if (register_inotify (path, state, batch)) { + state->listen_data->shutdown =SHUTDOWN_ERROR ; + goto done; + } + } + +if (em_dir_created (event->mask)) { +if (add_watch (path, state->listen_data)) { + state->listen_data->shutdown =SHUTDOWN_ERROR ; + goto done; + } +if (register_inotify (path, state, batch)) { + state->listen_data->shutdown =SHUTDOWN_ERROR ; + goto done; + } + } +break ; +case IS_OUTSIDE_CONE : +default : +trace_printf_key (&trace_fsmonitor, +"ignoring '%s'" , path); +break ; + } +return 0 ; +done: +return -1 ; +} + +/* +* Read the inotify event stream and pre-process events before further +* processing and eventual publishing. +*/ +static void handle_events (struct fsmonitor_daemon_state *state) +{ +/* See https://man7.org/linux/man-pages/man7/inotify.7.html */ + char buf[4096] + __attribute__ ((aligned(__alignof__(struct inotify_event)))); + +struct hashmap watches = state->listen_data->watches; +struct fsmonitor_batch *batch =NULL ; +struct string_list cookie_list =STRING_LIST_INIT_DUP ; +struct watch_entry k, *w; +struct strbuf path; +const struct inotify_event *event; +int fd = state->listen_data->fd_inotify; +ssize_t len; +char *ptr, *p; + +strbuf_init (&path,PATH_MAX ); + +for (;;) { + len =read (fd, buf,sizeof (buf)); +if (len ==-1 && errno !=EAGAIN ) { +error_errno (_ ("reading inotify message stream failed" )); + state->listen_data->shutdown =SHUTDOWN_ERROR ; + goto done; + } + +/* nothing to read */ +if (len <=0 ) + goto done; + +/* Loop over all events in the buffer. */ +for (ptr = buf; ptr < buf + len; + ptr +=sizeof (struct inotify_event ) + event->len) { + + event = (const struct inotify_event *) ptr; + +if (em_ignore (event->mask)) +continue ; + +/* File system was unmounted or event queue overflowed */ +if (em_force_shutdown (event->mask)) { +if (trace_pass_fl (&trace_fsmonitor)) +log_mask_set ("Forcing shutdown" , event->mask); + state->listen_data->shutdown =SHUTDOWN_FORCE ; + goto done; + } + +hashmap_entry_init (&k.ent,memhash (&event->wd,sizeof (int ))); + k.wd = event->wd; + + w =hashmap_get_entry (&watches, &k, ent,NULL ); +if (!w)/* should never happen */ +BUG ("No watch for '%s'" , event->name); + +/* directory watch was removed */ +if (em_remove_watch (event->mask)) { +remove_watch (w, state->listen_data); +continue ; + } + +strbuf_reset (&path); +strbuf_add (&path, w->dir,strlen (w->dir)); +strbuf_addch (&path,'/' ); +strbuf_addstr (&path, event->name); + + p =fsmonitor__resolve_alias (path.buf, &state->alias); +if (!p) + p =strbuf_detach (&path,NULL ); + +if (!batch) + batch =fsmonitor_batch__new (); + +if (process_event (p, event, batch, &cookie_list, state)) { +free (p); + goto done; + } +free (p); + } +strbuf_reset (&path); +fsmonitor_publish (state, batch, &cookie_list); +string_list_clear (&cookie_list,0 ); + batch =NULL ; + } +done: +strbuf_release (&path); +fsmonitor_batch__free_list (batch); +string_list_clear (&cookie_list,0 ); +} + +/* +* Non-blocking read of the inotify events stream. The inotify fd is polled +* frequently to help minimize the number of queue overflows. +*/ +void fsm_listen__loop (struct fsmonitor_daemon_state *state) +{ +int poll_num; +const int interval =1000 ; +time_t checked =time (NULL ); +struct pollfd fds[1 ]; + fds[0 ].fd = state->listen_data->fd_inotify; + fds[0 ].events =POLLIN ; + +for (;;) { +switch (state->listen_data->shutdown) { +case SHUTDOWN_CONTINUE : + poll_num =poll (fds,1 ,1 ); +if (poll_num ==-1 ) { +if (errno ==EINTR ) +continue ; +error_errno (_ ("polling inotify message stream failed" )); + state->listen_data->shutdown =SHUTDOWN_ERROR ; +continue ; + } + +if ((time (NULL ) - checked) >= interval) { + checked =time (NULL ); +if (check_stale_dir_renames (&state->listen_data->renames, + checked - interval)) { +trace_printf_key (&trace_fsmonitor, +"Missed IN_MOVED_TO events, forcing shutdown" ); + state->listen_data->shutdown =SHUTDOWN_FORCE ; +continue ; + } + } + +if (poll_num >0 && (fds[0 ].revents &POLLIN )) +handle_events (state); + +continue ; +case SHUTDOWN_ERROR : + state->listen_error_code =-1 ; +ipc_server_stop_async (state->ipc_server_data); +break ; +case SHUTDOWN_FORCE : + state->listen_error_code =0 ; +ipc_server_stop_async (state->ipc_server_data); +break ; +case SHUTDOWN_STOP : +default : + state->listen_error_code =0 ; +break ; + } +return ; + } +}
Signature · 1 line
gitgitgadget
PATCH v3 4/6 fsmonitor: enable fsmonitor for Linux
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
Update build to enable fsmonitor for Linux.
--- config.mak.uname | 8 ++++++++ contrib/buildsystems/CMakeLists.txt | 11 ++++++++++- 2 files changed, 18 insertions(+), 1 deletion(-)
config.mak.uname +8 -0
@@ -68,6 +68,14 @@ ifeq ($(uname_S),Linux) ifneq ($(findstring .el7.,$(uname_R)),) BASIC_CFLAGS += -std=c99 endif + # The builtin FSMonitor on Linux builds upon Simple-IPC. Both require + # Unix domain sockets and PThreads. + ifndef NO_PTHREADS + ifndef NO_UNIX_SOCKETS + FSMONITOR_DAEMON_BACKEND = linux + FSMONITOR_OS_SETTINGS = linux + endif + endif endif ifeq ($(uname_S),GNU/kFreeBSD) HAVE_ALLOCA_H = YesPlease
contrib/buildsystems/CMakeLists.txt +10 -1
@@ -304,7 +304,16 @@ else() endif() if(SUPPORTS_SIMPLE_IPC) - if(CMAKE_SYSTEM_NAME STREQUAL "Windows") + if(CMAKE_SYSTEM_NAME STREQUAL "Linux") + add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-linux.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-linux.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-linux.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-linux.c) + + add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-linux.c) + elseif(CMAKE_SYSTEM_NAME STREQUAL "Windows") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c) list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c)
Signature · 1 line
gitgitgadget
PATCH v3 5/6 fsmonitor: test updates
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
t7527-builtin-fsmonitor was leaking fsmonitor--daemon processes in some cases.
Accomodate slight difference in the number of events generated on Linux.
On lower-powered systems, spin a little to give the daemon time to respond to and log filesystem events.
--- t/t7527-builtin-fsmonitor.sh | 94 ++++++++++++++++++++++++++++++------ 1 file changed, 80 insertions(+), 14 deletions(-)
t/t7527-builtin-fsmonitor.sh +80 -14
@@ -13,7 +13,7 @@ fi stop_daemon_delete_repo () { r=$1 && test_might_fail git -C $r fsmonitor--daemon stop && - rm -rf $1 + rm -rf $r } start_daemon () {
@@ -72,6 +72,34 @@ start_daemon () { ) } +IMPLICIT_TIMEOUT=5 + +wait_for_update () { + func=$1 && + file=$2 && + sz=$(wc -c < "$file") && + last=0 && + $func && + k=0 && + while test "$k" -lt $IMPLICIT_TIMEOUT + do + nsz=$(wc -c < "$file") + if test "$nsz" -gt "$sz" + then + if test "$last" -eq "$nsz" + then + cat "$file" && + return 0 + fi + last=$nsz + fi + sleep 1 + k=$(( $k + 1 )) + done && + cat "$file" && + return 0 +} + # Is a Trace2 data event present with the given catetory and key? # We do not care what the value is. #
@@ -137,7 +165,6 @@ test_expect_success 'implicit daemon start' ' # machines (where it might take a moment to wake and reschedule the # daemon process) to avoid false alarms during test runs.) # -IMPLICIT_TIMEOUT=5 verify_implicit_shutdown () { r=$1 &&
@@ -373,6 +400,15 @@ create_files () { echo 3 >dir2/new } +rename_directory () { + mv dirtorename dirrenamed +} + +rename_directory_file () { + mv dirtorename dirrenamed && + echo 1 > dirrenamed/new +} + rename_files () { mv rename renamed && mv dir1/rename dir1/renamed &&
@@ -427,10 +463,12 @@ test_expect_success 'edit some files' ' start_daemon --tf "$PWD/.git/trace" && - edit_files && + wait_for_update edit_files "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 && + test_might_fail git fsmonitor--daemon stop && + grep "^event: dir1/modified$" .git/trace && grep "^event: dir2/modified$" .git/trace && grep "^event: modified$" .git/trace &&
@@ -442,10 +480,12 @@ test_expect_success 'create some files' ' start_daemon --tf "$PWD/.git/trace" && - create_files && + wait_for_update create_files "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 && + test_might_fail git fsmonitor--daemon stop && + grep "^event: dir1/new$" .git/trace && grep "^event: dir2/new$" .git/trace && grep "^event: new$" .git/trace
@@ -456,10 +496,12 @@ test_expect_success 'delete some files' ' start_daemon --tf "$PWD/.git/trace" && - delete_files && + wait_for_update delete_files "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 && + test_might_fail git fsmonitor--daemon stop && + grep "^event: dir1/delete$" .git/trace && grep "^event: dir2/delete$" .git/trace && grep "^event: delete$" .git/trace
@@ -470,10 +512,12 @@ test_expect_success 'rename some files' ' start_daemon --tf "$PWD/.git/trace" && - rename_files && + wait_for_update rename_files "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 && + test_might_fail git fsmonitor--daemon stop && + grep "^event: dir1/rename$" .git/trace && grep "^event: dir2/rename$" .git/trace && grep "^event: rename$" .git/trace &&
@@ -487,23 +531,42 @@ test_expect_success 'rename directory' ' start_daemon --tf "$PWD/.git/trace" && - mv dirtorename dirrenamed && + wait_for_update rename_directory "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 && + test_might_fail git fsmonitor--daemon stop && + grep "^event: dirtorename/*$" .git/trace && grep "^event: dirrenamed/*$" .git/trace ' +test_expect_success 'rename directory file' ' + test_when_finished clean_up_repo_and_stop_daemon && + + start_daemon --tf "$PWD/.git/trace" && + + wait_for_update rename_directory_file "$PWD/.git/trace" && + + test-tool fsmonitor-client query --token 0 && + + test_might_fail git fsmonitor--daemon stop && + + grep "^event: dirtorename/*$" .git/trace && + grep "^event: dirrenamed/*$" .git/trace && + grep "^event: dirrenamed/new$" .git/trace +' test_expect_success 'file changes to directory' ' test_when_finished clean_up_repo_and_stop_daemon && start_daemon --tf "$PWD/.git/trace" && - file_to_directory && + wait_for_update file_to_directory "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 && + test_might_fail git fsmonitor--daemon stop && + grep "^event: delete$" .git/trace && grep "^event: delete/new$" .git/trace '
@@ -513,10 +576,12 @@ test_expect_success 'directory changes to a file' ' start_daemon --tf "$PWD/.git/trace" && - directory_to_file && + wait_for_update directory_to_file "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 && + test_might_fail git fsmonitor--daemon stop && + grep "^event: dir1$" .git/trace '
@@ -561,7 +626,7 @@ test_expect_success 'flush cached data' ' test-tool -C test_flush fsmonitor-client query --token "builtin:test_00000002:0" >actual_2 && nul_to_q <actual_2 >actual_q2 && - grep "^builtin:test_00000002:0Q$" actual_q2 && + grep "^builtin:test_00000002:[0-1]Q$" actual_q2 && >test_flush/file_3 &&
@@ -732,7 +797,8 @@ u_values="$u1 $u2" for u in $u_values do test_expect_success "unicode in repo root path: $u" ' - test_when_finished "stop_daemon_delete_repo $u" && + test_when_finished \ + "stop_daemon_delete_repo `echo "$u" | sed 's:x:\\\\\\\\\\\\\\x:g'`" && git init "$u" && echo 1 >"$u"/file1 &&
@@ -818,8 +884,7 @@ test_expect_success 'submodule setup' ' ' test_expect_success 'submodule always visited' ' - test_when_finished "git -C super fsmonitor--daemon stop; \ - rm -rf super; \ + test_when_finished "rm -rf super; \ rm -rf sub" && create_super super &&
@@ -887,7 +952,8 @@ have_t2_error_event () { } test_expect_success "stray submodule super-prefix warning" ' - test_when_finished "rm -rf super; \ + test_when_finished "git -C super/dir_1/dir_2/sub fsmonitor--daemon stop; \ + rm -rf super; \ rm -rf sub; \ rm super-sub.trace" &&
Signature · 1 line
gitgitgadget
PATCH v3 6/6 fsmonitor: update doc for Linux
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
Update the documentation for Linux.
--- Documentation/config/fsmonitor--daemon.txt | 4 ++-- Documentation/git-fsmonitor--daemon.txt | 24 ++++++++++++++-------- 2 files changed, 18 insertions(+), 10 deletions(-)
Documentation/config/fsmonitor--daemon.txt +2 -2
@@ -4,8 +4,8 @@ fsmonitor.allowRemote:: behavior. Only respected when `core.fsmonitor` is set to `true`. fsmonitor.socketDir:: - This Mac OS-specific option, if set, specifies the directory in + Mac OS and Linux-specific option. If set, specifies the directory in which to create the Unix domain socket used for communication between the fsmonitor daemon and various Git commands. The directory must - reside on a native Mac OS filesystem. Only respected when `core.fsmonitor` + reside on a native filesystem. Only respected when `core.fsmonitor` is set to `true`.
Documentation/git-fsmonitor--daemon.txt +16 -8
@@ -76,23 +76,31 @@ repositories; this may be overridden by setting `fsmonitor.allowRemote` to correctly with all network-mounted repositories and such use is considered experimental. -On Mac OS, the inter-process communication (IPC) between various Git +On Linux and Mac OS, the inter-process communication (IPC) between various Git commands and the fsmonitor daemon is done via a Unix domain socket (UDS) -- a -special type of file -- which is supported by native Mac OS filesystems, -but not on network-mounted filesystems, NTFS, or FAT32. Other filesystems -may or may not have the needed support; the fsmonitor daemon is not guaranteed -to work with these filesystems and such use is considered experimental. +special type of file -- which is supported by many native Linux and Mac OS +filesystems, but not on network-mounted filesystems, NTFS, or FAT32. Other +filesystems may or may not have the needed support; the fsmonitor daemon is not +guaranteed to work with these filesystems and such use is considered +experimental. By default, the socket is created in the `.git` directory, however, if the `.git` directory is on a network-mounted filesystem, it will be instead be created at `$HOME/.git-fsmonitor-*` unless `$HOME` itself is on a network-mounted filesystem in which case you must set the configuration -variable `fsmonitor.socketDir` to the path of a directory on a Mac OS native +variable `fsmonitor.socketDir` to the path of a directory on a native filesystem in which to create the socket file. If none of the above directories (`.git`, `$HOME`, or `fsmonitor.socketDir`) -is on a native Mac OS file filesystem the fsmonitor daemon will report an -error that will cause the daemon and the currently running command to exit. +is on a native Linux or Mac OS filesystem the fsmonitor daemon will report +an error that will cause the daemon to exit and the currently running command +to issue a warning. + +On Linux, the fsmonitor daemon registers a watch for each directory in the +repository. The default per-user limit for the number of watches on most Linux +systems is 8192. This may not be sufficient for large repositories or if +multiple instances of the fsmonitor daemon are running. +See https://watchexec.github.io/docs/inotify-limits.html[Linux inotify limits] for more information. CONFIGURATION -------------
Signature · 1 line
gitgitgadget
[+cc Jeff Hostetler]
Quoted 7 lines · Eric DeCosta via GitGitGadget
Eric DeCosta (6):
fsmonitor: prepare to share code between Mac OS and Linux
fsmonitor: determine if filesystem is local or remote
fsmonitor: implement filesystem change listener for Linux
fsmonitor: enable fsmonitor for Linux
fsmonitor: test updates
fsmonitor: update doc for Linux
Thanks, I picked these up. Let's add Jeff Hostetler (cc'd) to the chain and see if he has any thoughts.
Thanks,
Taylor
PATCH v4 1/6 fsmonitor: prepare to share code between Mac OS and Linux
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
Linux and Mac OS can share some of the code originally developed for Mac OS.
Mac OS and Linux can share fsm-ipc-unix.c and fsm-settings-unix.c
--- compat/fsmonitor/fsm-health-linux.c | 24 ++++++++++ compat/fsmonitor/fsm-ipc-darwin.c | 53 +--------------------- compat/fsmonitor/fsm-ipc-linux.c | 1 + compat/fsmonitor/fsm-ipc-unix.c | 52 +++++++++++++++++++++ compat/fsmonitor/fsm-settings-darwin.c | 63 +------------------------- compat/fsmonitor/fsm-settings-linux.c | 1 + compat/fsmonitor/fsm-settings-unix.c | 61 +++++++++++++++++++++++++ 7 files changed, 141 insertions(+), 114 deletions(-) create mode 100644 compat/fsmonitor/fsm-health-linux.c create mode 100644 compat/fsmonitor/fsm-ipc-linux.c create mode 100644 compat/fsmonitor/fsm-ipc-unix.c create mode 100644 compat/fsmonitor/fsm-settings-linux.c create mode 100644 compat/fsmonitor/fsm-settings-unix.c
compat/fsmonitor/fsm-health-linux.c new file +24 -0
@@ -0,0 +1,24 @@ +#include "cache.h" +#include "config.h" +#include "fsmonitor.h" +#include "fsm-health.h" +#include "fsmonitor--daemon.h" + +int fsm_health__ctor (struct fsmonitor_daemon_state *state) +{ +return 0 ; +} + +void fsm_health__dtor (struct fsmonitor_daemon_state *state) +{ +return ; +} + +void fsm_health__loop (struct fsmonitor_daemon_state *state) +{ +return ; +} + +void fsm_health__stop_async (struct fsmonitor_daemon_state *state) +{ +}
compat/fsmonitor/fsm-ipc-darwin.c +1 -52
53 lines not on this page · raw message
compat/fsmonitor/fsm-ipc-linux.c new file +1 -0
@@ -0,0 +1 @@ +#include "fsm-ipc-unix.c"
compat/fsmonitor/fsm-ipc-unix.c new file +52 -0
52 lines not on this page · raw message
compat/fsmonitor/fsm-settings-darwin.c +1 -62
63 lines not on this page · raw message
compat/fsmonitor/fsm-settings-linux.c new file +1 -0
@@ -0,0 +1 @@ +#include "fsm-settings-unix.c"
compat/fsmonitor/fsm-settings-unix.c new file +61 -0
61 lines not on this page · raw message
Signature · 1 line
gitgitgadget
v4 cover fsmonitor: Implement fsmonitor for Linux
In reply to Eric DeCosta via GitGitGadgetGoal is to deliver fsmonitor for Linux that is on par with fsmonitor for Windows and Mac OS.
This patch set builds upon previous work for done for Windows and Mac OS to implement a fsmonitor back-end for Linux based on the Linux inotify API. inotify differs significantly from the equivalent Windows and Mac OS APIs in that a watch must be registered for every directory of interest (rather than a singular watch at the root of the directory tree) and special care must be taken to handle directory renames correctly.
More information about inotify:
https://man7.org/linux/man-pages/man7/inotify.7.html
v3 differs from v2:
* Avoid potential entanglements with GPLv3
* Classify a reasonable set of filesystems as being remote
v2 differs from v1:
* Prior work for Windows and Mac OS has been merged to master, reducing the patch set from 12 to 6 patches
* Code review feedback
* Identified and resolved race condition revealed by CI test system, see "Limitations and caveats" regarding monitoring of directory trees from the man page, above
* Apologies for being away from this for so long, but my attention was needed elsewhere
v1 differs from v0:
* Code review feedback
* Update how and which code can be shared between Mac OS and Linux
* Increase polling frequency to every 1ms (matches Mac OS)
* Updates to t7527 to improve test stability
Eric DeCosta (6):
fsmonitor: prepare to share code between Mac OS and Linux
fsmonitor: determine if filesystem is local or remote
fsmonitor: implement filesystem change listener for Linux
fsmonitor: enable fsmonitor for Linux
fsmonitor: test updates
fsmonitor: update doc for Linux
Documentation/config/fsmonitor--daemon.txt | 4 +- Documentation/git-fsmonitor--daemon.txt | 24 +- compat/fsmonitor/fsm-health-linux.c | 24 + compat/fsmonitor/fsm-ipc-darwin.c | 53 +- compat/fsmonitor/fsm-ipc-linux.c | 1 + compat/fsmonitor/fsm-ipc-unix.c | 52 ++ compat/fsmonitor/fsm-listen-linux.c | 676 +++++++++++++++++++++ compat/fsmonitor/fsm-path-utils-linux.c | 186 ++++++ compat/fsmonitor/fsm-settings-darwin.c | 63 +- compat/fsmonitor/fsm-settings-linux.c | 1 + compat/fsmonitor/fsm-settings-unix.c | 61 ++ config.mak.uname | 8 + contrib/buildsystems/CMakeLists.txt | 11 +- t/t7527-builtin-fsmonitor.sh | 94 ++- 14 files changed, 1119 insertions(+), 139 deletions(-) create mode 100644 compat/fsmonitor/fsm-health-linux.c create mode 100644 compat/fsmonitor/fsm-ipc-linux.c create mode 100644 compat/fsmonitor/fsm-ipc-unix.c create mode 100644 compat/fsmonitor/fsm-listen-linux.c create mode 100644 compat/fsmonitor/fsm-path-utils-linux.c create mode 100644 compat/fsmonitor/fsm-settings-linux.c create mode 100644 compat/fsmonitor/fsm-settings-unix.c
base-commit: 319605f8f00e402f3ea758a02c63534ff800a711
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1352%2Fedecosta-mw%2Ffsmonitor_linux-v4
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1352/edecosta-mw/fsmonitor_linux-v4
Pull-Request: https://github.com/git/git/pull/1352
Range-diff vs v3:
1: 99d684c7bdf = 1: 99d684c7bdf fsmonitor: prepare to share code between Mac OS and Linux
2: aa405379891 ! 2: e53fc077540 fsmonitor: determine if filesystem is local or remote
@@ compat/fsmonitor/fsm-path-utils-linux.c (new)
+#include <errno.h>
+#include <mntent.h>
+#include <sys/mount.h>
++#include <sys/vfs.h>
+#include <sys/statvfs.h>
+
-+/*
-+ * https://github.com/coreutils/gnulib/blob/master/lib/mountlist.c
-+ */
-+#ifndef ME_REMOTE
-+/* A file system is "remote" if its Fs_name contains a ':'
-+ or if (it is of type (smbfs or cifs) and its Fs_name starts with '//')
-+ or if it is of any other of the listed types
-+ or Fs_name is equal to "-hosts" (used by autofs to mount remote fs).
-+ "VM" file systems like prl_fs or vboxsf are not considered remote here. */
-+# define ME_REMOTE(Fs_name, Fs_type) \
-+ (strchr (Fs_name, ':') != NULL \
-+ || ((Fs_name)[0] == '/' \
-+ && (Fs_name)[1] == '/' \
-+ && (strcmp (Fs_type, "smbfs") == 0 \
-+ || strcmp (Fs_type, "smb3") == 0 \
-+ || strcmp (Fs_type, "cifs") == 0)) \
-+ || strcmp (Fs_type, "acfs") == 0 \
-+ || strcmp (Fs_type, "afs") == 0 \
-+ || strcmp (Fs_type, "coda") == 0 \
-+ || strcmp (Fs_type, "auristorfs") == 0 \
-+ || strcmp (Fs_type, "fhgfs") == 0 \
-+ || strcmp (Fs_type, "gpfs") == 0 \
-+ || strcmp (Fs_type, "ibrix") == 0 \
-+ || strcmp (Fs_type, "ocfs2") == 0 \
-+ || strcmp (Fs_type, "vxfs") == 0 \
-+ || strcmp ("-hosts", Fs_name) == 0)
-+#endif
++static int is_remote_fs(const char* path) {
++ struct statfs fs;
++
++ if (statfs(path, &fs)) {
++ error_errno(_("statfs('%s') failed"), path);
++ return -1;
++ }
++
++ switch (fs.f_type) {
++ case 0x61636673: /* ACFS */
++ case 0x5346414F: /* AFS */
++ case 0x00C36400: /* CEPH */
++ case 0xFF534D42: /* CIFS */
++ case 0x73757245: /* CODA */
++ case 0x19830326: /* FHGFS */
++ case 0x1161970: /* GFS */
++ case 0x47504653: /* GPFS */
++ case 0x013111A8: /* IBRIX */
++ case 0x6B414653: /* KAFS */
++ case 0x0BD00BD0: /* LUSTRE */
++ case 0x564C: /* NCP */
++ case 0x6969: /* NFS */
++ case 0x6E667364: /* NFSD */
++ case 0x7461636f: /* OCFS2 */
++ case 0xAAD7AAEA: /* PANFS */
++ case 0x517B: /* SMB */
++ case 0xBEEFDEAD: /* SNFS */
++ case 0xFE534D42: /* SMB2 */
++ case 0xBACBACBC: /* VMHGFS */
++ case 0xA501FCF5: /* VXFS */
++ return 1;
++ default:
++ break;
++ }
++
++ return 0;
++}
+
+static int find_mount(const char *path, const struct statvfs *fs,
+ struct mntent *ent)
@@ compat/fsmonitor/fsm-path-utils-linux.c (new)
+ "statvfs('%s') [flags 0x%08lx] '%s' '%s'",
+ path, fs.f_flag, ment.mnt_type, ment.mnt_fsname);
+
-+ fs_info->is_remote = ME_REMOTE(ment.mnt_fsname, ment.mnt_type);
++ fs_info->is_remote = is_remote_fs(ment.mnt_dir);
+ fs_info->typename = ment.mnt_fsname;
+ free(ment.mnt_dir);
+ free(ment.mnt_type);
+
++ if (fs_info->is_remote < 0) {
++ free(ment.mnt_fsname);
++ return -1;
++ }
++
+ trace_printf_key(&trace_fsmonitor,
+ "'%s' is_remote: %d",
+ path, fs_info->is_remote);
++
+ return 0;
+}
+
3: c2e5a7201aa = 3: 80282efef57 fsmonitor: implement filesystem change listener for Linux
4: 05f5b2dd4fb = 4: cb03803e355 fsmonitor: enable fsmonitor for Linux
5: 754355ca44f = 5: 8d9d469b356 fsmonitor: test updates
6: f56175e097a = 6: 5afd03fa6ca fsmonitor: update doc for LinuxSignature · 1 line
gitgitgadget
PATCH v4 2/6 fsmonitor: determine if filesystem is local or remote
From: Eric DeCosta <[email protected]>
Compare the given path to the mounted filesystems. Find the mount that is the longest prefix of the path (if any) and determine if that mount is on a local or remote filesystem.
--- compat/fsmonitor/fsm-path-utils-linux.c | 186 ++++++++++++++++++++++++ 1 file changed, 186 insertions(+) create mode 100644 compat/fsmonitor/fsm-path-utils-linux.c
compat/fsmonitor/fsm-path-utils-linux.c new file +186 -0
@@ -0,0 +1,186 @@ +#include "fsmonitor.h" +#include "fsmonitor-path-utils.h" +#include <errno.h> +#include <mntent.h> +#include <sys/mount.h> +#include <sys/vfs.h> +#include <sys/statvfs.h> + +static int is_remote_fs (const char * path) { +struct statfs fs; + +if (statfs (path, &fs)) { +error_errno (_ ("statfs('%s') failed" ), path); +return -1 ; + } + +switch (fs.f_type) { +case 0x61636673 :/* ACFS */ +case 0x5346414F :/* AFS */ +case 0x00C36400 :/* CEPH */ +case 0xFF534D42 :/* CIFS */ +case 0x73757245 :/* CODA */ +case 0x19830326 :/* FHGFS */ +case 0x1161970 :/* GFS */ +case 0x47504653 :/* GPFS */ +case 0x013111A8 :/* IBRIX */ +case 0x6B414653 :/* KAFS */ +case 0x0BD00BD0 :/* LUSTRE */ +case 0x564C :/* NCP */ +case 0x6969 :/* NFS */ +case 0x6E667364 :/* NFSD */ +case 0x7461636f :/* OCFS2 */ +case 0xAAD7AAEA :/* PANFS */ +case 0x517B :/* SMB */ +case 0xBEEFDEAD :/* SNFS */ +case 0xFE534D42 :/* SMB2 */ +case 0xBACBACBC :/* VMHGFS */ +case 0xA501FCF5 :/* VXFS */ +return 1 ; +default : +break ; + } + +return 0 ; +} + +static int find_mount (const char *path,const struct statvfs *fs, +struct mntent *ent) +{ +const char *const mounts ="/proc/mounts" ; +const char *rp =real_pathdup (path,1 ); +struct mntent *ment =NULL ; +struct statvfs mntfs; +FILE *fp; +int found =0 ; +int dlen, plen, flen =0 ; + + ent->mnt_fsname =NULL ; + ent->mnt_dir =NULL ; + ent->mnt_type =NULL ; + + fp =setmntent (mounts,"r" ); +if (!fp) { +error_errno (_ ("setmntent('%s') failed" ), mounts); +return -1 ; + } + + plen =strlen (rp); + +/* read all the mount information and compare to path */ +while ((ment =getmntent (fp)) !=NULL ) { +if (statvfs (ment->mnt_dir, &mntfs)) { +switch (errno) { +case EPERM : +case ESRCH : +case EACCES : +continue ; +default : +error_errno (_ ("statvfs('%s') failed" ), ment->mnt_dir); +endmntent (fp); +return -1 ; + } + } + +/* is mount on the same filesystem and is a prefix of the path */ +if ((fs->f_fsid == mntfs.f_fsid) && + !strncmp (ment->mnt_dir, rp,strlen (ment->mnt_dir))) { + dlen =strlen (ment->mnt_dir); +if (dlen > plen) +continue ; +/* +* root is always a potential match; otherwise look for +* directory prefix +*/ +if ((dlen ==1 && ment->mnt_dir[0 ] =='/' ) || + (dlen > flen && (!rp[dlen] || rp[dlen] =='/' ))) { + flen = dlen; +/* +* https://man7.org/linux/man-pages/man3/getmntent.3.html +* +* The pointer points to a static area of memory which is +* overwritten by subsequent calls to getmntent(). +*/ + found =1 ; +free (ent->mnt_fsname); +free (ent->mnt_dir); +free (ent->mnt_type); + ent->mnt_fsname =xstrdup (ment->mnt_fsname); + ent->mnt_dir =xstrdup (ment->mnt_dir); + ent->mnt_type =xstrdup (ment->mnt_type); + } + } + } +endmntent (fp); + +if (!found) +return -1 ; + +return 0 ; +} + +int fsmonitor__get_fs_info (const char *path,struct fs_info *fs_info) +{ +struct mntent ment; +struct statvfs fs; + +if (statvfs (path, &fs)) +return error_errno (_ ("statvfs('%s') failed" ), path); + + +if (find_mount (path, &fs, &ment) <0 ) { +free (ment.mnt_fsname); +free (ment.mnt_dir); +free (ment.mnt_type); +return -1 ; + } + +trace_printf_key (&trace_fsmonitor, +"statvfs('%s') [flags 0x%08lx] '%s' '%s'" , + path, fs.f_flag, ment.mnt_type, ment.mnt_fsname); + + fs_info->is_remote =is_remote_fs (ment.mnt_dir); + fs_info->typename = ment.mnt_fsname; +free (ment.mnt_dir); +free (ment.mnt_type); + +if (fs_info->is_remote <0 ) { +free (ment.mnt_fsname); +return -1 ; + } + +trace_printf_key (&trace_fsmonitor, +"'%s' is_remote: %d" , + path, fs_info->is_remote); + +return 0 ; +} + +int fsmonitor__is_fs_remote (const char *path) +{ +struct fs_info fs; + +if (fsmonitor__get_fs_info (path, &fs)) +return -1 ; + +free (fs.typename); + +return fs.is_remote; +} + +/* +* No-op for now. +*/ +int fsmonitor__get_alias (const char *path,struct alias_info *info) +{ +return 0 ; +} + +/* +* No-op for now. +*/ +char *fsmonitor__resolve_alias (const char *path, +const struct alias_info *info) +{ +return NULL ; +}
Signature · 1 line
gitgitgadget
PATCH v4 3/6 fsmonitor: implement filesystem change listener for Linux
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
Implement a filesystem change listener for Linux based on the inotify API:
https://man7.org/linux/man-pages/man7/inotify.7.html
inotify requires registering a watch on every directory in the worktree and special handling of moves/renames.
--- compat/fsmonitor/fsm-listen-linux.c | 676 ++++++++++++++++++++++++++++ 1 file changed, 676 insertions(+) create mode 100644 compat/fsmonitor/fsm-listen-linux.c
compat/fsmonitor/fsm-listen-linux.c new file +676 -0
@@ -0,0 +1,676 @@ +#include "cache.h" +#include "fsmonitor.h" +#include "fsm-listen.h" +#include "fsmonitor--daemon.h" +#include <dirent.h> +#include <fcntl.h> +#include <sys/inotify.h> +#include <sys/stat.h> + +/* +* Safe value to bitwise OR with rest of mask for +* kernels that do not support IN_MASK_CREATE +*/ +#ifndef IN_MASK_CREATE +#define IN_MASK_CREATE 0x00000000 +#endif + +enum shutdown_reason { +SHUTDOWN_CONTINUE =0 , +SHUTDOWN_STOP , +SHUTDOWN_ERROR , +SHUTDOWN_FORCE +}; + +struct watch_entry { +struct hashmap_entry ent; +int wd; +uint32_t cookie; +const char *dir; +}; + +struct rename_entry { +struct hashmap_entry ent; +time_t whence; +uint32_t cookie; +const char *dir; +}; + +struct fsm_listen_data { +int fd_inotify; +enum shutdown_reason shutdown; +struct hashmap watches; +struct hashmap renames; +struct hashmap revwatches; +}; + +static int watch_entry_cmp (const void *cmp_data, +const struct hashmap_entry *eptr, +const struct hashmap_entry *entry_or_key, +const void *keydata) +{ +const struct watch_entry *e1, *e2; + + e1 = container_of(eptr, const struct watch_entry, ent); + e2 = container_of(eptr, const struct watch_entry, ent); +return e1->wd != e2->wd; +} + +static int revwatches_entry_cmp (const void *cmp_data, +const struct hashmap_entry *eptr, +const struct hashmap_entry *entry_or_key, +const void *keydata) +{ +const struct watch_entry *e1, *e2; + + e1 = container_of(eptr, const struct watch_entry, ent); + e2 = container_of(eptr, const struct watch_entry, ent); +return strcmp (e1->dir, e2->dir); +} + +static int rename_entry_cmp (const void *cmp_data, +const struct hashmap_entry *eptr, +const struct hashmap_entry *entry_or_key, +const void *keydata) +{ +const struct rename_entry *e1, *e2; + + e1 = container_of(eptr, const struct rename_entry, ent); + e2 = container_of(eptr, const struct rename_entry, ent); +return e1->cookie != e2->cookie; +} + +/* +* Register an inotify watch, add watch descriptor to path mapping +* and the reverse mapping. +*/ +static int add_watch (const char *path,struct fsm_listen_data *data) +{ +const char *interned =strintern (path); +struct watch_entry *w1, *w2; + +/* add the inotify watch, don't allow watches to be modified */ +int wd =inotify_add_watch (data->fd_inotify, interned, + (IN_ALL_EVENTS |IN_ONLYDIR |IN_MASK_CREATE ) + ^IN_ACCESS ^IN_CLOSE ^IN_OPEN ); +if (wd <0 ) +return error_errno ("inotify_add_watch('%s') failed" , interned); + +/* add watch descriptor -> directory mapping */ +CALLOC_ARRAY (w1,1 ); + w1->wd = wd; + w1->dir = interned; +hashmap_entry_init (&w1->ent,memhash (&w1->wd,sizeof (int ))); +hashmap_add (&data->watches, &w1->ent); + +/* add directory -> watch descriptor mapping */ +CALLOC_ARRAY (w2,1 ); + w2->wd = wd; + w2->dir = interned; +hashmap_entry_init (&w2->ent,memhash (w2->dir,strlen (w2->dir))); +hashmap_add (&data->revwatches, &w2->ent); + +return 0 ; +} + +/* +* Remove the inotify watch, the watch descriptor to path mapping +* and the reverse mapping. +*/ +static void remove_watch (struct watch_entry *w, +struct fsm_listen_data *data) +{ +struct watch_entry k1, k2, *w1, *w2; + +/* remove watch, ignore error if kernel already did it */ +if (inotify_rm_watch (data->fd_inotify, w->wd) && errno !=EINVAL ) +error_errno ("inotify_rm_watch() failed" ); + +hashmap_entry_init (&k1.ent,memhash (&w->wd,sizeof (int ))); + w1 =hashmap_remove_entry (&data->watches, &k1, ent,NULL ); +if (!w1) +BUG ("Double remove of watch for '%s'" , w->dir); + +if (w1->cookie) +BUG ("Removing watch for '%s' which has a pending rename" , w1->dir); + +hashmap_entry_init (&k2.ent,memhash (w->dir,strlen (w->dir))); + w2 =hashmap_remove_entry (&data->revwatches, &k2, ent,NULL ); +if (!w2) +BUG ("Double remove of reverse watch for '%s'" , w->dir); + +/* w1->dir and w2->dir are interned strings, we don't own them */ +free (w1); +free (w2); +} + +/* +* Check for stale directory renames. +* +* https://man7.org/linux/man-pages/man7/inotify.7.html +* +* Allow for some small timeout to account for the fact that insertion of the +* IN_MOVED_FROM+IN_MOVED_TO event pair is not atomic, and the possibility that +* there may not be any IN_MOVED_TO event. +* +* If the IN_MOVED_TO event is not received within the timeout then events have +* been missed and the monitor is in an inconsistent state with respect to the +* filesystem. +*/ +static int check_stale_dir_renames (struct hashmap *renames,time_t max_age) +{ +struct rename_entry *re; +struct hashmap_iter iter; + +hashmap_for_each_entry (renames, &iter, re, ent) { +if (re->whence <= max_age) +return -1 ; + } +return 0 ; +} + +/* +* Track pending renames. +* +* Tracking is done via a event cookie to watch descriptor mapping. +* +* A rename is not complete until matching a IN_MOVED_TO event is received +* for a corresponding IN_MOVED_FROM event. +*/ +static void add_dir_rename (uint32_t cookie,const char *path, +struct fsm_listen_data *data) +{ +struct watch_entry k, *w; +struct rename_entry *re; + +/* lookup the watch descriptor for the given path */ +hashmap_entry_init (&k.ent,memhash (path,strlen (path))); + w =hashmap_get_entry (&data->revwatches, &k, ent,NULL ); +if (!w)/* should never happen */ +BUG ("No watch for '%s'" , path); + w->cookie = cookie; + +/* add the pending rename to match against later */ +CALLOC_ARRAY (re,1 ); + re->dir = w->dir; + re->cookie = w->cookie; + re->whence =time (NULL ); +hashmap_entry_init (&re->ent,memhash (&re->cookie,sizeof (uint32_t ))); +hashmap_add (&data->renames, &re->ent); +} + +/* +* Handle directory renames +* +* Once a IN_MOVED_TO event is received, lookup the rename tracking information +* via the event cookie and use this information to update the watch. +*/ +static void rename_dir (uint32_t cookie,const char *path, +struct fsm_listen_data *data) +{ +struct rename_entry rek, *re; +struct watch_entry k, *w; + +/* lookup a pending rename to match */ + rek.cookie = cookie; +hashmap_entry_init (&rek.ent,memhash (&rek.cookie,sizeof (uint32_t ))); + re =hashmap_get_entry (&data->renames, &rek, ent,NULL ); +if (re) { + k.dir = re->dir; +hashmap_entry_init (&k.ent,memhash (k.dir,strlen (k.dir))); + w =hashmap_get_entry (&data->revwatches, &k, ent,NULL ); +if (w) { + w->cookie =0 ;/* rename handled */ +remove_watch (w, data); +add_watch (path, data); + }else { +BUG ("No matching watch" ); + } + }else { +BUG ("No matching cookie" ); + } +} + +/* +* Recursively add watches to every directory under path +*/ +static int register_inotify (const char *path, +struct fsmonitor_daemon_state *state, +struct fsmonitor_batch *batch) +{ +DIR *dir; +const char *rel; +struct strbuf current =STRBUF_INIT ; +struct dirent *de; +struct stat fs; +int ret =-1 ; + + dir =opendir (path); +if (!dir) +return error_errno ("opendir('%s') failed" , path); + +while ((de =readdir_skip_dot_and_dotdot (dir)) !=NULL ) { +strbuf_reset (¤t); +strbuf_addf (¤t,"%s/%s" , path, de->d_name); +if (lstat (current.buf, &fs)) { +error_errno ("lstat('%s') failed" , current.buf); + goto failed; + } + +/* recurse into directory */ +if (S_ISDIR (fs.st_mode)) { +if (add_watch (current.buf, state->listen_data)) + goto failed; +if (register_inotify (current.buf, state, batch)) + goto failed; + }else if (batch) { + rel = current.buf + state->path_worktree_watch.len +1 ; +trace_printf_key (&trace_fsmonitor,"explicitly adding '%s'" , rel); +fsmonitor_batch__add_path (batch, rel); + } + } + ret =0 ; + +failed: +strbuf_release (¤t); +if (closedir (dir) <0 ) +return error_errno ("closedir('%s') failed" , path); +return ret; +} + +static int em_rename_dir_from (u_int32_t mask) +{ +return ((mask &IN_ISDIR ) && (mask &IN_MOVED_FROM )); +} + +static int em_rename_dir_to (u_int32_t mask) +{ +return ((mask &IN_ISDIR ) && (mask &IN_MOVED_TO )); +} + +static int em_remove_watch (u_int32_t mask) +{ +return (mask &IN_DELETE_SELF ); +} + +static int em_dir_renamed (u_int32_t mask) +{ +return ((mask &IN_ISDIR ) && (mask &IN_MOVE )); +} + +static int em_dir_created (u_int32_t mask) +{ +return ((mask &IN_ISDIR ) && (mask &IN_CREATE )); +} + +static int em_dir_deleted (uint32_t mask) +{ +return ((mask &IN_ISDIR ) && (mask &IN_DELETE )); +} + +static int em_force_shutdown (u_int32_t mask) +{ +return (mask &IN_UNMOUNT ) || (mask &IN_Q_OVERFLOW ); +} + +static int em_ignore (u_int32_t mask) +{ +return (mask &IN_IGNORED ) || (mask &IN_MOVE_SELF ); +} + +static void log_mask_set (const char *path,u_int32_t mask) +{ +struct strbuf msg =STRBUF_INIT ; + +if (mask &IN_ACCESS ) +strbuf_addstr (&msg,"IN_ACCESS|" ); +if (mask &IN_MODIFY ) +strbuf_addstr (&msg,"IN_MODIFY|" ); +if (mask &IN_ATTRIB ) +strbuf_addstr (&msg,"IN_ATTRIB|" ); +if (mask &IN_CLOSE_WRITE ) +strbuf_addstr (&msg,"IN_CLOSE_WRITE|" ); +if (mask &IN_CLOSE_NOWRITE ) +strbuf_addstr (&msg,"IN_CLOSE_NOWRITE|" ); +if (mask &IN_OPEN ) +strbuf_addstr (&msg,"IN_OPEN|" ); +if (mask &IN_MOVED_FROM ) +strbuf_addstr (&msg,"IN_MOVED_FROM|" ); +if (mask &IN_MOVED_TO ) +strbuf_addstr (&msg,"IN_MOVED_TO|" ); +if (mask &IN_CREATE ) +strbuf_addstr (&msg,"IN_CREATE|" ); +if (mask &IN_DELETE ) +strbuf_addstr (&msg,"IN_DELETE|" ); +if (mask &IN_DELETE_SELF ) +strbuf_addstr (&msg,"IN_DELETE_SELF|" ); +if (mask &IN_MOVE_SELF ) +strbuf_addstr (&msg,"IN_MOVE_SELF|" ); +if (mask &IN_UNMOUNT ) +strbuf_addstr (&msg,"IN_UNMOUNT|" ); +if (mask &IN_Q_OVERFLOW ) +strbuf_addstr (&msg,"IN_Q_OVERFLOW|" ); +if (mask &IN_IGNORED ) +strbuf_addstr (&msg,"IN_IGNORED|" ); +if (mask &IN_ISDIR ) +strbuf_addstr (&msg,"IN_ISDIR|" ); + +trace_printf_key (&trace_fsmonitor,"inotify_event: '%s', mask=%#8.8x %s" , + path, mask, msg.buf); + +strbuf_release (&msg); +} + +int fsm_listen__ctor (struct fsmonitor_daemon_state *state) +{ +int fd; +int ret =0 ; +struct fsm_listen_data *data; + +CALLOC_ARRAY (data,1 ); + state->listen_data = data; + state->listen_error_code =-1 ; + data->shutdown =SHUTDOWN_ERROR ; + + fd =inotify_init1 (O_NONBLOCK ); +if (fd <0 ) +return error_errno ("inotify_init1() failed" ); + + data->fd_inotify = fd; + +hashmap_init (&data->watches, watch_entry_cmp,NULL ,0 ); +hashmap_init (&data->renames, rename_entry_cmp,NULL ,0 ); +hashmap_init (&data->revwatches, revwatches_entry_cmp,NULL ,0 ); + +if (add_watch (state->path_worktree_watch.buf, data)) + ret =-1 ; +else if (register_inotify (state->path_worktree_watch.buf, state,NULL )) + ret =-1 ; +else if (state->nr_paths_watching >1 ) { +if (add_watch (state->path_gitdir_watch.buf, data)) + ret =-1 ; +else if (register_inotify (state->path_gitdir_watch.buf, state,NULL )) + ret =-1 ; + } + +if (!ret) { + state->listen_error_code =0 ; + data->shutdown =SHUTDOWN_CONTINUE ; + } + +return ret; +} + +void fsm_listen__dtor (struct fsmonitor_daemon_state *state) +{ +struct fsm_listen_data *data; +struct hashmap_iter iter; +struct watch_entry *w; +int fd; + +if (!state || !state->listen_data) +return ; + + data = state->listen_data; + fd = data->fd_inotify; + +hashmap_for_each_entry (&data->watches, &iter, w, ent) { + w->cookie =0 ;/* ignore any pending renames */ +remove_watch (w, data); + } +hashmap_clear (&data->watches); + +hashmap_clear (&data->revwatches);/* remove_watch freed the entries */ + + hashmap_clear_and_free(&data->renames, struct rename_entry, ent); + +FREE_AND_NULL (state->listen_data); + +if (fd && (close (fd) <0 )) +error_errno (_ ("closing inotify file descriptor failed" )); +} + +void fsm_listen__stop_async (struct fsmonitor_daemon_state *state) +{ +if (!state->listen_data->shutdown) + state->listen_data->shutdown =SHUTDOWN_STOP ; +} + +/* +* Process a single inotify event and queue for publication. +*/ +static int process_event (const char *path, +const struct inotify_event *event, +struct fsmonitor_batch *batch, +struct string_list *cookie_list, +struct fsmonitor_daemon_state *state) +{ +const char *rel; +const char *last_sep; + +switch (fsmonitor_classify_path_absolute (state, path)) { +case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX : +case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX : +/* Use just the filename of the cookie file. */ + last_sep =find_last_dir_sep (path); +string_list_append (cookie_list, + last_sep ? last_sep +1 : path); +break ; +case IS_INSIDE_DOT_GIT : +case IS_INSIDE_GITDIR : +break ; +case IS_DOT_GIT : +case IS_GITDIR : +/* +* If .git directory is deleted or renamed away, +* we have to quit. +*/ +if (em_dir_deleted (event->mask)) { +trace_printf_key (&trace_fsmonitor, +"event: gitdir removed" ); + state->listen_data->shutdown =SHUTDOWN_FORCE ; + goto done; + } + +if (em_dir_renamed (event->mask)) { +trace_printf_key (&trace_fsmonitor, +"event: gitdir renamed" ); + state->listen_data->shutdown =SHUTDOWN_FORCE ; + goto done; + } +break ; +case IS_WORKDIR_PATH : +/* normal events in the working directory */ +if (trace_pass_fl (&trace_fsmonitor)) +log_mask_set (path, event->mask); + + rel = path + state->path_worktree_watch.len +1 ; +fsmonitor_batch__add_path (batch, rel); + +if (em_dir_deleted (event->mask)) +break ; + +/* received IN_MOVE_FROM, add tracking for expected IN_MOVE_TO */ +if (em_rename_dir_from (event->mask)) +add_dir_rename (event->cookie, path, state->listen_data); + +/* received IN_MOVE_TO, update watch to reflect new path */ +if (em_rename_dir_to (event->mask)) { +rename_dir (event->cookie, path, state->listen_data); +if (register_inotify (path, state, batch)) { + state->listen_data->shutdown =SHUTDOWN_ERROR ; + goto done; + } + } + +if (em_dir_created (event->mask)) { +if (add_watch (path, state->listen_data)) { + state->listen_data->shutdown =SHUTDOWN_ERROR ; + goto done; + } +if (register_inotify (path, state, batch)) { + state->listen_data->shutdown =SHUTDOWN_ERROR ; + goto done; + } + } +break ; +case IS_OUTSIDE_CONE : +default : +trace_printf_key (&trace_fsmonitor, +"ignoring '%s'" , path); +break ; + } +return 0 ; +done: +return -1 ; +} + +/* +* Read the inotify event stream and pre-process events before further +* processing and eventual publishing. +*/ +static void handle_events (struct fsmonitor_daemon_state *state) +{ +/* See https://man7.org/linux/man-pages/man7/inotify.7.html */ + char buf[4096] + __attribute__ ((aligned(__alignof__(struct inotify_event)))); + +struct hashmap watches = state->listen_data->watches; +struct fsmonitor_batch *batch =NULL ; +struct string_list cookie_list =STRING_LIST_INIT_DUP ; +struct watch_entry k, *w; +struct strbuf path; +const struct inotify_event *event; +int fd = state->listen_data->fd_inotify; +ssize_t len; +char *ptr, *p; + +strbuf_init (&path,PATH_MAX ); + +for (;;) { + len =read (fd, buf,sizeof (buf)); +if (len ==-1 && errno !=EAGAIN ) { +error_errno (_ ("reading inotify message stream failed" )); + state->listen_data->shutdown =SHUTDOWN_ERROR ; + goto done; + } + +/* nothing to read */ +if (len <=0 ) + goto done; + +/* Loop over all events in the buffer. */ +for (ptr = buf; ptr < buf + len; + ptr +=sizeof (struct inotify_event ) + event->len) { + + event = (const struct inotify_event *) ptr; + +if (em_ignore (event->mask)) +continue ; + +/* File system was unmounted or event queue overflowed */ +if (em_force_shutdown (event->mask)) { +if (trace_pass_fl (&trace_fsmonitor)) +log_mask_set ("Forcing shutdown" , event->mask); + state->listen_data->shutdown =SHUTDOWN_FORCE ; + goto done; + } + +hashmap_entry_init (&k.ent,memhash (&event->wd,sizeof (int ))); + k.wd = event->wd; + + w =hashmap_get_entry (&watches, &k, ent,NULL ); +if (!w)/* should never happen */ +BUG ("No watch for '%s'" , event->name); + +/* directory watch was removed */ +if (em_remove_watch (event->mask)) { +remove_watch (w, state->listen_data); +continue ; + } + +strbuf_reset (&path); +strbuf_add (&path, w->dir,strlen (w->dir)); +strbuf_addch (&path,'/' ); +strbuf_addstr (&path, event->name); + + p =fsmonitor__resolve_alias (path.buf, &state->alias); +if (!p) + p =strbuf_detach (&path,NULL ); + +if (!batch) + batch =fsmonitor_batch__new (); + +if (process_event (p, event, batch, &cookie_list, state)) { +free (p); + goto done; + } +free (p); + } +strbuf_reset (&path); +fsmonitor_publish (state, batch, &cookie_list); +string_list_clear (&cookie_list,0 ); + batch =NULL ; + } +done: +strbuf_release (&path); +fsmonitor_batch__free_list (batch); +string_list_clear (&cookie_list,0 ); +} + +/* +* Non-blocking read of the inotify events stream. The inotify fd is polled +* frequently to help minimize the number of queue overflows. +*/ +void fsm_listen__loop (struct fsmonitor_daemon_state *state) +{ +int poll_num; +const int interval =1000 ; +time_t checked =time (NULL ); +struct pollfd fds[1 ]; + fds[0 ].fd = state->listen_data->fd_inotify; + fds[0 ].events =POLLIN ; + +for (;;) { +switch (state->listen_data->shutdown) { +case SHUTDOWN_CONTINUE : + poll_num =poll (fds,1 ,1 ); +if (poll_num ==-1 ) { +if (errno ==EINTR ) +continue ; +error_errno (_ ("polling inotify message stream failed" )); + state->listen_data->shutdown =SHUTDOWN_ERROR ; +continue ; + } + +if ((time (NULL ) - checked) >= interval) { + checked =time (NULL ); +if (check_stale_dir_renames (&state->listen_data->renames, + checked - interval)) { +trace_printf_key (&trace_fsmonitor, +"Missed IN_MOVED_TO events, forcing shutdown" ); + state->listen_data->shutdown =SHUTDOWN_FORCE ; +continue ; + } + } + +if (poll_num >0 && (fds[0 ].revents &POLLIN )) +handle_events (state); + +continue ; +case SHUTDOWN_ERROR : + state->listen_error_code =-1 ; +ipc_server_stop_async (state->ipc_server_data); +break ; +case SHUTDOWN_FORCE : + state->listen_error_code =0 ; +ipc_server_stop_async (state->ipc_server_data); +break ; +case SHUTDOWN_STOP : +default : + state->listen_error_code =0 ; +break ; + } +return ; + } +}
Signature · 1 line
gitgitgadget
PATCH v4 4/6 fsmonitor: enable fsmonitor for Linux
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
Update build to enable fsmonitor for Linux.
--- config.mak.uname | 8 ++++++++ contrib/buildsystems/CMakeLists.txt | 11 ++++++++++- 2 files changed, 18 insertions(+), 1 deletion(-)
config.mak.uname +8 -0
@@ -68,6 +68,14 @@ ifeq ($(uname_S),Linux) ifneq ($(findstring .el7.,$(uname_R)),) BASIC_CFLAGS += -std=c99 endif + # The builtin FSMonitor on Linux builds upon Simple-IPC. Both require + # Unix domain sockets and PThreads. + ifndef NO_PTHREADS + ifndef NO_UNIX_SOCKETS + FSMONITOR_DAEMON_BACKEND = linux + FSMONITOR_OS_SETTINGS = linux + endif + endif endif ifeq ($(uname_S),GNU/kFreeBSD) HAVE_ALLOCA_H = YesPlease
contrib/buildsystems/CMakeLists.txt +10 -1
@@ -304,7 +304,16 @@ else() endif() if(SUPPORTS_SIMPLE_IPC) - if(CMAKE_SYSTEM_NAME STREQUAL "Windows") + if(CMAKE_SYSTEM_NAME STREQUAL "Linux") + add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-linux.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-linux.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-linux.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-linux.c) + + add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-linux.c) + elseif(CMAKE_SYSTEM_NAME STREQUAL "Windows") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c) list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c)
Signature · 1 line
gitgitgadget
PATCH v4 5/6 fsmonitor: test updates
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
t7527-builtin-fsmonitor was leaking fsmonitor--daemon processes in some cases.
Accomodate slight difference in the number of events generated on Linux.
On lower-powered systems, spin a little to give the daemon time to respond to and log filesystem events.
--- t/t7527-builtin-fsmonitor.sh | 94 ++++++++++++++++++++++++++++++------ 1 file changed, 80 insertions(+), 14 deletions(-)
t/t7527-builtin-fsmonitor.sh +80 -14
@@ -13,7 +13,7 @@ fi stop_daemon_delete_repo () { r=$1 && test_might_fail git -C $r fsmonitor--daemon stop && - rm -rf $1 + rm -rf $r } start_daemon () {
@@ -72,6 +72,34 @@ start_daemon () { ) } +IMPLICIT_TIMEOUT=5 + +wait_for_update () { + func=$1 && + file=$2 && + sz=$(wc -c < "$file") && + last=0 && + $func && + k=0 && + while test "$k" -lt $IMPLICIT_TIMEOUT + do + nsz=$(wc -c < "$file") + if test "$nsz" -gt "$sz" + then + if test "$last" -eq "$nsz" + then + cat "$file" && + return 0 + fi + last=$nsz + fi + sleep 1 + k=$(( $k + 1 )) + done && + cat "$file" && + return 0 +} + # Is a Trace2 data event present with the given catetory and key? # We do not care what the value is. #
@@ -137,7 +165,6 @@ test_expect_success 'implicit daemon start' ' # machines (where it might take a moment to wake and reschedule the # daemon process) to avoid false alarms during test runs.) # -IMPLICIT_TIMEOUT=5 verify_implicit_shutdown () { r=$1 &&
@@ -373,6 +400,15 @@ create_files () { echo 3 >dir2/new } +rename_directory () { + mv dirtorename dirrenamed +} + +rename_directory_file () { + mv dirtorename dirrenamed && + echo 1 > dirrenamed/new +} + rename_files () { mv rename renamed && mv dir1/rename dir1/renamed &&
@@ -427,10 +463,12 @@ test_expect_success 'edit some files' ' start_daemon --tf "$PWD/.git/trace" && - edit_files && + wait_for_update edit_files "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 && + test_might_fail git fsmonitor--daemon stop && + grep "^event: dir1/modified$" .git/trace && grep "^event: dir2/modified$" .git/trace && grep "^event: modified$" .git/trace &&
@@ -442,10 +480,12 @@ test_expect_success 'create some files' ' start_daemon --tf "$PWD/.git/trace" && - create_files && + wait_for_update create_files "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 && + test_might_fail git fsmonitor--daemon stop && + grep "^event: dir1/new$" .git/trace && grep "^event: dir2/new$" .git/trace && grep "^event: new$" .git/trace
@@ -456,10 +496,12 @@ test_expect_success 'delete some files' ' start_daemon --tf "$PWD/.git/trace" && - delete_files && + wait_for_update delete_files "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 && + test_might_fail git fsmonitor--daemon stop && + grep "^event: dir1/delete$" .git/trace && grep "^event: dir2/delete$" .git/trace && grep "^event: delete$" .git/trace
@@ -470,10 +512,12 @@ test_expect_success 'rename some files' ' start_daemon --tf "$PWD/.git/trace" && - rename_files && + wait_for_update rename_files "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 && + test_might_fail git fsmonitor--daemon stop && + grep "^event: dir1/rename$" .git/trace && grep "^event: dir2/rename$" .git/trace && grep "^event: rename$" .git/trace &&
@@ -487,23 +531,42 @@ test_expect_success 'rename directory' ' start_daemon --tf "$PWD/.git/trace" && - mv dirtorename dirrenamed && + wait_for_update rename_directory "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 && + test_might_fail git fsmonitor--daemon stop && + grep "^event: dirtorename/*$" .git/trace && grep "^event: dirrenamed/*$" .git/trace ' +test_expect_success 'rename directory file' ' + test_when_finished clean_up_repo_and_stop_daemon && + + start_daemon --tf "$PWD/.git/trace" && + + wait_for_update rename_directory_file "$PWD/.git/trace" && + + test-tool fsmonitor-client query --token 0 && + + test_might_fail git fsmonitor--daemon stop && + + grep "^event: dirtorename/*$" .git/trace && + grep "^event: dirrenamed/*$" .git/trace && + grep "^event: dirrenamed/new$" .git/trace +' test_expect_success 'file changes to directory' ' test_when_finished clean_up_repo_and_stop_daemon && start_daemon --tf "$PWD/.git/trace" && - file_to_directory && + wait_for_update file_to_directory "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 && + test_might_fail git fsmonitor--daemon stop && + grep "^event: delete$" .git/trace && grep "^event: delete/new$" .git/trace '
@@ -513,10 +576,12 @@ test_expect_success 'directory changes to a file' ' start_daemon --tf "$PWD/.git/trace" && - directory_to_file && + wait_for_update directory_to_file "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 && + test_might_fail git fsmonitor--daemon stop && + grep "^event: dir1$" .git/trace '
@@ -561,7 +626,7 @@ test_expect_success 'flush cached data' ' test-tool -C test_flush fsmonitor-client query --token "builtin:test_00000002:0" >actual_2 && nul_to_q <actual_2 >actual_q2 && - grep "^builtin:test_00000002:0Q$" actual_q2 && + grep "^builtin:test_00000002:[0-1]Q$" actual_q2 && >test_flush/file_3 &&
@@ -732,7 +797,8 @@ u_values="$u1 $u2" for u in $u_values do test_expect_success "unicode in repo root path: $u" ' - test_when_finished "stop_daemon_delete_repo $u" && + test_when_finished \ + "stop_daemon_delete_repo `echo "$u" | sed 's:x:\\\\\\\\\\\\\\x:g'`" && git init "$u" && echo 1 >"$u"/file1 &&
@@ -818,8 +884,7 @@ test_expect_success 'submodule setup' ' ' test_expect_success 'submodule always visited' ' - test_when_finished "git -C super fsmonitor--daemon stop; \ - rm -rf super; \ + test_when_finished "rm -rf super; \ rm -rf sub" && create_super super &&
@@ -887,7 +952,8 @@ have_t2_error_event () { } test_expect_success "stray submodule super-prefix warning" ' - test_when_finished "rm -rf super; \ + test_when_finished "git -C super/dir_1/dir_2/sub fsmonitor--daemon stop; \ + rm -rf super; \ rm -rf sub; \ rm super-sub.trace" &&
Signature · 1 line
gitgitgadget
PATCH v4 6/6 fsmonitor: update doc for Linux
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
Update the documentation for Linux.
--- Documentation/config/fsmonitor--daemon.txt | 4 ++-- Documentation/git-fsmonitor--daemon.txt | 24 ++++++++++++++-------- 2 files changed, 18 insertions(+), 10 deletions(-)
Documentation/config/fsmonitor--daemon.txt +2 -2
@@ -4,8 +4,8 @@ fsmonitor.allowRemote:: behavior. Only respected when `core.fsmonitor` is set to `true`. fsmonitor.socketDir:: - This Mac OS-specific option, if set, specifies the directory in + Mac OS and Linux-specific option. If set, specifies the directory in which to create the Unix domain socket used for communication between the fsmonitor daemon and various Git commands. The directory must - reside on a native Mac OS filesystem. Only respected when `core.fsmonitor` + reside on a native filesystem. Only respected when `core.fsmonitor` is set to `true`.
Documentation/git-fsmonitor--daemon.txt +16 -8
@@ -76,23 +76,31 @@ repositories; this may be overridden by setting `fsmonitor.allowRemote` to correctly with all network-mounted repositories and such use is considered experimental. -On Mac OS, the inter-process communication (IPC) between various Git +On Linux and Mac OS, the inter-process communication (IPC) between various Git commands and the fsmonitor daemon is done via a Unix domain socket (UDS) -- a -special type of file -- which is supported by native Mac OS filesystems, -but not on network-mounted filesystems, NTFS, or FAT32. Other filesystems -may or may not have the needed support; the fsmonitor daemon is not guaranteed -to work with these filesystems and such use is considered experimental. +special type of file -- which is supported by many native Linux and Mac OS +filesystems, but not on network-mounted filesystems, NTFS, or FAT32. Other +filesystems may or may not have the needed support; the fsmonitor daemon is not +guaranteed to work with these filesystems and such use is considered +experimental. By default, the socket is created in the `.git` directory, however, if the `.git` directory is on a network-mounted filesystem, it will be instead be created at `$HOME/.git-fsmonitor-*` unless `$HOME` itself is on a network-mounted filesystem in which case you must set the configuration -variable `fsmonitor.socketDir` to the path of a directory on a Mac OS native +variable `fsmonitor.socketDir` to the path of a directory on a native filesystem in which to create the socket file. If none of the above directories (`.git`, `$HOME`, or `fsmonitor.socketDir`) -is on a native Mac OS file filesystem the fsmonitor daemon will report an -error that will cause the daemon and the currently running command to exit. +is on a native Linux or Mac OS filesystem the fsmonitor daemon will report +an error that will cause the daemon to exit and the currently running command +to issue a warning. + +On Linux, the fsmonitor daemon registers a watch for each directory in the +repository. The default per-user limit for the number of watches on most Linux +systems is 8192. This may not be sufficient for large repositories or if +multiple instances of the fsmonitor daemon are running. +See https://watchexec.github.io/docs/inotify-limits.html[Linux inotify limits] for more information. CONFIGURATION -------------
Signature · 1 line
gitgitgadget
Re: [PATCH v4 2/6] fsmonitor: determine if filesystem is local or remote
In reply to Eric DeCosta via GitGitGadgetQuoted 1 line · Eric DeCosta via GitGitGadget
+static int
is_remote_fs(const char* path){
Style (asterisk sticks to the variable not to the type).
compat/fsmonitor/fsm-path-utils-linux.c line 42
Quoted 8 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +7
-0+switch (fs.f_type) { +case 0x61636673 :/* ACFS */ +case 0xA501FCF5 :/* VXFS */ +return 1 ; +default : +break ; + }
Align case/default to switch by de-denting one level, just like you did the switch() statement in find_mount().
compat/fsmonitor/fsm-path-utils-linux.c line 51
Quoted 5 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +5
-0+static int find_mount (const char *path,const struct statvfs *fs, +struct mntent *ent) +{ +const char *const mounts ="/proc/mounts" ; +const char *rp =real_pathdup (path,1 );
Nobody seems to free() this once we are done with this function.
compat/fsmonitor/fsm-path-utils-linux.c line 60
Quoted 9 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +9
-0+struct mntent *ment =NULL ; +struct statvfs mntfs; +FILE *fp; +int found =0 ; +int dlen, plen, flen =0 ; + + ent->mnt_fsname =NULL ; + ent->mnt_dir =NULL ; + ent->mnt_type =NULL ;
Quoted 40 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +40
-0+ fp =setmntent (mounts,"r" ); +if (!fp) { +error_errno (_ ("setmntent('%s') failed" ), mounts); +return -1 ; + } + + plen =strlen (rp); + +/* read all the mount information and compare to path */ +while ((ment =getmntent (fp)) !=NULL ) { +if (statvfs (ment->mnt_dir, &mntfs)) { +switch (errno) { +case EPERM : +case ESRCH : +case EACCES : +continue ; +default : +error_errno (_ ("statvfs('%s') failed" ), ment->mnt_dir); +endmntent (fp); +return -1 ; + } + } + +/* is mount on the same filesystem and is a prefix of the path */ +if ((fs->f_fsid == mntfs.f_fsid) && + !strncmp (ment->mnt_dir, rp,strlen (ment->mnt_dir))) { + dlen =strlen (ment->mnt_dir); +if (dlen > plen) +continue ; +/* +* root is always a potential match; otherwise look for +* directory prefix +*/ +if ((dlen ==1 && ment->mnt_dir[0 ] =='/' ) || + (dlen > flen && (!rp[dlen] || rp[dlen] =='/' ))) { + flen = dlen; +/* +* https://man7.org/linux/man-pages/man3/getmntent.3.html +* +* The pointer points to a static area of memory which is
compat/fsmonitor/fsm-path-utils-linux.c line 111
Quoted 10 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +10
-0+* overwritten by subsequent calls to getmntent(). +*/ + found =1 ; +free (ent->mnt_fsname); +free (ent->mnt_dir); +free (ent->mnt_type); + ent->mnt_fsname =xstrdup (ment->mnt_fsname); + ent->mnt_dir =xstrdup (ment->mnt_dir); + ent->mnt_type =xstrdup (ment->mnt_type); + }
So more than one mount entries could match the given path. This loop implements "the last one wins", but is that a sensible thing to do? Shouldn't it be more like "the longest one, being the most specific, wins"? If /usr mounts on / and /usr/local mounts on /usr, asking for /usr/local/me would want to discover that it is on the filesystem mounted at /usr/local regardless of the order in which getmntent() returns the entries, no?
Quoted 15 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +13
-0+ } + } +endmntent (fp); + +if (!found) +return -1 ; + +return 0 ; +} + +int fsmonitor__get_fs_info (const char *path,struct fs_info *fs_info) +{ +struct mntent ment;
compat/fsmonitor/fsm-path-utils-linux.c line 134
Quoted 10 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +10
-0+struct statvfs fs; + +if (statvfs (path, &fs)) +return error_errno (_ ("statvfs('%s') failed" ), path); + + +if (find_mount (path, &fs, &ment) <0 ) { +free (ment.mnt_fsname); +free (ment.mnt_dir); +free (ment.mnt_type);
It is a good idea to free, but I _think_ the code is easier to follow if you _clear_ ment before calling find_mount(), not in find_mount().
Quoted 5 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +5
-0+return -1 ; + } + + trace_printf_key(&trace_fsmonitor, + "statvfs('%s') [flags 0x%08lx] '%s' '%s'",
compat/fsmonitor/fsm-path-utils-linux.c line 149
Quoted 10 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +10
-0+ path, fs.f_flag, ment.mnt_type, ment.mnt_fsname); + + fs_info->is_remote =is_remote_fs (ment.mnt_dir); + fs_info->typename = ment.mnt_fsname; +free (ment.mnt_dir); +free (ment.mnt_type); + +if (fs_info->is_remote <0 ) { +free (ment.mnt_fsname); +return -1 ;
fs_info->typename just started to point into an already freed memory at this point, which is a very safe thing to do to the caller of this function.
Rather, perhaps delay the setting of typename after this statement, which would be more friendly to the caller than telling them that they are not allowed to touch the member when the function returns negative.
Quoted 37 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +37
-0+ } + +trace_printf_key (&trace_fsmonitor, +"'%s' is_remote: %d" , + path, fs_info->is_remote); + +return 0 ; +} + +int fsmonitor__is_fs_remote (const char *path) +{ +struct fs_info fs; + +if (fsmonitor__get_fs_info (path, &fs)) +return -1 ; + +free (fs.typename); + +return fs.is_remote; +} + +/* +* No-op for now. +*/ +int fsmonitor__get_alias (const char *path,struct alias_info *info) +{ +return 0 ; +} + +/* +* No-op for now. +*/ +char *fsmonitor__resolve_alias (const char *path, +const struct alias_info *info) +{ +return NULL ; +}
Re: [PATCH v4 2/6] fsmonitor: determine if filesystem is local or remote
In reply to Eric DeCosta via GitGitGadgetQuoted 23 lines · Eric DeCosta via GitGitGadget
From: Eric DeCosta <[email protected]>
Compare the given path to the mounted filesystems. Find the mount that is the longest prefix of the path (if any) and determine if that mount is on a local or remote filesystem.
Signed-off-by Eric DeCosta <[email protected]>--- compat/fsmonitor/fsm-path-utils-linux.c | 186 ++++++++++++++++++++++++ 1 file changed, 186 insertions(+) create mode 100644 compat/fsmonitor/fsm-path-utils-linux.ccompat/fsmonitor/fsm-path-utils-linux.c new file +5
-0@@ -0,0 +1,186 @@ +#include "fsmonitor.h" +#include "fsmonitor-path-utils.h" +#include <errno.h> +#include <mntent.h> +#include <sys/mount.h>
compat/fsmonitor/fsm-path-utils-linux.c line 15
Quoted 10 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +10
-0+#include <sys/vfs.h> +#include <sys/statvfs.h> + +static int is_remote_fs (const char * path) { +struct statfs fs; + +if (statfs (path, &fs)) { +error_errno (_ ("statfs('%s') failed" ), path); +return -1 ; + }
Nit: Drop the braces and do:
if (statfs(...) == -1) return error_errno(...)
Quoted 12 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +12
-0+switch (fs.f_type) { +case 0x61636673 :/* ACFS */ +case 0x5346414F :/* AFS */ +case 0x00C36400 :/* CEPH */ +case 0xFF534D42 :/* CIFS */ +case 0x73757245 :/* CODA */ +case 0x19830326 :/* FHGFS */ +case 0x1161970 :/* GFS */ +case 0x47504653 :/* GPFS */ +case 0x013111A8 :/* IBRIX */ +case 0x6B414653 :/* KAFS */ +case 0x0BD00BD0 :/* LUSTRE */
compat/fsmonitor/fsm-path-utils-linux.c line 38
Quoted 10 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +10
-0+case 0x564C :/* NCP */ +case 0x6969 :/* NFS */ +case 0x6E667364 :/* NFSD */ +case 0x7461636f :/* OCFS2 */ +case 0xAAD7AAEA :/* PANFS */ +case 0x517B :/* SMB */ +case 0xBEEFDEAD :/* SNFS */ +case 0xFE534D42 :/* SMB2 */ +case 0xBACBACBC :/* VMHGFS */ +case 0xA501FCF5 :/* VXFS */
So, before we'd compare against the name, but to avoid the GPLv3 copy/pasting we're now comparing against the fs.f_type.
If we are hardcoding them, our usual convention is to lower-case hexdigits, so 0xbacbacbc not 0xBACBACBC.
But at least my statfs() manpage documents the named defines in linux/magic.h for most of these. Why not use those?
compat/fsmonitor/fsm-path-utils-linux.c line 41
Quoted 3 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +3
-0+return 1 ; +default : +break ;
You could just "return 0" here, and...
compat/fsmonitor/fsm-path-utils-linux.c line 44
Quoted 3 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +3
-0+ } + +return 0 ;
...drop this "return 0".
compat/fsmonitor/fsm-path-utils-linux.c line 48
Quoted 4 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +4
-0+} + +static int find_mount (const char *path,const struct statvfs *fs, +struct mntent *ent)
Misindentation.
Quoted 7 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +7
-0+{ +const char *const mounts ="/proc/mounts" ; +const char *rp =real_pathdup (path,1 ); +struct mntent *ment =NULL ; +struct statvfs mntfs; +FILE *fp; +int found =0 ;
compat/fsmonitor/fsm-path-utils-linux.c line 65
Quoted 10 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +10
-0+int dlen, plen, flen =0 ; + + ent->mnt_fsname =NULL ; + ent->mnt_dir =NULL ; + ent->mnt_type =NULL ; + + fp =setmntent (mounts,"r" ); +if (!fp) { +error_errno (_ ("setmntent('%s') failed" ), mounts); +return -1 ;
Ditto "return error_errno()"
compat/fsmonitor/fsm-path-utils-linux.c line 68
Quoted 3 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +3
-0+ } + + plen =strlen (rp);
Let's make "plen", "dlen" and "flen" a "size_t", not "int"
compat/fsmonitor/fsm-path-utils-linux.c line 71
Quoted 3 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +3
-0+ +/* read all the mount information and compare to path */ +while ((ment =getmntent (fp)) !=NULL ) {
Drop the "!= NULL"
compat/fsmonitor/fsm-path-utils-linux.c line 80
Quoted 9 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +9
-0+if (statvfs (ment->mnt_dir, &mntfs)) { +switch (errno) { +case EPERM : +case ESRCH : +case EACCES : +continue ; +default : +error_errno (_ ("statvfs('%s') failed" ), ment->mnt_dir); +endmntent (fp);
Shouldn't we check the endmntent() error too? Now, from the manpage the interface is funny, and always returns 1.
But since this is linux-specific code it seems safe enough to go with it & glibc assumptions and:
errno = 0;
endmntent(fp);
if (errno)
return error_errno(....);I.e. it'll just call fclose(), which might set errno() on failure.
Maybe it's not worth it...
compat/fsmonitor/fsm-path-utils-linux.c line 128
Quoted 2 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +2
-0+if (statvfs (path, &fs)) +return error_errno (_ ("statvfs('%s') failed" ), path);
Here you do use that "return error_errno(...)" pattern...", yay!
Quoted 7 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +7
-0+ + +if (find_mount (path, &fs, &ment) <0 ) { +free (ment.mnt_fsname); +free (ment.mnt_dir); +free (ment.mnt_type); +return -1 ;
compat/fsmonitor/fsm-path-utils-linux.c line 145
Quoted 10 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +10
-0+ } + +trace_printf_key (&trace_fsmonitor, +"statvfs('%s') [flags 0x%08lx] '%s' '%s'" , + path, fs.f_flag, ment.mnt_type, ment.mnt_fsname); + + fs_info->is_remote =is_remote_fs (ment.mnt_dir); + fs_info->typename = ment.mnt_fsname; +free (ment.mnt_dir); +free (ment.mnt_type);
If you're going to \n\n-seperate this and the trace_printf_key() above I think moving the second free() here to that "block" would make sense, sinec here is the last time we use mnt_dir, but the last time we used mnt_type was in the trace_printf_key().
But...
compat/fsmonitor/fsm-path-utils-linux.c line 148
Quoted 3 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +3
-0+ +if (fs_info->is_remote <0 ) { +free (ment.mnt_fsname);
...aren't you NULL init-ing these, why not just for all of these:
goto error;
Then....
compat/fsmonitor/fsm-path-utils-linux.c line 156
Quoted 8 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +8
-0+return -1 ; + } + +trace_printf_key (&trace_fsmonitor, +"'%s' is_remote: %d" , + path, fs_info->is_remote); + +return 0 ;
Have this be:
int ret = -1; /* earlier */ ret = 0; cleanup: free(...); free(...); return ret;
compat/fsmonitor/fsm-path-utils-linux.c line 166
Quoted 10 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +10
-0+} + +int fsmonitor__is_fs_remote (const char *path) +{ +struct fs_info fs; + +if (fsmonitor__get_fs_info (path, &fs)) +return -1 ; + +free (fs.typename);
This will segfault if you take the part through fsmonitor__get_fs_info() where we don't have the fs.typename yet, i.e. if statfs() fails.
There's the trivial NULL-init way to work around it, but I think this suggests a leaky abstraction. If we fail to get the fs info, then the function itself should have free'd that, shouldn't it?
compat/fsmonitor/fsm-path-utils-linux.c line 183
Quoted 5 lines
compat/fsmonitor/fsm-path-utils-linux.c new file +5
-0+/* +* No-op for now. +*/ +char *fsmonitor__resolve_alias (const char *path, +const struct alias_info *info)
Ditto misindentatione
Re: [PATCH v4 3/6] fsmonitor: implement filesystem change listener for Linux
In reply to Eric DeCosta via GitGitGadgetQuoted 6 lines · Eric DeCosta via GitGitGadget
From: Eric DeCosta <[email protected]>
[...]compat/fsmonitor/fsm-listen-linux.c new file +4
-0+} + +/* +* Remove the inotify watch, the watch descriptor to path mapping
compat/fsmonitor/fsm-listen-linux.c line 127
Quoted 10 lines
compat/fsmonitor/fsm-listen-linux.c new file +10
-0+* and the reverse mapping. +*/ +static void remove_watch (struct watch_entry *w, +struct fsm_listen_data *data) +{ +struct watch_entry k1, k2, *w1, *w2; + +/* remove watch, ignore error if kernel already did it */ +if (inotify_rm_watch (data->fd_inotify, w->wd) && errno !=EINVAL ) +error_errno ("inotify_rm_watch() failed" );
If we error_errno(), shouldn't this function have a return value?
compat/fsmonitor/fsm-listen-linux.c line 140
Quoted 13 lines
compat/fsmonitor/fsm-listen-linux.c new file +13
-0+ +hashmap_entry_init (&k1.ent,memhash (&w->wd,sizeof (int ))); + w1 =hashmap_remove_entry (&data->watches, &k1, ent,NULL ); +if (!w1) +BUG ("Double remove of watch for '%s'" , w->dir); + +if (w1->cookie) +BUG ("Removing watch for '%s' which has a pending rename" , w1->dir); + +hashmap_entry_init (&k2.ent,memhash (w->dir,strlen (w->dir))); + w2 =hashmap_remove_entry (&data->revwatches, &k2, ent,NULL ); +if (!w2) +BUG ("Double remove of reverse watch for '%s'" , w->dir);
For the BUG() additions: Start with lower-case in messages, see CodingGuidelines. I.e. "double remove of ..." etc. Ditto below..
Quoted 15 lines
[...]
compat/fsmonitor/fsm-listen-linux.c new file +14
-0+/* +* Handle directory renames +* +* Once a IN_MOVED_TO event is received, lookup the rename tracking information +* via the event cookie and use this information to update the watch. +*/ +static void rename_dir (uint32_t cookie,const char *path, +struct fsm_listen_data *data) +{ +struct rename_entry rek, *re; +struct watch_entry k, *w; + +/* lookup a pending rename to match */ + rek.cookie = cookie;
compat/fsmonitor/fsm-listen-linux.c line 225
Quoted 10 lines
compat/fsmonitor/fsm-listen-linux.c new file +10
-0+hashmap_entry_init (&rek.ent,memhash (&rek.cookie,sizeof (uint32_t ))); + re =hashmap_get_entry (&data->renames, &rek, ent,NULL ); +if (re) { + k.dir = re->dir; +hashmap_entry_init (&k.ent,memhash (k.dir,strlen (k.dir))); + w =hashmap_get_entry (&data->revwatches, &k, ent,NULL ); +if (w) { + w->cookie =0 ;/* rename handled */ +remove_watch (w, data); +add_watch (path, data);
Elsewhere you check the add_watch() return value, but not here?
Quoted 17 lines
compat/fsmonitor/fsm-listen-linux.c new file +17
-0+ }else { +BUG ("No matching watch" ); + } + }else { +BUG ("No matching cookie" ); + } +} + +/* +* Recursively add watches to every directory under path +*/ +static int register_inotify (const char *path, +struct fsmonitor_daemon_state *state, +struct fsmonitor_batch *batch) +{ +DIR *dir; +const char *rel;
compat/fsmonitor/fsm-listen-linux.c line 252
Quoted 10 lines
compat/fsmonitor/fsm-listen-linux.c new file +10
-0+struct strbuf current =STRBUF_INIT ; +struct dirent *de; +struct stat fs; +int ret =-1 ; + + dir =opendir (path); +if (!dir) +return error_errno ("opendir('%s') failed" , path); + +while ((de =readdir_skip_dot_and_dotdot (dir)) !=NULL ) {
ditto drop the "!= NULL"
Quoted 24 lines
compat/fsmonitor/fsm-listen-linux.c new file +24
-0+struct strbuf msg =STRBUF_INIT ; + +if (mask &IN_ACCESS ) +strbuf_addstr (&msg,"IN_ACCESS|" ); +if (mask &IN_MODIFY ) +strbuf_addstr (&msg,"IN_MODIFY|" ); +if (mask &IN_ATTRIB ) +strbuf_addstr (&msg,"IN_ATTRIB|" ); +if (mask &IN_CLOSE_WRITE ) +strbuf_addstr (&msg,"IN_CLOSE_WRITE|" ); +if (mask &IN_CLOSE_NOWRITE ) +strbuf_addstr (&msg,"IN_CLOSE_NOWRITE|" ); +if (mask &IN_OPEN ) +strbuf_addstr (&msg,"IN_OPEN|" ); +if (mask &IN_MOVED_FROM ) +strbuf_addstr (&msg,"IN_MOVED_FROM|" ); +if (mask &IN_MOVED_TO ) +strbuf_addstr (&msg,"IN_MOVED_TO|" ); +if (mask &IN_CREATE ) +strbuf_addstr (&msg,"IN_CREATE|" ); +if (mask &IN_DELETE ) +strbuf_addstr (&msg,"IN_DELETE|" ); +if (mask &IN_DELETE_SELF ) +strbuf_addstr (&msg,"IN_DELETE_SELF|" );
compat/fsmonitor/fsm-listen-linux.c line 356
Quoted 10 lines
compat/fsmonitor/fsm-listen-linux.c new file +10
-0+if (mask &IN_MOVE_SELF ) +strbuf_addstr (&msg,"IN_MOVE_SELF|" ); +if (mask &IN_UNMOUNT ) +strbuf_addstr (&msg,"IN_UNMOUNT|" ); +if (mask &IN_Q_OVERFLOW ) +strbuf_addstr (&msg,"IN_Q_OVERFLOW|" ); +if (mask &IN_IGNORED ) +strbuf_addstr (&msg,"IN_IGNORED|" ); +if (mask &IN_ISDIR ) +strbuf_addstr (&msg,"IN_ISDIR|" );
Shouldn't the very last addition to mask omit the "|"?
I think it would be worth just making this a NULL-delimited "int, const char *" array like:
{
{ IN_ACCESS, "IN_ACCESS" },
...
NULL,
};Then looping over it, or maybe not...
Quoted 10 lines
compat/fsmonitor/fsm-listen-linux.c new file +10
-0+trace_printf_key (&trace_fsmonitor,"inotify_event: '%s', mask=%#8.8x %s" , + path, mask, msg.buf); + +strbuf_release (&msg); +} + +int fsm_listen__ctor (struct fsmonitor_daemon_state *state) +{ +int fd; +int ret =0 ;
compat/fsmonitor/fsm-listen-linux.c line 377
Quoted 10 lines
compat/fsmonitor/fsm-listen-linux.c new file +10
-0+struct fsm_listen_data *data; + +CALLOC_ARRAY (data,1 ); + state->listen_data = data; + state->listen_error_code =-1 ; + data->shutdown =SHUTDOWN_ERROR ; + + fd =inotify_init1 (O_NONBLOCK ); +if (fd <0 ) +return error_errno ("inotify_init1() failed" );
Here you leak the "data" you just allocated on error.
compat/fsmonitor/fsm-listen-linux.c line 385
Quoted 8 lines
compat/fsmonitor/fsm-listen-linux.c new file +8
-0+ + data->fd_inotify = fd; + +hashmap_init (&data->watches, watch_entry_cmp,NULL ,0 ); +hashmap_init (&data->renames, rename_entry_cmp,NULL ,0 ); +hashmap_init (&data->revwatches, revwatches_entry_cmp,NULL ,0 ); + +if (add_watch (state->path_worktree_watch.buf, data))
I.e. we should only avoid free()-ing it if we can successfully hand it over to add_watch(), shouldn't we?n
compat/fsmonitor/fsm-listen-linux.c line 388
Quoted 3 lines
compat/fsmonitor/fsm-listen-linux.c new file +3
-0+ ret =-1 ; + else if (register_inotify(state->path_worktree_watch.buf, state, NULL)) + ret =-1 ;
We add {}'s to all if/else if branches if one needs it, see CodingGuidelines.
compat/fsmonitor/fsm-listen-linux.c line 393
Quoted 5 lines
compat/fsmonitor/fsm-listen-linux.c new file +5
-0+else if (state->nr_paths_watching >1 ) { +if (add_watch (state->path_gitdir_watch.buf, data)) + ret =-1 ; +else if (register_inotify (state->path_gitdir_watch.buf, state,NULL )) + ret =-1 ;
Can't this be:
else if (state->nr_paths_watching > 1 && (add_watch(...) || register_inotify(...))) ret = -1;
compat/fsmonitor/fsm-listen-linux.c line 452
Quoted 2 lines
compat/fsmonitor/fsm-listen-linux.c new file +2
-0+switch (fsmonitor_classify_path_absolute (state, path)) { +case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX :
Don't indent the "case" label here more than the "switch", see CodingGuidelines.
compat/fsmonitor/fsm-listen-linux.c line 534
Quoted 3 lines
compat/fsmonitor/fsm-listen-linux.c new file +3
-0+static void handle_events (struct fsmonitor_daemon_state *state) +{ +/* See https://man7.org/linux/man-pages/man7/inotify.7.html */
Extra " " here after the "\t".
Quoted 4 lines
compat/fsmonitor/fsm-listen-linux.c new file +4
-0+ char buf[4096] + __attribute__ ((aligned(__alignof__(struct inotify_event)))); + +struct hashmap watches = state->listen_data->watches;
compat/fsmonitor/fsm-listen-linux.c line 548
Quoted 10 lines
compat/fsmonitor/fsm-listen-linux.c new file +10
-0+struct fsmonitor_batch *batch =NULL ; +struct string_list cookie_list =STRING_LIST_INIT_DUP ; +struct watch_entry k, *w; +struct strbuf path; +const struct inotify_event *event; +int fd = state->listen_data->fd_inotify; +ssize_t len; +char *ptr, *p; + +strbuf_init (&path,PATH_MAX );
Just use "struct strbuf path = STRBUF_INIT" instead? I.e...
Quoted 37 lines
compat/fsmonitor/fsm-listen-linux.c new file +37
-0+ +for (;;) { + len =read (fd, buf,sizeof (buf)); +if (len ==-1 && errno !=EAGAIN ) { +error_errno (_ ("reading inotify message stream failed" )); + state->listen_data->shutdown =SHUTDOWN_ERROR ; + goto done; + } + +/* nothing to read */ +if (len <=0 ) + goto done; + +/* Loop over all events in the buffer. */ +for (ptr = buf; ptr < buf + len; + ptr +=sizeof (struct inotify_event ) + event->len) { + + event = (const struct inotify_event *) ptr; + +if (em_ignore (event->mask)) +continue ; + +/* File system was unmounted or event queue overflowed */ +if (em_force_shutdown (event->mask)) { +if (trace_pass_fl (&trace_fsmonitor)) +log_mask_set ("Forcing shutdown" , event->mask); + state->listen_data->shutdown =SHUTDOWN_FORCE ; + goto done; + } + +hashmap_entry_init (&k.ent,memhash (&event->wd,sizeof (int ))); + k.wd = event->wd; + + w =hashmap_get_entry (&watches, &k, ent,NULL ); +if (!w)/* should never happen */ +BUG ("No watch for '%s'" , event->name); +
compat/fsmonitor/fsm-listen-linux.c line 595
Quoted 10 lines
compat/fsmonitor/fsm-listen-linux.c new file +10
-0+/* directory watch was removed */ +if (em_remove_watch (event->mask)) { +remove_watch (w, state->listen_data); +continue ; + } + +strbuf_reset (&path); +strbuf_add (&path, w->dir,strlen (w->dir)); +strbuf_addch (&path,'/' ); +strbuf_addstr (&path, event->name);
... we may not even get to this, so we may pointlessly pre-grow it, if we're considering the micro-optimization.
But if we do need it it'll quickly get up to size in the loop, which is probably smaller than PATH_MAX, and paths can exceed PATH_MAX....
Quoted 4 lines
compat/fsmonitor/fsm-listen-linux.c new file +4
-0+for (;;) { +switch (state->listen_data->shutdown) { +case SHUTDOWN_CONTINUE : + poll_num =poll (fds,1 ,1 );
compat/fsmonitor/fsm-listen-linux.c line 647
Quoted 10 lines
compat/fsmonitor/fsm-listen-linux.c new file +10
-0+if (poll_num ==-1 ) { +if (errno ==EINTR ) +continue ; +error_errno (_ ("polling inotify message stream failed" )); + state->listen_data->shutdown =SHUTDOWN_ERROR ; +continue ; + } + +if ((time (NULL ) - checked) >= interval) { + checked =time (NULL );
As this is linux-specific code, shouldn't we use the linux-specific API to get the guaranteed atomically growing time here, arther than time()?
Maybe not, but I wonder if this has funny side-effects with ntpd adjusting the time concurrently...
v5 cover fsmonitor: Implement fsmonitor for Linux
In reply to Eric DeCosta via GitGitGadgetGoal is to deliver fsmonitor for Linux that is on par with fsmonitor for Windows and Mac OS.
This patch set builds upon previous work for done for Windows and Mac OS to implement a fsmonitor back-end for Linux based on the Linux inotify API. inotify differs significantly from the equivalent Windows and Mac OS APIs in that a watch must be registered for every directory of interest (rather than a singular watch at the root of the directory tree) and special care must be taken to handle directory renames correctly.
More information about inotify:
https://man7.org/linux/man-pages/man7/inotify.7.html
v4 differs from v3:
* Code review feedback
v3 differs from v2:
* Avoid potential entanglements with GPLv3
* Classify a reasonable set of filesystems as being remote
v2 differs from v1:
* Prior work for Windows and Mac OS has been merged to master, reducing the patch set from 12 to 6 patches
* Code review feedback
* Identified and resolved race condition revealed by CI test system, see "Limitations and caveats" regarding monitoring of directory trees from the man page, above
* Apologies for being away from this for so long, but my attention was needed elsewhere
v1 differs from v0:
* Code review feedback
* Update how and which code can be shared between Mac OS and Linux
* Increase polling frequency to every 1ms (matches Mac OS)
* Updates to t7527 to improve test stability
Eric DeCosta (6):
fsmonitor: prepare to share code between Mac OS and Linux
fsmonitor: determine if filesystem is local or remote
fsmonitor: implement filesystem change listener for Linux
fsmonitor: enable fsmonitor for Linux
fsmonitor: test updates
fsmonitor: update doc for Linux
Documentation/config/fsmonitor--daemon.txt | 4 +- Documentation/git-fsmonitor--daemon.txt | 24 +- compat/fsmonitor/fsm-health-linux.c | 24 + compat/fsmonitor/fsm-ipc-darwin.c | 53 +- compat/fsmonitor/fsm-ipc-linux.c | 1 + compat/fsmonitor/fsm-ipc-unix.c | 52 ++ compat/fsmonitor/fsm-listen-linux.c | 676 +++++++++++++++++++++ compat/fsmonitor/fsm-path-utils-linux.c | 193 ++++++ compat/fsmonitor/fsm-settings-darwin.c | 63 +- compat/fsmonitor/fsm-settings-linux.c | 1 + compat/fsmonitor/fsm-settings-unix.c | 62 ++ config.mak.uname | 8 + contrib/buildsystems/CMakeLists.txt | 11 +- t/t7527-builtin-fsmonitor.sh | 94 ++- 14 files changed, 1127 insertions(+), 139 deletions(-) create mode 100644 compat/fsmonitor/fsm-health-linux.c create mode 100644 compat/fsmonitor/fsm-ipc-linux.c create mode 100644 compat/fsmonitor/fsm-ipc-unix.c create mode 100644 compat/fsmonitor/fsm-listen-linux.c create mode 100644 compat/fsmonitor/fsm-path-utils-linux.c create mode 100644 compat/fsmonitor/fsm-settings-linux.c create mode 100644 compat/fsmonitor/fsm-settings-unix.c
base-commit: c48035d29b4e524aed3a32f0403676f0d9128863
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1352%2Fedecosta-mw%2Ffsmonitor_linux-v5
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1352/edecosta-mw/fsmonitor_linux-v5
Pull-Request: https://github.com/git/git/pull/1352
Range-diff vs v4:
1: 99d684c7bdf ! 1: 7f659603c9a fsmonitor: prepare to share code between Mac OS and Linux
@@ compat/fsmonitor/fsm-settings-unix.c (new)
+ strbuf_addstr(&path, ipc_path);
+
+ if (fsmonitor__get_fs_info(dirname(path.buf), &fs) == -1) {
++ free(fs.typename);
+ strbuf_release(&path);
+ return FSMONITOR_REASON_ERROR;
+ }
2: e53fc077540 ! 2: eb3ff9d1c05 fsmonitor: determine if filesystem is local or remote
@@ compat/fsmonitor/fsm-path-utils-linux.c (new)
+#include <sys/vfs.h>
+#include <sys/statvfs.h>
+
-+static int is_remote_fs(const char* path) {
++static int is_remote_fs(const char *path) {
+ struct statfs fs;
+
+ if (statfs(path, &fs)) {
@@ compat/fsmonitor/fsm-path-utils-linux.c (new)
+ }
+
+ switch (fs.f_type) {
-+ case 0x61636673: /* ACFS */
-+ case 0x5346414F: /* AFS */
-+ case 0x00C36400: /* CEPH */
-+ case 0xFF534D42: /* CIFS */
-+ case 0x73757245: /* CODA */
-+ case 0x19830326: /* FHGFS */
-+ case 0x1161970: /* GFS */
-+ case 0x47504653: /* GPFS */
-+ case 0x013111A8: /* IBRIX */
-+ case 0x6B414653: /* KAFS */
-+ case 0x0BD00BD0: /* LUSTRE */
-+ case 0x564C: /* NCP */
-+ case 0x6969: /* NFS */
-+ case 0x6E667364: /* NFSD */
-+ case 0x7461636f: /* OCFS2 */
-+ case 0xAAD7AAEA: /* PANFS */
-+ case 0x517B: /* SMB */
-+ case 0xBEEFDEAD: /* SNFS */
-+ case 0xFE534D42: /* SMB2 */
-+ case 0xBACBACBC: /* VMHGFS */
-+ case 0xA501FCF5: /* VXFS */
-+ return 1;
-+ default:
-+ break;
++ case 0x61636673: /* ACFS */
++ case 0x5346414F: /* AFS */
++ case 0x00C36400: /* CEPH */
++ case 0xFF534D42: /* CIFS */
++ case 0x73757245: /* CODA */
++ case 0x19830326: /* FHGFS */
++ case 0x1161970: /* GFS */
++ case 0x47504653: /* GPFS */
++ case 0x013111A8: /* IBRIX */
++ case 0x6B414653: /* KAFS */
++ case 0x0BD00BD0: /* LUSTRE */
++ case 0x564C: /* NCP */
++ case 0x6969: /* NFS */
++ case 0x6E667364: /* NFSD */
++ case 0x7461636f: /* OCFS2 */
++ case 0xAAD7AAEA: /* PANFS */
++ case 0x517B: /* SMB */
++ case 0xBEEFDEAD: /* SNFS */
++ case 0xFE534D42: /* SMB2 */
++ case 0xBACBACBC: /* VMHGFS */
++ case 0xA501FCF5: /* VXFS */
++ return 1;
++ default:
++ break;
+ }
+
+ return 0;
+}
+
+static int find_mount(const char *path, const struct statvfs *fs,
-+ struct mntent *ent)
++ struct mntent *entry)
+{
+ const char *const mounts = "/proc/mounts";
-+ const char *rp = real_pathdup(path, 1);
++ char *rp = real_pathdup(path, 1);
+ struct mntent *ment = NULL;
+ struct statvfs mntfs;
+ FILE *fp;
+ int found = 0;
+ int dlen, plen, flen = 0;
+
-+ ent->mnt_fsname = NULL;
-+ ent->mnt_dir = NULL;
-+ ent->mnt_type = NULL;
++ entry->mnt_fsname = NULL;
++ entry->mnt_dir = NULL;
++ entry->mnt_type = NULL;
+
+ fp = setmntent(mounts, "r");
+ if (!fp) {
++ free(rp);
+ error_errno(_("setmntent('%s') failed"), mounts);
+ return -1;
+ }
@@ compat/fsmonitor/fsm-path-utils-linux.c (new)
+ default:
+ error_errno(_("statvfs('%s') failed"), ment->mnt_dir);
+ endmntent(fp);
++ free(rp);
+ return -1;
+ }
+ }
@@ compat/fsmonitor/fsm-path-utils-linux.c (new)
+ if (dlen > plen)
+ continue;
+ /*
-+ * root is always a potential match; otherwise look for
-+ * directory prefix
++ * look for the longest prefix (including root)
+ */
-+ if ((dlen == 1 && ment->mnt_dir[0] == '/') ||
-+ (dlen > flen && (!rp[dlen] || rp[dlen] == '/'))) {
++ if (dlen > flen &&
++ ((dlen == 1 && ment->mnt_dir[0] == '/') ||
++ (!rp[dlen] || rp[dlen] == '/'))) {
+ flen = dlen;
++ found = 1;
++
+ /*
+ * https://man7.org/linux/man-pages/man3/getmntent.3.html
+ *
+ * The pointer points to a static area of memory which is
+ * overwritten by subsequent calls to getmntent().
+ */
-+ found = 1;
-+ free(ent->mnt_fsname);
-+ free(ent->mnt_dir);
-+ free(ent->mnt_type);
-+ ent->mnt_fsname = xstrdup(ment->mnt_fsname);
-+ ent->mnt_dir = xstrdup(ment->mnt_dir);
-+ ent->mnt_type = xstrdup(ment->mnt_type);
++ free(entry->mnt_fsname);
++ free(entry->mnt_dir);
++ free(entry->mnt_type);
++ entry->mnt_fsname = xstrdup(ment->mnt_fsname);
++ entry->mnt_dir = xstrdup(ment->mnt_dir);
++ entry->mnt_type = xstrdup(ment->mnt_type);
+ }
+ }
+ }
+ endmntent(fp);
++ free(rp);
+
+ if (!found)
+ return -1;
@@ compat/fsmonitor/fsm-path-utils-linux.c (new)
+
+int fsmonitor__get_fs_info(const char *path, struct fs_info *fs_info)
+{
-+ struct mntent ment;
++ struct mntent entry;
+ struct statvfs fs;
+
++ fs_info->is_remote = -1;
++ fs_info->typename = NULL;
++
+ if (statvfs(path, &fs))
+ return error_errno(_("statvfs('%s') failed"), path);
+
-+
-+ if (find_mount(path, &fs, &ment) < 0) {
-+ free(ment.mnt_fsname);
-+ free(ment.mnt_dir);
-+ free(ment.mnt_type);
++ if (find_mount(path, &fs, &entry) < 0) {
++ free(entry.mnt_fsname);
++ free(entry.mnt_dir);
++ free(entry.mnt_type);
+ return -1;
+ }
+
+ trace_printf_key(&trace_fsmonitor,
+ "statvfs('%s') [flags 0x%08lx] '%s' '%s'",
-+ path, fs.f_flag, ment.mnt_type, ment.mnt_fsname);
++ path, fs.f_flag, entry.mnt_type, entry.mnt_fsname);
+
-+ fs_info->is_remote = is_remote_fs(ment.mnt_dir);
-+ fs_info->typename = ment.mnt_fsname;
-+ free(ment.mnt_dir);
-+ free(ment.mnt_type);
++ fs_info->is_remote = is_remote_fs(entry.mnt_dir);
++ fs_info->typename = xstrdup(entry.mnt_fsname);
++ free(entry.mnt_fsname);
++ free(entry.mnt_dir);
++ free(entry.mnt_type);
+
-+ if (fs_info->is_remote < 0) {
-+ free(ment.mnt_fsname);
++ if (fs_info->is_remote < 0)
+ return -1;
-+ }
+
+ trace_printf_key(&trace_fsmonitor,
+ "'%s' is_remote: %d",
@@ compat/fsmonitor/fsm-path-utils-linux.c (new)
+{
+ struct fs_info fs;
+
-+ if (fsmonitor__get_fs_info(path, &fs))
++ if (fsmonitor__get_fs_info(path, &fs)) {
++ free(fs.typename);
+ return -1;
++ }
+
+ free(fs.typename);
+
3: 80282efef57 = 3: e093a2703b1 fsmonitor: implement filesystem change listener for Linux
4: cb03803e355 = 4: c03070fb0a2 fsmonitor: enable fsmonitor for Linux
5: 8d9d469b356 = 5: 6a7b554642c fsmonitor: test updates
6: 5afd03fa6ca = 6: 827410c22ee fsmonitor: update doc for LinuxSignature · 1 line
gitgitgadget
PATCH v5 1/6 fsmonitor: prepare to share code between Mac OS and Linux
From: Eric DeCosta <[email protected]>
Linux and Mac OS can share some of the code originally developed for Mac OS.
Mac OS and Linux can share fsm-ipc-unix.c and fsm-settings-unix.c
--- compat/fsmonitor/fsm-health-linux.c | 24 ++++++++++ compat/fsmonitor/fsm-ipc-darwin.c | 53 +--------------------- compat/fsmonitor/fsm-ipc-linux.c | 1 + compat/fsmonitor/fsm-ipc-unix.c | 52 +++++++++++++++++++++ compat/fsmonitor/fsm-settings-darwin.c | 63 +------------------------- compat/fsmonitor/fsm-settings-linux.c | 1 + compat/fsmonitor/fsm-settings-unix.c | 62 +++++++++++++++++++++++++ 7 files changed, 142 insertions(+), 114 deletions(-) create mode 100644 compat/fsmonitor/fsm-health-linux.c create mode 100644 compat/fsmonitor/fsm-ipc-linux.c create mode 100644 compat/fsmonitor/fsm-ipc-unix.c create mode 100644 compat/fsmonitor/fsm-settings-linux.c create mode 100644 compat/fsmonitor/fsm-settings-unix.c
compat/fsmonitor/fsm-health-linux.c new file +24 -0
@@ -0,0 +1,24 @@ +#include "cache.h" +#include "config.h" +#include "fsmonitor.h" +#include "fsm-health.h" +#include "fsmonitor--daemon.h" + +int fsm_health__ctor (struct fsmonitor_daemon_state *state) +{ +return 0 ; +} + +void fsm_health__dtor (struct fsmonitor_daemon_state *state) +{ +return ; +} + +void fsm_health__loop (struct fsmonitor_daemon_state *state) +{ +return ; +} + +void fsm_health__stop_async (struct fsmonitor_daemon_state *state) +{ +}
compat/fsmonitor/fsm-ipc-darwin.c +1 -52
53 lines not on this page · raw message
compat/fsmonitor/fsm-ipc-linux.c new file +1 -0
@@ -0,0 +1 @@ +#include "fsm-ipc-unix.c"
compat/fsmonitor/fsm-ipc-unix.c new file +52 -0
52 lines not on this page · raw message
compat/fsmonitor/fsm-settings-darwin.c +1 -62
63 lines not on this page · raw message
compat/fsmonitor/fsm-settings-linux.c new file +1 -0
@@ -0,0 +1 @@ +#include "fsm-settings-unix.c"
compat/fsmonitor/fsm-settings-unix.c new file +62 -0
62 lines not on this page · raw message
Signature · 1 line
gitgitgadget
PATCH v5 2/6 fsmonitor: determine if filesystem is local or remote
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
Compare the given path to the mounted filesystems. Find the mount that is the longest prefix of the path (if any) and determine if that mount is on a local or remote filesystem.
--- compat/fsmonitor/fsm-path-utils-linux.c | 193 ++++++++++++++++++++++++ 1 file changed, 193 insertions(+) create mode 100644 compat/fsmonitor/fsm-path-utils-linux.c
compat/fsmonitor/fsm-path-utils-linux.c new file +193 -0
@@ -0,0 +1,193 @@ +#include "fsmonitor.h" +#include "fsmonitor-path-utils.h" +#include <errno.h> +#include <mntent.h> +#include <sys/mount.h> +#include <sys/vfs.h> +#include <sys/statvfs.h> + +static int is_remote_fs (const char *path) { +struct statfs fs; + +if (statfs (path, &fs)) { +error_errno (_ ("statfs('%s') failed" ), path); +return -1 ; + } + +switch (fs.f_type) { +case 0x61636673 :/* ACFS */ +case 0x5346414F :/* AFS */ +case 0x00C36400 :/* CEPH */ +case 0xFF534D42 :/* CIFS */ +case 0x73757245 :/* CODA */ +case 0x19830326 :/* FHGFS */ +case 0x1161970 :/* GFS */ +case 0x47504653 :/* GPFS */ +case 0x013111A8 :/* IBRIX */ +case 0x6B414653 :/* KAFS */ +case 0x0BD00BD0 :/* LUSTRE */ +case 0x564C :/* NCP */ +case 0x6969 :/* NFS */ +case 0x6E667364 :/* NFSD */ +case 0x7461636f :/* OCFS2 */ +case 0xAAD7AAEA :/* PANFS */ +case 0x517B :/* SMB */ +case 0xBEEFDEAD :/* SNFS */ +case 0xFE534D42 :/* SMB2 */ +case 0xBACBACBC :/* VMHGFS */ +case 0xA501FCF5 :/* VXFS */ +return 1 ; +default : +break ; + } + +return 0 ; +} + +static int find_mount (const char *path,const struct statvfs *fs, +struct mntent *entry) +{ +const char *const mounts ="/proc/mounts" ; +char *rp =real_pathdup (path,1 ); +struct mntent *ment =NULL ; +struct statvfs mntfs; +FILE *fp; +int found =0 ; +int dlen, plen, flen =0 ; + + entry->mnt_fsname =NULL ; + entry->mnt_dir =NULL ; + entry->mnt_type =NULL ; + + fp =setmntent (mounts,"r" ); +if (!fp) { +free (rp); +error_errno (_ ("setmntent('%s') failed" ), mounts); +return -1 ; + } + + plen =strlen (rp); + +/* read all the mount information and compare to path */ +while ((ment =getmntent (fp)) !=NULL ) { +if (statvfs (ment->mnt_dir, &mntfs)) { +switch (errno) { +case EPERM : +case ESRCH : +case EACCES : +continue ; +default : +error_errno (_ ("statvfs('%s') failed" ), ment->mnt_dir); +endmntent (fp); +free (rp); +return -1 ; + } + } + +/* is mount on the same filesystem and is a prefix of the path */ +if ((fs->f_fsid == mntfs.f_fsid) && + !strncmp (ment->mnt_dir, rp,strlen (ment->mnt_dir))) { + dlen =strlen (ment->mnt_dir); +if (dlen > plen) +continue ; +/* +* look for the longest prefix (including root) +*/ +if (dlen > flen && + ((dlen ==1 && ment->mnt_dir[0 ] =='/' ) || + (!rp[dlen] || rp[dlen] =='/' ))) { + flen = dlen; + found =1 ; + +/* +* https://man7.org/linux/man-pages/man3/getmntent.3.html +* +* The pointer points to a static area of memory which is +* overwritten by subsequent calls to getmntent(). +*/ +free (entry->mnt_fsname); +free (entry->mnt_dir); +free (entry->mnt_type); + entry->mnt_fsname =xstrdup (ment->mnt_fsname); + entry->mnt_dir =xstrdup (ment->mnt_dir); + entry->mnt_type =xstrdup (ment->mnt_type); + } + } + } +endmntent (fp); +free (rp); + +if (!found) +return -1 ; + +return 0 ; +} + +int fsmonitor__get_fs_info (const char *path,struct fs_info *fs_info) +{ +struct mntent entry; +struct statvfs fs; + + fs_info->is_remote =-1 ; + fs_info->typename =NULL ; + +if (statvfs (path, &fs)) +return error_errno (_ ("statvfs('%s') failed" ), path); + +if (find_mount (path, &fs, &entry) <0 ) { +free (entry.mnt_fsname); +free (entry.mnt_dir); +free (entry.mnt_type); +return -1 ; + } + +trace_printf_key (&trace_fsmonitor, +"statvfs('%s') [flags 0x%08lx] '%s' '%s'" , + path, fs.f_flag, entry.mnt_type, entry.mnt_fsname); + + fs_info->is_remote =is_remote_fs (entry.mnt_dir); + fs_info->typename =xstrdup (entry.mnt_fsname); +free (entry.mnt_fsname); +free (entry.mnt_dir); +free (entry.mnt_type); + +if (fs_info->is_remote <0 ) +return -1 ; + +trace_printf_key (&trace_fsmonitor, +"'%s' is_remote: %d" , + path, fs_info->is_remote); + +return 0 ; +} + +int fsmonitor__is_fs_remote (const char *path) +{ +struct fs_info fs; + +if (fsmonitor__get_fs_info (path, &fs)) { +free (fs.typename); +return -1 ; + } + +free (fs.typename); + +return fs.is_remote; +} + +/* +* No-op for now. +*/ +int fsmonitor__get_alias (const char *path,struct alias_info *info) +{ +return 0 ; +} + +/* +* No-op for now. +*/ +char *fsmonitor__resolve_alias (const char *path, +const struct alias_info *info) +{ +return NULL ; +}
Signature · 1 line
gitgitgadget
PATCH v5 4/6 fsmonitor: enable fsmonitor for Linux
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
Update build to enable fsmonitor for Linux.
--- config.mak.uname | 8 ++++++++ contrib/buildsystems/CMakeLists.txt | 11 ++++++++++- 2 files changed, 18 insertions(+), 1 deletion(-)
config.mak.uname +8 -0
@@ -68,6 +68,14 @@ ifeq ($(uname_S),Linux) ifneq ($(findstring .el7.,$(uname_R)),) BASIC_CFLAGS += -std=c99 endif + # The builtin FSMonitor on Linux builds upon Simple-IPC. Both require + # Unix domain sockets and PThreads. + ifndef NO_PTHREADS + ifndef NO_UNIX_SOCKETS + FSMONITOR_DAEMON_BACKEND = linux + FSMONITOR_OS_SETTINGS = linux + endif + endif endif ifeq ($(uname_S),GNU/kFreeBSD) HAVE_ALLOCA_H = YesPlease
contrib/buildsystems/CMakeLists.txt +10 -1
@@ -304,7 +304,16 @@ else() endif() if(SUPPORTS_SIMPLE_IPC) - if(CMAKE_SYSTEM_NAME STREQUAL "Windows") + if(CMAKE_SYSTEM_NAME STREQUAL "Linux") + add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-linux.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-linux.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-linux.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-linux.c) + + add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-linux.c) + elseif(CMAKE_SYSTEM_NAME STREQUAL "Windows") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c) list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c)
Signature · 1 line
gitgitgadget
PATCH v5 3/6 fsmonitor: implement filesystem change listener for Linux
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
Implement a filesystem change listener for Linux based on the inotify API:
https://man7.org/linux/man-pages/man7/inotify.7.html
inotify requires registering a watch on every directory in the worktree and special handling of moves/renames.
--- compat/fsmonitor/fsm-listen-linux.c | 676 ++++++++++++++++++++++++++++ 1 file changed, 676 insertions(+) create mode 100644 compat/fsmonitor/fsm-listen-linux.c
compat/fsmonitor/fsm-listen-linux.c new file +676 -0
@@ -0,0 +1,676 @@ +#include "cache.h" +#include "fsmonitor.h" +#include "fsm-listen.h" +#include "fsmonitor--daemon.h" +#include <dirent.h> +#include <fcntl.h> +#include <sys/inotify.h> +#include <sys/stat.h> + +/* +* Safe value to bitwise OR with rest of mask for +* kernels that do not support IN_MASK_CREATE +*/ +#ifndef IN_MASK_CREATE +#define IN_MASK_CREATE 0x00000000 +#endif + +enum shutdown_reason { +SHUTDOWN_CONTINUE =0 , +SHUTDOWN_STOP , +SHUTDOWN_ERROR , +SHUTDOWN_FORCE +}; + +struct watch_entry { +struct hashmap_entry ent; +int wd; +uint32_t cookie; +const char *dir; +}; + +struct rename_entry { +struct hashmap_entry ent; +time_t whence; +uint32_t cookie; +const char *dir; +}; + +struct fsm_listen_data { +int fd_inotify; +enum shutdown_reason shutdown; +struct hashmap watches; +struct hashmap renames; +struct hashmap revwatches; +}; + +static int watch_entry_cmp (const void *cmp_data, +const struct hashmap_entry *eptr, +const struct hashmap_entry *entry_or_key, +const void *keydata) +{ +const struct watch_entry *e1, *e2; + + e1 = container_of(eptr, const struct watch_entry, ent); + e2 = container_of(eptr, const struct watch_entry, ent); +return e1->wd != e2->wd; +} + +static int revwatches_entry_cmp (const void *cmp_data, +const struct hashmap_entry *eptr, +const struct hashmap_entry *entry_or_key, +const void *keydata) +{ +const struct watch_entry *e1, *e2; + + e1 = container_of(eptr, const struct watch_entry, ent); + e2 = container_of(eptr, const struct watch_entry, ent); +return strcmp (e1->dir, e2->dir); +} + +static int rename_entry_cmp (const void *cmp_data, +const struct hashmap_entry *eptr, +const struct hashmap_entry *entry_or_key, +const void *keydata) +{ +const struct rename_entry *e1, *e2; + + e1 = container_of(eptr, const struct rename_entry, ent); + e2 = container_of(eptr, const struct rename_entry, ent); +return e1->cookie != e2->cookie; +} + +/* +* Register an inotify watch, add watch descriptor to path mapping +* and the reverse mapping. +*/ +static int add_watch (const char *path,struct fsm_listen_data *data) +{ +const char *interned =strintern (path); +struct watch_entry *w1, *w2; + +/* add the inotify watch, don't allow watches to be modified */ +int wd =inotify_add_watch (data->fd_inotify, interned, + (IN_ALL_EVENTS |IN_ONLYDIR |IN_MASK_CREATE ) + ^IN_ACCESS ^IN_CLOSE ^IN_OPEN ); +if (wd <0 ) +return error_errno ("inotify_add_watch('%s') failed" , interned); + +/* add watch descriptor -> directory mapping */ +CALLOC_ARRAY (w1,1 ); + w1->wd = wd; + w1->dir = interned; +hashmap_entry_init (&w1->ent,memhash (&w1->wd,sizeof (int ))); +hashmap_add (&data->watches, &w1->ent); + +/* add directory -> watch descriptor mapping */ +CALLOC_ARRAY (w2,1 ); + w2->wd = wd; + w2->dir = interned; +hashmap_entry_init (&w2->ent,memhash (w2->dir,strlen (w2->dir))); +hashmap_add (&data->revwatches, &w2->ent); + +return 0 ; +} + +/* +* Remove the inotify watch, the watch descriptor to path mapping +* and the reverse mapping. +*/ +static void remove_watch (struct watch_entry *w, +struct fsm_listen_data *data) +{ +struct watch_entry k1, k2, *w1, *w2; + +/* remove watch, ignore error if kernel already did it */ +if (inotify_rm_watch (data->fd_inotify, w->wd) && errno !=EINVAL ) +error_errno ("inotify_rm_watch() failed" ); + +hashmap_entry_init (&k1.ent,memhash (&w->wd,sizeof (int ))); + w1 =hashmap_remove_entry (&data->watches, &k1, ent,NULL ); +if (!w1) +BUG ("Double remove of watch for '%s'" , w->dir); + +if (w1->cookie) +BUG ("Removing watch for '%s' which has a pending rename" , w1->dir); + +hashmap_entry_init (&k2.ent,memhash (w->dir,strlen (w->dir))); + w2 =hashmap_remove_entry (&data->revwatches, &k2, ent,NULL ); +if (!w2) +BUG ("Double remove of reverse watch for '%s'" , w->dir); + +/* w1->dir and w2->dir are interned strings, we don't own them */ +free (w1); +free (w2); +} + +/* +* Check for stale directory renames. +* +* https://man7.org/linux/man-pages/man7/inotify.7.html +* +* Allow for some small timeout to account for the fact that insertion of the +* IN_MOVED_FROM+IN_MOVED_TO event pair is not atomic, and the possibility that +* there may not be any IN_MOVED_TO event. +* +* If the IN_MOVED_TO event is not received within the timeout then events have +* been missed and the monitor is in an inconsistent state with respect to the +* filesystem. +*/ +static int check_stale_dir_renames (struct hashmap *renames,time_t max_age) +{ +struct rename_entry *re; +struct hashmap_iter iter; + +hashmap_for_each_entry (renames, &iter, re, ent) { +if (re->whence <= max_age) +return -1 ; + } +return 0 ; +} + +/* +* Track pending renames. +* +* Tracking is done via a event cookie to watch descriptor mapping. +* +* A rename is not complete until matching a IN_MOVED_TO event is received +* for a corresponding IN_MOVED_FROM event. +*/ +static void add_dir_rename (uint32_t cookie,const char *path, +struct fsm_listen_data *data) +{ +struct watch_entry k, *w; +struct rename_entry *re; + +/* lookup the watch descriptor for the given path */ +hashmap_entry_init (&k.ent,memhash (path,strlen (path))); + w =hashmap_get_entry (&data->revwatches, &k, ent,NULL ); +if (!w)/* should never happen */ +BUG ("No watch for '%s'" , path); + w->cookie = cookie; + +/* add the pending rename to match against later */ +CALLOC_ARRAY (re,1 ); + re->dir = w->dir; + re->cookie = w->cookie; + re->whence =time (NULL ); +hashmap_entry_init (&re->ent,memhash (&re->cookie,sizeof (uint32_t ))); +hashmap_add (&data->renames, &re->ent); +} + +/* +* Handle directory renames +* +* Once a IN_MOVED_TO event is received, lookup the rename tracking information +* via the event cookie and use this information to update the watch. +*/ +static void rename_dir (uint32_t cookie,const char *path, +struct fsm_listen_data *data) +{ +struct rename_entry rek, *re; +struct watch_entry k, *w; + +/* lookup a pending rename to match */ + rek.cookie = cookie; +hashmap_entry_init (&rek.ent,memhash (&rek.cookie,sizeof (uint32_t ))); + re =hashmap_get_entry (&data->renames, &rek, ent,NULL ); +if (re) { + k.dir = re->dir; +hashmap_entry_init (&k.ent,memhash (k.dir,strlen (k.dir))); + w =hashmap_get_entry (&data->revwatches, &k, ent,NULL ); +if (w) { + w->cookie =0 ;/* rename handled */ +remove_watch (w, data); +add_watch (path, data); + }else { +BUG ("No matching watch" ); + } + }else { +BUG ("No matching cookie" ); + } +} + +/* +* Recursively add watches to every directory under path +*/ +static int register_inotify (const char *path, +struct fsmonitor_daemon_state *state, +struct fsmonitor_batch *batch) +{ +DIR *dir; +const char *rel; +struct strbuf current =STRBUF_INIT ; +struct dirent *de; +struct stat fs; +int ret =-1 ; + + dir =opendir (path); +if (!dir) +return error_errno ("opendir('%s') failed" , path); + +while ((de =readdir_skip_dot_and_dotdot (dir)) !=NULL ) { +strbuf_reset (¤t); +strbuf_addf (¤t,"%s/%s" , path, de->d_name); +if (lstat (current.buf, &fs)) { +error_errno ("lstat('%s') failed" , current.buf); + goto failed; + } + +/* recurse into directory */ +if (S_ISDIR (fs.st_mode)) { +if (add_watch (current.buf, state->listen_data)) + goto failed; +if (register_inotify (current.buf, state, batch)) + goto failed; + }else if (batch) { + rel = current.buf + state->path_worktree_watch.len +1 ; +trace_printf_key (&trace_fsmonitor,"explicitly adding '%s'" , rel); +fsmonitor_batch__add_path (batch, rel); + } + } + ret =0 ; + +failed: +strbuf_release (¤t); +if (closedir (dir) <0 ) +return error_errno ("closedir('%s') failed" , path); +return ret; +} + +static int em_rename_dir_from (u_int32_t mask) +{ +return ((mask &IN_ISDIR ) && (mask &IN_MOVED_FROM )); +} + +static int em_rename_dir_to (u_int32_t mask) +{ +return ((mask &IN_ISDIR ) && (mask &IN_MOVED_TO )); +} + +static int em_remove_watch (u_int32_t mask) +{ +return (mask &IN_DELETE_SELF ); +} + +static int em_dir_renamed (u_int32_t mask) +{ +return ((mask &IN_ISDIR ) && (mask &IN_MOVE )); +} + +static int em_dir_created (u_int32_t mask) +{ +return ((mask &IN_ISDIR ) && (mask &IN_CREATE )); +} + +static int em_dir_deleted (uint32_t mask) +{ +return ((mask &IN_ISDIR ) && (mask &IN_DELETE )); +} + +static int em_force_shutdown (u_int32_t mask) +{ +return (mask &IN_UNMOUNT ) || (mask &IN_Q_OVERFLOW ); +} + +static int em_ignore (u_int32_t mask) +{ +return (mask &IN_IGNORED ) || (mask &IN_MOVE_SELF ); +} + +static void log_mask_set (const char *path,u_int32_t mask) +{ +struct strbuf msg =STRBUF_INIT ; + +if (mask &IN_ACCESS ) +strbuf_addstr (&msg,"IN_ACCESS|" ); +if (mask &IN_MODIFY ) +strbuf_addstr (&msg,"IN_MODIFY|" ); +if (mask &IN_ATTRIB ) +strbuf_addstr (&msg,"IN_ATTRIB|" ); +if (mask &IN_CLOSE_WRITE ) +strbuf_addstr (&msg,"IN_CLOSE_WRITE|" ); +if (mask &IN_CLOSE_NOWRITE ) +strbuf_addstr (&msg,"IN_CLOSE_NOWRITE|" ); +if (mask &IN_OPEN ) +strbuf_addstr (&msg,"IN_OPEN|" ); +if (mask &IN_MOVED_FROM ) +strbuf_addstr (&msg,"IN_MOVED_FROM|" ); +if (mask &IN_MOVED_TO ) +strbuf_addstr (&msg,"IN_MOVED_TO|" ); +if (mask &IN_CREATE ) +strbuf_addstr (&msg,"IN_CREATE|" ); +if (mask &IN_DELETE ) +strbuf_addstr (&msg,"IN_DELETE|" ); +if (mask &IN_DELETE_SELF ) +strbuf_addstr (&msg,"IN_DELETE_SELF|" ); +if (mask &IN_MOVE_SELF ) +strbuf_addstr (&msg,"IN_MOVE_SELF|" ); +if (mask &IN_UNMOUNT ) +strbuf_addstr (&msg,"IN_UNMOUNT|" ); +if (mask &IN_Q_OVERFLOW ) +strbuf_addstr (&msg,"IN_Q_OVERFLOW|" ); +if (mask &IN_IGNORED ) +strbuf_addstr (&msg,"IN_IGNORED|" ); +if (mask &IN_ISDIR ) +strbuf_addstr (&msg,"IN_ISDIR|" ); + +trace_printf_key (&trace_fsmonitor,"inotify_event: '%s', mask=%#8.8x %s" , + path, mask, msg.buf); + +strbuf_release (&msg); +} + +int fsm_listen__ctor (struct fsmonitor_daemon_state *state) +{ +int fd; +int ret =0 ; +struct fsm_listen_data *data; + +CALLOC_ARRAY (data,1 ); + state->listen_data = data; + state->listen_error_code =-1 ; + data->shutdown =SHUTDOWN_ERROR ; + + fd =inotify_init1 (O_NONBLOCK ); +if (fd <0 ) +return error_errno ("inotify_init1() failed" ); + + data->fd_inotify = fd; + +hashmap_init (&data->watches, watch_entry_cmp,NULL ,0 ); +hashmap_init (&data->renames, rename_entry_cmp,NULL ,0 ); +hashmap_init (&data->revwatches, revwatches_entry_cmp,NULL ,0 ); + +if (add_watch (state->path_worktree_watch.buf, data)) + ret =-1 ; +else if (register_inotify (state->path_worktree_watch.buf, state,NULL )) + ret =-1 ; +else if (state->nr_paths_watching >1 ) { +if (add_watch (state->path_gitdir_watch.buf, data)) + ret =-1 ; +else if (register_inotify (state->path_gitdir_watch.buf, state,NULL )) + ret =-1 ; + } + +if (!ret) { + state->listen_error_code =0 ; + data->shutdown =SHUTDOWN_CONTINUE ; + } + +return ret; +} + +void fsm_listen__dtor (struct fsmonitor_daemon_state *state) +{ +struct fsm_listen_data *data; +struct hashmap_iter iter; +struct watch_entry *w; +int fd; + +if (!state || !state->listen_data) +return ; + + data = state->listen_data; + fd = data->fd_inotify; + +hashmap_for_each_entry (&data->watches, &iter, w, ent) { + w->cookie =0 ;/* ignore any pending renames */ +remove_watch (w, data); + } +hashmap_clear (&data->watches); + +hashmap_clear (&data->revwatches);/* remove_watch freed the entries */ + + hashmap_clear_and_free(&data->renames, struct rename_entry, ent); + +FREE_AND_NULL (state->listen_data); + +if (fd && (close (fd) <0 )) +error_errno (_ ("closing inotify file descriptor failed" )); +} + +void fsm_listen__stop_async (struct fsmonitor_daemon_state *state) +{ +if (!state->listen_data->shutdown) + state->listen_data->shutdown =SHUTDOWN_STOP ; +} + +/* +* Process a single inotify event and queue for publication. +*/ +static int process_event (const char *path, +const struct inotify_event *event, +struct fsmonitor_batch *batch, +struct string_list *cookie_list, +struct fsmonitor_daemon_state *state) +{ +const char *rel; +const char *last_sep; + +switch (fsmonitor_classify_path_absolute (state, path)) { +case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX : +case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX : +/* Use just the filename of the cookie file. */ + last_sep =find_last_dir_sep (path); +string_list_append (cookie_list, + last_sep ? last_sep +1 : path); +break ; +case IS_INSIDE_DOT_GIT : +case IS_INSIDE_GITDIR : +break ; +case IS_DOT_GIT : +case IS_GITDIR : +/* +* If .git directory is deleted or renamed away, +* we have to quit. +*/ +if (em_dir_deleted (event->mask)) { +trace_printf_key (&trace_fsmonitor, +"event: gitdir removed" ); + state->listen_data->shutdown =SHUTDOWN_FORCE ; + goto done; + } + +if (em_dir_renamed (event->mask)) { +trace_printf_key (&trace_fsmonitor, +"event: gitdir renamed" ); + state->listen_data->shutdown =SHUTDOWN_FORCE ; + goto done; + } +break ; +case IS_WORKDIR_PATH : +/* normal events in the working directory */ +if (trace_pass_fl (&trace_fsmonitor)) +log_mask_set (path, event->mask); + + rel = path + state->path_worktree_watch.len +1 ; +fsmonitor_batch__add_path (batch, rel); + +if (em_dir_deleted (event->mask)) +break ; + +/* received IN_MOVE_FROM, add tracking for expected IN_MOVE_TO */ +if (em_rename_dir_from (event->mask)) +add_dir_rename (event->cookie, path, state->listen_data); + +/* received IN_MOVE_TO, update watch to reflect new path */ +if (em_rename_dir_to (event->mask)) { +rename_dir (event->cookie, path, state->listen_data); +if (register_inotify (path, state, batch)) { + state->listen_data->shutdown =SHUTDOWN_ERROR ; + goto done; + } + } + +if (em_dir_created (event->mask)) { +if (add_watch (path, state->listen_data)) { + state->listen_data->shutdown =SHUTDOWN_ERROR ; + goto done; + } +if (register_inotify (path, state, batch)) { + state->listen_data->shutdown =SHUTDOWN_ERROR ; + goto done; + } + } +break ; +case IS_OUTSIDE_CONE : +default : +trace_printf_key (&trace_fsmonitor, +"ignoring '%s'" , path); +break ; + } +return 0 ; +done: +return -1 ; +} + +/* +* Read the inotify event stream and pre-process events before further +* processing and eventual publishing. +*/ +static void handle_events (struct fsmonitor_daemon_state *state) +{ +/* See https://man7.org/linux/man-pages/man7/inotify.7.html */ + char buf[4096] + __attribute__ ((aligned(__alignof__(struct inotify_event)))); + +struct hashmap watches = state->listen_data->watches; +struct fsmonitor_batch *batch =NULL ; +struct string_list cookie_list =STRING_LIST_INIT_DUP ; +struct watch_entry k, *w; +struct strbuf path; +const struct inotify_event *event; +int fd = state->listen_data->fd_inotify; +ssize_t len; +char *ptr, *p; + +strbuf_init (&path,PATH_MAX ); + +for (;;) { + len =read (fd, buf,sizeof (buf)); +if (len ==-1 && errno !=EAGAIN ) { +error_errno (_ ("reading inotify message stream failed" )); + state->listen_data->shutdown =SHUTDOWN_ERROR ; + goto done; + } + +/* nothing to read */ +if (len <=0 ) + goto done; + +/* Loop over all events in the buffer. */ +for (ptr = buf; ptr < buf + len; + ptr +=sizeof (struct inotify_event ) + event->len) { + + event = (const struct inotify_event *) ptr; + +if (em_ignore (event->mask)) +continue ; + +/* File system was unmounted or event queue overflowed */ +if (em_force_shutdown (event->mask)) { +if (trace_pass_fl (&trace_fsmonitor)) +log_mask_set ("Forcing shutdown" , event->mask); + state->listen_data->shutdown =SHUTDOWN_FORCE ; + goto done; + } + +hashmap_entry_init (&k.ent,memhash (&event->wd,sizeof (int ))); + k.wd = event->wd; + + w =hashmap_get_entry (&watches, &k, ent,NULL ); +if (!w)/* should never happen */ +BUG ("No watch for '%s'" , event->name); + +/* directory watch was removed */ +if (em_remove_watch (event->mask)) { +remove_watch (w, state->listen_data); +continue ; + } + +strbuf_reset (&path); +strbuf_add (&path, w->dir,strlen (w->dir)); +strbuf_addch (&path,'/' ); +strbuf_addstr (&path, event->name); + + p =fsmonitor__resolve_alias (path.buf, &state->alias); +if (!p) + p =strbuf_detach (&path,NULL ); + +if (!batch) + batch =fsmonitor_batch__new (); + +if (process_event (p, event, batch, &cookie_list, state)) { +free (p); + goto done; + } +free (p); + } +strbuf_reset (&path); +fsmonitor_publish (state, batch, &cookie_list); +string_list_clear (&cookie_list,0 ); + batch =NULL ; + } +done: +strbuf_release (&path); +fsmonitor_batch__free_list (batch); +string_list_clear (&cookie_list,0 ); +} + +/* +* Non-blocking read of the inotify events stream. The inotify fd is polled +* frequently to help minimize the number of queue overflows. +*/ +void fsm_listen__loop (struct fsmonitor_daemon_state *state) +{ +int poll_num; +const int interval =1000 ; +time_t checked =time (NULL ); +struct pollfd fds[1 ]; + fds[0 ].fd = state->listen_data->fd_inotify; + fds[0 ].events =POLLIN ; + +for (;;) { +switch (state->listen_data->shutdown) { +case SHUTDOWN_CONTINUE : + poll_num =poll (fds,1 ,1 ); +if (poll_num ==-1 ) { +if (errno ==EINTR ) +continue ; +error_errno (_ ("polling inotify message stream failed" )); + state->listen_data->shutdown =SHUTDOWN_ERROR ; +continue ; + } + +if ((time (NULL ) - checked) >= interval) { + checked =time (NULL ); +if (check_stale_dir_renames (&state->listen_data->renames, + checked - interval)) { +trace_printf_key (&trace_fsmonitor, +"Missed IN_MOVED_TO events, forcing shutdown" ); + state->listen_data->shutdown =SHUTDOWN_FORCE ; +continue ; + } + } + +if (poll_num >0 && (fds[0 ].revents &POLLIN )) +handle_events (state); + +continue ; +case SHUTDOWN_ERROR : + state->listen_error_code =-1 ; +ipc_server_stop_async (state->ipc_server_data); +break ; +case SHUTDOWN_FORCE : + state->listen_error_code =0 ; +ipc_server_stop_async (state->ipc_server_data); +break ; +case SHUTDOWN_STOP : +default : + state->listen_error_code =0 ; +break ; + } +return ; + } +}
Signature · 1 line
gitgitgadget
PATCH v5 5/6 fsmonitor: test updates
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
t7527-builtin-fsmonitor was leaking fsmonitor--daemon processes in some cases.
Accomodate slight difference in the number of events generated on Linux.
On lower-powered systems, spin a little to give the daemon time to respond to and log filesystem events.
--- t/t7527-builtin-fsmonitor.sh | 94 ++++++++++++++++++++++++++++++------ 1 file changed, 80 insertions(+), 14 deletions(-)
t/t7527-builtin-fsmonitor.sh +80 -14
@@ -13,7 +13,7 @@ fi stop_daemon_delete_repo () { r=$1 && test_might_fail git -C $r fsmonitor--daemon stop && - rm -rf $1 + rm -rf $r } start_daemon () {
@@ -72,6 +72,34 @@ start_daemon () { ) } +IMPLICIT_TIMEOUT=5 + +wait_for_update () { + func=$1 && + file=$2 && + sz=$(wc -c < "$file") && + last=0 && + $func && + k=0 && + while test "$k" -lt $IMPLICIT_TIMEOUT + do + nsz=$(wc -c < "$file") + if test "$nsz" -gt "$sz" + then + if test "$last" -eq "$nsz" + then + cat "$file" && + return 0 + fi + last=$nsz + fi + sleep 1 + k=$(( $k + 1 )) + done && + cat "$file" && + return 0 +} + # Is a Trace2 data event present with the given catetory and key? # We do not care what the value is. #
@@ -137,7 +165,6 @@ test_expect_success 'implicit daemon start' ' # machines (where it might take a moment to wake and reschedule the # daemon process) to avoid false alarms during test runs.) # -IMPLICIT_TIMEOUT=5 verify_implicit_shutdown () { r=$1 &&
@@ -373,6 +400,15 @@ create_files () { echo 3 >dir2/new } +rename_directory () { + mv dirtorename dirrenamed +} + +rename_directory_file () { + mv dirtorename dirrenamed && + echo 1 > dirrenamed/new +} + rename_files () { mv rename renamed && mv dir1/rename dir1/renamed &&
@@ -427,10 +463,12 @@ test_expect_success 'edit some files' ' start_daemon --tf "$PWD/.git/trace" && - edit_files && + wait_for_update edit_files "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 && + test_might_fail git fsmonitor--daemon stop && + grep "^event: dir1/modified$" .git/trace && grep "^event: dir2/modified$" .git/trace && grep "^event: modified$" .git/trace &&
@@ -442,10 +480,12 @@ test_expect_success 'create some files' ' start_daemon --tf "$PWD/.git/trace" && - create_files && + wait_for_update create_files "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 && + test_might_fail git fsmonitor--daemon stop && + grep "^event: dir1/new$" .git/trace && grep "^event: dir2/new$" .git/trace && grep "^event: new$" .git/trace
@@ -456,10 +496,12 @@ test_expect_success 'delete some files' ' start_daemon --tf "$PWD/.git/trace" && - delete_files && + wait_for_update delete_files "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 && + test_might_fail git fsmonitor--daemon stop && + grep "^event: dir1/delete$" .git/trace && grep "^event: dir2/delete$" .git/trace && grep "^event: delete$" .git/trace
@@ -470,10 +512,12 @@ test_expect_success 'rename some files' ' start_daemon --tf "$PWD/.git/trace" && - rename_files && + wait_for_update rename_files "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 && + test_might_fail git fsmonitor--daemon stop && + grep "^event: dir1/rename$" .git/trace && grep "^event: dir2/rename$" .git/trace && grep "^event: rename$" .git/trace &&
@@ -487,23 +531,42 @@ test_expect_success 'rename directory' ' start_daemon --tf "$PWD/.git/trace" && - mv dirtorename dirrenamed && + wait_for_update rename_directory "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 && + test_might_fail git fsmonitor--daemon stop && + grep "^event: dirtorename/*$" .git/trace && grep "^event: dirrenamed/*$" .git/trace ' +test_expect_success 'rename directory file' ' + test_when_finished clean_up_repo_and_stop_daemon && + + start_daemon --tf "$PWD/.git/trace" && + + wait_for_update rename_directory_file "$PWD/.git/trace" && + + test-tool fsmonitor-client query --token 0 && + + test_might_fail git fsmonitor--daemon stop && + + grep "^event: dirtorename/*$" .git/trace && + grep "^event: dirrenamed/*$" .git/trace && + grep "^event: dirrenamed/new$" .git/trace +' test_expect_success 'file changes to directory' ' test_when_finished clean_up_repo_and_stop_daemon && start_daemon --tf "$PWD/.git/trace" && - file_to_directory && + wait_for_update file_to_directory "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 && + test_might_fail git fsmonitor--daemon stop && + grep "^event: delete$" .git/trace && grep "^event: delete/new$" .git/trace '
@@ -513,10 +576,12 @@ test_expect_success 'directory changes to a file' ' start_daemon --tf "$PWD/.git/trace" && - directory_to_file && + wait_for_update directory_to_file "$PWD/.git/trace" && test-tool fsmonitor-client query --token 0 && + test_might_fail git fsmonitor--daemon stop && + grep "^event: dir1$" .git/trace '
@@ -561,7 +626,7 @@ test_expect_success 'flush cached data' ' test-tool -C test_flush fsmonitor-client query --token "builtin:test_00000002:0" >actual_2 && nul_to_q <actual_2 >actual_q2 && - grep "^builtin:test_00000002:0Q$" actual_q2 && + grep "^builtin:test_00000002:[0-1]Q$" actual_q2 && >test_flush/file_3 &&
@@ -732,7 +797,8 @@ u_values="$u1 $u2" for u in $u_values do test_expect_success "unicode in repo root path: $u" ' - test_when_finished "stop_daemon_delete_repo $u" && + test_when_finished \ + "stop_daemon_delete_repo `echo "$u" | sed 's:x:\\\\\\\\\\\\\\x:g'`" && git init "$u" && echo 1 >"$u"/file1 &&
@@ -818,8 +884,7 @@ test_expect_success 'submodule setup' ' ' test_expect_success 'submodule always visited' ' - test_when_finished "git -C super fsmonitor--daemon stop; \ - rm -rf super; \ + test_when_finished "rm -rf super; \ rm -rf sub" && create_super super &&
@@ -887,7 +952,8 @@ have_t2_error_event () { } test_expect_success "stray submodule super-prefix warning" ' - test_when_finished "rm -rf super; \ + test_when_finished "git -C super/dir_1/dir_2/sub fsmonitor--daemon stop; \ + rm -rf super; \ rm -rf sub; \ rm super-sub.trace" &&
Signature · 1 line
gitgitgadget
PATCH v5 6/6 fsmonitor: update doc for Linux
In reply to Eric DeCosta via GitGitGadgetFrom: Eric DeCosta <[email protected]>
Update the documentation for Linux.
--- Documentation/config/fsmonitor--daemon.txt | 4 ++-- Documentation/git-fsmonitor--daemon.txt | 24 ++++++++++++++-------- 2 files changed, 18 insertions(+), 10 deletions(-)
Documentation/config/fsmonitor--daemon.txt +2 -2
@@ -4,8 +4,8 @@ fsmonitor.allowRemote:: behavior. Only respected when `core.fsmonitor` is set to `true`. fsmonitor.socketDir:: - This Mac OS-specific option, if set, specifies the directory in + Mac OS and Linux-specific option. If set, specifies the directory in which to create the Unix domain socket used for communication between the fsmonitor daemon and various Git commands. The directory must - reside on a native Mac OS filesystem. Only respected when `core.fsmonitor` + reside on a native filesystem. Only respected when `core.fsmonitor` is set to `true`.
Documentation/git-fsmonitor--daemon.txt +16 -8
@@ -76,23 +76,31 @@ repositories; this may be overridden by setting `fsmonitor.allowRemote` to correctly with all network-mounted repositories and such use is considered experimental. -On Mac OS, the inter-process communication (IPC) between various Git +On Linux and Mac OS, the inter-process communication (IPC) between various Git commands and the fsmonitor daemon is done via a Unix domain socket (UDS) -- a -special type of file -- which is supported by native Mac OS filesystems, -but not on network-mounted filesystems, NTFS, or FAT32. Other filesystems -may or may not have the needed support; the fsmonitor daemon is not guaranteed -to work with these filesystems and such use is considered experimental. +special type of file -- which is supported by many native Linux and Mac OS +filesystems, but not on network-mounted filesystems, NTFS, or FAT32. Other +filesystems may or may not have the needed support; the fsmonitor daemon is not +guaranteed to work with these filesystems and such use is considered +experimental. By default, the socket is created in the `.git` directory, however, if the `.git` directory is on a network-mounted filesystem, it will be instead be created at `$HOME/.git-fsmonitor-*` unless `$HOME` itself is on a network-mounted filesystem in which case you must set the configuration -variable `fsmonitor.socketDir` to the path of a directory on a Mac OS native +variable `fsmonitor.socketDir` to the path of a directory on a native filesystem in which to create the socket file. If none of the above directories (`.git`, `$HOME`, or `fsmonitor.socketDir`) -is on a native Mac OS file filesystem the fsmonitor daemon will report an -error that will cause the daemon and the currently running command to exit. +is on a native Linux or Mac OS filesystem the fsmonitor daemon will report +an error that will cause the daemon to exit and the currently running command +to issue a warning. + +On Linux, the fsmonitor daemon registers a watch for each directory in the +repository. The default per-user limit for the number of watches on most Linux +systems is 8192. This may not be sufficient for large repositories or if +multiple instances of the fsmonitor daemon are running. +See https://watchexec.github.io/docs/inotify-limits.html[Linux inotify limits] for more information. CONFIGURATION -------------
Signature · 1 line
gitgitgadget
Quoted 10 lines · Eric DeCosta via GitGitGadget
Goal is to deliver fsmonitor for Linux that is on par with fsmonitor for Windows and Mac OS.
This patch set builds upon previous work for done for Windows and Mac OS to implement a fsmonitor back-end for Linux based on the Linux inotify API. inotify differs significantly from the equivalent Windows and Mac OS APIs in that a watch must be registered for every directory of interest (rather than a singular watch at the root of the directory tree) and special care must be taken to handle directory renames correctly.
Quoted 6 lines
More information about inotify:
https://man7.org/linux/man-pages/man7/inotify.7.htmlv4 differs from v3:
* Code review feedback
This has been dormant for full four months, and it seems to break linux-asan CI job when merged to 'seen',
https://github.com/git/git/actions/runs/4672116751/jobs/8273938089
while the same 'seen' excluding this topic passes everything.
https://github.com/git/git/actions/runs/4674694371
For now, I'll drop this topic from 'seen'.
Thanks.