Builtin FSMonitor Feature
Subsystems (2)
v2 cover Builtin FSMonitor Feature
In reply to Jeff Hostetler via GitGitGadgetHere is V2 of my patch series to add a builtin FSMonitor daemon to Git.
V2 includes addresses most of the review comments from the RFC and many of the issues seen during out beta-testing with selected GVFS users. However, there are still a few items that I need to address:
[ ] Revisit the how the client handles the IPC_STATE__NOT_LISTENING state (where a daemon appears to be running, but is non-responsive) [ ] Revisit use of global core_fsmonitor as both a pathname and a boolean. The existing fsmonitor code uses it as the pathname to the fsmonitor hook and as a flag to indicate that a hook is configured. [ ] Consider having daemon chdir() out of the working directory to avoid directory handle issues on Windows. [ ] Some documentation recommendations. [ ] Split up the commit containing the tests and move some earlier in patch series. [ ] Move my FSMonitor PREREQ to test-lib.sh instead of having it in my test scripts. [ ] Document performance gains. [ ] On Windows, If the daemon is started as an elevated process, then client commands might not have access to communicate with it. [ ] Review if/how we decide to shutdown the FSMonitor daemon after and a significant idle period. [ ] Investigate ways to temporarily shutdown FSMonitor daemons processes so that the Git for Windows installer can install an upgrade.
In this version, the first commit updates the Simple IPC API to make it easier to pass binary data using {char *, size_t} rather than assuming that the message is a null-terminated string. FSMonitor does not use binary messages and doesn't really need this API change, but I thought it best to fix the API now before we have other callers of IPC.
This V2 version will hopefully be previewed as an experimental feature in Git for Windows v2.32.0.windows.*.
Jeff Hostetler (26):
simple-ipc: preparations for supporting binary messages.
fsmonitor--daemon: man page
fsmonitor--daemon: update fsmonitor documentation
fsmonitor-ipc: create client routines for git-fsmonitor--daemon
help: include fsmonitor--daemon feature flag in version info
fsmonitor--daemon: add a built-in fsmonitor daemon
fsmonitor--daemon: implement client command options t/helper/fsmonitor-client: create IPC client to talk to FSMonitor
Daemon
fsmonitor-fs-listen-win32: stub in backend for Windows
fsmonitor-fs-listen-macos: stub in backend for MacOS
fsmonitor--daemon: implement daemon command options
fsmonitor--daemon: add pathname classification
fsmonitor--daemon: define token-ids
fsmonitor--daemon: create token-based changed path cache
fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows
fsmonitor-fs-listen-macos: add macos header files for FSEvent
fsmonitor-fs-listen-macos: implement FSEvent listener on MacOS
fsmonitor--daemon: implement handle_client callback
fsmonitor--daemon: periodically truncate list of modified files
fsmonitor--daemon: use a cookie file to sync with file system
fsmonitor: enhance existing comments
fsmonitor: force update index after large responses
t7527: create test for fsmonitor--daemon
p7519: add fsmonitor--daemon
t7527: test status with untracked-cache and fsmonitor--daemon t/perf: avoid copying builtin fsmonitor files into test repo
Johannes Schindelin (2):
config: FSMonitor is repository-specific
fsmonitor: introduce core.useBuiltinFSMonitor to call the daemon via
IPC
.gitignore | 1 + Documentation/config/core.txt | 56 +- Documentation/git-fsmonitor--daemon.txt | 75 + Documentation/git-update-index.txt | 27 +- Documentation/githooks.txt | 3 +- Makefile | 16 + builtin.h | 1 + builtin/fsmonitor--daemon.c | 1511 ++++++++++++++++++ builtin/update-index.c | 4 +- compat/fsmonitor/fsmonitor-fs-listen-macos.c | 497 ++++++ compat/fsmonitor/fsmonitor-fs-listen-win32.c | 553 +++++++ compat/fsmonitor/fsmonitor-fs-listen.h | 49 + compat/simple-ipc/ipc-unix-socket.c | 14 +- compat/simple-ipc/ipc-win32.c | 14 +- config.c | 9 +- config.h | 2 +- config.mak.uname | 4 + contrib/buildsystems/CMakeLists.txt | 8 + fsmonitor--daemon.h | 140 ++ fsmonitor-ipc.c | 179 +++ fsmonitor-ipc.h | 48 + fsmonitor.c | 132 +- git.c | 1 + help.c | 4 + repo-settings.c | 3 + repository.h | 2 + simple-ipc.h | 7 +- t/helper/test-fsmonitor-client.c | 125 ++ t/helper/test-simple-ipc.c | 34 +- t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + t/perf/p7519-fsmonitor.sh | 42 +- t/perf/perf-lib.sh | 2 +- t/t7527-builtin-fsmonitor.sh | 572 +++++++ 34 files changed, 4069 insertions(+), 68 deletions(-) create mode 100644 Documentation/git-fsmonitor--daemon.txt create mode 100644 builtin/fsmonitor--daemon.c create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-macos.c create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-win32.c create mode 100644 compat/fsmonitor/fsmonitor-fs-listen.h create mode 100644 fsmonitor--daemon.h create mode 100644 fsmonitor-ipc.c create mode 100644 fsmonitor-ipc.h create mode 100644 t/helper/test-fsmonitor-client.c create mode 100755 t/t7527-builtin-fsmonitor.sh
base-commit: b0c09ab8796fb736efa432b8e817334f3e5ee75a
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-923%2Fjeffhostetler%2Fbuiltin-fsmonitor-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-923/jeffhostetler/builtin-fsmonitor-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/923
Range-diff vs v1:
-: ------------ > 1: 763fa1ee7bb6 simple-ipc: preparations for supporting binary messages. -: ------------ > 2: fc180e8591bf fsmonitor--daemon: man page 1: 074273330f8d ! 3: d56f3e91db9f fsmonitor--daemon: man page and documentation
@@ Metadata
Author: Jeff Hostetler <[email protected]>
## Commit message ##
- fsmonitor--daemon: man page and documentation
+ fsmonitor--daemon: update fsmonitor documentation
- Create a manual page describing the git fsmonitor--daemon feature.
-
- Update references to core.fsmonitor, core.fsmonitorHookVersion and
- pointers to watchman to mention the built-in FSMonitor.
+ Update references to core.fsmonitor and core.fsmonitorHookVersion and
+ pointers to Watchman to mention the new built-in fsmonitor--daemon.
## Documentation/config/core.txt ##
-@@ Documentation/config/core.txt: core.fsmonitor::
- will identify all files that may have changed since the
- requested date/time. This information is used to speed up git by
- avoiding unnecessary processing of files that have not changed.
+@@ Documentation/config/core.txt: core.protectNTFS::
+ Defaults to true on Windows, and false elsewhere.
+
+ core.fsmonitor::
+- If set, the value of this variable is used as a command which
+- will identify all files that may have changed since the
+- requested date/time. This information is used to speed up git by
+- avoiding unnecessary processing of files that have not changed.
- See the "fsmonitor-watchman" section of linkgit:githooks[5].
++ If set, this variable contains the pathname of the "fsmonitor"
++ hook command.
+++
++This hook command is used to identify all files that may have changed ++since the requested date/time. This information is used to speed up ++git by avoiding unnecessary scanning of files that have not changed.
++
+See the "fsmonitor-watchman" section of linkgit:githooks[5]. ++
-+Note: FSMonitor hooks (and this config setting) are ignored if the -+built-in FSMonitor is enabled (see core.useBuiltinFSMonitor). ++Note: The value of this config setting is ignored if the ++built-in file system monitor is enabled (see core.useBuiltinFSMonitor).
core.fsmonitorHookVersion::
- Sets the version of hook that is to be used when calling fsmonitor.
@@ Documentation/config/core.txt: core.fsmonitor::
- Version 2 uses an opaque string so that the monitor can return
- something that can be used to determine what files have changed
- without race conditions.
-+ Sets the version of hook that is to be used when calling the
-+ FSMonitor hook (as configured via core.fsmonitor).
++ Sets the protocol version to be used when invoking the
++ "fsmonitor" hook.
++
+There are currently versions 1 and 2. When this is not set, +version 2 will be tried first and if it fails then version 1 +will be tried. Version 1 uses a timestamp as input to determine +which files have changes since that time but some monitors
-+like watchman have race conditions when used with a timestamp. ++like Watchman have race conditions when used with a timestamp.
+Version 2 uses an opaque string so that the monitor can return +something that can be used to determine what files have changed +without race conditions.
++
-+Note: FSMonitor hooks (and this config setting) are ignored if the -+built-in FSMonitor is enabled (see core.useBuiltinFSMonitor). ++Note: The value of this config setting is ignored if the ++built-in file system monitor is enabled (see core.useBuiltinFSMonitor).
+
+core.useBuiltinFSMonitor::
-+ If set to true, enable the built-in filesystem event watcher (for
-+ technical details, see linkgit:git-fsmonitor--daemon[1]).
++ If set to true, enable the built-in file system monitor
++ daemon for this working directory (linkgit:git-fsmonitor--daemon[1]).
++
-+Like external (hook-based) FSMonitors, the built-in FSMonitor can speed up -+Git commands that need to refresh the Git index (e.g. git status) in a -+worktree with many files. The built-in FSMonitor facility eliminates the -+need to install and maintain an external third-party monitoring tool. ++Like hook-based file system monitors, the built-in file system monitor ++can speed up Git commands that need to refresh the Git index ++(e.g. git status) in a working directory with many files. The ++built-in monitor eliminates the need to install and maintain an ++external third-party tool.
++
-+The built-in FSMonitor is currently available only on a limited set of -+supported platforms.
++The built-in file system monitor is currently available only on a ++limited set of supported platforms. Currently, this includes Windows ++and MacOS.
++
-+Note: if this config setting is set to true, any FSMonitor hook -+configured via core.fsmonitor (and possibly core.fsmonitorHookVersion) -+is ignored.
++Note: if this config setting is set to true, the values of ++core.fsmonitor and core.fsmonitorHookVersion are ignored.
core.trustctime::
If false, the ctime differences between the index and the
- ## Documentation/git-fsmonitor--daemon.txt (new) ##
-@@
-+git-fsmonitor--daemon(1)
-+========================
-+
-+NAME
-+----
-+git-fsmonitor--daemon - Builtin file system monitor daemon
-+
-+SYNOPSIS
-+--------
-+[verse]
-+'git fsmonitor--daemon' --start
-+'git fsmonitor--daemon' --run
-+'git fsmonitor--daemon' --stop
-+'git fsmonitor--daemon' --is-running
-+'git fsmonitor--daemon' --is-supported
-+'git fsmonitor--daemon' --query <token>
-+'git fsmonitor--daemon' --query-index
-+'git fsmonitor--daemon' --flush
-+
-+DESCRIPTION
-+-----------
-+
-+Monitors files and directories in the working directory for changes using -+platform-specific file system notification facilities.
-+
-+It communicates directly with commands like git status using the -+link:technical/api-simple-ipc.html[simple IPC] interface instead of
-+the slower linkgit:githooks[5] interface.
-+
-+OPTIONS
-+-------
-+
-+--start::
-+ Starts the fsmonitor daemon in the background.
-+
-+--run::
-+ Runs the fsmonitor daemon in the foreground.
-+
-+--stop::
-+ Stops the fsmonitor daemon running for the current working
-+ directory, if present.
-+
-+--is-running::
-+ Exits with zero status if the fsmonitor daemon is watching the
-+ current working directory.
-+
-+--is-supported::
-+ Exits with zero status if the fsmonitor daemon feature is supported
-+ on this platform.
-+
-+--query <token>::
-+ Connects to the fsmonitor daemon (starting it if necessary) and
-+ requests the list of changed files and directories since the
-+ given token.
-+ This is intended for testing purposes.
-+
-+--query-index::
-+ Read the current <token> from the File System Monitor index
-+ extension (if present) and use it to query the fsmonitor daemon.
-+ This is intended for testing purposes.
-+
-+--flush::
-+ Force the fsmonitor daemon to flush its in-memory cache and
-+ re-sync with the file system.
-+ This is intended for testing purposes.
-+
-+REMARKS
-+-------
-+The fsmonitor daemon is a long running process that will watch a single
-+working directory. Commands, such as git status, should automatically
-+start it (if necessary) when core.useBuiltinFSMonitor is set to true
-+(see linkgit:git-config[1]).
-+
-+Configure the built-in FSMonitor via core.useBuiltinFSMonitor in each
-+working directory separately, or globally via git config --global -+core.useBuiltinFSMonitor true.
-+
-+Tokens are opaque strings. They are used by the fsmonitor daemon to
-+mark a point in time and the associated internal state. Callers should
-+make no assumptions about the content of the token. In particular,
-+the should not assume that it is a timestamp.
-+
-+Query commands send a request-token to the daemon and it responds with
-+a summary of the changes that have occurred since that token was
-+created. The daemon also returns a response-token that the client can
-+use in a future query.
-+
-+For more information see the "File System Monitor" section in
-+linkgit:git-update-index[1].
-+
-+CAVEATS
-+-------
-+
-+The fsmonitor daemon does not currently know about submodules and does
-+not know to filter out file system events that happen within a
-+submodule. If fsmonitor daemon is watching a super repo and a file is
-+modified within the working directory of a submodule, it will report
-+the change (as happening against the super repo). However, the client
-+should properly ignore these extra events, so performance may be affected
-+but it should not cause an incorrect result.
-+
-+GIT
-+---
-+Part of the linkgit:git[1] suite
-
## Documentation/git-update-index.txt ##
@@ Documentation/git-update-index.txt: FILE SYSTEM MONITOR
This feature is intended to speed up git operations for repos that have
@@ Documentation/git-update-index.txt: FILE SYSTEM MONITOR
"fsmonitor-watchman" section of linkgit:githooks[5]) that can
inform it as to what files have been modified. This enables git to avoid
having to lstat() every file to find modified files.
+@@ Documentation/git-update-index.txt: performance by avoiding the cost of scanning the entire working directory
+ looking for new files.
+
+ If you want to enable (or disable) this feature, it is easier to use
+-the core.fsmonitor configuration variable (see
+-linkgit:git-config[1]) than using the --fsmonitor option to
+-git update-index in each repository, especially if you want to do so
+-across all repositories you use, because you can set the configuration
+-variable in your $HOME/.gitconfig just once and have it affect all
+-repositories you touch.
+-
+-When the core.fsmonitor configuration variable is changed, the
+-file system monitor is added to or removed from the index the next time
+-a command reads the index. When --[no-]fsmonitor are used, the file
+-system monitor is immediately added to or removed from the index.
++the core.fsmonitor or core.useBuiltinFSMonitor configuration
++variable (see linkgit:git-config[1]) than using the --fsmonitor
++option to git update-index in each repository, especially if you
++want to do so across all repositories you use, because you can set the
++configuration variable in your $HOME/.gitconfig just once and have
++it affect all repositories you touch.
++
++When the core.fsmonitor or core.useBuiltinFSMonitor configuration
++variable is changed, the file system monitor is added to or removed
++from the index the next time a command reads the index. When
++--[no-]fsmonitor are used, the file system monitor is immediately
++added to or removed from the index.
+
+ CONFIGURATION
+ -------------
## Documentation/githooks.txt ##
@@ Documentation/githooks.txt: fsmonitor-watchman
2: 3dac63eae201 ! 4: e4a263728773 fsmonitor-ipc: create client routines for git-fsmonitor--daemon
@@ Metadata
## Commit message ##
fsmonitor-ipc: create client routines for git-fsmonitor--daemon
- Create client routines to spawn a fsmonitor daemon and send it an IPC
- request using `simple-ipc`.
+ Create fsmonitor_ipc__*() client routines to spawn the built-in file
+ system monitor daemon and send it an IPC request using the `Simple
+ IPC` API.
+
+ Stub in empty fsmonitor_ipc__*() functions for unsupported platforms.
Signed-off-by: Jeff Hostetler <[email protected]>
@@ fsmonitor-ipc.c (new)
@@
+#include "cache.h"
+#include "fsmonitor.h"
++#include "simple-ipc.h"
+#include "fsmonitor-ipc.h"
+#include "run-command.h"
+#include "strbuf.h"
+#include "trace2.h"
+
+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND
-+#define FSMONITOR_DAEMON_IS_SUPPORTED 1
-+#else
-+#define FSMONITOR_DAEMON_IS_SUPPORTED 0
-+#endif
+
-+/*
-+ * A trivial function so that this source file always defines at least
-+ * one symbol even when the feature is not supported. This quiets an
-+ * annoying compiler error.
-+ */
+int fsmonitor_ipc__is_supported(void)
+{
-+ return FSMONITOR_DAEMON_IS_SUPPORTED;
++ return 1;
+}
+
-+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND
-+
-+GIT_PATH_FUNC(fsmonitor_ipc__get_path, "fsmonitor")
++GIT_PATH_FUNC(fsmonitor_ipc__get_path, "fsmonitor--daemon.ipc")
+
+enum ipc_active_state fsmonitor_ipc__get_state(void)
+{
@@ fsmonitor-ipc.c (new)
+
+static int spawn_daemon(void)
+{
-+ const char *args[] = { "fsmonitor--daemon", "--start", NULL };
++ const char *args[] = { "fsmonitor--daemon", "start", NULL };
+
+ return run_command_v_opt_tr2(args, RUN_COMMAND_NO_STDIN | RUN_GIT_CMD,
+ "fsmonitor");
@@ fsmonitor-ipc.c (new)
+ switch (state) {
+ case IPC_STATE__LISTENING:
+ ret = ipc_client_send_command_to_connection(
-+ connection, since_token, answer);
++ connection, since_token, strlen(since_token), answer);
+ ipc_client_close_connection(connection);
+
+ trace2_data_intmax("fsm_client", NULL,
@@ fsmonitor-ipc.c (new)
+ return -1;
+ }
+
-+ ret = ipc_client_send_command_to_connection(connection, command, answer);
++ ret = ipc_client_send_command_to_connection(connection,
++ command, strlen(command),
++ answer);
+ ipc_client_close_connection(connection);
+
+ if (ret == -1) {
@@ fsmonitor-ipc.c (new)
+ return 0;
+}
+
++#else
++
++/*
++ * A trivial implementation of the fsmonitor_ipc__ API for unsupported
++ * platforms.
++ */
++
++int fsmonitor_ipc__is_supported(void)
++{
++ return 0;
++}
++
++const char *fsmonitor_ipc__get_path(void)
++{
++ return NULL;
++}
++
++enum ipc_active_state fsmonitor_ipc__get_state(void)
++{
++ return IPC_STATE__OTHER_ERROR;
++}
++
++int fsmonitor_ipc__send_query(const char *since_token,
++ struct strbuf *answer)
++{
++ return -1;
++}
++
++int fsmonitor_ipc__send_command(const char *command,
++ struct strbuf *answer)
++{
++ return -1;
++}
++
+#endif
## fsmonitor-ipc.h (new) ##
@@ fsmonitor-ipc.h (new)
+#define FSMONITOR_IPC_H
+
+/*
-+ * Returns true if a filesystem notification backend is defined
-+ * for this platform. This symbol must always be visible and
-+ * outside of the HAVE_ ifdef.
++ * Returns true if built-in file system monitor daemon is defined
++ * for this platform.
+ */
+int fsmonitor_ipc__is_supported(void);
+
-+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND
-+#include "run-command.h"
-+#include "simple-ipc.h"
-+
+/*
+ * Returns the pathname to the IPC named pipe or Unix domain socket
+ * where a `git-fsmonitor--daemon` process will listen. This is a
+ * per-worktree value.
++ *
++ * Returns NULL if the daemon is not supported on this platform.
+ */
+const char *fsmonitor_ipc__get_path(void);
+
@@ fsmonitor-ipc.h (new)
+ * This DOES NOT use the hook interface.
+ *
+ * Spawn a daemon process in the background if necessary.
++ *
++ * Returns -1 on error; 0 on success.
+ */
+int fsmonitor_ipc__send_query(const char *since_token,
+ struct strbuf *answer);
@@ fsmonitor-ipc.h (new)
+ * Connect to a `git-fsmonitor--daemon` process via simple-ipc and
+ * send a command verb. If no daemon is available, we DO NOT try to
+ * start one.
++ *
++ * Returns -1 on error; 0 on success.
+ */
+int fsmonitor_ipc__send_command(const char *command,
+ struct strbuf *answer);
+
-+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */
+#endif /* FSMONITOR_IPC_H */
-
- ## help.c ##
-@@
- #include "version.h"
- #include "refs.h"
- #include "parse-options.h"
-+#include "fsmonitor-ipc.h"
-
- struct category_description {
- uint32_t category;
-@@ help.c: void get_version_info(struct strbuf *buf, int show_build_options)
- strbuf_addf(buf, "sizeof-size_t: %d\n", (int)sizeof(size_t));
- strbuf_addf(buf, "shell-path: %s\n", SHELL_PATH);
- /* NEEDSWORK: also save and output GIT-BUILD_OPTIONS? */
-+
-+ if (fsmonitor_ipc__is_supported())
-+ strbuf_addstr(buf, "feature: fsmonitor--daemon\n");
- }
- }
-
-: ------------ > 5: d5d09eb1635b help: include fsmonitor--daemon feature flag in version info
3: 18c125ec73dc ! 6: 67bcf57f5948 config: FSMonitor is repository-specific @@ Commit message
This commit refactors git_config_get_fsmonitor() into the repo_*() form that takes a parameter struct repository *r.
- That change prepares for the upcoming core.useFSMonitorDaemon flag which
+ That change prepares for the upcoming core.useBuiltinFSMonitor flag which
will be stored in the repo_settings struct.
4: 7082528d8f7c ! 7: 7e097cebc143 fsmonitor: introduce core.useBuiltinFSMonitor to call the daemon via IPC
@@ Metadata
## Commit message ##
fsmonitor: introduce core.useBuiltinFSMonitor to call the daemon via IPC
- The core.fsmonitor setting is supposed to be set to a path pointing
- to a script or executable that (via the Hook API) queries an fsmonitor
- process such as watchman.
+ Use simple IPC to directly communicate with the new builtin file
+ system monitor daemon.
- We are about to implement our own fsmonitor backend, and do not want
- to spawn hook processes just to query it. Let's use Simple IPC to
- directly communicate with the daemon (and start it if necessary),
- guarded by the brand-new core.useBuiltinFSMonitor toggle.
+ Define a new config setting core.useBuiltinFSMonitor to enable the
+ builtin file system monitor.
+
+ The core.fsmonitor setting has already been defined as a HOOK
+ pathname. Historically, this has been set to a HOOK script that will
+ talk with Watchman. For compatibility reasons, we do not want to
+ overload that definition (and cause problems if users have multiple
+ versions of Git installed).
@@ fsmonitor.c
#include "run-command.h"
#include "strbuf.h"
-@@ fsmonitor.c: void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)
- /*
- * Call the query-fsmonitor hook passing the last update token of the saved results.
- */
--static int query_fsmonitor(int version, const char *last_update, struct strbuf *query_result) -+static int query_fsmonitor(int version, struct index_state *istate, struct strbuf *query_result) +@@ fsmonitor.c: static void fsmonitor_refresh_callback(struct index_state *istate, char *name) +
+ void refresh_fsmonitor(struct index_state *istate)
{
+ struct repository *r = istate->repo ? istate->repo : the_repository;
-+ const char *last_update = istate->fsmonitor_last_update;
- struct child_process cp = CHILD_PROCESS_INIT;
- int result;
-
- if (!core_fsmonitor)
- return -1;
+ struct strbuf query_result = STRBUF_INIT;
+ int query_success = 0, hook_version = -1;
+ size_t bol = 0; /* beginning of line */
+@@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)
+ istate->fsmonitor_has_run_once = 1;
+ trace_printf_key(&trace_fsmonitor, "refresh fsmonitor");
++
+ if (r->settings.use_builtin_fsmonitor > 0) {
-+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND
-+ return fsmonitor_ipc__send_query(last_update, query_result);
-+#else
-+ /* Fake a trivial response. */
-+ warning(_("fsmonitor--daemon unavailable; falling back"));
-+ strbuf_add(query_result, "/", 2);
-+ return 0;
-+#endif
++ query_success = !fsmonitor_ipc__send_query(
++ istate->fsmonitor_last_update, &query_result);
++ if (query_success) {
++ /*
++ * The response contains a series of nul terminated
++ * strings. The first is the new token.
++ *
++ * Use char *buf as an interlude to trick the CI
++ * static analysis to let us use strbuf_addstr()
++ * here (and only copy the token) rather than
++ * strbuf_addbuf().
++ */
++ buf = query_result.buf;
++ strbuf_addstr(&last_update_token, buf);
++ bol = last_update_token.len + 1;
++ } else {
++ /*
++ * The builtin daemon is not available on this
++ * platform -OR- we failed to get a response.
++ *
++ * Generate a fake token (rather than a V1
++ * timestamp) for the index extension. (If
++ * they switch back to the hook API, we don't
++ * want ambiguous state.)
++ */
++ strbuf_addstr(&last_update_token, "builtin:fake");
++ }
++
++ /*
++ * Regardless of whether we successfully talked to a
++ * fsmonitor daemon or not, we skip over and do not
++ * try to use the hook. The "core.useBuiltinFSMonitor"
++ * config setting ALWAYS overrides the "core.fsmonitor"
++ * hook setting.
++ */
++ goto apply_results;
+ }
+
- strvec_push(&cp.args, core_fsmonitor);
- strvec_pushf(&cp.args, "%d", version);
- strvec_pushf(&cp.args, "%s", last_update);
+ /*
+ * This could be racy so save the date/time now and query_fsmonitor
+ * should be inclusive to ensure we don't miss potential changes.
@@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)
- if (istate->fsmonitor_last_update) {
- if (hook_version == -1 || hook_version == HOOK_INTERFACE_VERSION2) {
- query_success = !query_fsmonitor(HOOK_INTERFACE_VERSION2,
-- istate->fsmonitor_last_update, &query_result);
-+ istate, &query_result);
+ core_fsmonitor, query_success ? "success" : "failure");
+ }
- if (query_success) {
- if (hook_version < 0)
-@@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate) -
- if (hook_version == HOOK_INTERFACE_VERSION1) {
- query_success = !query_fsmonitor(HOOK_INTERFACE_VERSION1,
-- istate->fsmonitor_last_update, &query_result);
-+ istate, &query_result);
- }
-
- trace_performance_since(last_update, "fsmonitor process '%s'", core_fsmonitor);
++apply_results:
+ /* a fsmonitor process can return '/' to indicate all entries are invalid */
+ if (query_success && query_result.buf[bol] != '/') {
+ /* Mark all entries returned by the monitor as dirty */
## repo-settings.c ##
@@ repo-settings.c: void prepare_repo_settings(struct repository *r)
@@ repo-settings.c: void prepare_repo_settings(struct repository *r)
UPDATE_DEFAULT_BOOL(r->settings.core_untracked_cache, UNTRACKED_CACHE_WRITE);
## repository.h ##
-@@ repository.h: struct repo_settings {
- enum fetch_negotiation_setting fetch_negotiation_algorithm;
+@@ repository.h: enum fetch_negotiation_setting {
+ struct repo_settings {
+ int initialized;
- int core_multi_pack_index;
-+
+ int use_builtin_fsmonitor;
- };
-
- struct repository {
++
+ int core_commit_graph;
+ int commit_graph_read_changed_paths;
+ int gc_write_commit_graph;
5: 95d511d83b12 ! 8: f362a88632e4 fsmonitor--daemon: add a built-in fsmonitor daemon
@@ builtin/fsmonitor--daemon.c (new)
+
+int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)
+{
-+ enum daemon_mode {
-+ UNDEFINED_MODE,
-+ } mode = UNDEFINED_MODE;
++ const char *subcmd;
+
+ struct option options[] = {
+ OPT_END()
+ };
+
++ if (argc < 2)
++ usage_with_options(builtin_fsmonitor__daemon_usage, options);
++
+ if (argc == 2 && !strcmp(argv[1], "-h"))
+ usage_with_options(builtin_fsmonitor__daemon_usage, options);
+
+ git_config(git_default_config, NULL);
+
++ subcmd = argv[1];
++ argv--;
++ argc++;
++
+ argc = parse_options(argc, argv, prefix, options,
+ builtin_fsmonitor__daemon_usage, 0);
+
-+ switch (mode) {
-+ case UNDEFINED_MODE:
-+ default:
-+ die(_("Unhandled command mode %d"), mode);
-+ }
++ die(_("Unhandled subcommand '%s'"), subcmd);
+}
+
+#else
6: 77170e521f67 ! 9: 4f401310539e fsmonitor--daemon: implement client command options
@@ Metadata
## Commit message ##
fsmonitor--daemon: implement client command options
- Implement command options `--stop`, `--is-running`, `--query`,
- `--query-index`, and `--flush` to control and query the status of a
- `fsmonitor--daemon` server process (and implicitly start a server
- process if necessary).
+ Implement `stop` and `status` client commands to control and query the
+ status of a `fsmonitor--daemon` server process (and implicitly start a
+ server process if necessary).
- Later commits will implement the actual server and monitor
- the file system.
+ Later commits will implement the actual server and monitor the file
+ system.
Signed-off-by: Jeff Hostetler <[email protected]>
@@ builtin/fsmonitor--daemon.c
#include "khash.h"
static const char * const builtin_fsmonitor__daemon_usage[] = {
-+ N_("git fsmonitor--daemon --stop"),
-+ N_("git fsmonitor--daemon --is-running"),
-+ N_("git fsmonitor--daemon --query <token>"),
-+ N_("git fsmonitor--daemon --query-index"),
-+ N_("git fsmonitor--daemon --flush"),
++ N_("git fsmonitor--daemon stop"),
++ N_("git fsmonitor--daemon status"),
NULL
};
@@ builtin/fsmonitor--daemon.c
+/*
+ * Acting as a CLIENT.
+ *
-+ * Send an IPC query to a `git-fsmonitor--daemon` SERVER process and
-+ * ask for the changes since the given token. This will implicitly
-+ * start a daemon process if necessary. The daemon process will
-+ * persist after we exit.
-+ *
-+ * This feature is primarily used by the test suite.
-+ */
-+static int do_as_client__query_token(const char *token)
-+{
-+ struct strbuf answer = STRBUF_INIT;
-+ int ret;
-+
-+ ret = fsmonitor_ipc__send_query(token, &answer);
-+ if (ret < 0)
-+ die(_("could not query fsmonitor--daemon"));
-+
-+ write_in_full(1, answer.buf, answer.len);
-+ strbuf_release(&answer);
-+
-+ return 0;
-+}
-+
-+/*
-+ * Acting as a CLIENT.
-+ *
-+ * Read the `.git/index` to get the last token written to the FSMonitor index
-+ * extension and use that to make a query.
-+ *
-+ * This feature is primarily used by the test suite.
-+ */
-+static int do_as_client__query_from_index(void)
-+{
-+ struct index_state *istate = the_repository->index;
-+
-+ setup_git_directory();
-+ if (do_read_index(istate, the_repository->index_file, 0) < 0)
-+ die("unable to read index file");
-+ if (!istate->fsmonitor_last_update)
-+ die("index file does not have fsmonitor extension");
-+
-+ return do_as_client__query_token(istate->fsmonitor_last_update);
-+}
-+
-+/*
-+ * Acting as a CLIENT.
-+ *
+ * Send a "quit" command to the `git-fsmonitor--daemon` (if running)
+ * and wait for it to shutdown.
+ */
@@ builtin/fsmonitor--daemon.c
+ return 0;
+}
+
-+/*
-+ * Acting as a CLIENT.
-+ *
-+ * Send a "flush" command to the `git-fsmonitor--daemon` (if running)
-+ * and tell it to flush its cache.
-+ *
-+ * This feature is primarily used by the test suite to simulate a loss of
-+ * sync with the filesystem where we miss kernel events.
-+ */
-+static int do_as_client__send_flush(void)
++static int do_as_client__status(void)
+{
-+ struct strbuf answer = STRBUF_INIT;
-+ int ret;
-+
-+ ret = fsmonitor_ipc__send_command("flush", &answer);
-+ if (ret)
-+ return ret;
-+
-+ write_in_full(1, answer.buf, answer.len);
-+ strbuf_release(&answer);
++ enum ipc_active_state state = fsmonitor_ipc__get_state();
+
-+ return 0;
-+}
++ switch (state) {
++ case IPC_STATE__LISTENING:
++ printf(_("The built-in file system monitor is active\n"));
++ return 0;
+
-+static int is_ipc_daemon_listening(void)
-+{
-+ return fsmonitor_ipc__get_state() == IPC_STATE__LISTENING;
++ default:
++ printf(_("The built-in file system monitor is not active\n"));
++ return 1;
++ }
+}
int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)
{
- enum daemon_mode {
- UNDEFINED_MODE,
-+ STOP,
-+ IS_RUNNING,
-+ QUERY,
-+ QUERY_INDEX,
-+ FLUSH,
- } mode = UNDEFINED_MODE;
-
- struct option options[] = {
-+ OPT_CMDMODE(0, "stop", &mode, N_("stop the running daemon"),
-+ STOP),
-+
-+ OPT_CMDMODE(0, "is-running", &mode,
-+ N_("test whether the daemon is running"),
-+ IS_RUNNING),
-+
-+ OPT_CMDMODE(0, "query", &mode,
-+ N_("query the daemon (starting if necessary)"),
-+ QUERY),
-+ OPT_CMDMODE(0, "query-index", &mode,
-+ N_("query the daemon (starting if necessary) using token from index"),
-+ QUERY_INDEX),
-+ OPT_CMDMODE(0, "flush", &mode, N_("flush cached filesystem events"),
-+ FLUSH),
- OPT_END()
- };
-
@@ builtin/fsmonitor--daemon.c: int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)
+ argc = parse_options(argc, argv, prefix, options,
builtin_fsmonitor__daemon_usage, 0);
- switch (mode) {
-+ case STOP:
++ if (!strcmp(subcmd, "stop"))
+ return !!do_as_client__send_stop();
+
-+ case IS_RUNNING:
-+ return !is_ipc_daemon_listening();
-+
-+ case QUERY:
-+ if (argc != 1)
-+ usage_with_options(builtin_fsmonitor__daemon_usage,
-+ options);
-+ return !!do_as_client__query_token(argv[0]);
++ if (!strcmp(subcmd, "status"))
++ return !!do_as_client__status();
+
-+ case QUERY_INDEX:
-+ return !!do_as_client__query_from_index();
-+
-+ case FLUSH:
-+ return !!do_as_client__send_flush();
-+
- case UNDEFINED_MODE:
- default:
- die(_("Unhandled command mode %d"), mode);
+ die(_("Unhandled subcommand '%s'"), subcmd);
+ }
+
-: ------------ > 10: d21af7ff842c t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon
7: 27f47dfbd9cf ! 11: 49f9e2e3d49c fsmonitor-fs-listen-win32: stub in backend for Windows
@@ Makefile: all::
# directory, and the JSON compilation database 'compile_commands.json' will be
# created at the root of the repository.
#
-+# If your platform supports an built-in fsmonitor backend, set
-+# FSMONITOR_DAEMON_BACKEND to the name of the corresponding
++# If your platform supports a built-in fsmonitor backend, set
++# FSMONITOR_DAEMON_BACKEND to the "<name>" of the corresponding
+# `compat/fsmonitor/fsmonitor-fs-listen-<name>.c` that implements the
+# `fsmonitor_fs_listen__*()` routines.
+#
8: a84dee24e13e = 12: 2aa85151f03c fsmonitor-fs-listen-macos: stub in backend for MacOS
9: 2b291d805d59 ! 13: 2aa05ad5c67f fsmonitor--daemon: implement daemon command options
@@ Metadata
## Commit message ##
fsmonitor--daemon: implement daemon command options
- Implement command options `--run` and `--start` to try to
+ Implement `run` and `start` commands to try to
begin listening for file system events.
This version defines the thread structure with a single
@@ builtin/fsmonitor--daemon.c
#include "khash.h"
static const char * const builtin_fsmonitor__daemon_usage[] = {
-+ N_("git fsmonitor--daemon --start [<options>]"),
-+ N_("git fsmonitor--daemon --run [<options>]"),
- N_("git fsmonitor--daemon --stop"),
- N_("git fsmonitor--daemon --is-running"),
- N_("git fsmonitor--daemon --query <token>"),
-@@ builtin/fsmonitor--daemon.c: static const char * const builtin_fsmonitor__daemon_usage[] = {
++ N_("git fsmonitor--daemon start [<options>]"),
++ N_("git fsmonitor--daemon run [<options>]"),
+ N_("git fsmonitor--daemon stop"),
+ N_("git fsmonitor--daemon status"),
+ NULL
};
#ifdef HAVE_FSMONITOR_DAEMON_BACKEND
@@ builtin/fsmonitor--daemon.c: static const char * const builtin_fsmonitor__daemon
/*
* Acting as a CLIENT.
*
-@@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)
- return 0;
+@@ builtin/fsmonitor--daemon.c: static int do_as_client__status(void)
+ }
}
+static ipc_server_application_cb handle_client;
+
-+static int handle_client(void *data, const char *command,
++static int handle_client(void *data,
++ const char *command, size_t command_len,
+ ipc_server_reply_cb *reply,
+ struct ipc_server_reply_data *reply_data)
+{
+ /* struct fsmonitor_daemon_state *state = data; */
+ int result;
+
++ /*
++ * The Simple IPC API now supports {char*, len} arguments, but
++ * FSMonitor always uses proper null-terminated strings, so
++ * we can ignore the command_len argument. (Trust, but verify.)
++ */
++ if (command_len != strlen(command))
++ BUG("FSMonitor assumes text messages");
++
+ trace2_region_enter("fsmonitor", "handle_client", the_repository);
+ trace2_data_string("fsmonitor", the_repository, "request", command);
+
@@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)
+ pthread_mutex_init(&state.main_lock, NULL);
+ state.error_code = 0;
+ state.current_token_data = NULL;
-+ state.test_client_delay_ms = 0;
+
+ /* Prepare to (recursively) watch the <worktree-root> directory. */
+ strbuf_init(&state.path_worktree_watch, 0);
@@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)
+ state.nr_paths_watching = 1;
+
+ /*
-+ * If ".git" is not a directory, then <gitdir> is not inside the
-+ * cone of <worktree-root>, so set up a second watch for it.
++ * We create/delete cookie files inside the .git directory to
++ * help us keep sync with the file system. If ".git" is not a
++ * directory, then <gitdir> is not inside the cone of
++ * <worktree-root>, so set up a second watch for it.
+ */
+ strbuf_init(&state.path_gitdir_watch, 0);
+ strbuf_addbuf(&state.path_gitdir_watch, &state.path_worktree_watch);
@@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)
+ return err;
+}
+
- static int is_ipc_daemon_listening(void)
- {
- return fsmonitor_ipc__get_state() == IPC_STATE__LISTENING;
- }
-
+static int try_to_run_foreground_daemon(void)
+{
+ /*
@@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)
+ * However, this method gives us a nicer error message for a
+ * common error case.
+ */
-+ if (is_ipc_daemon_listening())
++ if (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)
+ die("fsmonitor--daemon is already running.");
+
+ return !!fsmonitor_run_daemon();
@@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)
+
+ strvec_push(&args, git_exe);
+ strvec_push(&args, "fsmonitor--daemon");
-+ strvec_push(&args, "--run");
++ strvec_push(&args, "run");
+
+ *pid = mingw_spawnvpe(args.v[0], args.v, NULL, NULL, in, out, out);
+ close(in);
@@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)
+
+ if (pid_seen == -1)
+ return error_errno(_("waitpid failed"));
-+
+ else if (pid_seen == 0) {
+ /*
+ * The child is still running (this should be
@@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)
+ time(&now);
+ if (now > time_limit)
+ return error(_("fsmonitor--daemon not online yet"));
-+
-+ continue;
-+ }
-+
-+ else if (pid_seen == pid_child) {
++ } else if (pid_seen == pid_child) {
+ /*
+ * The new child daemon process shutdown while
+ * it was starting up, so it is not listening
@@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)
+ * early shutdown as an error.
+ */
+ return error(_("fsmonitor--daemon failed to start"));
-+ }
-+
-+ else
++ } else
+ return error(_("waitpid is confused"));
+ }
+}
@@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)
+ * of creating the background process (and not whether it
+ * immediately exited).
+ */
-+ if (is_ipc_daemon_listening())
++ if (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)
+ die("fsmonitor--daemon is already running.");
+
+ /*
@@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)
+
int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)
{
- enum daemon_mode {
- UNDEFINED_MODE,
-+ START,
-+ RUN,
- STOP,
- IS_RUNNING,
- QUERY,
-@@ builtin/fsmonitor--daemon.c: int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)
- } mode = UNDEFINED_MODE;
+ const char *subcmd;
struct option options[] = {
-+ OPT_CMDMODE(0, "start", &mode,
-+ N_("run the daemon in the background"),
-+ START),
-+ OPT_CMDMODE(0, "run", &mode,
-+ N_("run the daemon in the foreground"), RUN),
- OPT_CMDMODE(0, "stop", &mode, N_("stop the running daemon"),
- STOP),
-
-@@ builtin/fsmonitor--daemon.c: int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)
- QUERY_INDEX),
- OPT_CMDMODE(0, "flush", &mode, N_("flush cached filesystem events"),
- FLUSH),
-+
-+ OPT_GROUP(N_("Daemon options")),
+ OPT_INTEGER(0, "ipc-threads",
+ &fsmonitor__ipc_threads,
+ N_("use <n> ipc worker threads")),
+ OPT_INTEGER(0, "start-timeout",
+ &fsmonitor__start_timeout_sec,
+ N_("Max seconds to wait for background daemon startup")),
++
OPT_END()
};
+@@ builtin/fsmonitor--daemon.c: int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)
if (argc == 2 && !strcmp(argv[1], "-h"))
usage_with_options(builtin_fsmonitor__daemon_usage, options);
- git_config(git_default_config, NULL);
+ git_config(fsmonitor_config, NULL);
+ subcmd = argv[1];
+ argv--;
+@@ builtin/fsmonitor--daemon.c: int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)
+
argc = parse_options(argc, argv, prefix, options,
builtin_fsmonitor__daemon_usage, 0);
+ if (fsmonitor__ipc_threads < 1)
+ die(_("invalid 'ipc-threads' value (%d)"),
+ fsmonitor__ipc_threads);
-
- switch (mode) {
-+ case START:
++
++ if (!strcmp(subcmd, "start"))
+ return !!try_to_start_background_daemon();
+
-+ case RUN:
++ if (!strcmp(subcmd, "run"))
+ return !!try_to_run_foreground_daemon();
-+
- case STOP:
- return !!do_as_client__send_stop();
+ if (!strcmp(subcmd, "stop"))
+ return !!do_as_client__send_stop();
## fsmonitor--daemon.h (new) ##
@@
@@ fsmonitor--daemon.h (new)
+ struct fsmonitor_daemon_backend_data *backend_data;
+
+ struct ipc_server_data *ipc_server_data;
-+
-+ int test_client_delay_ms;
+};
+
+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */ 10: 451563314d84 ! 14: d5ababfd03e9 fsmonitor--daemon: add pathname classification
@@ Commit message
## builtin/fsmonitor--daemon.c ##
-@@ builtin/fsmonitor--daemon.c: static int handle_client(void *data, const char *command, +@@ builtin/fsmonitor--daemon.c: static int handle_client(void *data,
return result;
}
@@ builtin/fsmonitor--daemon.c: static int handle_client(void *data, const char *co
## fsmonitor--daemon.h ##
@@ fsmonitor--daemon.h: struct fsmonitor_daemon_state {
- int test_client_delay_ms;
+ struct ipc_server_data *ipc_server_data;
};
+/*
11: 304fe03034f8 ! 15: c092cdf2c8b7 fsmonitor--daemon: define token-ids
@@ Commit message
## builtin/fsmonitor--daemon.c ##
-@@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)
- return 0;
+@@ builtin/fsmonitor--daemon.c: static int do_as_client__status(void)
+ }
}
+/*
@@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)
+ *
+ * "builtin" ":" <token_id> ":" <sequence_nr>
+ *
++ * The "builtin" prefix is used as a namespace to avoid conflicts ++ * with other providers (such as Watchman).
++ *
+ * The <token_id> is an arbitrary OPAQUE string, such as a GUID,
+ * UUID, or {timestamp,pid}. It is used to group all filesystem
+ * events that happened while the daemon was monitoring (and in-sync
@@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)
+ * (There are too many race conditions to rely on file system
+ * event timestamps.)
+ *
-+ * The <sequence_nr> is a simple integer incremented for each event -+ * received. When a new <token_id> is created, the <sequence_nr> is -+ * reset to zero.
++ * The <sequence_nr> is a simple integer incremented whenever the ++ * daemon needs to make its state public. For example, if 1000 file ++ * system events come in, but no clients have requested the data, ++ * the daemon can continue to accumulate file changes in the same ++ * bin and does not need to advance the sequence number. However, ++ * as soon as a client does arrive, the daemon needs to start a new ++ * bin and increment the sequence number.
++ *
++ * The sequence number serves as the boundary between 2 sets
++ * of bins -- the older ones that the client has already seen
++ * and the newer ones that it hasn't.
++ *
++ * When a new <token_id> is created, the <sequence_nr> is reset to ++ * zero.
+ *
+ *
+ * About Token Ids
@@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)
+ * [3] in response to a client "flush" command (for dropped event
+ * testing).
+ *
-+ * [4] MAYBE We might want to change the token_id after very complex
-+ * filesystem operations are performed, such as a directory move
-+ * sequence that affects many files within. It might be simpler
-+ * to just give up and fake a re-sync (and let the client do a
-+ * full scan) than try to enumerate the effects of such a change.
-+ *
+ * When a new token_id is created, the daemon is free to discard all
+ * cached filesystem events associated with any previous token_ids.
+ * Events associated with a non-current token_id will never be sent
@@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)
+ static uint64_t flush_count = 0;
+ struct fsmonitor_token_data *token;
+
-+ token = (struct fsmonitor_token_data *)xcalloc(1, sizeof(*token));
++ CALLOC_ARRAY(token, 1);
+
+ strbuf_init(&token->token_id, 0);
+ token->batch_head = NULL;
@@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)
+
static ipc_server_application_cb handle_client;
- static int handle_client(void *data, const char *command,
+ static int handle_client(void *data,
@@ builtin/fsmonitor--daemon.c: static int fsmonitor_run_daemon(void)
pthread_mutex_init(&state.main_lock, NULL); state.error_code = 0;
- state.current_token_data = NULL;
+ state.current_token_data = fsmonitor_new_token_data();
- state.test_client_delay_ms = 0;
/* Prepare to (recursively) watch the <worktree-root> directory. */
+ strbuf_init(&state.path_worktree_watch, 0);
12: f1fa803ebe9c ! 16: 2ed7bc3fae7a fsmonitor--daemon: create token-based changed path cache
@@ Metadata
## Commit message ##
fsmonitor--daemon: create token-based changed path cache
- Teach fsmonitor--daemon to build lists of changed paths and associate
+ Teach fsmonitor--daemon to build a list of changed paths and associate
them with a token-id. This will be used by the platform-specific
backends to accumulate changed paths in response to filesystem events.
- The platform-specific event loops receive batches containing one or
- more changed paths. Their fs listener thread will accumulate them in
- a `fsmonitor_batch` (and without locking) and then "publish" them to
- associate them with the current token and to make them visible to the
- client worker threads.
+ The platform-specific file system listener thread receives file system
+ events containing one or more changed pathnames (with whatever bucketing
+ or grouping that is convenient for the file system). These paths are
+ accumulated (without locking) by the file system layer into a `fsmonitor_batch`.
+
+ When the file system layer has drained the kernel event queue, it will
+ "publish" them to our token queue and make them visible to concurrent
+ client worker threads. The token layer is free to combine and/or de-dup
+ paths within these batches for efficient presentation to clients.
Signed-off-by: Jeff Hostetler <[email protected]>
## builtin/fsmonitor--daemon.c ##
-@@ builtin/fsmonitor--daemon.c: static struct fsmonitor_token_data *fsmonitor_new_token_data(void)
- return token;
- }
+@@ builtin/fsmonitor--daemon.c: struct fsmonitor_token_data {
+ uint64_t client_ref_count;
+ };
+struct fsmonitor_batch {
+ struct fsmonitor_batch *next;
@@ builtin/fsmonitor--daemon.c: static struct fsmonitor_token_data *fsmonitor_new_t
+ time_t pinned_time;
+};
+
+ static struct fsmonitor_token_data *fsmonitor_new_token_data(void)
+ {
+ static int test_env_value = -1;
+ static uint64_t flush_count = 0;
+ struct fsmonitor_token_data *token;
++ struct fsmonitor_batch *batch;
+
+ CALLOC_ARRAY(token, 1);
++ batch = fsmonitor_batch__new();
+
+ strbuf_init(&token->token_id, 0);
+- token->batch_head = NULL;
+- token->batch_tail = NULL;
++ token->batch_head = batch;
++ token->batch_tail = batch;
+ token->client_ref_count = 0;
+
+ if (test_env_value < 0)
+@@ builtin/fsmonitor--daemon.c: static struct fsmonitor_token_data *fsmonitor_new_token_data(void)
+ strbuf_addf(&token->token_id, "test_%08x", test_env_value++);
+ }
+
++ /*
++ * We created a new <token_id> and are starting a new series
++ * of tokens with a zero <seq_nr>.
++ *
++ * Since clients cannot guess our new (non test) <token_id>
++ * they will always receive a trivial response (because of the
++ * mismatch on the <token_id>). The trivial response will
++ * tell them our new <token_id> so that subsequent requests
++ * will be relative to our new series. (And when sending that
++ * response, we pin the current head of the batch list.)
++ *
++ * Even if the client correctly guesses the <token_id>, their
++ * request of "builtin:<token_id>:0" asks for all changes MORE
++ * RECENT than batch/bin 0.
++ *
++ * This implies that it is a waste to accumulate paths in the
++ * initial batch/bin (because they will never be transmitted).
++ *
++ * So the daemon could be running for days and watching the
++ * file system, but doesn't need to actually accumulate any
++ * paths UNTIL we need to set a reference point for a later
++ * relative request.
++ *
++ * However, it is very useful for testing to always have a
++ * reference point set. Pin batch 0 to force early file system
++ * events to accumulate.
++ */
++ if (test_env_value)
++ batch->pinned_time = time(NULL);
++
+ return token;
+ }
+
+struct fsmonitor_batch *fsmonitor_batch__new(void)
+{
-+ struct fsmonitor_batch *batch = xcalloc(1, sizeof(*batch));
++ struct fsmonitor_batch *batch;
++
++ CALLOC_ARRAY(batch, 1);
+
+ return batch;
+}
+
-+struct fsmonitor_batch *fsmonitor_batch__free(struct fsmonitor_batch *batch)
++struct fsmonitor_batch *fsmonitor_batch__pop(struct fsmonitor_batch *batch)
+{
+ struct fsmonitor_batch *next;
+
@@ builtin/fsmonitor--daemon.c: static struct fsmonitor_token_data *fsmonitor_new_t
+static void fsmonitor_batch__combine(struct fsmonitor_batch *batch_dest,
+ const struct fsmonitor_batch *batch_src)
+{
-+ /* assert state->main_lock */
-+
+ size_t k;
+
+ ALLOC_GROW(batch_dest->interned_paths,
@@ builtin/fsmonitor--daemon.c: static struct fsmonitor_token_data *fsmonitor_new_t
+
+ strbuf_release(&token->token_id);
+
-+ for (p = token->batch_head; p; p = fsmonitor_batch__free(p))
++ for (p = token->batch_head; p; p = fsmonitor_batch__pop(p))
+ ;
+
+ free(token);
@@ builtin/fsmonitor--daemon.c: static struct fsmonitor_token_data *fsmonitor_new_t
+ * We should create a new token and start fresh (as if we just
+ * booted up).
+ *
-+ * If there are no readers of the the current token data series, we
-+ * can free it now. Otherwise, let the last reader free it. Either
-+ * way, the old token data series is no longer associated with our
-+ * state data.
++ * If there are no concurrent threads readering the current token data
++ * series, we can free it now. Otherwise, let the last reader free
++ * it.
++ *
++ * Either way, the old token data series is no longer associated with
++ * our state data.
+ */
-+void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)
++static void with_lock__do_force_resync(struct fsmonitor_daemon_state *state)
+{
++ /* assert current thread holding state->main_lock */
++
+ struct fsmonitor_token_data *free_me = NULL;
+ struct fsmonitor_token_data *new_one = NULL;
+
+ new_one = fsmonitor_new_token_data();
+
-+ pthread_mutex_lock(&state->main_lock);
-+
-+ trace_printf_key(&trace_fsmonitor,
-+ "force resync [old '%s'][new '%s']",
-+ state->current_token_data->token_id.buf,
-+ new_one->token_id.buf);
-+
+ if (state->current_token_data->client_ref_count == 0)
+ free_me = state->current_token_data;
+ state->current_token_data = new_one;
+
-+ pthread_mutex_unlock(&state->main_lock);
-+
+ fsmonitor_free_token_data(free_me);
+}
++
++void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)
++{
++ pthread_mutex_lock(&state->main_lock);
++ with_lock__do_force_resync(state);
++ pthread_mutex_unlock(&state->main_lock);
++}
+
static ipc_server_application_cb handle_client;
- static int handle_client(void *data, const char *command,
+ static int handle_client(void *data,
@@ builtin/fsmonitor--daemon.c: enum fsmonitor_path_type fsmonitor_classify_path_absolute(
return fsmonitor_classify_path_gitdir_relative(rel);
}
@@ builtin/fsmonitor--daemon.c: enum fsmonitor_path_type fsmonitor_classify_path_ab
+
+ head = state->current_token_data->batch_head;
+ if (!head) {
-+ batch->batch_seq_nr = 0;
-+ batch->next = NULL;
-+ state->current_token_data->batch_head = batch;
-+ state->current_token_data->batch_tail = batch;
++ BUG("token does not have batch");
+ } else if (head->pinned_time) {
+ /*
+ * We cannot alter the current batch list
@@ builtin/fsmonitor--daemon.c: enum fsmonitor_path_type fsmonitor_classify_path_ab
+ batch->batch_seq_nr = head->batch_seq_nr + 1;
+ batch->next = head;
+ state->current_token_data->batch_head = batch;
++ } else if (!head->batch_seq_nr) {
++ /*
++ * Batch 0 is unpinned. See the note in
++ * `fsmonitor_new_token_data()` about why we
++ * don't need to accumulate these paths.
++ */
++ fsmonitor_batch__pop(batch);
+ } else if (head->nr + batch->nr > MY_COMBINE_LIMIT) {
+ /*
+ * The head batch in the list has never been
@@ builtin/fsmonitor--daemon.c: enum fsmonitor_path_type fsmonitor_classify_path_ab
+ * batch onto the end of the current head batch.
+ */
+ fsmonitor_batch__combine(head, batch);
-+ fsmonitor_batch__free(batch);
++ fsmonitor_batch__pop(batch);
+ }
+ }
+
@@ fsmonitor--daemon.h
+/*
+ * Free this batch and return the value of the batch->next field.
+ */
-+struct fsmonitor_batch *fsmonitor_batch__free(struct fsmonitor_batch *batch);
++struct fsmonitor_batch *fsmonitor_batch__pop(struct fsmonitor_batch *batch);
+
+/*
+ * Add this path to this batch of modified files.
13: a57ddb3bc7cc ! 17: 9ea4b04b8215 fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows
@@ compat/fsmonitor/fsmonitor-fs-listen-win32.c
+ return NULL;
+ }
+
-+ watch = xcalloc(1, sizeof(*watch));
++ CALLOC_ARRAY(watch, 1);
+
+ watch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */
+
@@ compat/fsmonitor/fsmonitor-fs-listen-win32.c
+ watch->overlapped.hEvent = watch->hEvent;
+
+start_watch:
++ /*
++ * Queue an async call using Overlapped IO. This returns immediately.
++ * Our event handle will be signalled when the real result is available.
++ *
++ * The return value here just means that we successfully queued it.
++ * We won't know if the Read...() actually produces data until later.
++ */
+ watch->is_active = ReadDirectoryChangesW(
+ watch->hDir, watch->buffer, watch->buf_len, TRUE,
+ dwNotifyFilter, &watch->count, &watch->overlapped, NULL);
+
++ /*
++ * The kernel throws an invalid parameter error when our buffer
++ * is too big and we are pointed at a remote directory (and possibly
++ * for other reasons). Quietly set it down and try again.
++ *
++ * See note about MAX_RDCW_BUF at the top.
++ */
+ if (!watch->is_active &&
+ GetLastError() == ERROR_INVALID_PARAMETER &&
+ watch->buf_len > MAX_RDCW_BUF_FALLBACK) {
@@ compat/fsmonitor/fsmonitor-fs-listen-win32.c
+{
+ watch->is_active = FALSE;
+
++ /*
++ * The overlapped result is ready. If the Read...() was successful
++ * we finally receive the actual result into our buffer.
++ */
+ if (GetOverlappedResult(watch->hDir, &watch->overlapped, &watch->count,
+ TRUE))
+ return 0;
+
-+ // TODO If an external <gitdir> is deleted, the above returns an error.
-+ // TODO I'm not sure that there's anything that we can do here other
-+ // TODO than failing -- the <worktree>/.git link file would be broken
-+ // TODO anyway. We might try to check for that and return a better
-+ // TODO error message.
++ /*
++ * NEEDSWORK: If an external <gitdir> is deleted, the above
++ * returns an error. I'm not sure that there's anything that
++ * we can do here other than failing -- the <worktree>/.git
++ * link file would be broken anyway. We might try to check
++ * for that and return a better error message, but I'm not
++ * sure it is worth it.
++ */
+
+ error("GetOverlappedResult failed on '%s' [GLE %ld]",
+ watch->path.buf, GetLastError());
@@ compat/fsmonitor/fsmonitor-fs-listen-win32.c
+ if (!watch || !watch->is_active)
+ return;
+
++ /*
++ * The calls to ReadDirectoryChangesW() and GetOverlappedResult()
++ * form a "pair" (my term) where we queue an IO and promise to
++ * hang around and wait for the kernel to give us the result.
++ *
++ * If for some reason after we queue the IO, we have to quit
++ * or otherwise not stick around for the second half, we must
++ * tell the kernel to abort the IO. This prevents the kernel
++ * from writing to our buffer and/or signalling our event
++ * after we free them.
++ *
++ * (Ask me how much fun it was to track that one down).
++ */
+ CancelIoEx(watch->hDir, &watch->overlapped);
+ GetOverlappedResult(watch->hDir, &watch->overlapped, &count, TRUE);
+ watch->is_active = FALSE;
@@ compat/fsmonitor/fsmonitor-fs-listen-win32.c
+ /*
+ * If the kernel gets more events than will fit in the kernel
+ * buffer associated with our RDCW handle, it drops them and
-+ * returns a count of zero. (A successful call, but with
-+ * length zero.)
++ * returns a count of zero.
++ *
++ * Yes, the call returns WITHOUT error and with length zero.
++ *
++ * (The "overflow" case is not ambiguous with the "no data" case
++ * because we did an INFINITE wait.)
++ *
++ * This means we have a gap in coverage. Tell the daemon layer
++ * to resync.
+ */
+ if (!watch->count) {
+ trace2_data_string("fsmonitor", NULL, "fsm-listen/kernel",
@@ compat/fsmonitor/fsmonitor-fs-listen-win32.c
+ default:
+ BUG("unexpected path classification '%d' for '%s'",
+ t, path.buf);
-+ goto skip_this_path;
+ }
+
+skip_this_path:
@@ compat/fsmonitor/fsmonitor-fs-listen-win32.c
+ return LISTENER_HAVE_DATA_WORKTREE;
+
+force_shutdown:
-+ fsmonitor_batch__free(batch);
++ fsmonitor_batch__pop(batch);
+ string_list_clear(&cookie_list, 0);
+ strbuf_release(&path);
+ return LISTENER_SHUTDOWN;
+}
+
+/*
-+ * Process filesystem events that happend anywhere (recursively) under the
++ * Process filesystem events that happened anywhere (recursively) under the
+ * external <gitdir> (such as non-primary worktrees or submodules).
+ * We only care about cookie files that our client threads created here.
+ *
@@ compat/fsmonitor/fsmonitor-fs-listen-win32.c
+ default:
+ BUG("unexpected path classification '%d' for '%s'",
+ t, path.buf);
-+ goto skip_this_path;
+ }
+
+skip_this_path:
@@ compat/fsmonitor/fsmonitor-fs-listen-win32.c
{
+ struct fsmonitor_daemon_backend_data *data;
+
-+ data = xcalloc(1, sizeof(*data));
++ CALLOC_ARRAY(data, 1);
+
+ data->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL);
+
14: 67fa7c7b8ac7 = 18: 21b2b4f941b2 fsmonitor-fs-listen-macos: add macos header files for FSEvent
15: d469d3f02e33 ! 19: 08474bad8303 fsmonitor-fs-listen-macos: implement FSEvent listener on MacOS
@@ compat/fsmonitor/fsmonitor-fs-listen-macos.c: void FSEventStreamRelease(FSEventS
+ ef & kFSEventStreamEventFlagItemRenamed);
+}
+
++static int ef_is_dropped(const FSEventStreamEventFlags ef)
++{
++ return (ef & kFSEventStreamEventFlagKernelDropped ||
++ ef & kFSEventStreamEventFlagUserDropped);
++}
++
+static void fsevent_callback(ConstFSEventStreamRef streamRef,
+ void *ctx,
+ size_t num_of_events,
@@ compat/fsmonitor/fsmonitor-fs-listen-macos.c: void FSEventStreamRelease(FSEventS
+ const char *path_k;
+ const char *slash;
+ int k;
++ struct strbuf tmp = STRBUF_INIT;
+
+ /*
+ * Build a list of all filesystem changes into a private/local
@@ compat/fsmonitor/fsmonitor-fs-listen-macos.c: void FSEventStreamRelease(FSEventS
+ * If you want to debug FSEvents, log them to GIT_TRACE_FSMONITOR.
+ * Please don't log them to Trace2.
+ *
-+ * trace_printf_key(&trace_fsmonitor, "XXX '%s'", path_k);
++ * trace_printf_key(&trace_fsmonitor, "Path: '%s'", path_k);
+ */
+
+ /*
@@ compat/fsmonitor/fsmonitor-fs-listen-macos.c: void FSEventStreamRelease(FSEventS
+ * they are conceptually relative to the just flushed
+ * token).
+ */
-+ if ((event_flags[k] & kFSEventStreamEventFlagKernelDropped) ||
-+ (event_flags[k] & kFSEventStreamEventFlagUserDropped)) {
++ if (ef_is_dropped(event_flags[k])) {
+ /*
+ * see also kFSEventStreamEventFlagMustScanSubDirs
+ */
-+ trace2_data_string("fsmonitor", NULL,
-+ "fsm-listen/kernel", "dropped");
++ trace_printf_key(&trace_fsmonitor, "event: dropped");
+
+ fsmonitor_force_resync(state);
-+
-+ if (fsmonitor_batch__free(batch))
-+ BUG("batch should not have a next");
++ fsmonitor_batch__pop(batch);
+ string_list_clear(&cookie_list, 0);
+
+ /*
@@ compat/fsmonitor/fsmonitor-fs-listen-macos.c: void FSEventStreamRelease(FSEventS
+ * we have to quit.
+ */
+ if (ef_is_root_delete(event_flags[k])) {
-+ trace2_data_string("fsmonitor", NULL,
-+ "fsm-listen/gitdir",
-+ "removed");
++ trace_printf_key(&trace_fsmonitor,
++ "event: gitdir removed");
+ goto force_shutdown;
+ }
+ if (ef_is_root_renamed(event_flags[k])) {
-+ trace2_data_string("fsmonitor", NULL,
-+ "fsm-listen/gitdir",
-+ "renamed");
++ trace_printf_key(&trace_fsmonitor,
++ "event: gitdir renamed");
+ goto force_shutdown;
+ }
+ break;
@@ compat/fsmonitor/fsmonitor-fs-listen-macos.c: void FSEventStreamRelease(FSEventS
+ if (trace_pass_fl(&trace_fsmonitor))
+ log_flags_set(path_k, event_flags[k]);
+
-+ /* fsevent could be marked as both a file and directory */
++ /*
++ * Because of the implicit "binning" (the
++ * kernel calls us at a given frequency) and
++ * de-duping (the kernel is free to combine
++ * multiple events for a given pathname), an
++ * individual fsevent could be marked as both
++ * a file and directory. Add it to the queue
++ * with both spellings so that the client will
++ * know how much to invalidate/refresh.
++ */
+
+ if (event_flags[k] & kFSEventStreamEventFlagItemIsFile) {
+ const char *rel = path_k +
@@ compat/fsmonitor/fsmonitor-fs-listen-macos.c: void FSEventStreamRelease(FSEventS
+ if (event_flags[k] & kFSEventStreamEventFlagItemIsDir) {
+ const char *rel = path_k +
+ state->path_worktree_watch.len + 1;
-+ char *p = xstrfmt("%s/", rel);
++
++ strbuf_reset(&tmp);
++ strbuf_addstr(&tmp, rel);
++ strbuf_addch(&tmp, '/');
+
+ if (!batch)
+ batch = fsmonitor_batch__new();
-+ fsmonitor_batch__add_path(batch, p);
-+
-+ free(p);
++ fsmonitor_batch__add_path(batch, tmp.buf);
+ }
+
+ break;
@@ compat/fsmonitor/fsmonitor-fs-listen-macos.c: void FSEventStreamRelease(FSEventS
+
+ fsmonitor_publish(state, batch, &cookie_list);
+ string_list_clear(&cookie_list, 0);
++ strbuf_release(&tmp);
+ return;
+
+force_shutdown:
-+ if (fsmonitor_batch__free(batch))
-+ BUG("batch should not have a next");
++ fsmonitor_batch__pop(batch);
+ string_list_clear(&cookie_list, 0);
+
+ data->shutdown_style = FORCE_SHUTDOWN;
+ CFRunLoopStop(data->rl);
++ strbuf_release(&tmp);
+ return;
+}
+
+/*
-+ * TODO Investigate the proper value for the `latency` argument in the call
-+ * TODO to `FSEventStreamCreate()`. I'm not sure that this needs to be a
-+ * TODO config setting or just something that we tune after some testing.
-+ * TODO
-+ * TODO With a latency of 0.1, I was seeing lots of dropped events during
-+ * TODO the "touch 100000" files test within t/perf/p7519, but with a
-+ * TODO latency of 0.001 I did not see any dropped events. So the "correct"
-+ * TODO value may be somewhere in between.
-+ * TODO
-+ * TODO https://developer.apple.com/documentation/coreservices/1443980-fseventstreamcreate
++ * NEEDSWORK: Investigate the proper value for the `latency` argument
++ * in the call to `FSEventStreamCreate()`. I'm not sure that this
++ * needs to be a config setting or just something that we tune after
++ * some testing.
++ *
++ * With a latency of 0.1, I was seeing lots of dropped events during
++ * the "touch 100000" files test within t/perf/p7519, but with a
++ * latency of 0.001 I did not see any dropped events. So the
++ * "correct" value may be somewhere in between.
++ *
++ * https://developer.apple.com/documentation/coreservices/1443980-fseventstreamcreate
+ */
int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state)
@@ compat/fsmonitor/fsmonitor-fs-listen-macos.c: void FSEventStreamRelease(FSEventS
+ struct fsmonitor_daemon_backend_data *data;
+ const void *dir_array[2];
+
-+ data = xcalloc(1, sizeof(*data));
++ CALLOC_ARRAY(data, 1);
+ state->backend_data = data;
+
+ data->cfsr_worktree_path = CFStringCreateWithCString(
16: 2b4ae4fc3d62 ! 20: cc4a596d17c7 fsmonitor--daemon: implement handle_client callback
@@ builtin/fsmonitor--daemon.c
+#include "pkt-line.h"
static const char * const builtin_fsmonitor__daemon_usage[] = {
- N_("git fsmonitor--daemon --start [<options>]"),
+ N_("git fsmonitor--daemon start [<options>]"),
@@ builtin/fsmonitor--daemon.c: void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)
- fsmonitor_free_token_data(free_me);
+ pthread_mutex_unlock(&state->main_lock);
}
+/*
+ * Format an opaque token string to send to the client.
+ */
-+static void fsmonitor_format_response_token(
++static void with_lock__format_response_token(
+ struct strbuf *response_token,
+ const struct strbuf *response_token_id,
+ const struct fsmonitor_batch *batch)
+{
-+ uint64_t seq_nr = (batch) ? batch->batch_seq_nr + 1 : 0;
++ /* assert current thread holding state->main_lock */
+
+ strbuf_reset(response_token);
+ strbuf_addf(response_token, "builtin:%s:%"PRIu64,
-+ response_token_id->buf, seq_nr);
++ response_token_id->buf, batch->batch_seq_nr);
+}
+
+/*
+ * Parse an opaque token from the client.
++ * Returns -1 on error.
+ */
+static int fsmonitor_parse_client_token(const char *buf_token,
+ struct strbuf *requested_token_id,
@@ builtin/fsmonitor--daemon.c: void fsmonitor_force_resync(struct fsmonitor_daemon
+ *seq_nr = 0;
+
+ if (!skip_prefix(buf_token, "builtin:", &p))
-+ return 1;
++ return -1;
+
+ while (*p && *p != ':')
+ strbuf_addch(requested_token_id, *p++);
+ if (!*p++)
-+ return 1;
++ return -1;
+
+ *seq_nr = (uint64_t)strtoumax(p, &p_end, 10);
+ if (*p_end)
-+ return 1;
++ return -1;
+
+ return 0;
+}
@@ builtin/fsmonitor--daemon.c: void fsmonitor_force_resync(struct fsmonitor_daemon
+ intmax_t count = 0, duplicates = 0;
+ kh_str_t *shown;
+ int hash_ret;
-+ int result;
++ int do_trivial = 0;
++ int do_flush = 0;
+
+ /*
+ * We expect `command` to be of the form:
@@ builtin/fsmonitor--daemon.c: void fsmonitor_force_resync(struct fsmonitor_daemon
+ * There is no reply to the client.
+ */
+ return SIMPLE_IPC_QUIT;
-+ }
+
-+ if (!strcmp(command, "flush")) {
++ } else if (!strcmp(command, "flush")) {
+ /*
+ * Flush all of our cached data and generate a new token
+ * just like if we lost sync with the filesystem.
+ *
+ * Then send a trivial response using the new token.
+ */
-+ fsmonitor_force_resync(state);
-+ result = 0;
-+ goto send_trivial_response;
-+ }
++ do_flush = 1;
++ do_trivial = 1;
+
-+ if (!skip_prefix(command, "builtin:", &p)) {
++ } else if (!skip_prefix(command, "builtin:", &p)) {
+ /* assume V1 timestamp or garbage */
+
+ char *p_end;
@@ builtin/fsmonitor--daemon.c: void fsmonitor_force_resync(struct fsmonitor_daemon
+ "fsmonitor: invalid command line '%s'" :
+ "fsmonitor: unsupported V1 protocol '%s'"),
+ command);
-+ result = -1;
-+ goto send_trivial_response;
++ do_trivial = 1;
++
++ } else {
++ /* We have "builtin:*" */
++ if (fsmonitor_parse_client_token(command, &requested_token_id,
++ &requested_oldest_seq_nr)) {
++ trace_printf_key(&trace_fsmonitor,
++ "fsmonitor: invalid V2 protocol token '%s'",
++ command);
++ do_trivial = 1;
++
++ } else {
++ /*
++ * We have a V2 valid token:
++ * "builtin:<token_id>:<seq_nr>"
++ */
++ }
+ }
+
-+ /* try V2 token */
++ pthread_mutex_lock(&state->main_lock);
+
-+ if (fsmonitor_parse_client_token(command, &requested_token_id,
-+ &requested_oldest_seq_nr)) {
-+ trace_printf_key(&trace_fsmonitor,
-+ "fsmonitor: invalid V2 protocol token '%s'",
-+ command);
-+ result = -1;
-+ goto send_trivial_response;
-+ }
++ if (!state->current_token_data)
++ BUG("fsmonitor state does not have a current token");
+
-+ pthread_mutex_lock(&state->main_lock);
++ if (do_flush)
++ with_lock__do_force_resync(state);
+
-+ if (!state->current_token_data) {
-+ /*
-+ * We don't have a current token. This may mean that
-+ * the listener thread has not yet started.
-+ */
-+ pthread_mutex_unlock(&state->main_lock);
-+ result = 0;
-+ goto send_trivial_response;
-+ }
-+ if (strcmp(requested_token_id.buf,
-+ state->current_token_data->token_id.buf)) {
-+ /*
-+ * The client last spoke to a different daemon
-+ * instance -OR- the daemon had to resync with
-+ * the filesystem (and lost events), so reject.
-+ */
-+ pthread_mutex_unlock(&state->main_lock);
-+ result = 0;
-+ trace2_data_string("fsmonitor", the_repository,
-+ "response/token", "different");
-+ goto send_trivial_response;
-+ }
-+ if (!state->current_token_data->batch_tail) {
-+ /*
-+ * The listener has not received any filesystem
-+ * events yet since we created the current token.
-+ * We can respond with an empty list, since the
-+ * client has already seen the current token and
-+ * we have nothing new to report. (This is
-+ * instead of sending a trivial response.)
-+ */
-+ pthread_mutex_unlock(&state->main_lock);
-+ result = 0;
-+ goto send_empty_response;
++ /*
++ * We mark the current head of the batch list as "pinned" so
++ * that the listener thread will treat this item as read-only
++ * (and prevent any more paths from being added to it) from
++ * now on.
++ */
++ token_data = state->current_token_data;
++ batch_head = token_data->batch_head;
++ ((struct fsmonitor_batch *)batch_head)->pinned_time = time(NULL);
++
++ /*
++ * FSMonitor Protocol V2 requires that we send a response header
++ * with a "new current token" and then all of the paths that changed
++ * since the "requested token". We send the seq_nr of the just-pinned
++ * head batch so that future requests from a client will be relative
++ * to it.
++ */
++ with_lock__format_response_token(&response_token,
++ &token_data->token_id, batch_head);
++
++ reply(reply_data, response_token.buf, response_token.len + 1);
++ total_response_len += response_token.len + 1;
++
++ trace2_data_string("fsmonitor", the_repository, "response/token",
++ response_token.buf);
++ trace_printf_key(&trace_fsmonitor, "response token: %s",
++ response_token.buf);
++
++ if (!do_trivial) {
++ if (strcmp(requested_token_id.buf, token_data->token_id.buf)) {
++ /*
++ * The client last spoke to a different daemon
++ * instance -OR- the daemon had to resync with
++ * the filesystem (and lost events), so reject.
++ */
++ trace2_data_string("fsmonitor", the_repository,
++ "response/token", "different");
++ do_trivial = 1;
++
++ } else if (requested_oldest_seq_nr <
++ token_data->batch_tail->batch_seq_nr) {
++ /*
++ * The client wants older events than we have for
++ * this token_id. This means that the end of our
++ * batch list was truncated and we cannot give the
++ * client a complete snapshot relative to their
++ * request.
++ */
++ trace_printf_key(&trace_fsmonitor,
++ "client requested truncated data");
++ do_trivial = 1;
++ }
+ }
-+ if (requested_oldest_seq_nr <
-+ state->current_token_data->batch_tail->batch_seq_nr) {
-+ /*
-+ * The client wants older events than we have for
-+ * this token_id. This means that the end of our
-+ * batch list was truncated and we cannot give the
-+ * client a complete snapshot relative to their
-+ * request.
-+ */
++
++ if (do_trivial) {
+ pthread_mutex_unlock(&state->main_lock);
+
-+ trace_printf_key(&trace_fsmonitor,
-+ "client requested truncated data");
-+ result = 0;
-+ goto send_trivial_response;
++ reply(reply_data, "/", 2);
++
++ trace2_data_intmax("fsmonitor", the_repository,
++ "response/trivial", 1);
++
++ strbuf_release(&response_token);
++ strbuf_release(&requested_token_id);
++ return 0;
+ }
+
+ /*
@@ builtin/fsmonitor--daemon.c: void fsmonitor_force_resync(struct fsmonitor_daemon
+ *
+ * AND it allows the listener thread to do a token-reset
+ * (and install a new `current_token_data`).
-+ *
-+ * We mark the current head of the batch list as "pinned" so
-+ * that the listener thread will treat this item as read-only
-+ * (and prevent any more paths from being added to it) from
-+ * now on.
+ */
-+ token_data = state->current_token_data;
+ token_data->client_ref_count++;
+
-+ batch_head = token_data->batch_head;
-+ ((struct fsmonitor_batch *)batch_head)->pinned_time = time(NULL);
-+
+ pthread_mutex_unlock(&state->main_lock);
+
+ /*
-+ * FSMonitor Protocol V2 requires that we send a response header
-+ * with a "new current token" and then all of the paths that changed
-+ * since the "requested token".
++ * The client request is relative to the token that they sent,
++ * so walk the batch list backwards from the current head back
++ * to the batch (sequence number) they named.
++ *
++ * We use khash to de-dup the list of pathnames.
++ *
++ * NEEDSWORK: each batch contains a list of interned strings,
++ * so we only need to do pointer comparisons here to build the
++ * hash table. Currently, we're still comparing the string
++ * values.
+ */
-+ fsmonitor_format_response_token(&response_token,
-+ &token_data->token_id,
-+ batch_head);
-+
-+ reply(reply_data, response_token.buf, response_token.len + 1);
-+ total_response_len += response_token.len + 1;
-+
-+ trace2_data_string("fsmonitor", the_repository, "response/token",
-+ response_token.buf);
-+ trace_printf_key(&trace_fsmonitor, "response token: %s", response_token.buf);
-+
+ shown = kh_init_str();
+ for (batch = batch_head;
-+ batch && batch->batch_seq_nr >= requested_oldest_seq_nr;
++ batch && batch->batch_seq_nr > requested_oldest_seq_nr;
+ batch = batch->next) {
+ size_t k;
+
@@ builtin/fsmonitor--daemon.c: void fsmonitor_force_resync(struct fsmonitor_daemon
+ kh_release_str(shown);
+
+ pthread_mutex_lock(&state->main_lock);
++
+ if (token_data->client_ref_count > 0)
+ token_data->client_ref_count--;
+
@@ builtin/fsmonitor--daemon.c: void fsmonitor_force_resync(struct fsmonitor_daemon
+ strbuf_release(&payload);
+
+ return 0;
-+
-+send_trivial_response:
-+ pthread_mutex_lock(&state->main_lock);
-+ fsmonitor_format_response_token(&response_token,
-+ &state->current_token_data->token_id,
-+ state->current_token_data->batch_head);
-+ pthread_mutex_unlock(&state->main_lock);
-+
-+ reply(reply_data, response_token.buf, response_token.len + 1);
-+ trace2_data_string("fsmonitor", the_repository, "response/token",
-+ response_token.buf);
-+ reply(reply_data, "/", 2);
-+ trace2_data_intmax("fsmonitor", the_repository, "response/trivial", 1);
-+
-+ strbuf_release(&response_token);
-+ strbuf_release(&requested_token_id);
-+
-+ return result;
-+
-+send_empty_response:
-+ pthread_mutex_lock(&state->main_lock);
-+ fsmonitor_format_response_token(&response_token,
-+ &state->current_token_data->token_id,
-+ NULL);
-+ pthread_mutex_unlock(&state->main_lock);
-+
-+ reply(reply_data, response_token.buf, response_token.len + 1);
-+ trace2_data_string("fsmonitor", the_repository, "response/token",
-+ response_token.buf);
-+ trace2_data_intmax("fsmonitor", the_repository, "response/empty", 1);
-+
-+ strbuf_release(&response_token);
-+ strbuf_release(&requested_token_id);
-+
-+ return 0;
+}
+
static ipc_server_application_cb handle_client;
- static int handle_client(void *data, const char *command,
+ static int handle_client(void *data,
+@@ builtin/fsmonitor--daemon.c: static int handle_client(void *data,
ipc_server_reply_cb *reply,
struct ipc_server_reply_data *reply_data)
{
@@ builtin/fsmonitor--daemon.c: void fsmonitor_force_resync(struct fsmonitor_daemon
+ struct fsmonitor_daemon_state *state = data;
int result;
+ /*
+@@ builtin/fsmonitor--daemon.c: static int handle_client(void *data,
+ if (command_len != strlen(command))
+ BUG("FSMonitor assumes text messages");
+
+ trace_printf_key(&trace_fsmonitor, "requested token: %s", command);
+
trace2_region_enter("fsmonitor", "handle_client", the_repository);
17: 9f263e70c724 ! 21: f0da90e9b050 fsmonitor--daemon: periodically truncate list of modified files
@@ Commit message
relative to this new token).
Therefore, the daemon can gradually truncate the in-memory list of
- changed paths as they become obsolete (older that the previous token).
+ changed paths as they become obsolete (older than the previous token).
Since we may have multiple clients making concurrent requests with a
skew of tokens and clients may be racing to the talk to the daemon,
we lazily truncate the list.
@@ builtin/fsmonitor--daemon.c: static void fsmonitor_batch__combine(struct fsmonit
+ * artificial (based on when we pinned the batch item) and not on any
+ * filesystem activity.
+ */
-+#define MY_TIME_DELAY (5 * 60) /* seconds */
++#define MY_TIME_DELAY_SECONDS (5 * 60) /* seconds */
+
-+static void fsmonitor_batch__truncate(struct fsmonitor_daemon_state *state,
-+ const struct fsmonitor_batch *batch_marker)
++static void with_lock__truncate_old_batches(
++ struct fsmonitor_daemon_state *state,
++ const struct fsmonitor_batch *batch_marker)
+{
-+ /* assert state->main_lock */
++ /* assert current thread holding state->main_lock */
+
+ const struct fsmonitor_batch *batch;
+ struct fsmonitor_batch *rest;
+ struct fsmonitor_batch *p;
-+ time_t t;
+
+ if (!batch_marker)
+ return;
+
-+ trace_printf_key(&trace_fsmonitor, "TRNC mark (%"PRIu64",%"PRIu64")",
++ trace_printf_key(&trace_fsmonitor, "Truncate: mark (%"PRIu64",%"PRIu64")",
+ batch_marker->batch_seq_nr,
+ (uint64_t)batch_marker->pinned_time);
+
+ for (batch = batch_marker; batch; batch = batch->next) {
++ time_t t;
++
+ if (!batch->pinned_time) /* an overflow batch */
+ continue;
+
-+ t = batch->pinned_time + MY_TIME_DELAY;
++ t = batch->pinned_time + MY_TIME_DELAY_SECONDS;
+ if (t > batch_marker->pinned_time) /* too close to marker */
+ continue;
+
@@ builtin/fsmonitor--daemon.c: static void fsmonitor_batch__combine(struct fsmonit
+ rest = ((struct fsmonitor_batch *)batch)->next;
+ ((struct fsmonitor_batch *)batch)->next = NULL;
+
-+ for (p = rest; p; p = fsmonitor_batch__free(p)) {
++ for (p = rest; p; p = fsmonitor_batch__pop(p)) {
+ trace_printf_key(&trace_fsmonitor,
-+ "TRNC kill (%"PRIu64",%"PRIu64")",
++ "Truncate: kill (%"PRIu64",%"PRIu64")",
+ p->batch_seq_nr, (uint64_t)p->pinned_time);
+ }
+}
@@ builtin/fsmonitor--daemon.c: static int do_handle_client(struct fsmonitor_daemon
+ * obsolete. See if we can truncate the list
+ * and save some memory.
+ */
-+ fsmonitor_batch__truncate(state, batch);
++ with_lock__truncate_old_batches(state, batch);
}
} 18: c6d5f045fb56 < -: ------------ fsmonitor--daemon:: introduce client delay for testing
19: 038b62dc6744 ! 22: bb7b1912bb47 fsmonitor--daemon: use a cookie file to sync with file system
@@ Commit message
Signed-off-by: Jeff Hostetler <[email protected]>
## builtin/fsmonitor--daemon.c ##
-@@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)
- return 0;
+@@ builtin/fsmonitor--daemon.c: static int do_as_client__status(void)
+ }
}
+enum fsmonitor_cookie_item_result {
@@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)
+ return strcmp(a->name, keydata ? keydata : b->name);
+}
+
-+static enum fsmonitor_cookie_item_result fsmonitor_wait_for_cookie(
++static enum fsmonitor_cookie_item_result with_lock__wait_for_cookie(
+ struct fsmonitor_daemon_state *state)
+{
++ /* assert current thread holding state->main_lock */
++
+ int fd;
-+ struct fsmonitor_cookie_item cookie;
++ struct fsmonitor_cookie_item *cookie;
+ struct strbuf cookie_pathname = STRBUF_INIT;
+ struct strbuf cookie_filename = STRBUF_INIT;
-+ const char *slash;
++ enum fsmonitor_cookie_item_result result;
+ int my_cookie_seq;
+
-+ pthread_mutex_lock(&state->main_lock);
++ CALLOC_ARRAY(cookie, 1);
+
+ my_cookie_seq = state->cookie_seq++;
+
++ strbuf_addf(&cookie_filename, "%i-%i", getpid(), my_cookie_seq);
++
+ strbuf_addbuf(&cookie_pathname, &state->path_cookie_prefix);
-+ strbuf_addf(&cookie_pathname, "%i-%i", getpid(), my_cookie_seq);
-+
-+ slash = find_last_dir_sep(cookie_pathname.buf);
-+ if (slash)
-+ strbuf_addstr(&cookie_filename, slash + 1);
-+ else
-+ strbuf_addbuf(&cookie_filename, &cookie_pathname);
-+ cookie.name = strbuf_detach(&cookie_filename, NULL);
-+ cookie.result = FCIR_INIT;
-+ // TODO should we have case-insenstive hash (and in cookie_cmp()) ??
-+ hashmap_entry_init(&cookie.entry, strhash(cookie.name));
++ strbuf_addbuf(&cookie_pathname, &cookie_filename);
+
-+ /*
-+ * Warning: we are putting the address of a stack variable into a
-+ * global hashmap. This feels dodgy. We must ensure that we remove
-+ * it before this thread and stack frame returns.
-+ */
-+ hashmap_add(&state->cookies, &cookie.entry);
++ cookie->name = strbuf_detach(&cookie_filename, NULL);
++ cookie->result = FCIR_INIT;
++ hashmap_entry_init(&cookie->entry, strhash(cookie->name));
++
++ hashmap_add(&state->cookies, &cookie->entry);
+
+ trace_printf_key(&trace_fsmonitor, "cookie-wait: '%s' '%s'",
-+ cookie.name, cookie_pathname.buf);
++ cookie->name, cookie_pathname.buf);
+
+ /*
+ * Create the cookie file on disk and then wait for a notification
@@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)
+ fd = open(cookie_pathname.buf, O_WRONLY | O_CREAT | O_EXCL, 0600);
+ if (fd >= 0) {
+ close(fd);
-+ unlink_or_warn(cookie_pathname.buf);
++ unlink(cookie_pathname.buf);
+
-+ while (cookie.result == FCIR_INIT)
++ /*
++ * NEEDSWORK: This is an infinite wait (well, unless another
++ * thread sends us an abort). I'd like to change this to
++ * use `pthread_cond_timedwait()` and return an error/timeout
++ * and let the caller do the trivial response thing.
++ */
++ while (cookie->result == FCIR_INIT)
+ pthread_cond_wait(&state->cookies_cond,
+ &state->main_lock);
-+
-+ hashmap_remove(&state->cookies, &cookie.entry, NULL);
+ } else {
+ error_errno(_("could not create fsmonitor cookie '%s'"),
-+ cookie.name);
++ cookie->name);
+
-+ cookie.result = FCIR_ERROR;
-+ hashmap_remove(&state->cookies, &cookie.entry, NULL);
++ cookie->result = FCIR_ERROR;
+ }
+
-+ pthread_mutex_unlock(&state->main_lock);
++ hashmap_remove(&state->cookies, &cookie->entry, NULL);
++
++ result = cookie->result;
+
-+ free((char*)cookie.name);
++ free((char*)cookie->name);
++ free(cookie);
+ strbuf_release(&cookie_pathname);
-+ return cookie.result;
++
++ return result;
+}
+
+/*
+ * Mark these cookies as _SEEN and wake up the corresponding client threads.
+ */
-+static void fsmonitor_cookie_mark_seen(struct fsmonitor_daemon_state *state,
-+ const struct string_list *cookie_names)
++static void with_lock__mark_cookies_seen(struct fsmonitor_daemon_state *state,
++ const struct string_list *cookie_names)
+{
-+ /* assert state->main_lock */
++ /* assert current thread holding state->main_lock */
+
+ int k;
+ int nr_seen = 0;
@@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)
+/*
+ * Set _ABORT on all pending cookies and wake up all client threads.
+ */
-+static void fsmonitor_cookie_abort_all(struct fsmonitor_daemon_state *state)
++static void with_lock__abort_all_cookies(struct fsmonitor_daemon_state *state)
+{
-+ /* assert state->main_lock */
++ /* assert current thread holding state->main_lock */
+
+ struct hashmap_iter iter;
+ struct fsmonitor_cookie_item *cookie;
@@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)
+ pthread_cond_broadcast(&state->cookies_cond);
+}
+
- static int lookup_client_test_delay(void)
- {
- static int delay_ms = -1;
+ /*
+ * Requests to and from a FSMonitor Protocol V2 provider use an opaque
+ * "token" as a virtual timestamp. Clients can request a summary of all
@@ builtin/fsmonitor--daemon.c: static void fsmonitor_free_token_data(struct fsmonitor_token_data *token)
* We should create a new token and start fresh (as if we just
* booted up).
@@ builtin/fsmonitor--daemon.c: static void fsmonitor_free_token_data(struct fsmoni
+ * [2] Some of those lost events may have been for cookie files. We
+ * should assume the worst and abort them rather letting them starve.
+ *
- * If there are no readers of the the current token data series, we
- * can free it now. Otherwise, let the last reader free it. Either
- * way, the old token data series is no longer associated with our
-@@ builtin/fsmonitor--daemon.c: void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)
- state->current_token_data->token_id.buf,
- new_one->token_id.buf);
+ * If there are no concurrent threads readering the current token data
+ * series, we can free it now. Otherwise, let the last reader free
+ * it.
+@@ builtin/fsmonitor--daemon.c: static void with_lock__do_force_resync(struct fsmonitor_daemon_state *state)
+ state->current_token_data = new_one;
-+ fsmonitor_cookie_abort_all(state);
+ fsmonitor_free_token_data(free_me);
+
- if (state->current_token_data->client_ref_count == 0)
- free_me = state->current_token_data;
- state->current_token_data = new_one;
++ with_lock__abort_all_cookies(state);
+ }
+
+ void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)
@@ builtin/fsmonitor--daemon.c: static int do_handle_client(struct fsmonitor_daemon_state *state,
- kh_str_t *shown;
int hash_ret;
- int result;
+ int do_trivial = 0;
+ int do_flush = 0;
++ int do_cookie = 0;
+ enum fsmonitor_cookie_item_result cookie_result;
/*
* We expect `command` to be of the form:
@@ builtin/fsmonitor--daemon.c: static int do_handle_client(struct fsmonitor_daemon_state *state,
- goto send_trivial_response;
+ */
+ do_flush = 1;
+ do_trivial = 1;
++ do_cookie = 1;
+
+ } else if (!skip_prefix(command, "builtin:", &p)) {
+ /* assume V1 timestamp or garbage */
+@@ builtin/fsmonitor--daemon.c: static int do_handle_client(struct fsmonitor_daemon_state *state,
+ "fsmonitor: unsupported V1 protocol '%s'"),
+ command);
+ do_trivial = 1;
++ do_cookie = 1;
+
+ } else {
+ /* We have "builtin:*" */
+@@ builtin/fsmonitor--daemon.c: static int do_handle_client(struct fsmonitor_daemon_state *state,
+ "fsmonitor: invalid V2 protocol token '%s'",
+ command);
+ do_trivial = 1;
++ do_cookie = 1;
+
+ } else {
+ /*
+ * We have a V2 valid token:
+ * "builtin:<token_id>:<seq_nr>"
+ */
++ do_cookie = 1;
+ }
}
-+ pthread_mutex_unlock(&state->main_lock);
-+
+@@ builtin/fsmonitor--daemon.c: static int do_handle_client(struct fsmonitor_daemon_state *state,
+ if (!state->current_token_data)
+ BUG("fsmonitor state does not have a current token");
+
+ /*
-+ * Write a cookie file inside the directory being watched in an
-+ * effort to flush out existing filesystem events that we actually
-+ * care about. Suspend this client thread until we see the filesystem
-+ * events for this cookie file.
++ * Write a cookie file inside the directory being watched in
++ * an effort to flush out existing filesystem events that we
++ * actually care about. Suspend this client thread until we
++ * see the filesystem events for this cookie file.
++ *
++ * Creating the cookie lets us guarantee that our FS listener
++ * thread has drained the kernel queue and we are caught up
++ * with the kernel.
++ *
++ * If we cannot create the cookie (or otherwise guarantee that
++ * we are caught up), we send a trivial response. We have to
++ * assume that there might be some very, very recent activity
++ * on the FS still in flight.
+ */
-+ cookie_result = fsmonitor_wait_for_cookie(state);
-+ if (cookie_result != FCIR_SEEN) {
-+ error(_("fsmonitor: cookie_result '%d' != SEEN"),
-+ cookie_result);
-+ result = 0;
-+ goto send_trivial_response;
-+ }
-+
-+ pthread_mutex_lock(&state->main_lock);
-+
-+ if (strcmp(requested_token_id.buf,
-+ state->current_token_data->token_id.buf)) {
-+ /*
-+ * Ack! The listener thread lost sync with the filesystem
-+ * and created a new token while we were waiting for the
-+ * cookie file to be created! Just give up.
-+ */
-+ pthread_mutex_unlock(&state->main_lock);
-+
-+ trace_printf_key(&trace_fsmonitor,
-+ "lost filesystem sync");
-+ result = 0;
-+ goto send_trivial_response;
++ if (do_cookie) {
++ cookie_result = with_lock__wait_for_cookie(state);
++ if (cookie_result != FCIR_SEEN) {
++ error(_("fsmonitor: cookie_result '%d' != SEEN"),
++ cookie_result);
++ do_trivial = 1;
++ }
+ }
+
- /*
- * We're going to hold onto a pointer to the current
- * token-data while we walk the list of batches of files.
+ if (do_flush)
+ with_lock__do_force_resync(state);
+
+@@ builtin/fsmonitor--daemon.c: static int handle_client(void *data,
+ return result;
+ }
+
+-#define FSMONITOR_COOKIE_PREFIX ".fsmonitor-daemon-"
++#define FSMONITOR_DIR "fsmonitor--daemon"
++#define FSMONITOR_COOKIE_DIR "cookies"
++#define FSMONITOR_COOKIE_PREFIX (FSMONITOR_DIR "/" FSMONITOR_COOKIE_DIR "/")
+
+ enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative(
+ const char *rel)
@@ builtin/fsmonitor--daemon.c: void fsmonitor_publish(struct fsmonitor_daemon_state *state,
}
}
+ if (cookie_names->nr)
-+ fsmonitor_cookie_mark_seen(state, cookie_names);
++ with_lock__mark_cookies_seen(state, cookie_names);
+
pthread_mutex_unlock(&state->main_lock);
}
@@ builtin/fsmonitor--daemon.c: static int fsmonitor_run_daemon(void)
+ pthread_cond_init(&state.cookies_cond, NULL);
state.error_code = 0;
state.current_token_data = fsmonitor_new_token_data();
- state.test_client_delay_ms = lookup_client_test_delay();
+
@@ builtin/fsmonitor--daemon.c: static int fsmonitor_run_daemon(void)
state.nr_paths_watching = 2;
}
+ /*
+ * We will write filesystem syncing cookie files into
-+ * <gitdir>/<cookie-prefix><pid>-<seq>.
++ * <gitdir>/<fsmonitor-dir>/<cookie-dir>/<pid>-<seq>.
+ */
+ strbuf_init(&state.path_cookie_prefix, 0);
+ strbuf_addbuf(&state.path_cookie_prefix, &state.path_gitdir_watch);
++
++ strbuf_addch(&state.path_cookie_prefix, '/');
++ strbuf_addstr(&state.path_cookie_prefix, FSMONITOR_DIR);
++ mkdir(state.path_cookie_prefix.buf, 0777);
++
++ strbuf_addch(&state.path_cookie_prefix, '/');
++ strbuf_addstr(&state.path_cookie_prefix, FSMONITOR_COOKIE_DIR);
++ mkdir(state.path_cookie_prefix.buf, 0777);
++
+ strbuf_addch(&state.path_cookie_prefix, '/');
-+ strbuf_addstr(&state.path_cookie_prefix, FSMONITOR_COOKIE_PREFIX);
+
/*
* Confirm that we can create platform-specific resources for the
@@ builtin/fsmonitor--daemon.c: static int fsmonitor_run_daemon(void)
strbuf_release(&state.path_worktree_watch);
strbuf_release(&state.path_gitdir_watch);
+ strbuf_release(&state.path_cookie_prefix);
++
++ /*
++ * NEEDSWORK: Consider "rm -rf <gitdir>/<fsmonitor-dir>"
++ */
return err;
} 20: d699ad597d2c ! 23: 102e17cbc875 fsmonitor: force update index when fsmonitor token advances
@@ Metadata
Author: Jeff Hostetler <[email protected]>
## Commit message ##
- fsmonitor: force update index when fsmonitor token advances
-
- Set the FSMONITOR_CHANGED bit on istate->cache_changed when the
- fsmonitor response contains a different token to ensure that the index
- is written to disk.
-
- Normally, when the fsmonitor response includes a tracked file, the
- index is always updated. Similarly, the index might be updated when
- the response alters the untracked-cache (when enabled). However, in
- cases where neither of those cause the index to be considered changed,
- the fsmonitor response is wasted. And subsequent commands will
- continue to make requests with the same token and if there have not
- been any changes in the working directory, they will receive the same
- response.
-
- This was observed on Windows after a large checkout. On Windows, the
- kernel emits events for the files that are changed as they are
- changed. However, it might delay events for the containing
- directories until the system is more idle (or someone scans the
- directory (so it seems)). The first status following a checkout would
- get the list of files. The subsequent status commands would get the
- list of directories as the events trickled out. But they would never
- catch up because the token was not advanced because the index wasn't
- updated.
-
- This list of directories caused wt_status_collect_untracked() to
- unnecessarily spend time actually scanning them during each command.
+ fsmonitor: enhance existing comments
## fsmonitor.c ##
@@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)
}
- strbuf_release(&query_result);
+ apply_results:
+- /* a fsmonitor process can return '/' to indicate all entries are invalid */
+ /*
-+ * If the fsmonitor response and the subsequent scan of the disk
-+ * did not cause the in-memory index to be marked dirty, then force
-+ * it so that we advance the fsmonitor token in our extension, so
-+ * that future requests don't keep re-requesting the same range.
++ * The response from FSMonitor (excluding the header token) is
++ * either:
++ *
++ * [a] a (possibly empty) list of NUL delimited relative
++ * pathnames of changed paths. This list can contain
++ * files and directories. Directories have a trailing
++ * slash.
++ *
++ * [b] a single '/' to indicate the provider had no
++ * information and that we should consider everything
++ * invalid. We call this a trivial response.
+ */
-+ if (istate->fsmonitor_last_update &&
-+ strcmp(istate->fsmonitor_last_update, last_update_token.buf))
-+ istate->cache_changed |= FSMONITOR_CHANGED;
+ if (query_success && query_result.buf[bol] != '/') {
+- /* Mark all entries returned by the monitor as dirty */
++ /*
++ * Mark all pathnames returned by the monitor as dirty.
++ *
++ * This updates both the cache-entries and the untracked-cache.
++ */
+ buf = query_result.buf;
+ for (i = bol; i < query_result.len; i++) {
+ if (buf[i] != '\0')
+@@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)
+ if (istate->untracked)
+ istate->untracked->use_fsmonitor = 1;
+ } else {
+-
+- /* We only want to run the post index changed hook if we've actually changed entries, so keep track
+- * if we actually changed entries or not */
++ /*
++ * We received a trivial response, so invalidate everything.
++ *
++ * We only want to run the post index changed hook if
++ * we've actually changed entries, so keep track if we
++ * actually changed entries or not.
++ */
+ int is_cache_changed = 0;
+- /* Mark all entries invalid */
+
- /* Now that we've updated istate, save the last_update_token */
- FREE_AND_NULL(istate->fsmonitor_last_update);
- istate->fsmonitor_last_update = strbuf_detach(&last_update_token, NULL);
+ for (i = 0; i < istate->cache_nr; i++) {
+ if (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID) {
+ is_cache_changed = 1;
+@@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)
+ }
+ }
+
+- /* If we're going to check every file, ensure we save the results */
++ /*
++ * If we're going to check every file, ensure we save
++ * the results.
++ */
+ if (is_cache_changed)
+ istate->cache_changed |= FSMONITOR_CHANGED;
+
-: ------------ > 24: 11ea2f97def6 fsmonitor: force update index after large responses
21: 8b2280e5c4d2 ! 25: c9159db718a7 t7527: create test for fsmonitor--daemon
@@ t/t7527-builtin-fsmonitor.sh (new)
+
+. ./test-lib.sh
+
-+# Ask the fsmonitor daemon to insert a little delay before responding to
-+# client commands like `git status` and `git fsmonitor--daemon --query` to
-+# allow recent filesystem events to be received by the daemon. This helps
-+# the CI/PR builds be more stable.
-+#
-+# An arbitrary millisecond value.
-+#
-+GIT_TEST_FSMONITOR_CLIENT_DELAY=1000
-+export GIT_TEST_FSMONITOR_CLIENT_DELAY
-+
+git version --build-options | grep "feature:" | grep "fsmonitor--daemon" || {
+ skip_all="The built-in FSMonitor is not supported on this platform"
+ test_done
@@ t/t7527-builtin-fsmonitor.sh (new)
+
+kill_repo () {
+ r=$1
-+ git -C $r fsmonitor--daemon --stop >/dev/null 2>/dev/null
++ git -C $r fsmonitor--daemon stop >/dev/null 2>/dev/null
+ rm -rf $1
+ return 0
+}
@@ t/t7527-builtin-fsmonitor.sh (new)
+ *) r="";
+ esac
+
-+ git $r fsmonitor--daemon --start || return $?
-+ git $r fsmonitor--daemon --is-running || return $?
++ git $r fsmonitor--daemon start || return $?
++ git $r fsmonitor--daemon status || return $?
+
+ return 0
+}
@@ t/t7527-builtin-fsmonitor.sh (new)
+ git init test_explicit &&
+ start_daemon test_explicit &&
+
-+ git -C test_explicit fsmonitor--daemon --stop &&
-+ test_must_fail git -C test_explicit fsmonitor--daemon --is-running
++ git -C test_explicit fsmonitor--daemon stop &&
++ test_must_fail git -C test_explicit fsmonitor--daemon status
+'
+
+test_expect_success 'implicit daemon start' '
+ test_when_finished "kill_repo test_implicit" &&
+
+ git init test_implicit &&
-+ test_must_fail git -C test_implicit fsmonitor--daemon --is-running &&
++ test_must_fail git -C test_implicit fsmonitor--daemon status &&
+
+ # query will implicitly start the daemon.
+ #
@@ t/t7527-builtin-fsmonitor.sh (new)
+ # implicitly started.)
+
+ GIT_TRACE2_EVENT="$PWD/.git/trace" \
-+ git -C test_implicit fsmonitor--daemon --query 0 >actual &&
++ test-tool -C test_implicit fsmonitor-client query --token 0 >actual &&
+ nul_to_q <actual >actual.filtered &&
+ grep "builtin:" actual.filtered &&
+
@@ t/t7527-builtin-fsmonitor.sh (new)
+
+ grep :\"query/response-length\" .git/trace &&
+
-+ git -C test_implicit fsmonitor--daemon --is-running &&
-+ git -C test_implicit fsmonitor--daemon --stop &&
-+ test_must_fail git -C test_implicit fsmonitor--daemon --is-running
++ git -C test_implicit fsmonitor--daemon status &&
++ git -C test_implicit fsmonitor--daemon stop &&
++ test_must_fail git -C test_implicit fsmonitor--daemon status
+'
+
+test_expect_success 'implicit daemon stop (delete .git)' '
@@ t/t7527-builtin-fsmonitor.sh (new)
+ sleep 1 &&
+ mkdir test_implicit_1/.git &&
+
-+ test_must_fail git -C test_implicit_1 fsmonitor--daemon --is-running
++ test_must_fail git -C test_implicit_1 fsmonitor--daemon status
+'
+
+test_expect_success 'implicit daemon stop (rename .git)' '
@@ t/t7527-builtin-fsmonitor.sh (new)
+ sleep 1 &&
+ mkdir test_implicit_2/.git &&
+
-+ test_must_fail git -C test_implicit_2 fsmonitor--daemon --is-running
++ test_must_fail git -C test_implicit_2 fsmonitor--daemon status
+'
+
+test_expect_success 'cannot start multiple daemons' '
@@ t/t7527-builtin-fsmonitor.sh (new)
+
+ start_daemon test_multiple &&
+
-+ test_must_fail git -C test_multiple fsmonitor--daemon --start 2>actual &&
++ test_must_fail git -C test_multiple fsmonitor--daemon start 2>actual &&
+ grep "fsmonitor--daemon is already running" actual &&
+
-+ git -C test_multiple fsmonitor--daemon --stop &&
-+ test_must_fail git -C test_multiple fsmonitor--daemon --is-running
++ git -C test_multiple fsmonitor--daemon stop &&
++ test_must_fail git -C test_multiple fsmonitor--daemon status
+'
+
+test_expect_success 'setup' '
@@ t/t7527-builtin-fsmonitor.sh (new)
+'
+
+test_expect_success 'update-index implicitly starts daemon' '
-+ test_must_fail git fsmonitor--daemon --is-running &&
++ test_must_fail git fsmonitor--daemon status &&
+
+ GIT_TRACE2_EVENT="$PWD/.git/trace_implicit_1" \
+ git update-index --fsmonitor &&
+
-+ git fsmonitor--daemon --is-running &&
-+ test_might_fail git fsmonitor--daemon --stop &&
++ git fsmonitor--daemon status &&
++ test_might_fail git fsmonitor--daemon stop &&
+
+ grep \"event\":\"start\".*\"fsmonitor--daemon\" .git/trace_implicit_1
+'
+
+test_expect_success 'status implicitly starts daemon' '
-+ test_must_fail git fsmonitor--daemon --is-running &&
++ test_must_fail git fsmonitor--daemon status &&
+
+ GIT_TRACE2_EVENT="$PWD/.git/trace_implicit_2" \
+ git status >actual &&
+
-+ git fsmonitor--daemon --is-running &&
-+ test_might_fail git fsmonitor--daemon --stop &&
++ git fsmonitor--daemon status &&
++ test_might_fail git fsmonitor--daemon stop &&
+
+ grep \"event\":\"start\".*\"fsmonitor--daemon\" .git/trace_implicit_2
+'
@@ t/t7527-builtin-fsmonitor.sh (new)
+clean_up_repo_and_stop_daemon () {
+ git reset --hard HEAD
+ git clean -fd
-+ git fsmonitor--daemon --stop
++ git fsmonitor--daemon stop
+ rm -f .git/trace
+}
+
@@ t/t7527-builtin-fsmonitor.sh (new)
+
+ edit_files &&
+
-+ git fsmonitor--daemon --query 0 >/dev/null 2>&1 &&
++ test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&
+
+ grep "^event: dir1/modified$" .git/trace &&
+ grep "^event: dir2/modified$" .git/trace &&
@@ t/t7527-builtin-fsmonitor.sh (new)
+
+ create_files &&
+
-+ git fsmonitor--daemon --query 0 >/dev/null 2>&1 &&
++ test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&
+
+ grep "^event: dir1/new$" .git/trace &&
+ grep "^event: dir2/new$" .git/trace &&
@@ t/t7527-builtin-fsmonitor.sh (new)
+
+ delete_files &&
+
-+ git fsmonitor--daemon --query 0 >/dev/null 2>&1 &&
++ test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&
+
+ grep "^event: dir1/delete$" .git/trace &&
+ grep "^event: dir2/delete$" .git/trace &&
@@ t/t7527-builtin-fsmonitor.sh (new)
+
+ rename_files &&
+
-+ git fsmonitor--daemon --query 0 >/dev/null 2>&1 &&
++ test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&
+
+ grep "^event: dir1/rename$" .git/trace &&
+ grep "^event: dir2/rename$" .git/trace &&
@@ t/t7527-builtin-fsmonitor.sh (new)
+
+ mv dirtorename dirrenamed &&
+
-+ git fsmonitor--daemon --query 0 >/dev/null 2>&1 &&
++ test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&
+
+ grep "^event: dirtorename/*$" .git/trace &&
+ grep "^event: dirrenamed/*$" .git/trace
@@ t/t7527-builtin-fsmonitor.sh (new)
+
+ file_to_directory &&
+
-+ git fsmonitor--daemon --query 0 >/dev/null 2>&1 &&
++ test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&
+
+ grep "^event: delete$" .git/trace &&
+ grep "^event: delete/new$" .git/trace
@@ t/t7527-builtin-fsmonitor.sh (new)
+
+ directory_to_file &&
+
-+ git fsmonitor--daemon --query 0 >/dev/null 2>&1 &&
++ test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&
+
+ grep "^event: dir1$" .git/trace
+'
@@ t/t7527-builtin-fsmonitor.sh (new)
+# polling fast enough), we need to discard the cached data (relative to the
+# current token) and start collecting events under a new token.
+#
-+# the 'git fsmonitor--daemon --flush' command can be used to send a "flush"
-+# message to a running daemon and ask it to do a flush/resync.
++# the 'test-tool fsmonitor-client flush' command can be used to send a
++# "flush" message to a running daemon and ask it to do a flush/resync.
+
+test_expect_success 'flush cached data' '
+ test_when_finished "kill_repo test_flush" &&
@@ t/t7527-builtin-fsmonitor.sh (new)
+ # then a few (probably platform-specific number of) events in _1.
+ # These should both have the same <token_id>.
+
-+ git -C test_flush fsmonitor--daemon --query "builtin:test_00000001:0" >actual_0 &&
++ test-tool -C test_flush fsmonitor-client query --token "builtin:test_00000001:0" >actual_0 &&
+ nul_to_q <actual_0 >actual_q0 &&
+
+ touch test_flush/file_1 &&
+ touch test_flush/file_2 &&
+
-+ git -C test_flush fsmonitor--daemon --query "builtin:test_00000001:0" >actual_1 &&
++ test-tool -C test_flush fsmonitor-client query --token "builtin:test_00000001:0" >actual_1 &&
+ nul_to_q <actual_1 >actual_q1 &&
+
+ grep "file_1" actual_q1 &&
@@ t/t7527-builtin-fsmonitor.sh (new)
+ # flush the file data. Then create some events and ensure that the file
+ # again appears in the cache. It should have the new <token_id>.
+
-+ git -C test_flush fsmonitor--daemon --flush >flush_0 &&
++ test-tool -C test_flush fsmonitor-client flush >flush_0 &&
+ nul_to_q <flush_0 >flush_q0 &&
+ grep "^builtin:test_00000002:0Q/Q$" flush_q0 &&
+
-+ git -C test_flush fsmonitor--daemon --query "builtin:test_00000002:0" >actual_2 &&
++ test-tool -C test_flush fsmonitor-client query --token "builtin:test_00000002:0" >actual_2 &&
+ nul_to_q <actual_2 >actual_q2 &&
+
+ grep "^builtin:test_00000002:0Q$" actual_q2 &&
+
+ touch test_flush/file_3 &&
+
-+ git -C test_flush fsmonitor--daemon --query "builtin:test_00000002:0" >actual_3 &&
++ test-tool -C test_flush fsmonitor-client query --token "builtin:test_00000002:0" >actual_3 &&
+ nul_to_q <actual_3 >actual_q3 &&
+
+ grep "file_3" actual_q3
@@ t/t7527-builtin-fsmonitor.sh (new)
+ start_daemon wt-secondary
+ ) &&
+
-+ git -C wt-secondary fsmonitor--daemon --stop &&
-+ test_must_fail git -C wt-secondary fsmonitor--daemon --is-running
++ git -C wt-secondary fsmonitor--daemon stop &&
++ test_must_fail git -C wt-secondary fsmonitor--daemon status
+'
+
+test_done
22: da5094e52032 ! 26: 5b035c6e0d60 p7519: add fsmonitor--daemon
@@ t/perf/p7519-fsmonitor.sh: test_expect_success "one time repo setup" '
#
# Choose integration script based on existence of Watchman.
# Fall back to an empty integration script.
+@@ t/perf/p7519-fsmonitor.sh: test_perf_w_drop_caches () {
+ }
+
+ test_fsmonitor_suite() {
+- if test -n "$INTEGRATION_SCRIPT"; then
++ if test -n "$USE_FSMONITOR_DAEMON"
++ then
++ DESC="builtin fsmonitor--daemon"
++ elif test -n "$INTEGRATION_SCRIPT"; then
+ DESC="fsmonitor=$(basename $INTEGRATION_SCRIPT)"
+ else
+ DESC="fsmonitor=disabled"
@@ t/perf/p7519-fsmonitor.sh: test_expect_success "setup without fsmonitor" '
test_fsmonitor_suite
trace_stop
@@ t/perf/p7519-fsmonitor.sh: test_expect_success "setup without fsmonitor" '
+ USE_FSMONITOR_DAEMON=t
+
+ trace_start fsmonitor--daemon--server
-+ git fsmonitor--daemon --start
++ git fsmonitor--daemon start
+
+ trace_start fsmonitor--daemon--client
+ test_expect_success "setup for fsmonitor--daemon" 'setup_for_fsmonitor'
+ test_fsmonitor_suite
+
-+ git fsmonitor--daemon --stop
++ git fsmonitor--daemon stop
+ trace_stop
+fi
+
23: 3eafd0b5cb09 ! 27: 1483c68855cb t7527: test status with untracked-cache and fsmonitor--daemon
@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'setup' '
git -c core.useBuiltinFSMonitor= add . &&
@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'worktree with .git file' '
- test_must_fail git -C wt-secondary fsmonitor--daemon --is-running
+ test_must_fail git -C wt-secondary fsmonitor--daemon status
'
+# TODO Repeat one of the "edit" tests on wt-secondary and confirm that
@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'worktree with .git file' '
+test_expect_success 'Matrix: setup for untracked-cache,fsmonitor matrix' '
+ test_might_fail git config --unset core.useBuiltinFSMonitor &&
+ git update-index --no-fsmonitor &&
-+ test_might_fail git fsmonitor--daemon --stop
++ test_might_fail git fsmonitor--daemon stop
+'
+
+matrix_clean_up_repo () {
@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'worktree with .git file' '
+ test_expect_success "Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor" '
+ test_might_fail git config --unset core.useBuiltinFSMonitor &&
+ git update-index --no-fsmonitor &&
-+ test_might_fail git fsmonitor--daemon --stop 2>/dev/null
++ test_might_fail git fsmonitor--daemon stop 2>/dev/null
+ '
+ else
+ test_expect_success "Matrix[uc:$uc_val][fsm:$fsm_val] enable fsmonitor" '
+ git config core.useBuiltinFSMonitor true &&
-+ git fsmonitor--daemon --start &&
++ git fsmonitor--daemon start &&
+ git update-index --fsmonitor
+ '
+ fi -: ------------ > 28: 96a3eab819f4 t/perf: avoid copying builtin fsmonitor files into test repo
Signature · 1 line
gitgitgadget
PATCH v2 1/28 simple-ipc: preparations for supporting binary messages.
From: Jeff Hostetler <[email protected]>
Add command_len argument to the Simple IPC API.
In my original Simple IPC API, I assumed that the request would always be a null-terminated string of text characters. The command arg was just a const char *.
I found a caller that would like to pass a binary command to the daemon, so I want to ammend the Simple IPC API to take const char *command, size_t command_len and pass that to the daemon. (Really, the first arg should just be a void * or const unsigned byte * to make that clearer.)
Note, the response side has always been a struct strbuf which includes the buffer and length, so we already support returning a binary answer. (Yes, it feels a little weird returning a binary buffer in a strbuf, but it works.)
--- compat/simple-ipc/ipc-unix-socket.c | 14 +++++++----- compat/simple-ipc/ipc-win32.c | 14 +++++++----- simple-ipc.h | 7 ++++-- t/helper/test-simple-ipc.c | 34 +++++++++++++++++++---------- 4 files changed, 46 insertions(+), 23 deletions(-)
compat/simple-ipc/ipc-unix-socket.c +9 -5
@@ -164,7 +164,8 @@ void ipc_client_close_connection(struct ipc_client_connection *connection)int ipc_client_send_command_to_connection (struct ipc_client_connection *connection, -const char *message,struct strbuf *answer) +const char *message,size_t message_len, +struct strbuf *answer) {int ret =0 ;
@@ -172,7 +173,7 @@ int ipc_client_send_command_to_connection(trace2_region_enter ("ipc-client" ,"send-command" ,NULL ); -if (write_packetized_from_buf_no_flush (message,strlen (message), +if (write_packetized_from_buf_no_flush (message, message_len, connection->fd) <0 ||packet_flush_gently (connection->fd) <0 ) { ret =error (_ ("could not send IPC command" ));
@@ -193,7 +194,8 @@ int ipc_client_send_command_to_connection(int ipc_client_send_command (const char *path,const struct ipc_client_connect_options *options, -const char *message,struct strbuf *answer) +const char *message,size_t message_len, +struct strbuf *answer) {int ret =-1 ;enum ipc_active_state state;
@@ -204,7 +206,9 @@ int ipc_client_send_command(const char *path,if (state !=IPC_STATE__LISTENING )return ret; - ret =ipc_client_send_command_to_connection (connection, message, answer); + ret =ipc_client_send_command_to_connection (connection, + message, message_len, + answer);ipc_client_close_connection (connection);
@@ -499,7 +503,7 @@ static int worker_thread__do_io(if (ret >=0 ) { ret = worker_thread_data->server_data->application_cb ( worker_thread_data->server_data->application_data, - buf.buf, do_io_reply_callback, &reply_data); + buf.buf, buf.len, do_io_reply_callback, &reply_data);packet_flush_gently (reply_data.fd); }
compat/simple-ipc/ipc-win32.c +9 -5
@@ -204,7 +204,8 @@ void ipc_client_close_connection(struct ipc_client_connection *connection)int ipc_client_send_command_to_connection (struct ipc_client_connection *connection, -const char *message,struct strbuf *answer) +const char *message,size_t message_len, +struct strbuf *answer) {int ret =0 ;
@@ -212,7 +213,7 @@ int ipc_client_send_command_to_connection(trace2_region_enter ("ipc-client" ,"send-command" ,NULL ); -if (write_packetized_from_buf_no_flush (message,strlen (message), +if (write_packetized_from_buf_no_flush (message, message_len, connection->fd) <0 ||packet_flush_gently (connection->fd) <0 ) { ret =error (_ ("could not send IPC command" ));
@@ -235,7 +236,8 @@ int ipc_client_send_command_to_connection(int ipc_client_send_command (const char *path,const struct ipc_client_connect_options *options, -const char *message,struct strbuf *response) +const char *message,size_t message_len, +struct strbuf *response) {int ret =-1 ;enum ipc_active_state state;
@@ -246,7 +248,9 @@ int ipc_client_send_command(const char *path,if (state !=IPC_STATE__LISTENING )return ret; - ret =ipc_client_send_command_to_connection (connection, message, response); + ret =ipc_client_send_command_to_connection (connection, + message, message_len, + response);ipc_client_close_connection (connection);
@@ -454,7 +458,7 @@ static int do_io(struct ipc_server_thread_data *server_thread_data)if (ret >=0 ) { ret = server_thread_data->server_data->application_cb ( server_thread_data->server_data->application_data, - buf.buf, do_io_reply_callback, &reply_data); + buf.buf, buf.len, do_io_reply_callback, &reply_data);packet_flush_gently (reply_data.fd);
simple-ipc.h +5 -2
@@ -111,7 +111,8 @@ void ipc_client_close_connection(struct ipc_client_connection *connection);*/ int ipc_client_send_command_to_connection (struct ipc_client_connection *connection, -const char *message,struct strbuf *answer); +const char *message,size_t message_len, +struct strbuf *answer);/* * Used by the client to synchronously connect and send and receive a
@@ -123,7 +124,8 @@ int ipc_client_send_command_to_connection(*/ int ipc_client_send_command (const char *path,const struct ipc_client_connect_options *options, -const char *message,struct strbuf *answer); +const char *message,size_t message_len, +struct strbuf *answer);/* * Simple IPC Server Side API.
@@ -148,6 +150,7 @@ typedef int (ipc_server_reply_cb)(struct ipc_server_reply_data *,*/ typedef int (ipc_server_application_cb )(void *application_data,const char *request, +size_t request_len,ipc_server_reply_cb *reply_cb,struct ipc_server_reply_data *reply_data);
t/helper/test-simple-ipc.c +23 -11
@@ -112,7 +112,7 @@ static int app__slow_command(ipc_server_reply_cb *reply_cb,/* * The client sent a command followed by a (possibly very) large buffer. */ -static int app__sendbytes_command (const char *received, +static int app__sendbytes_command (const char *received,size_t received_len,ipc_server_reply_cb *reply_cb,struct ipc_server_reply_data *reply_data) {
@@ -123,6 +123,13 @@ static int app__sendbytes_command(const char *received,int errs =0 ;int ret; +/* +* The test is setup to send: +* "sendbytes" SP <n * char> +*/ +if (received_len <strlen ("sendbytes " )) +BUG ("received_len is short in app__sendbytes_command" ); +if (skip_prefix (received,"sendbytes " , &p)) len_ballast =strlen (p);
@@ -160,7 +167,7 @@ static ipc_server_application_cb test_app_cb;* by this application. */ static int test_app_cb (void *application_data, -const char *command, +const char *command,size_t command_len,ipc_server_reply_cb *reply_cb,struct ipc_server_reply_data *reply_data) {
@@ -173,7 +180,7 @@ static int test_app_cb(void *application_data,if (application_data != (void *)&my_app_data)BUG ("application_cb: application_data pointer wrong" ); -if (!strcmp (command,"quit" )) { +if (command_len ==4 && !strncmp (command,"quit" ,4 )) {/* * The client sent a "quit" command. This is an async * request for the server to shutdown.
@@ -193,22 +200,23 @@ static int test_app_cb(void *application_data,return SIMPLE_IPC_QUIT ; } -if (!strcmp (command,"ping" )) { +if (command_len ==4 && !strncmp (command,"ping" ,4 )) {const char *answer ="pong" ;return reply_cb (reply_data, answer,strlen (answer)); } -if (!strcmp (command,"big" )) +if (command_len ==3 && !strncmp (command,"big" ,3 ))return app__big_command (reply_cb, reply_data); -if (!strcmp (command,"chunk" )) +if (command_len ==5 && !strncmp (command,"chunk" ,5 ))return app__chunk_command (reply_cb, reply_data); -if (!strcmp (command,"slow" )) +if (command_len ==4 && !strncmp (command,"slow" ,4 ))return app__slow_command (reply_cb, reply_data); -if (starts_with (command,"sendbytes " )) -return app__sendbytes_command (command, reply_cb, reply_data); +if (command_len >=10 &&starts_with (command,"sendbytes " )) +return app__sendbytes_command (command, command_len, + reply_cb, reply_data);return app__unhandled_command (command, reply_cb, reply_data); }
@@ -488,7 +496,9 @@ static int client__send_ipc(void) options.wait_if_busy =1 ; options.wait_if_not_found =0 ; -if (!ipc_client_send_command (cl_args.path, &options, command, &buf)) { +if (!ipc_client_send_command (cl_args.path, &options, + command,strlen (command), + &buf)) {if (buf.len) {printf ("%s\n" , buf.buf);fflush (stdout);
@@ -556,7 +566,9 @@ static int do_sendbytes(int bytecount, char byte, const char *path,strbuf_addstr (&buf_send,"sendbytes " );strbuf_addchars (&buf_send, byte, bytecount); -if (!ipc_client_send_command (path, options, buf_send.buf, &buf_resp)) { +if (!ipc_client_send_command (path, options, + buf_send.buf, buf_send.len, + &buf_resp)) {strbuf_rtrim (&buf_resp);printf ("sent:%c%08d %s\n" , byte, bytecount, buf_resp.buf);fflush (stdout);
Signature · 1 line
gitgitgadget
PATCH v2 2/28 fsmonitor--daemon: man page
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Create a manual page describing the git fsmonitor--daemon feature.
--- Documentation/git-fsmonitor--daemon.txt | 75 +++++++++++++++++++++++++ 1 file changed, 75 insertions(+) create mode 100644 Documentation/git-fsmonitor--daemon.txt
Documentation/git-fsmonitor--daemon.txt new file +75 -0
@@ -0,0 +1,75 @@ +git-fsmonitor--daemon(1) +======================== + +NAME +---- +git-fsmonitor--daemon - A Built-in File System Monitor + +SYNOPSIS +-------- +[verse] +'git fsmonitor--daemon' start +'git fsmonitor--daemon' run +'git fsmonitor--daemon' stop +'git fsmonitor--daemon' status + +DESCRIPTION +----------- + +A daemon to watch the working directory for file and directory +changes using platform-specific file system notification facilities. + +This daemon communicates directly with commands like `git status` +using the link:technical/api-simple-ipc.html[simple IPC] interface +instead of the slower linkgit:githooks[5] interface. + +This daemon is built into Git so that no third-party tools are +required. + +OPTIONS +------- + +start:: + Starts a daemon in the background. + +run:: + Runs a daemon in the foreground. + +stop:: + Stops the daemon running in the current working + directory, if present. + +status:: + Exits with zero status if a daemon is watching the + current working directory. + +REMARKS +------- + +This daemon is a long running process used to watch a single working +directory and maintain a list of the recently changed files and +directories. Performance of commands such as `git status` can be +increased if they just ask for a summary of changes to the working +directory and can avoid scanning the disk. + +When `core.useBuiltinFSMonitor` is set to `true` (see +linkgit:git-config[1]) commands, such as `git status`, will ask the +daemon for changes and automatically start it (if necessary). + +For more information see the "File System Monitor" section in +linkgit:git-update-index[1]. + +CAVEATS +------- + +The fsmonitor daemon does not currently know about submodules and does +not know to filter out file system events that happen within a +submodule. If fsmonitor daemon is watching a super repo and a file is +modified within the working directory of a submodule, it will report +the change (as happening against the super repo). However, the client +will properly ignore these extra events, so performance may be affected +but it will not cause an incorrect result. + +GIT +--- +Part of the linkgit:git[1] suite
Signature · 1 line
gitgitgadget
PATCH v2 3/28 fsmonitor--daemon: update fsmonitor documentation
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Update references to core.fsmonitor and core.fsmonitorHookVersion and pointers to Watchman to mention the new built-in fsmonitor--daemon.
--- Documentation/config/core.txt | 56 ++++++++++++++++++++++-------- Documentation/git-update-index.txt | 27 +++++++------- Documentation/githooks.txt | 3 +- 3 files changed, 59 insertions(+), 27 deletions(-)
Documentation/config/core.txt +42 -14
@@ -62,22 +62,50 @@ core.protectNTFS:: Defaults to `true` on Windows, and `false` elsewhere. core.fsmonitor:: - If set, the value of this variable is used as a command which - will identify all files that may have changed since the - requested date/time. This information is used to speed up git by - avoiding unnecessary processing of files that have not changed. - See the "fsmonitor-watchman" section of linkgit:githooks[5]. + If set, this variable contains the pathname of the "fsmonitor" + hook command. ++ +This hook command is used to identify all files that may have changed +since the requested date/time. This information is used to speed up +git by avoiding unnecessary scanning of files that have not changed. ++ +See the "fsmonitor-watchman" section of linkgit:githooks[5]. ++ +Note: The value of this config setting is ignored if the +built-in file system monitor is enabled (see `core.useBuiltinFSMonitor`). core.fsmonitorHookVersion:: - Sets the version of hook that is to be used when calling fsmonitor. - There are currently versions 1 and 2. When this is not set, - version 2 will be tried first and if it fails then version 1 - will be tried. Version 1 uses a timestamp as input to determine - which files have changes since that time but some monitors - like watchman have race conditions when used with a timestamp. - Version 2 uses an opaque string so that the monitor can return - something that can be used to determine what files have changed - without race conditions. + Sets the protocol version to be used when invoking the + "fsmonitor" hook. ++ +There are currently versions 1 and 2. When this is not set, +version 2 will be tried first and if it fails then version 1 +will be tried. Version 1 uses a timestamp as input to determine +which files have changes since that time but some monitors +like Watchman have race conditions when used with a timestamp. +Version 2 uses an opaque string so that the monitor can return +something that can be used to determine what files have changed +without race conditions. ++ +Note: The value of this config setting is ignored if the +built-in file system monitor is enabled (see `core.useBuiltinFSMonitor`). + +core.useBuiltinFSMonitor:: + If set to true, enable the built-in file system monitor + daemon for this working directory (linkgit:git-fsmonitor--daemon[1]). ++ +Like hook-based file system monitors, the built-in file system monitor +can speed up Git commands that need to refresh the Git index +(e.g. `git status`) in a working directory with many files. The +built-in monitor eliminates the need to install and maintain an +external third-party tool. ++ +The built-in file system monitor is currently available only on a +limited set of supported platforms. Currently, this includes Windows +and MacOS. ++ +Note: if this config setting is set to `true`, the values of +`core.fsmonitor` and `core.fsmonitorHookVersion` are ignored. core.trustctime:: If false, the ctime differences between the index and the
Documentation/git-update-index.txt +15 -12
@@ -498,7 +498,9 @@ FILE SYSTEM MONITOR This feature is intended to speed up git operations for repos that have large working directories. -It enables git to work together with a file system monitor (see the +It enables git to work together with a file system monitor (see +linkgit:git-fsmonitor--daemon[1] +and the "fsmonitor-watchman" section of linkgit:githooks[5]) that can inform it as to what files have been modified. This enables git to avoid having to lstat() every file to find modified files.
@@ -508,17 +510,18 @@ performance by avoiding the cost of scanning the entire working directory looking for new files. If you want to enable (or disable) this feature, it is easier to use -the `core.fsmonitor` configuration variable (see -linkgit:git-config[1]) than using the `--fsmonitor` option to -`git update-index` in each repository, especially if you want to do so -across all repositories you use, because you can set the configuration -variable in your `$HOME/.gitconfig` just once and have it affect all -repositories you touch. - -When the `core.fsmonitor` configuration variable is changed, the -file system monitor is added to or removed from the index the next time -a command reads the index. When `--[no-]fsmonitor` are used, the file -system monitor is immediately added to or removed from the index. +the `core.fsmonitor` or `core.useBuiltinFSMonitor` configuration +variable (see linkgit:git-config[1]) than using the `--fsmonitor` +option to `git update-index` in each repository, especially if you +want to do so across all repositories you use, because you can set the +configuration variable in your `$HOME/.gitconfig` just once and have +it affect all repositories you touch. + +When the `core.fsmonitor` or `core.useBuiltinFSMonitor` configuration +variable is changed, the file system monitor is added to or removed +from the index the next time a command reads the index. When +`--[no-]fsmonitor` are used, the file system monitor is immediately +added to or removed from the index. CONFIGURATION -------------
Documentation/githooks.txt +2 -1
@@ -593,7 +593,8 @@ fsmonitor-watchman This hook is invoked when the configuration option `core.fsmonitor` is set to `.git/hooks/fsmonitor-watchman` or `.git/hooks/fsmonitor-watchmanv2` -depending on the version of the hook to use. +depending on the version of the hook to use, unless overridden via +`core.useBuiltinFSMonitor` (see linkgit:git-config[1]). Version 1 takes two arguments, a version (1) and the time in elapsed nanoseconds since midnight, January 1, 1970.
Signature · 1 line
gitgitgadget
PATCH v2 4/28 fsmonitor-ipc: create client routines for git-fsmonitor--daemon
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Create fsmonitor_ipc__*() client routines to spawn the built-in file system monitor daemon and send it an IPC request using the Simple IPC API.
Stub in empty fsmonitor_ipc__*() functions for unsupported platforms.
--- Makefile | 1 + fsmonitor-ipc.c | 179 ++++++++++++++++++++++++++++++++++++++++++++++++ fsmonitor-ipc.h | 48 +++++++++++++ 3 files changed, 228 insertions(+) create mode 100644 fsmonitor-ipc.c create mode 100644 fsmonitor-ipc.h
Makefile +1 -0
@@ -892,6 +892,7 @@ LIB_OBJS += fetch-pack.o LIB_OBJS += fmt-merge-msg.o LIB_OBJS += fsck.o LIB_OBJS += fsmonitor.o +LIB_OBJS += fsmonitor-ipc.o LIB_OBJS += gettext.o LIB_OBJS += gpg-interface.o LIB_OBJS += graph.o
fsmonitor-ipc.c new file +179 -0
179 lines not on this page · raw message
fsmonitor-ipc.h new file +48 -0
48 lines not on this page · raw message
Signature · 1 line
gitgitgadget
Re: [PATCH v2 04/28] fsmonitor-ipc: create client routines for git-fsmonitor--daemon
Hi Jeff,
I know you're on vacation, therefore I would like to apologize for adding to your post-vacation notification overload, but...
Quoted 26 lines · Jeff Hostetler via GitGitGadget
From: Jeff Hostetler <[email protected]>
fsmonitor-ipc.c new file +17
-0@@ -0,0 +1,179 @@ + +int fsmonitor_ipc__send_query (const char *since_token, +struct strbuf *answer) +{ +int ret =-1 ; +int tried_to_spawn =0 ; +enum ipc_active_state state =IPC_STATE__OTHER_ERROR ; +struct ipc_client_connection *connection =NULL ; +struct ipc_client_connect_options options + =IPC_CLIENT_CONNECT_OPTIONS_INIT ; + + options.wait_if_busy =1 ; + options.wait_if_not_found =0 ; + +trace2_region_enter ("fsm_client" ,"query" ,NULL ); + + trace2_data_string("fsm_client", NULL, "query/command",
fsmonitor-ipc.c line 56
Quoted 10 lines
fsmonitor-ipc.c new file +10
-0+ since_token); + +try_again: + state =ipc_client_try_connect (fsmonitor_ipc__get_path (), &options, + &connection); + +switch (state) { +case IPC_STATE__LISTENING : + ret =ipc_client_send_command_to_connection ( + connection, since_token,strlen (since_token), answer);
Here, since_token can be NULL (and hence the strlen(since_token) can lead to a segmentation fault). I ran into this situation while git rebase -i --autostash wanted to apply the stashed changes.
Since I picked up your v2 and included it in Git for Windows v2.32.0-rc2, I needed this hotfix: https://github.com/git-for-windows/git/pull/3241
Thanks,
Dscho
Re: [PATCH v2 04/28] fsmonitor-ipc: create client routines for git-fsmonitor--daemon
Hi Jeff,
Quoted 2 lines · Johannes Schindelin
I know you're on vacation, therefore I would like to apologize for adding to your post-vacation notification overload, but...
Now that you're back from vacation...
Quoted 43 lines · Jeff Hostetler via GitGitGadget
Quoted 27 lines
From: Jeff Hostetler <[email protected]>
fsmonitor-ipc.c new file +17
-0@@ -0,0 +1,179 @@ + +int fsmonitor_ipc__send_query (const char *since_token, +struct strbuf *answer) +{ +int ret =-1 ; +int tried_to_spawn =0 ; +enum ipc_active_state state =IPC_STATE__OTHER_ERROR ; +struct ipc_client_connection *connection =NULL ; +struct ipc_client_connect_options options + =IPC_CLIENT_CONNECT_OPTIONS_INIT ; + + options.wait_if_busy =1 ; + options.wait_if_not_found =0 ; + +trace2_region_enter ("fsm_client" ,"query" ,NULL ); + + trace2_data_string("fsm_client", NULL, "query/command",fsmonitor-ipc.c
Quoted 10 lines
fsmonitor-ipc.c new file +10
-0+ since_token); + +try_again: + state =ipc_client_try_connect (fsmonitor_ipc__get_path (), &options, + &connection); + +switch (state) { +case IPC_STATE__LISTENING : + ret =ipc_client_send_command_to_connection ( + connection, since_token,strlen (since_token), answer);Here,
since_tokencan beNULL(and hence thestrlen(since_token)can lead to a segmentation fault). I ran into this situation whilegit rebase -i --autostashwanted to apply the stashed changes.
Quoted 2 lines
Since I picked up your v2 and included it in Git for Windows v2.32.0-rc2, I needed this hotfix: https://github.com/git-for-windows/git/pull/3241
I actually noticed another similar issue and fixed it in time for Git for Windows v2.32.0, but eventually figured out the actual culprit, with a much better fix:
-- snip --
commit bc40a560d3c95040b55fd7be6fe5b7012d267f8f
Author: Johannes Schindelin <[email protected]> Date: Wed Jun 9 09:49:50 2021 +0200
fixup! fsmonitor: introduce core.useBuiltinFSMonitor to call the daemon via IPC
In FSMonitor v1, we made sure to only use a valid since_token when querying the FSMonitor. This condition was accidentally lost in v2, and caused segmentation faults uncovered by Scalar's Functional Tests.
I had tried to fix this in https://github.com/git-for-windows/pull/3241,
but the fix was incomplete, and I had to follow up with
https://github.com/git-for-windows/pull/3258. However, it turns out that both of them were actually only work-arounds; I should have dug deeper to figure out _why_ the since_token was no longer guaranteed not to be NULL, and I finally did.
fsmonitor.c +3 -2
@@ -290,8 +290,9 @@ void refresh_fsmonitor(struct index_state *istate)trace_printf_key (&trace_fsmonitor,"refresh fsmonitor" );if (r->settings.use_builtin_fsmonitor >0 ) { - query_success = !fsmonitor_ipc__send_query ( - istate->fsmonitor_last_update, &query_result); + query_success = istate->fsmonitor_last_update && + !fsmonitor_ipc__send_query (istate->fsmonitor_last_update, + &query_result);if (query_success) {/* * The response contains a series of nul terminated
-- snap --
Would you mind squashing this in when you re-roll?
Ciao,
Dscho
PATCH v2 5/28 help: include fsmonitor--daemon feature flag in version info
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Add the "feature: fsmonitor--daemon" message to the output of git version --build-options.
This allows users to know if the built-in fsmonitor feature is supported on their platform.
--- help.c | 4 ++++ 1 file changed, 4 insertions(+)
help.c +4 -0
@@ -11,6 +11,7 @@#include "version.h" #include "refs.h" #include "parse-options.h" +#include "fsmonitor-ipc.h" struct category_description {uint32_t category;
@@ -664,6 +665,9 @@ void get_version_info(struct strbuf *buf, int show_build_options)strbuf_addf (buf,"sizeof-size_t: %d\n" , (int )sizeof (size_t ));strbuf_addf (buf,"shell-path: %s\n" ,SHELL_PATH );/* NEEDSWORK: also save and output GIT-BUILD_OPTIONS? */ + +if (fsmonitor_ipc__is_supported ()) +strbuf_addstr (buf,"feature: fsmonitor--daemon\n" ); } }
Signature · 1 line
gitgitgadget
PATCH v2 6/28 config: FSMonitor is repository-specific
In reply to Jeff Hostetler via GitGitGadgetFrom: Johannes Schindelin <[email protected]>
This commit refactors git_config_get_fsmonitor() into the repo_*() form that takes a parameter struct repository *r.
That change prepares for the upcoming core.useBuiltinFSMonitor flag which will be stored in the repo_settings struct.
--- builtin/update-index.c | 4 ++-- config.c | 4 ++-- config.h | 2 +- fsmonitor.c | 2 +- 4 files changed, 6 insertions(+), 6 deletions(-)
builtin/update-index.c +2 -2
@@ -1214,14 +1214,14 @@ int cmd_update_index(int argc, const char **argv, const char *prefix) }if (fsmonitor >0 ) { -if (git_config_get_fsmonitor () ==0 ) +if (repo_config_get_fsmonitor (r) ==0 )warning (_ ("core.fsmonitor is unset; " "set it if you really want to " "enable fsmonitor" ));add_fsmonitor (&the_index);report (_ ("fsmonitor enabled" )); }else if (!fsmonitor) { -if (git_config_get_fsmonitor () ==1 ) +if (repo_config_get_fsmonitor (r) ==1 )warning (_ ("core.fsmonitor is set; " "remove it if you really want to " "disable fsmonitor" ));
config.c +2 -2
@@ -2499,9 +2499,9 @@ int git_config_get_max_percent_split_change(void)return -1 ;/* default value */ } -int git_config_get_fsmonitor (void ) +int repo_config_get_fsmonitor (struct repository *r) { -if (git_config_get_pathname ("core.fsmonitor" , &core_fsmonitor)) +if (repo_config_get_pathname (r,"core.fsmonitor" , &core_fsmonitor)) core_fsmonitor =getenv ("GIT_TEST_FSMONITOR" );if (core_fsmonitor && !*core_fsmonitor)
config.h +1 -1
@@ -607,7 +607,7 @@ int git_config_get_index_threads(int *dest);int git_config_get_untracked_cache (void );int git_config_get_split_index (void );int git_config_get_max_percent_split_change (void ); -int git_config_get_fsmonitor (void ); +int repo_config_get_fsmonitor (struct repository *r);/* This dies if the configured or default date is in the future */ int git_config_get_expiry (const char *key,const char **output);
fsmonitor.c +1 -1
@@ -411,7 +411,7 @@ void remove_fsmonitor(struct index_state *istate)void tweak_fsmonitor (struct index_state *istate) {unsigned int i; -int fsmonitor_enabled =git_config_get_fsmonitor (); +int fsmonitor_enabled =repo_config_get_fsmonitor (istate->repo ? istate->repo : the_repository);if (istate->fsmonitor_dirty) {if (fsmonitor_enabled) {
Signature · 1 line
gitgitgadget
PATCH v2 7/28 fsmonitor: introduce `core.useBuiltinFSMonitor` to call the daemon via IPC
In reply to Jeff Hostetler via GitGitGadgetFrom: Johannes Schindelin <[email protected]>
Use simple IPC to directly communicate with the new builtin file system monitor daemon.
Define a new config setting core.useBuiltinFSMonitor to enable the builtin file system monitor.
The core.fsmonitor setting has already been defined as a HOOK pathname. Historically, this has been set to a HOOK script that will talk with Watchman. For compatibility reasons, we do not want to overload that definition (and cause problems if users have multiple versions of Git installed).
--- config.c | 5 +++++ fsmonitor.c | 43 +++++++++++++++++++++++++++++++++++++++++++ repo-settings.c | 3 +++ repository.h | 2 ++ 4 files changed, 53 insertions(+)
config.c +5 -0
@@ -2501,6 +2501,11 @@ int git_config_get_max_percent_split_change(void)int repo_config_get_fsmonitor (struct repository *r) { +if (r->settings.use_builtin_fsmonitor >0 ) { + core_fsmonitor ="(built-in daemon)" ; +return 1 ; + } +if (repo_config_get_pathname (r,"core.fsmonitor" , &core_fsmonitor)) core_fsmonitor =getenv ("GIT_TEST_FSMONITOR" );
fsmonitor.c +43 -0
67 lines not on this page · raw message
repo-settings.c +3 -0
@@ -58,6 +58,9 @@ void prepare_repo_settings(struct repository *r) r->settings.core_multi_pack_index = value;UPDATE_DEFAULT_BOOL (r->settings.core_multi_pack_index,1 ); +if (!repo_config_get_bool (r,"core.usebuiltinfsmonitor" , &value) && value) + r->settings.use_builtin_fsmonitor =1 ; +if (!repo_config_get_bool (r,"feature.manyfiles" , &value) && value) {UPDATE_DEFAULT_BOOL (r->settings.index_version,4 );UPDATE_DEFAULT_BOOL (r->settings.core_untracked_cache,UNTRACKED_CACHE_WRITE );
repository.h +2 -0
@@ -29,6 +29,8 @@ enum fetch_negotiation_setting {struct repo_settings {int initialized; +int use_builtin_fsmonitor; +int core_commit_graph;int commit_graph_read_changed_paths;int gc_write_commit_graph;
Signature · 1 line
gitgitgadget
Re: [PATCH v2 07/28] fsmonitor: introduce `core.useBuiltinFSMonitor` to call the daemon via IPC
Hi Jeff,
Quoted 21 lines · Johannes Schindelin via GitGitGadget
fsmonitor.c +2
-0@@ -3,6 +3,7 @@#include "dir.h" #include "ewah/ewok.h" #include "fsmonitor.h" +#include "fsmonitor-ipc.h" #include "run-command.h" #include "strbuf.h" @@ -231,6 +232,7 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)void refresh_fsmonitor (struct index_state *istate) { +struct repository *r = istate->repo ? istate->repo : the_repository;struct strbuf query_result =STRBUF_INIT ;int query_success =0 , hook_version =-1 ;size_t bol =0 ;/* beginning of line */
fsmonitor.c void refresh_fsmonitor(struct index_state *istate) · line 255
Quoted 7 lines
fsmonitor.c +4
-0@@ -247,6 +249,46 @@ void refresh_fsmonitor(struct index_state *istate) istate->fsmonitor_has_run_once =1 ;trace_printf_key (&trace_fsmonitor,"refresh fsmonitor" ); + +if (r->settings.use_builtin_fsmonitor >0 ) { + query_success = !fsmonitor_ipc__send_query ( + istate->fsmonitor_last_update, &query_result);
As I pointed out elsewhere in the thread, this is a slight change in
behavior: in the previous iteration, we had this call in query_fsmonitor(), which was only ever called if istate->fsmonitor_last_update is non-NULL.
The code in fsmonitor_ipc__send_query() does actually depend on this, therefore we need this change to be squashed in:
-- snip --
fsmonitor.c +3 -2
@@ -290,8 +290,9 @@ void refresh_fsmonitor(struct index_state *istate)trace_printf_key (&trace_fsmonitor,"refresh fsmonitor" );if (r->settings.use_builtin_fsmonitor >0 ) { - query_success = !fsmonitor_ipc__send_query ( - istate->fsmonitor_last_update, &query_result); + query_success = istate->fsmonitor_last_update && + !fsmonitor_ipc__send_query (istate->fsmonitor_last_update, + &query_result);if (query_success) {/* * The response contains a series of nul terminated
-- snap --
Thanks,
Dscho
Quoted 78 lines
fsmonitor.c +37
-0+if (query_success) { +/* +* The response contains a series of nul terminated +* strings. The first is the new token. +* +* Use `char *buf` as an interlude to trick the CI +* static analysis to let us use `strbuf_addstr()` +* here (and only copy the token) rather than +* `strbuf_addbuf()`. +*/ + buf = query_result.buf; +strbuf_addstr (&last_update_token, buf); + bol = last_update_token.len +1 ; + }else { +/* +* The builtin daemon is not available on this +* platform -OR- we failed to get a response. +* +* Generate a fake token (rather than a V1 +* timestamp) for the index extension. (If +* they switch back to the hook API, we don't +* want ambiguous state.) +*/ +strbuf_addstr (&last_update_token,"builtin:fake" ); + } + +/* +* Regardless of whether we successfully talked to a +* fsmonitor daemon or not, we skip over and do not +* try to use the hook. The "core.useBuiltinFSMonitor" +* config setting ALWAYS overrides the "core.fsmonitor" +* hook setting. +*/ + goto apply_results; + } +/* * This could be racy so save the date/time now and query_fsmonitor * should be inclusive to ensure we don't miss potential changes. @@ -301,6 +343,7 @@ void refresh_fsmonitor(struct index_state *istate) core_fsmonitor, query_success ? "success" : "failure"); } +apply_results:/* a fsmonitor process can return '/' to indicate all entries are invalid */ if (query_success && query_result.buf[bol] !='/' ) {/* Mark all entries returned by the monitor as dirty */ repo-settings.c +3
-0@@ -58,6 +58,9 @@ void prepare_repo_settings(struct repository *r) r->settings.core_multi_pack_index = value;UPDATE_DEFAULT_BOOL (r->settings.core_multi_pack_index,1 ); +if (!repo_config_get_bool (r,"core.usebuiltinfsmonitor" , &value) && value) + r->settings.use_builtin_fsmonitor =1 ; +if (!repo_config_get_bool (r,"feature.manyfiles" , &value) && value) {UPDATE_DEFAULT_BOOL (r->settings.index_version,4 );UPDATE_DEFAULT_BOOL (r->settings.core_untracked_cache,UNTRACKED_CACHE_WRITE );repository.h +2
-1@@ -29,6 +29,8 @@ enum fetch_negotiation_setting {struct repo_settings {int initialized; +int use_builtin_fsmonitor; +int core_commit_graph;int commit_graph_read_changed_paths;int gc_write_commit_graph; --gitgitgadget
PATCH v2 8/28 fsmonitor--daemon: add a built-in fsmonitor daemon
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Create a built-in file system monitoring daemon that can be used by the existing fsmonitor feature (protocol API and index extension) to improve the performance of various Git commands, such as status.
The fsmonitor--daemon feature builds upon the Simple IPC API and provides an alternative to hook access to existing fsmonitors such as watchman.
This commit merely adds the new command without any functionality.
--- .gitignore | 1 + Makefile | 1 + builtin.h | 1 + builtin/fsmonitor--daemon.c | 53 +++++++++++++++++++++++++++++++++++++ git.c | 1 + 5 files changed, 57 insertions(+) create mode 100644 builtin/fsmonitor--daemon.c
.gitignore +1 -0
@@ -71,6 +71,7 @@ /git-format-patch /git-fsck /git-fsck-objects +/git-fsmonitor--daemon /git-gc /git-get-tar-commit-id /git-grep
Makefile +1 -0
@@ -1092,6 +1092,7 @@ BUILTIN_OBJS += builtin/fmt-merge-msg.o BUILTIN_OBJS += builtin/for-each-ref.o BUILTIN_OBJS += builtin/for-each-repo.o BUILTIN_OBJS += builtin/fsck.o +BUILTIN_OBJS += builtin/fsmonitor--daemon.o BUILTIN_OBJS += builtin/gc.o BUILTIN_OBJS += builtin/get-tar-commit-id.o BUILTIN_OBJS += builtin/grep.o
builtin.h +1 -0
@@ -158,6 +158,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix);int cmd_for_each_repo (int argc,const char **argv,const char *prefix);int cmd_format_patch (int argc,const char **argv,const char *prefix);int cmd_fsck (int argc,const char **argv,const char *prefix); +int cmd_fsmonitor__daemon (int argc,const char **argv,const char *prefix);int cmd_gc (int argc,const char **argv,const char *prefix);int cmd_get_tar_commit_id (int argc,const char **argv,const char *prefix);int cmd_grep (int argc,const char **argv,const char *prefix);
builtin/fsmonitor--daemon.c new file +53 -0
53 lines not on this page · raw message
git.c +1 -0
@@ -523,6 +523,7 @@ static struct cmd_struct commands[] = { { "format-patch", cmd_format_patch, RUN_SETUP }, { "fsck", cmd_fsck, RUN_SETUP }, { "fsck-objects", cmd_fsck, RUN_SETUP }, + { "fsmonitor--daemon", cmd_fsmonitor__daemon, RUN_SETUP }, { "gc", cmd_gc, RUN_SETUP }, { "get-tar-commit-id", cmd_get_tar_commit_id, NO_PARSEOPT }, { "grep", cmd_grep, RUN_SETUP_GENTLY },
Signature · 1 line
gitgitgadget
PATCH v2 9/28 fsmonitor--daemon: implement client command options
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Implement stop and status client commands to control and query the status of a fsmonitor--daemon server process (and implicitly start a server process if necessary).
Later commits will implement the actual server and monitor the file system.
--- builtin/fsmonitor--daemon.c | 49 +++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+)
builtin/fsmonitor--daemon.c +49 -0
@@ -7,10 +7,53 @@#include "khash.h" static const char *const builtin_fsmonitor__daemon_usage[] = { +N_ ("git fsmonitor--daemon stop" ), +N_ ("git fsmonitor--daemon status" ),NULL };#ifdef HAVE_FSMONITOR_DAEMON_BACKEND +/* +* Acting as a CLIENT. +* +* Send a "quit" command to the `git-fsmonitor--daemon` (if running) +* and wait for it to shutdown. +*/ +static int do_as_client__send_stop (void ) +{ +struct strbuf answer =STRBUF_INIT ; +int ret; + + ret =fsmonitor_ipc__send_command ("quit" , &answer); + +/* The quit command does not return any response data. */ +strbuf_release (&answer); + +if (ret) +return ret; + +trace2_region_enter ("fsm_client" ,"polling-for-daemon-exit" ,NULL ); +while (fsmonitor_ipc__get_state () ==IPC_STATE__LISTENING ) +sleep_millisec (50 ); +trace2_region_leave ("fsm_client" ,"polling-for-daemon-exit" ,NULL ); + +return 0 ; +} + +static int do_as_client__status (void ) +{ +enum ipc_active_state state =fsmonitor_ipc__get_state (); + +switch (state) { +case IPC_STATE__LISTENING : +printf (_ ("The built-in file system monitor is active\n" )); +return 0 ; + +default : +printf (_ ("The built-in file system monitor is not active\n" )); +return 1 ; + } +}int cmd_fsmonitor__daemon (int argc,const char **argv,const char *prefix) {
@@ -35,6 +78,12 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) argc =parse_options (argc, argv, prefix, options, builtin_fsmonitor__daemon_usage,0 ); +if (!strcmp (subcmd,"stop" )) +return !!do_as_client__send_stop (); + +if (!strcmp (subcmd,"status" )) +return !!do_as_client__status (); +die (_ ("Unhandled subcommand '%s'" ), subcmd); }
Signature · 1 line
gitgitgadget
PATCH v2 10/28 t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Create an IPC client to send query and flush commands to the daemon.
--- Makefile | 1 + t/helper/test-fsmonitor-client.c | 125 +++++++++++++++++++++++++++++++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + 4 files changed, 128 insertions(+) create mode 100644 t/helper/test-fsmonitor-client.c
Makefile +1 -0
@@ -709,6 +709,7 @@ TEST_BUILTINS_OBJS += test-dump-split-index.o TEST_BUILTINS_OBJS += test-dump-untracked-cache.o TEST_BUILTINS_OBJS += test-example-decorate.o TEST_BUILTINS_OBJS += test-fast-rebase.o +TEST_BUILTINS_OBJS += test-fsmonitor-client.o TEST_BUILTINS_OBJS += test-genrandom.o TEST_BUILTINS_OBJS += test-genzeros.o TEST_BUILTINS_OBJS += test-hash-speed.o
t/helper/test-fsmonitor-client.c new file +125 -0
125 lines not on this page · raw message
t/helper/test-tool.c +1 -0
@@ -31,6 +31,7 @@ static struct test_cmd cmds[] = { { "dump-untracked-cache", cmd__dump_untracked_cache }, { "example-decorate", cmd__example_decorate }, { "fast-rebase", cmd__fast_rebase }, + { "fsmonitor-client", cmd__fsmonitor_client }, { "genrandom", cmd__genrandom }, { "genzeros", cmd__genzeros }, { "hashmap", cmd__hashmap },
t/helper/test-tool.h +1 -0
@@ -21,6 +21,7 @@ int cmd__dump_split_index(int argc, const char **argv);int cmd__dump_untracked_cache (int argc,const char **argv);int cmd__example_decorate (int argc,const char **argv);int cmd__fast_rebase (int argc,const char **argv); +int cmd__fsmonitor_client (int argc,const char **argv);int cmd__genrandom (int argc,const char **argv);int cmd__genzeros (int argc,const char **argv);int cmd__hashmap (int argc,const char **argv);
Signature · 1 line
gitgitgadget
Re: [PATCH v2 10/28] t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon
t/helper/test-fsmonitor-client.c line 13
Quoted 8 lines · Jeff Hostetler via GitGitGadget
t/helper/test-fsmonitor-client.c new file +8
-0+#include "test-tool.h" +#include "cache.h" +#include "parse-options.h" +//#include "fsmonitor.h" +#include "fsmonitor-ipc.h" +//#include "compat/fsmonitor/fsmonitor-fs-listen.h" +//#include "fsmonitor--daemon.h" +//#include "simple-ipc.h"
Please never commit a commented-out code. Comments are for humans (and they shouldn't use the // style in this project)---use removal for machines.
Thanks.
PATCH v2 11/28 fsmonitor-fs-listen-win32: stub in backend for Windows
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Stub in empty backend for fsmonitor--daemon on Windows.
--- Makefile | 13 ++++++ compat/fsmonitor/fsmonitor-fs-listen-win32.c | 21 +++++++++ compat/fsmonitor/fsmonitor-fs-listen.h | 49 ++++++++++++++++++++ config.mak.uname | 2 + contrib/buildsystems/CMakeLists.txt | 5 ++ 5 files changed, 90 insertions(+) create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-win32.c create mode 100644 compat/fsmonitor/fsmonitor-fs-listen.h
Makefile +13 -0
@@ -467,6 +467,11 @@ all:: # directory, and the JSON compilation database 'compile_commands.json' will be # created at the root of the repository. # +# If your platform supports a built-in fsmonitor backend, set +# FSMONITOR_DAEMON_BACKEND to the "<name>" of the corresponding +# `compat/fsmonitor/fsmonitor-fs-listen-<name>.c` that implements the +# `fsmonitor_fs_listen__*()` routines. +# # Define DEVELOPER to enable more compiler warnings. Compiler version # and family are auto detected, but could be overridden by defining # COMPILER_FEATURES (see config.mak.dev). You can still set
@@ -1906,6 +1911,11 @@ ifdef NEED_ACCESS_ROOT_HANDLER COMPAT_OBJS += compat/access.o endif +ifdef FSMONITOR_DAEMON_BACKEND + COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND + COMPAT_OBJS += compat/fsmonitor/fsmonitor-fs-listen-$(FSMONITOR_DAEMON_BACKEND).o +endif + ifeq ($(TCLTK_PATH),) NO_TCLTK = NoThanks endif
@@ -2765,6 +2775,9 @@ GIT-BUILD-OPTIONS: FORCE @echo PAGER_ENV=\''$(subst ','\'',$(subst ','\'',$(PAGER_ENV)))'\' >>$@+ @echo DC_SHA1=\''$(subst ','\'',$(subst ','\'',$(DC_SHA1)))'\' >>$@+ @echo X=\'$(X)\' >>$@+ +ifdef FSMONITOR_DAEMON_BACKEND + @echo FSMONITOR_DAEMON_BACKEND=\''$(subst ','\'',$(subst ','\'',$(FSMONITOR_DAEMON_BACKEND)))'\' >>$@+ +endif ifdef TEST_OUTPUT_DIRECTORY @echo TEST_OUTPUT_DIRECTORY=\''$(subst ','\'',$(subst ','\'',$(TEST_OUTPUT_DIRECTORY)))'\' >>$@+ endif
compat/fsmonitor/fsmonitor-fs-listen-win32.c new file +21 -0
@@ -0,0 +1,21 @@ +#include "cache.h" +#include "config.h" +#include "fsmonitor.h" +#include "fsmonitor-fs-listen.h" + +void fsmonitor_fs_listen__stop_async (struct fsmonitor_daemon_state *state) +{ +} + +void fsmonitor_fs_listen__loop (struct fsmonitor_daemon_state *state) +{ +} + +int fsmonitor_fs_listen__ctor (struct fsmonitor_daemon_state *state) +{ +return -1 ; +} + +void fsmonitor_fs_listen__dtor (struct fsmonitor_daemon_state *state) +{ +}
compat/fsmonitor/fsmonitor-fs-listen.h new file +49 -0
49 lines not on this page · raw message
config.mak.uname +2 -0
@@ -420,6 +420,7 @@ ifeq ($(uname_S),Windows) # so we don't need this: # # SNPRINTF_RETURNS_BOGUS = YesPlease + FSMONITOR_DAEMON_BACKEND = win32 NO_SVN_TESTS = YesPlease RUNTIME_PREFIX = YesPlease HAVE_WPGMPTR = YesWeDo
@@ -598,6 +599,7 @@ ifneq (,$(findstring MINGW,$(uname_S))) NO_STRTOUMAX = YesPlease NO_MKDTEMP = YesPlease NO_SVN_TESTS = YesPlease + FSMONITOR_DAEMON_BACKEND = win32 RUNTIME_PREFIX = YesPlease HAVE_WPGMPTR = YesWeDo NO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease
contrib/buildsystems/CMakeLists.txt +5 -0
@@ -256,6 +256,11 @@ else() list(APPEND compat_SOURCES compat/simple-ipc/ipc-shared.c compat/simple-ipc/ipc-unix-socket.c) endif() +if(CMAKE_SYSTEM_NAME STREQUAL "Windows") + add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) + list(APPEND compat_SOURCES compat/fsmonitor/fsmonitor-fs-listen-win32.c) +endif() + set(EXE_EXTENSION ${CMAKE_EXECUTABLE_SUFFIX}) #header checks
Signature · 1 line
gitgitgadget
PATCH v2 12/28 fsmonitor-fs-listen-macos: stub in backend for MacOS
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Stub in empty implementation of fsmonitor--daemon
backend for MacOS.
--- compat/fsmonitor/fsmonitor-fs-listen-macos.c | 20 ++++++++++++++++++++ config.mak.uname | 2 ++ contrib/buildsystems/CMakeLists.txt | 3 +++ 3 files changed, 25 insertions(+) create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-macos.c
compat/fsmonitor/fsmonitor-fs-listen-macos.c new file +20 -0
@@ -0,0 +1,20 @@ +#include "cache.h" +#include "fsmonitor.h" +#include "fsmonitor-fs-listen.h" + +int fsmonitor_fs_listen__ctor (struct fsmonitor_daemon_state *state) +{ +return -1 ; +} + +void fsmonitor_fs_listen__dtor (struct fsmonitor_daemon_state *state) +{ +} + +void fsmonitor_fs_listen__stop_async (struct fsmonitor_daemon_state *state) +{ +} + +void fsmonitor_fs_listen__loop (struct fsmonitor_daemon_state *state) +{ +}
config.mak.uname +2 -0
@@ -147,6 +147,8 @@ ifeq ($(uname_S),Darwin) MSGFMT = /usr/local/opt/gettext/bin/msgfmt endif endif + FSMONITOR_DAEMON_BACKEND = macos + BASIC_LDFLAGS += -framework CoreServices endif ifeq ($(uname_S),SunOS) NEEDS_SOCKET = YesPlease
contrib/buildsystems/CMakeLists.txt +3 -0
@@ -259,6 +259,9 @@ endif() if(CMAKE_SYSTEM_NAME STREQUAL "Windows") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsmonitor-fs-listen-win32.c) +elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") + add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) + list(APPEND compat_SOURCES compat/fsmonitor/fsmonitor-fs-listen-macos.c) endif() set(EXE_EXTENSION ${CMAKE_EXECUTABLE_SUFFIX})
Signature · 1 line
gitgitgadget
PATCH v2 13/28 fsmonitor--daemon: implement daemon command options
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Implement run and start commands to try to begin listening for file system events.
This version defines the thread structure with a single fsmonitor_fs_listen thread to watch for file system events and a simple IPC thread pool to wait for connections from Git clients over a well-known named pipe or Unix domain socket.
This version does not actually do anything yet because the backends are still just stubs.
--- builtin/fsmonitor--daemon.c | 390 +++++++++++++++++++++++++++++++++++- fsmonitor--daemon.h | 34 ++++ 2 files changed, 423 insertions(+), 1 deletion(-) create mode 100644 fsmonitor--daemon.h
builtin/fsmonitor--daemon.c +389 -1
@@ -3,16 +3,52 @@#include "parse-options.h" #include "fsmonitor.h" #include "fsmonitor-ipc.h" +#include "compat/fsmonitor/fsmonitor-fs-listen.h" +#include "fsmonitor--daemon.h" #include "simple-ipc.h" #include "khash.h" static const char *const builtin_fsmonitor__daemon_usage[] = { +N_ ("git fsmonitor--daemon start [<options>]" ), +N_ ("git fsmonitor--daemon run [<options>]" ),N_ ("git fsmonitor--daemon stop" ),N_ ("git fsmonitor--daemon status" ),NULL };#ifdef HAVE_FSMONITOR_DAEMON_BACKEND +/* +* Global state loaded from config. +*/ +#define FSMONITOR__IPC_THREADS "fsmonitor.ipcthreads" +static int fsmonitor__ipc_threads =8 ; + +#define FSMONITOR__START_TIMEOUT "fsmonitor.starttimeout" +static int fsmonitor__start_timeout_sec =60 ; + +static int fsmonitor_config (const char *var,const char *value,void *cb) +{ +if (!strcmp (var,FSMONITOR__IPC_THREADS )) { +int i =git_config_int (var, value); +if (i <1 ) +return error (_ ("value of '%s' out of range: %d" ), +FSMONITOR__IPC_THREADS , i); + fsmonitor__ipc_threads = i; +return 0 ; + } + +if (!strcmp (var,FSMONITOR__START_TIMEOUT )) { +int i =git_config_int (var, value); +if (i <0 ) +return error (_ ("value of '%s' out of range: %d" ), +FSMONITOR__START_TIMEOUT , i); + fsmonitor__start_timeout_sec = i; +return 0 ; + } + +return git_default_config (var, value, cb); +} +/* * Acting as a CLIENT. *
@@ -55,11 +91,354 @@ static int do_as_client__status(void) } } +static ipc_server_application_cb handle_client; + +static int handle_client (void *data, +const char *command,size_t command_len, +ipc_server_reply_cb *reply, +struct ipc_server_reply_data *reply_data) +{ +/* struct fsmonitor_daemon_state *state = data; */ +int result; + +/* +* The Simple IPC API now supports {char*, len} arguments, but +* FSMonitor always uses proper null-terminated strings, so +* we can ignore the command_len argument. (Trust, but verify.) +*/ +if (command_len !=strlen (command)) +BUG ("FSMonitor assumes text messages" ); + +trace2_region_enter ("fsmonitor" ,"handle_client" , the_repository); +trace2_data_string ("fsmonitor" , the_repository,"request" , command); + + result =0 ;/* TODO Do something here. */ + +trace2_region_leave ("fsmonitor" ,"handle_client" , the_repository); + +return result; +} + +static void *fsmonitor_fs_listen__thread_proc (void *_state) +{ +struct fsmonitor_daemon_state *state = _state; + +trace2_thread_start ("fsm-listen" ); + +trace_printf_key (&trace_fsmonitor,"Watching: worktree '%s'" , + state->path_worktree_watch.buf); +if (state->nr_paths_watching >1 ) +trace_printf_key (&trace_fsmonitor,"Watching: gitdir '%s'" , + state->path_gitdir_watch.buf); + +fsmonitor_fs_listen__loop (state); + +trace2_thread_exit (); +return NULL ; +} + +static int fsmonitor_run_daemon_1 (struct fsmonitor_daemon_state *state) +{ +struct ipc_server_opts ipc_opts = { + .nr_threads = fsmonitor__ipc_threads, + +/* +* We know that there are no other active threads yet, +* so we can let the IPC layer temporarily chdir() if +* it needs to when creating the server side of the +* Unix domain socket. +*/ + .uds_disallow_chdir =0 + }; + +/* +* Start the IPC thread pool before the we've started the file +* system event listener thread so that we have the IPC handle +* before we need it. +*/ +if (ipc_server_run_async (&state->ipc_server_data, +fsmonitor_ipc__get_path (), &ipc_opts, + handle_client, state)) +return error (_ ("could not start IPC thread pool" )); + +/* +* Start the fsmonitor listener thread to collect filesystem +* events. +*/ +if (pthread_create (&state->listener_thread,NULL , + fsmonitor_fs_listen__thread_proc, state) <0 ) { +ipc_server_stop_async (state->ipc_server_data); +ipc_server_await (state->ipc_server_data); + +return error (_ ("could not start fsmonitor listener thread" )); + } + +/* +* The daemon is now fully functional in background threads. +* Wait for the IPC thread pool to shutdown (whether by client +* request or from filesystem activity). +*/ +ipc_server_await (state->ipc_server_data); + +/* +* The fsmonitor listener thread may have received a shutdown +* event from the IPC thread pool, but it doesn't hurt to tell +* it again. And wait for it to shutdown. +*/ +fsmonitor_fs_listen__stop_async (state); +pthread_join (state->listener_thread,NULL ); + +return state->error_code; +} + +static int fsmonitor_run_daemon (void ) +{ +struct fsmonitor_daemon_state state; +int err; + +memset (&state,0 ,sizeof (state)); + +pthread_mutex_init (&state.main_lock,NULL ); + state.error_code =0 ; + state.current_token_data =NULL ; + +/* Prepare to (recursively) watch the <worktree-root> directory. */ +strbuf_init (&state.path_worktree_watch,0 ); +strbuf_addstr (&state.path_worktree_watch,absolute_path (get_git_work_tree ())); + state.nr_paths_watching =1 ; + +/* +* We create/delete cookie files inside the .git directory to +* help us keep sync with the file system. If ".git" is not a +* directory, then <gitdir> is not inside the cone of +* <worktree-root>, so set up a second watch for it. +*/ +strbuf_init (&state.path_gitdir_watch,0 ); +strbuf_addbuf (&state.path_gitdir_watch, &state.path_worktree_watch); +strbuf_addstr (&state.path_gitdir_watch,"/.git" ); +if (!is_directory (state.path_gitdir_watch.buf)) { +strbuf_reset (&state.path_gitdir_watch); +strbuf_addstr (&state.path_gitdir_watch,absolute_path (get_git_dir ())); + state.nr_paths_watching =2 ; + } + +/* +* Confirm that we can create platform-specific resources for the +* filesystem listener before we bother starting all the threads. +*/ +if (fsmonitor_fs_listen__ctor (&state)) { + err =error (_ ("could not initialize listener thread" )); + goto done; + } + + err =fsmonitor_run_daemon_1 (&state); + +done: +pthread_mutex_destroy (&state.main_lock); +fsmonitor_fs_listen__dtor (&state); + +ipc_server_free (state.ipc_server_data); + +strbuf_release (&state.path_worktree_watch); +strbuf_release (&state.path_gitdir_watch); + +return err; +} + +static int try_to_run_foreground_daemon (void ) +{ +/* +* Technically, we don't need to probe for an existing daemon +* process, since we could just call `fsmonitor_run_daemon()` +* and let it fail if the pipe/socket is busy. +* +* However, this method gives us a nicer error message for a +* common error case. +*/ +if (fsmonitor_ipc__get_state () ==IPC_STATE__LISTENING ) +die ("fsmonitor--daemon is already running." ); + +return !!fsmonitor_run_daemon (); +} + +#ifndef GIT_WINDOWS_NATIVE +/* +* This is adapted from `daemonize()`. Use `fork()` to directly create +* and run the daemon in a child process. The fork-parent returns the +* child PID so that we can wait for the child to startup before exiting. +*/ +static int spawn_background_fsmonitor_daemon (pid_t *pid) +{ + *pid =fork (); + +switch (*pid) { +case 0 : +if (setsid () ==-1 ) +error_errno (_ ("setsid failed" )); +close (0 ); +close (1 ); +close (2 ); +sanitize_stdfds (); + +return !!fsmonitor_run_daemon (); + +case -1 : +return error_errno (_ ("could not spawn fsmonitor--daemon in the background" )); + +default : +return 0 ; + } +} +#else +/* +* Conceptually like `daemonize()` but different because Windows does not +* have `fork(2)`. Spawn a normal Windows child process but without the +* limitations of `start_command()` and `finish_command()`. +*/ +static int spawn_background_fsmonitor_daemon (pid_t *pid) +{ +char git_exe[MAX_PATH ]; +struct strvec args =STRVEC_INIT ; +int in, out; + +GetModuleFileNameA (NULL , git_exe,MAX_PATH ); + + in =open ("/dev/null" ,O_RDONLY ); + out =open ("/dev/null" ,O_WRONLY ); + +strvec_push (&args, git_exe); +strvec_push (&args,"fsmonitor--daemon" ); +strvec_push (&args,"run" ); + + *pid =mingw_spawnvpe (args.v[0 ], args.v,NULL ,NULL , in, out, out); +close (in); +close (out); + +strvec_clear (&args); + +if (*pid <0 ) +return error (_ ("could not spawn fsmonitor--daemon in the background" )); + +return 0 ; +} +#endif + +/* +* This is adapted from `wait_or_whine()`. Watch the child process and +* let it get started and begin listening for requests on the socket +* before reporting our success. +*/ +static int wait_for_background_startup (pid_t pid_child) +{ +int status; +pid_t pid_seen; +enum ipc_active_state s; +time_t time_limit, now; + +time (&time_limit); + time_limit += fsmonitor__start_timeout_sec; + +for (;;) { + pid_seen =waitpid (pid_child, &status,WNOHANG ); + +if (pid_seen ==-1 ) +return error_errno (_ ("waitpid failed" )); +else if (pid_seen ==0 ) { +/* +* The child is still running (this should be +* the normal case). Try to connect to it on +* the socket and see if it is ready for +* business. +* +* If there is another daemon already running, +* our child will fail to start (possibly +* after a timeout on the lock), but we don't +* care (who responds) if the socket is live. +*/ + s =fsmonitor_ipc__get_state (); +if (s ==IPC_STATE__LISTENING ) +return 0 ; + +time (&now); +if (now > time_limit) +return error (_ ("fsmonitor--daemon not online yet" )); + }else if (pid_seen == pid_child) { +/* +* The new child daemon process shutdown while +* it was starting up, so it is not listening +* on the socket. +* +* Try to ping the socket in the odd chance +* that another daemon started (or was already +* running) while our child was starting. +* +* Again, we don't care who services the socket. +*/ + s =fsmonitor_ipc__get_state (); +if (s ==IPC_STATE__LISTENING ) +return 0 ; + +/* +* We don't care about the WEXITSTATUS() nor +* any of the WIF*(status) values because +* `cmd_fsmonitor__daemon()` does the `!!result` +* trick on all function return values. +* +* So it is sufficient to just report the +* early shutdown as an error. +*/ +return error (_ ("fsmonitor--daemon failed to start" )); + }else +return error (_ ("waitpid is confused" )); + } +} + +static int try_to_start_background_daemon (void ) +{ +pid_t pid_child; +int ret; + +/* +* Before we try to create a background daemon process, see +* if a daemon process is already listening. This makes it +* easier for us to report an already-listening error to the +* console, since our spawn/daemon can only report the success +* of creating the background process (and not whether it +* immediately exited). +*/ +if (fsmonitor_ipc__get_state () ==IPC_STATE__LISTENING ) +die ("fsmonitor--daemon is already running." ); + +/* +* Run the actual daemon in a background process. +*/ + ret =spawn_background_fsmonitor_daemon (&pid_child); +if (pid_child <=0 ) +return ret; + +/* +* Wait (with timeout) for the background child process get +* started and begin listening on the socket/pipe. This makes +* the "start" command more synchronous and more reliable in +* tests. +*/ + ret =wait_for_background_startup (pid_child); + +return ret; +} +int cmd_fsmonitor__daemon (int argc,const char **argv,const char *prefix) {const char *subcmd;struct option options[] = { +OPT_INTEGER (0 ,"ipc-threads" , + &fsmonitor__ipc_threads, +N_ ("use <n> ipc worker threads" )), +OPT_INTEGER (0 ,"start-timeout" , + &fsmonitor__start_timeout_sec, +N_ ("Max seconds to wait for background daemon startup" )), +OPT_END () };
@@ -69,7 +448,7 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)if (argc ==2 && !strcmp (argv[1 ],"-h" ))usage_with_options (builtin_fsmonitor__daemon_usage, options); -git_config (git_default_config,NULL ); +git_config (fsmonitor_config,NULL ); subcmd = argv[1 ]; argv--;
@@ -77,6 +456,15 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) argc =parse_options (argc, argv, prefix, options, builtin_fsmonitor__daemon_usage,0 ); +if (fsmonitor__ipc_threads <1 ) +die (_ ("invalid 'ipc-threads' value (%d)" ), + fsmonitor__ipc_threads); + +if (!strcmp (subcmd,"start" )) +return !!try_to_start_background_daemon (); + +if (!strcmp (subcmd,"run" )) +return !!try_to_run_foreground_daemon ();if (!strcmp (subcmd,"stop" ))return !!do_as_client__send_stop ();
fsmonitor--daemon.h new file +34 -0
@@ -0,0 +1,34 @@ +#ifndef FSMONITOR_DAEMON_H +#define FSMONITOR_DAEMON_H + +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND + +#include "cache.h" +#include "dir.h" +#include "run-command.h" +#include "simple-ipc.h" +#include "thread-utils.h" + +struct fsmonitor_batch ; +struct fsmonitor_token_data ; + +struct fsmonitor_daemon_backend_data ;/* opaque platform-specific data */ + +struct fsmonitor_daemon_state { +pthread_t listener_thread; +pthread_mutex_t main_lock; + +struct strbuf path_worktree_watch; +struct strbuf path_gitdir_watch; +int nr_paths_watching; + +struct fsmonitor_token_data *current_token_data; + +int error_code; +struct fsmonitor_daemon_backend_data *backend_data; + +struct ipc_server_data *ipc_server_data; +}; + +#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */ +#endif /* FSMONITOR_DAEMON_H */
Signature · 1 line
gitgitgadget
PATCH v2 14/28 fsmonitor--daemon: add pathname classification
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Teach fsmonitor--daemon to classify relative and absolute pathnames and decide how they should be handled. This will be used by the platform-specific backend to respond to each filesystem event.
When we register for filesystem notifications on a directory, we get events for everything (recursively) in the directory. We want to report to clients changes to tracked and untracked paths within the working directory. We do not want to report changes within the .git directory, for example.
This classification will be used in a later commit by the different backends to classify paths as events are received.
--- builtin/fsmonitor--daemon.c | 81 +++++++++++++++++++++++++++++++++++++ fsmonitor--daemon.h | 61 ++++++++++++++++++++++++++++ 2 files changed, 142 insertions(+)
builtin/fsmonitor--daemon.c +81 -0
@@ -119,6 +119,87 @@ static int handle_client(void *data,return result; } +#define FSMONITOR_COOKIE_PREFIX ".fsmonitor-daemon-" + +enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative ( +const char *rel) +{ +if (fspathncmp (rel,".git" ,4 )) +return IS_WORKDIR_PATH ; + rel +=4 ; + +if (!*rel) +return IS_DOT_GIT ; +if (*rel !='/' ) +return IS_WORKDIR_PATH ;/* e.g. .gitignore */ + rel++; + +if (!fspathncmp (rel,FSMONITOR_COOKIE_PREFIX , +strlen (FSMONITOR_COOKIE_PREFIX ))) +return IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX ; + +return IS_INSIDE_DOT_GIT ; +} + +enum fsmonitor_path_type fsmonitor_classify_path_gitdir_relative ( +const char *rel) +{ +if (!fspathncmp (rel,FSMONITOR_COOKIE_PREFIX , +strlen (FSMONITOR_COOKIE_PREFIX ))) +return IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX ; + +return IS_INSIDE_GITDIR ; +} + +static enum fsmonitor_path_type try_classify_workdir_abs_path ( +struct fsmonitor_daemon_state *state, +const char *path) +{ +const char *rel; + +if (fspathncmp (path, state->path_worktree_watch.buf, + state->path_worktree_watch.len)) +return IS_OUTSIDE_CONE ; + + rel = path + state->path_worktree_watch.len; + +if (!*rel) +return IS_WORKDIR_PATH ;/* it is the root dir exactly */ +if (*rel !='/' ) +return IS_OUTSIDE_CONE ; + rel++; + +return fsmonitor_classify_path_workdir_relative (rel); +} + +enum fsmonitor_path_type fsmonitor_classify_path_absolute ( +struct fsmonitor_daemon_state *state, +const char *path) +{ +const char *rel; +enum fsmonitor_path_type t; + + t =try_classify_workdir_abs_path (state, path); +if (state->nr_paths_watching ==1 ) +return t; +if (t !=IS_OUTSIDE_CONE ) +return t; + +if (fspathncmp (path, state->path_gitdir_watch.buf, + state->path_gitdir_watch.len)) +return IS_OUTSIDE_CONE ; + + rel = path + state->path_gitdir_watch.len; + +if (!*rel) +return IS_GITDIR ;/* it is the <gitdir> exactly */ +if (*rel !='/' ) +return IS_OUTSIDE_CONE ; + rel++; + +return fsmonitor_classify_path_gitdir_relative (rel); +} +static void *fsmonitor_fs_listen__thread_proc (void *_state) {struct fsmonitor_daemon_state *state = _state;
fsmonitor--daemon.h +61 -0
66 lines not on this page · raw message
Signature · 1 line
gitgitgadget
PATCH v2 15/28 fsmonitor--daemon: define token-ids
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Teach fsmonitor--daemon to create token-ids and define the overall token naming scheme.
--- builtin/fsmonitor--daemon.c | 116 +++++++++++++++++++++++++++++++++++- 1 file changed, 115 insertions(+), 1 deletion(-)
builtin/fsmonitor--daemon.c +115 -1
@@ -91,6 +91,120 @@ static int do_as_client__status(void) } } +/* +* Requests to and from a FSMonitor Protocol V2 provider use an opaque +* "token" as a virtual timestamp. Clients can request a summary of all +* created/deleted/modified files relative to a token. In the response, +* clients receive a new token for the next (relative) request. +* +* +* Token Format +* ============ +* +* The contents of the token are private and provider-specific. +* +* For the built-in fsmonitor--daemon, we define a token as follows: +* +* "builtin" ":" <token_id> ":" <sequence_nr> +* +* The "builtin" prefix is used as a namespace to avoid conflicts +* with other providers (such as Watchman). +* +* The <token_id> is an arbitrary OPAQUE string, such as a GUID, +* UUID, or {timestamp,pid}. It is used to group all filesystem +* events that happened while the daemon was monitoring (and in-sync +* with the filesystem). +* +* Unlike FSMonitor Protocol V1, it is not defined as a timestamp +* and does not define less-than/greater-than relationships. +* (There are too many race conditions to rely on file system +* event timestamps.) +* +* The <sequence_nr> is a simple integer incremented whenever the +* daemon needs to make its state public. For example, if 1000 file +* system events come in, but no clients have requested the data, +* the daemon can continue to accumulate file changes in the same +* bin and does not need to advance the sequence number. However, +* as soon as a client does arrive, the daemon needs to start a new +* bin and increment the sequence number. +* +* The sequence number serves as the boundary between 2 sets +* of bins -- the older ones that the client has already seen +* and the newer ones that it hasn't. +* +* When a new <token_id> is created, the <sequence_nr> is reset to +* zero. +* +* +* About Token Ids +* =============== +* +* A new token_id is created: +* +* [1] each time the daemon is started. +* +* [2] any time that the daemon must re-sync with the filesystem +* (such as when the kernel drops or we miss events on a very +* active volume). +* +* [3] in response to a client "flush" command (for dropped event +* testing). +* +* When a new token_id is created, the daemon is free to discard all +* cached filesystem events associated with any previous token_ids. +* Events associated with a non-current token_id will never be sent +* to a client. A token_id change implicitly means that the daemon +* has gap in its event history. +* +* Therefore, clients that present a token with a stale (non-current) +* token_id will always be given a trivial response. +*/ +struct fsmonitor_token_data { +struct strbuf token_id; +struct fsmonitor_batch *batch_head; +struct fsmonitor_batch *batch_tail; +uint64_t client_ref_count; +}; + +static struct fsmonitor_token_data *fsmonitor_new_token_data (void ) +{ +static int test_env_value =-1 ; +static uint64_t flush_count =0 ; +struct fsmonitor_token_data *token; + +CALLOC_ARRAY (token,1 ); + +strbuf_init (&token->token_id,0 ); + token->batch_head =NULL ; + token->batch_tail =NULL ; + token->client_ref_count =0 ; + +if (test_env_value <0 ) + test_env_value =git_env_bool ("GIT_TEST_FSMONITOR_TOKEN" ,0 ); + +if (!test_env_value) { +struct timeval tv; +struct tm tm; +time_t secs; + +gettimeofday (&tv,NULL ); + secs = tv.tv_sec; +gmtime_r (&secs, &tm); + +strbuf_addf (&token->token_id, +"%" PRIu64".%d.%4d%02d%02dT%02d%02d%02d.%06ldZ" , + flush_count++, +getpid (), + tm.tm_year +1900 , tm.tm_mon +1 , tm.tm_mday, + tm.tm_hour, tm.tm_min, tm.tm_sec, + (long )tv.tv_usec); + }else { +strbuf_addf (&token->token_id,"test_%08x" , test_env_value++); + } + +return token; +} +static ipc_server_application_cb handle_client; static int handle_client(void *data,
@@ -281,7 +395,7 @@ static int fsmonitor_run_daemon(void)pthread_mutex_init (&state.main_lock,NULL ); state.error_code =0 ; - state.current_token_data =NULL ; + state.current_token_data =fsmonitor_new_token_data ();/* Prepare to (recursively) watch the <worktree-root> directory. */ strbuf_init (&state.path_worktree_watch,0 );
Signature · 1 line
gitgitgadget
PATCH v2 16/28 fsmonitor--daemon: create token-based changed path cache
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Teach fsmonitor--daemon to build a list of changed paths and associate them with a token-id. This will be used by the platform-specific backends to accumulate changed paths in response to filesystem events.
The platform-specific file system listener thread receives file system events containing one or more changed pathnames (with whatever bucketing or grouping that is convenient for the file system). These paths are accumulated (without locking) by the file system layer into a fsmonitor_batch.
When the file system layer has drained the kernel event queue, it will "publish" them to our token queue and make them visible to concurrent client worker threads. The token layer is free to combine and/or de-dup paths within these batches for efficient presentation to clients.
--- builtin/fsmonitor--daemon.c | 234 +++++++++++++++++++++++++++++++++++- fsmonitor--daemon.h | 40 ++++++ 2 files changed, 272 insertions(+), 2 deletions(-)
builtin/fsmonitor--daemon.c +232 -2
@@ -166,17 +166,27 @@ struct fsmonitor_token_data {uint64_t client_ref_count; }; +struct fsmonitor_batch { +struct fsmonitor_batch *next; +uint64_t batch_seq_nr; +const char **interned_paths; +size_t nr, alloc; +time_t pinned_time; +}; +static struct fsmonitor_token_data *fsmonitor_new_token_data (void ) {static int test_env_value =-1 ;static uint64_t flush_count =0 ;struct fsmonitor_token_data *token; +struct fsmonitor_batch *batch;CALLOC_ARRAY (token,1 ); + batch =fsmonitor_batch__new ();strbuf_init (&token->token_id,0 ); - token->batch_head =NULL ; - token->batch_tail =NULL ; + token->batch_head = batch; + token->batch_tail = batch; token->client_ref_count =0 ;if (test_env_value <0 )
@@ -202,9 +212,147 @@ static struct fsmonitor_token_data *fsmonitor_new_token_data(void)strbuf_addf (&token->token_id,"test_%08x" , test_env_value++); } +/* +* We created a new <token_id> and are starting a new series +* of tokens with a zero <seq_nr>. +* +* Since clients cannot guess our new (non test) <token_id> +* they will always receive a trivial response (because of the +* mismatch on the <token_id>). The trivial response will +* tell them our new <token_id> so that subsequent requests +* will be relative to our new series. (And when sending that +* response, we pin the current head of the batch list.) +* +* Even if the client correctly guesses the <token_id>, their +* request of "builtin:<token_id>:0" asks for all changes MORE +* RECENT than batch/bin 0. +* +* This implies that it is a waste to accumulate paths in the +* initial batch/bin (because they will never be transmitted). +* +* So the daemon could be running for days and watching the +* file system, but doesn't need to actually accumulate any +* paths UNTIL we need to set a reference point for a later +* relative request. +* +* However, it is very useful for testing to always have a +* reference point set. Pin batch 0 to force early file system +* events to accumulate. +*/ +if (test_env_value) + batch->pinned_time =time (NULL ); +return token; } +struct fsmonitor_batch *fsmonitor_batch__new (void ) +{ +struct fsmonitor_batch *batch; + +CALLOC_ARRAY (batch,1 ); + +return batch; +} + +struct fsmonitor_batch *fsmonitor_batch__pop (struct fsmonitor_batch *batch) +{ +struct fsmonitor_batch *next; + +if (!batch) +return NULL ; + + next = batch->next; + +/* +* The actual strings within the array are interned, so we don't +* own them. +*/ +free (batch->interned_paths); + +return next; +} + +void fsmonitor_batch__add_path (struct fsmonitor_batch *batch, +const char *path) +{ +const char *interned_path =strintern (path); + +trace_printf_key (&trace_fsmonitor,"event: %s" , interned_path); + +ALLOC_GROW (batch->interned_paths, batch->nr +1 , batch->alloc); + batch->interned_paths[batch->nr++] = interned_path; +} + +static void fsmonitor_batch__combine (struct fsmonitor_batch *batch_dest, +const struct fsmonitor_batch *batch_src) +{ +size_t k; + +ALLOC_GROW (batch_dest->interned_paths, + batch_dest->nr + batch_src->nr +1 , + batch_dest->alloc); + +for (k =0 ; k < batch_src->nr; k++) + batch_dest->interned_paths[batch_dest->nr++] = + batch_src->interned_paths[k]; +} + +static void fsmonitor_free_token_data (struct fsmonitor_token_data *token) +{ +struct fsmonitor_batch *p; + +if (!token) +return ; + +assert (token->client_ref_count ==0 ); + +strbuf_release (&token->token_id); + +for (p = token->batch_head; p; p =fsmonitor_batch__pop (p)) + ; + +free (token); +} + +/* +* Flush all of our cached data about the filesystem. Call this if we +* lose sync with the filesystem and miss some notification events. +* +* [1] If we are missing events, then we no longer have a complete +* history of the directory (relative to our current start token). +* We should create a new token and start fresh (as if we just +* booted up). +* +* If there are no concurrent threads readering the current token data +* series, we can free it now. Otherwise, let the last reader free +* it. +* +* Either way, the old token data series is no longer associated with +* our state data. +*/ +static void with_lock__do_force_resync (struct fsmonitor_daemon_state *state) +{ +/* assert current thread holding state->main_lock */ + +struct fsmonitor_token_data *free_me =NULL ; +struct fsmonitor_token_data *new_one =NULL ; + + new_one =fsmonitor_new_token_data (); + +if (state->current_token_data->client_ref_count ==0 ) + free_me = state->current_token_data; + state->current_token_data = new_one; + +fsmonitor_free_token_data (free_me); +} + +void fsmonitor_force_resync (struct fsmonitor_daemon_state *state) +{ +pthread_mutex_lock (&state->main_lock); +with_lock__do_force_resync (state); +pthread_mutex_unlock (&state->main_lock); +} +static ipc_server_application_cb handle_client; static int handle_client(void *data,
@@ -314,6 +462,81 @@ enum fsmonitor_path_type fsmonitor_classify_path_absolute(return fsmonitor_classify_path_gitdir_relative (rel); } +/* +* We try to combine small batches at the front of the batch-list to avoid +* having a long list. This hopefully makes it a little easier when we want +* to truncate and maintain the list. However, we don't want the paths array +* to just keep growing and growing with realloc, so we insert an arbitrary +* limit. +*/ +#define MY_COMBINE_LIMIT (1024) + +void fsmonitor_publish (struct fsmonitor_daemon_state *state, +struct fsmonitor_batch *batch, +const struct string_list *cookie_names) +{ +if (!batch && !cookie_names->nr) +return ; + +pthread_mutex_lock (&state->main_lock); + +if (batch) { +struct fsmonitor_batch *head; + + head = state->current_token_data->batch_head; +if (!head) { +BUG ("token does not have batch" ); + }else if (head->pinned_time) { +/* +* We cannot alter the current batch list +* because: +* +* [a] it is being transmitted to at least one +* client and the handle_client() thread has a +* ref-count, but not a lock on the batch list +* starting with this item. +* +* [b] it has been transmitted in the past to +* at least one client such that future +* requests are relative to this head batch. +* +* So, we can only prepend a new batch onto +* the front of the list. +*/ + batch->batch_seq_nr = head->batch_seq_nr +1 ; + batch->next = head; + state->current_token_data->batch_head = batch; + }else if (!head->batch_seq_nr) { +/* +* Batch 0 is unpinned. See the note in +* `fsmonitor_new_token_data()` about why we +* don't need to accumulate these paths. +*/ +fsmonitor_batch__pop (batch); + }else if (head->nr + batch->nr >MY_COMBINE_LIMIT ) { +/* +* The head batch in the list has never been +* transmitted to a client, but folding the +* contents of the new batch onto it would +* exceed our arbitrary limit, so just prepend +* the new batch onto the list. +*/ + batch->batch_seq_nr = head->batch_seq_nr +1 ; + batch->next = head; + state->current_token_data->batch_head = batch; + }else { +/* +* We are free to append the paths in the given +* batch onto the end of the current head batch. +*/ +fsmonitor_batch__combine (head, batch); +fsmonitor_batch__pop (batch); + } + } + +pthread_mutex_unlock (&state->main_lock); +} +static void *fsmonitor_fs_listen__thread_proc (void *_state) {struct fsmonitor_daemon_state *state = _state;
@@ -328,6 +551,13 @@ static void *fsmonitor_fs_listen__thread_proc(void *_state)fsmonitor_fs_listen__loop (state); +pthread_mutex_lock (&state->main_lock); +if (state->current_token_data && + state->current_token_data->client_ref_count ==0 ) +fsmonitor_free_token_data (state->current_token_data); + state->current_token_data =NULL ; +pthread_mutex_unlock (&state->main_lock); +trace2_thread_exit ();return NULL ; }
fsmonitor--daemon.h +40 -0
@@ -12,6 +12,27 @@struct fsmonitor_batch ;struct fsmonitor_token_data ; +/* +* Create a new batch of path(s). The returned batch is considered +* private and not linked into the fsmonitor daemon state. The caller +* should fill this batch with one or more paths and then publish it. +*/ +struct fsmonitor_batch *fsmonitor_batch__new (void ); + +/* +* Free this batch and return the value of the batch->next field. +*/ +struct fsmonitor_batch *fsmonitor_batch__pop (struct fsmonitor_batch *batch); + +/* +* Add this path to this batch of modified files. +* +* The batch should be private and NOT (yet) linked into the fsmonitor +* daemon state and therefore not yet visible to worker threads and so +* no locking is required. +*/ +void fsmonitor_batch__add_path (struct fsmonitor_batch *batch,const char *path); +struct fsmonitor_daemon_backend_data ;/* opaque platform-specific data */ struct fsmonitor_daemon_state {
@@ -91,5 +112,24 @@ enum fsmonitor_path_type fsmonitor_classify_path_absolute(struct fsmonitor_daemon_state *state, const char *path); +/* +* Prepend the this batch of path(s) onto the list of batches associated +* with the current token. This makes the batch visible to worker threads. +* +* The caller no longer owns the batch and must not free it. +* +* Wake up the client threads waiting on these cookies. +*/ +void fsmonitor_publish (struct fsmonitor_daemon_state *state, +struct fsmonitor_batch *batch, +const struct string_list *cookie_names); + +/* +* If the platform-specific layer loses sync with the filesystem, +* it should call this to invalidate cached data and abort waiting +* threads. +*/ +void fsmonitor_force_resync (struct fsmonitor_daemon_state *state); +#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */ #endif /* FSMONITOR_DAEMON_H */
Signature · 1 line
gitgitgadget
PATCH v2 17/28 fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Teach the win32 backend to register a watch on the working tree root directory (recursively). Also watch the <gitdir> if it is not inside the working tree. And to collect path change notifications into batches and publish.
--- compat/fsmonitor/fsmonitor-fs-listen-win32.c | 532 +++++++++++++++++++ 1 file changed, 532 insertions(+)
compat/fsmonitor/fsmonitor-fs-listen-win32.c +532 -0
@@ -2,20 +2,552 @@#include "config.h" #include "fsmonitor.h" #include "fsmonitor-fs-listen.h" +#include "fsmonitor--daemon.h" + +/* +* The documentation of ReadDirectoryChangesW() states that the maximum +* buffer size is 64K when the monitored directory is remote. +* +* Larger buffers may be used when the monitored directory is local and +* will help us receive events faster from the kernel and avoid dropped +* events. +* +* So we try to use a very large buffer and silently fallback to 64K if +* we get an error. +*/ +#define MAX_RDCW_BUF_FALLBACK (65536) +#define MAX_RDCW_BUF (65536 * 8) + +struct one_watch +{ +char buffer[MAX_RDCW_BUF ]; +DWORD buf_len; +DWORD count; + +struct strbuf path; +HANDLE hDir; +HANDLE hEvent; +OVERLAPPED overlapped; + +/* +* Is there an active ReadDirectoryChangesW() call pending. If so, we +* need to later call GetOverlappedResult() and possibly CancelIoEx(). +*/ +BOOL is_active; +}; + +struct fsmonitor_daemon_backend_data +{ +struct one_watch *watch_worktree; +struct one_watch *watch_gitdir; + +HANDLE hEventShutdown; + +HANDLE hListener[3 ];/* we don't own these handles */ +#define LISTENER_SHUTDOWN 0 +#define LISTENER_HAVE_DATA_WORKTREE 1 +#define LISTENER_HAVE_DATA_GITDIR 2 +int nr_listener_handles; +}; + +/* +* Convert the WCHAR path from the notification into UTF8 and +* then normalize it. +*/ +static int normalize_path_in_utf8 (FILE_NOTIFY_INFORMATION *info, +struct strbuf *normalized_path) +{ +int reserve; +int len =0 ; + +strbuf_reset (normalized_path); +if (!info->FileNameLength) + goto normalize; + +/* +* Pre-reserve enough space in the UTF8 buffer for +* each Unicode WCHAR character to be mapped into a +* sequence of 2 UTF8 characters. That should let us +* avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time. +*/ + reserve = info->FileNameLength +1 ; +strbuf_grow (normalized_path, reserve); + +for (;;) { + len =WideCharToMultiByte (CP_UTF8 ,0 , info->FileName, + info->FileNameLength /sizeof (WCHAR ), + normalized_path->buf, +strbuf_avail (normalized_path) -1 , +NULL ,NULL ); +if (len >0 ) + goto normalize; +if (GetLastError () !=ERROR_INSUFFICIENT_BUFFER ) { +error ("[GLE %ld] could not convert path to UTF-8: '%.*ls'" , +GetLastError (), + (int )(info->FileNameLength /sizeof (WCHAR )), + info->FileName); +return -1 ; + } + +strbuf_grow (normalized_path, +strbuf_avail (normalized_path) + reserve); + } + +normalize: +strbuf_setlen (normalized_path, len); +return strbuf_normalize_path (normalized_path); +}void fsmonitor_fs_listen__stop_async (struct fsmonitor_daemon_state *state) { +SetEvent (state->backend_data->hListener[LISTENER_SHUTDOWN ]); +} + +static struct one_watch *create_watch (struct fsmonitor_daemon_state *state, +const char *path) +{ +struct one_watch *watch =NULL ; +DWORD desired_access =FILE_LIST_DIRECTORY ; +DWORD share_mode = +FILE_SHARE_WRITE |FILE_SHARE_READ |FILE_SHARE_DELETE ; +HANDLE hDir; + + hDir =CreateFileA (path, + desired_access, share_mode,NULL ,OPEN_EXISTING , +FILE_FLAG_BACKUP_SEMANTICS |FILE_FLAG_OVERLAPPED , +NULL ); +if (hDir ==INVALID_HANDLE_VALUE ) { +error (_ ("[GLE %ld] could not watch '%s'" ), +GetLastError (), path); +return NULL ; + } + +CALLOC_ARRAY (watch,1 ); + + watch->buf_len =sizeof (watch->buffer);/* assume full MAX_RDCW_BUF */ + +strbuf_init (&watch->path,0 ); +strbuf_addstr (&watch->path, path); + + watch->hDir = hDir; + watch->hEvent =CreateEvent (NULL , TRUE, FALSE,NULL ); + +return watch; +} + +static void destroy_watch (struct one_watch *watch) +{ +if (!watch) +return ; + +strbuf_release (&watch->path); +if (watch->hDir !=INVALID_HANDLE_VALUE ) +CloseHandle (watch->hDir); +if (watch->hEvent !=INVALID_HANDLE_VALUE ) +CloseHandle (watch->hEvent); + +free (watch); +} + +static int start_rdcw_watch (struct fsmonitor_daemon_backend_data *data, +struct one_watch *watch) +{ +DWORD dwNotifyFilter = +FILE_NOTIFY_CHANGE_FILE_NAME | +FILE_NOTIFY_CHANGE_DIR_NAME | +FILE_NOTIFY_CHANGE_ATTRIBUTES | +FILE_NOTIFY_CHANGE_SIZE | +FILE_NOTIFY_CHANGE_LAST_WRITE | +FILE_NOTIFY_CHANGE_CREATION ; + +ResetEvent (watch->hEvent); + +memset (&watch->overlapped,0 ,sizeof (watch->overlapped)); + watch->overlapped.hEvent = watch->hEvent; + +start_watch: +/* +* Queue an async call using Overlapped IO. This returns immediately. +* Our event handle will be signalled when the real result is available. +* +* The return value here just means that we successfully queued it. +* We won't know if the Read...() actually produces data until later. +*/ + watch->is_active =ReadDirectoryChangesW ( + watch->hDir, watch->buffer, watch->buf_len, TRUE, + dwNotifyFilter, &watch->count, &watch->overlapped,NULL ); + +/* +* The kernel throws an invalid parameter error when our buffer +* is too big and we are pointed at a remote directory (and possibly +* for other reasons). Quietly set it down and try again. +* +* See note about MAX_RDCW_BUF at the top. +*/ +if (!watch->is_active && +GetLastError () ==ERROR_INVALID_PARAMETER && + watch->buf_len >MAX_RDCW_BUF_FALLBACK ) { + watch->buf_len =MAX_RDCW_BUF_FALLBACK ; + goto start_watch; + } + +if (watch->is_active) +return 0 ; + +error ("ReadDirectoryChangedW failed on '%s' [GLE %ld]" , + watch->path.buf,GetLastError ()); +return -1 ; +} + +static int recv_rdcw_watch (struct one_watch *watch) +{ + watch->is_active = FALSE; + +/* +* The overlapped result is ready. If the Read...() was successful +* we finally receive the actual result into our buffer. +*/ +if (GetOverlappedResult (watch->hDir, &watch->overlapped, &watch->count, + TRUE)) +return 0 ; + +/* +* NEEDSWORK: If an external <gitdir> is deleted, the above +* returns an error. I'm not sure that there's anything that +* we can do here other than failing -- the <worktree>/.git +* link file would be broken anyway. We might try to check +* for that and return a better error message, but I'm not +* sure it is worth it. +*/ + +error ("GetOverlappedResult failed on '%s' [GLE %ld]" , + watch->path.buf,GetLastError ()); +return -1 ; +} + +static void cancel_rdcw_watch (struct one_watch *watch) +{ +DWORD count; + +if (!watch || !watch->is_active) +return ; + +/* +* The calls to ReadDirectoryChangesW() and GetOverlappedResult() +* form a "pair" (my term) where we queue an IO and promise to +* hang around and wait for the kernel to give us the result. +* +* If for some reason after we queue the IO, we have to quit +* or otherwise not stick around for the second half, we must +* tell the kernel to abort the IO. This prevents the kernel +* from writing to our buffer and/or signalling our event +* after we free them. +* +* (Ask me how much fun it was to track that one down). +*/ +CancelIoEx (watch->hDir, &watch->overlapped); +GetOverlappedResult (watch->hDir, &watch->overlapped, &count, TRUE); + watch->is_active = FALSE; +} + +/* +* Process filesystem events that happen anywhere (recursively) under the +* <worktree> root directory. For a normal working directory, this includes +* both version controlled files and the contents of the .git/ directory. +* +* If <worktree>/.git is a file, then we only see events for the file +* itself. +*/ +static int process_worktree_events (struct fsmonitor_daemon_state *state) +{ +struct fsmonitor_daemon_backend_data *data = state->backend_data; +struct one_watch *watch = data->watch_worktree; +struct strbuf path =STRBUF_INIT ; +struct string_list cookie_list =STRING_LIST_INIT_DUP ; +struct fsmonitor_batch *batch =NULL ; +const char *p = watch->buffer; + +/* +* If the kernel gets more events than will fit in the kernel +* buffer associated with our RDCW handle, it drops them and +* returns a count of zero. +* +* Yes, the call returns WITHOUT error and with length zero. +* +* (The "overflow" case is not ambiguous with the "no data" case +* because we did an INFINITE wait.) +* +* This means we have a gap in coverage. Tell the daemon layer +* to resync. +*/ +if (!watch->count) { +trace2_data_string ("fsmonitor" ,NULL ,"fsm-listen/kernel" , +"overflow" ); +fsmonitor_force_resync (state); +return LISTENER_HAVE_DATA_WORKTREE ; + } + +/* +* On Windows, `info` contains an "array" of paths that are +* relative to the root of whichever directory handle received +* the event. +*/ +for (;;) { +FILE_NOTIFY_INFORMATION *info = (void *)p; +const char *slash; +enum fsmonitor_path_type t; + +strbuf_reset (&path); +if (normalize_path_in_utf8 (info, &path) ==-1 ) + goto skip_this_path; + + t =fsmonitor_classify_path_workdir_relative (path.buf); + +switch (t) { +case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX : +/* special case cookie files within .git */ + +/* Use just the filename of the cookie file. */ + slash =find_last_dir_sep (path.buf); +string_list_append (&cookie_list, + slash ? slash +1 : path.buf); +break ; + +case IS_INSIDE_DOT_GIT : +/* ignore everything inside of "<worktree>/.git/" */ +break ; + +case IS_DOT_GIT : +/* "<worktree>/.git" was deleted (or renamed away) */ +if ((info->Action ==FILE_ACTION_REMOVED ) || + (info->Action ==FILE_ACTION_RENAMED_OLD_NAME )) { +trace2_data_string ("fsmonitor" ,NULL , +"fsm-listen/dotgit" , +"removed" ); + goto force_shutdown; + } +break ; + +case IS_WORKDIR_PATH : +/* queue normal pathname */ +if (!batch) + batch =fsmonitor_batch__new (); +fsmonitor_batch__add_path (batch, path.buf); +break ; + +case IS_GITDIR : +case IS_INSIDE_GITDIR : +case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX : +default : +BUG ("unexpected path classification '%d' for '%s'" , + t, path.buf); + } + +skip_this_path: +if (!info->NextEntryOffset) +break ; + p += info->NextEntryOffset; + } + +fsmonitor_publish (state, batch, &cookie_list); + batch =NULL ; +string_list_clear (&cookie_list,0 ); +strbuf_release (&path); +return LISTENER_HAVE_DATA_WORKTREE ; + +force_shutdown: +fsmonitor_batch__pop (batch); +string_list_clear (&cookie_list,0 ); +strbuf_release (&path); +return LISTENER_SHUTDOWN ; +} + +/* +* Process filesystem events that happened anywhere (recursively) under the +* external <gitdir> (such as non-primary worktrees or submodules). +* We only care about cookie files that our client threads created here. +* +* Note that we DO NOT get filesystem events on the external <gitdir> +* itself (it is not inside something that we are watching). In particular, +* we do not get an event if the external <gitdir> is deleted. +*/ +static int process_gitdir_events (struct fsmonitor_daemon_state *state) +{ +struct fsmonitor_daemon_backend_data *data = state->backend_data; +struct one_watch *watch = data->watch_gitdir; +struct strbuf path =STRBUF_INIT ; +struct string_list cookie_list =STRING_LIST_INIT_DUP ; +const char *p = watch->buffer; + +if (!watch->count) { +trace2_data_string ("fsmonitor" ,NULL ,"fsm-listen/kernel" , +"overflow" ); +fsmonitor_force_resync (state); +return LISTENER_HAVE_DATA_GITDIR ; + } + +for (;;) { +FILE_NOTIFY_INFORMATION *info = (void *)p; +const char *slash; +enum fsmonitor_path_type t; + +strbuf_reset (&path); +if (normalize_path_in_utf8 (info, &path) ==-1 ) + goto skip_this_path; + + t =fsmonitor_classify_path_gitdir_relative (path.buf); + +trace_printf_key (&trace_fsmonitor,"BBB: %s" , path.buf); + +switch (t) { +case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX : +/* special case cookie files within gitdir */ + +/* Use just the filename of the cookie file. */ + slash =find_last_dir_sep (path.buf); +string_list_append (&cookie_list, + slash ? slash +1 : path.buf); +break ; + +case IS_INSIDE_GITDIR : + goto skip_this_path; + +default : +BUG ("unexpected path classification '%d' for '%s'" , + t, path.buf); + } + +skip_this_path: +if (!info->NextEntryOffset) +break ; + p += info->NextEntryOffset; + } + +fsmonitor_publish (state,NULL , &cookie_list); +string_list_clear (&cookie_list,0 ); +strbuf_release (&path); +return LISTENER_HAVE_DATA_GITDIR ; }void fsmonitor_fs_listen__loop (struct fsmonitor_daemon_state *state) { +struct fsmonitor_daemon_backend_data *data = state->backend_data; +DWORD dwWait; + + state->error_code =0 ; + +if (start_rdcw_watch (data, data->watch_worktree) ==-1 ) + goto force_error_stop; + +if (data->watch_gitdir && +start_rdcw_watch (data, data->watch_gitdir) ==-1 ) + goto force_error_stop; + +for (;;) { + dwWait =WaitForMultipleObjects (data->nr_listener_handles, + data->hListener, + FALSE,INFINITE ); + +if (dwWait ==WAIT_OBJECT_0 +LISTENER_HAVE_DATA_WORKTREE ) { +if (recv_rdcw_watch (data->watch_worktree) ==-1 ) + goto force_error_stop; +if (process_worktree_events (state) ==LISTENER_SHUTDOWN ) + goto force_shutdown; +if (start_rdcw_watch (data, data->watch_worktree) ==-1 ) + goto force_error_stop; +continue ; + } + +if (dwWait ==WAIT_OBJECT_0 +LISTENER_HAVE_DATA_GITDIR ) { +if (recv_rdcw_watch (data->watch_gitdir) ==-1 ) + goto force_error_stop; +if (process_gitdir_events (state) ==LISTENER_SHUTDOWN ) + goto force_shutdown; +if (start_rdcw_watch (data, data->watch_gitdir) ==-1 ) + goto force_error_stop; +continue ; + } + +if (dwWait ==WAIT_OBJECT_0 +LISTENER_SHUTDOWN ) + goto clean_shutdown; + +error (_ ("could not read directory changes [GLE %ld]" ), +GetLastError ()); + goto force_error_stop; + } + +force_error_stop: + state->error_code =-1 ; + +force_shutdown: +/* +* Tell the IPC thead pool to stop (which completes the await +* in the main thread (which will also signal this thread (if +* we are still alive))). +*/ +ipc_server_stop_async (state->ipc_server_data); + +clean_shutdown: +cancel_rdcw_watch (data->watch_worktree); +cancel_rdcw_watch (data->watch_gitdir); }int fsmonitor_fs_listen__ctor (struct fsmonitor_daemon_state *state) { +struct fsmonitor_daemon_backend_data *data; + +CALLOC_ARRAY (data,1 ); + + data->hEventShutdown =CreateEvent (NULL , TRUE, FALSE,NULL ); + + data->watch_worktree =create_watch (state, + state->path_worktree_watch.buf); +if (!data->watch_worktree) + goto failed; + +if (state->nr_paths_watching >1 ) { + data->watch_gitdir =create_watch (state, + state->path_gitdir_watch.buf); +if (!data->watch_gitdir) + goto failed; + } + + data->hListener[LISTENER_SHUTDOWN ] = data->hEventShutdown; + data->nr_listener_handles++; + + data->hListener[LISTENER_HAVE_DATA_WORKTREE ] = + data->watch_worktree->hEvent; + data->nr_listener_handles++; + +if (data->watch_gitdir) { + data->hListener[LISTENER_HAVE_DATA_GITDIR ] = + data->watch_gitdir->hEvent; + data->nr_listener_handles++; + } + + state->backend_data = data; +return 0 ; + +failed: +CloseHandle (data->hEventShutdown); +destroy_watch (data->watch_worktree); +destroy_watch (data->watch_gitdir); +return -1 ; }void fsmonitor_fs_listen__dtor (struct fsmonitor_daemon_state *state) { +struct fsmonitor_daemon_backend_data *data; + +if (!state || !state->backend_data) +return ; + + data = state->backend_data; + +CloseHandle (data->hEventShutdown); +destroy_watch (data->watch_worktree); +destroy_watch (data->watch_gitdir); + +FREE_AND_NULL (state->backend_data); }
Signature · 1 line
gitgitgadget
PATCH v2 18/28 fsmonitor-fs-listen-macos: add macos header files for FSEvent
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Include MacOS system declarations to allow us to use FSEvent and CoreFoundation APIs. We need GCC and clang versions because of compiler and header file conflicts.
While it is quite possible to #include Apple's CoreServices.h when compiling C source code with clang, trying to build it with GCC currently fails with this error:
In file included
from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework/Headers/AuthSession.h:32,
from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework/Headers/Security.h:42,
from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/OSServices.framework/Headers/CSIdentity.h:43, from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/OSServices.framework/Headers/OSServices.h:29, from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Headers/IconsCore.h:23, from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Headers/LaunchServices.h:23, from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:45,
/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework/Headers/Authorization.h:193:7: error: variably modified 'bytes' at file scope
193 | char bytes[kAuthorizationExternalFormLength];
| ^~~~~The underlying reason is that GCC (rightfully) objects that an enum value such as kAuthorizationExternalFormLength is not a constant (because it is not, the preprocessor has no knowledge of it, only the actual C compiler does) and can therefore not be used to define the size of a C array.
This is a known problem and tracked in GCC's bug tracker:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93082
In the meantime, let's not block things and go the slightly ugly route of declaring/defining the FSEvents constants, data structures and functions that we need, so that we can avoid above-mentioned issue.
Let's do this _only_ for GCC, though, so that the CI/PR builds (which build both with clang and with GCC) can guarantee that we _are_ using the correct data types.
--- compat/fsmonitor/fsmonitor-fs-listen-macos.c | 96 ++++++++++++++++++++ 1 file changed, 96 insertions(+)
compat/fsmonitor/fsmonitor-fs-listen-macos.c +96 -0
@@ -1,3 +1,99 @@ +#if defined(__GNUC__) +/* +* It is possible to #include CoreFoundation/CoreFoundation.h when compiling +* with clang, but not with GCC as of time of writing. +* +* See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93082 for details. +*/ +typedef unsigned int FSEventStreamCreateFlags ; +#define kFSEventStreamEventFlagNone 0x00000000 +#define kFSEventStreamEventFlagMustScanSubDirs 0x00000001 +#define kFSEventStreamEventFlagUserDropped 0x00000002 +#define kFSEventStreamEventFlagKernelDropped 0x00000004 +#define kFSEventStreamEventFlagEventIdsWrapped 0x00000008 +#define kFSEventStreamEventFlagHistoryDone 0x00000010 +#define kFSEventStreamEventFlagRootChanged 0x00000020 +#define kFSEventStreamEventFlagMount 0x00000040 +#define kFSEventStreamEventFlagUnmount 0x00000080 +#define kFSEventStreamEventFlagItemCreated 0x00000100 +#define kFSEventStreamEventFlagItemRemoved 0x00000200 +#define kFSEventStreamEventFlagItemInodeMetaMod 0x00000400 +#define kFSEventStreamEventFlagItemRenamed 0x00000800 +#define kFSEventStreamEventFlagItemModified 0x00001000 +#define kFSEventStreamEventFlagItemFinderInfoMod 0x00002000 +#define kFSEventStreamEventFlagItemChangeOwner 0x00004000 +#define kFSEventStreamEventFlagItemXattrMod 0x00008000 +#define kFSEventStreamEventFlagItemIsFile 0x00010000 +#define kFSEventStreamEventFlagItemIsDir 0x00020000 +#define kFSEventStreamEventFlagItemIsSymlink 0x00040000 +#define kFSEventStreamEventFlagOwnEvent 0x00080000 +#define kFSEventStreamEventFlagItemIsHardlink 0x00100000 +#define kFSEventStreamEventFlagItemIsLastHardlink 0x00200000 +#define kFSEventStreamEventFlagItemCloned 0x00400000 + +typedef struct __FSEventStream *FSEventStreamRef ; +typedef const FSEventStreamRef ConstFSEventStreamRef ; + +typedef unsigned int CFStringEncoding ; +#define kCFStringEncodingUTF8 0x08000100 + +typedef const struct __CFString *CFStringRef ; +typedef const struct __CFArray *CFArrayRef ; +typedef const struct __CFRunLoop *CFRunLoopRef ; + +struct FSEventStreamContext { +long long version; +void *cb_data, *retain, *release, *copy_description; +}; + +typedef struct FSEventStreamContext FSEventStreamContext ; +typedef unsigned int FSEventStreamEventFlags ; +#define kFSEventStreamCreateFlagNoDefer 0x02 +#define kFSEventStreamCreateFlagWatchRoot 0x04 +#define kFSEventStreamCreateFlagFileEvents 0x10 + +typedef unsigned long long FSEventStreamEventId ; +#define kFSEventStreamEventIdSinceNow 0xFFFFFFFFFFFFFFFFULL + +typedef void (*FSEventStreamCallback )(ConstFSEventStreamRef streamRef, +void *context, +__SIZE_TYPE__ num_of_events, +void *event_paths, +const FSEventStreamEventFlags event_flags[], +const FSEventStreamEventId event_ids[]); +typedef double CFTimeInterval ; +FSEventStreamRef FSEventStreamCreate (void *allocator, +FSEventStreamCallback callback, +FSEventStreamContext *context, +CFArrayRef paths_to_watch, +FSEventStreamEventId since_when, +CFTimeInterval latency, +FSEventStreamCreateFlags flags); +CFStringRef CFStringCreateWithCString (void *allocator,const char *string, +CFStringEncoding encoding); +CFArrayRef CFArrayCreate (void *allocator,const void **items,long long count, +void *callbacks); +void CFRunLoopRun (void ); +void CFRunLoopStop (CFRunLoopRef run_loop); +CFRunLoopRef CFRunLoopGetCurrent (void ); +extern CFStringRef kCFRunLoopDefaultMode; +void FSEventStreamScheduleWithRunLoop (FSEventStreamRef stream, +CFRunLoopRef run_loop, +CFStringRef run_loop_mode); +unsigned char FSEventStreamStart (FSEventStreamRef stream); +void FSEventStreamStop (FSEventStreamRef stream); +void FSEventStreamInvalidate (FSEventStreamRef stream); +void FSEventStreamRelease (FSEventStreamRef stream); +#else +/* +* Let Apple's headers declare `isalnum()` first, before +* Git's headers override it via a constant +*/ +#include <string.h> +#include <CoreFoundation/CoreFoundation.h> +#include <CoreServices/CoreServices.h> +#endif +#include "cache.h" #include "fsmonitor.h" #include "fsmonitor-fs-listen.h"
Signature · 1 line
gitgitgadget
PATCH v2 19/28 fsmonitor-fs-listen-macos: implement FSEvent listener on MacOS
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Implement file system event listener on MacOS using FSEvent, CoreFoundation, and CoreServices.
--- compat/fsmonitor/fsmonitor-fs-listen-macos.c | 381 +++++++++++++++++++ 1 file changed, 381 insertions(+)
compat/fsmonitor/fsmonitor-fs-listen-macos.c +381 -0
@@ -97,20 +97,401 @@ void FSEventStreamRelease(FSEventStreamRef stream);#include "cache.h" #include "fsmonitor.h" #include "fsmonitor-fs-listen.h" +#include "fsmonitor--daemon.h" + +struct fsmonitor_daemon_backend_data +{ +CFStringRef cfsr_worktree_path; +CFStringRef cfsr_gitdir_path; + +CFArrayRef cfar_paths_to_watch; +int nr_paths_watching; + +FSEventStreamRef stream; + +CFRunLoopRef rl; + +enum shutdown_style { +SHUTDOWN_EVENT =0 , +FORCE_SHUTDOWN , +FORCE_ERROR_STOP , + } shutdown_style; + +unsigned int stream_scheduled:1 ; +unsigned int stream_started:1 ; +}; + +static void log_flags_set (const char *path,const FSEventStreamEventFlags flag) +{ +struct strbuf msg =STRBUF_INIT ; + +if (flag & kFSEventStreamEventFlagMustScanSubDirs) +strbuf_addstr (&msg,"MustScanSubDirs|" ); +if (flag & kFSEventStreamEventFlagUserDropped) +strbuf_addstr (&msg,"UserDropped|" ); +if (flag & kFSEventStreamEventFlagKernelDropped) +strbuf_addstr (&msg,"KernelDropped|" ); +if (flag & kFSEventStreamEventFlagEventIdsWrapped) +strbuf_addstr (&msg,"EventIdsWrapped|" ); +if (flag & kFSEventStreamEventFlagHistoryDone) +strbuf_addstr (&msg,"HistoryDone|" ); +if (flag & kFSEventStreamEventFlagRootChanged) +strbuf_addstr (&msg,"RootChanged|" ); +if (flag & kFSEventStreamEventFlagMount) +strbuf_addstr (&msg,"Mount|" ); +if (flag & kFSEventStreamEventFlagUnmount) +strbuf_addstr (&msg,"Unmount|" ); +if (flag & kFSEventStreamEventFlagItemChangeOwner) +strbuf_addstr (&msg,"ItemChangeOwner|" ); +if (flag & kFSEventStreamEventFlagItemCreated) +strbuf_addstr (&msg,"ItemCreated|" ); +if (flag & kFSEventStreamEventFlagItemFinderInfoMod) +strbuf_addstr (&msg,"ItemFinderInfoMod|" ); +if (flag & kFSEventStreamEventFlagItemInodeMetaMod) +strbuf_addstr (&msg,"ItemInodeMetaMod|" ); +if (flag & kFSEventStreamEventFlagItemIsDir) +strbuf_addstr (&msg,"ItemIsDir|" ); +if (flag & kFSEventStreamEventFlagItemIsFile) +strbuf_addstr (&msg,"ItemIsFile|" ); +if (flag & kFSEventStreamEventFlagItemIsHardlink) +strbuf_addstr (&msg,"ItemIsHardlink|" ); +if (flag & kFSEventStreamEventFlagItemIsLastHardlink) +strbuf_addstr (&msg,"ItemIsLastHardlink|" ); +if (flag & kFSEventStreamEventFlagItemIsSymlink) +strbuf_addstr (&msg,"ItemIsSymlink|" ); +if (flag & kFSEventStreamEventFlagItemModified) +strbuf_addstr (&msg,"ItemModified|" ); +if (flag & kFSEventStreamEventFlagItemRemoved) +strbuf_addstr (&msg,"ItemRemoved|" ); +if (flag & kFSEventStreamEventFlagItemRenamed) +strbuf_addstr (&msg,"ItemRenamed|" ); +if (flag & kFSEventStreamEventFlagItemXattrMod) +strbuf_addstr (&msg,"ItemXattrMod|" ); +if (flag & kFSEventStreamEventFlagOwnEvent) +strbuf_addstr (&msg,"OwnEvent|" ); +if (flag & kFSEventStreamEventFlagItemCloned) +strbuf_addstr (&msg,"ItemCloned|" ); + +trace_printf_key (&trace_fsmonitor,"fsevent: '%s', flags=%u %s" , + path, flag, msg.buf); + +strbuf_release (&msg); +} + +static int ef_is_root_delete (const FSEventStreamEventFlags ef) +{ +return (ef & kFSEventStreamEventFlagItemIsDir && + ef & kFSEventStreamEventFlagItemRemoved); +} + +static int ef_is_root_renamed (const FSEventStreamEventFlags ef) +{ +return (ef & kFSEventStreamEventFlagItemIsDir && + ef & kFSEventStreamEventFlagItemRenamed); +} + +static int ef_is_dropped (const FSEventStreamEventFlags ef) +{ +return (ef & kFSEventStreamEventFlagKernelDropped || + ef & kFSEventStreamEventFlagUserDropped); +} + +static void fsevent_callback (ConstFSEventStreamRef streamRef, +void *ctx, +size_t num_of_events, +void *event_paths, +const FSEventStreamEventFlags event_flags[], +const FSEventStreamEventId event_ids[]) +{ +struct fsmonitor_daemon_state *state = ctx; +struct fsmonitor_daemon_backend_data *data = state->backend_data; +char **paths = (char **)event_paths; +struct fsmonitor_batch *batch =NULL ; +struct string_list cookie_list =STRING_LIST_INIT_DUP ; +const char *path_k; +const char *slash; +int k; +struct strbuf tmp =STRBUF_INIT ; + +/* +* Build a list of all filesystem changes into a private/local +* list and without holding any locks. +*/ +for (k =0 ; k < num_of_events; k++) { +/* +* On Mac, we receive an array of absolute paths. +*/ + path_k = paths[k]; + +/* +* If you want to debug FSEvents, log them to GIT_TRACE_FSMONITOR. +* Please don't log them to Trace2. +* +* trace_printf_key(&trace_fsmonitor, "Path: '%s'", path_k); +*/ + +/* +* If event[k] is marked as dropped, we assume that we have +* lost sync with the filesystem and should flush our cached +* data. We need to: +* +* [1] Abort/wake any client threads waiting for a cookie and +* flush the cached state data (the current token), and +* create a new token. +* +* [2] Discard the batch that we were locally building (since +* they are conceptually relative to the just flushed +* token). +*/ +if (ef_is_dropped (event_flags[k])) { +/* +* see also kFSEventStreamEventFlagMustScanSubDirs +*/ +trace_printf_key (&trace_fsmonitor,"event: dropped" ); + +fsmonitor_force_resync (state); +fsmonitor_batch__pop (batch); +string_list_clear (&cookie_list,0 ); + +/* +* We assume that any events that we received +* in this callback after this dropped event +* may still be valid, so we continue rather +* than break. (And just in case there is a +* delete of ".git" hiding in there.) +*/ +continue ; + } + +switch (fsmonitor_classify_path_absolute (state, path_k)) { + +case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX : +case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX : +/* special case cookie files within .git or gitdir */ + +/* Use just the filename of the cookie file. */ + slash =find_last_dir_sep (path_k); +string_list_append (&cookie_list, + slash ? slash +1 : path_k); +break ; + +case IS_INSIDE_DOT_GIT : +case IS_INSIDE_GITDIR : +/* ignore all other paths inside of .git or gitdir */ +break ; + +case IS_DOT_GIT : +case IS_GITDIR : +/* +* If .git directory is deleted or renamed away, +* we have to quit. +*/ +if (ef_is_root_delete (event_flags[k])) { +trace_printf_key (&trace_fsmonitor, +"event: gitdir removed" ); + goto force_shutdown; + } +if (ef_is_root_renamed (event_flags[k])) { +trace_printf_key (&trace_fsmonitor, +"event: gitdir renamed" ); + goto force_shutdown; + } +break ; + +case IS_WORKDIR_PATH : +/* try to queue normal pathnames */ + +if (trace_pass_fl (&trace_fsmonitor)) +log_flags_set (path_k, event_flags[k]); + +/* +* Because of the implicit "binning" (the +* kernel calls us at a given frequency) and +* de-duping (the kernel is free to combine +* multiple events for a given pathname), an +* individual fsevent could be marked as both +* a file and directory. Add it to the queue +* with both spellings so that the client will +* know how much to invalidate/refresh. +*/ + +if (event_flags[k] & kFSEventStreamEventFlagItemIsFile) { +const char *rel = path_k + + state->path_worktree_watch.len +1 ; + +if (!batch) + batch =fsmonitor_batch__new (); +fsmonitor_batch__add_path (batch, rel); + } + +if (event_flags[k] & kFSEventStreamEventFlagItemIsDir) { +const char *rel = path_k + + state->path_worktree_watch.len +1 ; + +strbuf_reset (&tmp); +strbuf_addstr (&tmp, rel); +strbuf_addch (&tmp,'/' ); + +if (!batch) + batch =fsmonitor_batch__new (); +fsmonitor_batch__add_path (batch, tmp.buf); + } + +break ; + +case IS_OUTSIDE_CONE : +default : +trace_printf_key (&trace_fsmonitor, +"ignoring '%s'" , path_k); +break ; + } + } + +fsmonitor_publish (state, batch, &cookie_list); +string_list_clear (&cookie_list,0 ); +strbuf_release (&tmp); +return ; + +force_shutdown: +fsmonitor_batch__pop (batch); +string_list_clear (&cookie_list,0 ); + + data->shutdown_style =FORCE_SHUTDOWN ; +CFRunLoopStop (data->rl); +strbuf_release (&tmp); +return ; +} + +/* +* NEEDSWORK: Investigate the proper value for the `latency` argument +* in the call to `FSEventStreamCreate()`. I'm not sure that this +* needs to be a config setting or just something that we tune after +* some testing. +* +* With a latency of 0.1, I was seeing lots of dropped events during +* the "touch 100000" files test within t/perf/p7519, but with a +* latency of 0.001 I did not see any dropped events. So the +* "correct" value may be somewhere in between. +* +* https://developer.apple.com/documentation/coreservices/1443980-fseventstreamcreate +*/ int fsmonitor_fs_listen__ctor (struct fsmonitor_daemon_state *state) { +FSEventStreamCreateFlags flags = kFSEventStreamCreateFlagNoDefer | + kFSEventStreamCreateFlagWatchRoot | + kFSEventStreamCreateFlagFileEvents; +FSEventStreamContext ctx = { +0 , + state, +NULL , +NULL , +NULL + }; +struct fsmonitor_daemon_backend_data *data; +const void *dir_array[2 ]; + +CALLOC_ARRAY (data,1 ); + state->backend_data = data; + + data->cfsr_worktree_path =CFStringCreateWithCString ( +NULL , state->path_worktree_watch.buf, kCFStringEncodingUTF8); + dir_array[data->nr_paths_watching++] = data->cfsr_worktree_path; + +if (state->nr_paths_watching >1 ) { + data->cfsr_gitdir_path =CFStringCreateWithCString ( +NULL , state->path_gitdir_watch.buf, + kCFStringEncodingUTF8); + dir_array[data->nr_paths_watching++] = data->cfsr_gitdir_path; + } + + data->cfar_paths_to_watch =CFArrayCreate (NULL , dir_array, + data->nr_paths_watching, +NULL ); + data->stream =FSEventStreamCreate (NULL , fsevent_callback, &ctx, + data->cfar_paths_to_watch, + kFSEventStreamEventIdSinceNow, +0.001 , flags); +if (data->stream ==NULL ) + goto failed; + +/* +* `data->rl` needs to be set inside the listener thread. +*/ + +return 0 ; + +failed: +error ("Unable to create FSEventStream." ); + +FREE_AND_NULL (state->backend_data);return -1 ; }void fsmonitor_fs_listen__dtor (struct fsmonitor_daemon_state *state) { +struct fsmonitor_daemon_backend_data *data; + +if (!state || !state->backend_data) +return ; + + data = state->backend_data; + +if (data->stream) { +if (data->stream_started) +FSEventStreamStop (data->stream); +if (data->stream_scheduled) +FSEventStreamInvalidate (data->stream); +FSEventStreamRelease (data->stream); + } + +FREE_AND_NULL (state->backend_data); }void fsmonitor_fs_listen__stop_async (struct fsmonitor_daemon_state *state) { +struct fsmonitor_daemon_backend_data *data; + + data = state->backend_data; + data->shutdown_style =SHUTDOWN_EVENT ; + +CFRunLoopStop (data->rl); }void fsmonitor_fs_listen__loop (struct fsmonitor_daemon_state *state) { +struct fsmonitor_daemon_backend_data *data; + + data = state->backend_data; + + data->rl =CFRunLoopGetCurrent (); + +FSEventStreamScheduleWithRunLoop (data->stream, data->rl, kCFRunLoopDefaultMode); + data->stream_scheduled =1 ; + +if (!FSEventStreamStart (data->stream)) { +error ("Failed to start the FSEventStream" ); + goto force_error_stop_without_loop; + } + data->stream_started =1 ; + +CFRunLoopRun (); + +switch (data->shutdown_style) { +case FORCE_ERROR_STOP : + state->error_code =-1 ; +/* fall thru */ +case FORCE_SHUTDOWN : +ipc_server_stop_async (state->ipc_server_data); +/* fall thru */ +case SHUTDOWN_EVENT : +default : +break ; + } +return ; + +force_error_stop_without_loop: + state->error_code =-1 ; +ipc_server_stop_async (state->ipc_server_data); +return ; }
Signature · 1 line
gitgitgadget
PATCH v2 20/28 fsmonitor--daemon: implement handle_client callback
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Teach fsmonitor--daemon to respond to IPC requests from client Git processes and respond with a list of modified pathnames relative to the provided token.
--- builtin/fsmonitor--daemon.c | 312 +++++++++++++++++++++++++++++++++++- 1 file changed, 310 insertions(+), 2 deletions(-)
builtin/fsmonitor--daemon.c +310 -2
@@ -7,6 +7,7 @@#include "fsmonitor--daemon.h" #include "simple-ipc.h" #include "khash.h" +#include "pkt-line.h" static const char *const builtin_fsmonitor__daemon_usage[] = {N_ ("git fsmonitor--daemon start [<options>]" ),
@@ -353,6 +354,311 @@ void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)pthread_mutex_unlock (&state->main_lock); } +/* +* Format an opaque token string to send to the client. +*/ +static void with_lock__format_response_token ( +struct strbuf *response_token, +const struct strbuf *response_token_id, +const struct fsmonitor_batch *batch) +{ +/* assert current thread holding state->main_lock */ + +strbuf_reset (response_token); +strbuf_addf (response_token,"builtin:%s:%" PRIu64, + response_token_id->buf, batch->batch_seq_nr); +} + +/* +* Parse an opaque token from the client. +* Returns -1 on error. +*/ +static int fsmonitor_parse_client_token (const char *buf_token, +struct strbuf *requested_token_id, +uint64_t *seq_nr) +{ +const char *p; +char *p_end; + +strbuf_reset (requested_token_id); + *seq_nr =0 ; + +if (!skip_prefix (buf_token,"builtin:" , &p)) +return -1 ; + +while (*p && *p !=':' ) +strbuf_addch (requested_token_id, *p++); +if (!*p++) +return -1 ; + + *seq_nr = (uint64_t )strtoumax (p, &p_end,10 ); +if (*p_end) +return -1 ; + +return 0 ; +} + +KHASH_INIT(str, const char *, int, 0, kh_str_hash_func, kh_str_hash_equal); + +static int do_handle_client (struct fsmonitor_daemon_state *state, +const char *command, +ipc_server_reply_cb *reply, +struct ipc_server_reply_data *reply_data) +{ +struct fsmonitor_token_data *token_data =NULL ; +struct strbuf response_token =STRBUF_INIT ; +struct strbuf requested_token_id =STRBUF_INIT ; +struct strbuf payload =STRBUF_INIT ; +uint64_t requested_oldest_seq_nr =0 ; +uint64_t total_response_len =0 ; +const char *p; +const struct fsmonitor_batch *batch_head; +const struct fsmonitor_batch *batch; +intmax_t count =0 , duplicates =0 ; +kh_str_t *shown; +int hash_ret; +int do_trivial =0 ; +int do_flush =0 ; + +/* +* We expect `command` to be of the form: +* +* <command> := quit NUL +* | flush NUL +* | <V1-time-since-epoch-ns> NUL +* | <V2-opaque-fsmonitor-token> NUL +*/ + +if (!strcmp (command,"quit" )) { +/* +* A client has requested over the socket/pipe that the +* daemon shutdown. +* +* Tell the IPC thread pool to shutdown (which completes +* the await in the main thread (which can stop the +* fsmonitor listener thread)). +* +* There is no reply to the client. +*/ +return SIMPLE_IPC_QUIT ; + + }else if (!strcmp (command,"flush" )) { +/* +* Flush all of our cached data and generate a new token +* just like if we lost sync with the filesystem. +* +* Then send a trivial response using the new token. +*/ + do_flush =1 ; + do_trivial =1 ; + + }else if (!skip_prefix (command,"builtin:" , &p)) { +/* assume V1 timestamp or garbage */ + +char *p_end; + +strtoumax (command, &p_end,10 ); +trace_printf_key (&trace_fsmonitor, + ((*p_end) ? +"fsmonitor: invalid command line '%s'" : +"fsmonitor: unsupported V1 protocol '%s'" ), + command); + do_trivial =1 ; + + }else { +/* We have "builtin:*" */ +if (fsmonitor_parse_client_token (command, &requested_token_id, + &requested_oldest_seq_nr)) { +trace_printf_key (&trace_fsmonitor, +"fsmonitor: invalid V2 protocol token '%s'" , + command); + do_trivial =1 ; + + }else { +/* +* We have a V2 valid token: +* "builtin:<token_id>:<seq_nr>" +*/ + } + } + +pthread_mutex_lock (&state->main_lock); + +if (!state->current_token_data) +BUG ("fsmonitor state does not have a current token" ); + +if (do_flush) +with_lock__do_force_resync (state); + +/* +* We mark the current head of the batch list as "pinned" so +* that the listener thread will treat this item as read-only +* (and prevent any more paths from being added to it) from +* now on. +*/ + token_data = state->current_token_data; + batch_head = token_data->batch_head; + ((struct fsmonitor_batch *)batch_head)->pinned_time =time (NULL ); + +/* +* FSMonitor Protocol V2 requires that we send a response header +* with a "new current token" and then all of the paths that changed +* since the "requested token". We send the seq_nr of the just-pinned +* head batch so that future requests from a client will be relative +* to it. +*/ +with_lock__format_response_token (&response_token, + &token_data->token_id, batch_head); + +reply (reply_data, response_token.buf, response_token.len +1 ); + total_response_len += response_token.len +1 ; + +trace2_data_string ("fsmonitor" , the_repository,"response/token" , + response_token.buf); +trace_printf_key (&trace_fsmonitor,"response token: %s" , + response_token.buf); + +if (!do_trivial) { +if (strcmp (requested_token_id.buf, token_data->token_id.buf)) { +/* +* The client last spoke to a different daemon +* instance -OR- the daemon had to resync with +* the filesystem (and lost events), so reject. +*/ +trace2_data_string ("fsmonitor" , the_repository, +"response/token" ,"different" ); + do_trivial =1 ; + + }else if (requested_oldest_seq_nr < + token_data->batch_tail->batch_seq_nr) { +/* +* The client wants older events than we have for +* this token_id. This means that the end of our +* batch list was truncated and we cannot give the +* client a complete snapshot relative to their +* request. +*/ +trace_printf_key (&trace_fsmonitor, +"client requested truncated data" ); + do_trivial =1 ; + } + } + +if (do_trivial) { +pthread_mutex_unlock (&state->main_lock); + +reply (reply_data,"/" ,2 ); + +trace2_data_intmax ("fsmonitor" , the_repository, +"response/trivial" ,1 ); + +strbuf_release (&response_token); +strbuf_release (&requested_token_id); +return 0 ; + } + +/* +* We're going to hold onto a pointer to the current +* token-data while we walk the list of batches of files. +* During this time, we will NOT be under the lock. +* So we ref-count it. +* +* This allows the listener thread to continue prepending +* new batches of items to the token-data (which we'll ignore). +* +* AND it allows the listener thread to do a token-reset +* (and install a new `current_token_data`). +*/ + token_data->client_ref_count++; + +pthread_mutex_unlock (&state->main_lock); + +/* +* The client request is relative to the token that they sent, +* so walk the batch list backwards from the current head back +* to the batch (sequence number) they named. +* +* We use khash to de-dup the list of pathnames. +* +* NEEDSWORK: each batch contains a list of interned strings, +* so we only need to do pointer comparisons here to build the +* hash table. Currently, we're still comparing the string +* values. +*/ + shown =kh_init_str (); +for (batch = batch_head; + batch && batch->batch_seq_nr > requested_oldest_seq_nr; + batch = batch->next) { +size_t k; + +for (k =0 ; k < batch->nr; k++) { +const char *s = batch->interned_paths[k]; +size_t s_len; + +if (kh_get_str (shown, s) !=kh_end (shown)) + duplicates++; +else { +kh_put_str (shown, s, &hash_ret); + +trace_printf_key (&trace_fsmonitor, +"send[%" PRIuMAX"]: %s" , + count, s); + +/* Each path gets written with a trailing NUL */ + s_len =strlen (s) +1 ; + +if (payload.len + s_len >= +LARGE_PACKET_DATA_MAX ) { +reply (reply_data, payload.buf, + payload.len); + total_response_len += payload.len; +strbuf_reset (&payload); + } + +strbuf_add (&payload, s, s_len); + count++; + } + } + } + +if (payload.len) { +reply (reply_data, payload.buf, payload.len); + total_response_len += payload.len; + } + +kh_release_str (shown); + +pthread_mutex_lock (&state->main_lock); + +if (token_data->client_ref_count >0 ) + token_data->client_ref_count--; + +if (token_data->client_ref_count ==0 ) { +if (token_data != state->current_token_data) { +/* +* The listener thread did a token-reset while we were +* walking the batch list. Therefore, this token is +* stale and can be discarded completely. If we are +* the last reader thread using this token, we own +* that work. +*/ +fsmonitor_free_token_data (token_data); + } + } + +pthread_mutex_unlock (&state->main_lock); + +trace2_data_intmax ("fsmonitor" , the_repository,"response/length" , total_response_len); +trace2_data_intmax ("fsmonitor" , the_repository,"response/count/files" , count); +trace2_data_intmax ("fsmonitor" , the_repository,"response/count/duplicates" , duplicates); + +strbuf_release (&response_token); +strbuf_release (&requested_token_id); +strbuf_release (&payload); + +return 0 ; +} +static ipc_server_application_cb handle_client; static int handle_client(void *data,
@@ -360,7 +666,7 @@ static int handle_client(void *data,ipc_server_reply_cb *reply, struct ipc_server_reply_data *reply_data) { - /* struct fsmonitor_daemon_state *state = data; */ + struct fsmonitor_daemon_state *state = data;int result;/*
@@ -371,10 +677,12 @@ static int handle_client(void *data,if (command_len !=strlen (command))BUG ("FSMonitor assumes text messages" ); +trace_printf_key (&trace_fsmonitor,"requested token: %s" , command); +trace2_region_enter ("fsmonitor" ,"handle_client" , the_repository);trace2_data_string ("fsmonitor" , the_repository,"request" , command); - result =0 ;/* TODO Do something here. */ + result =do_handle_client (state, command, reply, reply_data);trace2_region_leave ("fsmonitor" ,"handle_client" , the_repository);
Signature · 1 line
gitgitgadget
PATCH v2 21/28 fsmonitor--daemon: periodically truncate list of modified files
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Teach fsmonitor--daemon to periodically truncate the list of modified files to save some memory.
Clients will ask for the set of changes relative to a token that they found in the FSMN index extension in the index. (This token is like a point in time, but different). Clients will then update the index to contain the response token (so that subsequent commands will be relative to this new token).
Therefore, the daemon can gradually truncate the in-memory list of changed paths as they become obsolete (older than the previous token). Since we may have multiple clients making concurrent requests with a skew of tokens and clients may be racing to the talk to the daemon, we lazily truncate the list.
We introduce a 5 minute delay and truncate batches 5 minutes after they are considered obsolete.
--- builtin/fsmonitor--daemon.c | 80 +++++++++++++++++++++++++++++++++++++ 1 file changed, 80 insertions(+)
builtin/fsmonitor--daemon.c +80 -0
@@ -298,6 +298,77 @@ static void fsmonitor_batch__combine(struct fsmonitor_batch *batch_dest, batch_src->interned_paths[k]; } +/* +* To keep the batch list from growing unbounded in response to filesystem +* activity, we try to truncate old batches from the end of the list as +* they become irrelevant. +* +* We assume that the .git/index will be updated with the most recent token +* any time the index is updated. And future commands will only ask for +* recent changes *since* that new token. So as tokens advance into the +* future, older batch items will never be requested/needed. So we can +* truncate them without loss of functionality. +* +* However, multiple commands may be talking to the daemon concurrently +* or perform a slow command, so a little "token skew" is possible. +* Therefore, we want this to be a little bit lazy and have a generous +* delay. +* +* The current reader thread walked backwards in time from `token->batch_head` +* back to `batch_marker` somewhere in the middle of the batch list. +* +* Let's walk backwards in time from that marker an arbitrary delay +* and truncate the list there. Note that these timestamps are completely +* artificial (based on when we pinned the batch item) and not on any +* filesystem activity. +*/ +#define MY_TIME_DELAY_SECONDS (5 * 60)/* seconds */ + +static void with_lock__truncate_old_batches ( +struct fsmonitor_daemon_state *state, +const struct fsmonitor_batch *batch_marker) +{ +/* assert current thread holding state->main_lock */ + +const struct fsmonitor_batch *batch; +struct fsmonitor_batch *rest; +struct fsmonitor_batch *p; + +if (!batch_marker) +return ; + +trace_printf_key (&trace_fsmonitor,"Truncate: mark (%" PRIu64",%" PRIu64")" , + batch_marker->batch_seq_nr, + (uint64_t )batch_marker->pinned_time); + +for (batch = batch_marker; batch; batch = batch->next) { +time_t t; + +if (!batch->pinned_time)/* an overflow batch */ +continue ; + + t = batch->pinned_time +MY_TIME_DELAY_SECONDS ; +if (t > batch_marker->pinned_time)/* too close to marker */ +continue ; + + goto truncate_past_here; + } + +return ; + +truncate_past_here: + state->current_token_data->batch_tail = (struct fsmonitor_batch *)batch; + + rest = ((struct fsmonitor_batch *)batch)->next; + ((struct fsmonitor_batch *)batch)->next =NULL ; + +for (p = rest; p; p =fsmonitor_batch__pop (p)) { +trace_printf_key (&trace_fsmonitor, +"Truncate: kill (%" PRIu64",%" PRIu64")" , + p->batch_seq_nr, (uint64_t )p->pinned_time); + } +} +static void fsmonitor_free_token_data (struct fsmonitor_token_data *token) {struct fsmonitor_batch *p;
@@ -643,6 +714,15 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,* that work. */ fsmonitor_free_token_data (token_data); + }else if (batch) { +/* +* This batch is the first item in the list +* that is older than the requested sequence +* number and might be considered to be +* obsolete. See if we can truncate the list +* and save some memory. +*/ +with_lock__truncate_old_batches (state, batch); } }
Signature · 1 line
gitgitgadget
PATCH v2 22/28 fsmonitor--daemon: use a cookie file to sync with file system
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Teach fsmonitor--daemon client threads to create a cookie file inside the .git directory and then wait until FS events for the cookie are observed by the FS listener thread.
This helps address the racy nature of file system events by blocking the client response until the kernel has drained any event backlog.
This is especially important on MacOS where kernel events are only issued with a limited frequency. See the latency argument of FSeventStreamCreate(). The kernel only signals every latency seconds, but does not guarantee that the kernel queue is completely drained, so we may have to wait more than one interval. If we increase the frequency, the system is more likely to drop events. We avoid these issues by having each client thread create a unique cookie file and then wait until it is seen in the event stream.
--- builtin/fsmonitor--daemon.c | 210 +++++++++++++++++++++++++++++++++++- fsmonitor--daemon.h | 5 + 2 files changed, 214 insertions(+), 1 deletion(-)
builtin/fsmonitor--daemon.c +209 -1
@@ -92,6 +92,149 @@ static int do_as_client__status(void) } } +enum fsmonitor_cookie_item_result { +FCIR_ERROR =-1 ,/* could not create cookie file ? */ +FCIR_INIT =0 , +FCIR_SEEN , +FCIR_ABORT , +}; + +struct fsmonitor_cookie_item { +struct hashmap_entry entry; +const char *name; +enum fsmonitor_cookie_item_result result; +}; + +static int cookies_cmp (const void *data,const struct hashmap_entry *he1, +const struct hashmap_entry *he2,const void *keydata) +{ +const struct fsmonitor_cookie_item *a = + container_of(he1, const struct fsmonitor_cookie_item, entry); +const struct fsmonitor_cookie_item *b = + container_of(he2, const struct fsmonitor_cookie_item, entry); + +return strcmp (a->name, keydata ? keydata : b->name); +} + +static enum fsmonitor_cookie_item_result with_lock__wait_for_cookie ( +struct fsmonitor_daemon_state *state) +{ +/* assert current thread holding state->main_lock */ + +int fd; +struct fsmonitor_cookie_item *cookie; +struct strbuf cookie_pathname =STRBUF_INIT ; +struct strbuf cookie_filename =STRBUF_INIT ; +enum fsmonitor_cookie_item_result result; +int my_cookie_seq; + +CALLOC_ARRAY (cookie,1 ); + + my_cookie_seq = state->cookie_seq++; + +strbuf_addf (&cookie_filename,"%i-%i" ,getpid (), my_cookie_seq); + +strbuf_addbuf (&cookie_pathname, &state->path_cookie_prefix); +strbuf_addbuf (&cookie_pathname, &cookie_filename); + + cookie->name =strbuf_detach (&cookie_filename,NULL ); + cookie->result =FCIR_INIT ; +hashmap_entry_init (&cookie->entry,strhash (cookie->name)); + +hashmap_add (&state->cookies, &cookie->entry); + +trace_printf_key (&trace_fsmonitor,"cookie-wait: '%s' '%s'" , + cookie->name, cookie_pathname.buf); + +/* +* Create the cookie file on disk and then wait for a notification +* that the listener thread has seen it. +*/ + fd =open (cookie_pathname.buf,O_WRONLY |O_CREAT |O_EXCL ,0600 ); +if (fd >=0 ) { +close (fd); +unlink (cookie_pathname.buf); + +/* +* NEEDSWORK: This is an infinite wait (well, unless another +* thread sends us an abort). I'd like to change this to +* use `pthread_cond_timedwait()` and return an error/timeout +* and let the caller do the trivial response thing. +*/ +while (cookie->result ==FCIR_INIT ) +pthread_cond_wait (&state->cookies_cond, + &state->main_lock); + }else { +error_errno (_ ("could not create fsmonitor cookie '%s'" ), + cookie->name); + + cookie->result =FCIR_ERROR ; + } + +hashmap_remove (&state->cookies, &cookie->entry,NULL ); + + result = cookie->result; + +free ((char *)cookie->name); +free (cookie); +strbuf_release (&cookie_pathname); + +return result; +} + +/* +* Mark these cookies as _SEEN and wake up the corresponding client threads. +*/ +static void with_lock__mark_cookies_seen (struct fsmonitor_daemon_state *state, +const struct string_list *cookie_names) +{ +/* assert current thread holding state->main_lock */ + +int k; +int nr_seen =0 ; + +for (k =0 ; k < cookie_names->nr; k++) { +struct fsmonitor_cookie_item key; +struct fsmonitor_cookie_item *cookie; + + key.name = cookie_names->items[k].string; +hashmap_entry_init (&key.entry,strhash (key.name)); + + cookie =hashmap_get_entry (&state->cookies, &key, entry,NULL ); +if (cookie) { +trace_printf_key (&trace_fsmonitor,"cookie-seen: '%s'" , + cookie->name); + cookie->result =FCIR_SEEN ; + nr_seen++; + } + } + +if (nr_seen) +pthread_cond_broadcast (&state->cookies_cond); +} + +/* +* Set _ABORT on all pending cookies and wake up all client threads. +*/ +static void with_lock__abort_all_cookies (struct fsmonitor_daemon_state *state) +{ +/* assert current thread holding state->main_lock */ + +struct hashmap_iter iter; +struct fsmonitor_cookie_item *cookie; +int nr_aborted =0 ; + +hashmap_for_each_entry (&state->cookies, &iter, cookie, entry) { +trace_printf_key (&trace_fsmonitor,"cookie-abort: '%s'" , + cookie->name); + cookie->result =FCIR_ABORT ; + nr_aborted++; + } + +if (nr_aborted) +pthread_cond_broadcast (&state->cookies_cond); +} +/* * Requests to and from a FSMonitor Protocol V2 provider use an opaque * "token" as a virtual timestamp. Clients can request a summary of all
@@ -395,6 +538,9 @@ static void fsmonitor_free_token_data(struct fsmonitor_token_data *token)* We should create a new token and start fresh (as if we just * booted up). * +* [2] Some of those lost events may have been for cookie files. We +* should assume the worst and abort them rather letting them starve. +* * If there are no concurrent threads readering the current token data * series, we can free it now. Otherwise, let the last reader free * it.
@@ -416,6 +562,8 @@ static void with_lock__do_force_resync(struct fsmonitor_daemon_state *state) state->current_token_data = new_one;fsmonitor_free_token_data (free_me); + +with_lock__abort_all_cookies (state); }void fsmonitor_force_resync (struct fsmonitor_daemon_state *state)
@@ -490,6 +638,8 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,int hash_ret;int do_trivial =0 ;int do_flush =0 ; +int do_cookie =0 ; +enum fsmonitor_cookie_item_result cookie_result;/* * We expect `command` to be of the form:
@@ -522,6 +672,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,*/ do_flush =1 ; do_trivial =1 ; + do_cookie =1 ; }else if (!skip_prefix (command,"builtin:" , &p)) {/* assume V1 timestamp or garbage */
@@ -535,6 +686,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state, "fsmonitor: unsupported V1 protocol '%s'"), command); do_trivial =1 ; + do_cookie =1 ; }else {/* We have "builtin:*" */
@@ -544,12 +696,14 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,"fsmonitor: invalid V2 protocol token '%s'" , command); do_trivial =1 ; + do_cookie =1 ; }else {/* * We have a V2 valid token: * "builtin:<token_id>:<seq_nr>" */ + do_cookie =1 ; } }
@@ -558,6 +712,30 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,if (!state->current_token_data)BUG ("fsmonitor state does not have a current token" ); +/* +* Write a cookie file inside the directory being watched in +* an effort to flush out existing filesystem events that we +* actually care about. Suspend this client thread until we +* see the filesystem events for this cookie file. +* +* Creating the cookie lets us guarantee that our FS listener +* thread has drained the kernel queue and we are caught up +* with the kernel. +* +* If we cannot create the cookie (or otherwise guarantee that +* we are caught up), we send a trivial response. We have to +* assume that there might be some very, very recent activity +* on the FS still in flight. +*/ +if (do_cookie) { + cookie_result =with_lock__wait_for_cookie (state); +if (cookie_result !=FCIR_SEEN ) { +error (_ ("fsmonitor: cookie_result '%d' != SEEN" ), + cookie_result); + do_trivial =1 ; + } + } +if (do_flush)with_lock__do_force_resync (state);
@@ -769,7 +947,9 @@ static int handle_client(void *data,return result; } -#define FSMONITOR_COOKIE_PREFIX ".fsmonitor-daemon-" +#define FSMONITOR_DIR "fsmonitor--daemon" +#define FSMONITOR_COOKIE_DIR "cookies" +#define FSMONITOR_COOKIE_PREFIX (FSMONITOR_DIR "/" FSMONITOR_COOKIE_DIR "/")enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative (const char *rel)
@@ -922,6 +1102,9 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state, } } +if (cookie_names->nr) +with_lock__mark_cookies_seen (state, cookie_names); +pthread_mutex_unlock (&state->main_lock); }
@@ -1011,7 +1194,9 @@ static int fsmonitor_run_daemon(void)memset (&state,0 ,sizeof (state)); +hashmap_init (&state.cookies, cookies_cmp,NULL ,0 );pthread_mutex_init (&state.main_lock,NULL ); +pthread_cond_init (&state.cookies_cond,NULL ); state.error_code =0 ; state.current_token_data =fsmonitor_new_token_data ();
@@ -1035,6 +1220,23 @@ static int fsmonitor_run_daemon(void) state.nr_paths_watching =2 ; } +/* +* We will write filesystem syncing cookie files into +* <gitdir>/<fsmonitor-dir>/<cookie-dir>/<pid>-<seq>. +*/ +strbuf_init (&state.path_cookie_prefix,0 ); +strbuf_addbuf (&state.path_cookie_prefix, &state.path_gitdir_watch); + +strbuf_addch (&state.path_cookie_prefix,'/' ); +strbuf_addstr (&state.path_cookie_prefix,FSMONITOR_DIR ); +mkdir (state.path_cookie_prefix.buf,0777 ); + +strbuf_addch (&state.path_cookie_prefix,'/' ); +strbuf_addstr (&state.path_cookie_prefix,FSMONITOR_COOKIE_DIR ); +mkdir (state.path_cookie_prefix.buf,0777 ); + +strbuf_addch (&state.path_cookie_prefix,'/' ); +/* * Confirm that we can create platform-specific resources for the * filesystem listener before we bother starting all the threads.
@@ -1047,6 +1249,7 @@ static int fsmonitor_run_daemon(void) err =fsmonitor_run_daemon_1 (&state); done: +pthread_cond_destroy (&state.cookies_cond);pthread_mutex_destroy (&state.main_lock);fsmonitor_fs_listen__dtor (&state);
@@ -1054,6 +1257,11 @@ static int fsmonitor_run_daemon(void)strbuf_release (&state.path_worktree_watch);strbuf_release (&state.path_gitdir_watch); +strbuf_release (&state.path_cookie_prefix); + +/* +* NEEDSWORK: Consider "rm -rf <gitdir>/<fsmonitor-dir>" +*/ return err; }
fsmonitor--daemon.h +5 -0
@@ -45,6 +45,11 @@ struct fsmonitor_daemon_state {struct fsmonitor_token_data *current_token_data; +struct strbuf path_cookie_prefix; +pthread_cond_t cookies_cond; +int cookie_seq; +struct hashmap cookies; +int error_code;struct fsmonitor_daemon_backend_data *backend_data;
Signature · 1 line
gitgitgadget
Re: [PATCH v2 22/28] fsmonitor--daemon: use a cookie file to sync with file system
Hi Jeff,
I've found an issue with this patch that I unfortunately failed to fix in time for Git for Windows v2.32.0. It is a subtle issue, and it only reared its head in the form of flakiness when running Scalar's Functional Tests (which is the most comprehensive test suite we have in the absence of proper integration tests for Git).
The issue is the change in behavior between the previous iteration and this one when replying with the trivial response:
Quoted 22 lines · Jeff Hostetler via GitGitGadget
builtin/fsmonitor--daemon.c +2
-0@@ -522,6 +672,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,*/ do_flush =1 ; do_trivial =1 ; + do_cookie =1 ; }else if (!skip_prefix (command,"builtin:" , &p)) {/* assume V1 timestamp or garbage */ @@ -535,6 +686,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state, "fsmonitor: unsupported V1 protocol '%s'"), command); do_trivial =1 ; + do_cookie =1 ; }else {/* We have "builtin:*" */
builtin/fsmonitor--daemon.c static int do_handle_client(struct fsmonitor_daemon_state *state, · line 699
Quoted 4 lines
builtin/fsmonitor--daemon.c +1
-0@@ -544,12 +696,14 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,"fsmonitor: invalid V2 protocol token '%s'" , command); do_trivial =1 ; + do_cookie =1 ;
In the first iteration of this patch series, these three trivial responses were sent without first writing a cookie file and then waiting for the event to arrive.
The symptom of the issue here is that some of Scalar's Functional Test hung, ostensibly waiting for a cookie file that never arrived.
I am not 100% clear on why this only happened in Scalar's Functional Tests and not in the regression test in Git's test suite, but here are my thoughts on this:
- I _suspect_ that the .git/ directory gets deleted on the client side after receiving a trivial response and finishing the test case, and _before_ the daemon can receive the event. As the directory already got deleted, the event never arrives.
- I _think_ that running the test cases in parallel (10 concurrent test cases, if memory serves) exacerbates this problem.
- Simply _not_ writing that cookie file (i.e. removing those three do_cookie = 1 assignments above is sufficient to let Scalar's Functional Tests pass.
- Those trivial responses do not _actually_ need to be guarded behind that cookie file. The worst that could happen is for the FSMonitor daemon to over-report paths that need to be lstat()ed.
Therefore, I would like to suggest this diff to be squashed into this patch in a re-roll:
-- snip --
builtin/fsmonitor--daemon.c +0 -3
@@ -672,7 +672,6 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,*/ do_flush =1 ; do_trivial =1 ; - do_cookie =1 ; }else if (!skip_prefix (command,"builtin:" , &p)) {/* assume V1 timestamp or garbage */
@@ -686,7 +685,6 @@ static int do_handle_client(struct fsmonitor_daemon_state *state, "fsmonitor: unsupported V1 protocol '%s'"), command); do_trivial =1 ; - do_cookie =1 ; }else {/* We have "builtin:*" */
@@ -696,7 +694,6 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,"fsmonitor: invalid V2 protocol token '%s'" , command); do_trivial =1 ; - do_cookie =1 ; }else {/*
-- snap --
For the record, the Windows installer and macOS/Linux packages available at https://github.com/microsoft/git/releases/tag/v2.32.0.vfs.0.1 do come with this fix.
Thanks,
Dscho
Quoted 136 lines
builtin/fsmonitor--daemon.c +56
-1}else {/* * We have a V2 valid token: * "builtin:<token_id>:<seq_nr>" */ + do_cookie =1 ; } }@@ -558,6 +712,30 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,if (!state->current_token_data)BUG ("fsmonitor state does not have a current token" ); +/* +* Write a cookie file inside the directory being watched in +* an effort to flush out existing filesystem events that we +* actually care about. Suspend this client thread until we +* see the filesystem events for this cookie file. +* +* Creating the cookie lets us guarantee that our FS listener +* thread has drained the kernel queue and we are caught up +* with the kernel. +* +* If we cannot create the cookie (or otherwise guarantee that +* we are caught up), we send a trivial response. We have to +* assume that there might be some very, very recent activity +* on the FS still in flight. +*/ +if (do_cookie) { + cookie_result =with_lock__wait_for_cookie (state); +if (cookie_result !=FCIR_SEEN ) { +error (_ ("fsmonitor: cookie_result '%d' != SEEN" ), + cookie_result); + do_trivial =1 ; + } + } +if (do_flush)with_lock__do_force_resync (state);@@ -769,7 +947,9 @@ static int handle_client(void *data,return result; } -#define FSMONITOR_COOKIE_PREFIX ".fsmonitor-daemon-" +#define FSMONITOR_DIR "fsmonitor--daemon" +#define FSMONITOR_COOKIE_DIR "cookies" +#define FSMONITOR_COOKIE_PREFIX (FSMONITOR_DIR "/" FSMONITOR_COOKIE_DIR "/")enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative (const char *rel)@@ -922,6 +1102,9 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state, } } +if (cookie_names->nr) +with_lock__mark_cookies_seen (state, cookie_names); +pthread_mutex_unlock (&state->main_lock); }@@ -1011,7 +1194,9 @@ static int fsmonitor_run_daemon(void)memset (&state,0 ,sizeof (state)); +hashmap_init (&state.cookies, cookies_cmp,NULL ,0 );pthread_mutex_init (&state.main_lock,NULL ); +pthread_cond_init (&state.cookies_cond,NULL ); state.error_code =0 ; state.current_token_data =fsmonitor_new_token_data ();@@ -1035,6 +1220,23 @@ static int fsmonitor_run_daemon(void) state.nr_paths_watching =2 ; } +/* +* We will write filesystem syncing cookie files into +* <gitdir>/<fsmonitor-dir>/<cookie-dir>/<pid>-<seq>. +*/ +strbuf_init (&state.path_cookie_prefix,0 ); +strbuf_addbuf (&state.path_cookie_prefix, &state.path_gitdir_watch); + +strbuf_addch (&state.path_cookie_prefix,'/' ); +strbuf_addstr (&state.path_cookie_prefix,FSMONITOR_DIR ); +mkdir (state.path_cookie_prefix.buf,0777 ); + +strbuf_addch (&state.path_cookie_prefix,'/' ); +strbuf_addstr (&state.path_cookie_prefix,FSMONITOR_COOKIE_DIR ); +mkdir (state.path_cookie_prefix.buf,0777 ); + +strbuf_addch (&state.path_cookie_prefix,'/' ); +/* * Confirm that we can create platform-specific resources for the * filesystem listener before we bother starting all the threads. @@ -1047,6 +1249,7 @@ static int fsmonitor_run_daemon(void) err =fsmonitor_run_daemon_1 (&state); done: +pthread_cond_destroy (&state.cookies_cond);pthread_mutex_destroy (&state.main_lock);fsmonitor_fs_listen__dtor (&state);@@ -1054,6 +1257,11 @@ static int fsmonitor_run_daemon(void)strbuf_release (&state.path_worktree_watch);strbuf_release (&state.path_gitdir_watch); +strbuf_release (&state.path_cookie_prefix); + +/* +* NEEDSWORK: Consider "rm -rf <gitdir>/<fsmonitor-dir>" +*/ return err; }fsmonitor--daemon.h +5
-1@@ -45,6 +45,11 @@ struct fsmonitor_daemon_state {struct fsmonitor_token_data *current_token_data; +struct strbuf path_cookie_prefix; +pthread_cond_t cookies_cond; +int cookie_seq; +struct hashmap cookies; +int error_code;struct fsmonitor_daemon_backend_data *backend_data; --gitgitgadget
PATCH v2 23/28 fsmonitor: enhance existing comments
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
--- fsmonitor.c | 37 ++++++++++++++++++++++++++++++------- 1 file changed, 30 insertions(+), 7 deletions(-)
fsmonitor.c +30 -7
@@ -344,9 +344,25 @@ void refresh_fsmonitor(struct index_state *istate) } apply_results: -/* a fsmonitor process can return '/' to indicate all entries are invalid */ +/* +* The response from FSMonitor (excluding the header token) is +* either: +* +* [a] a (possibly empty) list of NUL delimited relative +* pathnames of changed paths. This list can contain +* files and directories. Directories have a trailing +* slash. +* +* [b] a single '/' to indicate the provider had no +* information and that we should consider everything +* invalid. We call this a trivial response. +*/ if (query_success && query_result.buf[bol] !='/' ) { -/* Mark all entries returned by the monitor as dirty */ +/* +* Mark all pathnames returned by the monitor as dirty. +* +* This updates both the cache-entries and the untracked-cache. +*/ buf = query_result.buf;for (i = bol; i < query_result.len; i++) {if (buf[i] !='\0' )
@@ -361,11 +377,15 @@ void refresh_fsmonitor(struct index_state *istate)if (istate->untracked) istate->untracked->use_fsmonitor =1 ; }else { - -/* We only want to run the post index changed hook if we've actually changed entries, so keep track -* if we actually changed entries or not */ +/* +* We received a trivial response, so invalidate everything. +* +* We only want to run the post index changed hook if +* we've actually changed entries, so keep track if we +* actually changed entries or not. +*/ int is_cache_changed =0 ; -/* Mark all entries invalid */ +for (i =0 ; i < istate->cache_nr; i++) {if (istate->cache[i]->ce_flags &CE_FSMONITOR_VALID ) { is_cache_changed =1 ;
@@ -373,7 +393,10 @@ void refresh_fsmonitor(struct index_state *istate) } } -/* If we're going to check every file, ensure we save the results */ +/* +* If we're going to check every file, ensure we save +* the results. +*/ if (is_cache_changed) istate->cache_changed |=FSMONITOR_CHANGED ;
Signature · 1 line
gitgitgadget
PATCH v2 24/28 fsmonitor: force update index after large responses
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Set the FSMONITOR_CHANGED bit on istate->cache_changed when FSMonitor returns a very large repsonse to ensure that the index is written to disk.
Normally, when the FSMonitor response includes a tracked file, the index is always updated. Similarly, the index might be updated when the response alters the untracked-cache (when enabled). However, in cases where neither of those cause the index to be considered changed, the FSMonitor response is wasted. Subsequent Git commands will make requests with the same token and receive the same response.
If that response is very large, performance may suffer. It would be more efficient to force update the index now (and the token in the index extension) in order to reduce the size of the response received by future commands.
This was observed on Windows after a large checkout. On Windows, the kernel emits events for the files that are changed as they are changed. However, it might delay events for the containing directories until the system is more idle (or someone scans the directory (so it seems)). The first status following a checkout would get the list of files. The subsequent status commands would get the list of directories as the events trickled out. But they would never catch up because the token was not advanced because the index wasn't updated.
This list of directories caused wt_status_collect_untracked() to unnecessarily spend time actually scanning them during each command.
--- fsmonitor.c | 50 +++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 49 insertions(+), 1 deletion(-)
fsmonitor.c +49 -1
@@ -230,6 +230,45 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)untracked_cache_invalidate_path (istate, name,0 ); } +/* +* The number of pathnames that we need to receive from FSMonitor +* before we force the index to be updated. +* +* Note that any pathname within the set of received paths MAY cause +* cache-entry or istate flag bits to be updated and thus cause the +* index to be updated on disk. +* +* However, the response may contain many paths (such as ignored +* paths) that will not update any flag bits. And thus not force the +* index to be updated. (This is fine and normal.) It also means +* that the token will not be updated in the FSMonitor index +* extension. So the next Git command will find the same token in the +* index, make the same token-relative request, and receive the same +* response (plus any newly changed paths). If this response is large +* (and continues to grow), performance could be impacted. +* +* For example, if the user runs a build and it writes 100K object +* files but doesn't modify any source files, the index would not need +* to be updated. The FSMonitor response (after the build and +* relative to a pre-build token) might be 5MB. Each subsequent Git +* command will receive that same 100K/5MB response until something +* causes the index to be updated. And `refresh_fsmonitor()` will +* have to iterate over those 100K paths each time. +* +* Performance could be improved if we optionally force update the +* index after a very large response and get an updated token into +* the FSMonitor index extension. This should allow subsequent +* commands to get smaller and more current responses. +* +* The value chosen here does not need to be precise. The index +* will be updated automatically the first time the user touches +* a tracked file and causes a command like `git status` to +* update an mtime to be updated and/or set a flag bit. +* +* NEEDSWORK: Does this need to be a config value? +*/ +static int fsmonitor_force_update_threshold =100 ; +void refresh_fsmonitor (struct index_state *istate) {struct repository *r = istate->repo ? istate->repo : the_repository;
@@ -363,19 +402,28 @@ void refresh_fsmonitor(struct index_state *istate)* * This updates both the cache-entries and the untracked-cache. */ +int count =0 ; + buf = query_result.buf;for (i = bol; i < query_result.len; i++) {if (buf[i] !='\0' )continue ;fsmonitor_refresh_callback (istate, buf + bol); bol = i +1 ; + count++; } -if (bol < query_result.len) +if (bol < query_result.len) {fsmonitor_refresh_callback (istate, buf + bol); + count++; + }/* Now mark the untracked cache for fsmonitor usage */ if (istate->untracked) istate->untracked->use_fsmonitor =1 ; + +if (count > fsmonitor_force_update_threshold) + istate->cache_changed |=FSMONITOR_CHANGED ; + }else {/* * We received a trivial response, so invalidate everything.
Signature · 1 line
gitgitgadget
PATCH v2 25/28 t7527: create test for fsmonitor--daemon
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
--- t/t7527-builtin-fsmonitor.sh | 475 +++++++++++++++++++++++++++++++++++ 1 file changed, 475 insertions(+) create mode 100755 t/t7527-builtin-fsmonitor.sh
t/t7527-builtin-fsmonitor.sh new file +475 -0
@@ -0,0 +1,475 @@ +#!/bin/sh + +test_description='built-in file system watcher' + +. ./test-lib.sh + +git version --build-options | grep "feature:" | grep "fsmonitor--daemon" || { + skip_all="The built-in FSMonitor is not supported on this platform" + test_done +} + +kill_repo () { + r=$1 + git -C $r fsmonitor--daemon stop >/dev/null 2>/dev/null + rm -rf $1 + return 0 +} + +start_daemon () { + case "$#" in + 1) r="-C $1";; + *) r=""; + esac + + git $r fsmonitor--daemon start || return $? + git $r fsmonitor--daemon status || return $? + + return 0 +} + +test_expect_success 'explicit daemon start and stop' ' + test_when_finished "kill_repo test_explicit" && + + git init test_explicit && + start_daemon test_explicit && + + git -C test_explicit fsmonitor--daemon stop && + test_must_fail git -C test_explicit fsmonitor--daemon status +' + +test_expect_success 'implicit daemon start' ' + test_when_finished "kill_repo test_implicit" && + + git init test_implicit && + test_must_fail git -C test_implicit fsmonitor--daemon status && + + # query will implicitly start the daemon. + # + # for test-script simplicity, we send a V1 timestamp rather than + # a V2 token. either way, the daemon response to any query contains + # a new V2 token. (the daemon may complain that we sent a V1 request, + # but this test case is only concerned with whether the daemon was + # implicitly started.) + + GIT_TRACE2_EVENT="$PWD/.git/trace" \ + test-tool -C test_implicit fsmonitor-client query --token 0 >actual && + nul_to_q <actual >actual.filtered && + grep "builtin:" actual.filtered && + + # confirm that a daemon was started in the background. + # + # since the mechanism for starting the background daemon is platform + # dependent, just confirm that the foreground command received a + # response from the daemon. + + grep :\"query/response-length\" .git/trace && + + git -C test_implicit fsmonitor--daemon status && + git -C test_implicit fsmonitor--daemon stop && + test_must_fail git -C test_implicit fsmonitor--daemon status +' + +test_expect_success 'implicit daemon stop (delete .git)' ' + test_when_finished "kill_repo test_implicit_1" && + + git init test_implicit_1 && + + start_daemon test_implicit_1 && + + # deleting the .git directory will implicitly stop the daemon. + rm -rf test_implicit_1/.git && + + # Create an empty .git directory so that the following Git command + # will stay relative to the `-C` directory. Without this, the Git + # command will (override the requested -C argument) and crawl out + # to the containing Git source tree. This would make the test + # result dependent upon whether we were using fsmonitor on our + # development worktree. + + sleep 1 && + mkdir test_implicit_1/.git && + + test_must_fail git -C test_implicit_1 fsmonitor--daemon status +' + +test_expect_success 'implicit daemon stop (rename .git)' ' + test_when_finished "kill_repo test_implicit_2" && + + git init test_implicit_2 && + + start_daemon test_implicit_2 && + + # renaming the .git directory will implicitly stop the daemon. + mv test_implicit_2/.git test_implicit_2/.xxx && + + # Create an empty .git directory so that the following Git command + # will stay relative to the `-C` directory. Without this, the Git + # command will (override the requested -C argument) and crawl out + # to the containing Git source tree. This would make the test + # result dependent upon whether we were using fsmonitor on our + # development worktree. + + sleep 1 && + mkdir test_implicit_2/.git && + + test_must_fail git -C test_implicit_2 fsmonitor--daemon status +' + +test_expect_success 'cannot start multiple daemons' ' + test_when_finished "kill_repo test_multiple" && + + git init test_multiple && + + start_daemon test_multiple && + + test_must_fail git -C test_multiple fsmonitor--daemon start 2>actual && + grep "fsmonitor--daemon is already running" actual && + + git -C test_multiple fsmonitor--daemon stop && + test_must_fail git -C test_multiple fsmonitor--daemon status +' + +test_expect_success 'setup' ' + >tracked && + >modified && + >delete && + >rename && + mkdir dir1 && + >dir1/tracked && + >dir1/modified && + >dir1/delete && + >dir1/rename && + mkdir dir2 && + >dir2/tracked && + >dir2/modified && + >dir2/delete && + >dir2/rename && + mkdir dirtorename && + >dirtorename/a && + >dirtorename/b && + + cat >.gitignore <<-\EOF && + .gitignore + expect* + actual* + EOF + + git -c core.useBuiltinFSMonitor= add . && + test_tick && + git -c core.useBuiltinFSMonitor= commit -m initial && + + git config core.useBuiltinFSMonitor true +' + +test_expect_success 'update-index implicitly starts daemon' ' + test_must_fail git fsmonitor--daemon status && + + GIT_TRACE2_EVENT="$PWD/.git/trace_implicit_1" \ + git update-index --fsmonitor && + + git fsmonitor--daemon status && + test_might_fail git fsmonitor--daemon stop && + + grep \"event\":\"start\".*\"fsmonitor--daemon\" .git/trace_implicit_1 +' + +test_expect_success 'status implicitly starts daemon' ' + test_must_fail git fsmonitor--daemon status && + + GIT_TRACE2_EVENT="$PWD/.git/trace_implicit_2" \ + git status >actual && + + git fsmonitor--daemon status && + test_might_fail git fsmonitor--daemon stop && + + grep \"event\":\"start\".*\"fsmonitor--daemon\" .git/trace_implicit_2 +' + +edit_files() { + echo 1 >modified + echo 2 >dir1/modified + echo 3 >dir2/modified + >dir1/untracked +} + +delete_files() { + rm -f delete + rm -f dir1/delete + rm -f dir2/delete +} + +create_files() { + echo 1 >new + echo 2 >dir1/new + echo 3 >dir2/new +} + +rename_files() { + mv rename renamed + mv dir1/rename dir1/renamed + mv dir2/rename dir2/renamed +} + +file_to_directory() { + rm -f delete + mkdir delete + echo 1 >delete/new +} + +directory_to_file() { + rm -rf dir1 + echo 1 >dir1 +} + +verify_status() { + git status >actual && + GIT_INDEX_FILE=.git/fresh-index git read-tree master && + GIT_INDEX_FILE=.git/fresh-index git -c core.useBuiltinFSMonitor= status >expect && + test_cmp expect actual && + echo HELLO AFTER && + cat .git/trace && + echo HELLO AFTER +} + +# The next few test cases confirm that our fsmonitor daemon sees each type +# of OS filesystem notification that we care about. At this layer we just +# ensure we are getting the OS notifications and do not try to confirm what +# is reported by `git status`. +# +# We run a simple query after modifying the filesystem just to introduce +# a bit of a delay so that the trace logging from the daemon has time to +# get flushed to disk. +# +# We `reset` and `clean` at the bottom of each test (and before stopping the +# daemon) because these commands might implicitly restart the daemon. + +clean_up_repo_and_stop_daemon () { + git reset --hard HEAD + git clean -fd + git fsmonitor--daemon stop + rm -f .git/trace +} + +test_expect_success 'edit some files' ' + test_when_finished "clean_up_repo_and_stop_daemon" && + + ( + GIT_TRACE_FSMONITOR="$PWD/.git/trace" && + export GIT_TRACE_FSMONITOR && + + start_daemon + ) && + + edit_files && + + test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 && + + grep "^event: dir1/modified$" .git/trace && + grep "^event: dir2/modified$" .git/trace && + grep "^event: modified$" .git/trace && + grep "^event: dir1/untracked$" .git/trace +' + +test_expect_success 'create some files' ' + test_when_finished "clean_up_repo_and_stop_daemon" && + + ( + GIT_TRACE_FSMONITOR="$PWD/.git/trace" && + export GIT_TRACE_FSMONITOR && + + start_daemon + ) && + + create_files && + + test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 && + + grep "^event: dir1/new$" .git/trace && + grep "^event: dir2/new$" .git/trace && + grep "^event: new$" .git/trace +' + +test_expect_success 'delete some files' ' + test_when_finished "clean_up_repo_and_stop_daemon" && + + ( + GIT_TRACE_FSMONITOR="$PWD/.git/trace" && + export GIT_TRACE_FSMONITOR && + + start_daemon + ) && + + delete_files && + + test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 && + + grep "^event: dir1/delete$" .git/trace && + grep "^event: dir2/delete$" .git/trace && + grep "^event: delete$" .git/trace +' + +test_expect_success 'rename some files' ' + test_when_finished "clean_up_repo_and_stop_daemon" && + + ( + GIT_TRACE_FSMONITOR="$PWD/.git/trace" && + export GIT_TRACE_FSMONITOR && + + start_daemon + ) && + + rename_files && + + test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 && + + grep "^event: dir1/rename$" .git/trace && + grep "^event: dir2/rename$" .git/trace && + grep "^event: rename$" .git/trace && + grep "^event: dir1/renamed$" .git/trace && + grep "^event: dir2/renamed$" .git/trace && + grep "^event: renamed$" .git/trace +' + +test_expect_success 'rename directory' ' + test_when_finished "clean_up_repo_and_stop_daemon" && + + ( + GIT_TRACE_FSMONITOR="$PWD/.git/trace" && + export GIT_TRACE_FSMONITOR && + + start_daemon + ) && + + mv dirtorename dirrenamed && + + test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 && + + grep "^event: dirtorename/*$" .git/trace && + grep "^event: dirrenamed/*$" .git/trace +' + +test_expect_success 'file changes to directory' ' + test_when_finished "clean_up_repo_and_stop_daemon" && + + ( + GIT_TRACE_FSMONITOR="$PWD/.git/trace" && + export GIT_TRACE_FSMONITOR && + + start_daemon + ) && + + file_to_directory && + + test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 && + + grep "^event: delete$" .git/trace && + grep "^event: delete/new$" .git/trace +' + +test_expect_success 'directory changes to a file' ' + test_when_finished "clean_up_repo_and_stop_daemon" && + + ( + GIT_TRACE_FSMONITOR="$PWD/.git/trace" && + export GIT_TRACE_FSMONITOR && + + start_daemon + ) && + + directory_to_file && + + test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 && + + grep "^event: dir1$" .git/trace +' + +# The next few test cases exercise the token-resync code. When filesystem +# drops events (because of filesystem velocity or because the daemon isn't +# polling fast enough), we need to discard the cached data (relative to the +# current token) and start collecting events under a new token. +# +# the 'test-tool fsmonitor-client flush' command can be used to send a +# "flush" message to a running daemon and ask it to do a flush/resync. + +test_expect_success 'flush cached data' ' + test_when_finished "kill_repo test_flush" && + + git init test_flush && + + ( + GIT_TEST_FSMONITOR_TOKEN=true && + export GIT_TEST_FSMONITOR_TOKEN && + + GIT_TRACE_FSMONITOR="$PWD/.git/trace_daemon" && + export GIT_TRACE_FSMONITOR && + + start_daemon test_flush + ) && + + # The daemon should have an initial token with no events in _0 and + # then a few (probably platform-specific number of) events in _1. + # These should both have the same <token_id>. + + test-tool -C test_flush fsmonitor-client query --token "builtin:test_00000001:0" >actual_0 && + nul_to_q <actual_0 >actual_q0 && + + touch test_flush/file_1 && + touch test_flush/file_2 && + + test-tool -C test_flush fsmonitor-client query --token "builtin:test_00000001:0" >actual_1 && + nul_to_q <actual_1 >actual_q1 && + + grep "file_1" actual_q1 && + + # Force a flush. This will change the <token_id>, reset the <seq_nr>, and + # flush the file data. Then create some events and ensure that the file + # again appears in the cache. It should have the new <token_id>. + + test-tool -C test_flush fsmonitor-client flush >flush_0 && + nul_to_q <flush_0 >flush_q0 && + grep "^builtin:test_00000002:0Q/Q$" flush_q0 && + + test-tool -C test_flush fsmonitor-client query --token "builtin:test_00000002:0" >actual_2 && + nul_to_q <actual_2 >actual_q2 && + + grep "^builtin:test_00000002:0Q$" actual_q2 && + + touch test_flush/file_3 && + + test-tool -C test_flush fsmonitor-client query --token "builtin:test_00000002:0" >actual_3 && + nul_to_q <actual_3 >actual_q3 && + + grep "file_3" actual_q3 +' + +# The next few test cases create repos where the .git directory is NOT +# inside the one of the working directory. That is, where .git is a file +# that points to a directory elsewhere. This happens for submodules and +# non-primary worktrees. + +test_expect_success 'setup worktree base' ' + git init wt-base && + echo 1 >wt-base/file1 && + git -C wt-base add file1 && + git -C wt-base commit -m "c1" +' + +test_expect_success 'worktree with .git file' ' + git -C wt-base worktree add ../wt-secondary && + + ( + GIT_TRACE2_PERF="$PWD/trace2_wt_secondary" && + export GIT_TRACE2_PERF && + + GIT_TRACE_FSMONITOR="$PWD/trace_wt_secondary" && + export GIT_TRACE_FSMONITOR && + + start_daemon wt-secondary + ) && + + git -C wt-secondary fsmonitor--daemon stop && + test_must_fail git -C wt-secondary fsmonitor--daemon status +' + +test_done
Signature · 1 line
gitgitgadget
PATCH v2 26/28 p7519: add fsmonitor--daemon
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Repeat all of the fsmonitor perf tests using git fsmonitor--daemon and the "Simple IPC" interface.
--- t/perf/p7519-fsmonitor.sh | 42 ++++++++++++++++++++++++++++++++++++--- 1 file changed, 39 insertions(+), 3 deletions(-)
t/perf/p7519-fsmonitor.sh +39 -3
@@ -24,7 +24,8 @@ test_description="Test core.fsmonitor" # GIT_PERF_7519_SPLIT_INDEX: used to configure core.splitIndex # GIT_PERF_7519_FSMONITOR: used to configure core.fsMonitor. May be an # absolute path to an integration. May be a space delimited list of -# absolute paths to integrations. +# absolute paths to integrations. (This hook or list of hooks does not +# include the built-in fsmonitor--daemon.) # # The big win for using fsmonitor is the elimination of the need to scan the # working directory looking for changed and untracked files. If the file
@@ -135,10 +136,16 @@ test_expect_success "one time repo setup" ' setup_for_fsmonitor() { # set INTEGRATION_SCRIPT depending on the environment - if test -n "$INTEGRATION_PATH" + if test -n "$USE_FSMONITOR_DAEMON" then + git config core.useBuiltinFSMonitor true && + INTEGRATION_SCRIPT=false + elif test -n "$INTEGRATION_PATH" + then + git config core.useBuiltinFSMonitor false && INTEGRATION_SCRIPT="$INTEGRATION_PATH" else + git config core.useBuiltinFSMonitor false && # # Choose integration script based on existence of Watchman. # Fall back to an empty integration script.
@@ -174,7 +181,10 @@ test_perf_w_drop_caches () { } test_fsmonitor_suite() { - if test -n "$INTEGRATION_SCRIPT"; then + if test -n "$USE_FSMONITOR_DAEMON" + then + DESC="builtin fsmonitor--daemon" + elif test -n "$INTEGRATION_SCRIPT"; then DESC="fsmonitor=$(basename $INTEGRATION_SCRIPT)" else DESC="fsmonitor=disabled"
@@ -285,4 +295,30 @@ test_expect_success "setup without fsmonitor" ' test_fsmonitor_suite trace_stop +# +# Run a full set of perf tests using the built-in fsmonitor--daemon. +# It does not use the Hook API, so it has a different setup. +# Explicitly start the daemon here and before we start client commands +# so that we can later add custom tracing. +# + +test_lazy_prereq HAVE_FSMONITOR_DAEMON ' + git version --build-options | grep "feature:" | grep "fsmonitor--daemon" +' + +if test_have_prereq HAVE_FSMONITOR_DAEMON +then + USE_FSMONITOR_DAEMON=t + + trace_start fsmonitor--daemon--server + git fsmonitor--daemon start + + trace_start fsmonitor--daemon--client + test_expect_success "setup for fsmonitor--daemon" 'setup_for_fsmonitor' + test_fsmonitor_suite + + git fsmonitor--daemon stop + trace_stop +fi + test_done
Signature · 1 line
gitgitgadget
PATCH v2 27/28 t7527: test status with untracked-cache and fsmonitor--daemon
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Create 2x2 test matrix with the untracked-cache and fsmonitor--daemon features and a series of edits and verify that status output is identical.
--- t/t7527-builtin-fsmonitor.sh | 97 ++++++++++++++++++++++++++++++++++++ 1 file changed, 97 insertions(+)
t/t7527-builtin-fsmonitor.sh +97 -0
@@ -153,6 +153,8 @@ test_expect_success 'setup' ' .gitignore expect* actual* + flush* + trace* EOF git -c core.useBuiltinFSMonitor= add . &&
@@ -472,4 +474,99 @@ test_expect_success 'worktree with .git file' ' test_must_fail git -C wt-secondary fsmonitor--daemon status ' +# TODO Repeat one of the "edit" tests on wt-secondary and confirm that +# TODO we get the same events and behavior -- that is, that fsmonitor--daemon +# TODO correctly listens to events on both the working directory and to the +# TODO referenced GITDIR. + +test_expect_success 'cleanup worktrees' ' + kill_repo wt-secondary && + kill_repo wt-base +' + +# The next few tests perform arbitrary/contrived file operations and +# confirm that status is correct. That is, that the data (or lack of +# data) from fsmonitor doesn't cause incorrect results. And doesn't +# cause incorrect results when the untracked-cache is enabled. + +test_lazy_prereq UNTRACKED_CACHE ' + { git update-index --test-untracked-cache; ret=$?; } && + test $ret -ne 1 +' + +test_expect_success 'Matrix: setup for untracked-cache,fsmonitor matrix' ' + test_might_fail git config --unset core.useBuiltinFSMonitor && + git update-index --no-fsmonitor && + test_might_fail git fsmonitor--daemon stop +' + +matrix_clean_up_repo () { + git reset --hard HEAD + git clean -fd +} + +matrix_try () { + uc=$1 + fsm=$2 + fn=$3 + + test_expect_success "Matrix[uc:$uc][fsm:$fsm] $fn" ' + matrix_clean_up_repo && + $fn && + if test $uc = false -a $fsm = false + then + git status --porcelain=v1 >.git/expect.$fn + else + git status --porcelain=v1 >.git/actual.$fn + test_cmp .git/expect.$fn .git/actual.$fn + fi + ' + + return $? +} + +uc_values="false" +test_have_prereq UNTRACKED_CACHE && uc_values="false true" +for uc_val in $uc_values +do + if test $uc_val = false + then + test_expect_success "Matrix[uc:$uc_val] disable untracked cache" ' + git config core.untrackedcache false && + git update-index --no-untracked-cache + ' + else + test_expect_success "Matrix[uc:$uc_val] enable untracked cache" ' + git config core.untrackedcache true && + git update-index --untracked-cache + ' + fi + + fsm_values="false true" + for fsm_val in $fsm_values + do + if test $fsm_val = false + then + test_expect_success "Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor" ' + test_might_fail git config --unset core.useBuiltinFSMonitor && + git update-index --no-fsmonitor && + test_might_fail git fsmonitor--daemon stop 2>/dev/null + ' + else + test_expect_success "Matrix[uc:$uc_val][fsm:$fsm_val] enable fsmonitor" ' + git config core.useBuiltinFSMonitor true && + git fsmonitor--daemon start && + git update-index --fsmonitor + ' + fi + + matrix_try $uc_val $fsm_val edit_files + matrix_try $uc_val $fsm_val delete_files + matrix_try $uc_val $fsm_val create_files + matrix_try $uc_val $fsm_val rename_files + matrix_try $uc_val $fsm_val file_to_directory + matrix_try $uc_val $fsm_val directory_to_file + done +done + test_done
Signature · 1 line
gitgitgadget
PATCH v2 28/28 t/perf: avoid copying builtin fsmonitor files into test repo
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Do not try to copy a fsmonitor--daemon socket from the current development directory into the test trash directory.
When we run the perf suite without an explicit source repo set, we copy of the current $GIT_DIR into the test trash directory. Unix domain sockets cannot be copied in that manner, so the test setup fails.
Additionally, omit any other fsmonitor--daemon temp files inside the $GIT_DIR directory.
--- t/perf/perf-lib.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
t/perf/perf-lib.sh +1 -1
@@ -74,7 +74,7 @@ test_perf_copy_repo_contents () { for stuff in "$1"/* do case "$stuff" in - */objects|*/hooks|*/config|*/commondir|*/gitdir|*/worktrees) + */objects|*/hooks|*/config|*/commondir|*/gitdir|*/worktrees|*/fsmonitor--daemon*) ;; *) cp -R "$stuff" "$repo/.git/" || exit 1
Signature · 1 line
gitgitgadget
These new global symbols are introduced by the series, but never used outside the file they are added to:
fsmonitor-ipc.o - fsmonitor_ipc__get_path fsmonitor-ipc.o - fsmonitor_ipc__get_state fsmonitor-ipc.o - fsmonitor_ipc__send_command
Perhaps make them file-scope static?
Thanks.
Hi Junio,
Quoted 8 lines · Junio C Hamano
These new global symbols are introduced by the series, but never used outside the file they are added to:
fsmonitor-ipc.o - fsmonitor_ipc__get_path fsmonitor-ipc.o - fsmonitor_ipc__get_state fsmonitor-ipc.o - fsmonitor_ipc__send_commandPerhaps make them file-scope static?
Good idea!
By the way, GitGitGadget keeps getting confused by the fact that one of Peff's patches looks very similar to the tip commit of this here patch series, and mislabels Jeff's PR as being closed.
Would you terribly mind picking up v2 some time soon so that I do not have to click "Reopen" on Jeff's PR all the time (I had to reopen it a couple times already: https://github.com/gitgitgadget/git/pull/923, and I do lack the time to teach GitGitGadget new tricks to avoid this mislabeling).
Thank you,
Dscho
Quoted 8 lines · Junio C Hamano
These new global symbols are introduced by the series, but never used outside the file they are added to:
fsmonitor-ipc.o - fsmonitor_ipc__get_path fsmonitor-ipc.o - fsmonitor_ipc__get_state fsmonitor-ipc.o - fsmonitor_ipc__send_commandPerhaps make them file-scope static?
I intended these to be part of the API for talking to a builtin FSMonitor. They are called from builtin/fsmonitor--daemon.c and t/helper/test-fsmonitor-client.c
Jeff
v3 cover Builtin FSMonitor Feature
In reply to Jeff Hostetler via GitGitGadgetHere is V3 of my patch series to add a builtin FSMonitor daemon to Git. I rebased this series onto v2.32.0.
V3 addresses most of the previous review comments and things we've learned from our experimental testing of V2. (A version of V2 was shipped as an experimental feature in the v2.32.0-based releases of Git for Windows and VFS for Git.)
There are still a few items that I need to address, but that list is getting very short.
The following items from my V2 cover have not yet been addressed:
[ ] Revisit the how the client handles the IPC_STATE__NOT_LISTENING state (where a daemon appears to be running, but is non-responsive)
[ ] Consider having daemon chdir() out of the working directory to avoid directory handle issues on Windows.
[ ] On Windows, If the daemon is started as an elevated process, then client commands might not have access to communicate with it.
[ ] Review if/how we decide to shutdown the FSMonitor daemon after and a significant idle period.
Also, there are potential problems with the untracked-cache that we have been looking at. Concurrently, Tao Klerks independently noticed similar problems with the untracked-cache and has reported/discussed them here on the mailing list. I would like to get to the bottom of them before going further -- at this point I don't know they are related to FSMonitor or not.
In this version, the first commit updates the Simple IPC API to make it easier to pass binary data using {char *, size_t} rather than assuming that the message is a null-terminated string. FSMonitor does not use binary messages and doesn't really need this API change, but I thought it best to fix the API now before we have other callers of IPC.
This patch series contains 34 commits and is rather large. If it would help with the review, I could try to divide it into a client-side part 1 and a daemon-side part 2 -- if there is interest.
Jeff Hostetler (34):
simple-ipc: preparations for supporting binary messages.
fsmonitor--daemon: man page
fsmonitor--daemon: update fsmonitor documentation
fsmonitor-ipc: create client routines for git-fsmonitor--daemon
help: include fsmonitor--daemon feature flag in version info
fsmonitor: config settings are repository-specific
fsmonitor: use IPC to query the builtin FSMonitor daemon
fsmonitor--daemon: add a built-in fsmonitor daemon
fsmonitor--daemon: implement 'stop' and 'status' commands t/helper/fsmonitor-client: create IPC client to talk to FSMonitor
Daemon
fsmonitor-fs-listen-win32: stub in backend for Windows
fsmonitor-fs-listen-macos: stub in backend for MacOS
fsmonitor--daemon: implement 'run' command
fsmonitor--daemon: implement 'start' command
fsmonitor: do not try to operate on bare repos
fsmonitor--daemon: add pathname classification
fsmonitor--daemon: define token-ids
fsmonitor--daemon: create token-based changed path cache
fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows
fsmonitor-fs-listen-macos: add macos header files for FSEvent
fsmonitor-fs-listen-macos: implement FSEvent listener on MacOS
fsmonitor--daemon: implement handle_client callback t/helper/test-touch: add helper to touch a series of files t/perf/p7519: speed up test using "test-tool touch" t/perf: avoid copying builtin fsmonitor files into test repo t/perf/p7519: add fsmonitor--daemon test cases
t7527: create test for fsmonitor--daemon
fsmonitor--daemon: periodically truncate list of modified files
fsmonitor--daemon: use a cookie file to sync with file system
fsmonitor: enhance existing comments
fsmonitor: force update index after large responses
t7527: test status with untracked-cache and fsmonitor--daemon
fsmonitor: handle shortname for .git
t7527: test FS event reporing on MacOS WRT case and Unicode
.gitignore | 1 + Documentation/config/core.txt | 56 +- Documentation/git-fsmonitor--daemon.txt | 75 + Documentation/git-update-index.txt | 27 +- Documentation/githooks.txt | 3 +- Makefile | 17 + builtin.h | 1 + builtin/fsmonitor--daemon.c | 1567 ++++++++++++++++++ builtin/update-index.c | 20 +- cache.h | 1 - compat/fsmonitor/fsmonitor-fs-listen-macos.c | 497 ++++++ compat/fsmonitor/fsmonitor-fs-listen-win32.c | 663 ++++++++ compat/fsmonitor/fsmonitor-fs-listen.h | 49 + compat/simple-ipc/ipc-unix-socket.c | 14 +- compat/simple-ipc/ipc-win32.c | 14 +- config.c | 14 - config.h | 1 - config.mak.uname | 4 + contrib/buildsystems/CMakeLists.txt | 8 + environment.c | 1 - fsmonitor--daemon.h | 140 ++ fsmonitor-ipc.c | 179 ++ fsmonitor-ipc.h | 48 + fsmonitor.c | 189 ++- fsmonitor.h | 14 +- git.c | 1 + help.c | 4 + repo-settings.c | 48 + repository.h | 11 + simple-ipc.h | 7 +- t/README | 4 +- t/helper/test-fsmonitor-client.c | 121 ++ t/helper/test-simple-ipc.c | 34 +- t/helper/test-tool.c | 2 + t/helper/test-tool.h | 2 + t/helper/test-touch.c | 126 ++ t/perf/p7519-fsmonitor.sh | 51 +- t/perf/perf-lib.sh | 2 +- t/t7519-status-fsmonitor.sh | 38 + t/t7527-builtin-fsmonitor.sh | 679 ++++++++ t/test-lib.sh | 6 + 41 files changed, 4618 insertions(+), 121 deletions(-) create mode 100644 Documentation/git-fsmonitor--daemon.txt create mode 100644 builtin/fsmonitor--daemon.c create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-macos.c create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-win32.c create mode 100644 compat/fsmonitor/fsmonitor-fs-listen.h create mode 100644 fsmonitor--daemon.h create mode 100644 fsmonitor-ipc.c create mode 100644 fsmonitor-ipc.h create mode 100644 t/helper/test-fsmonitor-client.c create mode 100644 t/helper/test-touch.c create mode 100755 t/t7527-builtin-fsmonitor.sh
base-commit: ebf3c04b262aa27fbb97f8a0156c2347fecafafb
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-923%2Fjeffhostetler%2Fbuiltin-fsmonitor-v3
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-923/jeffhostetler/builtin-fsmonitor-v3
Pull-Request: https://github.com/gitgitgadget/git/pull/923
Range-diff vs v2:
1: 763fa1ee7bb = 1: cafc71c8d7d simple-ipc: preparations for supporting binary messages.
2: fc180e8591b = 2: 5db2c0390a6 fsmonitor--daemon: man page
3: d56f3e91db9 = 3: 86413bfe347 fsmonitor--daemon: update fsmonitor documentation
4: e4a26372877 ! 4: dfcd3e5ac97 fsmonitor-ipc: create client routines for git-fsmonitor--daemon
@@ fsmonitor-ipc.c (new)
+ struct ipc_client_connection *connection = NULL;
+ struct ipc_client_connect_options options
+ = IPC_CLIENT_CONNECT_OPTIONS_INIT;
++ const char *tok = since_token ? since_token : "";
++ size_t tok_len = since_token ? strlen(since_token) : 0;
+
+ options.wait_if_busy = 1;
+ options.wait_if_not_found = 0;
+
+ trace2_region_enter("fsm_client", "query", NULL);
-+
-+ trace2_data_string("fsm_client", NULL, "query/command",
-+ since_token);
++ trace2_data_string("fsm_client", NULL, "query/command", tok);
+
+try_again:
+ state = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,
@@ fsmonitor-ipc.c (new)
+ switch (state) {
+ case IPC_STATE__LISTENING:
+ ret = ipc_client_send_command_to_connection(
-+ connection, since_token, strlen(since_token), answer);
++ connection, tok, tok_len, answer);
+ ipc_client_close_connection(connection);
+
+ trace2_data_intmax("fsm_client", NULL,
@@ fsmonitor-ipc.c (new)
+ = IPC_CLIENT_CONNECT_OPTIONS_INIT;
+ int ret;
+ enum ipc_active_state state;
++ const char *c = command ? command : "";
++ size_t c_len = command ? strlen(command) : 0;
+
+ strbuf_reset(answer);
+
@@ fsmonitor-ipc.c (new)
+ return -1;
+ }
+
-+ ret = ipc_client_send_command_to_connection(connection,
-+ command, strlen(command),
++ ret = ipc_client_send_command_to_connection(connection, c, c_len,
+ answer);
+ ipc_client_close_connection(connection);
+
+ if (ret == -1) {
-+ die("could not send '%s' command to fsmonitor--daemon",
-+ command);
++ die("could not send '%s' command to fsmonitor--daemon", c);
+ return -1;
+ }
+ 5: d5d09eb1635 ! 5: 0aaca2f9390 help: include fsmonitor--daemon feature flag in version info
@@ help.c: void get_version_info(struct strbuf *buf, int show_build_options)
}
}
+
+ ## t/test-lib.sh ##
+@@ t/test-lib.sh: test_lazy_prereq REBASE_P '
+ # Tests that verify the scheduler integration must set this locally
+ # to avoid errors.
+ GIT_TEST_MAINT_SCHEDULER="none:exit 1"
++
++# Does this platform support git fsmonitor--daemon
++#
++test_lazy_prereq FSMONITOR_DAEMON '
++ git version --build-options | grep "feature:" | grep "fsmonitor--daemon"
++'
6: 67bcf57f594 ! 6: 8b64b7cd367 config: FSMonitor is repository-specific
@@
## Metadata ##
-Author: Johannes Schindelin <[email protected]>
+Author: Jeff Hostetler <[email protected]>
## Commit message ##
- config: FSMonitor is repository-specific
+ fsmonitor: config settings are repository-specific
- This commit refactors `git_config_get_fsmonitor()` into the `repo_*()`
- form that takes a parameter `struct repository *r`.
+ Move FSMonitor config settings to `struct repo_settings`, get rid of
+ the `core_fsmonitor` global variable, and add support for the new
+ `core.useBuiltinFSMonitor` config setting. Move config code to lookup
+ `core.fsmonitor` into `prepare_repo_settings()` with the rest of the
+ setup code.
- That change prepares for the upcoming `core.useBuiltinFSMonitor` flag which
- will be stored in the `repo_settings` struct.
+ The `core_fsmonitor` global variable was used to store the pathname to
+ the FSMonitor hook and it was used as a boolean to see if FSMonitor
+ was enabled. This dual usage will lead to confusion when we add
+ support for a builtin FSMonitor based on IPC, since the builtin
+ FSMonitor doesn't need the hook pathname.
- Signed-off-by: Johannes Schindelin <[email protected]>
+ Replace the boolean usage with an `enum fsmonitor_mode` to represent
+ the state of FSMonitor. And only set the pathname when in HOOK mode.
+
+ Also, disable FSMonitor when the repository working directory is
+ incompatible. For example, in bare repositories, since there aren't
+ any files to watch.
+
+ Signed-off-by: Jeff Hostetler <[email protected]>
## builtin/update-index.c ##
@@ builtin/update-index.c: int cmd_update_index(int argc, const char **argv, const char *prefix)
@@ builtin/update-index.c: int cmd_update_index(int argc, const char **argv, const
if (fsmonitor > 0) {
- if (git_config_get_fsmonitor() == 0)
-+ if (repo_config_get_fsmonitor(r) == 0)
++ if (r->settings.fsmonitor_mode == FSMONITOR_MODE_INCOMPATIBLE)
++ return error(
++ _("repository is incompatible with fsmonitor"));
++
++ if (r->settings.fsmonitor_mode == FSMONITOR_MODE_DISABLED) {
++ warning(_("core.useBuiltinFSMonitor is unset; "
++ "set it if you really want to enable the "
++ "builtin fsmonitor"));
warning(_("core.fsmonitor is unset; "
- "set it if you really want to "
- "enable fsmonitor"));
+- "set it if you really want to "
+- "enable fsmonitor"));
++ "set it if you really want to enable the "
++ "hook-based fsmonitor"));
++ }
add_fsmonitor(&the_index);
report(_("fsmonitor enabled"));
} else if (!fsmonitor) {
- if (git_config_get_fsmonitor() == 1)
-+ if (repo_config_get_fsmonitor(r) == 1)
++ if (r->settings.fsmonitor_mode == FSMONITOR_MODE_IPC)
++ warning(_("core.useBuiltinFSMonitor is set; "
++ "remove it if you really want to "
++ "disable fsmonitor"));
++ if (r->settings.fsmonitor_mode == FSMONITOR_MODE_HOOK)
warning(_("core.fsmonitor is set; "
"remove it if you really want to "
"disable fsmonitor"));
+ ## cache.h ##
+@@ cache.h: extern int core_preload_index;
+ extern int precomposed_unicode;
+ extern int protect_hfs;
+ extern int protect_ntfs;
+-extern const char *core_fsmonitor;
+
+ extern int core_apply_sparse_checkout;
+ extern int core_sparse_checkout_cone;
+
## config.c ##
@@ config.c: int git_config_get_max_percent_split_change(void)
return -1; /* default value */
}
-int git_config_get_fsmonitor(void)
-+int repo_config_get_fsmonitor(struct repository *r)
- {
+-{
- if (git_config_get_pathname("core.fsmonitor", &core_fsmonitor))
-+ if (repo_config_get_pathname(r, "core.fsmonitor", &core_fsmonitor))
- core_fsmonitor = getenv("GIT_TEST_FSMONITOR");
-
- if (core_fsmonitor && !*core_fsmonitor)
+- core_fsmonitor = getenv("GIT_TEST_FSMONITOR");
+-
+- if (core_fsmonitor && !*core_fsmonitor)
+- core_fsmonitor = NULL;
+-
+- if (core_fsmonitor)
+- return 1;
+-
+- return 0;
+-}
+-
+ int git_config_get_index_threads(int *dest)
+ {
+ int is_bool, val;
## config.h ##
@@ config.h: int git_config_get_index_threads(int *dest);
@@ config.h: int git_config_get_index_threads(int *dest);
int git_config_get_split_index(void);
int git_config_get_max_percent_split_change(void);
-int git_config_get_fsmonitor(void);
-+int repo_config_get_fsmonitor(struct repository *r);
/* This dies if the configured or default date is in the future */
int git_config_get_expiry(const char *key, const char **output);
+ ## environment.c ##
+@@ environment.c: int protect_hfs = PROTECT_HFS_DEFAULT;
+ #define PROTECT_NTFS_DEFAULT 1
+ #endif
+ int protect_ntfs = PROTECT_NTFS_DEFAULT;
+-const char *core_fsmonitor;
+
+ /*
+ * The character that begins a commented line in user-editable file
+
## fsmonitor.c ##
+@@
+ #include "dir.h"
+ #include "ewah/ewok.h"
+ #include "fsmonitor.h"
++#include "fsmonitor-ipc.h"
+ #include "run-command.h"
+ #include "strbuf.h"
+
+@@ fsmonitor.c: void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)
+ /*
+ * Call the query-fsmonitor hook passing the last update token of the saved results.
+ */
+-static int query_fsmonitor(int version, const char *last_update, struct strbuf *query_result)
++static int query_fsmonitor_hook(struct repository *r,
++ int version,
++ const char *last_update,
++ struct strbuf *query_result)
+ {
+ struct child_process cp = CHILD_PROCESS_INIT;
+ int result;
+
+- if (!core_fsmonitor)
++ if (r->settings.fsmonitor_mode != FSMONITOR_MODE_HOOK)
+ return -1;
+
+- strvec_push(&cp.args, core_fsmonitor);
++ assert(r->settings.fsmonitor_hook_path);
++ assert(*r->settings.fsmonitor_hook_path);
++
++ strvec_push(&cp.args, r->settings.fsmonitor_hook_path);
+ strvec_pushf(&cp.args, "%d", version);
+ strvec_pushf(&cp.args, "%s", last_update);
+ cp.use_shell = 1;
+@@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)
+ struct strbuf last_update_token = STRBUF_INIT;
+ char *buf;
+ unsigned int i;
++ struct repository *r = istate->repo ? istate->repo : the_repository;
+
+- if (!core_fsmonitor || istate->fsmonitor_has_run_once)
++ if (r->settings.fsmonitor_mode <= FSMONITOR_MODE_DISABLED ||
++ istate->fsmonitor_has_run_once)
+ return;
+
+- hook_version = fsmonitor_hook_version();
+-
+ istate->fsmonitor_has_run_once = 1;
+
+ trace_printf_key(&trace_fsmonitor, "refresh fsmonitor");
++
++ if (r->settings.fsmonitor_mode == FSMONITOR_MODE_IPC) {
++ /* TODO */
++ return;
++ }
++
++ assert(r->settings.fsmonitor_mode == FSMONITOR_MODE_HOOK);
++
++ hook_version = fsmonitor_hook_version();
++
+ /*
+- * This could be racy so save the date/time now and query_fsmonitor
++ * This could be racy so save the date/time now and query_fsmonitor_hook
+ * should be inclusive to ensure we don't miss potential changes.
+ */
+ last_update = getnanotime();
+@@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)
+ strbuf_addf(&last_update_token, "%"PRIu64"", last_update);
+
+ /*
+- * If we have a last update token, call query_fsmonitor for the set of
++ * If we have a last update token, call query_fsmonitor_hook for the set of
+ * changes since that token, else assume everything is possibly dirty
+ * and check it all.
+ */
+ if (istate->fsmonitor_last_update) {
+ if (hook_version == -1 || hook_version == HOOK_INTERFACE_VERSION2) {
+- query_success = !query_fsmonitor(HOOK_INTERFACE_VERSION2,
++ query_success = !query_fsmonitor_hook(
++ r, HOOK_INTERFACE_VERSION2,
+ istate->fsmonitor_last_update, &query_result);
+
+ if (query_success) {
+@@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)
+ }
+
+ if (hook_version == HOOK_INTERFACE_VERSION1) {
+- query_success = !query_fsmonitor(HOOK_INTERFACE_VERSION1,
++ query_success = !query_fsmonitor_hook(
++ r, HOOK_INTERFACE_VERSION1,
+ istate->fsmonitor_last_update, &query_result);
+ }
+
+- trace_performance_since(last_update, "fsmonitor process '%s'", core_fsmonitor);
+- trace_printf_key(&trace_fsmonitor, "fsmonitor process '%s' returned %s",
+- core_fsmonitor, query_success ? "success" : "failure");
++ trace_performance_since(last_update, "fsmonitor process '%s'",
++ r->settings.fsmonitor_hook_path);
++ trace_printf_key(&trace_fsmonitor,
++ "fsmonitor process '%s' returned %s",
++ r->settings.fsmonitor_hook_path,
++ query_success ? "success" : "failure");
+ }
+
+ /* a fsmonitor process can return '/' to indicate all entries are invalid */
@@ fsmonitor.c: void remove_fsmonitor(struct index_state *istate)
void tweak_fsmonitor(struct index_state *istate)
{
unsigned int i;
- int fsmonitor_enabled = git_config_get_fsmonitor();
-+ int fsmonitor_enabled = repo_config_get_fsmonitor(istate->repo ? istate->repo : the_repository);
++ struct repository *r = istate->repo ? istate->repo : the_repository;
++ int fsmonitor_enabled = r->settings.fsmonitor_mode > FSMONITOR_MODE_DISABLED;
if (istate->fsmonitor_dirty) {
if (fsmonitor_enabled) {
+@@ fsmonitor.c: void tweak_fsmonitor(struct index_state *istate)
+ istate->fsmonitor_dirty = NULL;
+ }
+
+- switch (fsmonitor_enabled) {
+- case -1: /* keep: do nothing */
+- break;
+- case 0: /* false */
+- remove_fsmonitor(istate);
+- break;
+- case 1: /* true */
++ if (fsmonitor_enabled)
+ add_fsmonitor(istate);
+- break;
+- default: /* unknown value: do nothing */
+- break;
+- }
++ else
++ remove_fsmonitor(istate);
+ }
+
+ ## fsmonitor.h ##
+@@ fsmonitor.h: int fsmonitor_is_trivial_response(const struct strbuf *query_result);
+ */
+ static inline int is_fsmonitor_refreshed(const struct index_state *istate)
+ {
+- return !core_fsmonitor || istate->fsmonitor_has_run_once;
++ struct repository *r = istate->repo ? istate->repo : the_repository;
++
++ return r->settings.fsmonitor_mode <= FSMONITOR_MODE_DISABLED ||
++ istate->fsmonitor_has_run_once;
+ }
+
+ /*
+@@ fsmonitor.h: static inline int is_fsmonitor_refreshed(const struct index_state *istate)
+ */
+ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce)
+ {
+- if (core_fsmonitor && !(ce->ce_flags & CE_FSMONITOR_VALID)) {
++ struct repository *r = istate->repo ? istate->repo : the_repository;
++
++ if (r->settings.fsmonitor_mode > FSMONITOR_MODE_DISABLED &&
++ !(ce->ce_flags & CE_FSMONITOR_VALID)) {
+ istate->cache_changed = 1;
+ ce->ce_flags |= CE_FSMONITOR_VALID;
+ trace_printf_key(&trace_fsmonitor, "mark_fsmonitor_clean '%s'", ce->name);
+@@ fsmonitor.h: static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache
+ */
+ static inline void mark_fsmonitor_invalid(struct index_state *istate, struct cache_entry *ce)
+ {
+- if (core_fsmonitor) {
++ struct repository *r = istate->repo ? istate->repo : the_repository;
++
++ if (r->settings.fsmonitor_mode > FSMONITOR_MODE_DISABLED) {
+ ce->ce_flags &= ~CE_FSMONITOR_VALID;
+ untracked_cache_invalidate_path(istate, ce->name, 1);
+ trace_printf_key(&trace_fsmonitor, "mark_fsmonitor_invalid '%s'", ce->name);
+
+ ## repo-settings.c ##
+@@
+
+ #define UPDATE_DEFAULT_BOOL(s,v) do { if (s == -1) { s = v; } } while(0)
+
++/*
++ * Return 1 if the repo/workdir is incompatible with FSMonitor.
++ */
++static int is_repo_incompatible_with_fsmonitor(struct repository *r)
++{
++ const char *const_strval;
++
++ /*
++ * Bare repositories don't have a working directory and
++ * therefore, nothing to watch.
++ */
++ if (!r->worktree)
++ return 1;
++
++ /*
++ * GVFS (aka VFS for Git) is incompatible with FSMonitor.
++ *
++ * Granted, core Git does not know anything about GVFS and
++ * we shouldn't make assumptions about a downstream feature,
++ * but users can install both versions. And this can lead
++ * to incorrect results from core Git commands. So, without
++ * bringing in any of the GVFS code, do a simple config test
++ * for a published config setting. (We do not look at the
++ * various *_TEST_* environment variables.)
++ */
++ if (!repo_config_get_value(r, "core.virtualfilesystem", &const_strval))
++ return 1;
++
++ return 0;
++}
++
+ void prepare_repo_settings(struct repository *r)
+ {
+ int value;
+ char *strval;
++ const char *const_strval;
+
+ if (r->settings.initialized)
+ return;
+@@ repo-settings.c: void prepare_repo_settings(struct repository *r)
+ UPDATE_DEFAULT_BOOL(r->settings.commit_graph_read_changed_paths, 1);
+ UPDATE_DEFAULT_BOOL(r->settings.gc_write_commit_graph, 1);
+
++ r->settings.fsmonitor_hook_path = NULL;
++ r->settings.fsmonitor_mode = FSMONITOR_MODE_DISABLED;
++ if (is_repo_incompatible_with_fsmonitor(r))
++ r->settings.fsmonitor_mode = FSMONITOR_MODE_INCOMPATIBLE;
++ else if (!repo_config_get_bool(r, "core.usebuiltinfsmonitor", &value)
++ && value)
++ r->settings.fsmonitor_mode = FSMONITOR_MODE_IPC;
++ else {
++ if (repo_config_get_pathname(r, "core.fsmonitor", &const_strval))
++ const_strval = getenv("GIT_TEST_FSMONITOR");
++ if (const_strval && *const_strval) {
++ r->settings.fsmonitor_hook_path = strdup(const_strval);
++ r->settings.fsmonitor_mode = FSMONITOR_MODE_HOOK;
++ }
++ }
++
+ if (!repo_config_get_int(r, "index.version", &value))
+ r->settings.index_version = value;
+ if (!repo_config_get_maybe_bool(r, "core.untrackedcache", &value)) {
+
+ ## repository.h ##
+@@ repository.h: enum fetch_negotiation_setting {
+ FETCH_NEGOTIATION_NOOP = 3,
+ };
+
++enum fsmonitor_mode {
++ FSMONITOR_MODE_INCOMPATIBLE = -2,
++ FSMONITOR_MODE_UNSET = -1,
++ FSMONITOR_MODE_DISABLED = 0,
++ FSMONITOR_MODE_HOOK = 1, /* core.fsmonitor */
++ FSMONITOR_MODE_IPC = 2, /* core.useBuiltinFSMonitor */
++};
++
+ struct repo_settings {
+ int initialized;
+
+@@ repository.h: struct repo_settings {
+ int gc_write_commit_graph;
+ int fetch_write_commit_graph;
+
++ enum fsmonitor_mode fsmonitor_mode;
++ char *fsmonitor_hook_path;
++
+ int index_version;
+ enum untracked_cache_setting core_untracked_cache;
+
+
+ ## t/README ##
+@@ t/README: every 'git commit-graph write', as if the `--changed-paths` option was
+ passed in.
+
+ GIT_TEST_FSMONITOR=$PWD/t7519/fsmonitor-all exercises the fsmonitor
+-code path for utilizing a file system monitor to speed up detecting
+-new or changed files.
++code path for utilizing a (hook based) file system monitor to speed up
++detecting new or changed files.
+
+ GIT_TEST_INDEX_VERSION=<n> exercises the index read/write code path
+ for the index version specified. Can be set to any valid version
+
+ ## t/t7519-status-fsmonitor.sh ##
+@@ t/t7519-status-fsmonitor.sh: test_expect_success 'status succeeds after staging/unstaging' '
+ )
+ '
+
++# Test that we detect and disallow repos that are incompatible with FSMonitor.
++test_expect_success 'incompatible bare repo' '
++ test_when_finished "rm -rf ./bare-clone" &&
++ git clone --bare . ./bare-clone &&
++ cat >expect <<-\EOF &&
++ error: repository is incompatible with fsmonitor
++ EOF
++ test_must_fail git -C ./bare-clone update-index --fsmonitor 2>actual &&
++ test_cmp expect actual
++'
++
++test_expect_success 'incompatible core.virtualfilesystem' '
++ test_when_finished "rm -rf ./fake-gvfs-clone" &&
++ git clone . ./fake-gvfs-clone &&
++ git -C ./fake-gvfs-clone config core.virtualfilesystem true &&
++ cat >expect <<-\EOF &&
++ error: repository is incompatible with fsmonitor
++ EOF
++ test_must_fail git -C ./fake-gvfs-clone update-index --fsmonitor 2>actual &&
++ test_cmp expect actual
++'
++
+ test_done 7: 7e097cebc14 ! 7: c86b5651ecc fsmonitor: introduce core.useBuiltinFSMonitor to call the daemon via IPC
@@
## Metadata ##
-Author: Johannes Schindelin <[email protected]>
+Author: Jeff Hostetler <[email protected]>
## Commit message ##
- fsmonitor: introduce core.useBuiltinFSMonitor to call the daemon via IPC
+ fsmonitor: use IPC to query the builtin FSMonitor daemon
Use simple IPC to directly communicate with the new builtin file
- system monitor daemon.
-
- Define a new config setting core.useBuiltinFSMonitor to enable the
- builtin file system monitor.
+ system monitor daemon when core.useBuiltinFSMonitor is set.
The core.fsmonitor setting has already been defined as a HOOK pathname. Historically, this has been set to a HOOK script that will
@@ Commit message
- ## config.c ##
-@@ config.c: int git_config_get_max_percent_split_change(void)
-
- int repo_config_get_fsmonitor(struct repository *r)
- {
-+ if (r->settings.use_builtin_fsmonitor > 0) {
-+ core_fsmonitor = "(built-in daemon)";
-+ return 1;
-+ }
-+
- if (repo_config_get_pathname(r, "core.fsmonitor", &core_fsmonitor))
- core_fsmonitor = getenv("GIT_TEST_FSMONITOR");
-
-
## fsmonitor.c ##
-@@
- #include "dir.h"
- #include "ewah/ewok.h"
- #include "fsmonitor.h"
-+#include "fsmonitor-ipc.h"
- #include "run-command.h"
- #include "strbuf.h"
-
-@@ fsmonitor.c: static void fsmonitor_refresh_callback(struct index_state *istate, char *name) -
- void refresh_fsmonitor(struct index_state *istate)
- {
-+ struct repository *r = istate->repo ? istate->repo : the_repository;
- struct strbuf query_result = STRBUF_INIT;
- int query_success = 0, hook_version = -1;
- size_t bol = 0; /* beginning of line */
@@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)
- istate->fsmonitor_has_run_once = 1;
-
trace_printf_key(&trace_fsmonitor, "refresh fsmonitor");
-+
-+ if (r->settings.use_builtin_fsmonitor > 0) {
+
+ if (r->settings.fsmonitor_mode == FSMONITOR_MODE_IPC) {
+- /* TODO */
+- return;
+ query_success = !fsmonitor_ipc__send_query(
-+ istate->fsmonitor_last_update, &query_result);
++ istate->fsmonitor_last_update ?
++ istate->fsmonitor_last_update : "builtin:fake",
++ &query_result);
+ if (query_success) {
+ /*
+ * The response contains a series of nul terminated
@@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)
+ * hook setting.
+ */
+ goto apply_results;
-+ }
-+
- /*
- * This could be racy so save the date/time now and query_fsmonitor
- * should be inclusive to ensure we don't miss potential changes.
+ }
+
+ assert(r->settings.fsmonitor_mode == FSMONITOR_MODE_HOOK);
@@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)
- core_fsmonitor, query_success ? "success" : "failure");
+ query_success ? "success" : "failure");
}
+apply_results:
/* a fsmonitor process can return '/' to indicate all entries are invalid */ if (query_success && query_result.buf[bol] != '/') {
/* Mark all entries returned by the monitor as dirty */
-
- ## repo-settings.c ##
-@@ repo-settings.c: void prepare_repo_settings(struct repository *r)
- r->settings.core_multi_pack_index = value;
- UPDATE_DEFAULT_BOOL(r->settings.core_multi_pack_index, 1);
-
-+ if (!repo_config_get_bool(r, "core.usebuiltinfsmonitor", &value) && value)
-+ r->settings.use_builtin_fsmonitor = 1;
-+
- if (!repo_config_get_bool(r, "feature.manyfiles", &value) && value) {
- UPDATE_DEFAULT_BOOL(r->settings.index_version, 4);
- UPDATE_DEFAULT_BOOL(r->settings.core_untracked_cache, UNTRACKED_CACHE_WRITE);
-
- ## repository.h ##
-@@ repository.h: enum fetch_negotiation_setting {
- struct repo_settings {
- int initialized;
-
-+ int use_builtin_fsmonitor;
-+
- int core_commit_graph;
- int commit_graph_read_changed_paths;
- int gc_write_commit_graph;
8: f362a88632e = 8: f88db92d425 fsmonitor--daemon: add a built-in fsmonitor daemon
9: 4f401310539 ! 9: 02e21384ef0 fsmonitor--daemon: implement client command options
@@ Metadata
Author: Jeff Hostetler <[email protected]>
## Commit message ##
- fsmonitor--daemon: implement client command options
+ fsmonitor--daemon: implement 'stop' and 'status' commands
Implement `stop` and `status` client commands to control and query the
status of a `fsmonitor--daemon` server process (and implicitly start a
@@ builtin/fsmonitor--daemon.c
+
+ switch (state) {
+ case IPC_STATE__LISTENING:
-+ printf(_("The built-in file system monitor is active\n"));
++ printf(_("fsmonitor-daemon is watching '%s'\n"),
++ the_repository->worktree);
+ return 0;
+
+ default:
-+ printf(_("The built-in file system monitor is not active\n"));
++ printf(_("fsmonitor-daemon is not watching '%s'\n"),
++ the_repository->worktree);
+ return 1;
+ }
+}
10: d21af7ff842 ! 10: c2adac8ed4b t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon
@@ t/helper/test-fsmonitor-client.c (new)
+#include "test-tool.h"
+#include "cache.h"
+#include "parse-options.h"
-+//#include "fsmonitor.h"
+#include "fsmonitor-ipc.h"
-+//#include "compat/fsmonitor/fsmonitor-fs-listen.h"
-+//#include "fsmonitor--daemon.h"
-+//#include "simple-ipc.h"
+
+#ifndef HAVE_FSMONITOR_DAEMON_BACKEND
+int cmd__fsmonitor_client(int argc, const char **argv) 11: 49f9e2e3d49 ! 11: 5a9bda72203 fsmonitor-fs-listen-win32: stub in backend for Windows
@@ config.mak.uname: ifneq (,$(findstring MINGW,$(uname_S)))
## contrib/buildsystems/CMakeLists.txt ##
@@ contrib/buildsystems/CMakeLists.txt: else()
- list(APPEND compat_SOURCES compat/simple-ipc/ipc-shared.c compat/simple-ipc/ipc-unix-socket.c)
+ endif()
endif()
+if(CMAKE_SYSTEM_NAME STREQUAL "Windows")
12: 2aa85151f03 = 12: 58758048947 fsmonitor-fs-listen-macos: stub in backend for MacOS
13: 2aa05ad5c67 ! 13: 5d6646df93a fsmonitor--daemon: implement daemon command options
@@ Metadata
Author: Jeff Hostetler <[email protected]>
## Commit message ##
- fsmonitor--daemon: implement daemon command options
+ fsmonitor--daemon: implement 'run' command
- Implement `run` and `start` commands to try to
- begin listening for file system events.
+ Implement `run` command to try to begin listening for file system events.
- This version defines the thread structure with a single
- fsmonitor_fs_listen thread to watch for file system events
- and a simple IPC thread pool to wait for connections from
- Git clients over a well-known named pipe or Unix domain
- socket.
+ This version defines the thread structure with a single fsmonitor_fs_listen
+ thread to watch for file system events and a simple IPC thread pool to
+ watch for connection from Git clients over a well-known named pipe or
+ Unix domain socket.
- This version does not actually do anything yet because the
+ This commit does not actually do anything yet because the platform
backends are still just stubs.
Signed-off-by: Jeff Hostetler <[email protected]>
@@ builtin/fsmonitor--daemon.c
#include "khash.h"
static const char * const builtin_fsmonitor__daemon_usage[] = {
-+ N_("git fsmonitor--daemon start [<options>]"),
+ N_("git fsmonitor--daemon run [<options>]"),
N_("git fsmonitor--daemon stop"),
N_("git fsmonitor--daemon status"),
@@ builtin/fsmonitor--daemon.c
+#define FSMONITOR__IPC_THREADS "fsmonitor.ipcthreads"
+static int fsmonitor__ipc_threads = 8;
+
-+#define FSMONITOR__START_TIMEOUT "fsmonitor.starttimeout"
-+static int fsmonitor__start_timeout_sec = 60;
-+
+static int fsmonitor_config(const char *var, const char *value, void *cb)
+{
+ if (!strcmp(var, FSMONITOR__IPC_THREADS)) {
@@ builtin/fsmonitor--daemon.c
+ return 0;
+ }
+
-+ if (!strcmp(var, FSMONITOR__START_TIMEOUT)) {
-+ int i = git_config_int(var, value);
-+ if (i < 0)
-+ return error(_("value of '%s' out of range: %d"),
-+ FSMONITOR__START_TIMEOUT, i);
-+ fsmonitor__start_timeout_sec = i;
-+ return 0;
-+ }
-+
+ return git_default_config(var, value, cb);
+}
+
@@ builtin/fsmonitor--daemon.c: static int do_as_client__status(void)
+ state.nr_paths_watching = 1;
+
+ /*
-+ * We create/delete cookie files inside the .git directory to
-+ * help us keep sync with the file system. If ".git" is not a
-+ * directory, then <gitdir> is not inside the cone of
-+ * <worktree-root>, so set up a second watch for it.
++ * We create and delete cookie files somewhere inside the .git
++ * directory to help us keep sync with the file system. If
++ * ".git" is not a directory, then <gitdir> is not inside the
++ * cone of <worktree-root>, so set up a second watch to watch
++ * the <gitdir> so that we get events for the cookie files.
+ */
+ strbuf_init(&state.path_gitdir_watch, 0);
+ strbuf_addbuf(&state.path_gitdir_watch, &state.path_worktree_watch);
@@ builtin/fsmonitor--daemon.c: static int do_as_client__status(void)
+ * common error case.
+ */
+ if (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)
-+ die("fsmonitor--daemon is already running.");
-+
-+ return !!fsmonitor_run_daemon();
-+}
-+
-+#ifndef GIT_WINDOWS_NATIVE
-+/*
-+ * This is adapted from `daemonize()`. Use `fork()` to directly create
-+ * and run the daemon in a child process. The fork-parent returns the
-+ * child PID so that we can wait for the child to startup before exiting.
-+ */
-+static int spawn_background_fsmonitor_daemon(pid_t *pid)
-+{
-+ *pid = fork();
-+
-+ switch (*pid) {
-+ case 0:
-+ if (setsid() == -1)
-+ error_errno(_("setsid failed"));
-+ close(0);
-+ close(1);
-+ close(2);
-+ sanitize_stdfds();
-+
-+ return !!fsmonitor_run_daemon();
-+
-+ case -1:
-+ return error_errno(_("could not spawn fsmonitor--daemon in the background"));
-+
-+ default:
-+ return 0;
-+ }
-+}
-+#else
-+/*
-+ * Conceptually like `daemonize()` but different because Windows does not
-+ * have `fork(2)`. Spawn a normal Windows child process but without the
-+ * limitations of `start_command()` and `finish_command()`.
-+ */
-+static int spawn_background_fsmonitor_daemon(pid_t *pid)
-+{
-+ char git_exe[MAX_PATH];
-+ struct strvec args = STRVEC_INIT;
-+ int in, out;
-+
-+ GetModuleFileNameA(NULL, git_exe, MAX_PATH);
-+
-+ in = open("/dev/null", O_RDONLY);
-+ out = open("/dev/null", O_WRONLY);
-+
-+ strvec_push(&args, git_exe);
-+ strvec_push(&args, "fsmonitor--daemon");
-+ strvec_push(&args, "run");
++ die("fsmonitor--daemon is already running '%s'",
++ the_repository->worktree);
+
-+ *pid = mingw_spawnvpe(args.v[0], args.v, NULL, NULL, in, out, out);
-+ close(in);
-+ close(out);
++ printf(_("running fsmonitor-daemon in '%s'\n"),
++ the_repository->worktree);
++ fflush(stdout);
+
-+ strvec_clear(&args);
-+
-+ if (*pid < 0)
-+ return error(_("could not spawn fsmonitor--daemon in the background"));
-+
-+ return 0;
-+}
-+#endif
-+
-+/*
-+ * This is adapted from `wait_or_whine()`. Watch the child process and
-+ * let it get started and begin listening for requests on the socket
-+ * before reporting our success.
-+ */
-+static int wait_for_background_startup(pid_t pid_child)
-+{
-+ int status;
-+ pid_t pid_seen;
-+ enum ipc_active_state s;
-+ time_t time_limit, now;
-+
-+ time(&time_limit);
-+ time_limit += fsmonitor__start_timeout_sec;
-+
-+ for (;;) {
-+ pid_seen = waitpid(pid_child, &status, WNOHANG);
-+
-+ if (pid_seen == -1)
-+ return error_errno(_("waitpid failed"));
-+ else if (pid_seen == 0) {
-+ /*
-+ * The child is still running (this should be
-+ * the normal case). Try to connect to it on
-+ * the socket and see if it is ready for
-+ * business.
-+ *
-+ * If there is another daemon already running,
-+ * our child will fail to start (possibly
-+ * after a timeout on the lock), but we don't
-+ * care (who responds) if the socket is live.
-+ */
-+ s = fsmonitor_ipc__get_state();
-+ if (s == IPC_STATE__LISTENING)
-+ return 0;
-+
-+ time(&now);
-+ if (now > time_limit)
-+ return error(_("fsmonitor--daemon not online yet"));
-+ } else if (pid_seen == pid_child) {
-+ /*
-+ * The new child daemon process shutdown while
-+ * it was starting up, so it is not listening
-+ * on the socket.
-+ *
-+ * Try to ping the socket in the odd chance
-+ * that another daemon started (or was already
-+ * running) while our child was starting.
-+ *
-+ * Again, we don't care who services the socket.
-+ */
-+ s = fsmonitor_ipc__get_state();
-+ if (s == IPC_STATE__LISTENING)
-+ return 0;
-+
-+ /*
-+ * We don't care about the WEXITSTATUS() nor
-+ * any of the WIF*(status) values because
-+ * `cmd_fsmonitor__daemon()` does the `!!result`
-+ * trick on all function return values.
-+ *
-+ * So it is sufficient to just report the
-+ * early shutdown as an error.
-+ */
-+ return error(_("fsmonitor--daemon failed to start"));
-+ } else
-+ return error(_("waitpid is confused"));
-+ }
-+}
-+
-+static int try_to_start_background_daemon(void)
-+{
-+ pid_t pid_child;
-+ int ret;
-+
-+ /*
-+ * Before we try to create a background daemon process, see
-+ * if a daemon process is already listening. This makes it
-+ * easier for us to report an already-listening error to the
-+ * console, since our spawn/daemon can only report the success
-+ * of creating the background process (and not whether it
-+ * immediately exited).
-+ */
-+ if (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)
-+ die("fsmonitor--daemon is already running.");
-+
-+ /*
-+ * Run the actual daemon in a background process.
-+ */
-+ ret = spawn_background_fsmonitor_daemon(&pid_child);
-+ if (pid_child <= 0)
-+ return ret;
-+
-+ /*
-+ * Wait (with timeout) for the background child process get
-+ * started and begin listening on the socket/pipe. This makes
-+ * the "start" command more synchronous and more reliable in
-+ * tests.
-+ */
-+ ret = wait_for_background_startup(pid_child);
-+
-+ return ret;
++ return !!fsmonitor_run_daemon();
+}
+
int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)
@@ builtin/fsmonitor--daemon.c: static int do_as_client__status(void)
+ OPT_INTEGER(0, "ipc-threads",
+ &fsmonitor__ipc_threads,
+ N_("use <n> ipc worker threads")),
-+ OPT_INTEGER(0, "start-timeout",
-+ &fsmonitor__start_timeout_sec,
-+ N_("Max seconds to wait for background daemon startup")),
-+
OPT_END()
};
@@ builtin/fsmonitor--daemon.c: int cmd_fsmonitor__daemon(int argc, const char **ar
+ die(_("invalid 'ipc-threads' value (%d)"),
+ fsmonitor__ipc_threads);
+
-+ if (!strcmp(subcmd, "start"))
-+ return !!try_to_start_background_daemon();
-+
+ if (!strcmp(subcmd, "run"))
+ return !!try_to_run_foreground_daemon(); -: ----------- > 14: 9fe902aad87 fsmonitor--daemon: implement 'start' command
-: ----------- > 15: eef39aa168f fsmonitor: do not try to operate on bare repos
14: d5ababfd03e = 16: 3b12f668060 fsmonitor--daemon: add pathname classification
15: c092cdf2c8b = 17: 37fdce5ec3a fsmonitor--daemon: define token-ids
16: 2ed7bc3fae7 = 18: 84444c44c32 fsmonitor--daemon: create token-based changed path cache
17: 9ea4b04b821 ! 19: 5bba5eb3d1b fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows
@@ compat/fsmonitor/fsmonitor-fs-listen-win32.c
+
+ t = fsmonitor_classify_path_gitdir_relative(path.buf);
+
-+ trace_printf_key(&trace_fsmonitor, "BBB: %s", path.buf);
-+
+ switch (t) {
+ case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:
+ /* special case cookie files within gitdir */
18: 21b2b4f941b = 20: 175ae9a757e fsmonitor-fs-listen-macos: add macos header files for FSEvent
19: 08474bad830 = 21: 5d12b5d808a fsmonitor-fs-listen-macos: implement FSEvent listener on MacOS
20: cc4a596d17c = 22: 39df123143b fsmonitor--daemon: implement handle_client callback
-: ----------- > 23: 3cf8f3cd771 t/helper/test-touch: add helper to touch a series of files
-: ----------- > 24: f1ef9656fc3 t/perf/p7519: speed up test using "test-tool touch"
28: 96a3eab819f = 25: a83485fb10f t/perf: avoid copying builtin fsmonitor files into test repo
26: 5b035c6e0d6 ! 26: de517a8259a p7519: add fsmonitor--daemon @@ Metadata
Author: Jeff Hostetler <[email protected]>
## Commit message ##
- p7519: add fsmonitor--daemon
+ t/perf/p7519: add fsmonitor--daemon test cases
Repeat all of the fsmonitor perf tests using git fsmonitor--daemon and the "Simple IPC" interface.
@@ t/perf/p7519-fsmonitor.sh: test_expect_success "setup without fsmonitor" '
+# Explicitly start the daemon here and before we start client commands +# so that we can later add custom tracing.
+#
-+
-+test_lazy_prereq HAVE_FSMONITOR_DAEMON '
-+ git version --build-options | grep "feature:" | grep "fsmonitor--daemon"
-+'
-+
-+if test_have_prereq HAVE_FSMONITOR_DAEMON
++if test_have_prereq FSMONITOR_DAEMON
+then
+ USE_FSMONITOR_DAEMON=t
+
25: c9159db718a ! 27: 99279c0ebd2 t7527: create test for fsmonitor--daemon
@@ t/t7527-builtin-fsmonitor.sh (new)
+
+. ./test-lib.sh
+
-+git version --build-options | grep "feature:" | grep "fsmonitor--daemon" || {
-+ skip_all="The built-in FSMonitor is not supported on this platform"
++if ! test_have_prereq FSMONITOR_DAEMON
++then
++ skip_all="fsmonitor--daemon is not supported on this platform"
+ test_done
-+}
++fi
+
-+kill_repo () {
++stop_daemon_delete_repo () {
+ r=$1
+ git -C $r fsmonitor--daemon stop >/dev/null 2>/dev/null
+ rm -rf $1
@@ t/t7527-builtin-fsmonitor.sh (new)
+}
+
+test_expect_success 'explicit daemon start and stop' '
-+ test_when_finished "kill_repo test_explicit" &&
++ test_when_finished "stop_daemon_delete_repo test_explicit" &&
+
+ git init test_explicit &&
+ start_daemon test_explicit &&
@@ t/t7527-builtin-fsmonitor.sh (new)
+'
+
+test_expect_success 'implicit daemon start' '
-+ test_when_finished "kill_repo test_implicit" &&
++ test_when_finished "stop_daemon_delete_repo test_implicit" &&
+
+ git init test_implicit &&
+ test_must_fail git -C test_implicit fsmonitor--daemon status &&
@@ t/t7527-builtin-fsmonitor.sh (new)
+ # but this test case is only concerned with whether the daemon was
+ # implicitly started.)
+
-+ GIT_TRACE2_EVENT="$PWD/.git/trace" \
++ GIT_TRACE2_EVENT="$(pwd)/.git/trace" \
+ test-tool -C test_implicit fsmonitor-client query --token 0 >actual &&
+ nul_to_q <actual >actual.filtered &&
+ grep "builtin:" actual.filtered &&
@@ t/t7527-builtin-fsmonitor.sh (new)
+'
+
+test_expect_success 'implicit daemon stop (delete .git)' '
-+ test_when_finished "kill_repo test_implicit_1" &&
++ test_when_finished "stop_daemon_delete_repo test_implicit_1" &&
+
+ git init test_implicit_1 &&
+
@@ t/t7527-builtin-fsmonitor.sh (new)
+ # deleting the .git directory will implicitly stop the daemon.
+ rm -rf test_implicit_1/.git &&
+
-+ # Create an empty .git directory so that the following Git command
-+ # will stay relative to the `-C` directory. Without this, the Git
-+ # command will (override the requested -C argument) and crawl out
-+ # to the containing Git source tree. This would make the test
-+ # result dependent upon whether we were using fsmonitor on our
-+ # development worktree.
-+
++ # [1] Create an empty .git directory so that the following Git
++ # command will stay relative to the `-C` directory.
++ #
++ # Without this, the Git command will override the requested
++ # -C argument and crawl out to the containing Git source tree.
++ # This would make the test result dependent upon whether we
++ # were using fsmonitor on our development worktree.
++ #
+ sleep 1 &&
+ mkdir test_implicit_1/.git &&
+
@@ t/t7527-builtin-fsmonitor.sh (new)
+'
+
+test_expect_success 'implicit daemon stop (rename .git)' '
-+ test_when_finished "kill_repo test_implicit_2" &&
++ test_when_finished "stop_daemon_delete_repo test_implicit_2" &&
+
+ git init test_implicit_2 &&
+
@@ t/t7527-builtin-fsmonitor.sh (new)
+ # renaming the .git directory will implicitly stop the daemon.
+ mv test_implicit_2/.git test_implicit_2/.xxx &&
+
-+ # Create an empty .git directory so that the following Git command
-+ # will stay relative to the `-C` directory. Without this, the Git
-+ # command will (override the requested -C argument) and crawl out
-+ # to the containing Git source tree. This would make the test
-+ # result dependent upon whether we were using fsmonitor on our
-+ # development worktree.
-+
++ # See [1] above.
++ #
+ sleep 1 &&
+ mkdir test_implicit_2/.git &&
+
@@ t/t7527-builtin-fsmonitor.sh (new)
+'
+
+test_expect_success 'cannot start multiple daemons' '
-+ test_when_finished "kill_repo test_multiple" &&
++ test_when_finished "stop_daemon_delete_repo test_multiple" &&
+
+ git init test_multiple &&
+
@@ t/t7527-builtin-fsmonitor.sh (new)
+ test_must_fail git -C test_multiple fsmonitor--daemon status
+'
+
++# These tests use the main repo in the trash directory
++
+test_expect_success 'setup' '
+ >tracked &&
+ >modified &&
@@ t/t7527-builtin-fsmonitor.sh (new)
+ git config core.useBuiltinFSMonitor true
+'
+
++# The test already explicitly stopped (or tried to stop) the daemon.
++# This is here in case something else fails first.
++#
++redundant_stop_daemon () {
++ git fsmonitor--daemon stop
++ return 0
++}
++
+test_expect_success 'update-index implicitly starts daemon' '
++ test_when_finished redundant_stop_daemon &&
++
+ test_must_fail git fsmonitor--daemon status &&
+
-+ GIT_TRACE2_EVENT="$PWD/.git/trace_implicit_1" \
++ GIT_TRACE2_EVENT="$(pwd)/.git/trace_implicit_1" \
+ git update-index --fsmonitor &&
+
+ git fsmonitor--daemon status &&
@@ t/t7527-builtin-fsmonitor.sh (new)
+'
+
+test_expect_success 'status implicitly starts daemon' '
++ test_when_finished redundant_stop_daemon &&
++
+ test_must_fail git fsmonitor--daemon status &&
+
-+ GIT_TRACE2_EVENT="$PWD/.git/trace_implicit_2" \
++ GIT_TRACE2_EVENT="$(pwd)/.git/trace_implicit_2" \
+ git status >actual &&
+
+ git fsmonitor--daemon status &&
@@ t/t7527-builtin-fsmonitor.sh (new)
+}
+
+test_expect_success 'edit some files' '
-+ test_when_finished "clean_up_repo_and_stop_daemon" &&
++ test_when_finished clean_up_repo_and_stop_daemon &&
+
+ (
-+ GIT_TRACE_FSMONITOR="$PWD/.git/trace" &&
++ GIT_TRACE_FSMONITOR="$(pwd)/.git/trace" &&
+ export GIT_TRACE_FSMONITOR &&
+
+ start_daemon
@@ t/t7527-builtin-fsmonitor.sh (new)
+'
+
+test_expect_success 'create some files' '
-+ test_when_finished "clean_up_repo_and_stop_daemon" &&
++ test_when_finished clean_up_repo_and_stop_daemon &&
+
+ (
-+ GIT_TRACE_FSMONITOR="$PWD/.git/trace" &&
++ GIT_TRACE_FSMONITOR="$(pwd)/.git/trace" &&
+ export GIT_TRACE_FSMONITOR &&
+
+ start_daemon
@@ t/t7527-builtin-fsmonitor.sh (new)
+'
+
+test_expect_success 'delete some files' '
-+ test_when_finished "clean_up_repo_and_stop_daemon" &&
++ test_when_finished clean_up_repo_and_stop_daemon &&
+
+ (
-+ GIT_TRACE_FSMONITOR="$PWD/.git/trace" &&
++ GIT_TRACE_FSMONITOR="$(pwd)/.git/trace" &&
+ export GIT_TRACE_FSMONITOR &&
+
+ start_daemon
@@ t/t7527-builtin-fsmonitor.sh (new)
+'
+
+test_expect_success 'rename some files' '
-+ test_when_finished "clean_up_repo_and_stop_daemon" &&
++ test_when_finished clean_up_repo_and_stop_daemon &&
+
+ (
-+ GIT_TRACE_FSMONITOR="$PWD/.git/trace" &&
++ GIT_TRACE_FSMONITOR="$(pwd)/.git/trace" &&
+ export GIT_TRACE_FSMONITOR &&
+
+ start_daemon
@@ t/t7527-builtin-fsmonitor.sh (new)
+'
+
+test_expect_success 'rename directory' '
-+ test_when_finished "clean_up_repo_and_stop_daemon" &&
++ test_when_finished clean_up_repo_and_stop_daemon &&
+
+ (
-+ GIT_TRACE_FSMONITOR="$PWD/.git/trace" &&
++ GIT_TRACE_FSMONITOR="$(pwd)/.git/trace" &&
+ export GIT_TRACE_FSMONITOR &&
+
+ start_daemon
@@ t/t7527-builtin-fsmonitor.sh (new)
+'
+
+test_expect_success 'file changes to directory' '
-+ test_when_finished "clean_up_repo_and_stop_daemon" &&
++ test_when_finished clean_up_repo_and_stop_daemon &&
+
+ (
-+ GIT_TRACE_FSMONITOR="$PWD/.git/trace" &&
++ GIT_TRACE_FSMONITOR="$(pwd)/.git/trace" &&
+ export GIT_TRACE_FSMONITOR &&
+
+ start_daemon
@@ t/t7527-builtin-fsmonitor.sh (new)
+'
+
+test_expect_success 'directory changes to a file' '
-+ test_when_finished "clean_up_repo_and_stop_daemon" &&
++ test_when_finished clean_up_repo_and_stop_daemon &&
+
+ (
-+ GIT_TRACE_FSMONITOR="$PWD/.git/trace" &&
++ GIT_TRACE_FSMONITOR="$(pwd)/.git/trace" &&
+ export GIT_TRACE_FSMONITOR &&
+
+ start_daemon
@@ t/t7527-builtin-fsmonitor.sh (new)
+# "flush" message to a running daemon and ask it to do a flush/resync.
+
+test_expect_success 'flush cached data' '
-+ test_when_finished "kill_repo test_flush" &&
++ test_when_finished "stop_daemon_delete_repo test_flush" &&
+
+ git init test_flush &&
+
@@ t/t7527-builtin-fsmonitor.sh (new)
+ GIT_TEST_FSMONITOR_TOKEN=true &&
+ export GIT_TEST_FSMONITOR_TOKEN &&
+
-+ GIT_TRACE_FSMONITOR="$PWD/.git/trace_daemon" &&
++ GIT_TRACE_FSMONITOR="$(pwd)/.git/trace_daemon" &&
+ export GIT_TRACE_FSMONITOR &&
+
+ start_daemon test_flush
@@ t/t7527-builtin-fsmonitor.sh (new)
+ git -C wt-base worktree add ../wt-secondary &&
+
+ (
-+ GIT_TRACE2_PERF="$PWD/trace2_wt_secondary" &&
++ GIT_TRACE2_PERF="$(pwd)/trace2_wt_secondary" &&
+ export GIT_TRACE2_PERF &&
+
-+ GIT_TRACE_FSMONITOR="$PWD/trace_wt_secondary" &&
++ GIT_TRACE_FSMONITOR="$(pwd)/trace_wt_secondary" &&
+ export GIT_TRACE_FSMONITOR &&
+
+ start_daemon wt-secondary
@@ t/t7527-builtin-fsmonitor.sh (new)
+ test_must_fail git -C wt-secondary fsmonitor--daemon status
+'
+
++# NEEDSWORK: Repeat one of the "edit" tests on wt-secondary and
++# confirm that we get the same events and behavior -- that is, that
++# fsmonitor--daemon correctly watches BOTH the working directory and
++# the external GITDIR directory and behaves the same as when ".git"
++# is a directory inside the working directory.
++
++test_expect_success 'cleanup worktrees' '
++ stop_daemon_delete_repo wt-secondary &&
++ stop_daemon_delete_repo wt-base
++'
++
+test_done
21: f0da90e9b05 = 28: 3f36a31eb42 fsmonitor--daemon: periodically truncate list of modified files
22: bb7b1912bb4 ! 29: 555caca2216 fsmonitor--daemon: use a cookie file to sync with file system
@@ builtin/fsmonitor--daemon.c: static int do_handle_client(struct fsmonitor_daemon
/*
* We expect `command` to be of the form:
@@ builtin/fsmonitor--daemon.c: static int do_handle_client(struct fsmonitor_daemon_state *state,
- */
- do_flush = 1;
- do_trivial = 1;
-+ do_cookie = 1;
-
- } else if (!skip_prefix(command, "builtin:", &p)) {
- /* assume V1 timestamp or garbage */
-@@ builtin/fsmonitor--daemon.c: static int do_handle_client(struct fsmonitor_daemon_state *state,
- "fsmonitor: unsupported V1 protocol '%s'"),
- command);
- do_trivial = 1;
-+ do_cookie = 1;
-
- } else {
- /* We have "builtin:*" */
-@@ builtin/fsmonitor--daemon.c: static int do_handle_client(struct fsmonitor_daemon_state *state,
- "fsmonitor: invalid V2 protocol token '%s'",
- command);
- do_trivial = 1;
-+ do_cookie = 1;
-
- } else {
- /*
* We have a V2 valid token:
* "builtin:<token_id>:<seq_nr>"
*/
@@ builtin/fsmonitor--daemon.c: static int fsmonitor_run_daemon(void)
+ /*
+ * We will write filesystem syncing cookie files into
+ * <gitdir>/<fsmonitor-dir>/<cookie-dir>/<pid>-<seq>.
++ *
++ * The extra layers of subdirectories here keep us from
++ * changing the mtime on ".git/" or ".git/foo/" when we create
++ * or delete cookie files.
++ *
++ * There have been problems with some IDEs that do a
++ * non-recursive watch of the ".git/" directory and run a
++ * series of commands any time something happens.
++ *
++ * For example, if we place our cookie files directly in
++ * ".git/" or ".git/foo/" then a `git status` (or similar
++ * command) from the IDE will cause a cookie file to be
++ * created in one of those dirs. This causes the mtime of
++ * those dirs to change. This triggers the IDE's watch
++ * notification. This triggers the IDE to run those commands
++ * again. And the process repeats and the machine never goes
++ * idle.
++ *
++ * Adding the extra layers of subdirectories prevents the
++ * mtime of ".git/" and ".git/foo" from changing when a
++ * cookie file is created.
+ */
+ strbuf_init(&state.path_cookie_prefix, 0);
+ strbuf_addbuf(&state.path_cookie_prefix, &state.path_gitdir_watch);
23: 102e17cbc87 = 30: 75bb4bc8463 fsmonitor: enhance existing comments
24: 11ea2f97def ! 31: 8b3c4f4e6dd fsmonitor: force update index after large responses @@ fsmonitor.c: static void fsmonitor_refresh_callback(struct index_state *istate,
+
void refresh_fsmonitor(struct index_state *istate)
{
- struct repository *r = istate->repo ? istate->repo : the_repository;
+ struct strbuf query_result = STRBUF_INIT;
@@ fsmonitor.c: apply_results:
*
* This updates both the cache-entries and the untracked-cache.
27: 1483c68855c ! 32: 97dce46d1d0 t7527: test status with untracked-cache and fsmonitor--daemon
@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'setup' '
EOF
git -c core.useBuiltinFSMonitor= add . &&
-@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'worktree with .git file' '
- test_must_fail git -C wt-secondary fsmonitor--daemon status
+@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'cleanup worktrees' '
+ stop_daemon_delete_repo wt-base
'
-+# TODO Repeat one of the "edit" tests on wt-secondary and confirm that -+# TODO we get the same events and behavior -- that is, that fsmonitor--daemon -+# TODO correctly listens to events on both the working directory and to the -+# TODO referenced GITDIR.
-+
-+test_expect_success 'cleanup worktrees' '
-+ kill_repo wt-secondary &&
-+ kill_repo wt-base
-+'
-+
+# The next few tests perform arbitrary/contrived file operations and +# confirm that status is correct. That is, that the data (or lack of +# data) from fsmonitor doesn't cause incorrect results. And doesn't
-: ----------- > 33: e32ba686f7e fsmonitor: handle shortname for .git -: ----------- > 34: 627e27fe60b t7527: test FS event reporing on MacOS WRT case and Unicode
Signature · 1 line
gitgitgadget
PATCH v3 1/34 simple-ipc: preparations for supporting binary messages.
From: Jeff Hostetler <[email protected]>
Add command_len argument to the Simple IPC API.
In my original Simple IPC API, I assumed that the request would always be a null-terminated string of text characters. The command arg was just a const char *.
I found a caller that would like to pass a binary command to the daemon, so I want to ammend the Simple IPC API to take const char *command, size_t command_len and pass that to the daemon. (Really, the first arg should just be a void * or const unsigned byte * to make that clearer.)
Note, the response side has always been a struct strbuf which includes the buffer and length, so we already support returning a binary answer. (Yes, it feels a little weird returning a binary buffer in a strbuf, but it works.)
--- compat/simple-ipc/ipc-unix-socket.c | 14 +++++++----- compat/simple-ipc/ipc-win32.c | 14 +++++++----- simple-ipc.h | 7 ++++-- t/helper/test-simple-ipc.c | 34 +++++++++++++++++++---------- 4 files changed, 46 insertions(+), 23 deletions(-)
compat/simple-ipc/ipc-unix-socket.c +9 -5
@@ -168,7 +168,8 @@ void ipc_client_close_connection(struct ipc_client_connection *connection)int ipc_client_send_command_to_connection (struct ipc_client_connection *connection, -const char *message,struct strbuf *answer) +const char *message,size_t message_len, +struct strbuf *answer) {int ret =0 ;
@@ -176,7 +177,7 @@ int ipc_client_send_command_to_connection(trace2_region_enter ("ipc-client" ,"send-command" ,NULL ); -if (write_packetized_from_buf_no_flush (message,strlen (message), +if (write_packetized_from_buf_no_flush (message, message_len, connection->fd) <0 ||packet_flush_gently (connection->fd) <0 ) { ret =error (_ ("could not send IPC command" ));
@@ -197,7 +198,8 @@ done:int ipc_client_send_command (const char *path,const struct ipc_client_connect_options *options, -const char *message,struct strbuf *answer) +const char *message,size_t message_len, +struct strbuf *answer) {int ret =-1 ;enum ipc_active_state state;
@@ -208,7 +210,9 @@ int ipc_client_send_command(const char *path,if (state !=IPC_STATE__LISTENING )return ret; - ret =ipc_client_send_command_to_connection (connection, message, answer); + ret =ipc_client_send_command_to_connection (connection, + message, message_len, + answer);ipc_client_close_connection (connection);
@@ -503,7 +507,7 @@ static int worker_thread__do_io(if (ret >=0 ) { ret = worker_thread_data->server_data->application_cb ( worker_thread_data->server_data->application_data, - buf.buf, do_io_reply_callback, &reply_data); + buf.buf, buf.len, do_io_reply_callback, &reply_data);packet_flush_gently (reply_data.fd); }
compat/simple-ipc/ipc-win32.c +9 -5
@@ -208,7 +208,8 @@ void ipc_client_close_connection(struct ipc_client_connection *connection)int ipc_client_send_command_to_connection (struct ipc_client_connection *connection, -const char *message,struct strbuf *answer) +const char *message,size_t message_len, +struct strbuf *answer) {int ret =0 ;
@@ -216,7 +217,7 @@ int ipc_client_send_command_to_connection(trace2_region_enter ("ipc-client" ,"send-command" ,NULL ); -if (write_packetized_from_buf_no_flush (message,strlen (message), +if (write_packetized_from_buf_no_flush (message, message_len, connection->fd) <0 ||packet_flush_gently (connection->fd) <0 ) { ret =error (_ ("could not send IPC command" ));
@@ -239,7 +240,8 @@ done:int ipc_client_send_command (const char *path,const struct ipc_client_connect_options *options, -const char *message,struct strbuf *response) +const char *message,size_t message_len, +struct strbuf *response) {int ret =-1 ;enum ipc_active_state state;
@@ -250,7 +252,9 @@ int ipc_client_send_command(const char *path,if (state !=IPC_STATE__LISTENING )return ret; - ret =ipc_client_send_command_to_connection (connection, message, response); + ret =ipc_client_send_command_to_connection (connection, + message, message_len, + response);ipc_client_close_connection (connection);
@@ -458,7 +462,7 @@ static int do_io(struct ipc_server_thread_data *server_thread_data)if (ret >=0 ) { ret = server_thread_data->server_data->application_cb ( server_thread_data->server_data->application_data, - buf.buf, do_io_reply_callback, &reply_data); + buf.buf, buf.len, do_io_reply_callback, &reply_data);packet_flush_gently (reply_data.fd);
simple-ipc.h +5 -2
@@ -107,7 +107,8 @@ void ipc_client_close_connection(struct ipc_client_connection *connection);*/ int ipc_client_send_command_to_connection (struct ipc_client_connection *connection, -const char *message,struct strbuf *answer); +const char *message,size_t message_len, +struct strbuf *answer);/* * Used by the client to synchronously connect and send and receive a
@@ -119,7 +120,8 @@ int ipc_client_send_command_to_connection(*/ int ipc_client_send_command (const char *path,const struct ipc_client_connect_options *options, -const char *message,struct strbuf *answer); +const char *message,size_t message_len, +struct strbuf *answer);/* * Simple IPC Server Side API.
@@ -144,6 +146,7 @@ typedef int (ipc_server_reply_cb)(struct ipc_server_reply_data *,*/ typedef int (ipc_server_application_cb )(void *application_data,const char *request, +size_t request_len,ipc_server_reply_cb *reply_cb,struct ipc_server_reply_data *reply_data);
t/helper/test-simple-ipc.c +23 -11
@@ -112,7 +112,7 @@ static int app__slow_command(ipc_server_reply_cb *reply_cb,/* * The client sent a command followed by a (possibly very) large buffer. */ -static int app__sendbytes_command (const char *received, +static int app__sendbytes_command (const char *received,size_t received_len,ipc_server_reply_cb *reply_cb,struct ipc_server_reply_data *reply_data) {
@@ -123,6 +123,13 @@ static int app__sendbytes_command(const char *received,int errs =0 ;int ret; +/* +* The test is setup to send: +* "sendbytes" SP <n * char> +*/ +if (received_len <strlen ("sendbytes " )) +BUG ("received_len is short in app__sendbytes_command" ); +if (skip_prefix (received,"sendbytes " , &p)) len_ballast =strlen (p);
@@ -160,7 +167,7 @@ static ipc_server_application_cb test_app_cb;* by this application. */ static int test_app_cb (void *application_data, -const char *command, +const char *command,size_t command_len,ipc_server_reply_cb *reply_cb,struct ipc_server_reply_data *reply_data) {
@@ -173,7 +180,7 @@ static int test_app_cb(void *application_data,if (application_data != (void *)&my_app_data)BUG ("application_cb: application_data pointer wrong" ); -if (!strcmp (command,"quit" )) { +if (command_len ==4 && !strncmp (command,"quit" ,4 )) {/* * The client sent a "quit" command. This is an async * request for the server to shutdown.
@@ -193,22 +200,23 @@ static int test_app_cb(void *application_data,return SIMPLE_IPC_QUIT ; } -if (!strcmp (command,"ping" )) { +if (command_len ==4 && !strncmp (command,"ping" ,4 )) {const char *answer ="pong" ;return reply_cb (reply_data, answer,strlen (answer)); } -if (!strcmp (command,"big" )) +if (command_len ==3 && !strncmp (command,"big" ,3 ))return app__big_command (reply_cb, reply_data); -if (!strcmp (command,"chunk" )) +if (command_len ==5 && !strncmp (command,"chunk" ,5 ))return app__chunk_command (reply_cb, reply_data); -if (!strcmp (command,"slow" )) +if (command_len ==4 && !strncmp (command,"slow" ,4 ))return app__slow_command (reply_cb, reply_data); -if (starts_with (command,"sendbytes " )) -return app__sendbytes_command (command, reply_cb, reply_data); +if (command_len >=10 &&starts_with (command,"sendbytes " )) +return app__sendbytes_command (command, command_len, + reply_cb, reply_data);return app__unhandled_command (command, reply_cb, reply_data); }
@@ -488,7 +496,9 @@ static int client__send_ipc(void) options.wait_if_busy =1 ; options.wait_if_not_found =0 ; -if (!ipc_client_send_command (cl_args.path, &options, command, &buf)) { +if (!ipc_client_send_command (cl_args.path, &options, + command,strlen (command), + &buf)) {if (buf.len) {printf ("%s\n" , buf.buf);fflush (stdout);
@@ -556,7 +566,9 @@ static int do_sendbytes(int bytecount, char byte, const char *path,strbuf_addstr (&buf_send,"sendbytes " );strbuf_addchars (&buf_send, byte, bytecount); -if (!ipc_client_send_command (path, options, buf_send.buf, &buf_resp)) { +if (!ipc_client_send_command (path, options, + buf_send.buf, buf_send.len, + &buf_resp)) {strbuf_rtrim (&buf_resp);printf ("sent:%c%08d %s\n" , byte, bytecount, buf_resp.buf);fflush (stdout);
Signature · 1 line
gitgitgadget
PATCH v3 2/34 fsmonitor--daemon: man page
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Create a manual page describing the git fsmonitor--daemon feature.
--- Documentation/git-fsmonitor--daemon.txt | 75 +++++++++++++++++++++++++ 1 file changed, 75 insertions(+) create mode 100644 Documentation/git-fsmonitor--daemon.txt
Documentation/git-fsmonitor--daemon.txt new file +75 -0
@@ -0,0 +1,75 @@ +git-fsmonitor--daemon(1) +======================== + +NAME +---- +git-fsmonitor--daemon - A Built-in File System Monitor + +SYNOPSIS +-------- +[verse] +'git fsmonitor--daemon' start +'git fsmonitor--daemon' run +'git fsmonitor--daemon' stop +'git fsmonitor--daemon' status + +DESCRIPTION +----------- + +A daemon to watch the working directory for file and directory +changes using platform-specific file system notification facilities. + +This daemon communicates directly with commands like `git status` +using the link:technical/api-simple-ipc.html[simple IPC] interface +instead of the slower linkgit:githooks[5] interface. + +This daemon is built into Git so that no third-party tools are +required. + +OPTIONS +------- + +start:: + Starts a daemon in the background. + +run:: + Runs a daemon in the foreground. + +stop:: + Stops the daemon running in the current working + directory, if present. + +status:: + Exits with zero status if a daemon is watching the + current working directory. + +REMARKS +------- + +This daemon is a long running process used to watch a single working +directory and maintain a list of the recently changed files and +directories. Performance of commands such as `git status` can be +increased if they just ask for a summary of changes to the working +directory and can avoid scanning the disk. + +When `core.useBuiltinFSMonitor` is set to `true` (see +linkgit:git-config[1]) commands, such as `git status`, will ask the +daemon for changes and automatically start it (if necessary). + +For more information see the "File System Monitor" section in +linkgit:git-update-index[1]. + +CAVEATS +------- + +The fsmonitor daemon does not currently know about submodules and does +not know to filter out file system events that happen within a +submodule. If fsmonitor daemon is watching a super repo and a file is +modified within the working directory of a submodule, it will report +the change (as happening against the super repo). However, the client +will properly ignore these extra events, so performance may be affected +but it will not cause an incorrect result. + +GIT +--- +Part of the linkgit:git[1] suite
Signature · 1 line
gitgitgadget
Re: [PATCH v3 02/34] fsmonitor--daemon: man page
Quoted 81 lines · Jeff Hostetler via GitGitGadget
From: Jeff Hostetler <[email protected]>
Create a manual page describing the
git fsmonitor--daemonfeature.Signed-off-by Jeff Hostetler <[email protected]>--- Documentation/git-fsmonitor--daemon.txt | 75 +++++++++++++++++++++++++ 1 file changed, 75 insertions(+) create mode 100644 Documentation/git-fsmonitor--daemon.txtDocumentation/git-fsmonitor--daemon.txt new file +65
-0@@ -0,0 +1,75 @@ +git-fsmonitor--daemon(1) +======================== + +NAME +---- +git-fsmonitor--daemon - A Built-in File System Monitor + +SYNOPSIS +-------- +[verse] +'git fsmonitor--daemon' start +'git fsmonitor--daemon' run +'git fsmonitor--daemon' stop +'git fsmonitor--daemon' status + +DESCRIPTION +----------- + +A daemon to watch the working directory for file and directory +changes using platform-specific file system notification facilities. + +This daemon communicates directly with commands like `git status` +using the link:technical/api-simple-ipc.html[simple IPC] interface +instead of the slower linkgit:githooks[5] interface. + +This daemon is built into Git so that no third-party tools are +required. + +OPTIONS +------- + +start:: + Starts a daemon in the background. + +run:: + Runs a daemon in the foreground. + +stop:: + Stops the daemon running in the current working + directory, if present. + +status:: + Exits with zero status if a daemon is watching the + current working directory. + +REMARKS +------- + +This daemon is a long running process used to watch a single working +directory and maintain a list of the recently changed files and +directories. Performance of commands such as `git status` can be +increased if they just ask for a summary of changes to the working +directory and can avoid scanning the disk. + +When `core.useBuiltinFSMonitor` is set to `true` (see +linkgit:git-config[1]) commands, such as `git status`, will ask the +daemon for changes and automatically start it (if necessary). + +For more information see the "File System Monitor" section in +linkgit:git-update-index[1]. + +CAVEATS +------- + +The fsmonitor daemon does not currently know about submodules and does
Documentation/git-fsmonitor--daemon.txt line 75
Quoted 10 lines
Documentation/git-fsmonitor--daemon.txt new file +10
-0+not know to filter out file system events that happen within a +submodule. If fsmonitor daemon is watching a super repo and a file is +modified within the working directory of a submodule, it will report +the change (as happening against the super repo). However, the client +will properly ignore these extra events, so performance may be affected +but it will not cause an incorrect result. + +GIT +--- +Part of the linkgit:git[1] suite
Later in the series we incrementally add features to the daemon, so this is describing a state that doesn't exist yet at this point.
I think it would be better to start with a stup here and add documentation as we add features, e.g. the patch tha adds "start" should add that to the synopsis + options etc.
See the outstanding ab/config-based-hooks-base for a small example of that.
Re: [PATCH v3 02/34] fsmonitor--daemon: man page
Hi Ævar,
Quoted 97 lines · Ævar Arnfjörð Bjarmason
Quoted 81 lines · Jeff Hostetler via GitGitGadget
From: Jeff Hostetler <[email protected]>
Create a manual page describing the
git fsmonitor--daemonfeature.Signed-off-by Jeff Hostetler <[email protected]>--- Documentation/git-fsmonitor--daemon.txt | 75 +++++++++++++++++++++++++ 1 file changed, 75 insertions(+) create mode 100644 Documentation/git-fsmonitor--daemon.txtDocumentation/git-fsmonitor--daemon.txt new file +65
-0@@ -0,0 +1,75 @@ +git-fsmonitor--daemon(1) +======================== + +NAME +---- +git-fsmonitor--daemon - A Built-in File System Monitor + +SYNOPSIS +-------- +[verse] +'git fsmonitor--daemon' start +'git fsmonitor--daemon' run +'git fsmonitor--daemon' stop +'git fsmonitor--daemon' status + +DESCRIPTION +----------- + +A daemon to watch the working directory for file and directory +changes using platform-specific file system notification facilities. + +This daemon communicates directly with commands like `git status` +using the link:technical/api-simple-ipc.html[simple IPC] interface +instead of the slower linkgit:githooks[5] interface. + +This daemon is built into Git so that no third-party tools are +required. + +OPTIONS +------- + +start:: + Starts a daemon in the background. + +run:: + Runs a daemon in the foreground. + +stop:: + Stops the daemon running in the current working + directory, if present. + +status:: + Exits with zero status if a daemon is watching the + current working directory. + +REMARKS +------- + +This daemon is a long running process used to watch a single working +directory and maintain a list of the recently changed files and +directories. Performance of commands such as `git status` can be +increased if they just ask for a summary of changes to the working +directory and can avoid scanning the disk. + +When `core.useBuiltinFSMonitor` is set to `true` (see +linkgit:git-config[1]) commands, such as `git status`, will ask the +daemon for changes and automatically start it (if necessary). + +For more information see the "File System Monitor" section in +linkgit:git-update-index[1]. + +CAVEATS +------- + +The fsmonitor daemon does not currently know about submodules and doesDocumentation/git-fsmonitor--daemon.txt line 75
Quoted 10 lines
Documentation/git-fsmonitor--daemon.txt new file +10
-0+not know to filter out file system events that happen within a +submodule. If fsmonitor daemon is watching a super repo and a file is +modified within the working directory of a submodule, it will report +the change (as happening against the super repo). However, the client +will properly ignore these extra events, so performance may be affected +but it will not cause an incorrect result. + +GIT +--- +Part of the linkgit:git[1] suiteLater in the series we incrementally add features to the daemon, so this is describing a state that doesn't exist yet at this point.
Quoted 3 lines
I think it would be better to start with a stup here and add documentation as we add features, e.g. the patch tha adds "start" should add that to the synopsis + options etc.
Incidentally, I had structured the patch series that way until Jeff took over from me last year. And it was definitely not more reviewable because it was not clear from the get-go what I intended this command to do.
Therefore, I object to your suggestion.
Ciao,
Johannes
P.S.: I will not address your other reviews in this patch series, mostly because I am technically off from work this week.
Re: [PATCH v3 02/34] fsmonitor--daemon: man page
In reply to Ævar Arnfjörð BjarmasonQuoted 98 lines · Ævar Arnfjörð Bjarmason
Quoted 81 lines · Jeff Hostetler via GitGitGadget
From: Jeff Hostetler <[email protected]>
Create a manual page describing the
git fsmonitor--daemonfeature.Signed-off-by Jeff Hostetler <[email protected]>--- Documentation/git-fsmonitor--daemon.txt | 75 +++++++++++++++++++++++++ 1 file changed, 75 insertions(+)create mode 100644
Documentation/git-fsmonitor--daemon.txtDocumentation/git-fsmonitor--daemon.txt new file +65
-0@@ -0,0 +1,75 @@ +git-fsmonitor--daemon(1) +======================== + +NAME +---- +git-fsmonitor--daemon - A Built-in File System Monitor + +SYNOPSIS +-------- +[verse] +'git fsmonitor--daemon' start +'git fsmonitor--daemon' run +'git fsmonitor--daemon' stop +'git fsmonitor--daemon' status + +DESCRIPTION +----------- + +A daemon to watch the working directory for file and directory +changes using platform-specific file system notification facilities. + +This daemon communicates directly with commands like `git status` +using the link:technical/api-simple-ipc.html[simple IPC] interface +instead of the slower linkgit:githooks[5] interface. + +This daemon is built into Git so that no third-party tools are +required. + +OPTIONS +------- + +start:: + Starts a daemon in the background. + +run:: + Runs a daemon in the foreground. + +stop:: + Stops the daemon running in the current working + directory, if present. + +status:: + Exits with zero status if a daemon is watching the + current working directory. + +REMARKS +------- + +This daemon is a long running process used to watch a single working +directory and maintain a list of the recently changed files and +directories. Performance of commands such as `git status` can be +increased if they just ask for a summary of changes to the working +directory and can avoid scanning the disk. + +When `core.useBuiltinFSMonitor` is set to `true` (see +linkgit:git-config[1]) commands, such as `git status`, will ask the +daemon for changes and automatically start it (if necessary). + +For more information see the "File System Monitor" section in +linkgit:git-update-index[1]. + +CAVEATS +------- + +The fsmonitor daemon does not currently know about submodules and doesDocumentation/git-fsmonitor--daemon.txt line 75
Quoted 10 lines
Documentation/git-fsmonitor--daemon.txt new file +10
-0+not know to filter out file system events that happen within a +submodule. If fsmonitor daemon is watching a super repo and a file is +modified within the working directory of a submodule, it will report +the change (as happening against the super repo). However, the client +will properly ignore these extra events, so performance may be affected +but it will not cause an incorrect result. + +GIT +--- +Part of the linkgit:git[1] suiteLater in the series we incrementally add features to the daemon, so this is describing a state that doesn't exist yet at this point.
Quoted 7 lines
I think it would be better to start with a stup here and add documentation as we add features, e.g. the patch tha adds "start" should add that to the synopsis + options etc.
See the outstanding ab/config-based-hooks-base for a small example of that.
I like to lead the series with the documentation that summarizes the purpose of the entire feature or patch series. This gives the reviewer the context for the complete series that follows. In the past, we've had discussions on the list about how hard it is to review a series when the foo.c comes (alphabetically) before foo.h in the patch and all the documentation is attached to the prototypes in the .h file so the reviewer needs to bounce around in the patch or series to read the intent and then go back to the beginning to read the code. In that spirit, I think that having the complete man page come first provides necessary context and is helpful.
The argument that the man-page should grow as the feature grows presumes that there is a meaningful cut-point mid-series where you would adopt the first portion and delay the second to a later release or something. That division would not be useful/usable.
And it just clutters up later commits in the series with man-page deltas.
So I'd like to keep it as it unless there are further objections.
Jeff
Re: [PATCH v3 02/34] fsmonitor--daemon: man page
Quoted 116 lines · Jeff Hostetler
Quoted 94 lines · Ævar Arnfjörð Bjarmason
Quoted 81 lines · Jeff Hostetler via GitGitGadget
From: Jeff Hostetler <[email protected]>
Create a manual page describing the
git fsmonitor--daemonfeature.Signed-off-by Jeff Hostetler <[email protected]>--- Documentation/git-fsmonitor--daemon.txt | 75 +++++++++++++++++++++++++ 1 file changed, 75 insertions(+)create mode 100644
Documentation/git-fsmonitor--daemon.txtDocumentation/git-fsmonitor--daemon.txt new file +65
-0@@ -0,0 +1,75 @@ +git-fsmonitor--daemon(1) +======================== + +NAME +---- +git-fsmonitor--daemon - A Built-in File System Monitor + +SYNOPSIS +-------- +[verse] +'git fsmonitor--daemon' start +'git fsmonitor--daemon' run +'git fsmonitor--daemon' stop +'git fsmonitor--daemon' status + +DESCRIPTION +----------- + +A daemon to watch the working directory for file and directory +changes using platform-specific file system notification facilities. + +This daemon communicates directly with commands like `git status` +using the link:technical/api-simple-ipc.html[simple IPC] interface +instead of the slower linkgit:githooks[5] interface. + +This daemon is built into Git so that no third-party tools are +required. + +OPTIONS +------- + +start:: + Starts a daemon in the background. + +run:: + Runs a daemon in the foreground. + +stop:: + Stops the daemon running in the current working + directory, if present. + +status:: + Exits with zero status if a daemon is watching the + current working directory. + +REMARKS +------- + +This daemon is a long running process used to watch a single working +directory and maintain a list of the recently changed files and +directories. Performance of commands such as `git status` can be +increased if they just ask for a summary of changes to the working +directory and can avoid scanning the disk. + +When `core.useBuiltinFSMonitor` is set to `true` (see +linkgit:git-config[1]) commands, such as `git status`, will ask the +daemon for changes and automatically start it (if necessary). + +For more information see the "File System Monitor" section in +linkgit:git-update-index[1]. + +CAVEATS +------- + +The fsmonitor daemon does not currently know about submodules and doesDocumentation/git-fsmonitor--daemon.txt line 75
Quoted 10 lines
Documentation/git-fsmonitor--daemon.txt new file +10
-0+not know to filter out file system events that happen within a +submodule. If fsmonitor daemon is watching a super repo and a file is +modified within the working directory of a submodule, it will report +the change (as happening against the super repo). However, the client +will properly ignore these extra events, so performance may be affected +but it will not cause an incorrect result. + +GIT +--- +Part of the linkgit:git[1] suiteLater in the series we incrementally add features to the daemon, so
Quoted 9 lines
this
is describing a state that doesn't exist yet at this point. I think it would be better to start with a stup here and add documentation as we add features, e.g. the patch tha adds "start" should add that to the synopsis + options etc.
See the outstanding ab/config-based-hooks-base for a small example of
that.I like to lead the series with the documentation that summarizes the purpose of the entire feature or patch series. This gives the reviewer the context for the complete series that follows. In the past, we've had discussions on the list about how hard it is to review a series when the
foo.ccomes (alphabetically) beforefoo.hin the patch and all the documentation is attached to the prototypes in the .h file so the reviewer needs to bounce around in the patch or series to read the intent and then go back to the beginning to read the code. In that spirit, I think that having the complete man page come first provides necessary context and is helpful.
Quoted 4 lines
The argument that the man-page should grow as the feature grows presumes that there is a meaningful cut-point mid-series where you would adopt the first portion and delay the second to a later release or something. That division would not be useful/usable.
Isn't there such a meaningful cut-off point?
In 08/34[1] you add a skeleton of the daemon, so then the NAME/SYNOPSIS (empty)/DESCRIPTION/REMARKS/GIT sections could be added, in 09/34[3] you add stop/status commands, so then the SYNOPSIS/OPTIONS could be updated/created with those, same for the addition of run/start in 13/34[5] and 14/35[6] respectively.
Quoted 4 lines
And it just clutters up later commits in the series with man-page deltas.
So I'd like to keep it as it unless there are further objections.
I'll leave it to you to do with the feedback as you choose,
I suggested this because I think it's much easier to read patches that are larger because they incrementally update docs or tests along with code, than smaller ones that are e.g. "add docs" followed by incrementally modifying the code.
That's because you can consider those atomically. If earlier doc changes refer to later code changes you're left jumping back & forth and wondering if the code you're reading that doesn't match the docs yet is a bug, or if it's solved in some later change you're now needing to mentally keep track of.
Which is not some theoretical concern b.t.w., but exactly what I found myself doing when reading this series, hence the suggestion.
1. https://lore.kernel.org/git/f88db92d4259d1c29827e97e957daf6eda39c551.1625150864.git.gitgitgadget@gmail.com/
2. https://lore.kernel.org/git/[email protected]/
3. https://lore.kernel.org/git/02e21384ef0ca4909e0bda2c78fa63c06be22a50.1625150864.git.gitgitgadget@gmail.com/
4. https://lore.kernel.org/git/5d6646df93a17659af66f136295444d1bd834090.1625150864.git.gitgitgadget@gmail.com/
5. https://lore.kernel.org/git/5d6646df93a17659af66f136295444d1bd834090.1625150864.git.gitgitgadget@gmail.com/
6. https://lore.kernel.org/git/9fe902aad87f1192705fb69ea212a2d066d0286d.1625150864.git.gitgitgadget@gmail.com/
Re: [PATCH v3 02/34] fsmonitor--daemon: man page
Hi Ævar,
Quoted 8 lines · Ævar Arnfjörð Bjarmason
[snip Ævar's suggestion to populate the manual page incrementally, interspersed with the commits that finalize implementing the respective functionality]
I suggested this because I think it's much easier to read patches that are larger because they incrementally update docs or tests along with code, than smaller ones that are e.g. "add docs" followed by incrementally modifying the code.
My experience is the exact opposite of yours: shorter patches are easier to read.
Quoted 1 line
That's because you can consider those atomically.
No, in a patch series you cannot consider any patch completely atomically. Just like you don't consider any paragraph in any well-written book out of context.
Quoted 4 lines
If earlier doc changes refer to later code changes you're left jumping back & forth and wondering if the code you're reading that doesn't match the docs yet is a bug, or if it's solved in some later change you're now needing to mentally keep track of.
You only keep jumping back and forth when reviewing _patches_. We try to do code review on this mailing list, which means that you have the code locally and review it in the correct context. When you do that, a design document is quite helpful. And the proposed manual page serves as such a design document.
Quoted 2 lines
Which is not some theoretical concern b.t.w., but exactly what I found myself doing when reading this series, hence the suggestion.
Part of the problem here seems to be that this patch series saw many reviewer suggestions that forced it to increase in length. That was probably not very helpful, after all.
The first iteration had 23 patches. Reviews forced it to grow to 28 patches in the second iteration. And that was still not enough, therefore the third iteration consisted of 34 patches. And if you had had your way, requiring Jeff to include a Linux backend in the same patch series, it would have to increase in size again, and not just by a little.
This all sounds like we're truly falling into the trap of ignoring the rule that the perfect is the enemy of the good.
I really would like to come back to a focused review that truly improves the patches at hand, and avoids conflating the review of the actual patch series with matters of personal taste (which is a recipe for disagreement). After all, we are interested in getting this feature out to users who need to work with very large worktrees, right? At least that's my goal here.
Ciao,
Johannes
Re: [PATCH v3 02/34] fsmonitor--daemon: man page
Quoted 6 lines · Johannes Schindelin
Quoted 4 lines · Ævar Arnfjörð Bjarmason
[snip Ævar's suggestion to populate the manual page incrementally, interspersed with the commits that finalize implementing the respective functionality]
Quoted 7 lines
Quoted 4 lines
I suggested this because I think it's much easier to read patches that are larger because they incrementally update docs or tests along with code, than smaller ones that are e.g. "add docs" followed by incrementally modifying the code.
My experience is the exact opposite of yours: shorter patches are easier to read.
It depends.
Quoted 5 lines
Quoted 1 line
That's because you can consider those atomically.
No, in a patch series you cannot consider any patch completely atomically. Just like you don't consider any paragraph in any well-written book out of context.
But you do not put every sentence in a paragraph.
Sometimes a paragraph can contain a single sentence, or a single word even. But other times to properly read what is being tried to say you need a pretty big paragraph.
Quoted 2 lines
This all sounds like we're truly falling into the trap of ignoring the rule that the perfect is the enemy of the good.
Have you established that this is good enough?
Signature · 1 line
Felipe Contreras
PATCH v3 3/34 fsmonitor--daemon: update fsmonitor documentation
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Update references to core.fsmonitor and core.fsmonitorHookVersion and pointers to Watchman to mention the new built-in fsmonitor--daemon.
--- Documentation/config/core.txt | 56 ++++++++++++++++++++++-------- Documentation/git-update-index.txt | 27 +++++++------- Documentation/githooks.txt | 3 +- 3 files changed, 59 insertions(+), 27 deletions(-)
Documentation/config/core.txt +42 -14
@@ -62,22 +62,50 @@ core.protectNTFS:: Defaults to `true` on Windows, and `false` elsewhere. core.fsmonitor:: - If set, the value of this variable is used as a command which - will identify all files that may have changed since the - requested date/time. This information is used to speed up git by - avoiding unnecessary processing of files that have not changed. - See the "fsmonitor-watchman" section of linkgit:githooks[5]. + If set, this variable contains the pathname of the "fsmonitor" + hook command. ++ +This hook command is used to identify all files that may have changed +since the requested date/time. This information is used to speed up +git by avoiding unnecessary scanning of files that have not changed. ++ +See the "fsmonitor-watchman" section of linkgit:githooks[5]. ++ +Note: The value of this config setting is ignored if the +built-in file system monitor is enabled (see `core.useBuiltinFSMonitor`). core.fsmonitorHookVersion:: - Sets the version of hook that is to be used when calling fsmonitor. - There are currently versions 1 and 2. When this is not set, - version 2 will be tried first and if it fails then version 1 - will be tried. Version 1 uses a timestamp as input to determine - which files have changes since that time but some monitors - like watchman have race conditions when used with a timestamp. - Version 2 uses an opaque string so that the monitor can return - something that can be used to determine what files have changed - without race conditions. + Sets the protocol version to be used when invoking the + "fsmonitor" hook. ++ +There are currently versions 1 and 2. When this is not set, +version 2 will be tried first and if it fails then version 1 +will be tried. Version 1 uses a timestamp as input to determine +which files have changes since that time but some monitors +like Watchman have race conditions when used with a timestamp. +Version 2 uses an opaque string so that the monitor can return +something that can be used to determine what files have changed +without race conditions. ++ +Note: The value of this config setting is ignored if the +built-in file system monitor is enabled (see `core.useBuiltinFSMonitor`). + +core.useBuiltinFSMonitor:: + If set to true, enable the built-in file system monitor + daemon for this working directory (linkgit:git-fsmonitor--daemon[1]). ++ +Like hook-based file system monitors, the built-in file system monitor +can speed up Git commands that need to refresh the Git index +(e.g. `git status`) in a working directory with many files. The +built-in monitor eliminates the need to install and maintain an +external third-party tool. ++ +The built-in file system monitor is currently available only on a +limited set of supported platforms. Currently, this includes Windows +and MacOS. ++ +Note: if this config setting is set to `true`, the values of +`core.fsmonitor` and `core.fsmonitorHookVersion` are ignored. core.trustctime:: If false, the ctime differences between the index and the
Documentation/git-update-index.txt +15 -12
@@ -498,7 +498,9 @@ FILE SYSTEM MONITOR This feature is intended to speed up git operations for repos that have large working directories. -It enables git to work together with a file system monitor (see the +It enables git to work together with a file system monitor (see +linkgit:git-fsmonitor--daemon[1] +and the "fsmonitor-watchman" section of linkgit:githooks[5]) that can inform it as to what files have been modified. This enables git to avoid having to lstat() every file to find modified files.
@@ -508,17 +510,18 @@ performance by avoiding the cost of scanning the entire working directory looking for new files. If you want to enable (or disable) this feature, it is easier to use -the `core.fsmonitor` configuration variable (see -linkgit:git-config[1]) than using the `--fsmonitor` option to -`git update-index` in each repository, especially if you want to do so -across all repositories you use, because you can set the configuration -variable in your `$HOME/.gitconfig` just once and have it affect all -repositories you touch. - -When the `core.fsmonitor` configuration variable is changed, the -file system monitor is added to or removed from the index the next time -a command reads the index. When `--[no-]fsmonitor` are used, the file -system monitor is immediately added to or removed from the index. +the `core.fsmonitor` or `core.useBuiltinFSMonitor` configuration +variable (see linkgit:git-config[1]) than using the `--fsmonitor` +option to `git update-index` in each repository, especially if you +want to do so across all repositories you use, because you can set the +configuration variable in your `$HOME/.gitconfig` just once and have +it affect all repositories you touch. + +When the `core.fsmonitor` or `core.useBuiltinFSMonitor` configuration +variable is changed, the file system monitor is added to or removed +from the index the next time a command reads the index. When +`--[no-]fsmonitor` are used, the file system monitor is immediately +added to or removed from the index. CONFIGURATION -------------
Documentation/githooks.txt +2 -1
@@ -593,7 +593,8 @@ fsmonitor-watchman This hook is invoked when the configuration option `core.fsmonitor` is set to `.git/hooks/fsmonitor-watchman` or `.git/hooks/fsmonitor-watchmanv2` -depending on the version of the hook to use. +depending on the version of the hook to use, unless overridden via +`core.useBuiltinFSMonitor` (see linkgit:git-config[1]). Version 1 takes two arguments, a version (1) and the time in elapsed nanoseconds since midnight, January 1, 1970.
Signature · 1 line
gitgitgadget
Re: [PATCH v3 03/34] fsmonitor--daemon: update fsmonitor documentation
Quoted 131 lines · Jeff Hostetler via GitGitGadget
From: Jeff Hostetler <[email protected]>
Update references to
core.fsmonitorandcore.fsmonitorHookVersionand pointers toWatchmanto mention the new built-infsmonitor--daemon.Signed-off-by Jeff Hostetler <[email protected]>--- Documentation/config/core.txt | 56 ++++++++++++++++++++++-------- Documentation/git-update-index.txt | 27 +++++++------- Documentation/githooks.txt | 3 +- 3 files changed, 59 insertions(+), 27 deletions(-)Documentation/config/core.txt +42
-14@@ -62,22 +62,50 @@ core.protectNTFS:: Defaults to `true` on Windows, and `false` elsewhere. core.fsmonitor:: - If set, the value of this variable is used as a command which - will identify all files that may have changed since the - requested date/time. This information is used to speed up git by - avoiding unnecessary processing of files that have not changed. - See the "fsmonitor-watchman" section of linkgit:githooks[5]. + If set, this variable contains the pathname of the "fsmonitor" + hook command. ++ +This hook command is used to identify all files that may have changed +since the requested date/time. This information is used to speed up +git by avoiding unnecessary scanning of files that have not changed. ++ +See the "fsmonitor-watchman" section of linkgit:githooks[5]. ++ +Note: The value of this config setting is ignored if the +built-in file system monitor is enabled (see `core.useBuiltinFSMonitor`). core.fsmonitorHookVersion:: - Sets the version of hook that is to be used when calling fsmonitor. - There are currently versions 1 and 2. When this is not set, - version 2 will be tried first and if it fails then version 1 - will be tried. Version 1 uses a timestamp as input to determine - which files have changes since that time but some monitors - like watchman have race conditions when used with a timestamp. - Version 2 uses an opaque string so that the monitor can return - something that can be used to determine what files have changed - without race conditions. + Sets the protocol version to be used when invoking the + "fsmonitor" hook. ++ +There are currently versions 1 and 2. When this is not set, +version 2 will be tried first and if it fails then version 1 +will be tried. Version 1 uses a timestamp as input to determine +which files have changes since that time but some monitors +like Watchman have race conditions when used with a timestamp. +Version 2 uses an opaque string so that the monitor can return +something that can be used to determine what files have changed +without race conditions. ++ +Note: The value of this config setting is ignored if the +built-in file system monitor is enabled (see `core.useBuiltinFSMonitor`). + +core.useBuiltinFSMonitor:: + If set to true, enable the built-in file system monitor + daemon for this working directory (linkgit:git-fsmonitor--daemon[1]). ++ +Like hook-based file system monitors, the built-in file system monitor +can speed up Git commands that need to refresh the Git index +(e.g. `git status`) in a working directory with many files. The +built-in monitor eliminates the need to install and maintain an +external third-party tool. ++ +The built-in file system monitor is currently available only on a +limited set of supported platforms. Currently, this includes Windows +and MacOS. ++ +Note: if this config setting is set to `true`, the values of +`core.fsmonitor` and `core.fsmonitorHookVersion` are ignored. core.trustctime:: If false, the ctime differences between the index and theDocumentation/git-update-index.txt +15
-12@@ -498,7 +498,9 @@ FILE SYSTEM MONITOR This feature is intended to speed up git operations for repos that have large working directories. -It enables git to work together with a file system monitor (see the +It enables git to work together with a file system monitor (see +linkgit:git-fsmonitor--daemon[1] +and the "fsmonitor-watchman" section of linkgit:githooks[5]) that can inform it as to what files have been modified. This enables git to avoid having to lstat() every file to find modified files.@@ -508,17 +510,18 @@ performance by avoiding the cost of scanning the entire working directory looking for new files. If you want to enable (or disable) this feature, it is easier to use -the `core.fsmonitor` configuration variable (see -linkgit:git-config[1]) than using the `--fsmonitor` option to -`git update-index` in each repository, especially if you want to do so -across all repositories you use, because you can set the configuration -variable in your `$HOME/.gitconfig` just once and have it affect all -repositories you touch. - -When the `core.fsmonitor` configuration variable is changed, the -file system monitor is added to or removed from the index the next time -a command reads the index. When `--[no-]fsmonitor` are used, the file -system monitor is immediately added to or removed from the index. +the `core.fsmonitor` or `core.useBuiltinFSMonitor` configuration +variable (see linkgit:git-config[1]) than using the `--fsmonitor` +option to `git update-index` in each repository, especially if you +want to do so across all repositories you use, because you can set the +configuration variable in your `$HOME/.gitconfig` just once and have +it affect all repositories you touch. + +When the `core.fsmonitor` or `core.useBuiltinFSMonitor` configuration +variable is changed, the file system monitor is added to or removed +from the index the next time a command reads the index. When +`--[no-]fsmonitor` are used, the file system monitor is immediately +added to or removed from the index. CONFIGURATION -------------Documentation/githooks.txt +0
-0
Documentation/githooks.txt fsmonitor-watchman · line 600
Quoted 9 lines
Documentation/githooks.txt +2
-1@@ -593,7 +593,8 @@ fsmonitor-watchman This hook is invoked when the configuration option `core.fsmonitor` is set to `.git/hooks/fsmonitor-watchman` or `.git/hooks/fsmonitor-watchmanv2` -depending on the version of the hook to use. +depending on the version of the hook to use, unless overridden via +`core.useBuiltinFSMonitor` (see linkgit:git-config[1]). Version 1 takes two arguments, a version (1) and the time in elapsed nanoseconds since midnight, January 1, 1970.
Ditto the comment on 02/34, mostly docs for things that don't exist until in later patches.
PATCH v3 4/34 fsmonitor-ipc: create client routines for git-fsmonitor--daemon
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Create fsmonitor_ipc__*() client routines to spawn the built-in file system monitor daemon and send it an IPC request using the Simple IPC API.
Stub in empty fsmonitor_ipc__*() functions for unsupported platforms.
--- Makefile | 1 + fsmonitor-ipc.c | 179 ++++++++++++++++++++++++++++++++++++++++++++++++ fsmonitor-ipc.h | 48 +++++++++++++ 3 files changed, 228 insertions(+) create mode 100644 fsmonitor-ipc.c create mode 100644 fsmonitor-ipc.h
Makefile +1 -0
@@ -893,6 +893,7 @@ LIB_OBJS += fetch-pack.o LIB_OBJS += fmt-merge-msg.o LIB_OBJS += fsck.o LIB_OBJS += fsmonitor.o +LIB_OBJS += fsmonitor-ipc.o LIB_OBJS += gettext.o LIB_OBJS += gpg-interface.o LIB_OBJS += graph.o
fsmonitor-ipc.c new file +179 -0
179 lines not on this page · raw message
fsmonitor-ipc.h new file +48 -0
48 lines not on this page · raw message
Signature · 1 line
gitgitgadget
PATCH v3 5/34 help: include fsmonitor--daemon feature flag in version info
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Add the "feature: fsmonitor--daemon" message to the output of git version --build-options.
This allows users to know if the built-in fsmonitor feature is supported on their platform.
--- help.c | 4 ++++ t/test-lib.sh | 6 ++++++ 2 files changed, 10 insertions(+)
help.c +4 -0
@@ -11,6 +11,7 @@#include "version.h" #include "refs.h" #include "parse-options.h" +#include "fsmonitor-ipc.h" struct category_description {uint32_t category;
@@ -664,6 +665,9 @@ void get_version_info(struct strbuf *buf, int show_build_options)strbuf_addf (buf,"sizeof-size_t: %d\n" , (int )sizeof (size_t ));strbuf_addf (buf,"shell-path: %s\n" ,SHELL_PATH );/* NEEDSWORK: also save and output GIT-BUILD_OPTIONS? */ + +if (fsmonitor_ipc__is_supported ()) +strbuf_addstr (buf,"feature: fsmonitor--daemon\n" ); } }
t/test-lib.sh +6 -0
@@ -1674,3 +1674,9 @@ test_lazy_prereq REBASE_P ' # Tests that verify the scheduler integration must set this locally # to avoid errors. GIT_TEST_MAINT_SCHEDULER="none:exit 1" + +# Does this platform support `git fsmonitor--daemon` +# +test_lazy_prereq FSMONITOR_DAEMON ' + git version --build-options | grep "feature:" | grep "fsmonitor--daemon" +'
Signature · 1 line
gitgitgadget
PATCH v3 6/34 fsmonitor: config settings are repository-specific
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Move FSMonitor config settings to struct repo_settings, get rid of the core_fsmonitor global variable, and add support for the new core.useBuiltinFSMonitor config setting. Move config code to lookup core.fsmonitor into prepare_repo_settings() with the rest of the setup code.
The core_fsmonitor global variable was used to store the pathname to the FSMonitor hook and it was used as a boolean to see if FSMonitor was enabled. This dual usage will lead to confusion when we add support for a builtin FSMonitor based on IPC, since the builtin FSMonitor doesn't need the hook pathname.
Replace the boolean usage with an enum fsmonitor_mode to represent the state of FSMonitor. And only set the pathname when in HOOK mode.
Also, disable FSMonitor when the repository working directory is incompatible. For example, in bare repositories, since there aren't any files to watch.
--- builtin/update-index.c | 20 +++++++++--- cache.h | 1 - config.c | 14 -------- config.h | 1 - environment.c | 1 - fsmonitor.c | 65 +++++++++++++++++++++++-------------- fsmonitor.h | 14 ++++++-- repo-settings.c | 48 +++++++++++++++++++++++++++ repository.h | 11 +++++++ t/README | 4 +-- t/t7519-status-fsmonitor.sh | 22 +++++++++++++ 11 files changed, 150 insertions(+), 51 deletions(-)
builtin/update-index.c +16 -4
@@ -1216,14 +1216,26 @@ int cmd_update_index(int argc, const char **argv, const char *prefix) }if (fsmonitor >0 ) { -if (git_config_get_fsmonitor () ==0 ) +if (r->settings.fsmonitor_mode ==FSMONITOR_MODE_INCOMPATIBLE ) +return error ( +_ ("repository is incompatible with fsmonitor" )); + +if (r->settings.fsmonitor_mode ==FSMONITOR_MODE_DISABLED ) { +warning (_ ("core.useBuiltinFSMonitor is unset; " +"set it if you really want to enable the " +"builtin fsmonitor" ));warning (_ ("core.fsmonitor is unset; " -"set it if you really want to " -"enable fsmonitor" )); +"set it if you really want to enable the " +"hook-based fsmonitor" )); + }add_fsmonitor (&the_index);report (_ ("fsmonitor enabled" )); }else if (!fsmonitor) { -if (git_config_get_fsmonitor () ==1 ) +if (r->settings.fsmonitor_mode ==FSMONITOR_MODE_IPC ) +warning (_ ("core.useBuiltinFSMonitor is set; " +"remove it if you really want to " +"disable fsmonitor" )); +if (r->settings.fsmonitor_mode ==FSMONITOR_MODE_HOOK )warning (_ ("core.fsmonitor is set; " "remove it if you really want to " "disable fsmonitor" ));
cache.h +0 -1
@@ -981,7 +981,6 @@ extern int core_preload_index;extern int precomposed_unicode;extern int protect_hfs;extern int protect_ntfs; -extern const char *core_fsmonitor;extern int core_apply_sparse_checkout;extern int core_sparse_checkout_cone;
config.c +0 -14
@@ -2516,20 +2516,6 @@ int git_config_get_max_percent_split_change(void)return -1 ;/* default value */ } -int git_config_get_fsmonitor (void ) -{ -if (git_config_get_pathname ("core.fsmonitor" , &core_fsmonitor)) - core_fsmonitor =getenv ("GIT_TEST_FSMONITOR" ); - -if (core_fsmonitor && !*core_fsmonitor) - core_fsmonitor =NULL ; - -if (core_fsmonitor) -return 1 ; - -return 0 ; -} -int git_config_get_index_threads (int *dest) {int is_bool, val;
config.h +0 -1
@@ -609,7 +609,6 @@ int git_config_get_index_threads(int *dest);int git_config_get_untracked_cache (void );int git_config_get_split_index (void );int git_config_get_max_percent_split_change (void ); -int git_config_get_fsmonitor (void );/* This dies if the configured or default date is in the future */ int git_config_get_expiry (const char *key,const char **output);
environment.c +0 -1
@@ -84,7 +84,6 @@ int protect_hfs = PROTECT_HFS_DEFAULT;#define PROTECT_NTFS_DEFAULT 1#endif int protect_ntfs =PROTECT_NTFS_DEFAULT ; -const char *core_fsmonitor;/* * The character that begins a commented line in user-editable file
fsmonitor.c +40 -25
127 lines not on this page · raw message
fsmonitor.h +11 -3
@@ -57,7 +57,10 @@ int fsmonitor_is_trivial_response(const struct strbuf *query_result);*/ static inline int is_fsmonitor_refreshed (const struct index_state *istate) { -return !core_fsmonitor || istate->fsmonitor_has_run_once; +struct repository *r = istate->repo ? istate->repo : the_repository; + +return r->settings.fsmonitor_mode <=FSMONITOR_MODE_DISABLED || + istate->fsmonitor_has_run_once; }/*
@@ -67,7 +70,10 @@ static inline int is_fsmonitor_refreshed(const struct index_state *istate)*/ static inline void mark_fsmonitor_valid (struct index_state *istate,struct cache_entry *ce) { -if (core_fsmonitor && !(ce->ce_flags &CE_FSMONITOR_VALID )) { +struct repository *r = istate->repo ? istate->repo : the_repository; + +if (r->settings.fsmonitor_mode >FSMONITOR_MODE_DISABLED && + !(ce->ce_flags &CE_FSMONITOR_VALID )) { istate->cache_changed =1 ; ce->ce_flags |=CE_FSMONITOR_VALID ;trace_printf_key (&trace_fsmonitor,"mark_fsmonitor_clean '%s'" , ce->name);
@@ -83,7 +89,9 @@ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache*/ static inline void mark_fsmonitor_invalid (struct index_state *istate,struct cache_entry *ce) { -if (core_fsmonitor) { +struct repository *r = istate->repo ? istate->repo : the_repository; + +if (r->settings.fsmonitor_mode >FSMONITOR_MODE_DISABLED ) { ce->ce_flags &= ~CE_FSMONITOR_VALID ;untracked_cache_invalidate_path (istate, ce->name,1 );trace_printf_key (&trace_fsmonitor,"mark_fsmonitor_invalid '%s'" , ce->name);
repo-settings.c +48 -0
64 lines not on this page · raw message
repository.h +11 -0
@@ -26,6 +26,14 @@ enum fetch_negotiation_setting { FETCH_NEGOTIATION_NOOP = 3, }; +enum fsmonitor_mode { +FSMONITOR_MODE_INCOMPATIBLE =-2 , +FSMONITOR_MODE_UNSET =-1 , +FSMONITOR_MODE_DISABLED =0 , +FSMONITOR_MODE_HOOK =1 ,/* core.fsmonitor */ +FSMONITOR_MODE_IPC =2 ,/* core.useBuiltinFSMonitor */ +}; +struct repo_settings {int initialized;
@@ -34,6 +42,9 @@ struct repo_settings {int gc_write_commit_graph;int fetch_write_commit_graph; +enum fsmonitor_mode fsmonitor_mode; +char *fsmonitor_hook_path; +int index_version;enum untracked_cache_setting core_untracked_cache;
t/README +2 -2
@@ -398,8 +398,8 @@ every 'git commit-graph write', as if the `--changed-paths` option was passed in. GIT_TEST_FSMONITOR=$PWD/t7519/fsmonitor-all exercises the fsmonitor -code path for utilizing a file system monitor to speed up detecting -new or changed files. +code path for utilizing a (hook based) file system monitor to speed up +detecting new or changed files. GIT_TEST_INDEX_VERSION=<n> exercises the index read/write code path for the index version specified. Can be set to any valid version
t/t7519-status-fsmonitor.sh +22 -0
@@ -383,4 +383,26 @@ test_expect_success 'status succeeds after staging/unstaging' ' ) ' +# Test that we detect and disallow repos that are incompatible with FSMonitor. +test_expect_success 'incompatible bare repo' ' + test_when_finished "rm -rf ./bare-clone" && + git clone --bare . ./bare-clone && + cat >expect <<-\EOF && + error: repository is incompatible with fsmonitor + EOF + test_must_fail git -C ./bare-clone update-index --fsmonitor 2>actual && + test_cmp expect actual +' + +test_expect_success 'incompatible core.virtualfilesystem' ' + test_when_finished "rm -rf ./fake-gvfs-clone" && + git clone . ./fake-gvfs-clone && + git -C ./fake-gvfs-clone config core.virtualfilesystem true && + cat >expect <<-\EOF && + error: repository is incompatible with fsmonitor + EOF + test_must_fail git -C ./fake-gvfs-clone update-index --fsmonitor 2>actual && + test_cmp expect actual +' + test_done
Signature · 1 line
gitgitgadget
Re: [PATCH v3 06/34] fsmonitor: config settings are repository-specific
On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:
In a reference to a discussion[1] about an earlier version of this patch you said:
I'm going to ignore all of the thread responses to this patch dealing with how we acquire config settings and macros and etc. Those issues are completely independent of FSMonitor (which is already way too big).
Since then the changes to repo-settings.c have become a lot larger, so let's take a look...
1. https://lore.kernel.org/git/[email protected]/
2. https://lore.kernel.org/git/[email protected]/
Quoted 11 lines
repo-settings.c +3
-0@@ -5,10 +5,42 @@#define UPDATE_DEFAULT_BOOL (s,v) do { if (s == -1) { s = v; } } while(0) +/* +* Return 1 if the repo/workdir is incompatible with FSMonitor. +*/
repo-settings.c line 20
Quoted 10 lines
repo-settings.c +10
-0+static int is_repo_incompatible_with_fsmonitor (struct repository *r) +{ +const char *const_strval; + +/* +* Bare repositories don't have a working directory and +* therefore, nothing to watch. +*/ +if (!r->worktree) +return 1 ;
Looking ahead in this series you end up using FSMONITOR_MODE_INCOMPATIBLE in two places in the codebase. In builtin/update-index.c to throw a "repository is incompatible with fsmonitor" error.
Can't that case just be replaced with setup_work_tree()? Other sub-modes of update-index already die implicitly on that, e.g.:
$ git update-index test fatal: this operation must be run in a work tree
The other case is:
+ prepare_repo_settings(the_repository);
+ if (!the_repository->worktree)
+ return error(_("fsmonitor-daemon does not support bare repos '%s'"),
+ xgetcwd());
+ if (the_repository->settings.fsmonitor_mode == FSMONITOR_MODE_INCOMPATIBLE)
+ return error(_("fsmonitor-daemon is incompatible with this repo '%s'"),
+ the_repository->worktree);
I.e. we just checked the_repository->worktree, but it's not that, but....
Quoted 4 lines
repo-settings.c +4
-0+ +/* +* GVFS (aka VFS for Git) is incompatible with FSMonitor. +*
repo-settings.c line 34
Quoted 10 lines
repo-settings.c +10
-0+* Granted, core Git does not know anything about GVFS and +* we shouldn't make assumptions about a downstream feature, +* but users can install both versions. And this can lead +* to incorrect results from core Git commands. So, without +* bringing in any of the GVFS code, do a simple config test +* for a published config setting. (We do not look at the +* various *_TEST_* environment variables.) +*/ +if (!repo_config_get_value (r,"core.virtualfilesystem" , &const_strval)) +return 1 ;
I'm skeptical of us hardcoding a third-party software config variable. Can't GitVFS handle this somehow on its end?
But just in terms of implementation it seems the end result of that is to emit a very confusing error to the user. Sinc we already checked for bare repos we run into this and instead of sayingwhen we should really say "hey, maybe disable your core.virtualFileSystem setting", we say "your repo is incompatible".
repo-settings.c line 43
Quoted 9 lines
repo-settings.c +5
-0+ +return 0 ; +} +void prepare_repo_settings (struct repository *r) {int value;char *strval; +const char *const_strval;
Can be declared in the "else" below.
repo-settings.c void prepare_repo_settings(struct repository *r) · line 62
Quoted 9 lines
repo-settings.c +2
-0if (r->settings.initialized)return ;@@ -26,6 +58,22 @@ void prepare_repo_settings(struct repository *r)UPDATE_DEFAULT_BOOL (r->settings.commit_graph_read_changed_paths,1 );UPDATE_DEFAULT_BOOL (r->settings.gc_write_commit_graph,1 ); + r->settings.fsmonitor_hook_path =NULL ; + r->settings.fsmonitor_mode =FSMONITOR_MODE_DISABLED ;
With the memset earlier (b.t.w. I've got a patch to fix all this bizarre behavior in repo-settings.c, but have been waiting on this series we implicitly set it to FSMONITOR_MODE_UNSET (-1) with the memset, but then never use that ever.
Your code in update-index.c then for a check against "FSMONITOR_MODE_DISABLED" says "core.useBuiltinFSMonitor is unset;".
repo-settings.c void prepare_repo_settings(struct repository *r) · line 64
Quoted 2 lines
repo-settings.c +2
-0+if (is_repo_incompatible_with_fsmonitor (r)) + r->settings.fsmonitor_mode =FSMONITOR_MODE_INCOMPATIBLE ;
Style: should have {} braces on all arms.
repo-settings.c void prepare_repo_settings(struct repository *r) · line 67
Quoted 3 lines
repo-settings.c +3
-0+else if (!repo_config_get_bool (r,"core.usebuiltinfsmonitor" , &value) + && value) + r->settings.fsmonitor_mode =FSMONITOR_MODE_IPC ;
Here you're conflating false with whether the variable is set at all. I guess that works out here since if it's false we want to fall through to...
repo-settings.c void prepare_repo_settings(struct repository *r) · line 68
Quoted 1 line
repo-settings.c +1
-0+else {
...ignoring it and looing at core.fsmonitor instead.
repo-settings.c void prepare_repo_settings(struct repository *r) · line 70
Quoted 2 lines
repo-settings.c +2
-0+if (repo_config_get_pathname (r,"core.fsmonitor" , &const_strval)) + const_strval =getenv ("GIT_TEST_FSMONITOR" );
If it's not set we pay attention to GIT_TEST_FSMONITOR, so the behavior from the old git_config_get_fsmonitor(). So even if the env variable is set we want to take the config variable over it, correct?
repo-settings.c void prepare_repo_settings(struct repository *r) · line 72
Quoted 2 lines
repo-settings.c +2
-0+if (const_strval && *const_strval) { + r->settings.fsmonitor_hook_path =strdup (const_strval);
We had a strbuf_detach()'d string in the case of repo_config_get_pathname(), but here we strdup() it again in case we were in the getenv() codepath. This code probably leaks memory now anyway, but perhaps it's better to split up the two so we make it easier to deal with who owns/frees what in the future.
Re: [PATCH v3 06/34] fsmonitor: config settings are repository-specific
Quoted 11 lines · Ævar Arnfjörð Bjarmason
On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:
In a reference to a discussion[1] about an earlier version of this patch you said:
I'm going to ignore all of the thread responses to this patch dealing with how we acquire config settings and macros and etc. Those issues are completely independent of FSMonitor (which is already way too big).
Quoted 5 lines
Since then the changes to
repo-settings.chave become a lot larger, so let's take a look...1. https://lore.kernel.org/git/[email protected]/
2. https://lore.kernel.org/git/[email protected]/
Yes, there was a large conversation about re-doing how config values are acquired or whatever and it was nested inside of the context of a completely unrelated topic. It still is.
I would like to focus on FSMonitor using the existing config API. I'm only looking up ~2 config values. And that code is fairly minor considering everything else in this patch series.
Yes, there's a bizarre initialization with memset(-1), but it can wait.
Later (in a clean context) we can address and focus on the config API and/or structure initialization that you're talking about here.
FWIW, in V4 I'll refactor the block of code that I added into the body of prepare_repo_settings() to address your later concerns. This version will hopefully be more readable.
Quoted 29 lines
Quoted 13 lines
repo-settings.c +3
-0@@ -5,10 +5,42 @@#define UPDATE_DEFAULT_BOOL (s,v) do { if (s == -1) { s = v; } } while(0) +/* +* Return 1 if the repo/workdir is incompatible with FSMonitor. +*/ repo-settings.c line 20
Quoted 10 lines
repo-settings.c +10
-0+static int is_repo_incompatible_with_fsmonitor (struct repository *r) +{ +const char *const_strval; + +/* +* Bare repositories don't have a working directory and +* therefore, nothing to watch. +*/ +if (!r->worktree) +return 1 ;Looking ahead in this series you end up using
FSMONITOR_MODE_INCOMPATIBLEin two places in the codebase. Inbuiltin/update-index.cto throw a "repository is incompatible with fsmonitor" error.
Quoted 5 lines
Can't that case just be replaced with
setup_work_tree()? Other sub-modes of update-index already die implicitly on that, e.g.:$ git update-index test fatal: this operation must be run in a work tree
I will refactor that static function in V4, but I want it to return an indication of whether the repo is compatible only and let the command print the error/die as is appropriate for the daemon and/or update-index. The daemon should not start on an incompatible repo. Likewise, update-index should not enable the extension in the index.
We share some of that code with the client side code, like status, which wants to talk to the hook or daemon if supported/present/allowed. If the repo is incompatible, then status should just behave in the classic manner. So I don't want the detection code to print those error messages or die.
Quoted 11 lines
The other case is:
+
prepare_repo_settings(the_repository);
+ if (!the_repository->worktree)
+ return error(_("fsmonitor-daemon does not support bare repos '%s'"),
+xgetcwd());
+ if (the_repository->settings.fsmonitor_mode==FSMONITOR_MODE_INCOMPATIBLE)
+ return error(_("fsmonitor-daemon is incompatible with this repo '%s'"),
+the_repository->worktree);
Quoted 1 line
I.e. we just checked
the_repository->worktree, but it's not that, but....
yes, I currently have 2 types of repos that I want to disable both the hook and daemon version of FSMonitor. I'll update the error messages to specify the reason why we are incompatible.
Quoted 10 lines
Quoted 10 lines
repo-settings.c +9
-0+ +/* +* GVFS (aka VFS for Git) is incompatible with FSMonitor. +* +* Granted, core Git does not know anything about GVFS and +* we shouldn't make assumptions about a downstream feature, +* but users can install both versions. And this can lead +* to incorrect results from core Git commands. So, without +* bringing in any of the GVFS code, do a simple config test
Quoted 8 lines
repo-settings.c line 34
Quoted 5 lines
repo-settings.c +5
-0+* for a published config setting. (We do not look at the +* various *_TEST_* environment variables.) +*/ +if (!repo_config_get_value (r,"core.virtualfilesystem" , &const_strval)) +return 1 ;I'm skeptical of us hardcoding a third-party software config variable. Can't GitVFS handle this somehow on its end?
Adding the test for a GVFS-specific config setting is questionable. And perhaps we should move it to our downstream fork.
The value in putting it here for now (at least) is that it makes clear the structure for supporting other types of incompatible repos.
For example, perhaps we want to disallow repos that are on remote file systems (since we might not be able to get FS events). It would be nice to be able to prevent the daemon from starting and/or from status from trying to connect to a daemon that will be a disappointment. The code I have here serves as a model for adding such additional restrictions. And keeps us from trying to prematurely collapse the code into a simple expression.
Quoted 7 lines
But just in terms of implementation it seems the end result of that is to emit a very confusing error to the user. Sinc we already checked for bare repos we run into this and instead of sayingwhen we should really say "hey, maybe disable your core.virtualFileSystem setting", we say "your repo is incompatible".
I'll update the error messages to make that clear.
Quoted 19 lines
repo-settings.c line 43
Quoted 9 lines
repo-settings.c +5
-0+ +return 0 ; +} +void prepare_repo_settings (struct repository *r) {int value;char *strval; +const char *const_strval;Can be declared in the "else" below.
Quoted 7 lines
repo-settings.c +0
-0if (r->settings.initialized)return ;@@ -26,6 +58,22 @@ void prepare_repo_settings(struct repository *r)UPDATE_DEFAULT_BOOL (r->settings.commit_graph_read_changed_paths,1 );UPDATE_DEFAULT_BOOL (r->settings.gc_write_commit_graph,1 );
Quoted 7 lines
repo-settings.c void prepare_repo_settings(struct repository *r) · line 62
Quoted 2 lines
repo-settings.c +2
-0+ r->settings.fsmonitor_hook_path =NULL ; + r->settings.fsmonitor_mode =FSMONITOR_MODE_DISABLED ;With the memset earlier (b.t.w. I've got a patch to fix all this bizarre behavior in
repo-settings.c, but have been waiting on this series we implicitly set it toFSMONITOR_MODE_UNSET(-1) with the memset, but then never use that ever.
I'm working around the bogus -1 value that the structure has been initialize with and that I'm inheriting. I'll completely set my fsmonitor_mode variable so that I don't care what it is initialized to.
I created the _UNSET value as a reminder that there is this memset(-1) there and that must be attended to.
In my V4 I'll add comments to that effect.
Quoted 17 lines
Your code in
update-index.cthen for a check against "FSMONITOR_MODE_DISABLED" says "core.useBuiltinFSMonitor is unset;".repo-settings.c void prepare_repo_settings(struct repository *r) · line 64
Quoted 2 lines
repo-settings.c +2
-0+if (is_repo_incompatible_with_fsmonitor (r)) + r->settings.fsmonitor_mode =FSMONITOR_MODE_INCOMPATIBLE ;Style: should have {} braces on all arms.
repo-settings.c void prepare_repo_settings(struct repository *r) · line 67
Quoted 3 lines
repo-settings.c +3
-0+else if (!repo_config_get_bool (r,"core.usebuiltinfsmonitor" , &value) + && value) + r->settings.fsmonitor_mode =FSMONITOR_MODE_IPC ;Here you're conflating false with whether the variable is set at all. I guess that works out here since if it's false we want to fall through to...
Quoted 3 lines
repo-settings.c void prepare_repo_settings(struct repository *r) · line 68
Quoted 1 line
repo-settings.c +1
-0+else {...ignoring it and looing at core.fsmonitor instead.
yes, if core.useBuiltinFSMonitor is true, we do not need to look at the hook pathname.
Quoted 7 lines
repo-settings.c void prepare_repo_settings(struct repository *r) · line 70
Quoted 3 lines
repo-settings.c +2
-0+if (repo_config_get_pathname (r,"core.fsmonitor" , &const_strval)) + const_strval =getenv ("GIT_TEST_FSMONITOR" );If it's not set we pay attention to
GIT_TEST_FSMONITOR, so the behavior from the oldgit_config_get_fsmonitor(). So even if the env variable is set we want to take the config variable over it, correct?
GIT_TEST_FSMONITOR sets the hook path for testing and we're not using the hook API at all. So it is kind of ill-defined what that test env var should do if you have IPC turned on, so I'm ignoring it.
Quoted 10 lines
repo-settings.c void prepare_repo_settings(struct repository *r) · line 72
Quoted 3 lines
repo-settings.c +2
-0+if (const_strval && *const_strval) { + r->settings.fsmonitor_hook_path =strdup (const_strval);We had a strbuf_detach()'d string in the case of
repo_config_get_pathname(), but here westrdup()it again in case we were in thegetenv()codepath. This code probably leaks memory now anyway, but perhaps it's better to split up the two so we make it easier to deal with who owns/frees what in the future.
repo_config_get_pathname() returns a const char** which implies that it is not a detached buffer. it is a deep trek thru the config code, but it eventually gets to git_configset_get_*() which is documented as returning a pointer into a configset cache and that the caller should not free it. so dup'ing it is appropriate.
Similarly, getenv() is also returning a pointer to a buffer that the caller does not own. so dup'ing it here is also appropriate.
Thanks,
Jeff
PATCH v3 7/34 fsmonitor: use IPC to query the builtin FSMonitor daemon
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Use simple IPC to directly communicate with the new builtin file system monitor daemon when core.useBuiltinFSMonitor is set.
The core.fsmonitor setting has already been defined as a HOOK pathname. Historically, this has been set to a HOOK script that will talk with Watchman. For compatibility reasons, we do not want to overload that definition (and cause problems if users have multiple versions of Git installed).
--- fsmonitor.c | 41 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-)
fsmonitor.c +39 -2
@@ -256,8 +256,44 @@ void refresh_fsmonitor(struct index_state *istate)trace_printf_key (&trace_fsmonitor,"refresh fsmonitor" );if (r->settings.fsmonitor_mode ==FSMONITOR_MODE_IPC ) { -/* TODO */ -return ; + query_success = !fsmonitor_ipc__send_query ( + istate->fsmonitor_last_update ? + istate->fsmonitor_last_update :"builtin:fake" , + &query_result); +if (query_success) { +/* +* The response contains a series of nul terminated +* strings. The first is the new token. +* +* Use `char *buf` as an interlude to trick the CI +* static analysis to let us use `strbuf_addstr()` +* here (and only copy the token) rather than +* `strbuf_addbuf()`. +*/ + buf = query_result.buf; +strbuf_addstr (&last_update_token, buf); + bol = last_update_token.len +1 ; + }else { +/* +* The builtin daemon is not available on this +* platform -OR- we failed to get a response. +* +* Generate a fake token (rather than a V1 +* timestamp) for the index extension. (If +* they switch back to the hook API, we don't +* want ambiguous state.) +*/ +strbuf_addstr (&last_update_token,"builtin:fake" ); + } + +/* +* Regardless of whether we successfully talked to a +* fsmonitor daemon or not, we skip over and do not +* try to use the hook. The "core.useBuiltinFSMonitor" +* config setting ALWAYS overrides the "core.fsmonitor" +* hook setting. +*/ + goto apply_results; }assert (r->settings.fsmonitor_mode ==FSMONITOR_MODE_HOOK );
@@ -323,6 +359,7 @@ void refresh_fsmonitor(struct index_state *istate) query_success ? "success" : "failure"); } +apply_results:/* a fsmonitor process can return '/' to indicate all entries are invalid */ if (query_success && query_result.buf[bol] !='/' ) {/* Mark all entries returned by the monitor as dirty */
Signature · 1 line
gitgitgadget
PATCH v3 8/34 fsmonitor--daemon: add a built-in fsmonitor daemon
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Create a built-in file system monitoring daemon that can be used by the existing fsmonitor feature (protocol API and index extension) to improve the performance of various Git commands, such as status.
The fsmonitor--daemon feature builds upon the Simple IPC API and provides an alternative to hook access to existing fsmonitors such as watchman.
This commit merely adds the new command without any functionality.
--- .gitignore | 1 + Makefile | 1 + builtin.h | 1 + builtin/fsmonitor--daemon.c | 53 +++++++++++++++++++++++++++++++++++++ git.c | 1 + 5 files changed, 57 insertions(+) create mode 100644 builtin/fsmonitor--daemon.c
.gitignore +1 -0
@@ -72,6 +72,7 @@ /git-format-patch /git-fsck /git-fsck-objects +/git-fsmonitor--daemon /git-gc /git-get-tar-commit-id /git-grep
Makefile +1 -0
@@ -1097,6 +1097,7 @@ BUILTIN_OBJS += builtin/fmt-merge-msg.o BUILTIN_OBJS += builtin/for-each-ref.o BUILTIN_OBJS += builtin/for-each-repo.o BUILTIN_OBJS += builtin/fsck.o +BUILTIN_OBJS += builtin/fsmonitor--daemon.o BUILTIN_OBJS += builtin/gc.o BUILTIN_OBJS += builtin/get-tar-commit-id.o BUILTIN_OBJS += builtin/grep.o
builtin.h +1 -0
@@ -159,6 +159,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix);int cmd_for_each_repo (int argc,const char **argv,const char *prefix);int cmd_format_patch (int argc,const char **argv,const char *prefix);int cmd_fsck (int argc,const char **argv,const char *prefix); +int cmd_fsmonitor__daemon (int argc,const char **argv,const char *prefix);int cmd_gc (int argc,const char **argv,const char *prefix);int cmd_get_tar_commit_id (int argc,const char **argv,const char *prefix);int cmd_grep (int argc,const char **argv,const char *prefix);
builtin/fsmonitor--daemon.c new file +53 -0
53 lines not on this page · raw message
git.c +1 -0
@@ -533,6 +533,7 @@ static struct cmd_struct commands[] = { { "format-patch", cmd_format_patch, RUN_SETUP }, { "fsck", cmd_fsck, RUN_SETUP }, { "fsck-objects", cmd_fsck, RUN_SETUP }, + { "fsmonitor--daemon", cmd_fsmonitor__daemon, RUN_SETUP }, { "gc", cmd_gc, RUN_SETUP }, { "get-tar-commit-id", cmd_get_tar_commit_id, NO_PARSEOPT }, { "grep", cmd_grep, RUN_SETUP_GENTLY },
Signature · 1 line
gitgitgadget
Re: [PATCH v3 08/34] fsmonitor--daemon: add a built-in fsmonitor daemon
On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:
A general comment on this series (including previous patches). We've usually tried to bend over backwards in git's codebase not to have big ifdef blocks, so we compile most code the same everywhere. We waste a bit of object code, but that's fine.
See 9c897c5c2ad (pack-objects: remove #ifdef NO_PTHREADS, 2018-11-03) for a good exmaple of bad code being turned to good.
E.g. in this case:
Quoted 31 lines
builtin/fsmonitor--daemon.c new file +31
-0+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND + +int cmd_fsmonitor__daemon (int argc,const char **argv,const char *prefix) +{ +const char *subcmd; + +struct option options[] = { +OPT_END () + }; + +if (argc <2 ) +usage_with_options (builtin_fsmonitor__daemon_usage, options); + +if (argc ==2 && !strcmp (argv[1 ],"-h" )) +usage_with_options (builtin_fsmonitor__daemon_usage, options); + +git_config (git_default_config,NULL ); + + subcmd = argv[1 ]; + argv--; + argc++; + + argc =parse_options (argc, argv, prefix, options, + builtin_fsmonitor__daemon_usage,0 ); + +die (_ ("Unhandled subcommand '%s'" ), subcmd); +} + +#else +int cmd_fsmonitor__daemon (int argc,const char **argv,const char *prefix) +{
builtin/fsmonitor--daemon.c line 53
Quoted 10 lines
builtin/fsmonitor--daemon.c new file +10
-0+struct option options[] = { +OPT_END () + }; + +if (argc ==2 && !strcmp (argv[1 ],"-h" )) +usage_with_options (builtin_fsmonitor__daemon_usage, options); + + die(_("fsmonitor--daemon not supported on this platform")); +} +#endif
This whole thing could really just be a -DHAVE_FSMONITOR_DAEMON_BACKEND=1 or -DHAVE_FSMONITOR_DAEMON_BACKEND=0 somewhere (depending), and then somewhere in the middle of the first function:
if (!HAVE_FSMONITOR_DAEMON_BACKEND)
die(_("fsmonitor--daemon not supported on this platform"));Re: [PATCH v3 08/34] fsmonitor--daemon: add a built-in fsmonitor daemon
Quoted 60 lines · Ævar Arnfjörð Bjarmason
On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:
A general comment on this series (including previous patches). We've usually tried to bend over backwards in git's codebase not to have big ifdef blocks, so we compile most code the same everywhere. We waste a bit of object code, but that's fine.
See
9c897c5c2ad(pack-objects: remove #ifdefNO_PTHREADS, 2018-11-03) for a good exmaple of bad code being turned to good.E.g. in this case:
Quoted 31 lines
builtin/fsmonitor--daemon.c new file +31
-0+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND + +int cmd_fsmonitor__daemon (int argc,const char **argv,const char *prefix) +{ +const char *subcmd; + +struct option options[] = { +OPT_END () + }; + +if (argc <2 ) +usage_with_options (builtin_fsmonitor__daemon_usage, options); + +if (argc ==2 && !strcmp (argv[1 ],"-h" )) +usage_with_options (builtin_fsmonitor__daemon_usage, options); + +git_config (git_default_config,NULL ); + + subcmd = argv[1 ]; + argv--; + argc++; + + argc =parse_options (argc, argv, prefix, options, + builtin_fsmonitor__daemon_usage,0 ); + +die (_ ("Unhandled subcommand '%s'" ), subcmd); +} + +#else +int cmd_fsmonitor__daemon (int argc,const char **argv,const char *prefix) +{builtin/fsmonitor--daemon.c line 53
Quoted 10 lines
builtin/fsmonitor--daemon.c new file +10
-0+struct option options[] = { +OPT_END () + }; + +if (argc ==2 && !strcmp (argv[1 ],"-h" )) +usage_with_options (builtin_fsmonitor__daemon_usage, options); + + die(_("fsmonitor--daemon not supported on this platform")); +} +#endif This whole thing could really just be a -DHAVE_FSMONITOR_DAEMON_BACKEND=1 or -DHAVE_FSMONITOR_DAEMON_BACKEND=0 somewhere (depending), and then somewhere in the middle of the first function:
Quoted 3 lines
if (!HAVE_FSMONITOR_DAEMON_BACKEND)
die(_("fsmonitor--daemon not supported on this platform"));This whole file will be filled up with ~1500 lines of static functions that only make sense when the daemon is supported and that make calls to platform-specific backends.
I suppose we could stub in an empty backend (something like that in 11/34 and 12/34) and hack in all stuff in the makefile to link to it in the unsupported case, but that seems like a lot of effort just to avoid an ifdef here.
I mean, the intent of the #else block is quite clear and we're not fooling the reader with a large source file of code that will never be used on their platform.
We could consider splitting this source file into a supported and unsupported version and have the makefile select the right .c file. We'd have to move the usage and stuff to a shared header and etc. That would eliminate the ifdef, but it would break the convention of the source filename matching the command name.
I'm not sure it's worth the bother TBH.
Jeff
PATCH v3 9/34 fsmonitor--daemon: implement 'stop' and 'status' commands
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Implement stop and status client commands to control and query the status of a fsmonitor--daemon server process (and implicitly start a server process if necessary).
Later commits will implement the actual server and monitor the file system.
--- builtin/fsmonitor--daemon.c | 51 +++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+)
builtin/fsmonitor--daemon.c +51 -0
@@ -7,10 +7,55 @@#include "khash.h" static const char *const builtin_fsmonitor__daemon_usage[] = { +N_ ("git fsmonitor--daemon stop" ), +N_ ("git fsmonitor--daemon status" ),NULL };#ifdef HAVE_FSMONITOR_DAEMON_BACKEND +/* +* Acting as a CLIENT. +* +* Send a "quit" command to the `git-fsmonitor--daemon` (if running) +* and wait for it to shutdown. +*/ +static int do_as_client__send_stop (void ) +{ +struct strbuf answer =STRBUF_INIT ; +int ret; + + ret =fsmonitor_ipc__send_command ("quit" , &answer); + +/* The quit command does not return any response data. */ +strbuf_release (&answer); + +if (ret) +return ret; + +trace2_region_enter ("fsm_client" ,"polling-for-daemon-exit" ,NULL ); +while (fsmonitor_ipc__get_state () ==IPC_STATE__LISTENING ) +sleep_millisec (50 ); +trace2_region_leave ("fsm_client" ,"polling-for-daemon-exit" ,NULL ); + +return 0 ; +} + +static int do_as_client__status (void ) +{ +enum ipc_active_state state =fsmonitor_ipc__get_state (); + +switch (state) { +case IPC_STATE__LISTENING : +printf (_ ("fsmonitor-daemon is watching '%s'\n" ), + the_repository->worktree); +return 0 ; + +default : +printf (_ ("fsmonitor-daemon is not watching '%s'\n" ), + the_repository->worktree); +return 1 ; + } +}int cmd_fsmonitor__daemon (int argc,const char **argv,const char *prefix) {
@@ -35,6 +80,12 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) argc =parse_options (argc, argv, prefix, options, builtin_fsmonitor__daemon_usage,0 ); +if (!strcmp (subcmd,"stop" )) +return !!do_as_client__send_stop (); + +if (!strcmp (subcmd,"status" )) +return !!do_as_client__status (); +die (_ ("Unhandled subcommand '%s'" ), subcmd); }
Signature · 1 line
gitgitgadget
PATCH v3 10/34 t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Create an IPC client to send query and flush commands to the daemon.
--- Makefile | 1 + t/helper/test-fsmonitor-client.c | 121 +++++++++++++++++++++++++++++++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + 4 files changed, 124 insertions(+) create mode 100644 t/helper/test-fsmonitor-client.c
Makefile +1 -0
@@ -709,6 +709,7 @@ TEST_BUILTINS_OBJS += test-dump-split-index.o TEST_BUILTINS_OBJS += test-dump-untracked-cache.o TEST_BUILTINS_OBJS += test-example-decorate.o TEST_BUILTINS_OBJS += test-fast-rebase.o +TEST_BUILTINS_OBJS += test-fsmonitor-client.o TEST_BUILTINS_OBJS += test-genrandom.o TEST_BUILTINS_OBJS += test-genzeros.o TEST_BUILTINS_OBJS += test-hash-speed.o
t/helper/test-fsmonitor-client.c new file +121 -0
121 lines not on this page · raw message
t/helper/test-tool.c +1 -0
@@ -31,6 +31,7 @@ static struct test_cmd cmds[] = { { "dump-untracked-cache", cmd__dump_untracked_cache }, { "example-decorate", cmd__example_decorate }, { "fast-rebase", cmd__fast_rebase }, + { "fsmonitor-client", cmd__fsmonitor_client }, { "genrandom", cmd__genrandom }, { "genzeros", cmd__genzeros }, { "hashmap", cmd__hashmap },
t/helper/test-tool.h +1 -0
@@ -21,6 +21,7 @@ int cmd__dump_split_index(int argc, const char **argv);int cmd__dump_untracked_cache (int argc,const char **argv);int cmd__example_decorate (int argc,const char **argv);int cmd__fast_rebase (int argc,const char **argv); +int cmd__fsmonitor_client (int argc,const char **argv);int cmd__genrandom (int argc,const char **argv);int cmd__genzeros (int argc,const char **argv);int cmd__hashmap (int argc,const char **argv);
Signature · 1 line
gitgitgadget
Re: [PATCH v3 10/34] t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon
Quoted 37 lines · Jeff Hostetler via GitGitGadget
From: Jeff Hostetler <[email protected]>
Create an IPC client to send query and flush commands to the daemon.
Signed-off-by Jeff Hostetler <[email protected]>--- Makefile | 1 + t/helper/test-fsmonitor-client.c | 121 +++++++++++++++++++++++++++++++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + 4 files changed, 124 insertions(+) create mode 100644 t/helper/test-fsmonitor-client.cMakefile +1
-0@@ -709,6 +709,7 @@ TEST_BUILTINS_OBJS += test-dump-split-index.o TEST_BUILTINS_OBJS += test-dump-untracked-cache.o TEST_BUILTINS_OBJS += test-example-decorate.o TEST_BUILTINS_OBJS += test-fast-rebase.o +TEST_BUILTINS_OBJS += test-fsmonitor-client.o TEST_BUILTINS_OBJS += test-genrandom.o TEST_BUILTINS_OBJS += test-genzeros.o TEST_BUILTINS_OBJS += test-hash-speed.ot/helper/test-fsmonitor-client.c new file +6
-0@@ -0,0 +1,121 @@ +/* +* test-fsmonitor-client.c: client code to send commands/requests to +* a `git fsmonitor--daemon` daemon. +*/ + +#include "test-tool.h"
t/helper/test-fsmonitor-client.c line 16
Quoted 10 lines
t/helper/test-fsmonitor-client.c new file +10
-0+#include "cache.h" +#include "parse-options.h" +#include "fsmonitor-ipc.h" + +#ifndef HAVE_FSMONITOR_DAEMON_BACKEND +int cmd__fsmonitor_client (int argc,const char **argv) +{ +die ("fsmonitor--daemon not available on this platform" ); +} +#else
Re my earlier comments on excessive ifdefs: In this case don't we just want to not compile test-fsmonitor-client at all unless HAVE_FSMONITOR_DAEMON_BACKEND is true.
You'll get the same error as though you ran "helper/test-tool does-not-exist", but the tests check for the prerequisite earlier anyway, so why get this far on an unsupported platform for a pure test helper?
PATCH v3 11/34 fsmonitor-fs-listen-win32: stub in backend for Windows
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Stub in empty backend for fsmonitor--daemon on Windows.
--- Makefile | 13 ++++++ compat/fsmonitor/fsmonitor-fs-listen-win32.c | 21 +++++++++ compat/fsmonitor/fsmonitor-fs-listen.h | 49 ++++++++++++++++++++ config.mak.uname | 2 + contrib/buildsystems/CMakeLists.txt | 5 ++ 5 files changed, 90 insertions(+) create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-win32.c create mode 100644 compat/fsmonitor/fsmonitor-fs-listen.h
Makefile +13 -0
@@ -467,6 +467,11 @@ all:: # directory, and the JSON compilation database 'compile_commands.json' will be # created at the root of the repository. # +# If your platform supports a built-in fsmonitor backend, set +# FSMONITOR_DAEMON_BACKEND to the "<name>" of the corresponding +# `compat/fsmonitor/fsmonitor-fs-listen-<name>.c` that implements the +# `fsmonitor_fs_listen__*()` routines. +# # Define DEVELOPER to enable more compiler warnings. Compiler version # and family are auto detected, but could be overridden by defining # COMPILER_FEATURES (see config.mak.dev). You can still set
@@ -1929,6 +1934,11 @@ ifdef NEED_ACCESS_ROOT_HANDLER COMPAT_OBJS += compat/access.o endif +ifdef FSMONITOR_DAEMON_BACKEND + COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND + COMPAT_OBJS += compat/fsmonitor/fsmonitor-fs-listen-$(FSMONITOR_DAEMON_BACKEND).o +endif + ifeq ($(TCLTK_PATH),) NO_TCLTK = NoThanks endif
@@ -2793,6 +2803,9 @@ GIT-BUILD-OPTIONS: FORCE @echo PAGER_ENV=\''$(subst ','\'',$(subst ','\'',$(PAGER_ENV)))'\' >>$@+ @echo DC_SHA1=\''$(subst ','\'',$(subst ','\'',$(DC_SHA1)))'\' >>$@+ @echo X=\'$(X)\' >>$@+ +ifdef FSMONITOR_DAEMON_BACKEND + @echo FSMONITOR_DAEMON_BACKEND=\''$(subst ','\'',$(subst ','\'',$(FSMONITOR_DAEMON_BACKEND)))'\' >>$@+ +endif ifdef TEST_OUTPUT_DIRECTORY @echo TEST_OUTPUT_DIRECTORY=\''$(subst ','\'',$(subst ','\'',$(TEST_OUTPUT_DIRECTORY)))'\' >>$@+ endif
compat/fsmonitor/fsmonitor-fs-listen-win32.c new file +21 -0
@@ -0,0 +1,21 @@ +#include "cache.h" +#include "config.h" +#include "fsmonitor.h" +#include "fsmonitor-fs-listen.h" + +void fsmonitor_fs_listen__stop_async (struct fsmonitor_daemon_state *state) +{ +} + +void fsmonitor_fs_listen__loop (struct fsmonitor_daemon_state *state) +{ +} + +int fsmonitor_fs_listen__ctor (struct fsmonitor_daemon_state *state) +{ +return -1 ; +} + +void fsmonitor_fs_listen__dtor (struct fsmonitor_daemon_state *state) +{ +}
compat/fsmonitor/fsmonitor-fs-listen.h new file +49 -0
49 lines not on this page · raw message
config.mak.uname +2 -0
@@ -420,6 +420,7 @@ ifeq ($(uname_S),Windows) # so we don't need this: # # SNPRINTF_RETURNS_BOGUS = YesPlease + FSMONITOR_DAEMON_BACKEND = win32 NO_SVN_TESTS = YesPlease RUNTIME_PREFIX = YesPlease HAVE_WPGMPTR = YesWeDo
@@ -598,6 +599,7 @@ ifneq (,$(findstring MINGW,$(uname_S))) NO_STRTOUMAX = YesPlease NO_MKDTEMP = YesPlease NO_SVN_TESTS = YesPlease + FSMONITOR_DAEMON_BACKEND = win32 RUNTIME_PREFIX = YesPlease HAVE_WPGMPTR = YesWeDo NO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease
contrib/buildsystems/CMakeLists.txt +5 -0
@@ -263,6 +263,11 @@ else() endif() endif() +if(CMAKE_SYSTEM_NAME STREQUAL "Windows") + add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) + list(APPEND compat_SOURCES compat/fsmonitor/fsmonitor-fs-listen-win32.c) +endif() + set(EXE_EXTENSION ${CMAKE_EXECUTABLE_SUFFIX}) #header checks
Signature · 1 line
gitgitgadget
Re: [PATCH v3 11/34] fsmonitor-fs-listen-win32: stub in backend for Windows
Quoted 44 lines · Jeff Hostetler via GitGitGadget
From: Jeff Hostetler <[email protected]>
Stub in empty backend for fsmonitor--daemon on Windows.
Signed-off-by Jeff Hostetler <[email protected]>--- Makefile | 13 ++++++ compat/fsmonitor/fsmonitor-fs-listen-win32.c | 21 +++++++++ compat/fsmonitor/fsmonitor-fs-listen.h | 49 ++++++++++++++++++++ config.mak.uname | 2 + contrib/buildsystems/CMakeLists.txt | 5 ++ 5 files changed, 90 insertions(+) create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-win32.c create mode 100644 compat/fsmonitor/fsmonitor-fs-listen.hMakefile +10
-0@@ -467,6 +467,11 @@ all:: # directory, and the JSON compilation database 'compile_commands.json' will be # created at the root of the repository. # +# If your platform supports a built-in fsmonitor backend, set +# FSMONITOR_DAEMON_BACKEND to the "<name>" of the corresponding +# `compat/fsmonitor/fsmonitor-fs-listen-<name>.c` that implements the +# `fsmonitor_fs_listen__*()` routines. +# # Define DEVELOPER to enable more compiler warnings. Compiler version # and family are auto detected, but could be overridden by defining # COMPILER_FEATURES (see config.mak.dev). You can still set@@ -1929,6 +1934,11 @@ ifdef NEED_ACCESS_ROOT_HANDLER COMPAT_OBJS += compat/access.o endif +ifdef FSMONITOR_DAEMON_BACKEND + COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND + COMPAT_OBJS += compat/fsmonitor/fsmonitor-fs-listen-$(FSMONITOR_DAEMON_BACKEND).o +endif + ifeq ($(TCLTK_PATH),) NO_TCLTK = NoThanks endif
Makefile GIT-BUILD-OPTIONS: FORCE · line 2808
Quoted 6 lines
Makefile +3
-0@@ -2793,6 +2803,9 @@ GIT-BUILD-OPTIONS: FORCE @echo PAGER_ENV=\''$(subst ','\'',$(subst ','\'',$(PAGER_ENV)))'\' >>$@+ @echo DC_SHA1=\''$(subst ','\'',$(subst ','\'',$(DC_SHA1)))'\' >>$@+ @echo X=\'$(X)\' >>$@+ +ifdef FSMONITOR_DAEMON_BACKEND + @echo FSMONITOR_DAEMON_BACKEND=\''$(subst ','\'',$(subst ','\'',$(FSMONITOR_DAEMON_BACKEND)))'\' >>$@+ +endif
Why put this in an ifdef?
In 342e9ef2d9e (Introduce a performance testing framework, 2012-02-17) we started doing that for some perf/test options (which b.t.w., I don't really see the reason for, maybe it's some subtlety in how test-lib.sh picks those up).
But for all the other compile-time stuff we don't ifdef it, we just define it, and then you get an empty value or not.
This would AFAICT be the first build-time-for-the-C-program option we ifdef for writing a line to GIT-BUILD-OPTIONS.
Re: [PATCH v3 11/34] fsmonitor-fs-listen-win32: stub in backend for Windows
Hi Ævar,
Quoted 47 lines · Ævar Arnfjörð Bjarmason
Quoted 44 lines · Jeff Hostetler via GitGitGadget
From: Jeff Hostetler <[email protected]>
Stub in empty backend for fsmonitor--daemon on Windows.
Signed-off-by Jeff Hostetler <[email protected]>--- Makefile | 13 ++++++ compat/fsmonitor/fsmonitor-fs-listen-win32.c | 21 +++++++++ compat/fsmonitor/fsmonitor-fs-listen.h | 49 ++++++++++++++++++++ config.mak.uname | 2 + contrib/buildsystems/CMakeLists.txt | 5 ++ 5 files changed, 90 insertions(+) create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-win32.c create mode 100644 compat/fsmonitor/fsmonitor-fs-listen.hMakefile +10
-0@@ -467,6 +467,11 @@ all:: # directory, and the JSON compilation database 'compile_commands.json' will be # created at the root of the repository. # +# If your platform supports a built-in fsmonitor backend, set +# FSMONITOR_DAEMON_BACKEND to the "<name>" of the corresponding +# `compat/fsmonitor/fsmonitor-fs-listen-<name>.c` that implements the +# `fsmonitor_fs_listen__*()` routines. +# # Define DEVELOPER to enable more compiler warnings. Compiler version # and family are auto detected, but could be overridden by defining # COMPILER_FEATURES (see config.mak.dev). You can still set@@ -1929,6 +1934,11 @@ ifdef NEED_ACCESS_ROOT_HANDLER COMPAT_OBJS += compat/access.o endif +ifdef FSMONITOR_DAEMON_BACKEND + COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND + COMPAT_OBJS += compat/fsmonitor/fsmonitor-fs-listen-$(FSMONITOR_DAEMON_BACKEND).o +endif + ifeq ($(TCLTK_PATH),) NO_TCLTK = NoThanks endif@@ -2793,6 +2803,9 @@ GIT-BUILD-OPTIONS: FORCE
Quoted 8 lines
Makefile GIT-BUILD-OPTIONS: FORCE · line 2808
Quoted 6 lines
Makefile +3
-0@echo PAGER_ENV=\''$(subst ','\'',$(subst ','\'',$(PAGER_ENV)))'\' >>$@+ @echo DC_SHA1=\''$(subst ','\'',$(subst ','\'',$(DC_SHA1)))'\' >>$@+ @echo X=\'$(X)\' >>$@+ +ifdef FSMONITOR_DAEMON_BACKEND + @echo FSMONITOR_DAEMON_BACKEND=\''$(subst ','\'',$(subst ','\'',$(FSMONITOR_DAEMON_BACKEND)))'\' >>$@+ +endifWhy put this in an ifdef?
Why not? What benefit does this question bring to improving this patch series?
Ciao,
Dscho
Re: [PATCH v3 11/34] fsmonitor-fs-listen-win32: stub in backend for Windows
Quoted 60 lines · Johannes Schindelin
Hi Ævar,
Quoted 47 lines · Ævar Arnfjörð Bjarmason
Quoted 44 lines · Jeff Hostetler via GitGitGadget
From: Jeff Hostetler <[email protected]>
Stub in empty backend for fsmonitor--daemon on Windows.
Signed-off-by Jeff Hostetler <[email protected]>--- Makefile | 13 ++++++ compat/fsmonitor/fsmonitor-fs-listen-win32.c | 21 +++++++++ compat/fsmonitor/fsmonitor-fs-listen.h | 49 ++++++++++++++++++++ config.mak.uname | 2 + contrib/buildsystems/CMakeLists.txt | 5 ++ 5 files changed, 90 insertions(+) create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-win32.c create mode 100644 compat/fsmonitor/fsmonitor-fs-listen.hMakefile +10
-0@@ -467,6 +467,11 @@ all:: # directory, and the JSON compilation database 'compile_commands.json' will be # created at the root of the repository. # +# If your platform supports a built-in fsmonitor backend, set +# FSMONITOR_DAEMON_BACKEND to the "<name>" of the corresponding +# `compat/fsmonitor/fsmonitor-fs-listen-<name>.c` that implements the +# `fsmonitor_fs_listen__*()` routines. +# # Define DEVELOPER to enable more compiler warnings. Compiler version # and family are auto detected, but could be overridden by defining # COMPILER_FEATURES (see config.mak.dev). You can still set@@ -1929,6 +1934,11 @@ ifdef NEED_ACCESS_ROOT_HANDLER COMPAT_OBJS += compat/access.o endif +ifdef FSMONITOR_DAEMON_BACKEND + COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND + COMPAT_OBJS += compat/fsmonitor/fsmonitor-fs-listen-$(FSMONITOR_DAEMON_BACKEND).o +endif + ifeq ($(TCLTK_PATH),) NO_TCLTK = NoThanks endif@@ -2793,6 +2803,9 @@ GIT-BUILD-OPTIONS: FORCEQuoted 8 lines
Makefile GIT-BUILD-OPTIONS: FORCE · line 2808
Quoted 6 lines
Makefile +3
-0@echo PAGER_ENV=\''$(subst ','\'',$(subst ','\'',$(PAGER_ENV)))'\' >>$@+ @echo DC_SHA1=\''$(subst ','\'',$(subst ','\'',$(DC_SHA1)))'\' >>$@+ @echo X=\'$(X)\' >>$@+ +ifdef FSMONITOR_DAEMON_BACKEND + @echo FSMONITOR_DAEMON_BACKEND=\''$(subst ','\'',$(subst ','\'',$(FSMONITOR_DAEMON_BACKEND)))'\' >>$@+ +endifWhy put this in an ifdef?
Quoted 2 lines
Why not? What benefit does this question bring to improving this patch series?
I think that when adding code to the Makefile it makes sense to follow the prevailing pattern, unless there's a good reason to do otherwise, e.g. on my build:
$ grep "''" GIT-BUILD-OPTIONS NO_CURL='' NO_EXPAT='' NO_PERL='' NO_PTHREADS='' NO_PYTHON='' NO_UNIX_SOCKETS='' X=''
Why does the FSMONITOR_DAEMON_BACKEND option require a nonexistent line as opposed to an empty one?
Re: [PATCH v3 11/34] fsmonitor-fs-listen-win32: stub in backend for Windows
Quoted 28 lines · Ævar Arnfjörð Bjarmason
Quoted 10 lines
Quoted 10 lines
Quoted 8 lines
Makefile +5
-0+ifdef FSMONITOR_DAEMON_BACKEND + COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND + COMPAT_OBJS += compat/fsmonitor/fsmonitor-fs-listen-$(FSMONITOR_DAEMON_BACKEND).o +endif + ifeq ($(TCLTK_PATH),) NO_TCLTK = NoThanks endif...
Quoted 4 lines
Quoted 1 line
Why put this in an ifdef?
Why not? What benefit does this question bring to improving this patch series?
I think that when adding code to the
Makefileit makes sense to follow the prevailing pattern, unless there's a good reason to do otherwise, e.g. on my build:$ grep "''" GIT-BUILD-OPTIONS NO_CURL='' NO_EXPAT='' NO_PERL='' NO_PTHREADS='' NO_PYTHON='' NO_UNIX_SOCKETS='' X=''
Quoted 2 lines
Why does the
FSMONITOR_DAEMON_BACKENDoption require a nonexistent line as opposed to an empty one?
I do not quite get the question.
#!/bin/sh
cat >make.file <<\EOF
all::
ifeq ($(FSMONITOR_DAEMON_BACKEND),)
echo it is empty
endif
ifdef FSMONITOR_DAEMON_BACKEND
echo it is undefined
endif
EOF
echo "unset???"
make -f make.file
echo "set to empty???"
make -f make.file FSMONITOR_DAEMON_BACKEND=
These two make invocations will give us the same result, showing that "is it set to empty" and "is it unset" are the same.
Re: [PATCH v3 11/34] fsmonitor-fs-listen-win32: stub in backend for Windows
Quoted 7 lines · Junio C Hamano
#!/bin/sh
cat >make.file <<\EOF
all::
ifeq ($(FSMONITOR_DAEMON_BACKEND),)
echo it is empty
endif
ifdefFSMONITOR_DAEMON_BACKEND
An obvious typo. This must be "ifndef", of course.
Quoted 12 lines
echo it is undefined
endif
EOFecho "unset???"
make -f make.fileecho "set to empty???"
make -f make.file FSMONITOR_DAEMON_BACKEND=These two make invocations will give us the same result, showing that "is it set to empty" and "is it unset" are the same.
Re: [PATCH v3 11/34] fsmonitor-fs-listen-win32: stub in backend for Windows
In reply to Junio C HamanoQuoted 46 lines · Junio C Hamano
Quoted 28 lines · Ævar Arnfjörð Bjarmason
Quoted 10 lines
Quoted 10 lines
Quoted 8 lines
Makefile +5
-0+ifdef FSMONITOR_DAEMON_BACKEND + COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND + COMPAT_OBJS += compat/fsmonitor/fsmonitor-fs-listen-$(FSMONITOR_DAEMON_BACKEND).o +endif + ifeq ($(TCLTK_PATH),) NO_TCLTK = NoThanks endif...
Quoted 4 lines
Quoted 1 line
Why put this in an ifdef?
Why not? What benefit does this question bring to improving this patch series?
I think that when adding code to the
Makefileit makes sense to follow the prevailing pattern, unless there's a good reason to do otherwise, e.g. on my build:$ grep "''" GIT-BUILD-OPTIONS NO_CURL='' NO_EXPAT='' NO_PERL='' NO_PTHREADS='' NO_PYTHON='' NO_UNIX_SOCKETS='' X=''Quoted 2 lines
Why does the
FSMONITOR_DAEMON_BACKENDoption require a nonexistent line as opposed to an empty one?I do not quite get the question.
#!/bin/sh
cat >make.file <<\EOF
all::
ifeq ($(FSMONITOR_DAEMON_BACKEND),)
echo it is empty
endif
ifdefFSMONITOR_DAEMON_BACKEND
echo it is undefined
endif
EOF
Quoted 8 lines
echo "unset???"
make -f make.fileecho "set to empty???"
make -f make.file FSMONITOR_DAEMON_BACKEND=These two make invocations will give us the same result, showing that "is it set to empty" and "is it unset" are the same.
Indeed, which is why I'm pointing out that wrapping it in an ifdef is pointless, which is why we don't do it for the other ones.
We do have a bunch of ifdef'd things there for perf etc., I'm not sure if it matters or not for those.
Re: [PATCH v3 11/34] fsmonitor-fs-listen-win32: stub in backend for Windows
Quoted 24 lines · Ævar Arnfjörð Bjarmason
Quoted 24 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line
Why put this in an ifdef?
...
Why does the
FSMONITOR_DAEMON_BACKENDoption require a nonexistent line as opposed to an empty one?I do not quite get the question.
#!/bin/sh
cat >make.file <<\EOF
all::
ifeq ($(FSMONITOR_DAEMON_BACKEND),)
echo it is empty
endif
ifndefFSMONITOR_DAEMON_BACKEND
echo it is undefined
endif
EOFecho "unset???"
make -f make.fileecho "set to empty???"
make -f make.file FSMONITOR_DAEMON_BACKEND=
Quoted 8 lines
Quoted 2 lines
These two make invocations will give us the same result, showing that "is it set to empty" and "is it unset" are the same.
Indeed, which is why I'm pointing out that wrapping it in an ifdef is pointless, which is why we don't do it for the other ones.
We do have a bunch of ifdef'd things there for perf etc., I'm not sure if it matters or not for those.
Sorry, but I still do not get the question. There are bunch of ifndef in Makefile in addition to ifeq/ifneq and your question
FSMONITOR_DAEMON_BACKEND option require a nonexistent line as opposed to an empty one?
is asking "why is it X" when X is not quite true. I presume that your "wrapping it in an ifdef" refers to a construct like this:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Makefile ifdef NEED_ACCESS_ROOT_HANDLER · line 1940
Quoted 4 lines
Makefile +4
-0+ifdef FSMONITOR_DAEMON_BACKEND + COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND + COMPAT_OBJS += compat/fsmonitor/fsmonitor-fs-listen-$(FSMONITOR_DAEMON_BACKEND).o +endif
but is your suggestion that it should be written like this instead?
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Makefile ifdef NEED_ACCESS_ROOT_HANDLER · line 1940
Quoted 4 lines
Makefile +4
-0+ifneq ($(FSMONITOR_DAEMON_BACKEND),) + COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND + COMPAT_OBJS += compat/fsmonitor/fsmonitor-fs-listen-$(FSMONITOR_DAEMON_BACKEND).o +endif
I do not think the latter is any easier to follow (and we have many ifdef and ifndef in our Makefile already). Perhaps I will see what you mean when I see your "better alternative", but so far, I am not successfully guessing what it is.
Re: [PATCH v3 11/34] fsmonitor-fs-listen-win32: stub in backend for Windows
In reply to Johannes SchindelinQuoted 13 lines · Johannes Schindelin
Quoted 11 lines · Ævar Arnfjörð Bjarmason
Makefile GIT-BUILD-OPTIONS: FORCE · line 2808
Quoted 7 lines · Jeff Hostetler via GitGitGadget
Makefile +3
-0@@ -2793,6 +2803,9 @@ GIT-BUILD-OPTIONS: FORCE @echo PAGER_ENV=\''$(subst ','\'',$(subst ','\'',$(PAGER_ENV)))'\' >>$@+ @echo DC_SHA1=\''$(subst ','\'',$(subst ','\'',$(DC_SHA1)))'\' >>$@+ @echo X=\'$(X)\' >>$@+ +ifdef FSMONITOR_DAEMON_BACKEND + @echo FSMONITOR_DAEMON_BACKEND=\''$(subst ','\'',$(subst ','\'',$(FSMONITOR_DAEMON_BACKEND)))'\' >>$@+ +endifWhy put this in an ifdef?
Quoted 2 lines
Why not? What benefit does this question bring to improving this patch series?
This is a common debate tactic known as "shifting the burden of proof".
Ævar does not need to prove that your patch is undesirable, *you* have to prove that it is desirable.
You have the burden of proof, so you should answer the question.
https://www.logicallyfallacious.com/logicalfallacies/Shifting-of-the-Burden-of-Proof
Signature · 1 line
Felipe Contreras
Re: [PATCH v3 11/34] fsmonitor-fs-listen-win32: stub in backend for Windows
In reply to Ævar Arnfjörð BjarmasonQuoted 61 lines · Ævar Arnfjörð Bjarmason
Quoted 44 lines · Jeff Hostetler via GitGitGadget
From: Jeff Hostetler <[email protected]>
Stub in empty backend for fsmonitor--daemon on Windows.
Signed-off-by Jeff Hostetler <[email protected]>--- Makefile | 13 ++++++ compat/fsmonitor/fsmonitor-fs-listen-win32.c | 21 +++++++++ compat/fsmonitor/fsmonitor-fs-listen.h | 49 ++++++++++++++++++++ config.mak.uname | 2 + contrib/buildsystems/CMakeLists.txt | 5 ++ 5 files changed, 90 insertions(+)create mode 100644
compat/fsmonitor/fsmonitor-fs-listen-win32.ccreate mode 100644compat/fsmonitor/fsmonitor-fs-listen.hMakefile +10
-0@@ -467,6 +467,11 @@ all:: # directory, and the JSON compilation database 'compile_commands.json' will be # created at the root of the repository. # +# If your platform supports a built-in fsmonitor backend, set +# FSMONITOR_DAEMON_BACKEND to the "<name>" of the corresponding +# `compat/fsmonitor/fsmonitor-fs-listen-<name>.c` that implements the +# `fsmonitor_fs_listen__*()` routines. +# # Define DEVELOPER to enable more compiler warnings. Compiler version # and family are auto detected, but could be overridden by defining # COMPILER_FEATURES (see config.mak.dev). You can still set@@ -1929,6 +1934,11 @@ ifdef NEED_ACCESS_ROOT_HANDLER COMPAT_OBJS += compat/access.o endif +ifdef FSMONITOR_DAEMON_BACKEND + COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND + COMPAT_OBJS += compat/fsmonitor/fsmonitor-fs-listen-$(FSMONITOR_DAEMON_BACKEND).o +endif + ifeq ($(TCLTK_PATH),) NO_TCLTK = NoThanks endifMakefile GIT-BUILD-OPTIONS: FORCE · line 2808
Quoted 6 lines
Makefile +3
-0@@ -2793,6 +2803,9 @@ GIT-BUILD-OPTIONS: FORCE @echo PAGER_ENV=\''$(subst ','\'',$(subst ','\'',$(PAGER_ENV)))'\' >>$@+ @echo DC_SHA1=\''$(subst ','\'',$(subst ','\'',$(DC_SHA1)))'\' >>$@+ @echo X=\'$(X)\' >>$@+ +ifdef FSMONITOR_DAEMON_BACKEND + @echo FSMONITOR_DAEMON_BACKEND=\''$(subst ','\'',$(subst ','\'',$(FSMONITOR_DAEMON_BACKEND)))'\' >>$@+ +endifWhy put this in an ifdef?
In
342e9ef2d9e(Introduce a performance testing framework, 2012-02-17) we started doing that for some perf/test options (which b.t.w., I don't really see the reason for, maybe it's some subtlety in howtest-lib.shpicks those up).
Quoted 6 lines
But for all the other compile-time stuff we don't ifdef it, we just define it, and then you get an empty value or not.
This would AFAICT be the first build-time-for-the-C-program option we ifdef for writing a line to GIT-BUILD-OPTIONS.
(I'm going to respond here on the original question rather than on any of the follow up responses in an attempt at diffusing things a bit.)
I added the ifdef because I thought it to be the *most conservative* thing that I could do. The output of the generated file on unsupported platforms should be *identical* to what it was before my changes. I only alter the contents of the generated file on supported platforms.
Later, when the generated file is consumed, we don't need to worry about the effect (if any) on incremental compiles -- we will know that it won't be set -- just like it was not set in the original compile.
That change appears right before a 12 other ifdef'd symbols also being written to that generated file. Most are test and perf, but some are not. But my point is that the pattern is present already.
The original question also references a 9.5 year old commit which uses the same pattern as I've used here. It also muddies the water on why it was/wasn't needed back then. And hints at possible side-effects in some of our test scripts. So it is clear that the confusion/disagreements that we are having with the current patch and whether or not to ifdef are not new.
So, is there value in being explicit and having the ifdef ??
There are well defined Make rules (and Junio gave us a very elegant little script to demonstrate that), but the subtleties are there. Especially with our use generated files like GIT-BUILD-OPTIONS. We have a mailing list full of experts and yet this question received a lot more discussion than I thought possible or necessary, but it took a test script to demonstrate that the results are the same and it doesn't matter. Perhaps the clarity is worth it for the price of a simple ifdef.
So, how much time have we (collectively) wasted discussing this subtlety ??
To summarize, I added the ifdef to make it explicitly clear that I'm not altering behavior on unsupported platforms. I can remove it from V4 if desired or I can keep it. (We all now know that it doesn't functionally matter -- it does however, provide clarity.)
Sorry if this sounded like a rant,
Jeff
Re: [PATCH v3 11/34] fsmonitor-fs-listen-win32: stub in backend for Windows
Quoted 68 lines · Jeff Hostetler
Quoted 54 lines · Ævar Arnfjörð Bjarmason
Quoted 43 lines · Jeff Hostetler via GitGitGadget
From: Jeff Hostetler <[email protected]>
Stub in empty backend for fsmonitor--daemon on Windows.
Signed-off-by Jeff Hostetler <[email protected]>--- Makefile | 13 ++++++ compat/fsmonitor/fsmonitor-fs-listen-win32.c | 21 +++++++++ compat/fsmonitor/fsmonitor-fs-listen.h | 49 ++++++++++++++++++++ config.mak.uname | 2 + contrib/buildsystems/CMakeLists.txt | 5 ++ 5 files changed, 90 insertions(+)create mode 100644
compat/fsmonitor/fsmonitor-fs-listen-win32.ccreate mode 100644compat/fsmonitor/fsmonitor-fs-listen.hMakefile +9
-0@@ -467,6 +467,11 @@ all:: # directory, and the JSON compilation database 'compile_commands.json' will be # created at the root of the repository. # +# If your platform supports a built-in fsmonitor backend, set +# FSMONITOR_DAEMON_BACKEND to the "<name>" of the corresponding +# `compat/fsmonitor/fsmonitor-fs-listen-<name>.c` that implements the +# `fsmonitor_fs_listen__*()` routines. +# # Define DEVELOPER to enable more compiler warnings. Compiler version # and family are auto detected, but could be overridden by defining # COMPILER_FEATURES (see config.mak.dev). You can still set@@ -1929,6 +1934,11 @@ ifdef NEED_ACCESS_ROOT_HANDLER COMPAT_OBJS += compat/access.o endif +ifdef FSMONITOR_DAEMON_BACKEND + COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND + COMPAT_OBJS += compat/fsmonitor/fsmonitor-fs-listen-$(FSMONITOR_DAEMON_BACKEND).o +endif + ifeq ($(TCLTK_PATH),) NO_TCLTK = NoThanks endifMakefile GIT-BUILD-OPTIONS: FORCE · line 2808
Quoted 6 lines
Makefile +3
-0@@ -2793,6 +2803,9 @@ GIT-BUILD-OPTIONS: FORCE @echo PAGER_ENV=\''$(subst ','\'',$(subst ','\'',$(PAGER_ENV)))'\' >>$@+ @echo DC_SHA1=\''$(subst ','\'',$(subst ','\'',$(DC_SHA1)))'\' >>$@+ @echo X=\'$(X)\' >>$@+ +ifdef FSMONITOR_DAEMON_BACKEND + @echo FSMONITOR_DAEMON_BACKEND=\''$(subst ','\'',$(subst ','\'',$(FSMONITOR_DAEMON_BACKEND)))'\' >>$@+ +endifWhy put this in an ifdef?
In342e9ef2d9e(Introduce a performance testing framework, 2012-02-17)Quoted 9 lines
we started doing that for some perf/test options (which b.t.w., I don't really see the reason for, maybe it's some subtlety in how
test-lib.shpicks those up).
But for all the other compile-time stuff we don't ifdef it, we just define it, and then you get an empty value or not.
This would AFAICT be the first build-time-for-the-C-program option we
ifdef for writing a line to GIT-BUILD-OPTIONS.(I'm going to respond here on the original question rather than on any of the follow up responses in an attempt at diffusing things a bit.)
Quoted 8 lines
I added the ifdef because I thought it to be the *most conservative* thing that I could do. The output of the generated file on unsupported platforms should be *identical* to what it was before my changes. I only alter the contents of the generated file on supported platforms.
Later, when the generated file is consumed, we don't need to worry about the effect (if any) on incremental compiles -- we will know that it won't be set -- just like it was not set in the original compile.
Okey, so e.g. when we added e.g. USE_LIBPCRE2 we added it TO GIT-BUILD-OPTIONS unconditionally, so if you pulled that commit you'd trigger a rebuild on anything that cares about GIT-BUILD-OPTIONS (which is almost everything).
But you'd like to have the line not added to avoid that one-off recompile....
Quoted 29 lines
That change appears right before a 12 other ifdef'd symbols also being written to that generated file. Most are test and perf, but some are not. But my point is that the pattern is present already.
The original question also references a 9.5 year old commit which uses the same pattern as I've used here. It also muddies the water on why it was/wasn't needed back then. And hints at possible side-effects in some of our test scripts. So it is clear that the confusion/disagreements that we are having with the current patch and whether or not to ifdef are not new.
So, is there value in being explicit and having the ifdef ??
There are well defined Make rules (and Junio gave us a very elegant little script to demonstrate that), but the subtleties are there. Especially with our use generated files like
GIT-BUILD-OPTIONS. We have a mailing list full of experts and yet this question received a lot more discussion than I thought possible or necessary, but it took a test script to demonstrate that the results are the same and it doesn't matter. Perhaps the clarity is worth it for the price of a simple ifdef.So, how much time have we (collectively) wasted discussing this subtlety ??
Quoted 7 lines
To summarize, I added the ifdef to make it explicitly clear that I'm not altering behavior on unsupported platforms. I can remove it from V4 if desired or I can keep it. (We all now know that it doesn't functionally matter -- it does however, provide clarity.)
Sorry if this sounded like a rant,
...I asked because I've looked at that ifdef soup around GIT-BUILD-OPTIONS and wondered if I could make it go away, and before a patch lands is a good time to ask "what's this pattern for?", as opposed to inferring this after the fact.
For me it was just a minor curiosity, I didn't expect to start this big discussion about it. I expected just a "oh, I just copy/pasted that from the lines at the end" or something, which would be fair enough.
I really don't care which one we go for here. If you want to change it fine, if not that's fine too.
I have noticed a pattern where you seem to really carefully consider why you'd like X over Y. I.e. it wasn't just copy/pasting in this case if I understand you correctly, but a carefully thought out decision to not do it like the other C-level-GIT-BUILD-OPTIONS.
Okey, fair enough, but that decision then doesn't go into the commit message, and then when I innocently ask about it...
..I guess I'll stop before this starts resembling a rant on my part :)
Anyway, I have also had really non-trivial comments on this fsmonitor series, not just a few bikeshed comments. I.e. the un-addressed question about the wildly different performance numbers we seem to have seen in our respective testing:
https://lore.kernel.org/git/[email protected]
I think that's much more interesting than this relatively light-reading bikeshedding I had while giving this a read-through.
PATCH v3 12/34 fsmonitor-fs-listen-macos: stub in backend for MacOS
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Stub in empty implementation of fsmonitor--daemon
backend for MacOS.
--- compat/fsmonitor/fsmonitor-fs-listen-macos.c | 20 ++++++++++++++++++++ config.mak.uname | 2 ++ contrib/buildsystems/CMakeLists.txt | 3 +++ 3 files changed, 25 insertions(+) create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-macos.c
compat/fsmonitor/fsmonitor-fs-listen-macos.c new file +20 -0
@@ -0,0 +1,20 @@ +#include "cache.h" +#include "fsmonitor.h" +#include "fsmonitor-fs-listen.h" + +int fsmonitor_fs_listen__ctor (struct fsmonitor_daemon_state *state) +{ +return -1 ; +} + +void fsmonitor_fs_listen__dtor (struct fsmonitor_daemon_state *state) +{ +} + +void fsmonitor_fs_listen__stop_async (struct fsmonitor_daemon_state *state) +{ +} + +void fsmonitor_fs_listen__loop (struct fsmonitor_daemon_state *state) +{ +}
config.mak.uname +2 -0
@@ -147,6 +147,8 @@ ifeq ($(uname_S),Darwin) MSGFMT = /usr/local/opt/gettext/bin/msgfmt endif endif + FSMONITOR_DAEMON_BACKEND = macos + BASIC_LDFLAGS += -framework CoreServices endif ifeq ($(uname_S),SunOS) NEEDS_SOCKET = YesPlease
contrib/buildsystems/CMakeLists.txt +3 -0
@@ -266,6 +266,9 @@ endif() if(CMAKE_SYSTEM_NAME STREQUAL "Windows") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsmonitor-fs-listen-win32.c) +elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") + add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) + list(APPEND compat_SOURCES compat/fsmonitor/fsmonitor-fs-listen-macos.c) endif() set(EXE_EXTENSION ${CMAKE_EXECUTABLE_SUFFIX})
Signature · 1 line
gitgitgadget
Re: [PATCH v3 12/34] fsmonitor-fs-listen-macos: stub in backend for MacOS
Quoted 44 lines · Jeff Hostetler via GitGitGadget
From: Jeff Hostetler <[email protected]>
Stub in empty implementation of fsmonitor--daemon
backend for MacOS.Signed-off-by Jeff Hostetler <[email protected]>--- compat/fsmonitor/fsmonitor-fs-listen-macos.c | 20 ++++++++++++++++++++ config.mak.uname | 2 ++ contrib/buildsystems/CMakeLists.txt | 3 +++ 3 files changed, 25 insertions(+) create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-macos.ccompat/fsmonitor/fsmonitor-fs-listen-macos.c new file +20
-0@@ -0,0 +1,20 @@ +#include "cache.h" +#include "fsmonitor.h" +#include "fsmonitor-fs-listen.h" + +int fsmonitor_fs_listen__ctor (struct fsmonitor_daemon_state *state) +{ +return -1 ; +} + +void fsmonitor_fs_listen__dtor (struct fsmonitor_daemon_state *state) +{ +} + +void fsmonitor_fs_listen__stop_async (struct fsmonitor_daemon_state *state) +{ +} + +void fsmonitor_fs_listen__loop (struct fsmonitor_daemon_state *state) +{ +}config.mak.uname +0
-0
config.mak.uname ifeq ($(uname_S),Darwin) · line 150
Quoted 4 lines
config.mak.uname +1
-0@@ -147,6 +147,8 @@ ifeq ($(uname_S),Darwin) MSGFMT = /usr/local/opt/gettext/bin/msgfmt endif endif + FSMONITOR_DAEMON_BACKEND = macos
A rather trivial point, but can't we pick one of "macos" or "darwin" (I'd think going with the existing uname is better) and name the file after the uname (or lower-case thereof)?
Makes these make rules more consistent too, we could just set this to "YesPlease" here, and then lower case the uname for the file compilation/include.
Re: [PATCH v3 12/34] fsmonitor-fs-listen-macos: stub in backend for MacOS
Hi Ævar,
Quoted 55 lines · Ævar Arnfjörð Bjarmason
Quoted 44 lines · Jeff Hostetler via GitGitGadget
From: Jeff Hostetler <[email protected]>
Stub in empty implementation of fsmonitor--daemon
backend for MacOS.Signed-off-by Jeff Hostetler <[email protected]>--- compat/fsmonitor/fsmonitor-fs-listen-macos.c | 20 ++++++++++++++++++++ config.mak.uname | 2 ++ contrib/buildsystems/CMakeLists.txt | 3 +++ 3 files changed, 25 insertions(+) create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-macos.ccompat/fsmonitor/fsmonitor-fs-listen-macos.c new file +20
-0@@ -0,0 +1,20 @@ +#include "cache.h" +#include "fsmonitor.h" +#include "fsmonitor-fs-listen.h" + +int fsmonitor_fs_listen__ctor (struct fsmonitor_daemon_state *state) +{ +return -1 ; +} + +void fsmonitor_fs_listen__dtor (struct fsmonitor_daemon_state *state) +{ +} + +void fsmonitor_fs_listen__stop_async (struct fsmonitor_daemon_state *state) +{ +} + +void fsmonitor_fs_listen__loop (struct fsmonitor_daemon_state *state) +{ +}config.mak.uname +0
-0config.mak.uname ifeq ($(uname_S),Darwin) · line 150
Quoted 4 lines
config.mak.uname +1
-0@@ -147,6 +147,8 @@ ifeq ($(uname_S),Darwin) MSGFMT = /usr/local/opt/gettext/bin/msgfmt endif endif + FSMONITOR_DAEMON_BACKEND = macosA rather trivial point, but can't we pick one of "macos" or "darwin" (I'd think going with the existing uname is better) and name the file after the uname (or lower-case thereof)?
Quoted 3 lines
Makes these make rules more consistent too, we could just set this to "YesPlease" here, and then lower case the uname for the file compilation/include.
So you suggest that we name the new stuff after an uname that reflects a name that is no longer relevant? I haven't seen a real Darwin system in quite a long time, have you?
I don't find such a suggestion constructive, I have to admit.
Ciao,
Johannes
Re: [PATCH v3 12/34] fsmonitor-fs-listen-macos: stub in backend for MacOS
Quoted 59 lines · Johannes Schindelin
Hi Ævar,
Quoted 55 lines · Ævar Arnfjörð Bjarmason
Quoted 44 lines · Jeff Hostetler via GitGitGadget
From: Jeff Hostetler <[email protected]>
Stub in empty implementation of fsmonitor--daemon
backend for MacOS.Signed-off-by Jeff Hostetler <[email protected]>--- compat/fsmonitor/fsmonitor-fs-listen-macos.c | 20 ++++++++++++++++++++ config.mak.uname | 2 ++ contrib/buildsystems/CMakeLists.txt | 3 +++ 3 files changed, 25 insertions(+) create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-macos.ccompat/fsmonitor/fsmonitor-fs-listen-macos.c new file +20
-0@@ -0,0 +1,20 @@ +#include "cache.h" +#include "fsmonitor.h" +#include "fsmonitor-fs-listen.h" + +int fsmonitor_fs_listen__ctor (struct fsmonitor_daemon_state *state) +{ +return -1 ; +} + +void fsmonitor_fs_listen__dtor (struct fsmonitor_daemon_state *state) +{ +} + +void fsmonitor_fs_listen__stop_async (struct fsmonitor_daemon_state *state) +{ +} + +void fsmonitor_fs_listen__loop (struct fsmonitor_daemon_state *state) +{ +}config.mak.uname +0
-0config.mak.uname ifeq ($(uname_S),Darwin) · line 150
Quoted 4 lines
config.mak.uname +1
-0@@ -147,6 +147,8 @@ ifeq ($(uname_S),Darwin) MSGFMT = /usr/local/opt/gettext/bin/msgfmt endif endif + FSMONITOR_DAEMON_BACKEND = macosA rather trivial point, but can't we pick one of "macos" or "darwin" (I'd think going with the existing uname is better) and name the file after the uname (or lower-case thereof)?
Quoted 7 lines
Quoted 3 lines
Makes these make rules more consistent too, we could just set this to "YesPlease" here, and then lower case the uname for the file compilation/include.
So you suggest that we name the new stuff after an
unamethat reflects a name that is no longer relevant? I haven't seen a real Darwin system in quite a long time, have you?
It's not current? On an Mac Mini M1 which got released this year:
% uname -s
Darwin
We then have the same in config.mak.uname, it seemed the most obvious and consistent to carry that through to file inclusion.
Re: [PATCH v3 12/34] fsmonitor-fs-listen-macos: stub in backend for MacOS
Hi Ævar,
Quoted 6 lines · Ævar Arnfjörð Bjarmason
Quoted 3 lines · Johannes Schindelin
So you suggest that we name the new stuff after an
unamethat reflects a name that is no longer relevant? I haven't seen a real Darwin system in quite a long time, have you?
Quoted 7 lines
It's not current? On an Mac Mini M1 which got released this year:
% uname -s
DarwinWe then have the same in config.mak.uname, it seemed the most obvious and consistent to carry that through to file inclusion.
Sorry. I assumed that you knew that Darwin was the name for an open source Operating System. See
https://en.wikipedia.org/wiki/Darwin_%28operating_system%29 for more details.
Ciao,
Johannes
Re: [PATCH v3 12/34] fsmonitor-fs-listen-macos: stub in backend for MacOS
Quoted 18 lines · Johannes Schindelin
Hi Ævar,
Quoted 6 lines · Ævar Arnfjörð Bjarmason
Quoted 3 lines · Johannes Schindelin
So you suggest that we name the new stuff after an
unamethat reflects a name that is no longer relevant? I haven't seen a real Darwin system in quite a long time, have you?Quoted 7 lines
It's not current? On an Mac Mini M1 which got released this year:
% uname -s
DarwinWe then have the same in config.mak.uname, it seemed the most obvious and consistent to carry that through to file inclusion.
Quoted 7 lines
Sorry. I assumed that you knew that Darwin was the name for an open source Operating System. See
https://en.wikipedia.org/wiki/Darwin_%28operating_system%29 for more details.Ciao,
Johannes
Sorry, but I do not see that you are being more constructive than the other party, whom you blame to be not constructive, in this exchange.
The part of the file that the patch applies to uses $(uname_S) to implement platform specific special cases, and we are looking at
ifeq ($(uname_S),Darwin) ... FSMONITOR_DAEMON_BACKEND = macos ... endif
I find it a fair question why the name used there has to be different from the one we can automatically and mechanically get out of "uname -s".
Then you respond that uname output is no longer relevant because Darwin is a name that is no longer relevant? And when asked why the name is no longer relevant, you make a sniding comment implying that the other party does not know the name is an operating system?
What is going on here?
It does not really matter how "Darwin" is described in an encyclopedia in the context of this discussion. What matters is that it is what the system's "uname -s" currently uses to identify itself, and what we guard the section of makefile snippet with, isn't it?
ci/lib.sh seems to have an attempt to unify/translate among these names, and
* on azure-pipelines, it wants to translate darwin to osx
* on github-actions, it wants to translate macos to osx
Presumably that is because these two systems call the platform with these two different names, and you want to pick a middle ground that nobody uses to be neutral, or something?
Also, in contrib/vscode/init.sh, I see Darwin obtained from "uname -s" gets translated to "macOS".
In any case, if your argument was "we picked macos because we use the same token elsewhere, while trying to translate away from Darwin as much as possible for such and such reasons", I would have found it a productive exchange, but unfortunately that is not what I am seeing here.
Re: [PATCH v3 12/34] fsmonitor-fs-listen-macos: stub in backend for MacOS
Quoted 8 lines · Junio C Hamano
Quoted 6 lines · Johannes Schindelin
Hi Ævar,
Quoted 2 lines · Ævar Arnfjörð Bjarmason
On Fri, Jul 16 2021, Johannes Schindelin wrote:
...
I'm not sure that there is a "correct" answer here, but for the sake of harmony, in V4 I'll set this to "darwin" and update the name of the backend driver source file to match. So that we are consistently using 1 term throughout "Makefile" and "config.mak.uname".
ifeq ($(uname_S),Darwin) ... FSMONITOR_DAEMON_BACKEND = darwin endif
FWIW, I suspect that it is not worth the effort to directly set the backend name from $(uname_S). For example, on Windows we currently have two different uname values depending on which compiler is being used.
ifeq ($(uname_S),Windows) ... FSMONITOR_DAEMON_BACKEND = win32 endif ifneq (,$(findstring MINGW,$(uname_S))) ... FSMONITOR_DAEMON_BACKEND = win32 endif
Also, since the backend layer is highly platform-specific, it may be a while (if ever) before we have universal coverage for all platforms. Until then, we can simply set $FSMONITOR_DAEMON_BACKEND to a literal value on a platform-by-platform basis as support is added.
Thanks,
Jeff
Re: [PATCH v3 12/34] fsmonitor-fs-listen-macos: stub in backend for MacOS
Quoted 39 lines · Jeff Hostetler
Quoted 8 lines · Junio C Hamano
Quoted 6 lines · Johannes Schindelin
Hi Ævar,
Quoted 2 lines · Ævar Arnfjörð Bjarmason
On Fri, Jul 16 2021, Johannes Schindelin wrote:
...
I'm not sure that there is a "correct" answer here, but for the sake of harmony, in V4 I'll set this to "darwin" and update the name of the backend driver source file to match. So that we are consistently using 1 term throughout "
Makefile" and "config.mak.uname".ifeq ($(uname_S),Darwin) ... FSMONITOR_DAEMON_BACKEND = darwin endifFWIW, I suspect that it is not worth the effort to directly set the backend name from $(uname_S). For example, on Windows we currently have two different uname values depending on which compiler is being used.
ifeq ($(uname_S),Windows) ... FSMONITOR_DAEMON_BACKEND = win32 endif ifneq (,$(findstring MINGW,$(uname_S))) ... FSMONITOR_DAEMON_BACKEND = win32 endif
Quoted 4 lines
Also, since the backend layer is highly platform-specific, it may be a while (if ever) before we have universal coverage for all platforms. Until then, we can simply set $FSMONITOR_DAEMON_BACKEND to a literal value on a platform-by-platform basis as support is added.
Re "harmony": For what it's worth I don't think you should change it on my accord.
I should probably have more explicitly said (but I've also been trying to check the general verbosity of my E-Mails), that when I read a series like this and have some general trivial comments like this, I mean them as something like:
Just a thought while reading this through, i.e. a person familiar with the general codebase but not necessarily your specific are. Maybe this suggestion makes things easier/simpler, but if you think not and decide not to take the suggestion that's fine too.
I.e. that along with the general implicit suggestion that I'd say applies in general on list that if someone is perplexed by a patch by default that's a comment on the commit message.
That person (i.e. me in this case) could also just be hopelessly confused & nothing needs to change. When I get comments like that I sometimes change things, sometimes not. You should do the same.
As noted in another reply on this general thread & what's cooking I seem to have poked a bit of a hornet's nest here that I wasn't expecting to poke. I'd not been following earlier rounds of this topic, and didn't know that it had (seemingly) reached some phase of critical updates only in the minds of its authors.
Re: [PATCH v3 12/34] fsmonitor-fs-listen-macos: stub in backend for MacOS
In reply to Jeff HostetlerQuoted 10 lines · Jeff Hostetler
I'm not sure that there is a "correct" answer here, but for the sake of harmony, in V4 I'll set this to "darwin" and update the name of the backend driver source file to match. So that we are consistently using 1 term throughout "
Makefile" and "config.mak.uname".ifeq ($(uname_S),Darwin) ... FSMONITOR_DAEMON_BACKEND = darwin endif...
I do not think it would help "harmony" to change the name, but any one of 'darwin', 'macos' and 'osx' would be fine.
It was irritating to see a simple "why is this particular name chosen?" answered with such a hostility, when even a "we have no deep reasoning behind the choice of the name. It is only seen by developers in names of the source files, and it does not make much difference" would have been sufficient. I somehow view it more problematic.
I guess the blame goes both ways, though. We all have worked with each other long enough to know which of your recipients are prone to go overly defensive when asked questions, and we should know that it would help to prepend "I am just being curious but..." to your questions whose answers do not make a huge difference at the end either way (or not asking such questions at all).
Thanks.
Re: [PATCH v3 12/34] fsmonitor-fs-listen-macos: stub in backend for MacOS
Quoted 6 lines · Junio C Hamano
I guess the blame goes both ways, though. We all have worked with each other long enough to know which of your recipients are prone to go overly defensive when asked questions, and we should know that it would help to prepend "I am just being curious but..." to your questions whose answers do not make a huge difference at the end either way (or not asking such questions at all).
I disagree. It's not OK for contributors to react defensively when asked questions, and in particular I don't think it's OK that some contributors are punished for merely disagreeing, while others are given a pass for snarling at feedback. This is double standards.
The Git project should not play favorites, and all contributors should be asked to assume good faith.
https://en.wikipedia.org/wiki/Wikipedia:Assume_good_faith
It should be implied that the feedback given is to try to improve the project, it should not need to be stated.
Signature · 1 line
Felipe Contreras
PATCH v3 13/34 fsmonitor--daemon: implement 'run' command
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Implement run command to try to begin listening for file system events.
This version defines the thread structure with a single fsmonitor_fs_listen thread to watch for file system events and a simple IPC thread pool to watch for connection from Git clients over a well-known named pipe or Unix domain socket.
This commit does not actually do anything yet because the platform backends are still just stubs.
--- builtin/fsmonitor--daemon.c | 210 +++++++++++++++++++++++++++++++++++- fsmonitor--daemon.h | 34 ++++++ 2 files changed, 243 insertions(+), 1 deletion(-) create mode 100644 fsmonitor--daemon.h
builtin/fsmonitor--daemon.c +209 -1
@@ -3,16 +3,39 @@#include "parse-options.h" #include "fsmonitor.h" #include "fsmonitor-ipc.h" +#include "compat/fsmonitor/fsmonitor-fs-listen.h" +#include "fsmonitor--daemon.h" #include "simple-ipc.h" #include "khash.h" static const char *const builtin_fsmonitor__daemon_usage[] = { +N_ ("git fsmonitor--daemon run [<options>]" ),N_ ("git fsmonitor--daemon stop" ),N_ ("git fsmonitor--daemon status" ),NULL };#ifdef HAVE_FSMONITOR_DAEMON_BACKEND +/* +* Global state loaded from config. +*/ +#define FSMONITOR__IPC_THREADS "fsmonitor.ipcthreads" +static int fsmonitor__ipc_threads =8 ; + +static int fsmonitor_config (const char *var,const char *value,void *cb) +{ +if (!strcmp (var,FSMONITOR__IPC_THREADS )) { +int i =git_config_int (var, value); +if (i <1 ) +return error (_ ("value of '%s' out of range: %d" ), +FSMONITOR__IPC_THREADS , i); + fsmonitor__ipc_threads = i; +return 0 ; + } + +return git_default_config (var, value, cb); +} +/* * Acting as a CLIENT. *
@@ -57,11 +80,190 @@ static int do_as_client__status(void) } } +static ipc_server_application_cb handle_client; + +static int handle_client (void *data, +const char *command,size_t command_len, +ipc_server_reply_cb *reply, +struct ipc_server_reply_data *reply_data) +{ +/* struct fsmonitor_daemon_state *state = data; */ +int result; + +/* +* The Simple IPC API now supports {char*, len} arguments, but +* FSMonitor always uses proper null-terminated strings, so +* we can ignore the command_len argument. (Trust, but verify.) +*/ +if (command_len !=strlen (command)) +BUG ("FSMonitor assumes text messages" ); + +trace2_region_enter ("fsmonitor" ,"handle_client" , the_repository); +trace2_data_string ("fsmonitor" , the_repository,"request" , command); + + result =0 ;/* TODO Do something here. */ + +trace2_region_leave ("fsmonitor" ,"handle_client" , the_repository); + +return result; +} + +static void *fsmonitor_fs_listen__thread_proc (void *_state) +{ +struct fsmonitor_daemon_state *state = _state; + +trace2_thread_start ("fsm-listen" ); + +trace_printf_key (&trace_fsmonitor,"Watching: worktree '%s'" , + state->path_worktree_watch.buf); +if (state->nr_paths_watching >1 ) +trace_printf_key (&trace_fsmonitor,"Watching: gitdir '%s'" , + state->path_gitdir_watch.buf); + +fsmonitor_fs_listen__loop (state); + +trace2_thread_exit (); +return NULL ; +} + +static int fsmonitor_run_daemon_1 (struct fsmonitor_daemon_state *state) +{ +struct ipc_server_opts ipc_opts = { + .nr_threads = fsmonitor__ipc_threads, + +/* +* We know that there are no other active threads yet, +* so we can let the IPC layer temporarily chdir() if +* it needs to when creating the server side of the +* Unix domain socket. +*/ + .uds_disallow_chdir =0 + }; + +/* +* Start the IPC thread pool before the we've started the file +* system event listener thread so that we have the IPC handle +* before we need it. +*/ +if (ipc_server_run_async (&state->ipc_server_data, +fsmonitor_ipc__get_path (), &ipc_opts, + handle_client, state)) +return error (_ ("could not start IPC thread pool" )); + +/* +* Start the fsmonitor listener thread to collect filesystem +* events. +*/ +if (pthread_create (&state->listener_thread,NULL , + fsmonitor_fs_listen__thread_proc, state) <0 ) { +ipc_server_stop_async (state->ipc_server_data); +ipc_server_await (state->ipc_server_data); + +return error (_ ("could not start fsmonitor listener thread" )); + } + +/* +* The daemon is now fully functional in background threads. +* Wait for the IPC thread pool to shutdown (whether by client +* request or from filesystem activity). +*/ +ipc_server_await (state->ipc_server_data); + +/* +* The fsmonitor listener thread may have received a shutdown +* event from the IPC thread pool, but it doesn't hurt to tell +* it again. And wait for it to shutdown. +*/ +fsmonitor_fs_listen__stop_async (state); +pthread_join (state->listener_thread,NULL ); + +return state->error_code; +} + +static int fsmonitor_run_daemon (void ) +{ +struct fsmonitor_daemon_state state; +int err; + +memset (&state,0 ,sizeof (state)); + +pthread_mutex_init (&state.main_lock,NULL ); + state.error_code =0 ; + state.current_token_data =NULL ; + +/* Prepare to (recursively) watch the <worktree-root> directory. */ +strbuf_init (&state.path_worktree_watch,0 ); +strbuf_addstr (&state.path_worktree_watch,absolute_path (get_git_work_tree ())); + state.nr_paths_watching =1 ; + +/* +* We create and delete cookie files somewhere inside the .git +* directory to help us keep sync with the file system. If +* ".git" is not a directory, then <gitdir> is not inside the +* cone of <worktree-root>, so set up a second watch to watch +* the <gitdir> so that we get events for the cookie files. +*/ +strbuf_init (&state.path_gitdir_watch,0 ); +strbuf_addbuf (&state.path_gitdir_watch, &state.path_worktree_watch); +strbuf_addstr (&state.path_gitdir_watch,"/.git" ); +if (!is_directory (state.path_gitdir_watch.buf)) { +strbuf_reset (&state.path_gitdir_watch); +strbuf_addstr (&state.path_gitdir_watch,absolute_path (get_git_dir ())); + state.nr_paths_watching =2 ; + } + +/* +* Confirm that we can create platform-specific resources for the +* filesystem listener before we bother starting all the threads. +*/ +if (fsmonitor_fs_listen__ctor (&state)) { + err =error (_ ("could not initialize listener thread" )); + goto done; + } + + err =fsmonitor_run_daemon_1 (&state); + +done: +pthread_mutex_destroy (&state.main_lock); +fsmonitor_fs_listen__dtor (&state); + +ipc_server_free (state.ipc_server_data); + +strbuf_release (&state.path_worktree_watch); +strbuf_release (&state.path_gitdir_watch); + +return err; +} + +static int try_to_run_foreground_daemon (void ) +{ +/* +* Technically, we don't need to probe for an existing daemon +* process, since we could just call `fsmonitor_run_daemon()` +* and let it fail if the pipe/socket is busy. +* +* However, this method gives us a nicer error message for a +* common error case. +*/ +if (fsmonitor_ipc__get_state () ==IPC_STATE__LISTENING ) +die ("fsmonitor--daemon is already running '%s'" , + the_repository->worktree); + +printf (_ ("running fsmonitor-daemon in '%s'\n" ), + the_repository->worktree); +fflush (stdout); + +return !!fsmonitor_run_daemon (); +} +int cmd_fsmonitor__daemon (int argc,const char **argv,const char *prefix) {const char *subcmd;struct option options[] = { +OPT_INTEGER (0 ,"ipc-threads" , + &fsmonitor__ipc_threads, +N_ ("use <n> ipc worker threads" )),OPT_END () };
@@ -71,7 +273,7 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)if (argc ==2 && !strcmp (argv[1 ],"-h" ))usage_with_options (builtin_fsmonitor__daemon_usage, options); -git_config (git_default_config,NULL ); +git_config (fsmonitor_config,NULL ); subcmd = argv[1 ]; argv--;
@@ -79,6 +281,12 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) argc =parse_options (argc, argv, prefix, options, builtin_fsmonitor__daemon_usage,0 ); +if (fsmonitor__ipc_threads <1 ) +die (_ ("invalid 'ipc-threads' value (%d)" ), + fsmonitor__ipc_threads); + +if (!strcmp (subcmd,"run" )) +return !!try_to_run_foreground_daemon ();if (!strcmp (subcmd,"stop" ))return !!do_as_client__send_stop ();
fsmonitor--daemon.h new file +34 -0
@@ -0,0 +1,34 @@ +#ifndef FSMONITOR_DAEMON_H +#define FSMONITOR_DAEMON_H + +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND + +#include "cache.h" +#include "dir.h" +#include "run-command.h" +#include "simple-ipc.h" +#include "thread-utils.h" + +struct fsmonitor_batch ; +struct fsmonitor_token_data ; + +struct fsmonitor_daemon_backend_data ;/* opaque platform-specific data */ + +struct fsmonitor_daemon_state { +pthread_t listener_thread; +pthread_mutex_t main_lock; + +struct strbuf path_worktree_watch; +struct strbuf path_gitdir_watch; +int nr_paths_watching; + +struct fsmonitor_token_data *current_token_data; + +int error_code; +struct fsmonitor_daemon_backend_data *backend_data; + +struct ipc_server_data *ipc_server_data; +}; + +#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */ +#endif /* FSMONITOR_DAEMON_H */
Signature · 1 line
gitgitgadget
PATCH v3 14/34 fsmonitor--daemon: implement 'start' command
In reply to Jeff Hostetler via GitGitGadgetFrom: Jeff Hostetler <[email protected]>
Implement 'git fsmonitor--daemon start' command. This command tries to start a daemon in the background. It creates a background process to run the daemon.
The updated daemon does not actually do anything yet because the platform backends are still just stubs.
--- builtin/fsmonitor--daemon.c | 208 ++++++++++++++++++++++++++++++++++++ 1 file changed, 208 insertions(+)
builtin/fsmonitor--daemon.c +208 -0
@@ -9,6 +9,7 @@#include "khash.h" static const char *const builtin_fsmonitor__daemon_usage[] = { +N_ ("git fsmonitor--daemon start [<options>]" ),N_ ("git fsmonitor--daemon run [<options>]" ),N_ ("git fsmonitor--daemon stop" ),N_ ("git fsmonitor--daemon status" ),
@@ -22,6 +23,9 @@ static const char * const builtin_fsmonitor__daemon_usage[] = {#define FSMONITOR__IPC_THREADS "fsmonitor.ipcthreads"static int fsmonitor__ipc_threads =8 ; +#define FSMONITOR__START_TIMEOUT "fsmonitor.starttimeout" +static int fsmonitor__start_timeout_sec =60 ; +static int fsmonitor_config (const char *var,const char *value,void *cb) {if (!strcmp (var,FSMONITOR__IPC_THREADS )) {
@@ -33,6 +37,15 @@ static int fsmonitor_config(const char *var, const char *value, void *cb)return 0 ; } +if (!strcmp (var,FSMONITOR__START_TIMEOUT )) { +int i =git_config_int (var, value); +if (i <0 ) +return error (_ ("value of '%s' out of range: %d" ), +FSMONITOR__START_TIMEOUT , i); + fsmonitor__start_timeout_sec = i; +return 0 ; + } +return git_default_config (var, value, cb); }
@@ -256,6 +269,194 @@ static int try_to_run_foreground_daemon(void)return !!fsmonitor_run_daemon (); } +#ifdef GIT_WINDOWS_NATIVE +/* +* Create a background process to run the daemon. It should be completely +* disassociated from the terminal. +* +* Conceptually like `daemonize()` but different because Windows does not +* have `fork(2)`. Spawn a normal Windows child process but without the +* limitations of `start_command()` and `finish_command()`. +* +* The child process execs the "git fsmonitor--daemon run" command. +* +* The current process returns so that the caller can wait for the child +* to startup before exiting. +*/ +static int spawn_background_fsmonitor_daemon (pid_t *pid) +{ +char git_exe[MAX_PATH ]; +struct strvec args =STRVEC_INIT ; +int in, out; + +GetModuleFileNameA (NULL , git_exe,MAX_PATH ); + + in =open ("/dev/null" ,O_RDONLY ); + out =open ("/dev/null" ,O_WRONLY ); + +strvec_push (&args, git_exe); +strvec_push (&args,"fsmonitor--daemon" ); +strvec_push (&args,"run" ); +strvec_pushf (&args,"--ipc-threads=%d" , fsmonitor__ipc_threads); + + *pid =mingw_spawnvpe (args.v[0 ], args.v,NULL ,NULL , in, out, out); +close (in); +close (out); + +strvec_clear (&args); + +if (*pid <0 ) +return error (_ ("could not spawn fsmonitor--daemon in the background" )); + +return 0 ; +} +#else +/* +* Create a background process to run the daemon. It should be completely +* disassociated from the terminal. +* +* This is adapted from `daemonize()`. Use `fork()` to directly +* create and run the daemon in the child process. +* +* The fork-child can just call the run code; it does not need to exec +* it. +* +* The fork-parent returns the child PID so that we can wait for the +* child to startup before exiting. +*/ +static int spawn_background_fsmonitor_daemon (pid_t *pid) +{ + *pid =fork (); + +switch (*pid) { +case 0 : +if (setsid () ==-1 ) +error_errno (_ ("setsid failed" )); +close (0 ); +close (1 ); +close (2 ); +sanitize_stdfds (); + +return !!fsmonitor_run_daemon (); + +case -1 : +return error_errno (_ ("could not spawn fsmonitor--daemon in the background" )); + +default : +return 0 ; + } +} +#endif + +/* +* This is adapted from `wait_or_whine()`. Watch the child process and +* let it get started and begin listening for requests on the socket +* before reporting our success. +*/ +static int wait_for_background_startup (pid_t pid_child) +{ +int status; +pid_t pid_seen; +enum ipc_active_state s; +time_t time_limit, now; + +time (&time_limit); + time_limit += fsmonitor__start_timeout_sec; + +for (;;) { + pid_seen =waitpid (pid_child, &status,WNOHANG ); + +if (pid_seen ==-1 ) +return error_errno (_ ("waitpid failed" )); +else if (pid_seen ==0 ) { +/* +* The child is still running (this should be +* the normal case). Try to connect to it on +* the socket and see if it is ready for +* business. +* +* If there is another daemon already running, +* our child will fail to start (possibly +* after a timeout on the lock), but we don't +* care (who responds) if the socket is live. +*/ + s =fsmonitor_ipc__get_state (); +if (s ==IPC_STATE__LISTENING ) +return 0 ; + +time (&now); +if (now > time_limit) +return error (_ ("fsmonitor--daemon not online yet" )); + }else if (pid_seen == pid_child) { +/* +* The new child daemon process shutdown while +* it was starting up, so it is not listening +* on the socket. +* +* Try to ping the socket in the odd chance +* that another daemon started (or was already +* running) while our child was starting. +* +* Again, we don't care who services the socket. +*/ + s =fsmonitor_ipc__get_state (); +if (s ==IPC_STATE__LISTENING ) +return 0 ; + +/* +* We don't care about the WEXITSTATUS() nor +* any of the WIF*(status) values because +* `cmd_fsmonitor__daemon()` does the `!!result` +* trick on all function return values. +* +* So it is sufficient to just report the +* early shutdown as an error. +*/ +return error (_ ("fsmonitor--daemon failed to start" )); + }else +return error (_ ("waitpid is confused" )); + } +} + +static int try_to_start_background_daemon (void ) +{ +pid_t pid_child; +int ret; + +/* +* Before we try to create a background daemon process, see +* if a daemon process is already listening. This makes it +* easier for us to report an already-listening error to the +* console, since our spawn/daemon can only report the success +* of creating the background process (and not whether it +* immediately exited). +*/ +if (fsmonitor_ipc__get_state () ==IPC_STATE__LISTENING ) +die ("fsmonitor--daemon is already running '%s'" , + the_repository->worktree); + +printf (_ ("starting fsmonitor-daemon in '%s'\n" ), + the_repository->worktree); +fflush (stdout); + +/* +* Run the actual daemon in a background process. +*/ + ret =spawn_background_fsmonitor_daemon (&pid_child); +if (pid_child <=0 ) +return ret; + +/* +* Wait (with timeout) for the background child process get +* started and begin listening on the socket/pipe. This makes +* the "start" command more synchronous and more reliable in +* tests. +*/ + ret =wait_for_background_startup (pid_child); + +return ret; +} +int cmd_fsmonitor__daemon (int argc,const char **argv,const char *prefix) {const char *subcmd;
@@ -264,6 +465,10 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)OPT_INTEGER (0 ,"ipc-threads" , &fsmonitor__ipc_threads,N_ ("use <n> ipc worker threads" )), +OPT_INTEGER (0 ,"start-timeout" , + &fsmonitor__start_timeout_sec, +N_ ("Max seconds to wait for background daemon startup" )), + OPT_END() };
@@ -285,6 +490,9 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)die (_ ("invalid 'ipc-threads' value (%d)" ), fsmonitor__ipc_threads); +if (!strcmp (subcmd,"start" )) +return !!try_to_start_background_daemon (); +if (!strcmp (subcmd,"run" ))return !!try_to_run_foreground_daemon ();
Signature · 1 line
gitgitgadget
Re: [PATCH v3 14/34] fsmonitor--daemon: implement 'start' command
Quoted 68 lines · Jeff Hostetler via GitGitGadget
builtin/fsmonitor--daemon.c +68
-0+#ifdef GIT_WINDOWS_NATIVE +/* +* Create a background process to run the daemon. It should be completely +* disassociated from the terminal. +* +* Conceptually like `daemonize()` but different because Windows does not +* have `fork(2)`. Spawn a normal Windows child process but without the +* limitations of `start_command()` and `finish_command()`. +* +* The child process execs the "git fsmonitor--daemon run" command. +* +* The current process returns so that the caller can wait for the child +* to startup before exiting. +*/ +static int spawn_background_fsmonitor_daemon (pid_t *pid) +{ +char git_exe[MAX_PATH ]; +struct strvec args =STRVEC_INIT ; +int in, out; + +GetModuleFileNameA (NULL , git_exe,MAX_PATH ); + + in =open ("/dev/null" ,O_RDONLY ); + out =open ("/dev/null" ,O_WRONLY ); + +strvec_push (&args, git_exe); +strvec_push (&args,"fsmonitor--daemon" ); +strvec_push (&args,"run" ); +strvec_pushf (&args,"--ipc-threads=%d" , fsmonitor__ipc_threads); + + *pid =mingw_spawnvpe (args.v[0 ], args.v,NULL ,NULL , in, out, out); +close (in); +close (out); + +strvec_clear (&args); + +if (*pid <0 ) +return error (_ ("could not spawn fsmonitor--daemon in the background" )); + +return 0 ; +} +#else +/* +* Create a background process to run the daemon. It should be completely +* disassociated from the terminal. +* +* This is adapted from `daemonize()`. Use `fork()` to directly +* create and run the daemon in the child process. +* +* The fork-child can just call the run code; it does not need to exec +* it. +* +* The fork-parent returns the child PID so that we can wait for the +* child to startup before exiting. +*/ +static int spawn_background_fsmonitor_daemon (pid_t *pid) +{ + *pid =fork (); + +switch (*pid) { +case 0 : +if (setsid () ==-1 ) +error_errno (_ ("setsid failed" )); +close (0 ); +close (1 ); +close (2 ); +sanitize_stdfds (); +
builtin/fsmonitor--daemon.c static int try_to_run_foreground_daemon(void) · line 349
Quoted 10 lines
builtin/fsmonitor--daemon.c +10
-0+return !!fsmonitor_run_daemon (); + +case -1 : +return error_errno (_ ("could not spawn fsmonitor--daemon in the background" )); + +default : + return 0; + } +} +#endif
The spawn_background_fsmonitor_daemon() function here is almost the same as daemonize(). I wonder if this & the Windows-specific one you have here can't be refactored into an API from what's now in setup.c.
Then we could make builtin/gc.c and daemon.c use that, so Windows could have background GC, and we'd have a more battle-tested central codepath for this tricky bit.
It seems to me like the only limitations on it are to have this return slightly more general things (e.g. not set its own errors, return structured data), and maybe some callback for what to do in the child/parent.
Quoted 59 lines
builtin/fsmonitor--daemon.c +59
-0+/* +* This is adapted from `wait_or_whine()`. Watch the child process and +* let it get started and begin listening for requests on the socket +* before reporting our success. +*/ +static int wait_for_background_startup (pid_t pid_child) +{ +int status; +pid_t pid_seen; +enum ipc_active_state s; +time_t time_limit, now; + +time (&time_limit); + time_limit += fsmonitor__start_timeout_sec; + +for (;;) { + pid_seen =waitpid (pid_child, &status,WNOHANG ); + +if (pid_seen ==-1 ) +return error_errno (_ ("waitpid failed" )); +else if (pid_seen ==0 ) { +/* +* The child is still running (this should be +* the normal case). Try to connect to it on +* the socket and see if it is ready for +* business. +* +* If there is another daemon already running, +* our child will fail to start (possibly +* after a timeout on the lock), but we don't +* care (who responds) if the socket is live. +*/ + s =fsmonitor_ipc__get_state (); +if (s ==IPC_STATE__LISTENING ) +return 0 ; + +time (&now); +if (now > time_limit) +return error (_ ("fsmonitor--daemon not online yet" )); + }else if (pid_seen == pid_child) { +/* +* The new child daemon process shutdown while +* it was starting up, so it is not listening +* on the socket. +* +* Try to ping the socket in the odd chance +* that another daemon started (or was already +* running) while our child was starting. +* +* Again, we don't care who services the socket. +*/ + s =fsmonitor_ipc__get_state (); +if (s ==IPC_STATE__LISTENING ) +return 0 ; + +/* +* We don't care about the WEXITSTATUS() nor +* any of the WIF*(status) values because +* `cmd_fsmonitor__daemon()` does the `!!result`
builtin/fsmonitor--daemon.c static int try_to_run_foreground_daemon(void) · line 419
Quoted 10 lines
builtin/fsmonitor--daemon.c +10
-0+* trick on all function return values. +* +* So it is sufficient to just report the +* early shutdown as an error. +*/ +return error (_ ("fsmonitor--daemon failed to start" )); + }else +return error (_ ("waitpid is confused" )); + } +}
Ditto this. could we extend the wait_or_whine() function (or some extended version thereof) to do what you need with callbacks?
It seems the main difference is just being able to pass down a flag for waitpid(), and the loop needing to check EINTR or not depending on whether WNOHANG is passed.
For e.g. the "We don't care about the WEXITSTATUS()" you'd get that behavior with an adjusted wait_or_whine(). Wouldn't it be better to report what exit status it exits with e.g. if the top-level process is signalled? We do so in trace2 for other things we spawn...
Re: [PATCH v3 14/34] fsmonitor--daemon: implement 'start' command
Hi Ævar,
Quoted 75 lines · Ævar Arnfjörð Bjarmason
Quoted 73 lines · Jeff Hostetler via GitGitGadget
builtin/fsmonitor--daemon.c +73
-0+#ifdef GIT_WINDOWS_NATIVE +/* +* Create a background process to run the daemon. It should be completely +* disassociated from the terminal. +* +* Conceptually like `daemonize()` but different because Windows does not +* have `fork(2)`. Spawn a normal Windows child process but without the +* limitations of `start_command()` and `finish_command()`. +* +* The child process execs the "git fsmonitor--daemon run" command. +* +* The current process returns so that the caller can wait for the child +* to startup before exiting. +*/ +static int spawn_background_fsmonitor_daemon (pid_t *pid) +{ +char git_exe[MAX_PATH ]; +struct strvec args =STRVEC_INIT ; +int in, out; + +GetModuleFileNameA (NULL , git_exe,MAX_PATH ); + + in =open ("/dev/null" ,O_RDONLY ); + out =open ("/dev/null" ,O_WRONLY ); + +strvec_push (&args, git_exe); +strvec_push (&args,"fsmonitor--daemon" ); +strvec_push (&args,"run" ); +strvec_pushf (&args,"--ipc-threads=%d" , fsmonitor__ipc_threads); + + *pid =mingw_spawnvpe (args.v[0 ], args.v,NULL ,NULL , in, out, out); +close (in); +close (out); + +strvec_clear (&args); + +if (*pid <0 ) +return error (_ ("could not spawn fsmonitor--daemon in the background" )); + +return 0 ; +} +#else +/* +* Create a background process to run the daemon. It should be completely +* disassociated from the terminal. +* +* This is adapted from `daemonize()`. Use `fork()` to directly +* create and run the daemon in the child process. +* +* The fork-child can just call the run code; it does not need to exec +* it. +* +* The fork-parent returns the child PID so that we can wait for the +* child to startup before exiting. +*/ +static int spawn_background_fsmonitor_daemon (pid_t *pid) +{ + *pid =fork (); + +switch (*pid) { +case 0 : +if (setsid () ==-1 ) +error_errno (_ ("setsid failed" )); +close (0 ); +close (1 ); +close (2 ); +sanitize_stdfds (); + +return !!fsmonitor_run_daemon (); + +case -1 : +return error_errno (_ ("could not spawn fsmonitor--daemon in the background" )); +
Quoted 8 lines
builtin/fsmonitor--daemon.c static int try_to_run_foreground_daemon(void) · line 349
Quoted 5 lines
builtin/fsmonitor--daemon.c +5
-0+default : + return 0; + } +} +#endif The
spawn_background_fsmonitor_daemon()function here is almost the same asdaemonize().
Yes, the code comment above that function even says that it was adapted from daemonize().
And above that, of course, is a _completely different_ implementation that works on Windows (you will notice that this is in stark contrast of Windows, where the daemonize() function will simply fail with ENOSYS).
Quoted 2 lines
I wonder if this & the Windows-specific one you have here can't be refactored into an API from what's now in
setup.c.
Given that there is no fork() on Windows (which has been the subject of many a message to this mailing list), I think the answer to this question of yours is a resounding "No".
Quoted 3 lines
Then we could make
builtin/gc.canddaemon.cuse that, so Windows could have background GC, and we'd have a more battle-tested central codepath for this tricky bit.
Please. Not _more_ sidetracks.
The issue of getting git gc --auto to daemonize on Windows is a rather complicated one. I won't bore this list with the details, but link to
https://github.com/git-for-windows/git/issues/2221#issuecomment-542589590
(a ~950 word analysis of the problem).
Quoted 4 lines
It seems to me like the only limitations on it are to have this return slightly more general things (e.g. not set its own errors, return structured data), and maybe some callback for what to do in the child/parent.
And one version doesn't die(). Nor does it call exit(0) in the parent process. But it calls fsmonitor_listen() in the child process. And if you wanted to refactor daemonize() to do all that, it would have to be renamed (because it does no longer _necessarily_ daemonize), and it would have to have a gentle flag, and it would somehow have to indicate in its return value whether 0 means that the parent process returned successfully or the client process. And soon we'll end up with a function that is both longer and more unreadable than what we have right now.
Quoted 74 lines
Quoted 60 lines
builtin/fsmonitor--daemon.c +59
-0+/* +* This is adapted from `wait_or_whine()`. Watch the child process and +* let it get started and begin listening for requests on the socket +* before reporting our success. +*/ +static int wait_for_background_startup (pid_t pid_child) +{ +int status; +pid_t pid_seen; +enum ipc_active_state s; +time_t time_limit, now; + +time (&time_limit); + time_limit += fsmonitor__start_timeout_sec; + +for (;;) { + pid_seen =waitpid (pid_child, &status,WNOHANG ); + +if (pid_seen ==-1 ) +return error_errno (_ ("waitpid failed" )); +else if (pid_seen ==0 ) { +/* +* The child is still running (this should be +* the normal case). Try to connect to it on +* the socket and see if it is ready for +* business. +* +* If there is another daemon already running, +* our child will fail to start (possibly +* after a timeout on the lock), but we don't +* care (who responds) if the socket is live. +*/ + s =fsmonitor_ipc__get_state (); +if (s ==IPC_STATE__LISTENING ) +return 0 ; + +time (&now); +if (now > time_limit) +return error (_ ("fsmonitor--daemon not online yet" )); + }else if (pid_seen == pid_child) { +/* +* The new child daemon process shutdown while +* it was starting up, so it is not listening +* on the socket. +* +* Try to ping the socket in the odd chance +* that another daemon started (or was already +* running) while our child was starting. +* +* Again, we don't care who services the socket. +*/ + s =fsmonitor_ipc__get_state (); +if (s ==IPC_STATE__LISTENING ) +return 0 ; + +/* +* We don't care about the WEXITSTATUS() nor +* any of the WIF*(status) values because +* `cmd_fsmonitor__daemon()` does the `!!result` builtin/fsmonitor--daemon.c static int try_to_run_foreground_daemon(void) · line 419
Quoted 10 lines
builtin/fsmonitor--daemon.c +10
-0+* trick on all function return values. +* +* So it is sufficient to just report the +* early shutdown as an error. +*/ +return error (_ ("fsmonitor--daemon failed to start" )); + }else +return error (_ ("waitpid is confused" )); + } +}Ditto this. could we extend the
wait_or_whine()function (or some extended version thereof) to do what you need with callbacks?
Quoted 3 lines
It seems the main difference is just being able to pass down a flag for
waitpid(), and the loop needing to check EINTR or not depending on whether WNOHANG is passed.
Given that over half of wait_or_whine() is concerned with signals, which the wait_for_background_startup() function is not at all concerned with, I see another main difference.
Quoted 4 lines
For e.g. the "We don't care about the
WEXITSTATUS()" you'd get that behavior with an adjustedwait_or_whine(). Wouldn't it be better to report what exit status it exits with e.g. if the top-level process is signalled? We do so in trace2 for other things we spawn...
The wait_or_whine() function also adjusts atexit() behavior, which we would not want here.
Therefore, just like the suggestion about daemonize() above, it appears to me as if the suggested refactoring would make the code dramatically more complex and less readable.
In other words, it would be a refactoring for refactoring's sake. Definitely not something I would suggest.
Ciao,
Johannes
Re: [PATCH v3 14/34] fsmonitor--daemon: implement 'start' command
In reply to Ævar Arnfjörð BjarmasonMy response here is in addition to Dscho's remarks on this topic. He makes excellent points that I'll just #include here. I do want to add my own $0.02 here.
Quoted 86 lines · Ævar Arnfjörð Bjarmason
Quoted 68 lines · Jeff Hostetler via GitGitGadget
builtin/fsmonitor--daemon.c +68
-0+#ifdef GIT_WINDOWS_NATIVE +/* +* Create a background process to run the daemon. It should be completely +* disassociated from the terminal. +* +* Conceptually like `daemonize()` but different because Windows does not +* have `fork(2)`. Spawn a normal Windows child process but without the +* limitations of `start_command()` and `finish_command()`. +* +* The child process execs the "git fsmonitor--daemon run" command. +* +* The current process returns so that the caller can wait for the child +* to startup before exiting. +*/ +static int spawn_background_fsmonitor_daemon (pid_t *pid) +{ +char git_exe[MAX_PATH ]; +struct strvec args =STRVEC_INIT ; +int in, out; + +GetModuleFileNameA (NULL , git_exe,MAX_PATH ); + + in =open ("/dev/null" ,O_RDONLY ); + out =open ("/dev/null" ,O_WRONLY ); + +strvec_push (&args, git_exe); +strvec_push (&args,"fsmonitor--daemon" ); +strvec_push (&args,"run" ); +strvec_pushf (&args,"--ipc-threads=%d" , fsmonitor__ipc_threads); + + *pid =mingw_spawnvpe (args.v[0 ], args.v,NULL ,NULL , in, out, out); +close (in); +close (out); + +strvec_clear (&args); + +if (*pid <0 ) +return error (_ ("could not spawn fsmonitor--daemon in the background" )); + +return 0 ; +} +#else +/* +* Create a background process to run the daemon. It should be completely +* disassociated from the terminal. +* +* This is adapted from `daemonize()`. Use `fork()` to directly +* create and run the daemon in the child process. +* +* The fork-child can just call the run code; it does not need to exec +* it. +* +* The fork-parent returns the child PID so that we can wait for the +* child to startup before exiting. +*/ +static int spawn_background_fsmonitor_daemon (pid_t *pid) +{ + *pid =fork (); + +switch (*pid) { +case 0 : +if (setsid () ==-1 ) +error_errno (_ ("setsid failed" )); +close (0 ); +close (1 ); +close (2 ); +sanitize_stdfds (); +builtin/fsmonitor--daemon.c static int try_to_run_foreground_daemon(void) · line 349
Quoted 10 lines
builtin/fsmonitor--daemon.c +10
-0+return !!fsmonitor_run_daemon (); + +case -1 : +return error_errno (_ ("could not spawn fsmonitor--daemon in the background" )); + +default : + return 0; + } +} +#endif The
spawn_background_fsmonitor_daemon()function here is almost the same asdaemonize(). I wonder if this & the Windows-specific one you have here can't be refactored into an API from what's now insetup.c.
Quoted 4 lines
Then we could make
builtin/gc.canddaemon.cuse that, so Windows could have background GC, and we'd have a more battle-tested central codepath for this tricky bit.
I'd rather not refactor all of this and add unnecessary generality and complexity just to save duplicating some of the code in daemonize().
And I'd rather not destabilize existing commands like gc and daemon by changing the daemonize() layer on them. If those commands need help, let's have a separate conversation _later_ about what help they need and if it makes sense to combine them.
Quoted 4 lines
It seems to me like the only limitations on it are to have this return slightly more general things (e.g. not set its own errors, return structured data), and maybe some callback for what to do in the child/parent.
There are several issues here when trying to start a background process and we're already on the edge of the behavioral differences between Windows and Unix -- let's not make things more confusing with multiple callbacks, returning structures, custom errors, and etc.
Also, since Windows doesn't do fork(), we don't have child/parent branches in the call, so this whole "just pretend it's all Unix" model doesn't work.
Even if we did pretend I'd still need ifdef'd callback routines to either call fsmonitor_run_daemon() or build a command line (or have blocks of functions that "just happen to never be called on one platform or the other").
What I have here is an API that the primary (read: parent) calls and gets back a 0 or -1 (with error message). And that's it. The primary can then wait for the child (whether from fork or CreateProcess) to become responsive or fail to start. And then the primary can exit (with or without error).
So I think we're good. Yes, there is an ifdef here, but I think it is worth it.
Quoted 74 lines
Quoted 60 lines
builtin/fsmonitor--daemon.c +59
-0+/* +* This is adapted from `wait_or_whine()`. Watch the child process and +* let it get started and begin listening for requests on the socket +* before reporting our success. +*/ +static int wait_for_background_startup (pid_t pid_child) +{ +int status; +pid_t pid_seen; +enum ipc_active_state s; +time_t time_limit, now; + +time (&time_limit); + time_limit += fsmonitor__start_timeout_sec; + +for (;;) { + pid_seen =waitpid (pid_child, &status,WNOHANG ); + +if (pid_seen ==-1 ) +return error_errno (_ ("waitpid failed" )); +else if (pid_seen ==0 ) { +/* +* The child is still running (this should be +* the normal case). Try to connect to it on +* the socket and see if it is ready for +* business. +* +* If there is another daemon already running, +* our child will fail to start (possibly +* after a timeout on the lock), but we don't +* care (who responds) if the socket is live. +*/ + s =fsmonitor_ipc__get_state (); +if (s ==IPC_STATE__LISTENING ) +return 0 ; + +time (&now); +if (now > time_limit) +return error (_ ("fsmonitor--daemon not online yet" )); + }else if (pid_seen == pid_child) { +/* +* The new child daemon process shutdown while +* it was starting up, so it is not listening +* on the socket. +* +* Try to ping the socket in the odd chance +* that another daemon started (or was already +* running) while our child was starting. +* +* Again, we don't care who services the socket. +*/ + s =fsmonitor_ipc__get_state (); +if (s ==IPC_STATE__LISTENING ) +return 0 ; + +/* +* We don't care about the WEXITSTATUS() nor +* any of the WIF*(status) values because +* `cmd_fsmonitor__daemon()` does the `!!result` builtin/fsmonitor--daemon.c static int try_to_run_foreground_daemon(void) · line 419
Quoted 10 lines
builtin/fsmonitor--daemon.c +10
-0+* trick on all function return values. +* +* So it is sufficient to just report the +* early shutdown as an error. +*/ +return error (_ ("fsmonitor--daemon failed to start" )); + }else +return error (_ ("waitpid is confused" )); + } +}Ditto this. could we extend the
wait_or_whine()function (or some extended version thereof) to do what you need with callbacks?
Quoted 9 lines
It seems the main difference is just being able to pass down a flag for
waitpid(), and the loop needing to check EINTR or not depending on whether WNOHANG is passed.For e.g. the "We don't care about the
WEXITSTATUS()" you'd get that behavior with an adjustedwait_or_whine(). Wouldn't it be better to report what exit status it exits with e.g. if the top-level process is signalled? We do so in trace2 for other things we spawn...
Again, I don't want to mix my usage here with the existing code and destabilize all existing callers. Here we are spinning to give the child a chance to *start* and confirm that it is in a listening state and ready for connections. We do not wait for the child to exit (unless it dies quickly without becoming ready).
We want to end our wait as soon as we confirm that the child is ready and return. All I really need from the system is waitpid().
Also, since we started the child in my spawn_background...(), it is not in the children_to_clean list, so there is no need to mess with that.
So I'd like to leave this as is.
Jeff