Skip to contentSkip to conversation

fsmonitor: query watchman with right valid json

PATCH Superseded Son Luong Ngoc 3 participants 6 messages Jun 7, 2022

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 fsmonitor: query watchman with right valid json

In rare circumstances where the current git index does not carry the last_update_token, the fsmonitor v2 hook will be invoked with an empty string which would caused the final rendered json to be invalid.

  ["query", "/path/to/my/git/repository/", {
          "since": ,
          "fields": ["name"],
          "expression": ["not", ["dirname", ".git"]]
  }]

Which will left user with the following error message

Quoted 1 line

git status

failed to parse command from stdin: line 2, column 13, position 67: unexpected token near ','
Watchman: command returned no output.
Falling back to scanning...

Hide the "since" field in json query when "last_update_token" is empty.

---
 templates/hooks--fsmonitor-watchman.sample | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)
templates/hooks--fsmonitor-watchman.sample +13 -8
index 14ed0aa42d..b4ee86dfc4 100755
@@ -79,6 +79,12 @@ sub watchman_query {
 	or die "open2() failed: $!\n" .
 	"Falling back to scanning...\n";
 
+	my $query = <<"	END";
+		["query", "$git_work_tree", {
+			"fields": ["name"],
+			"expression": ["not", ["dirname", ".git"]]
+		}]
+	END
 	# In the query expression below we're asking for names of files that
 	# changed since $last_update_token but not from the .git folder.
 	#
@@ -87,15 +93,14 @@ sub watchman_query {
 	# output to file names only. Then we're using the "expression" term to
 	# further constrain the results.
 	if (substr($last_update_token, 0, 1) eq "c") {
-		$last_update_token = "\"$last_update_token\"";
+		$query = <<"		END";
+			["query", "$git_work_tree", {
+				"since": "$last_update_token",
+				"fields": ["name"],
+				"expression": ["not", ["dirname", ".git"]]
+			}]
+		END
 	}
-	my $query = <<"	END";
-		["query", "$git_work_tree", {
-			"since": $last_update_token,
-			"fields": ["name"],
-			"expression": ["not", ["dirname", ".git"]]
-		}]
-	END
 
 	# Uncomment for debugging the watchman query
 	# open (my $fh, ">", ".git/watchman-query.json");
Signature · 1 line
2.36.1.231.g6924ef9a07
#1 lore raw
Quoted 29 lines · Son Luong Ngoc

In rare circumstances where the current git index does not carry the last_update_token, the fsmonitor v2 hook will be invoked with an empty string which would caused the final rendered json to be invalid.

  ["query", "/path/to/my/git/repository/", {
          "since": ,
          "fields": ["name"],
          "expression": ["not", ["dirname", ".git"]]
  }]

Which will left user with the following error message

Quoted 1 line

git status

failed to parse command from stdin: line 2, column 13, position 67: unexpected token near ','
Watchman: command returned no output.
Falling back to scanning...

Hide the "since" field in json query when "last_update_token" is empty.

---
 templates/hooks--fsmonitor-watchman.sample | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)
templates/hooks--fsmonitor-watchman.sample +0 -0
index 14ed0aa42d..b4ee86dfc4 100755

templates/hooks--fsmonitor-watchman.sample sub watchman_query { · line 87

Quoted 9 lines
templates/hooks--fsmonitor-watchman.sample +6 -0
@@ -79,6 +79,12 @@ sub watchman_query {
 	or die "open2() failed: $!\n" .
 	"Falling back to scanning...\n";
 
+	my $query = <<"	END";
+		["query", "$git_work_tree", {
+			"fields": ["name"],
+			"expression": ["not", ["dirname", ".git"]]
+		}]
+	END

Wouldn't a more minimal & obvious patch here be....

templates/hooks--fsmonitor-watchman.sample sub watchman_query { · line 90

Quoted 8 lines
templates/hooks--fsmonitor-watchman.sample +0 -1
 	# In the query expression below we're asking for names of files that
 	# changed since $last_update_token but not from the .git folder.
 	#
@@ -87,15 +93,14 @@ sub watchman_query {
 	# output to file names only. Then we're using the "expression" term to
 	# further constrain the results.
 	if (substr($last_update_token, 0, 1) eq "c") {
-		$last_update_token = "\"$last_update_token\"";

To just change this to be:

	# same as now:
	$last_update_token = "\"$last_update_token\"";
        $last_update_line = qq["since": $last_update_token,];

Of course having declared the new $last_update_line variable earlier, then:

templates/hooks--fsmonitor-watchman.sample sub watchman_query { · line 94

Quoted 11 lines
templates/hooks--fsmonitor-watchman.sample +7 -3
+		$query = <<"		END";
+			["query", "$git_work_tree", {
+				"since": "$last_update_token",
+				"fields": ["name"],
+				"expression": ["not", ["dirname", ".git"]]
+			}]
+		END
 	}
-	my $query = <<"	END";
-		["query", "$git_work_tree", {
-			"since": $last_update_token,

Just change this line to:

$last_update_line

I.e. you don't need to duplicate the whole query just to omit/include a single line in it, or am I missing something?

(This suggestion *would* include a redundant line, but I'm assuming JSON/watchman deals with that just fine...).

#2 lore raw

Hi Ævar,

Quoted 56 lines · Ævar Arnfjörð Bjarmason
Quoted 29 lines · Son Luong Ngoc

In rare circumstances where the current git index does not carry the last_update_token, the fsmonitor v2 hook will be invoked with an empty string which would caused the final rendered json to be invalid.

  ["query", "/path/to/my/git/repository/", {
          "since": ,
          "fields": ["name"],
          "expression": ["not", ["dirname", ".git"]]
  }]

Which will left user with the following error message

Quoted 1 line

git status

failed to parse command from stdin: line 2, column 13, position 67: unexpected token near ','
Watchman: command returned no output.
Falling back to scanning...

Hide the "since" field in json query when "last_update_token" is empty.

---
 templates/hooks--fsmonitor-watchman.sample | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)
templates/hooks--fsmonitor-watchman.sample +0 -0
index 14ed0aa42d..b4ee86dfc4 100755

templates/hooks--fsmonitor-watchman.sample sub watchman_query { · line 87

Quoted 9 lines
templates/hooks--fsmonitor-watchman.sample +6 -0
@@ -79,6 +79,12 @@ sub watchman_query {
      or die "open2() failed: $!\n" .
      "Falling back to scanning...\n";
 
+     my $query = <<" END";
+             ["query", "$git_work_tree", {
+                     "fields": ["name"],
+                     "expression": ["not", ["dirname", ".git"]]
+             }]
+     END

Wouldn't a more minimal & obvious patch here be....

templates/hooks--fsmonitor-watchman.sample sub watchman_query { · line 90

Quoted 8 lines
templates/hooks--fsmonitor-watchman.sample +0 -1
      # In the query expression below we're asking for names of files that
      # changed since $last_update_token but not from the .git folder.
      #
@@ -87,15 +93,14 @@ sub watchman_query {
      # output to file names only. Then we're using the "expression" term to
      # further constrain the results.
      if (substr($last_update_token, 0, 1) eq "c") {
-             $last_update_token = "\"$last_update_token\"";

To just change this to be:

Quoted 6 lines
        # same as now:
        $last_update_token = "\"$last_update_token\"";
        $last_update_line = qq["since": $last_update_token,];

Of course having declared the new $last_update_line variable earlier, then:

Yup, I think this is a sensible suggestion.
I will fixup and send a V2 shortly.

Quoted 14 lines

templates/hooks--fsmonitor-watchman.sample sub watchman_query { · line 94

Quoted 11 lines
templates/hooks--fsmonitor-watchman.sample +7 -3
+             $query = <<"            END";
+                     ["query", "$git_work_tree", {
+                             "since": "$last_update_token",
+                             "fields": ["name"],
+                             "expression": ["not", ["dirname", ".git"]]
+                     }]
+             END
      }
-     my $query = <<" END";
-             ["query", "$git_work_tree", {
-                     "since": $last_update_token,

Just change this line to:

Quoted 7 lines

$last_update_line

I.e. you don't need to duplicate the whole query just to omit/include a single line in it, or am I missing something?

(This suggestion *would* include a redundant line, but I'm assuming JSON/watchman deals with that just fine...).

I think we can remove that redundant line by adding '\n' before $last_update_line.
I will be including this into the next version.

Thanks,
Son Luong.

#3 lore raw

PATCH v2 1/1 fsmonitor: query watchman with right valid json

In rare circumstances where the current git index does not carry the last_update_token, the fsmonitor v2 hook will be invoked with an empty string which would caused the final rendered json to be invalid.

  ["query", "/path/to/my/git/repository/", {
          "since": ,
          "fields": ["name"],
          "expression": ["not", ["dirname", ".git"]]
  }]

Which will left user with the following error message

Quoted 1 line

git status

failed to parse command from stdin: line 2, column 13, position 67: unexpected token near ','
Watchman: command returned no output.
Falling back to scanning...

Hide the "since" field in json query when "last_update_token" is empty.

---
 templates/hooks--fsmonitor-watchman.sample | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
templates/hooks--fsmonitor-watchman.sample +3 -2
index 14ed0aa42d..23e856f5de 100755
@@ -86,12 +86,13 @@ sub watchman_query {
 	# recency index to select candidate nodes and "fields" to limit the
 	# output to file names only. Then we're using the "expression" term to
 	# further constrain the results.
+	my $last_update_line = "";
 	if (substr($last_update_token, 0, 1) eq "c") {
 		$last_update_token = "\"$last_update_token\"";
+		$last_update_line = qq[\n"since": $last_update_token,];
 	}
 	my $query = <<"	END";
-		["query", "$git_work_tree", {
-			"since": $last_update_token,
+		["query", "$git_work_tree", {$last_update_line
 			"fields": ["name"],
 			"expression": ["not", ["dirname", ".git"]]
 		}]
Signature · 1 line
2.36.1.476.g0c4daa206d
#4 lore raw
Quoted 13 lines · Son Luong Ngoc

In rare circumstances where the current git index does not carry the last_update_token, the fsmonitor v2 hook will be invoked with an empty string which would caused the final rendered json to be invalid.

  ["query", "/path/to/my/git/repository/", {
          "since": ,
          "fields": ["name"],
          "expression": ["not", ["dirname", ".git"]]
  }]

Which will left user with the following error message

Quoted 1 line

git status

Quoted 8 lines

failed to parse command from stdin: line 2, column 13, position 67: unexpected token near ','
Watchman: command returned no output.
Falling back to scanning...

Hide the "since" field in json query when "last_update_token" is empty.

Thanks for the quick turnaround.

Quoted 8 lines
 templates/hooks--fsmonitor-watchman.sample | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
templates/hooks--fsmonitor-watchman.sample +0 -0
index 14ed0aa42d..23e856f5de 100755

templates/hooks--fsmonitor-watchman.sample sub watchman_query { · line 93

Quoted 8 lines
templates/hooks--fsmonitor-watchman.sample +2 -0
@@ -86,12 +86,13 @@ sub watchman_query {
 	# recency index to select candidate nodes and "fields" to limit the
 	# output to file names only. Then we're using the "expression" term to
 	# further constrain the results.
+	my $last_update_line = "";
 	if (substr($last_update_token, 0, 1) eq "c") {
 		$last_update_token = "\"$last_update_token\"";
+		$last_update_line = qq[\n"since": $last_update_token,];
 	}

This LGTM, just a note...

templates/hooks--fsmonitor-watchman.sample sub watchman_query { · line 95

Quoted 4 lines
templates/hooks--fsmonitor-watchman.sample +1 -2
 	my $query = <<"	END";
-		["query", "$git_work_tree", {
-			"since": $last_update_token,
+		["query", "$git_work_tree", {$last_update_line

...doesn't really need a re-roll, but doesn't this trade "we don't have too many \n" for not indenting the query properly anymore?

I think skipping both is fine, but between the two I think having indenting is better than having a redundant \n some of the time.

FWIW you could just add the variable on its own line, and then do this instead:

	(my $query = <<"        END") =~ s/(?<=\n)\t*\n//s; 

To post-hoc fix the extra \n in this case :)

But I think this is also fine as-is, thanks!

#5 lore raw
In reply to Son Luong Ngoc
Quoted 11 lines · Son Luong Ngoc

In rare circumstances where the current git index does not carry the last_update_token, the fsmonitor v2 hook will be invoked with an empty string which would caused the final rendered json to be invalid.

  ["query", "/path/to/my/git/repository/", {
          "since": ,
          "fields": ["name"],
          "expression": ["not", ["dirname", ".git"]]
  }]

Which will left user with the following error message

"left" -> "leave" (or "give")

Quoted 8 lines
Quoted 1 line

git status

failed to parse command from stdin: line 2, column 13, position 67: unexpected token near ','
Watchman: command returned no output.
Falling back to scanning...

Hide the "since" field in json query when "last_update_token" is empty.

It looked more like Helped-by to me, but I dunno.

Quoted 15 lines
---
 templates/hooks--fsmonitor-watchman.sample | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
templates/hooks--fsmonitor-watchman.sample +1 -0
index 14ed0aa42d..23e856f5de 100755
@@ -86,12 +86,13 @@ sub watchman_query {
 	# recency index to select candidate nodes and "fields" to limit the
 	# output to file names only. Then we're using the "expression" term to
 	# further constrain the results.
+	my $last_update_line = "";
 	if (substr($last_update_token, 0, 1) eq "c") {

templates/hooks--fsmonitor-watchman.sample sub watchman_query { · line 98

Quoted 10 lines
templates/hooks--fsmonitor-watchman.sample +2 -2
 		$last_update_token = "\"$last_update_token\"";
+		$last_update_line = qq[\n"since": $last_update_token,];
 	}
 	my $query = <<"	END";
-		["query", "$git_work_tree", {
-			"since": $last_update_token,
+		["query", "$git_work_tree", {$last_update_line
 			"fields": ["name"],
 			"expression": ["not", ["dirname", ".git"]]
 		}]

OK. Compared to v1, this looks much more reasonable.

This is totally unrelated to the "hide invalid since" topic, but I wonder if $git_work_tree needs a bit more careful quoting. It comes directly from get_working_dir() but can it contain say a double quote character, to make the resulting string in the variable $query not quite well formed?