is_hfs_dotgit: loosen over-eager match of \u{..47}

Our is_hfs_dotgit function relies on the hackily-implemented next_hfs_char to give us the next character that an HFS+ filename comparison would look at. It's hacky because it doesn't implement the full case-folding table of HFS+; it gives us just enough to see if the path matches ".git".

At the end of next_hfs_char, we use tolower() to convert our 32-bit code point to lowercase. Our tolower() implementation only takes an 8-bit char, though; it throws away the upper 24 bits. This means we can't have any false negatives for is_hfs_dotgit. We only care about matching 7-bit ASCII characters in ".git", and we will correctly process 'G' or 'g'.

However, we _can_ have false positives. Because we throw away the upper bits, code point \u{0147} (for example) will look like 'G' and get downcased to 'g'. It's not known whether a sequence of code points whose truncation ends up as ".git" is meaningful in any language, but it does not hurt to be more accurate here. We can just pass out the full 32-bit code point, and compare it manually to the upper and lowercase characters we care about.

---
I saw Linus ask about this on G+. I had done the "no false negative" analysis when writing the patch, but didn't consider the false positive.

Another way of accomplishing the same thing is for next_hfs_char to continue folding case, but _only_ do so for 8-bit code points. Like:

  return (out & 0xffffff00) ? out : tolower(out);

I think the what's below is more obvious (and is actually how I originally wrote it; I switched to using tolower() during development to try to make it more readable).

 t/t1450-fsck.sh | 16 ++++++++++++++++
 utf8.c          | 17 ++++++++++++-----
 2 files changed, 28 insertions(+), 5 deletions(-)

index 1f04b8a..3f5883d 100755
@@ -349,6 +349,22 @@ dot-backslash-case .\\\\.GIT\\\\foobar
 dotgit-case-backslash .git\\\\foobar
 EOF
 
+test_expect_success 'fsck allows .Ňit' '
+	(
+		git init not-dotgit &&
+		cd not-dotgit &&
+		echo content >file &&
+		git add file &&
+		git commit -m base &&
+		blob=$(git rev-parse :file) &&
+		printf "100644 blob $blob\t.\\305\\207it" >tree &&
+		tree=$(git mktree <tree) &&
+		git fsck 2>err &&
+		cat err &&
+		! test -s err
+	)
+'
+
 # create a static test repo which is broken by omitting
 # one particular object ($1, which is looked up via rev-parse
 # in the new repository).

index 9a3f4ad..34a779e 100644
@@ -606,7 +606,7 @@ static ucs_char_t next_hfs_char(const char **in)
 		 * but this is enough to catch anything that will convert
 		 * to ".git"
 		 */
-		return tolower(out);
+		return out;
 	}
 }
 

@@ -614,10 +614,17 @@ int is_hfs_dotgit(const char *path)
 {
 	ucs_char_t c;
 
-	if (next_hfs_char(&path) != '.' ||
-	    next_hfs_char(&path) != 'g' ||
-	    next_hfs_char(&path) != 'i' ||
-	    next_hfs_char(&path) != 't')
+	c = next_hfs_char(&path);
+	if (c != '.')
+		return 0;
+	c = next_hfs_char(&path);
+	if (c != 'g' && c != 'G')
+		return 0;
+	c = next_hfs_char(&path);
+	if (c != 'i' && c != 'I')
+		return 0;
+	c = next_hfs_char(&path);
+	if (c != 't' && c != 'T')
 		return 0;
 	c = next_hfs_char(&path);
 	if (c && !is_dir_sep(c))

2.2.1.376.gec59d43

Mph. Sorry I missed this before.

[...]

[...]

Thanks, I actually considered adding a "test_file_is_empty" for that reason, but empty line-count does the same thing.

-Peff

That's a good suggestion, I think. This is meant to apply on top of d08c13b, and we already had test_line_count back then.

So far I collected these follow-ups to squash into Peff's patch.

 t/t1450-fsck.sh |  3 +--
 utf8.c          | 15 ++++++++-------
 2 files changed, 9 insertions(+), 9 deletions(-)

index 5e42385..0279b2b 100755
@@ -284,8 +284,7 @@ test_expect_success 'fsck allows .Ňit' '
 		printf "100644 blob $blob\t.\\305\\207it" >tree &&
 		tree=$(git mktree <tree) &&
 		git fsck 2>err &&
-		cat err &&
-		! test -s err
+		test_line_count = 0 err
 	)
 '
 

index 18a8f42..9c9fa3a 100644
@@ -630,8 +630,8 @@ int mbs_chrlen(const char **text, size_t *remainder_p, const char *encoding)
 }
 
 /*
- * Pick the next char from the stream, folding as an HFS+ filename comparison
- * would. Note that this is _not_ complete by any means. It's just enough
+ * Pick the next char from the stream, ignoring codepoints an HFS+ would.
+ * Note that this is _not_ complete by any means. It's just enough
  * to make is_hfs_dotgit() work, and should not be used otherwise.
  */
 static ucs_char_t next_hfs_char(const char **in)

@@ -668,11 +668,6 @@ static ucs_char_t next_hfs_char(const char **in)
 			continue;
 		}
 
-		/*
-		 * there's a great deal of other case-folding that occurs,
-		 * but this is enough to catch anything that will convert
-		 * to ".git"
-		 */
 		return out;
 	}
 }

@@ -685,6 +680,12 @@ int is_hfs_dotgit(const char *path)
 	if (c != '.')
 		return 0;
 	c = next_hfs_char(&path);
+
+	/*
+	 * there's a great deal of other case-folding that occurs
+	 * in HFS+, but this is enough to catch anything that will
+	 * convert to ".git"
+	 */
 	if (c != 'g' && c != 'G')
 		return 0;
 	c = next_hfs_char(&path);

They all look good to me. Thanks.

-Peff

Don't we have the same possible problem under NTFS? Under Linux + VFAT ?
Under all OS + VFAT ?

And would it make sense to turn this

into this:
static ucs_char_t unicode_tolower(ucs_char_t ch) {
return (ch & 0xffffff00) ? ch : tolower(ch);
}

And what happens if I export NTFS to Mac OS X?
(Other combinations possible)
Shouldn't fsck under all OS warn for NTFS and hfs possible attacks ?

I'm not sure what you mean.

This code path is _only_ about checking for HFS+-specific problems. We check general case-insensitivity in another code path. And we check NTFS-specific problems in another code path.

(Technically we could make a "check all" function that runs over the data only once, which would be more efficient. But doing it this way makes the code much easier to follow).

Perhaps, but you would need a big warning at the top of that function that is _only_ downcasing ASCII characters. I.e., it is fine for our use here, but you would not want other unicode-aware code to call it. The next_hfs_char already has such a warning at the top.

Fsck already warns for all system types, no matter what platform you're on. And we do case-insensitive blocking of ".git" entering the index for all platforms.

We don't turn on filesystem-specific blocking of paths entering the index on all platforms. You get HFS+ blocking by default on OS X, and NTFS on Windows. If you are using HFS+ on Linux, you should set core.protectHFS.

Possibly we should just turn on all checks everywhere. That would be a safer default, at the cost to Linux folks of:

1. Some slight inefficiency in each read-tree, as we have to do extra processing on each name.

2. Some names would be disallowed that are otherwise OK. For the most part these are not names that would be used in practice, though (e.g., losing the ability to create ".git\u200c" is not a big loss to anyone). I think Git for Windows has started blocking other stuff like "CON" that is not specifically related to .git, though, and that is more plausible for somebody to use in real life.

-Peff

I do not think so, as NTFS codepath does not use the (previouly too wide but now fixed) is_hfs_dotgit() in the first place, and is_ntfs_dotgit() does not read one unicode codepoint at a time if I remember correctly.

Doesn't it? Have you read the whole series you are commenting on?

Am 23.12.2014 um 09:45 schrieb Jeff King:

I did consider it, but the comment that is there was actually written for the _original_ version, before I added tolower in the first place (it also applied equally to the tolower() version, so I left it).

So it was clear either way, at least in my brain. :)

Of course that does not say anything about people's brains who did not write the patch. The changes Junio suggested elsewhere in the thread do make it more clear.

-Peff