Skip to contentSkip to conversation

Honor core.precomposeUnicode in more places

PATCH Superseded Elijah Newren 4 participants 6 messages Apr 23, 2019 – Jul 26, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Honor core.precomposeUnicode in more places

On Mac's HFS ("Hilarious FileSystem"? "Halfwitted FileSystem"?) -- where git sets core.precomposeUnicode to true automatically by git init/clone -- when a user creates a simple unicode refname (in NFC format) such as españa:

  $ git branch españa

different commands would display the branch name differently. For example, git branch, git log --decorate, and git fast-export all used

65 73 70 61 c3 b1 61 (or "espa\xc3\xb1a")

(NFC form) while show-ref would use

65 73 70 61 6e cc 83 61 (or "espan\xcc\x83a")

(NFD form). A stress test for git filter-repo was tripped up by this inconsistency, though digging in I found that the problems could compound; for example, if the user ran

  $ git pack-refs --all

and then tried to check out the branch, they would be met with:

  $ git checkout españa
  error: pathspec 'españa' did not match any file(s) known to git
  $ git checkout españa --
  fatal: invalid reference: españa
  $ git branch
    españa
  * master

Note that the user could run the git branch command first and copy and paste the españa portion of the output and still see the same two errors. Also, if the user added --no-prune to the pack-refs command, then they would see three branches: master, españa, and españa (those last two are NFC vs. NFD forms, even if they render the same).

Further, if the user had the españa branch checked out before running git pack-refs --all, the user would be greeted with (note that I'm trimming trailing output with an ellipsis):

  $ git rev-parse HEAD
  fatal: ambiguous argument 'HEAD': unknown revision or path...
  $ git status
  On branch españa

No commits yet...

Or worse, if the user didn't check this stuff first, running git commit will create a new commit with all changes of all of history being squashed into it.

In addition to pack-refs, one could also get into this state with upload-pack or anything that calls either pack-refs or upload-pack (e.g. gc or clone).

Add code in a few places (pack-refs, show-ref, upload-pack) to check and honor the setting of core.precomposeUnicode to avoid these bugs.

---
 builtin/pack-refs.c | 2 ++
 builtin/show-ref.c  | 3 +++
 upload-pack.c       | 2 ++
 3 files changed, 7 insertions(+)
builtin/pack-refs.c +2 -0
index f3353564f9..cfbd5c36c7 100644
@@ -1,4 +1,5 @@
 #include "builtin.h"
+#include "config.h"
 #include "parse-options.h"
 #include "refs.h"
 #include "repository.h"
@@ -16,6 +17,7 @@ int cmd_pack_refs(int argc, const char **argv, const char *prefix)
 		OPT_BIT(0, "prune", &flags, N_("prune loose refs (default)"), PACK_REFS_PRUNE),
 		OPT_END(),
 	};
+	git_config(git_default_config, NULL);
 	if (parse_options(argc, argv, prefix, opts, pack_refs_usage, 0))
 		usage_with_options(pack_refs_usage, opts);
 	return refs_pack_refs(get_main_ref_store(the_repository), flags);
builtin/show-ref.c +3 -0
index 6a706c02a6..6456da70cc 100644
@@ -1,5 +1,6 @@
 #include "builtin.h"
 #include "cache.h"
+#include "config.h"
 #include "refs.h"
 #include "object-store.h"
 #include "object.h"
@@ -182,6 +183,8 @@ static const struct option show_ref_options[] = {
 
 int cmd_show_ref(int argc, const char **argv, const char *prefix)
 {
+	git_config(git_default_config, NULL);
+
 	argc = parse_options(argc, argv, prefix, show_ref_options,
 			     show_ref_usage, 0);
 
upload-pack.c +2 -0
index d098ef5982..159f751ea4 100644
@@ -1064,6 +1064,8 @@ static int upload_pack_config(const char *var, const char *value, void *unused)
 		allow_ref_in_want = git_config_bool(var, value);
 	} else if (!strcmp("uploadpack.allowsidebandall", var)) {
 		allow_sideband_all = git_config_bool(var, value);
+	} else if (!strcmp("core.precomposeunicode", var)) {
+		precomposed_unicode = git_config_bool(var, value);
 	}
 
 	if (current_config_scope() != CONFIG_SCOPE_REPO) {
Signature · 1 line
2.21.0.420.g4906d192b3
#1 lore raw
Quoted 1 line · Elijah Newren

On Mac's HFS ("Hilarious FileSystem"? "Halfwitted FileSystem"?) --

How about "Hierarchical File System" ?

Quoted 55 lines

where git sets core.precomposeUnicode to true automatically by git init/clone -- when a user creates a simple unicode refname (in NFC format) such as españa:

  $ git branch españa

different commands would display the branch name differently. For example, git branch, git log --decorate, and git fast-export all used

65 73 70 61 c3 b1 61 (or "espa\xc3\xb1a")

(NFC form) while show-ref would use

65 73 70 61 6e cc 83 61 (or "espan\xcc\x83a")

(NFD form). A stress test for git filter-repo was tripped up by this inconsistency, though digging in I found that the problems could compound; for example, if the user ran

  $ git pack-refs --all

and then tried to check out the branch, they would be met with:

  $ git checkout españa
  error: pathspec 'españa' did not match any file(s) known to git
  $ git checkout españa --
  fatal: invalid reference: españa
  $ git branch
    españa
  * master

Note that the user could run the git branch command first and copy and paste the españa portion of the output and still see the same two errors. Also, if the user added --no-prune to the pack-refs command, then they would see three branches: master, españa, and españa (those last two are NFC vs. NFD forms, even if they render the same).

Further, if the user had the españa branch checked out before running git pack-refs --all, the user would be greeted with (note that I'm trimming trailing output with an ellipsis):

  $ git rev-parse HEAD
  fatal: ambiguous argument 'HEAD': unknown revision or path...
  $ git status
  On branch españa

No commits yet...

Or worse, if the user didn't check this stuff first, running git commit will create a new commit with all changes of all of history being squashed into it.

Quoted 6 lines

In addition to pack-refs, one could also get into this state with upload-pack or anything that calls either pack-refs or upload-pack (e.g. gc or clone).

Add code in a few places (pack-refs, show-ref, upload-pack) to check and honor the setting of core.precomposeUnicode to avoid these bugs.

That's all correct, one minor question below.

Quoted 19 lines
---
 builtin/pack-refs.c | 2 ++
 builtin/show-ref.c  | 3 +++
 upload-pack.c       | 2 ++
 3 files changed, 7 insertions(+)
builtin/pack-refs.c +1 -0
index f3353564f9..cfbd5c36c7 100644
@@ -1,4 +1,5 @@
 #include "builtin.h"
+#include "config.h"
 #include "parse-options.h"
 #include "refs.h"
 #include "repository.h"

builtin/pack-refs.c int cmd_pack_refs(int argc, const char **argv, const char *prefix) · line 22

Quoted 6 lines
builtin/pack-refs.c +1 -0
@@ -16,6 +17,7 @@ int cmd_pack_refs(int argc, const char **argv, const char *prefix)
 		OPT_BIT(0, "prune", &flags, N_("prune loose refs (default)"), PACK_REFS_PRUNE),
 		OPT_END(),
 	};
+	git_config(git_default_config, NULL);
 	if (parse_options(argc, argv, prefix, opts, pack_refs_usage, 0))
 		usage_with_options(pack_refs_usage, opts);

I wonder if we could move the call to git_config() into parse_options(), (or another common place) but I haven't checked the details yet. Same below for show_ref().

And thankks for picking this up.

Quoted 37 lines
builtin/pack-refs.c +0 -0
 	return refs_pack_refs(get_main_ref_store(the_repository), flags);
builtin/show-ref.c +3 -0
index 6a706c02a6..6456da70cc 100644
@@ -1,5 +1,6 @@
 #include "builtin.h"
 #include "cache.h"
+#include "config.h"
 #include "refs.h"
 #include "object-store.h"
 #include "object.h"
@@ -182,6 +183,8 @@ static const struct option show_ref_options[] = {
 
 int cmd_show_ref(int argc, const char **argv, const char *prefix)
 {
+	git_config(git_default_config, NULL);
+
 	argc = parse_options(argc, argv, prefix, show_ref_options,
 			     show_ref_usage, 0);
 
upload-pack.c +2 -1
index d098ef5982..159f751ea4 100644
@@ -1064,6 +1064,8 @@ static int upload_pack_config(const char *var, const char *value, void *unused)
 		allow_ref_in_want = git_config_bool(var, value);
 	} else if (!strcmp("uploadpack.allowsidebandall", var)) {
 		allow_sideband_all = git_config_bool(var, value);
+	} else if (!strcmp("core.precomposeunicode", var)) {
+		precomposed_unicode = git_config_bool(var, value);
 	}
 
 	if (current_config_scope() != CONFIG_SCOPE_REPO) {
--

2.21.0.420.g4906d192b3

#2 lore raw
Quoted 5 lines · Torsten Bögershausen
Quoted 1 line · Elijah Newren

On Mac's HFS ("Hilarious FileSystem"? "Halfwitted FileSystem"?) --

How about "Hierarchical File System" ?

Sorry, I should have removed my draft commentary before submitting. Yes, you are of course right.

...

Quoted 4 lines
Quoted 2 lines

Add code in a few places (pack-refs, show-ref, upload-pack) to check and honor the setting of core.precomposeUnicode to avoid these bugs.

That's all correct, one minor question below.

...

Quoted 11 lines

builtin/pack-refs.c int cmd_pack_refs(int argc, const char **argv, const char *prefix) · line 22

Quoted 7 lines
builtin/pack-refs.c +1 -0
@@ -16,6 +17,7 @@ int cmd_pack_refs(int argc, const char **argv, const char *prefix)
              OPT_BIT(0, "prune", &flags, N_("prune loose refs (default)"), PACK_REFS_PRUNE),
              OPT_END(),
      };
+     git_config(git_default_config, NULL);
      if (parse_options(argc, argv, prefix, opts, pack_refs_usage, 0))
              usage_with_options(pack_refs_usage, opts);

I wonder if we could move the call to git_config() into parse_options(), (or another common place) but I haven't checked the details yet. Same below for show_ref().

Interesting idea. However, moving it into parse_options() presumes that either git_default_config will always be passed to git_config(), or else the arguments needed by git_config() will be explicitly passed to parse_options. A quick grep through the source code suggests that only about 1/3 of callsites pass git_default_config to git_config(), and passing a config-related callback function to parse_options seems a little weird to me. Plus, the fact that 2/3 of the callsites use a special config callback function suggests that we'd still not catch all cases with such a change, much like I had to update the upload_pack_config special callback below.

Quoted 39 lines

And thankks for picking this up.

Quoted 37 lines
builtin/pack-refs.c +0 -0
      return refs_pack_refs(get_main_ref_store(the_repository), flags);
builtin/show-ref.c +3 -0
index 6a706c02a6..6456da70cc 100644
@@ -1,5 +1,6 @@
 #include "builtin.h"
 #include "cache.h"
+#include "config.h"
 #include "refs.h"
 #include "object-store.h"
 #include "object.h"
@@ -182,6 +183,8 @@ static const struct option show_ref_options[] = {
 
 int cmd_show_ref(int argc, const char **argv, const char *prefix)
 {
+     git_config(git_default_config, NULL);
+
      argc = parse_options(argc, argv, prefix, show_ref_options,
                           show_ref_usage, 0);
 
upload-pack.c +2 -1
index d098ef5982..159f751ea4 100644
@@ -1064,6 +1064,8 @@ static int upload_pack_config(const char *var, const char *value, void *unused)
              allow_ref_in_want = git_config_bool(var, value);
      } else if (!strcmp("uploadpack.allowsidebandall", var)) {
              allow_sideband_all = git_config_bool(var, value);
+     } else if (!strcmp("core.precomposeunicode", var)) {
+             precomposed_unicode = git_config_bool(var, value);
      }
 
      if (current_config_scope() != CONFIG_SCOPE_REPO) {
--

2.21.0.420.g4906d192b3

#3 lore raw
Quoted 2 lines · Elijah Newren

..., and passing a config-related callback function to parse_options seems a little weird to me.

A little? That's a moderate understatement.

If parse_options API were the ONLY thing that gets affected by precompose_unicode, having an "if the config has not been read yet, read only that configuration variable and nothing else" there might make sense, but that is not the case (i.e. the variable also affects how readdir() works). So the alternatives that make sense are

(1) we stick to the current "make sure we read that variable sufficiently early in the main flow of the program" pattern, which this patch does, or

(2) we switch to "make sure the variable is read before we need it" pattern, i.e. add that "read the single config variable from the file, if it is not yet read" call to both parse_options() and opendir()---if the set of operations affected by the precompose_unicode variable grows, we'd need to add the same call to the new places, too.

I think (1) is good at least for now.

Quoted 11 lines
Quoted 11 lines

upload-pack.c static int upload_pack_config(const char *var, const char *value, void *unused) · line 1069

Quoted 11 lines
upload-pack.c +2 -0
index d098ef5982..159f751ea4 100644
@@ -1064,6 +1064,8 @@ static int upload_pack_config(const char *var, const char *value, void *unused)
              allow_ref_in_want = git_config_bool(var, value);
      } else if (!strcmp("uploadpack.allowsidebandall", var)) {
              allow_sideband_all = git_config_bool(var, value);
+     } else if (!strcmp("core.precomposeunicode", var)) {
+             precomposed_unicode = git_config_bool(var, value);
      }

What the other hunks wanted to do was quite obvious (i.e. before calling parse_options(), make sure we know precomposed_unicode is set appropriately, so that argv[] can be tweaked correctly). But this one was a bit less clear.

It turns out that upload-pack deliberately avoids using the default config callback, but tries to limit itself to the minimally needed set, so this hunk adds the precomposed_unicode to it. By doing so, we trigger another effect of precomposed_unicode, i.e. tweaking the paths we read out of readdir(), so that the refs we have to offer to the other side are all normalzied to the precomposed form.

Makes sense. Thanks.

#4 lore raw

PATCH v2 1/1 Honor core.precomposeUnicode in more places

In reply to Elijah Newren

On Mac's HFS where git sets core.precomposeUnicode to true automatically by git init/clone, when a user creates a simple unicode refname (in NFC format) such as españa:

  $ git branch españa

different commands would display the branch name differently. For example, git branch, git log --decorate, and git fast-export all used

65 73 70 61 c3 b1 61 (or "espa\xc3\xb1a")

(NFC form) while show-ref would use

65 73 70 61 6e cc 83 61 (or "espan\xcc\x83a")

(NFD form). A stress test for git filter-repo was tripped up by this inconsistency, though digging in I found that the problems could compound; for example, if the user ran

  $ git pack-refs --all

and then tried to check out the branch, they would be met with:

  $ git checkout españa
  error: pathspec 'españa' did not match any file(s) known to git
  $ git checkout españa --
  fatal: invalid reference: españa
  $ git branch
    españa
  * master

Note that the user could run the git branch command first and copy and paste the españa portion of the output and still see the same two errors. Also, if the user added --no-prune to the pack-refs command, then they would see three branches: master, españa, and españa (those last two are NFC vs. NFD forms, even if they render the same).

Further, if the user had the españa branch checked out before running git pack-refs --all, the user would be greeted with (note that I'm trimming trailing output with an ellipsis):

  $ git rev-parse HEAD
  fatal: ambiguous argument 'HEAD': unknown revision or path...
  $ git status
  On branch españa

No commits yet...

Or worse, if the user didn't check this stuff first, running git commit will create a new commit with all changes of all of history being squashed into it.

In addition to pack-refs, one could also get into this state with upload-pack or anything that calls either pack-refs or upload-pack (e.g. gc or clone).

Add code in a few places (pack-refs, show-ref, upload-pack) to check and honor the setting of core.precomposeUnicode to avoid these bugs.

---
Changes since v1:
* Excised unnecessary portion of the first sentence of the commit message

 builtin/pack-refs.c | 2 ++
 builtin/show-ref.c  | 3 +++
 upload-pack.c       | 2 ++
 3 files changed, 7 insertions(+)
builtin/pack-refs.c +2 -0
index f3353564f9..cfbd5c36c7 100644
@@ -1,4 +1,5 @@
 #include "builtin.h"
+#include "config.h"
 #include "parse-options.h"
 #include "refs.h"
 #include "repository.h"
@@ -16,6 +17,7 @@ int cmd_pack_refs(int argc, const char **argv, const char *prefix)
 		OPT_BIT(0, "prune", &flags, N_("prune loose refs (default)"), PACK_REFS_PRUNE),
 		OPT_END(),
 	};
+	git_config(git_default_config, NULL);
 	if (parse_options(argc, argv, prefix, opts, pack_refs_usage, 0))
 		usage_with_options(pack_refs_usage, opts);
 	return refs_pack_refs(get_main_ref_store(the_repository), flags);
builtin/show-ref.c +3 -0
index 6a706c02a6..6456da70cc 100644
@@ -1,5 +1,6 @@
 #include "builtin.h"
 #include "cache.h"
+#include "config.h"
 #include "refs.h"
 #include "object-store.h"
 #include "object.h"
@@ -182,6 +183,8 @@ static const struct option show_ref_options[] = {
 
 int cmd_show_ref(int argc, const char **argv, const char *prefix)
 {
+	git_config(git_default_config, NULL);
+
 	argc = parse_options(argc, argv, prefix, show_ref_options,
 			     show_ref_usage, 0);
 
upload-pack.c +2 -0
index d098ef5982..159f751ea4 100644
@@ -1064,6 +1064,8 @@ static int upload_pack_config(const char *var, const char *value, void *unused)
 		allow_ref_in_want = git_config_bool(var, value);
 	} else if (!strcmp("uploadpack.allowsidebandall", var)) {
 		allow_sideband_all = git_config_bool(var, value);
+	} else if (!strcmp("core.precomposeunicode", var)) {
+		precomposed_unicode = git_config_bool(var, value);
 	}
 
 	if (current_config_scope() != CONFIG_SCOPE_REPO) {
Signature · 1 line
2.21.0.503.g24b69a0db9
#5 lore raw
In reply to Elijah Newren
Quoted 11 lines · Elijah Newren
builtin/pack-refs.c +1 -0
index f3353564f9..cfbd5c36c7 100644
@@ -1,4 +1,5 @@
 #include "builtin.h"
+#include "config.h"
 #include "parse-options.h"
 #include "refs.h"
 #include "repository.h"

builtin/pack-refs.c int cmd_pack_refs(int argc, const char **argv, const char *prefix) · line 23

Quoted 7 lines
builtin/pack-refs.c +1 -0
@@ -16,6 +17,7 @@ int cmd_pack_refs(int argc, const char **argv, const char *prefix)
 		OPT_BIT(0, "prune", &flags, N_("prune loose refs (default)"), PACK_REFS_PRUNE),
 		OPT_END(),
 	};
+	git_config(git_default_config, NULL);
 	if (parse_options(argc, argv, prefix, opts, pack_refs_usage, 0))
 		usage_with_options(pack_refs_usage, opts);
 	return refs_pack_refs(get_main_ref_store(the_repository), flags);

I coincidentally just wrote the same patch (working off a branch of v2.21, so I didn't yet have your fix here). Just for the record, this fixes other bugs, too!

Mine was:

t/t3210-pack-refs.sh +9 -0
index 9ea5fa4fd2..07d09ac67a 100755
@@ -253,4 +253,13 @@ test_expect_success SYMLINKS 'pack symlinked packed-refs' '
 	test "$(readlink .git/packed-refs)" = "my-deviant-packed-refs"
 '
 
+test_expect_success 'pack-refs respects core.useReplaceRefs' '
+	# this is broken because the object refers to itself
+	commit=$(git rev-parse HEAD) &&
+	test_when_finished "git update-ref -d refs/replace/$commit" &&
+	git update-ref refs/replace/$commit $commit &&
+	test_must_fail git pack-refs --all &&
+	git -c core.useReplaceRefs=false pack-refs --all
+'
+
 test_done

It's probably not worth adding the test independently, but I thought I'd throw it out here for the benefit of the list archive.

-Peff