gettext(windows): always use UTF-8

The main issue we work around here is that Windows does not have a UTF-8 "code page".

Side note: there is actually a code page for UTF-8: 65001 (see
https://docs.microsoft.com/en-us/windows/desktop/Intl/code-page-identifiers
). However, when experimenting with it, we ran into a multitude of issues in the Git for Windows project, ranging from various problems with Windows' default console to miscounted file writes. While these issues may have been mitigated in recent Windows 10 versions, older ones (in particular, Windows
7) still seem to have most of them, and Git for Windows specifically still supports even Windows Vista. So from a practical point of view, there is no UTF-8 code page.

Karsten Blees (1):
gettext: always use UTF-8 on native Windows

 gettext.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

base-commit: aa25c82427ae70aebf3b8f970f2afd54e9a2a8c6
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-217%2Fdscho%2Fgettext-force-utf-8-on-windows-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-217/dscho/gettext-force-utf-8-on-windows-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/217

gitgitgadget

From: Karsten Blees <[email protected]>

Git on native Windows exclusively uses UTF-8 for console output (both with MinTTY and native Console windows). Gettext uses setlocale() to determine the output encoding for translated text, however, MSVCRT's setlocale() doesn't support UTF-8. As a result, translated text is encoded in system encoding (GetAPC()), and non-ASCII chars are mangled in console output.

Use gettext's bind_textdomain_codeset() to force the encoding to UTF-8 on native Windows.

In this developers' setup, HAVE_LIBCHARSET_H is apparently defined, but we *really* want to override the locale_charset() here.

---
 gettext.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

index d4021d690c..d8423e5c41 100644
@@ -12,7 +12,9 @@
 #ifndef NO_GETTEXT
 #	include <locale.h>
 #	include <libintl.h>
-#	ifdef HAVE_LIBCHARSET_H
+#	ifdef GIT_WINDOWS_NATIVE
+#		define locale_charset() "UTF-8"
+#	elif defined HAVE_LIBCHARSET_H
 #		include <libcharset.h>
 #	else
 #		include <langinfo.h>

gitgitgadget

Hi,

index 7da80db453c4..35d2c1218db2 100644
@@ -13,7 +13,23 @@
 #	include <locale.h>
 #	include <libintl.h>
 #	ifdef GIT_WINDOWS_NATIVE
-#		define locale_charset() "UTF-8"
+
+static const char *locale_charset(void)
+{
+	const char *env = getenv("LC_ALL"), *dot;
+
+	if (!env || !*env)
+		env = getenv("LC_CTYPE");
+	if (!env || !*env)
+		env = getenv("LANG");
+
+	if (!env)
+		return "UTF-8";
+
+	dot = strchr(env, '.');
+	return !dot ? env : dot + 1;
+}
+
 #	elif defined HAVE_LIBCHARSET_H
 #		include <libcharset.h>
 #	else

OK, will do.

The main issue we work around here is that Windows does not have a UTF-8 "code page".

Side note: there is actually a code page for UTF-8: 65001 (see
https://docs.microsoft.com/en-us/windows/desktop/Intl/code-page-identifiers
). However, when experimenting with it, we ran into a multitude of issues in the Git for Windows project, ranging from various problems with Windows' default console to miscounted file writes. While these issues may have been mitigated in recent Windows 10 versions, older ones (in particular, Windows
7) still seem to have most of them, and Git for Windows specifically still supports even Windows Vista. So from a practical point of view, there is no UTF-8 code page.

Changes since v1:

* The LC_ALL=C method used by ab/no-kwset to prevent Git from assuming UTF-8-encoded input is now supported.
* The commit message was enhanced and revamped.

Karsten Blees (1):
gettext: always use UTF-8 on native Windows

 gettext.c | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

base-commit: aa25c82427ae70aebf3b8f970f2afd54e9a2a8c6
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-217%2Fdscho%2Fgettext-force-utf-8-on-windows-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-217/dscho/gettext-force-utf-8-on-windows-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/217

Range-diff vs v1:

1: ff37a2646a ! 1: 2d2253faef gettext: always use UTF-8 on native Windows
@@ -2,17 +2,34 @@

gettext: always use UTF-8 on native Windows

- Git on native Windows exclusively uses UTF-8 for console output (both with
- MinTTY and native Console windows). Gettext uses setlocale() to determine
- the output encoding for translated text, however, MSVCRT's setlocale()
- doesn't support UTF-8. As a result, translated text is encoded in system
- encoding (GetAPC()), and non-ASCII chars are mangled in console output.
+ On native Windows, Git exclusively uses UTF-8 for console output (both
+ with MinTTY and native Win32 Console). Gettext uses setlocale() to
+ determine the output encoding for translated text, however, MSVCRT's
+ setlocale() does not support UTF-8. As a result, translated text is
+ encoded in system encoding (as per GetAPC()), and non-ASCII chars are
+ mangled in console output.

- Use gettext's bind_textdomain_codeset() to force the encoding to UTF-8 on
- native Windows.
+ Side note: There is actually a code page for UTF-8: 65001. In practice,
+ it does not work as expected at least on Windows 7, though, so we cannot
+ use it in Git. Besides, if we overrode the code page, any process
+ spawned from Git would inherit that code page (as opposed to the code
+ page configured for the current user), which would quite possibly break
+ e.g. diff or merge helpers. So we really cannot override the code page.

- In this developers' setup, HAVE_LIBCHARSET_H is apparently defined, but
- we *really* want to override the locale_charset() here.
+ In init_gettext_charset(), Git calls gettext's
+ bind_textdomain_codeset() with the character set obtained via
+ locale_charset(); Let's override that latter function to force the
+ encoding to UTF-8 on native Windows.
+
+ In Git for Windows' SDK, there is a libcharset.h and therefore we
+ define HAVE_LIBCHARSET_H in the MINGW-specific section in
+ config.mak.uname, therefore we need to add the override before that
+ conditionally-compiled code block.
+
+ Rather than simply defining locale_charset() to return the string
+ "UTF-8", though, we are careful not to break LC_ALL=C: the
+ ab/no-kwset patch series, for example, needs to have a way to prevent
+ Git from expecting UTF-8-encoded input.

@@ -26,7 +43,23 @@
# include <libintl.h>
-# ifdef HAVE_LIBCHARSET_H
+# ifdef GIT_WINDOWS_NATIVE
-+# define locale_charset() "UTF-8"
++
++static const char *locale_charset(void)
++{
++ const char *env = getenv("LC_ALL"), *dot;
++
++ if (!env || !*env)
++ env = getenv("LC_CTYPE");
++ if (!env || !*env)
++ env = getenv("LANG");
++
++ if (!env)
++ return "UTF-8";
++
++ dot = strchr(env, '.');
++ return !dot ? env : dot + 1;
++}
++
+# elif defined HAVE_LIBCHARSET_H
# include <libcharset.h>
# else

gitgitgadget

From: Karsten Blees <[email protected]>

On native Windows, Git exclusively uses UTF-8 for console output (both with MinTTY and native Win32 Console). Gettext uses setlocale() to determine the output encoding for translated text, however, MSVCRT's setlocale() does not support UTF-8. As a result, translated text is encoded in system encoding (as per GetAPC()), and non-ASCII chars are mangled in console output.

Side note: There is actually a code page for UTF-8: 65001. In practice, it does not work as expected at least on Windows 7, though, so we cannot use it in Git. Besides, if we overrode the code page, any process spawned from Git would inherit that code page (as opposed to the code page configured for the current user), which would quite possibly break e.g. diff or merge helpers. So we really cannot override the code page.

In init_gettext_charset(), Git calls gettext's bind_textdomain_codeset() with the character set obtained via locale_charset(); Let's override that latter function to force the encoding to UTF-8 on native Windows.

In Git for Windows' SDK, there is a libcharset.h and therefore we define HAVE_LIBCHARSET_H in the MINGW-specific section in config.mak.uname, therefore we need to add the override before that conditionally-compiled code block.

Rather than simply defining locale_charset() to return the string "UTF-8", though, we are careful not to break LC_ALL=C: the ab/no-kwset patch series, for example, needs to have a way to prevent Git from expecting UTF-8-encoded input.

---
 gettext.c | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

index d4021d690c..3f2aca5c3b 100644
@@ -12,7 +12,25 @@
 #ifndef NO_GETTEXT
 #	include <locale.h>
 #	include <libintl.h>
-#	ifdef HAVE_LIBCHARSET_H
+#	ifdef GIT_WINDOWS_NATIVE
+
+static const char *locale_charset(void)
+{
+	const char *env = getenv("LC_ALL"), *dot;
+
+	if (!env || !*env)
+		env = getenv("LC_CTYPE");
+	if (!env || !*env)
+		env = getenv("LANG");
+
+	if (!env)
+		return "UTF-8";
+
+	dot = strchr(env, '.');
+	return !dot ? env : dot + 1;
+}
+
+#	elif defined HAVE_LIBCHARSET_H
 #		include <libcharset.h>
 #	else
 #		include <langinfo.h>

gitgitgadget

It's not just the ab/no-kwset I have cooking (but happy to have this take that into account), but also anything grep-like is usually must faster with LC_ALL=C. Isn't that also the case on Windows? Setting locales affects a large variety of libc functions and third party libraries (e.g. PCRE via us setting "use UTF-8" under locale).

Hi Ævar,

Probably. I have never tested this.

Yes, but as I mentioned in the commit message, setting locales in MINGW programs is murky at best. There is the idea of gettext, and there is the idea of Windows, and they are likely a bit different from one another.

I copy/edited that code from a later code block in is_utf8_locale() that is also conditional (under NO_GETTEXT, hence no attempt to refactor it, as that would make the code even less readable).

So I am fairly confident that the code is _correct_.

Whether it is elegant, I cannot really say. It strikes me as ugly, in those indented #ifdef..#endif guards, yet I did not find a way to make it less ugly.

Based on what? If the switch is a compile time switch, then this function must be under the same compile time guard, otherwise GCC will complain about an unused static function.

How does that even avoid complaints by GCC about dead code.

That comment says:

# Define HAVE_LIBCHARSET_H if you haven't set NO_GETTEXT and you can't

# trust the langinfo.h's nl_langinfo(CODESET) function to return the
# current character set. [...]

I think it still applies.

MinGW is a really old, outdated thing. These days, mingw-w64 is the rage (it even supports building 64-bit binaries, would you believe that? </sarcasm>). And neither is "native", strictly, although it is as native as you can get with GCC.

And as you say, the macro thing makes it hard/impossible to decide in Makefile whether we want to pass HAVE_LIBCHARSET_H or not. So we should do it independently of whether we're on Windows or not.

I don't think that this is wrong, as it is correct in pretty much all circumstances except Git for Windows. And even in Git for Windows it is
correct: we do have a libcharset.h. We just can't use it to determine the current locale.

Ciao,
Dscho

I do not think this change has much to do with HAVE_LIBCHARSET_H; it inserts "regardless of what we have been doing, do this new thing only and always on windows (persumably '... because libcharset would not be useful on that platform')".

Existing users of HAVE_LIBCHARSET_H and existing non-windows users that did not use HAVE_LIBCHARSET_H are not affected, and whatever Makefile documents the macro as still applies to them.

OK, so init_gettext_charset() is the only caller of locale_charset() in our codebase, and we supply our own locale_charset() if we do not have <libcharset.h>, either with nl_langinfo(), or with the code introduced by the patch in question for windows. Your suggestion is to add a block of #ifdef cascade in init_gettext_charset() to call locale_charset(), nl_langinfo(), or the windows-only code (perhaps inlined right there)?

I am not sure. We'd need the conditional inclusion of header files depending on HAVE_LIBCHARSET_H etc. anyway, so...