gettext CTYPE for libc

PATCH Trần Ngọc Quân 4 participants 5 messages Jun 15, 2016

Share & tools

Trần Ngọc Quân

author

2016-06-15 22:59 UTC #0 lore raw

Hello,

$ mkdir xyz
$ cd xyz
$ rmdir ../xyz
$ git status
fatal: Unable to read current working directory: Kh?ng c? t?p tin ho?c
th? m?c nh? v?y

So, somthing wrong with our charset.

$ strace git status 2>&1 | grep open
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib/i386-linux-gnu/libz.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/lib/i386-linux-gnu/libcrypto.so.1.0.0", O_RDONLY|O_CLOEXEC) = 3
open("/lib/i386-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
open("/lib/i386-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
open("/lib/i386-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/dev/null", O_RDWR|O_LARGEFILE)   = 3
open("/usr/lib/locale/locale-archive", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
open("/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 3
open("/usr/share/locale/vi_VN/LC_MESSAGES/libc.mo", O_RDONLY) = -1
ENOENT (No such file or directory)
open("/usr/share/locale/vi/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT
(No such file or directory)
open("/usr/share/locale-langpack/vi_VN/LC_MESSAGES/libc.mo", O_RDONLY) =
-1 ENOENT (No such file or directory)
open("/usr/share/locale-langpack/vi/LC_MESSAGES/libc.mo", O_RDONLY) = 3
open("/usr/lib/i386-linux-gnu/gconv/gconv-modules.cache", O_RDONLY) = 3

We will see, this string come from libc.mo

$ gettext --domain=libc "No such file or directory"
Không có tập tin hoặc thư mục như vậy

in git's gettext.c, it not allow CTYPE="" for all domain, so we will set this one individually. In this ex. I set it for libc:

$ git diff

gettext.c +1 -0

index 71e9545..abd3978 100644
@@ -115,6 +115,7 @@ static void init_gettext_charset(const char *domain)
        setlocale(LC_CTYPE, "");
        charset = locale_charset();
        bind_textdomain_codeset(domain, charset);
+       bind_textdomain_codeset("libc", charset);
        setlocale(LC_CTYPE, "C");
 }

And it work as I expect!

Signature · 1 line

Trần Ngọc Quân.

Thomas Rast

2016-06-15 22:59 UTC #1 lore raw

Quoted 5 lines · Trần Ngọc Quân

$ git status
fatal: Unable to read current working directory: Kh?ng c? t?p tin ho?c
th? m?c nh? v?y

So, somthing wrong with our charset.

[...]

Quoted 12 lines

$ gettext --domain=libc "No such file or directory"
Không có tập tin hoặc thư mục như vậy
in git's gettext.c, it not allow CTYPE="" for all domain, so we will set this one individually. In this ex. I set it for libc:
$ git diff
gettext.c +0 -0
index 71e9545..abd3978 100644

gettext.c static void init_gettext_charset(const char *domain) · line 120

Quoted 6 lines

gettext.c +1 -0

@@ -115,6 +115,7 @@ static void init_gettext_charset(const char *domain)
        setlocale(LC_CTYPE, "");
        charset = locale_charset();
        bind_textdomain_codeset(domain, charset);
+       bind_textdomain_codeset("libc", charset);
        setlocale(LC_CTYPE, "C");
 }

Do you know why this "suddenly" broke? The long comment in init_gettext_charset() suggests that the *existing* code is there to handle exactly this problem, and apparently it doesn't. Why? Has libc moved the perror() strings into a separate domain in some version?

Signature · 2 lines

Thomas Rast
[email protected]

Trần Ngọc Quân

author

2016-06-15 22:59 UTC #2 lore raw

Quoted 9 lines · Thomas Rast

Quoted 5 lines · Trần Ngọc Quân
$ git status
fatal: Unable to read current working directory: Kh?ng c? t?p tin ho?c
th? m?c nh? v?y
So, somthing wrong with our charset.
[...]
Do you know why this "suddenly" broke?

I think git set CTYPE="C" for libc, so charset become 7-bit ASCII, but it don't set LC_MESSAGES="C" for libc and libc will get this one from system variable.

Quoted 4 lines

The long comment in
init_gettext_charset() suggests that the *existing* code is there to handle exactly this problem, and apparently it doesn't. Why? Has libc moved the perror() strings into a separate domain in some version?

See setlocale(3) [1]
I'm a newbie in GIT. I'm not sure about git work correctly [2] if set git's charset to same with system.
I don't think libc moved perror() string in separate domain. It use its own domain.

[1] http://man7.org/linux/man-pages/man3/setlocale.3.html
[2] incorrect if some function need work in ASCII mode

Signature · 1 line

Trần Ngọc Quân.

Duy Nguyen

2016-06-15 22:59 UTC #3 lore raw

In reply to Trần Ngọc Quân

Jonathan I see you participated in bug 6530, mentioned in the big comment block in init_gettext_charset(). The bug seems fixed since glibc 2.17. Do you think it's ok to revert the workaround if we detect the running glibc is fixed (or if it does not run glibc at all)? I think we could use gnu_get_libc_version() to detect it.

[1] https://sourceware.org/bugzilla/show_bug.cgi?id=6530

Quoted 59 lines · Trần Ngọc Quân

Hello,

$ mkdir xyz
$ cd xyz
$ rmdir ../xyz
$ git status
fatal: Unable to read current working directory: Kh?ng c? t?p tin ho?c
th? m?c nh? v?y

So, somthing wrong with our charset.

$ strace git status 2>&1 | grep open
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib/i386-linux-gnu/libz.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/lib/i386-linux-gnu/libcrypto.so.1.0.0", O_RDONLY|O_CLOEXEC) = 3
open("/lib/i386-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
open("/lib/i386-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
open("/lib/i386-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/dev/null", O_RDWR|O_LARGEFILE)   = 3
open("/usr/lib/locale/locale-archive", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
open("/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 3
open("/usr/share/locale/vi_VN/LC_MESSAGES/libc.mo", O_RDONLY) = -1
ENOENT (No such file or directory)
open("/usr/share/locale/vi/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT
(No such file or directory)
open("/usr/share/locale-langpack/vi_VN/LC_MESSAGES/libc.mo", O_RDONLY) =
-1 ENOENT (No such file or directory)
open("/usr/share/locale-langpack/vi/LC_MESSAGES/libc.mo", O_RDONLY) = 3
open("/usr/lib/i386-linux-gnu/gconv/gconv-modules.cache", O_RDONLY) = 3

We will see, this string come from libc.mo

$ gettext --domain=libc "No such file or directory"
Không có tập tin hoặc thư mục như vậy

in git's gettext.c, it not allow CTYPE="" for all domain, so we will set this one individually. In this ex. I set it for libc:

$ git diff

gettext.c +1 -0

index 71e9545..abd3978 100644
@@ -115,6 +115,7 @@ static void init_gettext_charset(const char *domain)
        setlocale(LC_CTYPE, "");
        charset = locale_charset();
        bind_textdomain_codeset(domain, charset);
+       bind_textdomain_codeset("libc", charset);
        setlocale(LC_CTYPE, "C");
 }

And it work as I expect!

--
Trần Ngọc Quân.

--
To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 1 line

Duy

Jonathan Nieder

2016-06-15 22:59 UTC #4 lore raw

Quoted 3 lines · Duy Nguyen

Do you think it's ok to revert the workaround if we detect
the running glibc is fixed (or if it does not run glibc at all)? I think we could use gnu_get_libc_version() to detect it.

That would be wonderful.

Thanks,
Jonathan