Skip to contentSkip to conversation

gtk: use setlocale() for LC_MESSAGES only

PATCH In review Alberto Garcia 5 participants 8 messages Sep 10, 2015 – Dec 21, 2015

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 gtk: use setlocale() for LC_MESSAGES only

The QEMU code is not internationalized and assumes that it runs under the C locale, but if we use the GTK+ UI we'll end up importing the locale settings from the environment. This can break things, such as the JSON generator and iotest 120 in locales that use a decimal comma.

We do however have translations for a few simple strings for the GTK+ menu items, so in order to run QEMU using the C locale, and yet have a translated UI let's use setlocale() for LC_MESSAGES only.

---
 ui/gtk.c | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)
ui/gtk.c +20 -1
index df2a79e..11ea2cf 100644
@@ -1941,7 +1941,8 @@ void gtk_display_init(DisplayState *ds, bool full_screen, bool grab_on_hover)
 
     s->free_scale = FALSE;
 
-    setlocale(LC_ALL, "");
+    /* LC_MESSAGES only. See early_gtk_display_init() for details */
+    setlocale(LC_MESSAGES, "");
     bindtextdomain("qemu", CONFIG_QEMU_LOCALEDIR);
     textdomain("qemu");
 
@@ -2010,6 +2011,24 @@ void gtk_display_init(DisplayState *ds, bool full_screen, bool grab_on_hover)
 
 void early_gtk_display_init(int opengl)
 {
+    /* The QEMU code relies on the assumption that it's always run in
+     * the C locale. Therefore it is not prepared to deal with
+     * operations that produce different results depending on the
+     * locale, such as printf's formatting of decimal numbers, and
+     * possibly others.
+     *
+     * Since GTK+ calls setlocale() by default -importing the locale
+     * settings from the environment- we must prevent it from doing so
+     * using gtk_disable_setlocale().
+     *
+     * QEMU's GTK+ UI, however, _does_ have translations for some of
+     * the menu items. As a trade-off between a functionally correct
+     * QEMU and a fully internationalized UI we support importing
+     * LC_MESSAGES from the environment (see the setlocale() call
+     * earlier in this file). This allows us to display translated
+     * messages leaving everything else untouched.
+     */
+    gtk_disable_setlocale();
     gtkinit = gtk_init_check(NULL, NULL);
     if (!gtkinit) {
         /* don't exit yet, that'll break -help */
Signature · 1 line
2.5.1
#1 lore raw
Quoted 10 lines

void early_gtk_display_init(int opengl)
{

ui/gtk.c +8 -0
+    /* The QEMU code relies on the assumption that it's always run in
+     * the C locale. Therefore it is not prepared to deal with
+     * operations that produce different results depending on the
+     * locale, such as printf's formatting of decimal numbers, and
+     * possibly others.
+     *
+     * Since GTK+ calls setlocale() by default -importing the locale
+     * settings from the environment- we must prevent it from doing so

ui/gtk.c void gtk_display_init(DisplayState *ds, bool full_screen, bool grab_on_hover) · line 2031

Quoted 10 lines
ui/gtk.c +10 -0
+     * using gtk_disable_setlocale().
+     *
+     * QEMU's GTK+ UI, however, _does_ have translations for some of
+     * the menu items. As a trade-off between a functionally correct
+     * QEMU and a fully internationalized UI we support importing
+     * LC_MESSAGES from the environment (see the setlocale() call
+     * earlier in this file). This allows us to display translated
+     * messages leaving everything else untouched.
+     */
+    gtk_disable_setlocale();

Thanks. Replacing my version with this one.

cheers,
Gerd

#2 lore raw
In reply to Alberto Garcia

Am 10.09.2015 um 17:19 hat Alberto Garcia geschrieben:

Quoted 10 lines

The QEMU code is not internationalized and assumes that it runs under the C locale, but if we use the GTK+ UI we'll end up importing the locale settings from the environment. This can break things, such as the JSON generator and iotest 120 in locales that use a decimal comma.

We do however have translations for a few simple strings for the GTK+ menu items, so in order to run QEMU using the C locale, and yet have a translated UI let's use setlocale() for LC_MESSAGES only.

Not sure why I noticed it only now and if it's related to any recent package upgrade on my side (using RHEL 7), but I noticed that non-ASCII characters in the GTK UI strings are broken for me and git bisect pointed to this commit.

Kevin

Quoted 46 lines
 ui/gtk.c | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)
ui/gtk.c +20 -1
index df2a79e..11ea2cf 100644
@@ -1941,7 +1941,8 @@ void gtk_display_init(DisplayState *ds, bool full_screen, bool grab_on_hover)
 
     s->free_scale = FALSE;
 
-    setlocale(LC_ALL, "");
+    /* LC_MESSAGES only. See early_gtk_display_init() for details */
+    setlocale(LC_MESSAGES, "");
     bindtextdomain("qemu", CONFIG_QEMU_LOCALEDIR);
     textdomain("qemu");
 
@@ -2010,6 +2011,24 @@ void gtk_display_init(DisplayState *ds, bool full_screen, bool grab_on_hover)
 
 void early_gtk_display_init(int opengl)
 {
+    /* The QEMU code relies on the assumption that it's always run in
+     * the C locale. Therefore it is not prepared to deal with
+     * operations that produce different results depending on the
+     * locale, such as printf's formatting of decimal numbers, and
+     * possibly others.
+     *
+     * Since GTK+ calls setlocale() by default -importing the locale
+     * settings from the environment- we must prevent it from doing so
+     * using gtk_disable_setlocale().
+     *
+     * QEMU's GTK+ UI, however, _does_ have translations for some of
+     * the menu items. As a trade-off between a functionally correct
+     * QEMU and a fully internationalized UI we support importing
+     * LC_MESSAGES from the environment (see the setlocale() call
+     * earlier in this file). This allows us to display translated
+     * messages leaving everything else untouched.
+     */
+    gtk_disable_setlocale();
     gtkinit = gtk_init_check(NULL, NULL);
     if (!gtkinit) {
         /* don't exit yet, that'll break -help */
Signature · 1 line
2.5.1
#3 lore raw
Quoted 10 lines · Kevin Wolf

Am 10.09.2015 um 17:19 hat Alberto Garcia geschrieben:

Quoted 9 lines

The QEMU code is not internationalized and assumes that it runs under the C locale, but if we use the GTK+ UI we'll end up importing the locale settings from the environment. This can break things, such as the JSON generator and iotest 120 in locales that use a decimal comma.

We do however have translations for a few simple strings for the GTK+ menu items, so in order to run QEMU using the C locale, and yet have a translated UI let's use setlocale() for LC_MESSAGES only.

Quoted 6 lines
Quoted 1 line

Not sure why I noticed it only now and if it's related to any recent package upgrade on my side (using RHEL 7), but I noticed that non-ASCII characters in the GTK UI strings are broken for me and git bisect pointed to this commit.

I guess we need to set LC_CTYPE too.
Can you try whenever the attached patch fixes the issue?

thanks,
Gerd

From 54821a4b405ca31c997485b563ec5c43dd53e4ed Mon Sep 17 00:00:00 2001
From: Gerd Hoffmann <[email protected]>
Date: Fri, 18 Dec 2015 14:15:56 +0100
Subject: [PATCH] gtk: fix utf8 strings in the ui

Commit "2cb5d2a gtk: use setlocale() for LC_MESSAGES only" restricts locate settings to LC_MESSAGES, to avoid bugs caused by locale-specific number printing (LC_NUMERIC) and possibly others.

We need LC_CTYPE too to make messages with chars outside us-ascii work correctly. Add it.

---
 ui/gtk.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
ui/gtk.c +2 -1
index 47b37e1..30407a5 100644
@@ -2044,8 +2044,9 @@ void gtk_display_init(DisplayState *ds, bool full_screen, bool grab_on_hover)
 
     s->free_scale = FALSE;
 
-    /* LC_MESSAGES only. See early_gtk_display_init() for details */
+    /* LC_MESSAGES+LC_CTYPE only. See early_gtk_display_init() for details */
     setlocale(LC_MESSAGES, "");
+    setlocale(LC_CTYPE, "");
     bindtextdomain("qemu", CONFIG_QEMU_LOCALEDIR);
     textdomain("qemu");
 
Signature · 1 line
1.8.3.1
Tested-by
#4 lore raw

Am 18.12.2015 um 14:23 hat Gerd Hoffmann geschrieben:

Quoted 13 lines · Kevin Wolf
Quoted 12 lines

Am 10.09.2015 um 17:19 hat Alberto Garcia geschrieben:

Quoted 10 lines

The QEMU code is not internationalized and assumes that it runs under the C locale, but if we use the GTK+ UI we'll end up importing the locale settings from the environment. This can break things, such as the JSON generator and iotest 120 in locales that use a decimal comma.

We do however have translations for a few simple strings for the GTK+ menu items, so in order to run QEMU using the C locale, and yet have a translated UI let's use setlocale() for LC_MESSAGES only.

Quoted 7 lines
Quoted 4 lines

Not sure why I noticed it only now and if it's related to any recent package upgrade on my side (using RHEL 7), but I noticed that non-ASCII characters in the GTK UI strings are broken for me and git bisect pointed to this commit.

I guess we need to set LC_CTYPE too.
Can you try whenever the attached patch fixes the issue?

Yes, that works for me.

#5 lore raw
In reply to Gerd Hoffmann
Quoted 11 lines
Quoted 9 lines
Quoted 5 lines

We do however have translations for a few simple strings for the GTK+ menu items, so in order to run QEMU using the C locale, and yet have a translated UI let's use setlocale() for LC_MESSAGES only.

Not sure why I noticed it only now and if it's related to any recent package upgrade on my side (using RHEL 7), but I noticed that non-ASCII characters in the GTK UI strings are broken for me and git bisect pointed to this commit.

I guess we need to set LC_CTYPE too.

That affects functions in ctype.h (isalpha(), islower(), isupper(), ...) I guess that's safe?

ui/gtk.c void gtk_display_init(DisplayState *ds, bool full_screen, bool grab_on_hover) · line 2051

Quoted 10 lines
ui/gtk.c +2 -1
@@ -2044,8 +2044,9 @@ void gtk_display_init(DisplayState *ds, bool full_screen, bool grab_on_hover)
 
     s->free_scale = FALSE;
 
-    /* LC_MESSAGES only. See early_gtk_display_init() for details */
+    /* LC_MESSAGES+LC_CTYPE only. See early_gtk_display_init() for details */
     setlocale(LC_MESSAGES, "");
+    setlocale(LC_CTYPE, "");
     bindtextdomain("qemu", CONFIG_QEMU_LOCALEDIR);
     textdomain("qemu");

You can also modify the comment in early_gtk_display_init() to say that " we support importing LC_MESSAGES and LC_CTYPE from the environment ".

Berto

#6 lore raw
Quoted 9 lines · Alberto Garcia
Quoted 9 lines
Quoted 8 lines
Quoted 4 lines

We do however have translations for a few simple strings for the GTK+ menu items, so in order to run QEMU using the C locale, and yet have a translated UI let's use setlocale() for LC_MESSAGES only.

Not sure why I noticed it only now and if it's related to any recent package upgrade on my side (using RHEL 7), but I noticed that non-ASCII characters in the GTK UI strings are broken for me and git bisect pointed to this commit.

Quoted 4 lines
Quoted 1 line

I guess we need to set LC_CTYPE too.

That affects functions in ctype.h (isalpha(), islower(), isupper(), ...) I guess that's safe?

If we're guessing, then I guess it isn't. But we shouldn't be guessing.

"LC_CTYPE affects the behavior of the character handling functions and the multibyte and wide character functions."

I doubt there's much use for the latter in QEMU itself, but in libraries, all bets are off. I guess this is what actually screws up GTK.

We do use the former. LC_CTYPE set to some sufficiently funky locale is bound to upset these uses.

In short: nope, we can't just set LC_CTYPE, at least not without further analysis.

We should've stayed out of the GUI business.

[...]

#7 lore raw
Quoted 11 lines · Markus Armbruster
Quoted 9 lines · Alberto Garcia
Quoted 9 lines
Quoted 8 lines
Quoted 4 lines

We do however have translations for a few simple strings for the GTK+ menu items, so in order to run QEMU using the C locale, and yet have a translated UI let's use setlocale() for LC_MESSAGES only.

Not sure why I noticed it only now and if it's related to any recent package upgrade on my side (using RHEL 7), but I noticed that non-ASCII characters in the GTK UI strings are broken for me and git bisect pointed to this commit.

Quoted 4 lines
Quoted 4 lines
Quoted 1 line

I guess we need to set LC_CTYPE too.

That affects functions in ctype.h (isalpha(), islower(), isupper(), ...) I guess that's safe?

Gnulib introduces functions named c_isalpha(), c_islower(), and so forth, which behave identically regardless of the current locale, precisely because locale-dependent definitions on which byte sequences form a valid character can cause undesirable behavior. I don't know if glib does the same, but it does indeed have the potential to affect us, in at least util/id.c:id_wellformed(). It would be weird to let the user's choice of locale determine which ids they can create.

Quoted 10 lines

If we're guessing, then I guess it isn't. But we shouldn't be guessing.

"LC_CTYPE affects the behavior of the character handling functions and the multibyte and wide character functions."

I doubt there's much use for the latter in QEMU itself, but in libraries, all bets are off. I guess this is what actually screws up GTK.

Quoted 5 lines

We do use the former. LC_CTYPE set to some sufficiently funky locale is bound to upset these uses.

In short: nope, we can't just set LC_CTYPE, at least not without further analysis.

In fact, if LC_CTYPE and LC_COLLATE are incompatible, then strcoll() has undefined behavior. GNU coreutils warns:

Unless otherwise specified, all comparisons use the character collating sequence specified by the ‘LC_COLLATE’ locale.(1) [...]
(1) If you use a non-POSIX locale (e.g., by setting ‘LC_ALL’ to ‘en_US’), then ‘sort’ may produce output that is sorted differently than you’re accustomed to. In that case, set the ‘LC_ALL’ environment variable to ‘C’. Note that setting only ‘LC_COLLATE’ has two problems. First, it is ineffective if ‘LC_ALL’ is also set. Second, it has undefined behavior if ‘LC_CTYPE’ (or ‘LANG’, if ‘LC_CTYPE’ is unset) is set to an incompatible value. For example, you get undefined behavior if ‘LC_CTYPE’ is ‘ja_JP.PCK’ but ‘LC_COLLATE’ is ‘en_US.UTF-8’.

Off-hand, we are specifically NOT calling setlocale() for the categories that we want to leave in the C locale, so we don't have to worry about LC_ALL throwing us off. And I'm hard-pressed to think of an example where LC_COLLATE=C while LC_CTYPE is a multibyte character will cause unusual sorting artifacts (the one that coreutils is warning against is when you have two incompatibly different multibyte character sets involved, where our case is a multibyte character set for display but a unibyte set for collation). But it is indeed a can of worms, that requires special analysis.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org