Skip to contentSkip to conversation

package/systemd: further fix for locale-related issues

PATCH New Yann E. MORIN 2 participants 9 messages Mar 4, 2018 Kbuild

lore
Share & tools
lore
#0 lore raw

cover package/systemd: further fix for locale-related issues

Hello All!

As reported by Christian, not all distros have a way to provide C.UTF-8. For example, Gentoo does not have it, and it is noteven possible to generate it (damn...).

So we go another route: find any UTF-8 locale from the user's system, still preferring the one they expect to see messages in.

Also, unbreak the check for an UTF-8 locale.

Regards,
Yann E. MORIN.

The following changes since commit 63497e02607543c6b27c8b59c026e78b0337dbaf

package/tor: security bump to version 0.3.1.10 (2018-03-04 11:01:45 +0100)

are available in the git repository at:

git://git.buildroot.org/~ymorin/git/buildroot.git

for you to fetch changes up to 27e66e08ce8a1493c52a43e2a34aa1ed75a043f9

core: drop no-longer used C.UTF-8 locale option (2018-03-04 11:28:29 +0100)

----------------------------------------------------------------
Yann E. MORIN (4):
support/dependencies: unbreak check for UTF-8 locale
core: find a host UTF-8 locale package/systemd: needs any UTF-8 locale
core: drop no-longer used C.UTF-8 locale option

 Config.in                            |  5 -----
 Makefile                             | 16 ++++++++++++++++
 package/systemd/Config.in            |  2 +-
 package/systemd/systemd.mk           |  2 +-
 support/dependencies/dependencies.sh | 15 +--------------
 5 files changed, 19 insertions(+), 21 deletions(-)
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#1 lore raw

PATCH 1/4 support/dependencies: unbreak check for UTF-8 locale

Although the UTF-8 locales in mainstream distributions all are suffixed with just 'utf8', the nomenclature is a bit ambiguous with the way they are to be specified with the various LC_* variables, suffixed there with 'UTF-8'.

Also, POSIX, ISO, and IEC do not enforce any specific suffix in LC_* variables:
http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html#tag_08_02

"""
If the locale value has the form:
language[_territory][.codeset]

it refers to an implementation-provided locale, where settings of language, territory, and codeset are implementation-defined. """

To avoid any confusion, use a regexp that is a bit more lax when matching locales.

Also, quote the regexp, so that the '?' and '$' are not interpreted by the shell.

Cc Peter Korsgaard <[email protected]>
Cc Thomas Petazzoni <[email protected]>
---
 support/dependencies/dependencies.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
support/dependencies/dependencies.sh +1 -1
index 6698f21a7a..656cb09d42 100755
@@ -200,7 +200,7 @@ if grep ^BR2_NEEDS_HOST_UTF8_LOCALE=y $BR2_CONFIG > /dev/null; then
 		echo "You need locale support on your build machine to build a toolchain supporting locales"
 		exit 1 ;
 	fi
-	if ! locale -a | grep -q -i utf8$ ; then
+	if ! locale -a | grep -q -i -E 'utf-?8$' ; then
 		echo
 		echo "You need at least one UTF8 locale to build a toolchain supporting locales"
 		exit 1 ;
Signature · 1 line
2.14.1
#2 lore raw

PATCH 1/4 support/dependencies: unbreak check for UTF-8 locale

Quoted 22 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

Although the UTF-8 locales in mainstream distributions all are suffixed with just 'utf8', the nomenclature is a bit ambiguous with the way they are to be specified with the various LC_* variables, suffixed there with 'UTF-8'.

Also, POSIX, ISO, and IEC do not enforce any specific suffix in LC_* variables:
http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html#tag_08_02

"""
If the locale value has the form:
language[_territory][.codeset]

it refers to an implementation-provided locale, where settings of language, territory, and codeset are implementation-defined. """

To avoid any confusion, use a regexp that is a bit more lax when matching locales.

Quoted 6 lines

Also, quote the regexp, so that the '?' and '$' are not interpreted by the shell.

Cc Peter Korsgaard <[email protected]>
Cc Thomas Petazzoni <[email protected]>

Committed, thanks.

Signature · 1 line
Bye, Peter Korsgaard
#3 lore raw

PATCH 2/4 core: find a host UTF-8 locale

In reply to Yann E. MORIN

Some packages really want to use an UTF-8 locale, or they break.

However, there is no guarantee that any given locale is available on a system. For example,, while most mainstream distros (Debian and derivatives, Fedora...) do have the generic, language-agnostic C.UTF-8 locale, Gentoo does not provide it.

So, find the first UTF-8 locale available on the system, and take any that is available. We however do favour using the user-set current locale, then using the language-agnostic C.UTF-8, and eventually any random C.UTF-8 locale.

Note: we only need to enforce LC_ALL, because setting it implies everything else:

    http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html#tag_08_02

    """
    1. If the LC_ALL environment variable is defined and is not null,
    the value of LC_ALL shall be used.
    """
---
 Makefile | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
Makefile +16 -0
index 5fe3a34c47..aca4e66302 100644
@@ -363,6 +363,22 @@ ifneq ($(firstword $(HOSTCC_VERSION)),4)
 HOSTCC_VERSION := $(firstword $(HOSTCC_VERSION))
 endif
 
+ifeq ($(BR2_NEEDS_HOST_UTF8_LOCALE),y)
+# First, we try to use the user's configured locale (as that's the
+# language they'd expect messages to be displayed), then we favour
+# a non language-specific locale like C.UTF-8 if one is available,
+# so we sort with the C locale to get it at the top.
+# This is guaranteed to not be empty, because of the check in
+# support/dependencies/dependencies.sh
+HOST_UTF8_LOCALE := $(shell \
+			( echo $${LC_ALL:-$${LC_MESSAGES:-$${LANG}}}; \
+			  locale -a 2>/dev/null |LC_ALL=C sort \
+			) \
+			|grep -i -E 'utf-?8' \
+			|head -n 1)
+HOST_UTF8_LOCALE_ENV := LC_ALL=$(HOST_UTF8_LOCALE)
+endif
+
 # Make sure pkg-config doesn't look outside the buildroot tree
 HOST_PKG_CONFIG_PATH := $(PKG_CONFIG_PATH)
 unexport PKG_CONFIG_PATH
Signature · 1 line
2.14.1
#4 lore raw

PATCH 2/4 core: find a host UTF-8 locale

Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

Some packages really want to use an UTF-8 locale, or they break. However, there is no guarantee that any given locale is available on a system. For example,, while most mainstream distros (Debian and derivatives, Fedora...) do have the generic, language-agnostic C.UTF-8 locale, Gentoo does not provide it.

Quoted 4 lines

So, find the first UTF-8 locale available on the system, and take any that is available. We however do favour using the user-set current locale, then using the language-agnostic C.UTF-8, and eventually any random C.UTF-8 locale.

I guess this should be 'random UTF-8 locale', so I changed it to that.

Makefile ifneq ($(firstword $(HOSTCC_VERSION)),4) · line 377

Quoted 12 lines
Makefile +12 -0
+ifeq ($(BR2_NEEDS_HOST_UTF8_LOCALE),y)
+# First, we try to use the user's configured locale (as that's the
+# language they'd expect messages to be displayed), then we favour
+# a non language-specific locale like C.UTF-8 if one is available,
+# so we sort with the C locale to get it at the top.
+# This is guaranteed to not be empty, because of the check in
+# support/dependencies/dependencies.sh
+HOST_UTF8_LOCALE := $(shell \
+			( echo $${LC_ALL:-$${LC_MESSAGES:-$${LANG}}}; \
+			  locale -a 2>/dev/null |LC_ALL=C sort \
+			) \
+			|grep -i -E 'utf-?8' \

For consistency with dependencies.sh I changed the regexp to 'utf-?8$'.

Committed with these changes, thanks.

Signature · 1 line
Bye, Peter Korsgaard
#5 lore raw

PATCH 3/4 package/systemd: needs any UTF-8 locale

In reply to Yann E. MORIN

Not all distributions have the language-agnostic C.UTF-8 locale (Gentoo, I'm frowning at you!).

Instead, use any UTF-8 locale provided by the system.

Cc Maxime Hadjinlian <[email protected]>
Cc Julius Kriukas <[email protected]>
Cc Christian Stewart <[email protected]>
Cc Trent Piepho <[email protected]>
Cc Adam Duskett <[email protected]>
---
 package/systemd/Config.in  | 2 +-
 package/systemd/systemd.mk | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
package/systemd/Config.in +1 -1
index 8978e3bac1..7b613df02c 100644
@@ -27,7 +27,7 @@ menuconfig BR2_PACKAGE_SYSTEMD
 	select BR2_PACKAGE_BUSYBOX_SHOW_OTHERS # kmod-tools
 	select BR2_PACKAGE_KMOD_TOOLS
 	select BR2_TARGET_TZ_INFO
-	select BR2_NEEDS_HOST_C_UTF8_LOCALE
+	select BR2_NEEDS_HOST_UTF8_LOCALE
 	help
 	  systemd is a system and service manager for Linux,
 	  compatible with SysV and LSB init scripts. systemd provides
package/systemd/systemd.mk +1 -1
index 793b966ff5..f47dfe7b1d 100644
@@ -385,7 +385,7 @@ endef
 
 SYSTEMD_NINJA_OPTS = $(if $(VERBOSE),-v) -j$(PARALLEL_JOBS)
 
-SYSTEMD_ENV = $(TARGET_MAKE_ENV) LANG=C.UTF-8 LC_ALL=C.UTF-8
+SYSTEMD_ENV = $(TARGET_MAKE_ENV) $(HOST_UTF8_LOCALE_ENV)
 
 define SYSTEMD_CONFIGURE_CMDS
 	rm -rf $(@D)/build
Signature · 1 line
2.14.1
#6 lore raw

PATCH 3/4 package/systemd: needs any UTF-8 locale

Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

Not all distributions have the language-agnostic C.UTF-8 locale (Gentoo, I'm frowning at you!).

Instead, use any UTF-8 locale provided by the system.

Quoted 7 lines
Cc Maxime Hadjinlian <[email protected]>
Cc Julius Kriukas <[email protected]>
Cc Christian Stewart <[email protected]>
Cc Trent Piepho <[email protected]>
Cc Adam Duskett <[email protected]>

Committed, thanks.

Signature · 1 line
Bye, Peter Korsgaard
#7 lore raw

PATCH 4/4 core: drop no-longer used C.UTF-8 locale option

In reply to Yann E. MORIN

Its use has been globbed into the more generic
BR2_NEEDS_HOST_UTF8_LOCALE option now.

Cc Thomas Petazzoni <[email protected]>
Cc Peter Korsgaard <[email protected]>
Cc Julius Kriukas <[email protected]>
Cc Christian Stewart <[email protected]>
---
 Config.in                            |  5 -----
 support/dependencies/dependencies.sh | 13 -------------
 2 files changed, 18 deletions(-)
Config.in +0 -5
index 704840653e..0002df5176 100644
@@ -94,11 +94,6 @@ config BR2_HOSTARCH_NEEDS_IA32_COMPILER
 config BR2_NEEDS_HOST_UTF8_LOCALE
 	bool
 
-# Hidden boolean selected by packages that explicitly need C.UTF-8 locale
-# to be available on host system.
-config BR2_NEEDS_HOST_C_UTF8_LOCALE
-	bool
-
 source "arch/Config.in"
 
 menu "Build options"
support/dependencies/dependencies.sh +0 -13
index 656cb09d42..985b1d863b 100755
@@ -207,19 +207,6 @@ if grep ^BR2_NEEDS_HOST_UTF8_LOCALE=y $BR2_CONFIG > /dev/null; then
 	fi
 fi
 
-if grep -q ^BR2_NEEDS_HOST_C_UTF8_LOCALE=y $BR2_CONFIG; then
-	if ! which locale > /dev/null ; then
-		echo
-		echo "You need locale support on your build machine"
-		exit 1 ;
-	fi
-	if ! LC_ALL=C.UTF-8 locale -c charmap | grep -q '^UTF-8$'; then
-		echo
-		echo "You need C.UTF-8 locale suppport on the host system"
-		exit 1 ;
-	fi
-fi
-
 if grep -q ^BR2_NEEDS_HOST_JAVA=y $BR2_CONFIG ; then
 	check_prog_host "java"
 	JAVA_GCJ=$(java -version 2>&1 | grep gcj)
Signature · 1 line
2.14.1
#8 lore raw

PATCH 4/4 core: drop no-longer used C.UTF-8 locale option

Quoted 10 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

Its use has been globbed into the more generic
BR2_NEEDS_HOST_UTF8_LOCALE option now.

Cc Thomas Petazzoni <[email protected]>
Cc Peter Korsgaard <[email protected]>
Cc Julius Kriukas <[email protected]>
Cc Christian Stewart <[email protected]>

Committed, thanks.

Signature · 1 line
Bye, Peter Korsgaard