Change host locale from C to en_US.UTF-8 for reproducable builds

PATCH Superseded Julius Kriukas 7 participants 19 messages Jan 28, 2018 – Mar 4, 2018 1 message not in the archive Kbuild

lore

Share & tools

lore

Julius Kriukas

author

2018-01-28 11:22 UTC #0 lore raw

PATCH 1/1 Change host locale from C to en_US.UTF-8 for reproducable builds

When BR2_REPRODUCIBLE is enabled building systemd fails with error:

[1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command.
FAILED: catalog/systemd.bg.catalog
/buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat
Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale.
You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents.
Please switch to a UTF-8 locale for your platform.
...
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128)
package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed
make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1

Other packages using meson build system might fail as well.

This patch changes default host system locale from C to en_US.UTF-8 to fix this issue. Any Unicode-compatible locale would be suitable here.

Signed-off-by Julius Kriukas <[email protected]>

---
 Makefile | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Makefile +2 -2

index 7d8ab51a8c..5991c6a221 100644
@@ -250,8 +250,8 @@ endif
 # timezone and locale may affect build output
 ifeq ($(BR2_REPRODUCIBLE),y)
 export TZ = UTC
-export LANG = C
-export LC_ALL = C
+export LANG = en_US.UTF-8
+export LC_ALL = en_US.UTF-8
 export GZIP = -n
 BR2_VERSION_GIT_EPOCH = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
 export SOURCE_DATE_EPOCH ?= $(if $(wildcard $(TOPDIR)/.git),$(BR2_VERSION_GIT_EPOCH),$(BR2_VERSION_EPOCH))

Signature · 1 line

2.15.1

Thomas Petazzoni

2018-01-30 22:38 UTC #1 lore raw

PATCH 1/1 Change host locale from C to en_US.UTF-8 for reproducable builds

Hello,

Quoted 13 lines · Julius Kriukas

When BR2_REPRODUCIBLE is enabled building systemd fails with error:
[1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command.
FAILED: catalog/systemd.bg.catalog
/buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat
Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale.
You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents.
Please switch to a UTF-8 locale for your platform.
...
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128)
package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed
make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1

Quoted 6 lines

Other packages using meson build system might fail as well.
This patch changes default host system locale from C to en_US.UTF-8 to fix this issue. Any Unicode-compatible locale would be suitable here.
Signed-off-by Julius Kriukas <[email protected]>

Thanks for your patch! I was about to apply it, but one issue I see is that we have no guarantee that en_US.UTF-8 is available.

support/dependencies/dependencies.sh checks that we have at least one UTF-8 locale available if BR2_NEEDS_HOST_UTF8_LOCALE=y, but:

- BR2_NEEDS_HOST_UTF8_LOCALE is only selected when a uClibc toolchain is being built

- it only ensures one UTF-8 locale is available, not specifically en_US.UTF-8

Perhaps we need another BR2_NEEDS_HOST_EN_US_UTF8_LOCALE, which would be used by support/dependencies/dependencies.sh to ensure that en_US.UTF-8 is available. This new option would be selected by BR2_REPRODUCIBLE.

Arnout, Peter, Yann? Thoughts?

Best regards,

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

Julius Kriukas

author

2018-01-31 21:34 UTC #2 lore raw

PATCH 1/1 Change host locale from C to en_US.UTF-8 for reproducable builds

Quoted 40 lines · Thomas Petazzoni

Hello,
Quoted 13 lines · Julius Kriukas
When BR2_REPRODUCIBLE is enabled building systemd fails with error:
[1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command.
FAILED: catalog/systemd.bg.catalog
/buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat
Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale.
You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents.
Please switch to a UTF-8 locale for your platform.
...
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128)
package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed
make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1
Quoted 6 lines
Other packages using meson build system might fail as well.
This patch changes default host system locale from C to en_US.UTF-8 to fix this issue. Any Unicode-compatible locale would be suitable here.
Signed-off-by Julius Kriukas <[email protected]>
Thanks for your patch! I was about to apply it, but one issue I see is that we have no guarantee that en_US.UTF-8 is available.
support/dependencies/dependencies.sh checks that we have at least one UTF-8 locale available if BR2_NEEDS_HOST_UTF8_LOCALE=y, but:
- BR2_NEEDS_HOST_UTF8_LOCALE is only selected when a uClibc toolchain is being built
- it only ensures one UTF-8 locale is available, not specifically
    en_US.UTF-8
Perhaps we need another BR2_NEEDS_HOST_EN_US_UTF8_LOCALE, which would be used by support/dependencies/dependencies.sh to ensure that en_US.UTF-8 is available. This new option would be selected by BR2_REPRODUCIBLE.

Quoted 6 lines

Arnout, Peter, Yann? Thoughts?
Best regards,
Thomas

I think Debian uses C.UTF-8 locale in similar circumstances, but I suppose it is too much Debian specific (for example it seems C.UTF-8 locale is not available by default on CentOS 7 systems).

I favor Thomas proposal to add en_US.UTF-8 availability detection with a new flag. It seemed overly complicated at first but it solves the issue and I do not have anything better to suggest.

Signature · 1 line

Julius Kriukas

Julius Kriukas

author

2018-01-31 21:36 UTC #3 lore raw

PATCH v2 1/1 Change host locale from C to en_US.UTF-8 for reproducable builds

In reply to Julius Kriukas

When BR2_REPRODUCIBLE is enabled building systemd fails with error:

Other packages using meson build system might fail as well.

This patch changes default host system locale from C to en_US.UTF-8 to fix this issue. It also introduces BR2_NEEDS_HOST_EN_US_UTF8_LOCALE hidden flag that checks if this locale is available on the host system. If locale is not available error message is show and build process is stopped.

---
Changes v1 -> v2:
- Added BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that is selected by BR2_REPRODUCIBLE.
- Extended support/dependencies/dependencies.sh to check for en_US.UTF-8 locale availability if BR2_NEEDS_HOST_EN_US_UTF8_LOCALE is set.

Signed-off-by Julius Kriukas <[email protected]>

---
 Config.in                            |  6 ++++++
 Makefile                             |  4 ++--
 support/dependencies/dependencies.sh | 13 +++++++++++++
 3 files changed, 21 insertions(+), 2 deletions(-)

Config.in +6 -0

index 62d67ce957..bd14226edc 100644
@@ -94,6 +94,11 @@ config BR2_HOSTARCH_NEEDS_IA32_COMPILER
 config BR2_NEEDS_HOST_UTF8_LOCALE
 	bool
 
+# Hidden boolean selected by packages that explicitly need en_US.UTF-8 locale 
+# to be available on host system.
+config BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
+	bool
+
 source "arch/Config.in"
 
 menu "Build options"

@@ -715,6 +720,7 @@ config BR2_REPRODUCIBLE
 	bool "Make the build reproducible (experimental)"
 	# SOURCE_DATE_EPOCH support in toolchain-wrapper requires GCC 4.4
 	depends on BR2_TOOLCHAIN_GCC_AT_LEAST_4_4
+	select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
 	help
 	  This option will remove all sources of non-reproducibility
 	  from the build process. For a given Buildroot configuration,

Makefile +2 -2

index 7d8ab51a8c..5991c6a221 100644
@@ -250,8 +250,8 @@ endif
 # timezone and locale may affect build output
 ifeq ($(BR2_REPRODUCIBLE),y)
 export TZ = UTC
-export LANG = C
-export LC_ALL = C
+export LANG = en_US.UTF-8
+export LC_ALL = en_US.UTF-8
 export GZIP = -n
 BR2_VERSION_GIT_EPOCH = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
 export SOURCE_DATE_EPOCH ?= $(if $(wildcard $(TOPDIR)/.git),$(BR2_VERSION_GIT_EPOCH),$(BR2_VERSION_EPOCH))

support/dependencies/dependencies.sh +13 -0

index a195c62c8c..4a5b55e9da 100755
@@ -207,6 +207,19 @@ if grep ^BR2_NEEDS_HOST_UTF8_LOCALE=y $BR2_CONFIG > /dev/null; then
 	fi
 fi
 
+if grep ^BR2_NEEDS_HOST_EN_US_UTF8_LOCALE=y $BR2_CONFIG > /dev/null; then
+	if ! which locale > /dev/null ; then
+		echo
+		echo "You need locale support on your build machine"
+		exit 1 ;
+	fi
+	if ! LC_ALL=en_US.UTF-8 locale -c charmap | grep '^UTF-8$' > /dev/null; then
+		echo
+		echo "You need en_US.UTF-8 locale suppport on the host system"
+		exit 1 ;
+	fi
+fi
+
 if grep -q ^BR2_NEEDS_HOST_JAVA=y $BR2_CONFIG ; then
 	check_prog_host "java"
 	JAVA_GCJ=$(java -version 2>&1 | grep gcj)

Signature · 1 line

2.15.1

Yann E. MORIN

2018-02-04 15:19 UTC #4 lore raw

PATCH v2 1/1 Change host locale from C to en_US.UTF-8 for reproducable builds

Julius, All,

On 2018-01-31 23:36 +0200, Julius Kriukas spake thusly:

Quoted 13 lines

When BR2_REPRODUCIBLE is enabled building systemd fails with error:
[1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command.
FAILED: catalog/systemd.bg.catalog
/buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat
Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale.
You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents.
Please switch to a UTF-8 locale for your platform.
...
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128)
package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed
make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1

Quoted 7 lines

Other packages using meson build system might fail as well.
This patch changes default host system locale from C to en_US.UTF-8 to fix this issue. It also introduces BR2_NEEDS_HOST_EN_US_UTF8_LOCALE hidden flag that checks if this locale is available on the host system. If locale is not available error message is show and build process is stopped.

Yeah, I like this change.

Yet, people that already use an UTF-8 locale (e.g fr_FR.UTF-8, or en_GB.UTF-8 or whatever_else.UTF-8) will be very disapointed that their build now fail when they were previously successful... :-/

But see below...

Config.in config BR2_HOSTARCH_NEEDS_IA32_COMPILER · line 100

Quoted 12 lines

Config.in +4 -0

index 62d67ce957..bd14226edc 100644
@@ -94,6 +94,11 @@ config BR2_HOSTARCH_NEEDS_IA32_COMPILER
 config BR2_NEEDS_HOST_UTF8_LOCALE
 	bool
 
+# Hidden boolean selected by packages that explicitly need en_US.UTF-8 locale 
+# to be available on host system.
+config BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
+	bool

So, this option is supposed to be selectable by packages, but it is only ever selected by the reproducible option, and never used to directly set the locale.

What I would have expected, too, would be that systemd also selects BR2_NEEDS_HOST_EN_US_UTF8_LOCALE, and the build commands for systemd are changed such as (simplified for brevity):

    SYSTEMD_ENV = $(TARGET_MAKE_ENV) LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8

    define SYSTEMD_CONFIGURE_CMDS
        $(SYSTEMD_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build
    endef

    define SYSTEMD_BUILD_CMDS
        $(SYSTEMD_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build
    endef

... and so on...

Config.in config BR2_REPRODUCIBLE · line 723

Quoted 8 lines

Config.in +1 -0

 source "arch/Config.in"
 
 menu "Build options"

@@ -715,6 +720,7 @@ config BR2_REPRODUCIBLE
 	bool "Make the build reproducible (experimental)"
 	# SOURCE_DATE_EPOCH support in toolchain-wrapper requires GCC 4.4
 	depends on BR2_TOOLCHAIN_GCC_AT_LEAST_4_4
+	select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE

If it were me, this option would ultimately disappear, and we would only do reproducible builds. There is no point in hell to generate non-reproducible builds... :-/

(Yeah, I know, not everyone is at ease reading tools' output messages not in their language.)

Regards,
Yann E. MORIN.

Quoted 45 lines

Config.in +0 -0

 	help
 	  This option will remove all sources of non-reproducibility
 	  from the build process. For a given Buildroot configuration,

Makefile +2 -2

index 7d8ab51a8c..5991c6a221 100644
@@ -250,8 +250,8 @@ endif
 # timezone and locale may affect build output
 ifeq ($(BR2_REPRODUCIBLE),y)
 export TZ = UTC
-export LANG = C
-export LC_ALL = C
+export LANG = en_US.UTF-8
+export LC_ALL = en_US.UTF-8
 export GZIP = -n
 BR2_VERSION_GIT_EPOCH = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
 export SOURCE_DATE_EPOCH ?= $(if $(wildcard $(TOPDIR)/.git),$(BR2_VERSION_GIT_EPOCH),$(BR2_VERSION_EPOCH))

support/dependencies/dependencies.sh +13 -0

index a195c62c8c..4a5b55e9da 100755
@@ -207,6 +207,19 @@ if grep ^BR2_NEEDS_HOST_UTF8_LOCALE=y $BR2_CONFIG > /dev/null; then
 	fi
 fi
 
+if grep ^BR2_NEEDS_HOST_EN_US_UTF8_LOCALE=y $BR2_CONFIG > /dev/null; then
+	if ! which locale > /dev/null ; then
+		echo
+		echo "You need locale support on your build machine"
+		exit 1 ;
+	fi
+	if ! LC_ALL=en_US.UTF-8 locale -c charmap | grep '^UTF-8$' > /dev/null; then
+		echo
+		echo "You need en_US.UTF-8 locale suppport on the host system"
+		exit 1 ;
+	fi
+fi
+
 if grep -q ^BR2_NEEDS_HOST_JAVA=y $BR2_CONFIG ; then
 	check_prog_host "java"
 	JAVA_GCJ=$(java -version 2>&1 | grep gcj)

Signature · 1 line

2.15.1

Signature · 6 lines

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

Thomas Petazzoni

2018-02-06 16:32 UTC #5 lore raw

PATCH v2 1/1 Change host locale from C to en_US.UTF-8 for reproducable builds

In reply to Julius Kriukas

Hello,

Quoted 29 lines · Julius Kriukas

When BR2_REPRODUCIBLE is enabled building systemd fails with error:
[1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command.
FAILED: catalog/systemd.bg.catalog
/buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat
Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale.
You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents.
Please switch to a UTF-8 locale for your platform.
...
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128)
package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed
make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1
Other packages using meson build system might fail as well.
This patch changes default host system locale from C to en_US.UTF-8 to fix this issue. It also introduces BR2_NEEDS_HOST_EN_US_UTF8_LOCALE hidden flag that checks if this locale is available on the host system. If locale is not available error message is show and build process is stopped.
---
Changes v1 -> v2:
- Added BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that is selected by BR2_REPRODUCIBLE.
- Extended support/dependencies/dependencies.sh to check for en_US.UTF-8 locale availability if BR2_NEEDS_HOST_EN_US_UTF8_LOCALE is set.

Quoted 1 line

Signed-off-by Julius Kriukas <[email protected]>

Your SoB line should be before the "---" that separates the commit log and the changelog.

However, I have a bigger issue with the patch: you're fixing the issue for BR2_REPRODUCIBLE, but the problem actually appears also with a non-BR2_REPRODUCIBLE build. Look at:

http://autobuild.buildroot.net/results/257/25757c41d1a83edbe2a7c47b63ce2e43eb6141dc/build-end.log

This configuration does *not* have BR2_REPRODUCIBLE enabled. The system on which it runs has several UTF-8 locales enabled:

$ locale -a
C
C.UTF-8
POSIX
en_US.utf8
fr_FR.utf8

But the POSIX locale is used:

$ locale
LANG=
LANGUAGE=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=

And Meson fails in this situation. I believe it should be fixed in Meson. We should not have to set a specific locale for the thing to build. The need for the UTF-8 locale to be present is more or less OK, but the need to have such a locale in *use* is annoying.

So perhaps your fix is needed because Meson will in the end really need at least one UTF-8 locale available, but it is not complete because it doesn't fix the non-BR2_REPRODUCIBLE situation.

Best regards,

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
http://bootlin.com

Julius Kriukas

author

2018-02-12 21:34 UTC #6 lore raw

PATCH v3 1/1 systemd: switch to en_US.UTF-8 locale when building

In reply to Julius Kriukas

When BR2_REPRODUCIBLE is enabled or host uses non UTF-8 capable locale building systemd fails with an error:

This patch changes default host system locale from C to en_US.UTF-8 when building systemd package to fix this issue. It also introduces BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that checks if this locale is available on the host system. If locale is not available error message is show and build process is stopped.

Signed-off-by Julius Kriukas <[email protected]>

Changes v2 -> v3:
- Always select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE by systemd package.
- Do not select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE by BR2_REPRODUCIBLE.
- Keep global locale set to C if BR2_REPRODUCIBLE=y.
- Use 'grep -q' instead of '> /dev/null' in dependencies.sh script.

---
 Config.in                            |  5 +++++
 package/systemd/Config.in            |  1 +
 package/systemd/systemd.mk           | 10 ++++++----
 support/dependencies/dependencies.sh | 13 +++++++++++++
 4 files changed, 25 insertions(+), 4 deletions(-)

Config.in +5 -0

index 0002df5176..8d471dfb2e 100644
@@ -94,6 +94,11 @@ config BR2_HOSTARCH_NEEDS_IA32_COMPILER
 config BR2_NEEDS_HOST_UTF8_LOCALE
 	bool
 
+# Hidden boolean selected by packages that explicitly need en_US.UTF-8 locale
+# to be available on host system.
+config BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
+	bool
+
 source "arch/Config.in"
 
 menu "Build options"

package/systemd/Config.in +1 -0

index 661f40dee4..742b3f1f1d 100644
@@ -27,6 +27,7 @@ menuconfig BR2_PACKAGE_SYSTEMD
 	select BR2_PACKAGE_BUSYBOX_SHOW_OTHERS # kmod-tools
 	select BR2_PACKAGE_KMOD_TOOLS
 	select BR2_TARGET_TZ_INFO
+        select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
 	help
 	  systemd is a system and service manager for Linux,
 	  compatible with SysV and LSB init scripts. systemd provides

package/systemd/systemd.mk +6 -4

index 1c31ebb34c..58798f32df 100644
@@ -385,23 +385,25 @@ endef
 
 SYSTEMD_NINJA_OPTS = $(if $(VERBOSE),-v) -j$(PARALLEL_JOBS)
 
+SYSTEMD_ENV = $(TARGET_MAKE_ENV) LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8
+
 define SYSTEMD_CONFIGURE_CMDS
 	rm -rf $(@D)/build
 	mkdir -p $(@D)/build
-	$(TARGET_MAKE_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build
+	$(SYSTEMD_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build
 endef
 
 define SYSTEMD_BUILD_CMDS
-	$(TARGET_MAKE_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build
+	$(SYSTEMD_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build
 endef
 
 define SYSTEMD_INSTALL_TARGET_CMDS
-	$(TARGET_MAKE_ENV) DESTDIR=$(TARGET_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
+	$(SYSTEMD_ENV) DESTDIR=$(TARGET_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
 		-C $(@D)/build install
 endef
 
 define SYSTEMD_INSTALL_STAGING_CMDS
-	$(TARGET_MAKE_ENV) DESTDIR=$(STAGING_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
+	$(SYSTEMD_ENV) DESTDIR=$(STAGING_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
 		-C $(@D)/build install
 endef

support/dependencies/dependencies.sh +13 -0

index 1804e85508..881f07008c 100755
@@ -207,6 +207,19 @@ if grep ^BR2_NEEDS_HOST_UTF8_LOCALE=y $BR2_CONFIG > /dev/null; then
 	fi
 fi
 
+if grep -q ^BR2_NEEDS_HOST_EN_US_UTF8_LOCALE=y $BR2_CONFIG; then
+	if ! which locale > /dev/null ; then
+		echo
+		echo "You need locale support on your build machine"
+		exit 1 ;
+	fi
+	if ! LC_ALL=en_US.UTF-8 locale -c charmap | grep -q '^UTF-8$'; then
+		echo
+		echo "You need en_US.UTF-8 locale suppport on the host system"
+		exit 1 ;
+	fi
+fi
+
 if grep -q ^BR2_NEEDS_HOST_JAVA=y $BR2_CONFIG ; then
 	check_prog_host "java"
 	JAVA_GCJ=$(java -version 2>&1 | grep gcj)

Signature · 1 line

2.15.1

John Keeping

2018-02-17 18:06 UTC #7 lore raw

PATCH v3 1/1 systemd: switch to en_US.UTF-8 locale when building

Quoted 14 lines · Julius Kriukas

When BR2_REPRODUCIBLE is enabled or host uses non UTF-8 capable locale building systemd fails with an error:
[1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command.
FAILED: catalog/systemd.bg.catalog
/buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat
Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale.
You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents.
Please switch to a UTF-8 locale for your platform.
...
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128)
package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed
make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1

Quoted 7 lines

This patch changes default host system locale from C to en_US.UTF-8 when building systemd package to fix this issue. It also introduces BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that checks if this locale is available on the host system. If locale is not available error message is show and build process is stopped.
Signed-off-by Julius Kriukas <[email protected]>

Tested-by John Keeping <[email protected]>

And, FWIW, with the indentation in Config.in fixed,

Reviewed-by John Keeping <[email protected]>

Quoted 41 lines

---
Changes v1 -> v2:
- Added BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that is selected by BR2_REPRODUCIBLE.
- Extended support/dependencies/dependencies.sh to check for en_US.UTF-8 locale availability if BR2_NEEDS_HOST_EN_US_UTF8_LOCALE is set.
Changes v2 -> v3:
- Always select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE by systemd package.
- Do not select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE by BR2_REPRODUCIBLE.
- Keep global locale set to C if BR2_REPRODUCIBLE=y.
- Use 'grep -q' instead of '> /dev/null' in dependencies.sh script.
---
 Config.in                            |  5 +++++
 package/systemd/Config.in            |  1 +
 package/systemd/systemd.mk           | 10 ++++++----
 support/dependencies/dependencies.sh | 13 +++++++++++++
 4 files changed, 25 insertions(+), 4 deletions(-)
Config.in +5 -0
index 0002df5176..8d471dfb2e 100644
@@ -94,6 +94,11 @@ config BR2_HOSTARCH_NEEDS_IA32_COMPILER
 config BR2_NEEDS_HOST_UTF8_LOCALE
 	bool
 
+# Hidden boolean selected by packages that explicitly need en_US.UTF-8 locale
+# to be available on host system.
+config BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
+	bool
+
 source "arch/Config.in"
 
 menu "Build options"
package/systemd/Config.in +0 -0
index 661f40dee4..742b3f1f1d 100644

package/systemd/Config.in menuconfig BR2_PACKAGE_SYSTEMD · line 30

Quoted 4 lines

package/systemd/Config.in +1 -0

@@ -27,6 +27,7 @@ menuconfig BR2_PACKAGE_SYSTEMD
 	select BR2_PACKAGE_BUSYBOX_SHOW_OTHERS # kmod-tools
 	select BR2_PACKAGE_KMOD_TOOLS
 	select BR2_TARGET_TZ_INFO
+        select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE

This should use TAB for indentation.

Quoted 61 lines

package/systemd/Config.in +0 -0

 	help
 	  systemd is a system and service manager for Linux,
 	  compatible with SysV and LSB init scripts. systemd provides

package/systemd/systemd.mk +6 -4

index 1c31ebb34c..58798f32df 100644
@@ -385,23 +385,25 @@ endef
 
 SYSTEMD_NINJA_OPTS = $(if $(VERBOSE),-v) -j$(PARALLEL_JOBS)
 
+SYSTEMD_ENV = $(TARGET_MAKE_ENV) LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8
+
 define SYSTEMD_CONFIGURE_CMDS
 	rm -rf $(@D)/build
 	mkdir -p $(@D)/build
-	$(TARGET_MAKE_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build
+	$(SYSTEMD_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build
 endef
 
 define SYSTEMD_BUILD_CMDS
-	$(TARGET_MAKE_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build
+	$(SYSTEMD_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build
 endef
 
 define SYSTEMD_INSTALL_TARGET_CMDS
-	$(TARGET_MAKE_ENV) DESTDIR=$(TARGET_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
+	$(SYSTEMD_ENV) DESTDIR=$(TARGET_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
 		-C $(@D)/build install
 endef
 
 define SYSTEMD_INSTALL_STAGING_CMDS
-	$(TARGET_MAKE_ENV) DESTDIR=$(STAGING_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
+	$(SYSTEMD_ENV) DESTDIR=$(STAGING_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
 		-C $(@D)/build install
 endef

support/dependencies/dependencies.sh +13 -0

index 1804e85508..881f07008c 100755
@@ -207,6 +207,19 @@ if grep ^BR2_NEEDS_HOST_UTF8_LOCALE=y $BR2_CONFIG > /dev/null; then
 	fi
 fi
 
+if grep -q ^BR2_NEEDS_HOST_EN_US_UTF8_LOCALE=y $BR2_CONFIG; then
+	if ! which locale > /dev/null ; then
+		echo
+		echo "You need locale support on your build machine"
+		exit 1 ;
+	fi
+	if ! LC_ALL=en_US.UTF-8 locale -c charmap | grep -q '^UTF-8$'; then
+		echo
+		echo "You need en_US.UTF-8 locale suppport on the host system"
+		exit 1 ;
+	fi
+fi
+
 if grep -q ^BR2_NEEDS_HOST_JAVA=y $BR2_CONFIG ; then
 	check_prog_host "java"
 	JAVA_GCJ=$(java -version 2>&1 | grep gcj)

Julius Kriukas

author

2018-02-22 23:02 UTC #8 lore raw

PATCH v3 1/1 systemd: switch to en_US.UTF-8 locale when building

Quoted 123 lines · John Keeping

Quoted 14 lines · Julius Kriukas
When BR2_REPRODUCIBLE is enabled or host uses non UTF-8 capable locale building systemd fails with an error:
[1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command.
FAILED: catalog/systemd.bg.catalog
/buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat
Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale.
You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents.
Please switch to a UTF-8 locale for your platform.
...
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128)
package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed
make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1
Quoted 7 lines
This patch changes default host system locale from C to en_US.UTF-8 when building systemd package to fix this issue. It also introduces BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that checks if this locale is available on the host system. If locale is not available error message is show and build process is stopped.
Signed-off-by Julius Kriukas <[email protected]>
Tested-by John Keeping <[email protected]>
And, FWIW, with the indentation in Config.in fixed,
Reviewed-by John Keeping <[email protected]>
Quoted 41 lines
---
Changes v1 -> v2:
- Added BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that is selected by BR2_REPRODUCIBLE.
- Extended support/dependencies/dependencies.sh to check for en_US.UTF-8 locale availability if BR2_NEEDS_HOST_EN_US_UTF8_LOCALE is set.
Changes v2 -> v3:
- Always select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE by systemd package.
- Do not select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE by BR2_REPRODUCIBLE.
- Keep global locale set to C if BR2_REPRODUCIBLE=y.
- Use 'grep -q' instead of '> /dev/null' in dependencies.sh script.
---
  Config.in                            |  5 +++++
  package/systemd/Config.in            |  1 +
  package/systemd/systemd.mk           | 10 ++++++----
  support/dependencies/dependencies.sh | 13 +++++++++++++
  4 files changed, 25 insertions(+), 4 deletions(-)
Config.in +5 -0
index 0002df5176..8d471dfb2e 100644
@@ -94,6 +94,11 @@ config BR2_HOSTARCH_NEEDS_IA32_COMPILER
  config BR2_NEEDS_HOST_UTF8_LOCALE
  	bool
 
+# Hidden boolean selected by packages that explicitly need en_US.UTF-8 locale
+# to be available on host system.
+config BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
+	bool
+
  source "arch/Config.in"
 
  menu "Build options"
package/systemd/Config.in +0 -0
index 661f40dee4..742b3f1f1d 100644
package/systemd/Config.in menuconfig BR2_PACKAGE_SYSTEMD · line 30
Quoted 4 lines
package/systemd/Config.in +1 -0
@@ -27,6 +27,7 @@ menuconfig BR2_PACKAGE_SYSTEMD
  	select BR2_PACKAGE_BUSYBOX_SHOW_OTHERS # kmod-tools
  	select BR2_PACKAGE_KMOD_TOOLS
  	select BR2_TARGET_TZ_INFO
+        select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
This should use TAB for indentation.
Quoted 45 lines
package/systemd/Config.in +0 -0
  	help
  	  systemd is a system and service manager for Linux,
  	  compatible with SysV and LSB init scripts. systemd provides
package/systemd/systemd.mk +6 -4
index 1c31ebb34c..58798f32df 100644
@@ -385,23 +385,25 @@ endef
 
  SYSTEMD_NINJA_OPTS = $(if $(VERBOSE),-v) -j$(PARALLEL_JOBS)
 
+SYSTEMD_ENV = $(TARGET_MAKE_ENV) LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8
+
  define SYSTEMD_CONFIGURE_CMDS
  	rm -rf $(@D)/build
  	mkdir -p $(@D)/build
-	$(TARGET_MAKE_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build
+	$(SYSTEMD_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build
  endef
 
  define SYSTEMD_BUILD_CMDS
-	$(TARGET_MAKE_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build
+	$(SYSTEMD_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build
  endef
 
  define SYSTEMD_INSTALL_TARGET_CMDS
-	$(TARGET_MAKE_ENV) DESTDIR=$(TARGET_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
+	$(SYSTEMD_ENV) DESTDIR=$(TARGET_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
  		-C $(@D)/build install
  endef
 
  define SYSTEMD_INSTALL_STAGING_CMDS
-	$(TARGET_MAKE_ENV) DESTDIR=$(STAGING_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
+	$(SYSTEMD_ENV) DESTDIR=$(STAGING_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
  		-C $(@D)/build install
  endef
 
support/dependencies/dependencies.sh +0 -0
index 1804e85508..881f07008c 100755
@@ -207,6 +207,19 @@ if grep ^BR2_NEEDS_HOST_UTF8_LOCALE=y $BR2_CONFIG > /dev/null; then
  	fi
  fi
 

Quoted 16 lines

Quoted 6 lines

support/dependencies/dependencies.sh +6 -0

+if grep -q ^BR2_NEEDS_HOST_EN_US_UTF8_LOCALE=y $BR2_CONFIG; then
+	if ! which locale > /dev/null ; then
+		echo
+		echo "You need locale support on your build machine"
+		exit 1 ;
+	fi

support/dependencies/dependencies.sh if grep ^BR2_NEEDS_HOST_UTF8_LOCALE=y $BR2_CONFIG > /dev/null; then · line 225

Quoted 10 lines

support/dependencies/dependencies.sh +7 -0

+	if ! LC_ALL=en_US.UTF-8 locale -c charmap | grep -q '^UTF-8$'; then
+		echo
+		echo "You need en_US.UTF-8 locale suppport on the host system"
+		exit 1 ;
+	fi
+fi
+
  if grep -q ^BR2_NEEDS_HOST_JAVA=y $BR2_CONFIG ; then
  	check_prog_host "java"
  	JAVA_GCJ=$(java -version 2>&1 | grep gcj)

Hi John. Thanks for spotting the indentation issue and review. I am submitting v4 of this patch with the issue fixed.

Signature · 1 line

Julius Kriukas

Julius Kriukas

author

2018-02-22 23:00 UTC #9 lore raw

PATCH v4 1/1 systemd: switch to en_US.UTF-8 locale when building

In reply to Julius Kriukas

When BR2_REPRODUCIBLE is enabled or host uses non UTF-8 capable locale building systemd fails with an error:

Signed-off-by Julius Kriukas <[email protected]>

Changes v3 -> v4:
- Fix indentation in package/systemd/Config.in, spaces -> tab.

---
 Config.in                            |  5 +++++
 package/systemd/Config.in            |  1 +
 package/systemd/systemd.mk           | 10 ++++++----
 support/dependencies/dependencies.sh | 13 +++++++++++++
 4 files changed, 25 insertions(+), 4 deletions(-)

Config.in +5 -0

index 0002df5176..8d471dfb2e 100644
@@ -94,6 +94,11 @@ config BR2_HOSTARCH_NEEDS_IA32_COMPILER
 config BR2_NEEDS_HOST_UTF8_LOCALE
 	bool
 
+# Hidden boolean selected by packages that explicitly need en_US.UTF-8 locale
+# to be available on host system.
+config BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
+	bool
+
 source "arch/Config.in"
 
 menu "Build options"

package/systemd/Config.in +1 -0

index 661f40dee4..e83f75c733 100644
@@ -27,6 +27,7 @@ menuconfig BR2_PACKAGE_SYSTEMD
 	select BR2_PACKAGE_BUSYBOX_SHOW_OTHERS # kmod-tools
 	select BR2_PACKAGE_KMOD_TOOLS
 	select BR2_TARGET_TZ_INFO
+	select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
 	help
 	  systemd is a system and service manager for Linux,
 	  compatible with SysV and LSB init scripts. systemd provides

package/systemd/systemd.mk +6 -4

index 126727d602..9a6a9cfcbc 100644
@@ -385,23 +385,25 @@ endef
 
 SYSTEMD_NINJA_OPTS = $(if $(VERBOSE),-v) -j$(PARALLEL_JOBS)
 
+SYSTEMD_ENV = $(TARGET_MAKE_ENV) LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8
+
 define SYSTEMD_CONFIGURE_CMDS
 	rm -rf $(@D)/build
 	mkdir -p $(@D)/build
-	$(TARGET_MAKE_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build
+	$(SYSTEMD_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build
 endef
 
 define SYSTEMD_BUILD_CMDS
-	$(TARGET_MAKE_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build
+	$(SYSTEMD_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build
 endef
 
 define SYSTEMD_INSTALL_TARGET_CMDS
-	$(TARGET_MAKE_ENV) DESTDIR=$(TARGET_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
+	$(SYSTEMD_ENV) DESTDIR=$(TARGET_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
 		-C $(@D)/build install
 endef
 
 define SYSTEMD_INSTALL_STAGING_CMDS
-	$(TARGET_MAKE_ENV) DESTDIR=$(STAGING_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
+	$(SYSTEMD_ENV) DESTDIR=$(STAGING_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
 		-C $(@D)/build install
 endef

support/dependencies/dependencies.sh +13 -0

index 1804e85508..881f07008c 100755
@@ -207,6 +207,19 @@ if grep ^BR2_NEEDS_HOST_UTF8_LOCALE=y $BR2_CONFIG > /dev/null; then
 	fi
 fi
 
+if grep -q ^BR2_NEEDS_HOST_EN_US_UTF8_LOCALE=y $BR2_CONFIG; then
+	if ! which locale > /dev/null ; then
+		echo
+		echo "You need locale support on your build machine"
+		exit 1 ;
+	fi
+	if ! LC_ALL=en_US.UTF-8 locale -c charmap | grep -q '^UTF-8$'; then
+		echo
+		echo "You need en_US.UTF-8 locale suppport on the host system"
+		exit 1 ;
+	fi
+fi
+
 if grep -q ^BR2_NEEDS_HOST_JAVA=y $BR2_CONFIG ; then
 	check_prog_host "java"
 	JAVA_GCJ=$(java -version 2>&1 | grep gcj)

Signature · 1 line

2.16.1

Yann E. MORIN

2018-02-28 21:31 UTC #10 lore raw

PATCH v4 1/1 systemd: switch to en_US.UTF-8 locale when building

Julius, All,

On 2018-02-23 01:00 +0200, Julius Kriukas spake thusly:

Quoted 2 lines

When BR2_REPRODUCIBLE is enabled or host uses non UTF-8 capable locale building systemd fails with an error:

As far as I can see, this patch is now no longer about REPRODCIBLE, but only about systemd; that's good. But then, the commit log should drop the part about REPRODUCIBLE.

Quoted 11 lines

[1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command.
FAILED: catalog/systemd.bg.catalog
/buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat
Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale.
You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents.
Please switch to a UTF-8 locale for your platform.
...
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128)
package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed
make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1

Quoted 5 lines

This patch changes default host system locale from C to en_US.UTF-8 when building systemd package to fix this issue. It also introduces BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that checks if this locale is available on the host system. If locale is not available error message is show and build process is stopped.

OK, so I do not have en_USUTF-8 on my machine. I have en_GB.UTF-8 (which I am using). So I'm left out in the cold with this patch.

However, C.UTF-8 is virtually available everywhere. And I was able to build systemd with C.UTF-8 without any problem.

So, I would prefer, if we have to force anything, that we force the use of C.UTF-8.

But then that also means that we override the users' environments, users that may prefer to see localised messages.

So, as a stop-gap measure, I'm OK with this patch as long as it is switched to using C.UTF-8. Also, rename the variable, something like:

BR2_NEEDS_HOST_UTF8_LOCALE_CURRENT

Note: this is different from the existing BR2_NEEDS_HOST_UTF8_LOCALE: it onlhy requires that a UTF-8 locale be present on the system, while we now require that it be the current locale.

Regards,
Yann E. MORIN.

Quoted 114 lines

Signed-off-by Julius Kriukas <[email protected]>

Changes v3 -> v4:
- Fix indentation in package/systemd/Config.in, spaces -> tab.

---
 Config.in                            |  5 +++++
 package/systemd/Config.in            |  1 +
 package/systemd/systemd.mk           | 10 ++++++----
 support/dependencies/dependencies.sh | 13 +++++++++++++
 4 files changed, 25 insertions(+), 4 deletions(-)

Config.in +5 -0

index 0002df5176..8d471dfb2e 100644
@@ -94,6 +94,11 @@ config BR2_HOSTARCH_NEEDS_IA32_COMPILER
 config BR2_NEEDS_HOST_UTF8_LOCALE
 	bool
 
+# Hidden boolean selected by packages that explicitly need en_US.UTF-8 locale
+# to be available on host system.
+config BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
+	bool
+
 source "arch/Config.in"
 
 menu "Build options"

package/systemd/Config.in +1 -0

index 661f40dee4..e83f75c733 100644
@@ -27,6 +27,7 @@ menuconfig BR2_PACKAGE_SYSTEMD
 	select BR2_PACKAGE_BUSYBOX_SHOW_OTHERS # kmod-tools
 	select BR2_PACKAGE_KMOD_TOOLS
 	select BR2_TARGET_TZ_INFO
+	select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
 	help
 	  systemd is a system and service manager for Linux,
 	  compatible with SysV and LSB init scripts. systemd provides

package/systemd/systemd.mk +6 -4

index 126727d602..9a6a9cfcbc 100644
@@ -385,23 +385,25 @@ endef
 
 SYSTEMD_NINJA_OPTS = $(if $(VERBOSE),-v) -j$(PARALLEL_JOBS)
 
+SYSTEMD_ENV = $(TARGET_MAKE_ENV) LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8
+
 define SYSTEMD_CONFIGURE_CMDS
 	rm -rf $(@D)/build
 	mkdir -p $(@D)/build
-	$(TARGET_MAKE_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build
+	$(SYSTEMD_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build
 endef
 
 define SYSTEMD_BUILD_CMDS
-	$(TARGET_MAKE_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build
+	$(SYSTEMD_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build
 endef
 
 define SYSTEMD_INSTALL_TARGET_CMDS
-	$(TARGET_MAKE_ENV) DESTDIR=$(TARGET_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
+	$(SYSTEMD_ENV) DESTDIR=$(TARGET_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
 		-C $(@D)/build install
 endef
 
 define SYSTEMD_INSTALL_STAGING_CMDS
-	$(TARGET_MAKE_ENV) DESTDIR=$(STAGING_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
+	$(SYSTEMD_ENV) DESTDIR=$(STAGING_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
 		-C $(@D)/build install
 endef

support/dependencies/dependencies.sh +13 -0

index 1804e85508..881f07008c 100755
@@ -207,6 +207,19 @@ if grep ^BR2_NEEDS_HOST_UTF8_LOCALE=y $BR2_CONFIG > /dev/null; then
 	fi
 fi
 
+if grep -q ^BR2_NEEDS_HOST_EN_US_UTF8_LOCALE=y $BR2_CONFIG; then
+	if ! which locale > /dev/null ; then
+		echo
+		echo "You need locale support on your build machine"
+		exit 1 ;
+	fi
+	if ! LC_ALL=en_US.UTF-8 locale -c charmap | grep -q '^UTF-8$'; then
+		echo
+		echo "You need en_US.UTF-8 locale suppport on the host system"
+		exit 1 ;
+	fi
+fi
+
 if grep -q ^BR2_NEEDS_HOST_JAVA=y $BR2_CONFIG ; then
 	check_prog_host "java"
 	JAVA_GCJ=$(java -version 2>&1 | grep gcj)

Signature · 1 line

2.16.1

Signature · 6 lines

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

Thomas Petazzoni

2018-02-28 22:13 UTC #11 lore raw

PATCH v4 1/1 systemd: switch to en_US.UTF-8 locale when building

Hello,

Quoted 12 lines · Yann E. MORIN

OK, so I do not have en_USUTF-8 on my machine. I have en_GB.UTF-8 (which I am using). So I'm left out in the cold with this patch.
However, C.UTF-8 is virtually available everywhere. And I was able to build systemd with C.UTF-8 without any problem.
So, I would prefer, if we have to force anything, that we force the use of C.UTF-8.
But then that also means that we override the users' environments, users that may prefer to see localised messages.

Quoted 8 lines

So, as a stop-gap measure, I'm OK with this patch as long as it is switched to using C.UTF-8. Also, rename the variable, something like:
BR2_NEEDS_HOST_UTF8_LOCALE_CURRENT
Note: this is different from the existing BR2_NEEDS_HOST_UTF8_LOCALE: it onlhy requires that a UTF-8 locale be present on the system, while we now require that it be the current locale.

Now, we don't require it to be the current locale:

Quoted 1 line

Quoted 1 line
+SYSTEMD_ENV = $(TARGET_MAKE_ENV) LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8

The systemd package ensures to pass LANG/LC_ALL pointing to an UTF-8 locale.

So even if your current locale is something else, as long as en_US.UTF-8 is *available*, things are going to work.

That being said, I also dislike this solution, and think the sytemd build system should be fixed instead.

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
http://bootlin.com

Yann E. MORIN

2018-03-01 18:06 UTC #12 lore raw

PATCH v4 1/1 systemd: switch to en_US.UTF-8 locale when building

Thomas, All,

On 2018-02-28 23:13 +0100, Thomas Petazzoni spake thusly:

Quoted 1 line · Yann E. MORIN

[--SNIP--]

Quoted 7 lines

Quoted 3 lines
Note: this is different from the existing BR2_NEEDS_HOST_UTF8_LOCALE: it onlhy requires that a UTF-8 locale be present on the system, while we now require that it be the current locale.
Now, we don't require it to be the current locale:
Quoted 1 line
Quoted 1 line
+SYSTEMD_ENV = $(TARGET_MAKE_ENV) LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8
The systemd package ensures to pass LANG/LC_ALL pointing to an UTF-8 locale.

Ah, yes, you're right, sorry...

I was confused now, because (without this patch) I had to change my locale to be able to build systemd. My bad, sorry...

Quoted 2 lines

That being said, I also dislike this solution, and think the sytemd build system should be fixed instead.

Yup, that's really sad... But properly fixing it is a totally different endeavour... :-/

Regards,
Yann E. MORIN.

Signature · 6 lines

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

Peter Korsgaard

2018-03-01 18:50 UTC #13 lore raw

PATCH v4 1/1 systemd: switch to en_US.UTF-8 locale when building

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:

Hi,

Quoted 4 lines

Ah, yes, you're right, sorry...
I was confused now, because (without this patch) I had to change my locale to be able to build systemd. My bad, sorry...

Wait, are you saying that it doesn't build with en_GB.UTF-8 locale? That's odd.

Signature · 1 line

Bye, Peter Korsgaard

Yann E. MORIN

2018-03-01 19:12 UTC #14 lore raw

PATCH v4 1/1 systemd: switch to en_US.UTF-8 locale when building

Peter,All,

On 2018-03-01 19:50 +0100, Peter Korsgaard spake thusly:

Quoted 8 lines

Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Ah, yes, you're right, sorry...
I was confused now, because (without this patch) I had to change my locale to be able to build systemd. My bad, sorry...
Wait, are you saying that it doesn't build with en_GB.UTF-8 locale? That's odd.

No, no, don't worry!

In my locale, en_GB.UTF-8, systemd-v237 *does* build fine.

But on my autobuilder, which only has C, POSIX, and C.UTF-8, the default being POSIX, I had to explicitly set it to C.UTF-8 to test without the patch.

So, don't worry, all is well when the current locale is any of *.UTF-8.

Regards,
Yann E. MORIN.

Signature · 6 lines

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

Peter Korsgaard

2018-03-01 20:04 UTC #15 lore raw

PATCH v4 1/1 systemd: switch to en_US.UTF-8 locale when building

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:

Hi,

Quoted 5 lines

Quoted 2 lines
Wait, are you saying that it doesn't build with en_GB.UTF-8 locale? That's odd.
No, no, don't worry!

Quoted 7 lines

In my locale, en_GB.UTF-8, systemd-v237 *does* build fine.
But on my autobuilder, which only has C, POSIX, and C.UTF-8, the default being POSIX, I had to explicitly set it to C.UTF-8 to test without the patch.
So, don't worry, all is well when the current locale is any of *.UTF-8.

*Pheeuw* ;)

Signature · 1 line

Bye, Peter Korsgaard

Thomas Petazzoni

2018-03-02 21:45 UTC #16 lore raw

PATCH v4 1/1 systemd: switch to en_US.UTF-8 locale when building

In reply to Julius Kriukas

Hello,

Quoted 22 lines · Julius Kriukas

When BR2_REPRODUCIBLE is enabled or host uses non UTF-8 capable locale building systemd fails with an error:
[1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command.
FAILED: catalog/systemd.bg.catalog
/buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat
Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale.
You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents.
Please switch to a UTF-8 locale for your platform.
...
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128)
package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed
make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1
This patch changes default host system locale from C to en_US.UTF-8 when building systemd package to fix this issue. It also introduces BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that checks if this locale is available on the host system. If locale is not available error message is show and build process is stopped.
Signed-off-by Julius Kriukas <[email protected]>

Quoted 7 lines

---
Changes v1 -> v2:
- Added BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that is selected by BR2_REPRODUCIBLE.
- Extended support/dependencies/dependencies.sh to check for en_US.UTF-8 locale availability if BR2_NEEDS_HOST_EN_US_UTF8_LOCALE is set.

I've changed to use C.UTF-8 instead of en_US.UTF-8 and applied to master. Thanks!

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

A message that is not in the archive (sent off-list or not yet received): [email protected]

Yann E. MORIN

2018-03-04 09:01 UTC #18 lore raw

PATCH v4 1/1 systemd: switch to en_US.UTF-8 locale when building

Christian, All,

On 2018-03-03 19:22 -0500, Christian Stewart spake thusly:

Quoted 5 lines

Quoted 2 lines · Thomas Petazzoni
I've changed to use C.UTF-8 instead of en_US.UTF-8 and applied to master. Thanks!
This has broken the build on Gentoo, unfortunately.

Grr.. Gentoo... Grr...

The major distros (Debian and derivatives, Fedora...) all have C.UTF-8. Gentoo is again playing solo... Sigh... :-/

I'm onto a fix for that, now...

Quoted 2 lines

Is it possible to compile with an en_US.UTF-8 or other language-specific UTF-8 locale, such that the build again works on Gentoo?

Not everyone on Earth is American, by far, so we already ruled out using a hard-coded en_US.UTF-8 locale; for example, I don't have it on my system (although I do have, and do use, an english locale, en_GB.UTF-8).

I'm onto a fix now...

Regards,
Yann E. MORIN.

Signature · 6 lines

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

Arnout Vandecappelle

2018-02-23 09:06 UTC #19 lore raw

PATCH v3 1/1 systemd: switch to en_US.UTF-8 locale when building

In reply to Julius Kriukas

Hi Julius,

I wanted to take the time to think of better solutions, but since I don't find
the time, here is a very quick review.

Quoted 11 lines · Julius Kriukas

When BR2_REPRODUCIBLE is enabled or host uses non UTF-8 capable locale building systemd fails with an error:
[1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command.
FAILED: catalog/systemd.bg.catalog
/buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat
Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale.
You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents.
Please switch to a UTF-8 locale for your platform.
...
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128)

As someone mentioned (IIRC), it's really not OK that some Python script (is it
meson or ninja?) is relying on a locale being set to work correctly. So it would be much preferable to find out what the real problem is, and to have an upstreamable patch that fixes it.

That said, getting to the bottom of this is work, someone has to do it, and in
the mean time we've got a broken system, and we're about to release 2018.02. So I'm OK to accept this workaround for the time being (and as we all now, that can be a very long time :-).

I would, however, like to mention explicitly in the commit message that this is
just a workaround and that really one of the underlying scripts (either meson or ninja or something systemd-specific, TBD) is broken.

Quoted 8 lines

package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed
make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1
This patch changes default host system locale from C to en_US.UTF-8 when building systemd package to fix this issue. It also introduces BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that checks if this locale is available on the host system. If locale is not available error message is show and build process is stopped.

The part that introduces BR2_NEEDS_HOST_EN_US_UTF8_LOCALE and uses it for
BR2_REPRODUCIBLE should be a separate patch. Indeed, when we find a better solution for the systemd/meson issue, then we can revert this patch, but the BR2_REPRODUCIBLE bit should stay.

Regards,
Arnout

Quoted 2 lines

Signed-off-by Julius Kriukas <[email protected]>

Signature · 6 lines

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF