host-localedef: Compile against glibc

In glibc 2.27 the following change occured:
"Statically compiled applications attempting to load locales compiled for the GNU C Library version 2.27 will fail and fall back to the builtin C/POSIX locale."

This impacts us since upstream buildroot uses a localdef built against an older eglibc release [0].
Update the host-localedef package to build against glibc 2.27. Unfortunately we can't compile just the localedef utility, but even so this does not increase the build time too significantly. eg:

With glibc localedef:

real    33m11.278s
user    171m51.684s
sys     19m46.528s

With eglibc localedef:

real    31m12.715s
user    155m13.616s
sys     16m23.504s

[0] https://bugs.busybox.net/show_bug.cgi?id=11096

Signed-off-by Samuel Mendoza-Jonas <[email protected]>

---
This is a patch we're using in our own Buildroot branch to keep us going until a solution is found for the above bug report. While it does add a few minutes to our example build it avoids hacking glibc to only build localedef which is a solution some other users have gone for. Definitely an RFC as I'm sure some of the configuration in localedef.mk could be trimmed down but I took the safer bet of basing it off the existing glibc.mk.

 .../0001-Don-t-include-xlocale.h.patch        | 50 -----------
 package/localedef/localedef.hash              |  2 +-
 package/localedef/localedef.mk                | 87 +++++++++++++++++--
 3 files changed, 79 insertions(+), 60 deletions(-)
 delete mode 100644 package/localedef/0001-Don-t-include-xlocale.h.patch

package/localedef/0001-Don-t-include-xlocale.h.patch deleted +0 -50

deleted file mode 100644
index 9a6c2f8e90..0000000000
@@ -1,50 +0,0 @@
-From: Bernhard Walle <[email protected]>
-Date: Mon, 2 Oct 2017 16:55:23 +0200
-Subject: [PATCH] Don't include <xlocale.h>
-
-This header has been removed in glibc 2.26:
-
-https://abi-laboratory.pro/tracker/changelog/glibc/2.26/log.html
-
------------------------- >8 ------------------------
-* The nonstandard header <xlocale.h> has been removed.  Most programs should
-  use <locale.h> instead.  If you have a specific need for the definition of
-  locale_t with no other declarations, please contact
-  libc-alpha at sourceware.org and explain.
------------------------- 8< ------------------------
-
-Signed-off-by: Bernhard Walle <[email protected]>
-
-Upstream: https://git.pengutronix.de/cgit/ptxdist/tree/patches/localedef-eglibc-2.14.1-r17443-ptx1/0003-Don-t-include-xlocale.h.patch
-Signed-off-by: Peter Seiderer <[email protected]>
----
- eglibc/locale/langinfo.h | 2 +-
- eglibc/locale/locale.h   | 2 +-
- 2 files changed, 2 insertions(+), 2 deletions(-)
-
-diff --git a/eglibc/locale/langinfo.h b/eglibc/locale/langinfo.h
-index 0a5336507196..76707a4584c1 100644
---- a/eglibc/locale/langinfo.h
-+++ b/eglibc/locale/langinfo.h
-@@ -589,7 +589,7 @@ extern char *nl_langinfo (nl_item __item) __THROW;
-    more information.  */
-
- /* Get locale datatype definition.  */
--# include <xlocale.h>
-+# include <locale.h>
-
- /* Just like nl_langinfo but get the information from the locale object L.  */
- extern char *nl_langinfo_l (nl_item __item, __locale_t __l);
-diff --git a/eglibc/locale/locale.h b/eglibc/locale/locale.h
-index 2aa19e76acb2..18be711a2d40 100644
---- a/eglibc/locale/locale.h
-+++ b/eglibc/locale/locale.h
-@@ -143,7 +143,7 @@ __END_NAMESPACE_STD
-    This is a proof-of-concept implementation.  */
-
- /* Get locale datatype definition.  */
--# include <xlocale.h>
-+# include_next <locale.h>
-
- /* Return a reference to a data structure representing a set of locale
-    datasets.  Unlike for the CATEGORY parameter for `setlocale' the

package/localedef/localedef.hash +1 -1

index ee14fb16b8..6f2d6313b6 100644
@@ -1,2 +1,2 @@
 # Locally calculated
-sha256 9a60f7cdab6fb39adf23a12102f2d950d5f07f0cd7e51e85ec327e07440a79c6  localedef-eglibc-2.14.1-r17443-ptx1.tar.bz2
+sha256 33189b3f10c88730a1f686fac794bc01f31765f12ffd75bc5e8a0f2a690d217a  localedef-glibc-2.27-57-g6c99e37f6fb640a50a3113b2dbee5d5389843c1e.tar.gz

package/localedef/localedef.mk +78 -10

index 11d9ba3848..c93132cc3f 100644
@@ -4,19 +4,88 @@
 #
 ################################################################################
 
-LOCALEDEF_VERSION = 2.14.1-r17443-ptx1
-LOCALEDEF_SOURCE = localedef-eglibc-$(LOCALEDEF_VERSION).tar.bz2
-LOCALEDEF_SITE = http://www.pengutronix.de/software/ptxdist/temporary-src
+# Use the same VERSION and SITE as target glibc
+LOCALEDEF_VERSION = glibc-2.27-57-g6c99e37f6fb640a50a3113b2dbee5d5389843c1e
+LOCALEDEF_SITE = $(call github,bminor,glibc,$(GLIBC_VERSION))
 
-HOST_LOCALEDEF_CONF_OPTS += \
-       --prefix=/usr \
-       --with-glibc=./eglibc
+HOST_LOCALEDEF_LICENSE = GPL-2.0+ (programs), LGPL-2.1+, BSD-3-Clause, MIT (library)
+HOST_LOCALEDEF_LICENSE_FILES = COPYING COPYING.LIB LICENSES
 
-HOST_LOCALEDEF_CONF_ENV = CFLAGS="$(HOST_CFLAGS) -fgnu89-inline"
+# glibc is part of the toolchain so disable the toolchain dependency
+HOST_LOCALEDEF_ADD_TOOLCHAIN_DEPENDENCY = NO
 
-# The makefile does not implement an install target
+HOST_LOCALEDEF_SUBDIR = build
+
+# Thumb build is broken, build in ARM mode
+ifeq ($(BR2_ARM_INSTRUCTIONS_THUMB),y)
+HOST_LOCALEDEF_EXTRA_CFLAGS += -marm
+endif
+
+# MIPS64 defaults to n32 so pass the correct -mabi if
+# we are using a different ABI. OABI32 is also used
+# in MIPS so we pass -mabi=32 in this case as well
+# even though it's not strictly necessary.
+ifeq ($(BR2_MIPS_NABI64),y)
+HOST_LOCALEDEF_EXTRA_CFLAGS += -mabi=64
+else ifeq ($(BR2_MIPS_OABI32),y)
+HOST_LOCALEDEF_EXTRA_CFLAGS += -mabi=32
+endif
+
+ifeq ($(BR2_ENABLE_DEBUG),y)
+HOST_LOCALEDEF_EXTRA_CFLAGS += -g
+endif
+
+# The stubs.h header is not installed by install-headers, but is
+# needed for the gcc build. An empty stubs.h will work, as explained
+# in http://gcc.gnu.org/ml/gcc/2002-01/msg00900.html. The same trick
+# is used by Crosstool-NG.
+ifeq ($(BR2_TOOLCHAIN_BUILDROOT_HOST_LOCALEDEF),y)
+define HOST_LOCALEDEF_ADD_MISSING_STUB_H
+       mkdir -p $(STAGING_DIR)/usr/include/gnu
+       touch $(STAGING_DIR)/usr/include/gnu/stubs.h
+endef
+endif
+
+# Even though we use the autotools-package infrastructure, we have to
+# override the default configure commands for several reasons:
+#
+#  1. We have to build out-of-tree, but we can't use the same
+#     'symbolic link to configure' used with the gcc packages.
+#
+#  2. We have to execute the configure script with bash and not sh.
+#
+# Note that as mentionned in
+# http://patches.openembedded.org/patch/38849/, glibc must be
+# built with -O2, so we pass our own CFLAGS and CXXFLAGS below.
+define HOST_LOCALEDEF_CONFIGURE_CMDS
+       mkdir -p $(@D)/build
+       # Do the configuration
+       (cd $(@D)/build; \
+               $(HOST_CONFIGURE_OPTS) \
+               CFLAGS="-O2 $(HOST_LOCALEDEF_EXTRA_CFLAGS)" CPPFLAGS="" \
+               CXXFLAGS="-O2 $(HOST_LOCALEDEF_EXTRA_CFLAGS)" \
+               $(SHELL) $(@D)/configure \
+               ac_cv_path_BASH_SHELL=/bin/bash \
+               libc_cv_forced_unwind=yes \
+               libc_cv_ssp=no \
+               --target=$(GNU_HOST_NAME) \
+               --host=$(GNU_HOST_NAME) \
+               --build=$(GNU_HOST_NAME) \
+               --prefix=/usr \
+               $(if $(BR2_SOFT_FLOAT),--without-fp,--with-fp) \
+               $(if $(BR2_x86_64),--enable-lock-elision) \
+               --with-pkgversion="Buildroot" \
+               --without-cvs \
+               --disable-profile \
+               --without-gd \
+               --enable-obsolete-rpc)
+       $(HOST_LOCALEDEF_ADD_MISSING_STUB_H)
+endef
+
+$(eval $(autotools-package))
+# The makefile does not implement an install target for localedef
 define HOST_LOCALEDEF_INSTALL_CMDS
-       $(INSTALL) -D -m 0755 $(@D)/localedef $(HOST_DIR)/bin/localedef
+       $(INSTALL) -D -m 0755 $(@D)/build/locale/localedef $(HOST_DIR)/bin/localedef
 endef
 
 $(eval $(host-autotools-package))
--

2.18.0

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

In glibc 2.27 the following change occured:
"Statically compiled applications attempting to load locales compiled for the GNU C Library version 2.27 will fail and fall back to the builtin C/POSIX locale."

This impacts us since upstream buildroot uses a localdef built against an older eglibc release [0].
Update the host-localedef package to build against glibc 2.27. Unfortunately we can't compile just the localedef utility, but even so this does not increase the build time too significantly. eg:

With glibc localedef:

real    33m11.278s
user    171m51.684s
sys     19m46.528s

With eglibc localedef:

real    31m12.715s
user    155m13.616s
sys     16m23.504s

+HOST_LOCALEDEF_EXTRA_CFLAGS += -marm

package/localedef/localedef.mk +19 -0

+endef
+endif
+
+# Even though we use the autotools-package infrastructure, we have to
+# override the default configure commands for several reasons:
+#
+#  1. We have to build out-of-tree, but we can't use the same
+#     'symbolic link to configure' used with the gcc packages.
+#
+#  2. We have to execute the configure script with bash and not sh.
+#
+# Note that as mentionned in
+# http://patches.openembedded.org/patch/38849/, glibc must be
+# built with -O2, so we pass our own CFLAGS and CXXFLAGS below.
+define HOST_LOCALEDEF_CONFIGURE_CMDS
+	mkdir -p $(@D)/build
+	# Do the configuration
+	(cd $(@D)/build; \
+		$(HOST_CONFIGURE_OPTS) \

Hello Samuel,

Thanks for taking a look into this!

Quoted 4 lines · Samuel Mendoza-Jonas

In glibc 2.27 the following change occured:
"Statically compiled applications attempting to load locales compiled for the GNU C Library version 2.27 will fail and fall back to the builtin C/POSIX locale."

I don't really understand this explanation. Isn't the problem that glibc 2.27 doesn't read properly locale files generated by localedef from glibc 2.24 ?

I kind of fail to parse the explanation given here :-/

Quoted 8 lines

This impacts us since upstream buildroot uses a localdef built against an older eglibc release [0].
Update the host-localedef package to build against glibc 2.27. Unfortunately we can't compile just the localedef utility, but even so this does not increase the build time too significantly. eg:

With glibc localedef:

Quoted 8 lines

real    33m11.278s
user    171m51.684s
sys     19m46.528s

With eglibc localedef:

real    31m12.715s
user    155m13.616s
sys     16m23.504s

Meh, two more minutes just to build some not so complicated user-space program :-/

package/localedef/localedef.mk line 15

Quoted 3 lines

package/localedef/localedef.mk +2 -1

-HOST_LOCALEDEF_CONF_ENV = CFLAGS="$(HOST_CFLAGS) -fgnu89-inline"
+# glibc is part of the toolchain so disable the toolchain dependency
+HOST_LOCALEDEF_ADD_TOOLCHAIN_DEPENDENCY = NO

Not needed for host-localedef.

package/localedef/localedef.mk line 20

Quoted 5 lines

package/localedef/localedef.mk +4 -1

-# The makefile does not implement an install target
+HOST_LOCALEDEF_SUBDIR = build
+
+# Thumb build is broken, build in ARM mode
+ifeq ($(BR2_ARM_INSTRUCTIONS_THUMB),y)

This is a target related option...

Quoted 1 line

+HOST_LOCALEDEF_EXTRA_CFLAGS += -marm

... so it is not relevant to observe it when building a host program.

package/localedef/localedef.mk line 32

Quoted 11 lines

package/localedef/localedef.mk +11 -0

+endif
+
+# MIPS64 defaults to n32 so pass the correct -mabi if
+# we are using a different ABI. OABI32 is also used
+# in MIPS so we pass -mabi=32 in this case as well
+# even though it's not strictly necessary.
+ifeq ($(BR2_MIPS_NABI64),y)
+HOST_LOCALEDEF_EXTRA_CFLAGS += -mabi=64
+else ifeq ($(BR2_MIPS_OABI32),y)
+HOST_LOCALEDEF_EXTRA_CFLAGS += -mabi=32
+endif

Same comment.

package/localedef/localedef.mk line 36

Quoted 3 lines

package/localedef/localedef.mk +3 -0

+ifeq ($(BR2_ENABLE_DEBUG),y)
+HOST_LOCALEDEF_EXTRA_CFLAGS += -g
+endif

Same comment.

package/localedef/localedef.mk line 45

Quoted 9 lines

package/localedef/localedef.mk +9 -0

+
+# The stubs.h header is not installed by install-headers, but is
+# needed for the gcc build. An empty stubs.h will work, as explained
+# in http://gcc.gnu.org/ml/gcc/2002-01/msg00900.html. The same trick
+# is used by Crosstool-NG.
+ifeq ($(BR2_TOOLCHAIN_BUILDROOT_HOST_LOCALEDEF),y)
+define HOST_LOCALEDEF_ADD_MISSING_STUB_H
+	mkdir -p $(STAGING_DIR)/usr/include/gnu
+	touch $(STAGING_DIR)/usr/include/gnu/stubs.h

STAGING_DIR is observed when building target packages, not host packages, so this seems useless.

Quoted 23 lines

package/localedef/localedef.mk +23 -0

+endef
+endif
+
+# Even though we use the autotools-package infrastructure, we have to
+# override the default configure commands for several reasons:
+#
+#  1. We have to build out-of-tree, but we can't use the same
+#     'symbolic link to configure' used with the gcc packages.
+#
+#  2. We have to execute the configure script with bash and not sh.
+#
+# Note that as mentionned in
+# http://patches.openembedded.org/patch/38849/, glibc must be
+# built with -O2, so we pass our own CFLAGS and CXXFLAGS below.
+define HOST_LOCALEDEF_CONFIGURE_CMDS
+	mkdir -p $(@D)/build
+	# Do the configuration
+	(cd $(@D)/build; \
+		$(HOST_CONFIGURE_OPTS) \
+		CFLAGS="-O2 $(HOST_LOCALEDEF_EXTRA_CFLAGS)" CPPFLAGS="" \
+		CXXFLAGS="-O2 $(HOST_LOCALEDEF_EXTRA_CFLAGS)" \
+		$(SHELL) $(@D)/configure \
+		ac_cv_path_BASH_SHELL=/bin/bash \

package/localedef/localedef.mk line 74

Quoted 6 lines

package/localedef/localedef.mk +6 -0

+		libc_cv_forced_unwind=yes \
+		libc_cv_ssp=no \
+		--target=$(GNU_HOST_NAME) \
+		--host=$(GNU_HOST_NAME) \
+		--build=$(GNU_HOST_NAME) \
+		--prefix=/usr \

Should be --prefix=$(HOST_DIR)

package/localedef/localedef.mk line 76

Quoted 3 lines

package/localedef/localedef.mk +2 -0

+		$(if $(BR2_SOFT_FLOAT),--without-fp,--with-fp) \
+		$(if $(BR2_x86_64),--enable-lock-elision) \

Both BR2_SOFT_FLOAT and BR2_x86_64 are target options, so observing them when building a host package is almost always wrong.

package/localedef/localedef.mk line 83

Quoted 7 lines

package/localedef/localedef.mk +7 -0

+		--with-pkgversion="Buildroot" \
+		--without-cvs \
+		--disable-profile \
+		--without-gd \
+		--enable-obsolete-rpc)
+	$(HOST_LOCALEDEF_ADD_MISSING_STUB_H)
+endef

With all this, aren't we building a full glibc for the host ?

Can you use the locale/others and locale/install-others make targets instead ?

See how they are doing it in PTXdist:
https://git.pengutronix.de/cgit/ptxdist/tree/rules/host-localedef.make.
Most likely using the locale/others make target will help cut the build time significantly.

Thanks!

Thomas

I think it's a bit of both; 2.27 introduces new locale descriptions which localedef can't handle, eg:

Generating locale ru_RU.UTF-8 /scratch/jenkins-workspace/op-build-upstream/output/host/powerpc64le-buildroot-linux-gnu/sysroot/usr/share/i18n/locales/ru_RU:125: LC_TIME:syntax error /scratch/jenkins-workspace/op-build-upstream/output/host/powerpc64le-buildroot-linux-gnu/sysroot/usr/share/i18n/locales/ru_RU:149: LC_TIME:syntax error

And we fall back to the C locale at runtime (at least in my environment). I'll stare at the glibc release notes / bug report a bit more and see if I can make a more legible explanation.

Quoted 4 lines

See how they are doing it in PTXdist:
https://git.pengutronix.de/cgit/ptxdist/tree/rules/host-localedef.make.
Most likely using the locale/others make target will help cut the build time significantly.

Yes this is the contentious part, we're building a whole glibc. Unfortunately if we want to build just the locale/others target like PTXdist we also need their bonus patch to glibc:
https://git.pengutronix.de/cgit/ptxdist/tree/patches/localedef-glibc-2.27/0001-HACK-only-build-and-install-localedef.patch

Otherwise the build fails with, for example:

make[4]: *** No rule to make target '/scratch/builds/locales/build/host-localedef-glibc-2.27-57-g6c99e37f6fb640a50a3113b2dbee5d5389843c1e/build/elf/soinit.os', needed by '/scratch/builds/locales/build/host-localedef-glibc-2.27-57-g6c99e37f6fb640a50a3113b2dbee5d5389843c1e/build/libc.so'. Stop. Makefile:215: recipe for target 'locale/others' failed
make[3]: *** [locale/others] Error 2
Makefile:9: recipe for target 'locale/others' failed
make[2]: *** [locale/others] Error 2

So the tradeoff is adding a few minutes of extra build time versus carrying a bit of a hack to enable just the localedef/others target.

Hello,

Quoted 6 lines · Samuel Mendoza-Jonas

I think it's a bit of both; 2.27 introduces new locale descriptions which localedef can't handle, eg:

Generating locale ru_RU.UTF-8 /scratch/jenkins-workspace/op-build-upstream/output/host/powerpc64le-buildroot-linux-gnu/sysroot/usr/share/i18n/locales/ru_RU:125: LC_TIME:syntax error /scratch/jenkins-workspace/op-build-upstream/output/host/powerpc64le-buildroot-linux-gnu/sysroot/usr/share/i18n/locales/ru_RU:149: LC_TIME:syntax error

Ah, so I guess this update of localedef is going to fix
https://bugs.busybox.net/show_bug.cgi?id=11096.

Quoted 3 lines

And we fall back to the C locale at runtime (at least in my environment). I'll stare at the glibc release notes / bug report a bit more and see if I can make a more legible explanation.

Would be nice, though it generally makes sense to have a localedef implementation that is in sync with glibc.

Quoted 12 lines

Quoted 4 lines

See how they are doing it in PTXdist:
https://git.pengutronix.de/cgit/ptxdist/tree/rules/host-localedef.make.
Most likely using the locale/others make target will help cut the build time significantly.

Yes this is the contentious part, we're building a whole glibc. Unfortunately if we want to build just the locale/others target like PTXdist we also need their bonus patch to glibc:
https://git.pengutronix.de/cgit/ptxdist/tree/patches/localedef-glibc-2.27/0001-HACK-only-build-and-install-localedef.patch

Otherwise the build fails with, for example:

Quoted 8 lines

make[4]: *** No rule to make target '/scratch/builds/locales/build/host-localedef-glibc-2.27-57-g6c99e37f6fb640a50a3113b2dbee5d5389843c1e/build/elf/soinit.os', needed by '/scratch/builds/locales/build/host-localedef-glibc-2.27-57-g6c99e37f6fb640a50a3113b2dbee5d5389843c1e/build/libc.so'. Stop. Makefile:215: recipe for target 'locale/others' failed
make[3]: *** [locale/others] Error 2
Makefile:9: recipe for target 'locale/others' failed
make[2]: *** [locale/others] Error 2

So the tradeoff is adding a few minutes of extra build time versus carrying a bit of a hack to enable just the localedef/others target.

Their patch is not that crazy IMO, it should be too complicated to maintain. Perhaps we could even improve it a bit further to make it acceptable upstream ?

Best regards,
l
Thomas