bitbake.conf: change localedir

PATCH In review Chen Qi 6 participants 14 messages Aug 8, 2016 – Aug 12, 2016

lore

Share & tools

lore

Chen Qi

author

2016-08-08 06:04 UTC #0 lore raw

cover bitbake.conf: change localedir

The following changes since commit c4dda05cb9755cb865c1d1bd142196f8a9f45d3d:

Revert "packagegroup-core-x11-base.bb: replace pointercal with pointercal-xinput" (2016-08-04 20:54:57 +0100)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib ChenQi/locale-fix
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=ChenQi/locale-fix

Chen Qi (1):
bitbake.conf: change localedir

 meta/conf/bitbake.conf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Signature · 1 line

1.9.1

Chen Qi

author

2016-08-08 06:04 UTC #1 lore raw

PATCH 1/1 bitbake.conf: change localedir

Previously, localedir is set to "${libdir}/locale". This would result in locale database installed in '/usr/lib64/locale' in some multilib case. For example, if we build out a multilib x86-64 self-hosted image and we try to build projects on this host, things broke and the following error appears.

Please use a locale setting which supports utf-8.
Python can't change the filesystem locale after loading so we need a utf-8 when python starts or things won't work.

This is because '/usr/lib/locale' is the default one. And actually the nativesdk-glibc is now set to use '/usr/lib/locale'.

Thus, we change the setting of 'localedir' to '${nonarch_libdir}/locale' to fix the above problem.

Signed-off-by Chen Qi <[email protected]>

---
 meta/conf/bitbake.conf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

meta/conf/bitbake.conf +1 -1

index c32a3f8..ed77ee3 100644
@@ -44,7 +44,7 @@ export libdir = "${exec_prefix}/${baselib}"
 export libexecdir = "${exec_prefix}/libexec"
 export includedir = "${exec_prefix}/include"
 export oldincludedir = "${exec_prefix}/include"
-localedir = "${libdir}/locale"
+localedir = "${nonarch_libdir}/locale"
 
 # Linkage between native/cross/nativesdk layouts
 base_bindir_native = "/bin"

Signature · 1 line

1.9.1

Hongxu Jia

2016-08-10 05:50 UTC #2 lore raw

PATCH 1/1 glibc: apply complocaledir while localedir changed

Since localedir changed, we should apply complocaledir=/usr/lib/locale

Build image with IMAGE_LINGUAS="en-us en-gb"

Boot target and display all locales:
...

root@localhost:~# locale -a
C
POSIX
en_GB
en_US
...

Signed-off-by Hongxu Jia <[email protected]>

---
 meta/recipes-core/glibc/glibc_2.24.bb | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

meta/recipes-core/glibc/glibc_2.24.bb +1 -3

index 88bcbef2..219370c 100644
@@ -128,9 +128,7 @@ do_compile () {
 
 }
 
-# Use the host locale archive when built for nativesdk so that we don't need to
-# ship a complete (100MB) locale set.
-do_compile_prepend_class-nativesdk() {
+do_compile_prepend() {
     echo "complocaledir=/usr/lib/locale" >> ${S}/configparms
 }

Signature · 1 line

2.8.1

Hongxu Jia

2016-08-10 05:55 UTC #3 lore raw

Re: [PATCH] glibc: apply complocaledir while localedir changed

Quoted 23 lines · Hongxu Jia

Since localedir changed, we should apply complocaledir=/usr/lib/locale
Build image with IMAGE_LINGUAS="en-us en-gb"
Boot target and display all locales:
...
root@localhost:~# locale -a
C
POSIX
en_GB
en_US
...
Signed-off-by Hongxu Jia <[email protected]>
---
  meta/recipes-core/glibc/glibc_2.24.bb | 4 +---
  1 file changed, 1 insertion(+), 3 deletions(-)
meta/recipes-core/glibc/glibc_2.24.bb +0 -0
index 88bcbef2..219370c 100644

meta/recipes-core/glibc/glibc_2.24.bb do_compile () { · line 132

Quoted 8 lines

meta/recipes-core/glibc/glibc_2.24.bb +1 -3

@@ -128,9 +128,7 @@ do_compile () {
 
  }
 
-# Use the host locale archive when built for nativesdk so that we don't need to
-# ship a complete (100MB) locale set.
-do_compile_prepend_class-nativesdk() {
+do_compile_prepend() {
      echo "complocaledir=/usr/lib/locale" >> ${S}/configparms

Please drop this, It should be "complocaledir=${localedir}" better.

//Hongxu

Quoted 2 lines

meta/recipes-core/glibc/glibc_2.24.bb +0 -0
  }
 

Hongxu Jia

2016-08-10 07:31 UTC #4 lore raw

PATCH v2 1/1 glibc: apply complocaledir while localedir changed

In reply to Chen Qi

Since localedir changed, we should apply complocaledir=${localedir}

Build image with IMAGE_LINGUAS="en-us en-gb"

Boot target and display all locales:
...

root@localhost:~# locale -a
C
POSIX
en_GB
en_US
...

Signed-off-by Hongxu Jia <[email protected]>

---
 meta/recipes-core/glibc/glibc_2.24.bb | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

meta/recipes-core/glibc/glibc_2.24.bb +2 -4

index 88bcbef2..2f0c4be 100644
@@ -128,10 +128,8 @@ do_compile () {
 
 }
 
-# Use the host locale archive when built for nativesdk so that we don't need to
-# ship a complete (100MB) locale set.
-do_compile_prepend_class-nativesdk() {
-    echo "complocaledir=/usr/lib/locale" >> ${S}/configparms
+do_compile_prepend() {
+    echo "complocaledir=${localedir}" >> ${S}/configparms
 }
 
 require glibc-package.inc

Signature · 1 line

2.8.1

Hongxu Jia

2016-08-10 08:13 UTC #5 lore raw

PATCH v3 1/1 glibc: apply complocaledir while localedir changed

In reply to Chen Qi

Since localedir changed
-----------------------
commit bbca0646c7b55aa28dbf796ab9135bbc63a92a8d

Author: Chen Qi <[email protected]>
Date:   Mon Aug 8 13:55:42 2016 +0800

bitbake.conf: change localedir
-----------------------

Refer nativesdk, We should apply complocaledir=${localedir} for target.

Build image with IMAGE_LINGUAS="en-us en-gb"

Boot target and display all locales:
...

root@localhost:~# locale -a
C
POSIX
en_GB
en_US
...

(LOCAL REV: NOT UPSTREAM) -- sent to oe-core on 20160810

Signed-off-by Hongxu Jia <[email protected]>

---
 meta/recipes-core/glibc/glibc_2.24.bb | 4 ++++
 1 file changed, 4 insertions(+)

meta/recipes-core/glibc/glibc_2.24.bb +4 -0

index 88bcbef2..c1b5190 100644
@@ -134,6 +134,10 @@ do_compile_prepend_class-nativesdk() {
     echo "complocaledir=/usr/lib/locale" >> ${S}/configparms
 }
 
+do_compile_prepend_class-target() {
+    echo "complocaledir=${localedir}" >> ${S}/configparms
+}
+
 require glibc-package.inc
 
 BBCLASSEXTEND = "nativesdk"

Signature · 1 line

2.8.1

Burton, Ross

2016-08-11 16:26 UTC #6 lore raw

In reply to Chen Qi

Quoted 6 lines · Chen Qi

Previously, localedir is set to "${libdir}/locale". This would result in locale database installed in '/usr/lib64/locale' in some multilib case. For example, if we build out a multilib x86-64 self-hosted image and we try to build projects on this host, things broke and the following error appears.

Quoted 7 lines

Please use a locale setting which supports utf-8.
Python can't change the filesystem locale after loading so we need a
utf-8 when python starts or things won't work.
This is because '/usr/lib/locale' is the default one. And actually the nativesdk-glibc is now set to use '/usr/lib/locale'.

This is irrelevant as nativesdk-glibc is configured to read the *hosts* locale directory.

Quoted 4 lines

Thus, we change the setting of 'localedir' to '${nonarch_libdir}/locale' to
fix the above problem.

I see two issues here:
1) should binary locales be considered shared in multilib environments? (libdir vs nonarch_libdir)
2) what packages are not respecting this variable and hard-coding /usr/lib/locale?

I'm guessing WR think yes to (1), and is the glibc patch you also sent the fundamental fix to (2)?

Ross

Mark Hatle

2016-08-11 17:11 UTC #7 lore raw

Quoted 25 lines · Ross

On 8 August 2016 at 07:04, Chen Qi <[email protected] <mailto:[email protected]>> wrote:
Previously, localedir is set to "${libdir}/locale". This would result in locale database installed in '/usr/lib64/locale' in some multilib case. For example, if we build out a multilib x86-64 self-hosted image and we try to build projects on this host, things broke and the following error appears.
Please use a locale setting which supports utf-8.
Python can't change the filesystem locale after loading so we need a utf-8
when python starts or things won't work.
This is because '/usr/lib/locale' is the default one. And actually the nativesdk-glibc is now set to use '/usr/lib/locale'.
This is irrelevant as nativesdk-glibc is configured to read the *hosts* locale directory.
Thus, we change the setting of 'localedir' to '${nonarch_libdir}/locale' to fix the above problem.

Quoted 7 lines

I see two issues here:
1) should binary locales be considered shared in multilib environments? (libdir vs nonarch_libdir)
2) what packages are not respecting this variable and hard-coding /usr/lib/locale?
I'm guessing WR think yes to (1), and is the glibc patch you also sent the fundamental fix to (2)?

Binary locales have an endian and alignment setting to them. If a platform supports both big and little endian, this common locale would not work. (That is extremely rare....) Also if a platform supports different alignments in different libraries that could cause an impact as well. (This is also extremely unlikely.)

The not-binary locales have no such issues BTW.

--Mark

Quoted 3 lines

Ross

Burton, Ross

2016-08-11 19:15 UTC #8 lore raw

Quoted 8 lines · Mark Hatle

Binary locales have an endian and alignment setting to them. If a platform supports both big and little endian, this common locale would not work. (That
is extremely rare....) Also if a platform supports different alignments in different libraries that could cause an impact as well. (This is also extremely
unlikely.)

So by that logic this patch should be rejected, right?

This path is for the compiled locale definitions - LC_TYPE etc and the locale archive, right? So it's only glibc-locales that ships in here, not other packages (just trying to check my assumptions)

Ross

Mark Hatle

2016-08-11 19:32 UTC #9 lore raw

Quoted 4 lines · Ross

On 11 August 2016 at 18:11, Mark Hatle <[email protected] <mailto:[email protected]>> wrote:

Quoted 8 lines

Binary locales have an endian and alignment setting to them. If a platform supports both big and little endian, this common locale would not work. (That is extremely rare....) Also if a platform supports different alignments in different libraries that could cause an impact as well. (This is also extremely unlikely.)
So by that logic this patch should be rejected, right?

The binary bits was why it was not originally implemented this way. HOWEVER, are there are conflicting cases we have to worry about for the binaries? If there are not any reasonable cases -- then this may save a huge amount of space for a device that contains a number of binary locales.

(In otherwords, I don't object to the patch due to those reasons -- I just want it to be clear to folks that there are binary nuances to this, it is extremely rare that it's a problem, so rare it may simply NOT be an issue in our configuration. But these can never be 'noarch' packages as they are tied to an arch.)

--Mark

Quoted 5 lines

This path is for the compiled locale definitions - LC_TYPE etc and the locale archive, right? So it's only glibc-locales that ships in here, not other packages (just trying to check my assumptions)
Ross

Khem Raj

2016-08-11 19:47 UTC #10 lore raw

In reply to Mark Hatle

Quoted 34 lines · Mark Hatle

Quoted 30 lines · Ross
On 8 August 2016 at 07:04, Chen Qi <[email protected] <mailto:[email protected]>> wrote:
Previously, localedir is set to "${libdir}/locale". This would result in locale database installed in '/usr/lib64/locale' in some multilib case. For example, if we build out a multilib x86-64 self-hosted image and we try to build projects on this host, things broke and the following error appears.
Please use a locale setting which supports utf-8.
Python can't change the filesystem locale after loading so we need a utf-8
when python starts or things won't work.
This is because '/usr/lib/locale' is the default one. And actually the nativesdk-glibc is now set to use '/usr/lib/locale'.
This is irrelevant as nativesdk-glibc is configured to read the *hosts* locale directory.
Thus, we change the setting of 'localedir' to '${nonarch_libdir}/locale' to fix the above problem.
I see two issues here:
1) should binary locales be considered shared in multilib environments? (libdir vs nonarch_libdir)
2) what packages are not respecting this variable and hard-coding /usr/lib/locale?

Quoted 8 lines

Quoted 2 lines
I'm guessing WR think yes to (1), and is the glibc patch you also sent the fundamental fix to (2)?
Binary locales have an endian and alignment setting to them. If a platform supports both big and little endian, this common locale would not work. (That is extremely rare....) Also if a platform supports different alignments in different libraries that could cause an impact as well. (This is also extremely unlikely.)

Are there any practical existing usecases ?

Quoted 14 lines

The not-binary locales have no such issues BTW.
--Mark
Quoted 3 lines
Ross
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Mark Hatle

2016-08-12 13:50 UTC #11 lore raw

Quoted 37 lines · Khem Raj

Quoted 37 lines · Mark Hatle
Quoted 32 lines · Ross
On 8 August 2016 at 07:04, Chen Qi <[email protected] <mailto:[email protected]>> wrote:
Previously, localedir is set to "${libdir}/locale". This would result in locale database installed in '/usr/lib64/locale' in some multilib case. For example, if we build out a multilib x86-64 self-hosted image and we try to build projects on this host, things broke and the following error appears.
Please use a locale setting which supports utf-8.
Python can't change the filesystem locale after loading so we need a utf-8
when python starts or things won't work.
This is because '/usr/lib/locale' is the default one. And actually the nativesdk-glibc is now set to use '/usr/lib/locale'.
This is irrelevant as nativesdk-glibc is configured to read the *hosts* locale directory.
Thus, we change the setting of 'localedir' to '${nonarch_libdir}/locale' to fix the above problem.
I see two issues here:
1) should binary locales be considered shared in multilib environments? (libdir vs nonarch_libdir)
2) what packages are not respecting this variable and hard-coding /usr/lib/locale?
I'm guessing WR think yes to (1), and is the glibc patch you also sent the fundamental fix to (2)?

Quoted 7 lines

Quoted 5 lines
Binary locales have an endian and alignment setting to them. If a platform supports both big and little endian, this common locale would not work. (That is extremely rare....) Also if a platform supports different alignments in different libraries that could cause an impact as well. (This is also extremely unlikely.)
Are there any practical existing usecases ?

At one point there was talk about running little endian and big endian power on the same machine. Same with ARM. I've never actually seen this implemented.

The alignment was an issue between different S/390 multilibs at one point. But I've not followed it to see if that is still true. (Not that S/390 is a target for what we're doing.)

So to my knowledge I'm not aware of any use cases we regularly use in the Yocto Project -- but I'm also not aware of -all- use-cases.

I think the best compromise may be to ensure the value is configurable. Set it to the combined location -- with a note that if the multilibs have a different endian or alignment setting, to use the older setting breaking them into the multilibs. (I don't know what, if any changes this would require in glibc to know when to use which dir...)

Plan for the optimized case, support the unusual case.

--Mark

Quoted 15 lines

Quoted 14 lines
The not-binary locales have no such issues BTW.
--Mark
Quoted 3 lines
Ross
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Radzykewycz, T (Radzy)

2016-08-12 14:30 UTC #12 lore raw

________________________________________

Quoted 50 lines

From: [email protected] [[email protected]] on behalf of Mark Hatle [[email protected]]
Sent: Friday, August 12, 2016 6:50 AM
To: Khem Raj
Cc: OE-core
Subject: Re: [OE-core] [PATCH 1/1] bitbake.conf: change localedir
Quoted 43 lines · Khem Raj
Quoted 42 lines · Mark Hatle
Quoted 25 lines · Ross
On 8 August 2016 at 07:04, Chen Qi <[email protected] <mailto:[email protected]>> wrote:
Previously, localedir is set to "${libdir}/locale". This would result in locale database installed in '/usr/lib64/locale' in some multilib case. For example, if we build out a multilib x86-64 self-hosted image and we try to build projects on this host, things broke and the following error appears.
Please use a locale setting which supports utf-8.
Python can't change the filesystem locale after loading so we need a utf-8
when python starts or things won't work.
This is because '/usr/lib/locale' is the default one. And actually the nativesdk-glibc is now set to use '/usr/lib/locale'.
This is irrelevant as nativesdk-glibc is configured to read the *hosts* locale directory.
Thus, we change the setting of 'localedir' to '${nonarch_libdir}/locale' to fix the above problem.
Quoted 7 lines
I see two issues here:
1) should binary locales be considered shared in multilib environments? (libdir vs nonarch_libdir)
2) what packages are not respecting this variable and hard-coding /usr/lib/locale?
I'm guessing WR think yes to (1), and is the glibc patch you also sent the fundamental fix to (2)?
Binary locales have an endian and alignment setting to them. If a platform supports both big and little endian, this common locale would not work. (That is extremely rare....) Also if a platform supports different alignments in different libraries that could cause an impact as well. (This is also extremely unlikely.)

Quoted 4 lines

Quoted 1 line
Are there any practical existing usecases ?
At one point there was talk about running little endian and big endian power on the same machine. Same with ARM. I've never actually seen this implemented.

I have seen it implemented, but it was an odd situation.

IIRC: on Android, an app used a native library in one endian when the system was running the other. The context was a bug report, where the comment (and what brought it to my attention) was disgust that this was being done.

Not sure this is relevant to this discussion, but it's one data point to indicate that it was actually done, though not for a normal Linux system.

Enjoy!

-- radzy

Quoted 38 lines

The alignment was an issue between different S/390 multilibs at one point. But I've not followed it to see if that is still true. (Not that S/390 is a target for what we're doing.)
So to my knowledge I'm not aware of any use cases we regularly use in the Yocto Project -- but I'm also not aware of -all- use-cases.
I think the best compromise may be to ensure the value is configurable. Set it to the combined location -- with a note that if the multilibs have a different endian or alignment setting, to use the older setting breaking them into the multilibs. (I don't know what, if any changes this would require in glibc to know when to use which dir...)
Plan for the optimized case, support the unusual case.
--Mark
Quoted 9 lines
Quoted 9 lines
The not-binary locales have no such issues BTW.
--Mark
Quoted 3 lines
Ross
Quoted 6 lines
Quoted 5 lines
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

ChenQi

author

2016-08-12 03:28 UTC #13 lore raw

In reply to Burton, Ross

Quoted 15 lines · Ross

On 8 August 2016 at 07:04, Chen Qi <[email protected] <mailto:[email protected]>> wrote:
Previously, localedir is set to "${libdir}/locale". This would result in locale database installed in '/usr/lib64/locale' in some multilib case.
For example, if we build out a multilib x86-64 self-hosted image and we try
to build projects on this host, things broke and the following error appears.
Please use a locale setting which supports utf-8. Python can't change the filesystem locale after loading so we
need a utf-8 when python starts or things won't work.

Quoted 5 lines

This is because '/usr/lib/locale' is the default one. And actually the nativesdk-glibc is now set to use '/usr/lib/locale'.
This is irrelevant as nativesdk-glibc is configured to read the *hosts* locale directory.

Hi Ross,

I think I didn't state things clearly. Sorry for that.
I mentioned nativesdk-glibc because of the following use case.
On a self-hosted image, which is built out by Yocto, we install buildtools-tarball, make use of it and start building. This is why I mentioned nativesdk-glibc.

Quoted 7 lines

Thus, we change the setting of 'localedir' to
'${nonarch_libdir}/locale' to
fix the above problem.
I see two issues here:
1) should binary locales be considered shared in multilib environments? (libdir vs nonarch_libdir)

I think yes.
Below is something from glibc source. The file is git/sysdeps/unix/sysv/linux/x86_64/64/configure.

# Locale data can be shared between 32-bit and 64-bit libraries. libc_cv_complocaledir='${exec_prefix}/lib/locale'

You can see that it states clearly that the binary data could be shared and it uses 'lib'.

Quoted 2 lines

2) what packages are not respecting this variable and hard-coding /usr/lib/locale?

I don't know. But the manual of locale has stated clearly that '/usr/lib/locale' is the default directory. Something from the manual below.

LOCPATH

The directory where locale data is stored. By default,
/usr/lib/locale is used.

Quoted 2 lines

I'm guessing WR think yes to (1), and is the glibc patch you also sent the fundamental fix to (2)?

I don't know why the glibc patch is needed.
At a first glance, I think it might be related to GLIBC_INTERNAL_USE_BINARY_LOCALE handling. The value of this variable is overridden in our bbappend file. This might also be related to cross-localedef recipe.

Best Regards,
Chen Qi

Quoted 1 line

Ross