Fixes for build failures for multilibs + usrmerge, NLS and GCONV

PATCH In review Jason Wessel 3 participants 9 messages Jul 2, 2019 – Jul 8, 2019

lore

Share & tools

lore

Jason Wessel

author

2019-07-02 20:16 UTC #0 lore raw

cover Fixes for build failures for multilibs + usrmerge, NLS and GCONV

Here are a few more fixes for usrmerge, multilibs, NLS, and GCONV.

Thanks,
Jason.

Jason Wessel

author

2019-07-02 20:16 UTC #1 lore raw

PATCH 1/3 psmisc: Fix dependency for USE_NLS=no

When using USE_NLS="no" in the local.conf psmisc will fail to compile as follows:

| autoreconf: Entering directory `.'
| autoreconf: running: autopoint --force
| autoreconf: failed to run autopoint: No such file or directory
| autoreconf: autopoint is needed because this package uses Gettext
| ERROR: autoreconf execution failed.

This is because the gettext.bbclass returns gettext-minimal-native for the host dependency which does not include autopoint. The autopoint utility is required to build psmisc, so it needs to list gettext-native as a dependency.

Signed-off-by Jason Wessel <[email protected]>

---
 meta/recipes-extended/psmisc/psmisc.inc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

meta/recipes-extended/psmisc/psmisc.inc +1 -1

index 82ef947709..594a10cf22 100644
@@ -7,7 +7,7 @@ command sends a specified signal (SIGTERM if nothing is specified) to \
 processes identified by name.  The fuser command identifies the PIDs \
 of processes that are using specified files or filesystems."
 SECTION = "base"
-DEPENDS = "ncurses virtual/libintl"
+DEPENDS = "ncurses virtual/libintl gettext-native"
 LICENSE = "GPLv2"
 
 SRC_URI = "${SOURCEFORGE_MIRROR}/psmisc/psmisc-${PV}.tar.gz"

Signature · 1 line

2.21.0

Jason Wessel

author

2019-07-02 20:16 UTC #2 lore raw

PATCH 2/3 glibc-locale: Fix build error with PACKAGE_NO_GCONV = "1"

In reply to Jason Wessel

When the PACKAGE_NO_GCONV is set to 1 an empty directory is left behind from the do_install rule:

=====
ERROR: glibc-locale-2.29-r0 do_package: QA Issue: glibc-locale: Files/directories were installed but not shipped in any package:
/usr/lib
/usr/lib/locale
Please set FILES such that these items are packaged. Alternatively if they are unneeded, avoid installing them or delete them within do_install.
glibc-locale: 2 installed and not shipped files. [installed-vs-shipped]
ERROR: glibc-locale-2.29-r0 do_package: Fatal QA errors found, failing task.
=====

The simple fix is to prune the empty directory.

Signed-off-by Jason Wessel <[email protected]>

---
 meta/recipes-core/glibc/glibc-locale.inc | 3 +++
 1 file changed, 3 insertions(+)

meta/recipes-core/glibc/glibc-locale.inc +3 -0

index f033635bd0..bf5eaee938 100644
@@ -87,6 +87,9 @@ do_install() {
 	if [ ${PACKAGE_NO_GCONV} -eq 0 ]; then
 		copy_locale_files ${libdir}/gconv 0755
 		copy_locale_files ${datadir}/i18n 0644
+	else
+		# Remove the libdir if it is empty when gconv is not copied
+		find ${D}${libdir} -type d -empty -delete
 	fi
 	copy_locale_files ${datadir}/locale 0644
 	install -m 0644 ${LOCALETREESRC}/SUPPORTED ${WORKDIR}/SUPPORTED

Signature · 1 line

2.21.0

Jason Wessel

author

2019-07-02 20:16 UTC #3 lore raw

PATCH 3/3 glibc/glibc-locale: Fix do_stash_locale to work with usrmerge and multilibs

In reply to Jason Wessel

The do_stash_locale was not working consistently across the 4 build configurations and the multilib, usrmerge configuration would fail entirely with the obscure message:

| DEBUG: Executing shell function do_prep_locale_tree
| tar: i18n: Cannot stat: No such file or directory
| tar: Exiting with failure status due to previous errors
| gzip: /poky/build/tmp/work/core2-64-poky-linux/glibc-locale/2.29-r0/locale-tree//usr/share/i18n/charmaps/*gz.gz: No such file or directory
| WARNING: /poky/build/tmp/work/core2-64-poky-linux/glibc-locale/2.29-r0/temp/run.do_prep_locale_tree.124690:1 exit 1 from 'gunzip $i'

Here is the 4 build configurations without the patch applied:

A) x86-64 no multilibs, no usrmerge
find ./tmp/work/*/glibc/2.29-r0/stashed-locale -type f |grep -v nscd.service |wc -l 909
B) x86-64 no multilibs, usrmerge
find ./tmp/work/*/glibc/2.29-r0/stashed-locale -type f |grep -v nscd.service |wc -l 909
C) x86-64 multilibs, no usrmerge
find ./tmp/work/*/glibc/2.29-r0/stashed-locale -type f |grep -v nscd.service |wc -l 885
D) x86-64 multilibs, usrmerge
find ./tmp/work/*/glibc/2.29-r0/stashed-locale -type f |grep -v nscd.service |wc -l 864

The issue here is that all the moves should be processed first, then a copy should be made of the lib directories, but only in the case they are different when using the usrmerge feature. Even though the build worked for the multilib configuration without usrmerge, the content was not the same.

After applying the patch the same number of files are in all the configurations. The list of files was also diffed, after normalizing the directory names to ensure all the correct files were copied.

Ultimately there are probably additional files that should be pruned from what is copied to the stated_locale, but the purpose of this patch is make it 100% consistent between the build types and fix the builds.

Signed-off-by Jason Wessel <[email protected]>

---
 meta/recipes-core/glibc/glibc-package.inc | 19 +++++++++++++------
 1 file changed, 13 insertions(+), 6 deletions(-)

meta/recipes-core/glibc/glibc-package.inc +13 -6

index a1d79b3075..ff17a193c3 100644
@@ -162,21 +162,28 @@ bashscripts = "mtrace sotruss xtrace"
 do_stash_locale () {
 	dest=${LOCALESTASH}
 	install -d ${dest}${base_libdir} ${dest}${bindir} ${dest}${libdir} ${dest}${datadir}
-	if [ "${base_libdir}" != "${libdir}" ]; then
-		cp -fpPR ${D}${base_libdir}/* ${dest}${base_libdir}
-	fi
+	# Hide away the locale data from the deployment
 	if [ -e ${D}${bindir}/localedef ]; then
 		mv -f ${D}${bindir}/localedef ${dest}${bindir}
 	fi
 	if [ -e ${D}${libdir}/gconv ]; then
 		mv -f ${D}${libdir}/gconv ${dest}${libdir}
 	fi
-	if [ -e ${D}${exec_prefix}/lib ]; then
-		cp -fpPR ${D}${exec_prefix}/lib ${dest}${exec_prefix}
-	fi
 	if [ -e ${D}${datadir}/i18n ]; then
 		mv ${D}${datadir}/i18n ${dest}${datadir}
 	fi
+
+	# Make a copy of all the libraries into the locale stash
+	cp -fpPR ${D}${libdir}/* ${dest}${libdir}
+	if [ "${base_libdir}" != "${libdir}" ]; then
+		cp -fpPR ${D}${base_libdir}/* ${dest}${base_libdir}
+	fi
+	if [ -e ${D}${exec_prefix}/lib ]; then
+		if [ ${exec_prefix}/lib != ${base_libdir} ] && [ ${exec_prefix}/lib != ${libdir} ]; then
+			cp -fpPR ${D}${exec_prefix}/lib ${dest}${exec_prefix}
+		fi
+	fi
+
 	cp -fpPR ${D}${datadir}/* ${dest}${datadir}
 	rm -rf ${D}${datadir}/locale/
 	cp -fpPR ${WORKDIR}/SUPPORTED ${dest}

Signature · 1 line

2.21.0

Martin Jansa

2019-07-04 09:13 UTC #4 lore raw

Re: [PATCH 3/3] glibc/glibc-locale: Fix do_stash_locale to work with usrmerge and multilibs

I don't see how this change (or the previous glibc-locale one) would cause that, but since the oe-core upgrade yesterday I'm seeing following glibc-locale.do_package failure:
http://errors.yoctoproject.org/Errors/Details/250557/

DEBUG: Executing shell function do_prep_locale_tree
tar: i18n: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
gzip:
TOPDIR/tmp-glibc/work/core2-64-oe-linux/glibc-locale/2.29-r0/locale-tree//usr/share/i18n/charmaps/*gz.gz:
No such file or directory
WARNING:
TOPDIR/tmp-glibc/work/core2-64-oe-linux/glibc-locale/2.29-r0/temp/run.do_prep_locale_tree.23303:1 exit 1 from 'gunzip $i'

anyone else seeing this?

Quoted 105 lines · Jason Wessel

The do_stash_locale was not working consistently across the 4 build configurations and the multilib, usrmerge configuration would fail entirely with the obscure message:
| DEBUG: Executing shell function do_prep_locale_tree
| tar: i18n: Cannot stat: No such file or directory
| tar: Exiting with failure status due to previous errors
| gzip:
/poky/build/tmp/work/core2-64-poky-linux/glibc-locale/2.29-r0/locale-tree//usr/share/i18n/charmaps/*gz.gz:
No such file or directory
| WARNING:
/poky/build/tmp/work/core2-64-poky-linux/glibc-locale/2.29-r0/temp/run.do_prep_locale_tree.124690:1 exit 1 from 'gunzip $i'
Here is the 4 build configurations without the patch applied:
A) x86-64 no multilibs, no usrmerge
find ./tmp/work/*/glibc/2.29-r0/stashed-locale -type f |grep -v nscd.service |wc -l
909
B) x86-64 no multilibs, usrmerge
find ./tmp/work/*/glibc/2.29-r0/stashed-locale -type f |grep -v nscd.service |wc -l
909
C) x86-64 multilibs, no usrmerge
find ./tmp/work/*/glibc/2.29-r0/stashed-locale -type f |grep -v nscd.service |wc -l
885
D) x86-64 multilibs, usrmerge
find ./tmp/work/*/glibc/2.29-r0/stashed-locale -type f |grep -v nscd.service |wc -l
864
The issue here is that all the moves should be processed first, then a copy should be made of the lib directories, but only in the case they are different when using the usrmerge feature. Even though the build worked for the multilib configuration without usrmerge, the content was not the same.
After applying the patch the same number of files are in all the configurations. The list of files was also diffed, after normalizing the directory names to ensure all the correct files were copied.
Ultimately there are probably additional files that should be pruned from what is copied to the stated_locale, but the purpose of this patch is make it 100% consistent between the build types and fix the builds.
Signed-off-by Jason Wessel <[email protected]>
---
 meta/recipes-core/glibc/glibc-package.inc | 19 +++++++++++++------
 1 file changed, 13 insertions(+), 6 deletions(-)
meta/recipes-core/glibc/glibc-package.inc +9 -6
index a1d79b3075..ff17a193c3 100644
@@ -162,21 +162,28 @@ bashscripts = "mtrace sotruss xtrace"
 do_stash_locale () {
        dest=${LOCALESTASH}
        install -d ${dest}${base_libdir} ${dest}${bindir} ${dest}${libdir}
 ${dest}${datadir}
-       if [ "${base_libdir}" != "${libdir}" ]; then
-               cp -fpPR ${D}${base_libdir}/* ${dest}${base_libdir}
-       fi
+       # Hide away the locale data from the deployment
        if [ -e ${D}${bindir}/localedef ]; then
                mv -f ${D}${bindir}/localedef ${dest}${bindir}
        fi
        if [ -e ${D}${libdir}/gconv ]; then
                mv -f ${D}${libdir}/gconv ${dest}${libdir}
        fi
-       if [ -e ${D}${exec_prefix}/lib ]; then
-               cp -fpPR ${D}${exec_prefix}/lib ${dest}${exec_prefix}
-       fi
        if [ -e ${D}${datadir}/i18n ]; then
                mv ${D}${datadir}/i18n ${dest}${datadir}
        fi
+
+       # Make a copy of all the libraries into the locale stash
+       cp -fpPR ${D}${libdir}/* ${dest}${libdir}
+       if [ "${base_libdir}" != "${libdir}" ]; then
+               cp -fpPR ${D}${base_libdir}/* ${dest}${base_libdir}
+       fi
+       if [ -e ${D}${exec_prefix}/lib ]; then
+               if [ ${exec_prefix}/lib != ${base_libdir} ] && [
${exec_prefix}/lib != ${libdir} ]; then
+ cp -fpPR ${D}${exec_prefix}/lib
${dest}${exec_prefix}
meta/recipes-core/glibc/glibc-package.inc +3 -1
+               fi
+       fi
+
        cp -fpPR ${D}${datadir}/* ${dest}${datadir}
        rm -rf ${D}${datadir}/locale/
        cp -fpPR ${WORKDIR}/SUPPORTED ${dest}
--
2.21.0
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Adrian Bunk

2019-07-04 10:57 UTC #5 lore raw

Re: [PATCH 3/3] glibc/glibc-locale: Fix do_stash_locale to work with usrmerge and multilibs

Quoted 15 lines · Martin Jansa

I don't see how this change (or the previous glibc-locale one) would cause that, but since the oe-core upgrade yesterday I'm seeing following glibc-locale.do_package failure:
http://errors.yoctoproject.org/Errors/Details/250557/
DEBUG: Executing shell function do_prep_locale_tree
tar: i18n: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
gzip:
TOPDIR/tmp-glibc/work/core2-64-oe-linux/glibc-locale/2.29-r0/locale-tree//usr/share/i18n/charmaps/*gz.gz:
No such file or directory
WARNING:
TOPDIR/tmp-glibc/work/core2-64-oe-linux/glibc-locale/2.29-r0/temp/run.do_prep_locale_tree.23303:1 exit 1 from 'gunzip $i'

Quoted 1 line

anyone else seeing this?

Yes.

cu
Adrian

Signature · 4 lines

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Jason Wessel

author

2019-07-04 11:57 UTC #6 lore raw

Re: [PATCH 3/3] glibc/glibc-locale: Fix do_stash_locale to work with usrmerge and multilibs

Quoted 16 lines · Adrian Bunk

Quoted 15 lines · Martin Jansa
I don't see how this change (or the previous glibc-locale one) would cause that, but since the oe-core upgrade yesterday I'm seeing following glibc-locale.do_package failure:
http://errors.yoctoproject.org/Errors/Details/250557/
DEBUG: Executing shell function do_prep_locale_tree
tar: i18n: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
gzip:
TOPDIR/tmp-glibc/work/core2-64-oe-linux/glibc-locale/2.29-r0/locale-tree//usr/share/i18n/charmaps/*gz.gz:
No such file or directory
WARNING:
TOPDIR/tmp-glibc/work/core2-64-oe-linux/glibc-locale/2.29-r0/temp/run.do_prep_locale_tree.23303:1 exit 1 from 'gunzip $i'

Quoted 4 lines

Quoted 1 line
anyone else seeing this?
Yes.

I don't know that it was specifically this change that causes the problem but rather any change to this function. If that is not the case can either of you send me your local.conf file, and I'll try to reproduce it. If the following doesn't work around the problem, there is clearly another corner case that is not well understood.

bitbake -c cleansstate glibc glibc-locale

I believe that any change to the .inc file which has locale stash function will cause the problem due to the fact it performs and "mv" operation to move the files. When it runs a second time, there are no more files to move and you will not have them for use later on. When I was experimenting earlier, I tried running the glibc temp/run.do_install script to put the files back, but then hit a problem with localdef being missing, because it comes from the temp/run.do_compile functions.

My conclusion thus far has been is that the ties between the glibc and glibc-locale are very subtle and break easily. This is a separate problem than what I was attempting to fix, where the usrmerge + multilibs didn't even get the correct files in the first place.

Cheers,

Jason.

Adrian Bunk

2019-07-06 19:31 UTC #7 lore raw

Re: [PATCH 3/3] glibc/glibc-locale: Fix do_stash_locale to work with usrmerge and multilibs

Quoted 19 lines · Jason Wessel

Quoted 18 lines · Adrian Bunk
Quoted 16 lines · Martin Jansa
I don't see how this change (or the previous glibc-locale one) would cause that, but since the oe-core upgrade yesterday I'm seeing following glibc-locale.do_package failure:
http://errors.yoctoproject.org/Errors/Details/250557/
DEBUG: Executing shell function do_prep_locale_tree
tar: i18n: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
gzip:
TOPDIR/tmp-glibc/work/core2-64-oe-linux/glibc-locale/2.29-r0/locale-tree//usr/share/i18n/charmaps/*gz.gz:
No such file or directory
WARNING:
TOPDIR/tmp-glibc/work/core2-64-oe-linux/glibc-locale/2.29-r0/temp/run.do_prep_locale_tree.23303:1 exit 1 from 'gunzip $i'
anyone else seeing this?

Quoted 7 lines

Quoted 1 line
Yes.
I don't know that it was specifically this change that causes the problem but rather any change to this function. If that is not the case can either of you send me your local.conf file, and I'll try to reproduce it. If the following doesn't work around the problem, there is clearly another corner case that is not well understood.
bitbake -c cleansstate glibc glibc-locale
...

bitbake -c cleansstate glibc glibc-locale nativesdk-glibc nativesdk-glibc-locale

This seems to workaround the problem in the cases I have seen so far, so what you said about the cause looks plausible to me.

Quoted 3 lines

Cheers,
Jason.

cu
Adrian

Signature · 4 lines

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Jason Wessel

author

2019-07-08 19:21 UTC #8 lore raw

Re: [PATCH 3/3] glibc/glibc-locale: Fix do_stash_locale to work with usrmerge and multilibs

Quoted 25 lines · Adrian Bunk

Quoted 24 lines · Jason Wessel
Quoted 16 lines · Adrian Bunk
Quoted 15 lines · Martin Jansa
I don't see how this change (or the previous glibc-locale one) would cause that, but since the oe-core upgrade yesterday I'm seeing following glibc-locale.do_package failure:
http://errors.yoctoproject.org/Errors/Details/250557/
DEBUG: Executing shell function do_prep_locale_tree
tar: i18n: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
gzip:
TOPDIR/tmp-glibc/work/core2-64-oe-linux/glibc-locale/2.29-r0/locale-tree//usr/share/i18n/charmaps/*gz.gz:
No such file or directory
WARNING:
TOPDIR/tmp-glibc/work/core2-64-oe-linux/glibc-locale/2.29-r0/temp/run.do_prep_locale_tree.23303:1 exit 1 from 'gunzip $i'
Quoted 3 lines
Quoted 1 line
anyone else seeing this?
Yes.
I don't know that it was specifically this change that causes the problem but rather any change to this function. If that is not the case can either of you send me your local.conf file, and I'll try to reproduce it. If the following doesn't work around the problem, there is clearly another corner case that is not well understood.

Quoted 8 lines

Quoted 2 lines
bitbake -c cleansstate glibc glibc-locale
...
bitbake -c cleansstate glibc glibc-locale nativesdk-glibc nativesdk-glibc-locale
This seems to workaround the problem in the cases I have seen so far, so what you said about the cause looks plausible to me.

I chatted with Richard to figure out how to fix the original implementation, so this will not continue to happen. I am no fan of incremental or sstate based builds failing. The pending fix is below.

https://patchwork.openembedded.org/patch/162886/

Cheers,
Jason.