Morty Next pull request

Please consider these changes for Morty. Clean AB build

This included sdk locale and host glibc 2.27 fixes.

The following changes since commit 34b26ec18450a3bc72f7b8a0f2df0bfaae1140dd:

linux-firmware: Add reference to iwlwifi-8000C firmware (2018-03-19 15:58:33 +0000)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib stable/morty-next
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=stable/morty-next

Alexander Kanavin (2):
selftest/signing: add --batch to gpg invocation when importing keys populate_sdk_base.bbclass: add documentation packages to SDKs if
api-documentation is in DISTRO_FEATURES

Andreas Oberritter (1):
libc-package.bbclass: split binary localedata even more if asked to

Armin Kuster (3):
lib/oe/terminal.py: use an absolute path to execute
oe-gnome-terminal-phonehome
distcc: Change SRC_URI
package_manager: fix install_glob to use morty sdk pkgdir

Christopher Larson (1):
buildhistory: fix latest_srcrev in the common case

Ed Bartosh (1):
toolchain-shar-extract: compare SDK and host gcc versions

Gianfranco Costamagna (1):
glibc-locale.inc: fix typo in comment

Juro Bystricky (1):
glibc-locale.inc: avoid duplicate packages

Linus Wallgren (1):
lib/oe/package_manager: .deb pre/postinst args

Martin Jansa (1):
package_manager: flush installed_pkgs file before oe-pkgdata-util uses
it

Maxin B. John (1):
grub-efi: fix build failure

Richard Purdie (4):
lib/oe/package_manager/sdk: Ensure do_populate_sdk_ext and
do_populate_sdk repos don't conflict
glibc: Separate locale files to their own sstate task image/packagegroup/populate_sdk: Drop do_populate_sysroot task
properly
glibc-package: Avoid race sstate races with do_stash_locale

Ross Burton (15):
default-distrovars: don't rename locales for nativesdk
glibc: don't use host locales in nativesdk
glibc: relocate locale paths in nativesdk lib/oe/package_manager: bail if createrepo can't be found
package_manager: don't race on a file when installing complementary
packages
package_manager: improve install_complementary
package-manager: add install_glob()
sdk: install specified locales into SDK
sdk: only install locales if we're using glibc
populate_sdk: install UTF-8 locales in SDKs
populate_sdk_base: depend on nativesdk-glibc-locale
cross-localedef-native: add way to specify which locale archive to
write
qemu: fix memfd_create with glibc 2.27
sdk: generate locale archive and remove packages
sdk: streamline locale removal

Stephano Cetola (1):
package_manager: remove strings and migrate to direct arrays

Tanu Kaskinen (4):
e2fsprogs: fix compatibility with glibc 2.27
libvorbis: CVE-2017-14633
libvorbis: CVE-2017-14632
libvorbis: CVE-2018-5146

Zhenbo Gao (1):
archiver: avoid archiving source for glibc-locale

 meta/classes/archiver.bbclass                      |   6 +
 meta/classes/buildhistory.bbclass                  |   2 +-
 meta/classes/image.bbclass                         |   2 +-
 meta/classes/libc-package.bbclass                  |  39 ++-
 meta/classes/packagegroup.bbclass                  |   2 +-
 meta/classes/populate_sdk_base.bbclass             |  11 +-
 meta/classes/populate_sdk_ext.bbclass              |   3 +-
 meta/conf/distro/include/default-distrovars.inc    |   1 +
 meta/files/toolchain-shar-extract.sh               |   8 +
 meta/lib/oe/package.py                             |  13 +-
 meta/lib/oe/package_manager.py                     | 328 +++++++++++----------
 meta/lib/oe/sdk.py                                 |  85 +++++-
 meta/lib/oe/terminal.py                            |   2 +-
 meta/lib/oeqa/selftest/signing.py                  |   2 +-
 ...ix-check-for-sys-sysmacros.h-under-glibc-.patch |  48 +++
 meta/recipes-bsp/grub/grub-efi_2.00.bb             |   1 +
 .../glibc/cross-localedef-native_2.24.bb           |   1 +
 meta/recipes-core/glibc/glibc-collateral.inc       |   2 +-
 meta/recipes-core/glibc/glibc-initial.inc          |   2 +-
 meta/recipes-core/glibc/glibc-locale.inc           |   5 +-
 meta/recipes-core/glibc/glibc-mtrace.inc           |   2 +-
 meta/recipes-core/glibc/glibc-package.inc          |  45 +--
 meta/recipes-core/glibc/glibc-scripts.inc          |   2 +-
 meta/recipes-core/glibc/glibc/archive-path.patch   |  39 +++
 .../glibc/glibc/relocate-locales.patch             |  33 +++
 meta/recipes-core/glibc/glibc_2.24.bb              |   7 +-
 meta/recipes-core/meta/buildtools-tarball.bb       |   1 -
 meta/recipes-devtools/distcc/distcc_3.2.bb         |   2 +-
 ...rename-copy_file_range-to-copy_file_chunk.patch |  62 ++++
 meta/recipes-devtools/e2fsprogs/e2fsprogs_1.43.bb  |   1 +
 meta/recipes-devtools/qemu/qemu/memfd.patch        |  57 ++++
 meta/recipes-devtools/qemu/qemu_2.7.0.bb           |   1 +
 .../libvorbis/libvorbis/CVE-2017-14632.patch       |  62 ++++
 .../libvorbis/libvorbis/CVE-2017-14633.patch       |  42 +++
 .../libvorbis/libvorbis/CVE-2018-5146.patch        | 100 +++++++
 .../libvorbis/libvorbis_1.3.5.bb                   |   6 +-
 36 files changed, 809 insertions(+), 216 deletions(-)
 create mode 100644 meta/recipes-bsp/grub/files/0001-configure-fix-check-for-sys-sysmacros.h-under-glibc-.patch
 create mode 100644 meta/recipes-core/glibc/glibc/archive-path.patch
 create mode 100644 meta/recipes-core/glibc/glibc/relocate-locales.patch
 create mode 100644 meta/recipes-devtools/e2fsprogs/e2fsprogs/0001-misc-rename-copy_file_range-to-copy_file_chunk.patch
 create mode 100644 meta/recipes-devtools/qemu/qemu/memfd.patch
 create mode 100644 meta/recipes-multimedia/libvorbis/libvorbis/CVE-2017-14632.patch
 create mode 100644 meta/recipes-multimedia/libvorbis/libvorbis/CVE-2017-14633.patch
 create mode 100644 meta/recipes-multimedia/libvorbis/libvorbis/CVE-2018-5146.patch

Please consider these changes for Morty. Clean AB build

This included sdk locale and host glibc 2.27 fixes.

The following changes since commit 34b26ec18450a3bc72f7b8a0f2df0bfaae1140dd:

linux-firmware: Add reference to iwlwifi-8000C firmware (2018-03-19 15:58:33 +0000)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib stable/morty-next
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=stable/morty-next

Richard Purdie (4):
lib/oe/package_manager/sdk: Ensure do_populate_sdk_ext and
do_populate_sdk repos don't conflict
glibc: Separate locale files to their own sstate task

image/packagegroup/populate_sdk: Drop do_populate_sysroot task
properly
glibc-package: Avoid race sstate races with do_stash_locale

Is it safe to backport to morty, which doesn't have RSS?

Quoted 5 lines · Armin Kuster

Please consider these changes for Morty. Clean AB build

This included sdk locale and host glibc 2.27 fixes.

The following changes since commit 34b26ec18450a3bc72f7b8a0f2df0b

faae1140dd:

Quoted 2 lines

linux-firmware: Add reference to iwlwifi-8000C firmware (2018-03-19

15:58:33 +0000)

Quoted 5 lines

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib stable/morty-next
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=

stable/morty-next

Quoted 5 lines

Richard Purdie (4):
lib/oe/package_manager/sdk: Ensure do_populate_sdk_ext and
do_populate_sdk repos don't conflict
glibc: Separate locale files to their own sstate task

This commit comes with the comment:

Putting the locale and script files into the sysroot for use by their specific recipes used to be a simple way to share the files. With RSS, we don't want to copy these into many different recipes so put these files in their own sstate task.

Is it safe to backport to morty, which doesn't have RSS?

Quoted 3 lines

image/packagegroup/populate_sdk: Drop do_populate_sysroot task
properly
glibc-package: Avoid race sstate races with do_stash_locale

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Quoted 1 line

 Is it safe to backport to morty, which doesn't have RSS?

It doesn't work well for me, all the builds are now failing with:

ERROR: glibc-2.24-r0 do_stash_locale_setscene: The recipe glibc is trying to install files into a shared area when those files already exist. Those files and their manifest location are:
   TMPDIR/sysroots-components/aarch64/glibc-stash-locale/SUPPORTED Matched in b'' TMPDIR/sysroots-components/aarch64/glibc-stash- locale/lib/libnss_nisplus.so.2
 Matched in b'' TMPDIR/sysroots-components/aarch64/glibc-stash-locale/lib/ld- 2.24.so
 Matched in b''
...

Quoted 4 lines · Martin Jansa

Quoted 1 line

Is it safe to backport to morty, which doesn't have RSS?

It doesn't work well for me, all the builds are now failing with:

Quoted 12 lines

ERROR: glibc-2.24-r0 do_stash_locale_setscene: The recipe glibc is trying to install files into a shared area when those files already exist. Those files and their manifest location are:
TMPDIR/sysroots-components/aarch64/glibc-stash-locale/SUPPORTED
Matched in b''
TMPDIR/sysroots-components/aarch64/glibc-stash-
locale/lib/libnss_nisplus.so.2
Matched in b''
TMPDIR/sysroots-components/aarch64/glibc-stash-locale/lib/ld-
2.24.so
Matched in b''
...

I suspect using its own sstate task is ok in morty even if it didn't have RSS. What may have broken is the existing TMPDIR upgrade path though :(

If the files are removed, I suspect things will then work out ok. There were manifest handling changes in later versions which dealt with this kind of issue better (removals before installations).

Cheers,

Richard

In this build I was starting with empty TMPDIR, there shouldn't be any old files, but will retry manually in different builds.

Quoted 30 lines · Richard Purdie

Quoted 4 lines · Martin Jansa

Quoted 1 line

Is it safe to backport to morty, which doesn't have RSS?

It doesn't work well for me, all the builds are now failing with:

Quoted 12 lines

ERROR: glibc-2.24-r0 do_stash_locale_setscene: The recipe glibc is trying to install files into a shared area when those files already exist. Those files and their manifest location are:
TMPDIR/sysroots-components/aarch64/glibc-stash-locale/SUPPORTED
Matched in b''
TMPDIR/sysroots-components/aarch64/glibc-stash-
locale/lib/libnss_nisplus.so.2
Matched in b''
TMPDIR/sysroots-components/aarch64/glibc-stash-locale/lib/ld-
2.24.so
Matched in b''
...

I suspect using its own sstate task is ok in morty even if it didn't have RSS. What may have broken is the existing TMPDIR upgrade path though :(

If the files are removed, I suspect things will then work out ok. There were manifest handling changes in later versions which dealt with this kind of issue better (removals before installations).

Cheers,

Richard

I think I've found the reason why it was failing for me.

glibc is now the only component in morty release which is using TMPDIR/sysroots-components (normally used only after RSS):
oe-core$ git grep "\${STAGING_DIR}-components" meta/recipes-core/glibc/glibc-locale.inc:LOCALETREESRC = "${STAGING_DIR}-components/${PACKAGE_ARCH}/glibc-stash-locale" meta/recipes-core/glibc/glibc-mtrace.inc:SRC = "${STAGING_DIR}- components/${PACKAGE_ARCH}/glibc-stash-locale/scripts" meta/recipes-core/glibc/glibc-package.inc:do_stash_locale[sstate- outputdirs] = "${STAGING_DIR}-components/${PACKAGE_ARCH}/glibc-stash- locale" meta/recipes-core/glibc/glibc-package.inc:do_stash_locale[sstate- fixmedir] = "${STAGING_DIR}-components/${PACKAGE_ARCH}/glibc-stash- locale"
meta/recipes-core/glibc/glibc-scripts.inc:SRC = "${STAGING_DIR}- components/${PACKAGE_ARCH}/glibc-stash-locale/scripts"

and the workspace of my jenkins jobs is still running test- dependencies jobs for morty, which does only selective pruning of TMPDIR:
http://git.openembedded.org/openembedded-core/tree/scripts/test-depen
dencies.sh#n165

which left TMPDIR/sysroots-components files from qlibc.do_stash_locale behind and when next build started in the same workspace it failed, because it was trying to overwrite files already there.

Every time I wiped whole TMPDIR and restarted the jobs it worked for a while and then it got broken again during first test-dependencies job in each workspace.

I can obviously change test-dependencies.sh to
delete TMPDIR/sysroots-components as well and I'll send update for it after a bit more testing.

Ross's oe-core/scripts/wipe-sysroot and maybe some other scripts/tools need similar fix as well.