Status of M3

I'm not sure people realise quite how much pain we've been suffering this week trying to get things stabilised for M3. To illustrate the kinds of problems, let me give an idea of the issues in the past few days. The resolved list:

* gobject-introspection has sstate relocation issues
* gobject-introspection was missing a dependency
* allarch contamination issues from gnomebase defaulting to g-i
* gobject-introspection fails on x32
* oe-selftest failures from the vm class changes
* random createrepo issue
* autobuilder workers were replaced with new distros
- one had firewall problems
- two had network interface problems
- another had VNC issues causing sanity test failures
* there were autobuilder configuration changes
- eSDK changes failed initial
- increased ptest coverage failed initially
- uninative tarball publishing wasn't correct
- handle meta-poky transition
- handle adt-installer removal
* uninative output name issues (BUILD_ARCH verses SDK_ARCH)
* uninative sstate interaction issues (NATIVESDKSTRING)
* ongoing pseudo retry issues
* bitbake unpack improvements broke
* canterall fonts broke oe-selftest
* unsafe script references broke sanity tests
* unsafe binary references broke in sanity tests due to prelink problem
* meta-yocto -> meta-poky had multiple problems
* race in do_rootfs_wicenv
* eudev change broke oe-selftest
* rpm upgrade went through multiple different build failures
* toaster references to meta-yocto
* I screwed up manually fixing a simple merge breaking builds. Twice :(

Things which still break:

* rpm upgrade causes smart remove to not function
* gobject-introspection breaks on multilib with python-pygobject file location issue
* gobject-introspection fails on musl
* createrepo has occasional failure with checksum mismatch
* oe-selftest signing failure
* sato application launch failures from glib issues due to prelink
* meta-fsl-ppc breaks on eudev change (patch pending)
* meta-fsl-ppc breakage blocks AB artefact publishing

There are only a small number of us who dive in and try and untangle the twisted web of which patch is causing which issue and try and put these things on a path to resolution

With this level of issues, we're simply not able to consider things like "why aren't you testing X?" or "can you test this patch to this component to get debugging?". There are 101 things that many of us would love to do but we need to improve the turnaround time of the tests we have. There are some simple things that come to mind that we could do:

a) optimisation of oe-selftest. Currently it takes around 8 hours but we could cut that time massively with some optimisation around the sstate cache and the way the tests are written. This one does catch many valid issues but its way too slow. Parallelism is also an option here.

b) switch to uninative by default to improve native/cross artefact reuse on the autobuilder. I have patches queued in -next to test this, see if we could switch to it.

c) switch to the new AB cluster when its ready (newer/faster hardware)

Volunteers for a) would be most welcome, other ideas welcome too.

Cheers,

Richard

We're expecting the new AB cluster to be available this week. When it is, I'll need a day to run it through the paces and make sure everything is ok, but, expect this to be available around the 7th.

Elizabeth Flanagan
Yocto Project
Build and Release

Hi Richard,

[Luo Zhenhua-B19537] The eudev patch is merged, http://git.yoctoproject.org/cgit/cgit.cgi/meta-fsl-ppc/commit/?id=2642cf5e8a6f8d11603acf016b8c075ebce00ec0, is there any other blocker issue?

Best Regards,

Zhenhua

Thanks, I appreciate getting that sorted!

Unfortunately there is something else not quite right with meta-fsl -ppc:

https://autobuilder.yoctoproject.org/main/builders/nightly-fsl-ppc/buil
ds/667

Seems something is not working in python3 do_configure? Everything else is building python3 fine so not sure what is wrong there.

Cheers,

Richard

Hi Richard,

In the log, we have do_configure failed on python3:
...

| checking for the platform triplet based on compiler characteristics... powerpc-linux-gnuspe
| configure: error: internal configure error for the platform triplet, please file a bug report
| NOTE: The following config.log files may provide further information.

...

vim configure.ac in python3 source
...
852 if test x$PLATFORM_TRIPLET != x && test x$MULTIARCH != x; then
853 if test x$PLATFORM_TRIPLET != x$MULTIARCH; then
854 AC_MSG_ERROR([internal configure error for the platform
triplet, please file a bug report])
855 fi
856 fi
...

In this case, PLATFORM_TRIPLET is powerpc-linux-gnuspe, and MULTIARCH is MULTIARCH=$($CC --print-multiarch 2>/dev/null), and both of them are not equivalent.

On my local build with MACHINE = "qemux86-64", the MULTIARCH is empty.

On my local build with MACHINE = "p1022ds", $CC --print-multiarch powerpc-linux-gnuspev1

Extra 'v1' casued the failure.

I am going to see what happen on 'v1', file [YOCTO #9226] to trace it

//Hongxu

A patch is submitted to fix the issue.

http://patchwork.openembedded.org/patch/117783/

Best Regards,

Zhenhua

From: Hongxu Jia [mailto:[email protected]]
Sent: Tuesday, March 08, 2016 4:09 PM
To: Richard Purdie <[email protected]>; Zhenhua Luo <[email protected]>
Cc: openembedded-core <[email protected]>
Subject: Re: [OE-core] Status of M3

On 03/04/2016 06:18 PM, Richard Purdie wrote:

On Fri, 2016-03-04 at 09:41 +0000, Zhenhua Luo wrote:

* meta-fsl-ppc breaks on eudev change (patch pending)

* meta-fsl-ppc breakage blocks AB artefact publishing

[Luo Zhenhua-B19537] The eudev patch is merged, http://git.yoctoproje

ct.org/cgit/cgit.cgi/meta-fsl

-ppc/commit/?id=2642cf5e8a6f8d11603acf016b8c075ebce00ec0, is there

any other blocker issue?

Thanks, I appreciate getting that sorted!

Unfortunately there is something else not quite right with meta-fsl

-ppc:

https://autobuilder.yoctoproject.org/main/builders/nightly-fsl-ppc/buil

ds/667

Hi Richard,

In the log, we have do_configure failed on python3:
...

| checking for the platform triplet based on compiler characteristics... powerpc-linux-gnuspe
| configure: error: internal configure error for the platform triplet, please file a bug report
| NOTE: The following config.log files may provide further information.

...

vim configure.ac in python3 source
...
852 if test x$PLATFORM_TRIPLET != x && test x$MULTIARCH != x; then
853 if test x$PLATFORM_TRIPLET != x$MULTIARCH; then
854 AC_MSG_ERROR([internal configure error for the platform triplet, please file a bug report])
855 fi
856 fi
...

In this case, PLATFORM_TRIPLET is powerpc-linux-gnuspe, and MULTIARCH is MULTIARCH=$($CC --print-multiarch 2>/dev/null), and both of them are not equivalent.

On my local build with MACHINE = "qemux86-64", the MULTIARCH is empty.

On my local build with MACHINE = "p1022ds", $CC --print-multiarch powerpc-linux-gnuspev1

Extra 'v1' casued the failure.

I am going to see what happen on 'v1', file [YOCTO #9226] to trace it

//Hongxu

Seems something is not working in python3 do_configure? Everything else

is building python3 fine so not sure what is wrong there.

Cheers,

Richard

I am just aware that another patch was already merged to fix the issue, please ignore my patch.

Best Regards,

Zhenhua

From: [email protected] [mailto:[email protected]] On Behalf Of Zhenhua Luo
Sent: Monday, March 14, 2016 2:12 PM
To: Hongxu Jia <[email protected]>; Richard Purdie <[email protected]>
Cc: openembedded-core <[email protected]>
Subject: Re: [OE-core] Status of M3

A patch is submitted to fix the issue.

http://patchwork.openembedded.org/patch/117783/

Best Regards,

Zhenhua

From: Hongxu Jia [mailto:[email protected]]
Sent: Tuesday, March 08, 2016 4:09 PM
To: Richard Purdie <[email protected]<mailto:[email protected]>>; Zhenhua Luo <[email protected]<mailto:[email protected]>>
Cc: openembedded-core <[email protected]<mailto:[email protected]>>
Subject: Re: [OE-core] Status of M3

On 03/04/2016 06:18 PM, Richard Purdie wrote:

On Fri, 2016-03-04 at 09:41 +0000, Zhenhua Luo wrote:

* meta-fsl-ppc breaks on eudev change (patch pending)

* meta-fsl-ppc breakage blocks AB artefact publishing

[Luo Zhenhua-B19537] The eudev patch is merged, http://git.yoctoproje

ct.org/cgit/cgit.cgi/meta-fsl

-ppc/commit/?id=2642cf5e8a6f8d11603acf016b8c075ebce00ec0, is there

any other blocker issue?

Thanks, I appreciate getting that sorted!

Unfortunately there is something else not quite right with meta-fsl

-ppc:

https://autobuilder.yoctoproject.org/main/builders/nightly-fsl-ppc/buil

ds/667

Hi Richard,

In the log, we have do_configure failed on python3:
...

| checking for the platform triplet based on compiler characteristics... powerpc-linux-gnuspe
| configure: error: internal configure error for the platform triplet, please file a bug report
| NOTE: The following config.log files may provide further information.

...

vim configure.ac in python3 source
...
852 if test x$PLATFORM_TRIPLET != x && test x$MULTIARCH != x; then
853 if test x$PLATFORM_TRIPLET != x$MULTIARCH; then
854 AC_MSG_ERROR([internal configure error for the platform triplet, please file a bug report])
855 fi
856 fi
...

In this case, PLATFORM_TRIPLET is powerpc-linux-gnuspe, and MULTIARCH is MULTIARCH=$($CC --print-multiarch 2>/dev/null), and both of them are not equivalent.

On my local build with MACHINE = "qemux86-64", the MULTIARCH is empty.

On my local build with MACHINE = "p1022ds", $CC --print-multiarch powerpc-linux-gnuspev1

Extra 'v1' casued the failure.

I am going to see what happen on 'v1', file [YOCTO #9226] to trace it

//Hongxu

Seems something is not working in python3 do_configure? Everything else

is building python3 fine so not sure what is wrong there.

Cheers,

Richard

Most of the above are still broken. In addition, I thought uninative was going too well, turns out we broke its enablement on the autobuilders. With that "fixed" we've seen some additional issues (and some new 'random' race failures):

* a gcc4 verses gcc5 conflict in the libstdc++ ABI (uninative)
* a codepage issue with mcopy (uninative)
* an additional gobject-introspection dependency problem (simple?)
* parallel make race in ncurses make install
* parallel make race in openssl (borrow patches from other distro?)
* some xmlto execution issue

So work remains to get M3 into shape. Not sure how much of my weekend I'm planning to spend in trying to sort this out...

Cheers,

Richard

On 4 March 2016 at 22:51, Richard Purdie <[email protected]

Assuming my build test passes, expect a patch for this one shortly.

Ross

On 4 March 2016 at 22:51, Richard Purdie <[email protected]

/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-ppc-lsb/build/build/tmp/sysroots/x86_64-linux/usr/bin/xmlto.real:
Bad substitution

This is a bashism in a /bin/sh script. Patch incoming!

Ross

On 4 March 2016 at 22:51, Richard Purdie <[email protected]

My hunch is that we need to add the utf8 gconv module to the tarball?

Ross

My hunch was probably wrong.

stracing the mcopy shows that it's trying to open the gconv files from the proper native sysroot and not the uninative sysroot, but the files don't exist in the proper sysroot at all as they're meant to be in the uninative one. We also strip the uninative sysroot right down to just one gconv module which we probably don't want to do now we're trying to use it for more than just eSDK but it does contain the index at least.

(The strace says this:

13648 open("/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86/build/build/tmp/sysroots/x86_64-linux/usr/lib/gconv/gconv-modules", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
)

Presumably it's looking in $libdir which seems reasonable, so I'm not sure how best to solve this. Remove gconf from uninative and generate it as part of the native sysroot?

Anyway it's now gone 1am so I think I'll stop now...

Ross