Remaining issues detected with toolchain-builders

Hello,

As part of preparing new pre-built toolchains based on Buildroot 2024.05 (hosted at toolchains.bootlin.com), I'm encountering a number of issues, which I am summarizing in this e-mail.

The toolchains and tests are built based on the Buildroot branch at:

https://github.com/bootlin/buildroot-toolchains/tree/toolchains.bootlin.com-2024.05

Which is basically 2024.05 + a number of backports and a small number of extra fixes.

Based on this, for each architecture/libc configuration, a toolchain is built, packed as a tarball, and is then used as an external toolchain to build a kernel+rootfs that is booted under Qemu (using our Qemu defconfigs, but to use an external toolchain instead of the internal toolchain).

sh-sh4
======

https://gitlab.com/buildroot.org/toolchains-builder/-/pipelines/1391146782

All toolchains built fine, but the test system did not boot in any of the toolchain combinations.

In the sh-sh4--glibc--bleeding-edge_test
(https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/7446074815),
Qemu complained with:

**
ERROR:../hw/intc/sh_intc.c:168:sh_intc_locate: code should not be reached Bail out! ERROR:../hw/intc/sh_intc.c:168:sh_intc_locate: code should not be reached Aborted (core dumped)
Connection problem, exiting.

With all other toolchains, there were no messages at all from Qemu:

https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/7446074827
https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/7446074831
https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/7446074832
https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/7446074833
https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/7446074834

One point worth mentioning is that the tests were all sucessful for the sh-sh4aeb toolchains:

https://gitlab.com/buildroot.org/toolchains-builder/-/pipelines/1379541813

riscv32-ilp32d
==============

Here, only the riscv32-ilp32d--uclibc--stable_test fails to boot:

https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/7446076680

This has already been reported on the list, and an initial investigation has been done by Waldemar. It seems like it could be a time64 issue: the uClibc stable toolchain uses 4.19 kernel headers (which predate time64) and generates a broken system, while the uClibc bleeding edge toolchain, which uses 5.4 kernel headers, generates a working system.

mips32r5el
==========

For this architecture, what we do is we use the qemu_mips32r2el_malta defconfig, but we build it for MIPS32R5, and we boot it under Qemu with the "-cpu P5600" argument, which is a MIPS32R5 CPU.

And all toolchains generate a system that fails to boot:

Run /sbin/init as init process
process '/bin/busybox' started with executable stack

Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000004

mips32r5el--glibc--bleeding-edge_test
https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/7377823322
mips32r5el--glibc--stable_test
https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/7377823324

mips32r5el--musl--bleeding-edge_test
https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/7377823325
mips32r5el--musl--stable_test
https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/7377823326

mips32r5el--uclibc--bleeding-edge_test
https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/7377823327
mips32r5el--uclibc--stable_test
https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/7377823329

The interesting thing is that those images boot fine with Qemu v8.2.6, but they fail to boot with Qemu v9.0.2.

Is it a regression in Qemu, or something that we were doing that was working by chance and that is now better emulated and in fact not correct?

m68k-68xxx
==========

See https://gitlab.com/buildroot.org/toolchains-builder/-/pipelines/1379452858

The stable toolchains are working fine, but the bleeding edge configurations are generating systems that don't boot.

With glibc:
https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/7377699056

mmap: kernel BUG at mm/mmap.c:3259!
*** TRAP #7 *** FORMAT=0
Current process id is 41
BAD KERNEL TRAP: 00000000

Modules linked in:
PC: [<000a3c22>] insert_vm_struct+0x0/0xee
SR: 2000  SP: cecacd44  a2: 00ab49c0
d0: 00000023    d1: 00000000    d2: 00000008    d3: 00000048
d4: 000a1560    d5: 00a3e360    a0: 00abfd1c    a1: 004b93c6
Process rcS (pid: 41, task=d2eb12e9)
Frame format=0 
Stack from 00abfd98:
        00468a6f 00468a57 00000cbb 00000007 00abe0ff 400004d8 0000001e 00000006
        00000000 00a3e360 00ab49c0 00a3e390 00abfe78 400000a3 e3640000 0001ffff
        ffff00a5 af0c0000 0000ffff ffff0000 0000010a 000000a3 e360ffff ffffffff
        ffff8000 00000001 00abfe04 00000000 00000000 00000008 005d8ef8 005d8c70
        005d8e44 005d8d00 005d97f8 005d99cc 005d9a14 005d99f0 00016bea 00a3e360
        00a3e360 00a3e360 0001b8aa 00a3e360 00a3e390 00000007 00abe000 400004d8
Call Trace: [<0001ffff>] cap_validate_magic+0x5b/0x114
 [<0000ffff>] tblpre+0x28f/0x700
 [<00016bea>] mmput+0x24/0x98
 [<0001b8aa>] do_exit+0x1ea/0x8e8
 [<0001c14e>] do_group_exit+0x1e/0x80
 [<00025e02>] get_signal+0x782/0x7a6
 [<00002010>] initcall_blacklisted+0x8/0x98
 [<000043c4>] do_notify_resume+0x70/0x578
 [<00024a6c>] force_sig_info_to_task+0xc0/0x152
 [<00024d1e>] force_sig_fault+0x36/0x3e
 [<00002738>] do_signal_return+0x10/0x1a
 [<0010c00f>] sys_inotify_add_watch+0x3/0x3cc

With musl:
https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/7377699062

Run /sbin/init as init process
System did not boot in time, exiting.

With uClibc:
https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/7377699066

Run /sbin/init as init process
EXT4-fs (sda): re-mounted 8565f0ac-7559-4bf5-ad8c-bee4703f59fb r/w. Quota mode: disabled. Segmentation fault
Segmentation fault
Segmentation fault
Segmentation fault
Segmentation fault
Segmentation fault

arcle-750d
==========

Here we don't have a runtime failure, but a build time failure. I have already reported it to Synopsys people, who are investigating.

arcle-750d--uclibc--bleeding-edge_build
https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/7377622441

/builds/buildroot.org/toolchains-builder/build/arcle-750d--uclibc--bleeding-edge-2024.05-1/lib/gcc/arc-buildroot-linux-uclibc/14.1.0/../../../../arc -buildroot-linux-uclibc/bin/ld: /builds/buildroot.org/toolchains-builder/build/arcle-750d--uclibc--bleeding-edge-2024.05-1/lib/gcc/arc-buildroot-lin ux-uclibc/14.1.0/../../../../arc-buildroot-linux-uclibc/lib/libstdc++.so: undefined reference to `__atomic_test_and_set'
collect2: error: ld returned 1 exit status

I would definitely appreciate support in investigating/solving some of those issues. Thanks in advance for your help!

Thomas

Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot