Sanity Failures - Segfaults in qemu images

We're coming up to release however we're struggling with various sanity test failures that keep showing up on the autobuilder.

A lot of them have been caused by issues in the qemu scripts and the fact that the systems are being asked to do more in parallel due to the new autobuilder infrastructure. I believe we have these ones resolved now.

The ones that worry me are like two that happened in the last build for example:

http://autobuilder.yoctoproject.org:8011/builders/nightly-arm-lsb/builds/95/steps/Running%20Sanity%20Tests_1/logs/stdio
http://autobuilder.yoctoproject.org:8011/builders/nightly-x86-64-lsb/builds/87/steps/Running%20Sanity%20Tests/logs/stdio

In both cases we have a segfault happening in the guest, one directly triggered by a sanity test, the other being detected in dmesg.

We saw one of these on the previous build:

http://autobuilder.yoctoproject.org:8011/builders/nightly-x86/builds/92/steps/Running%20Sanity%20Tests/logs/stdio
(ignore the minimal failure, that was likely a timeout issue, resolved by a recent change)

I've also seen the smart help segfault on a qemumips image. I did download that one locally and saw the same fault the first time I booted it. I then didn't see it again, despite running the image many times. The booting was of a copy of the image so it wasn't a first boot issue. The checksum matched that on the autobuilder.

At this point I think it may well be a qemu issue but we don't know that for sure. I've not seen any report of this on real hardware.

The question is how do we debug this? Does anyone have any ideas?

The best idea I've heard so far is to generate a coredump in the image and save that off, maybe it would give some clue in later analysis. We could also upon failure move the actually booted somewhere for later analysis. I wondered if we could save off the qemu state too somehow. The trouble is none of these are simple coming up to release.

So if anyone has any ideas on what is causing this of how to debug/fix it, I'd be very receptive to them.

Cheers,

Richard

what does complete dmesg output looks like ? I have seen ld.so segfaults on real x86_64 hardware but havent narrowed it down since
it does not seem to bother my testing.​

I don't have the full dmesg output however I have put a wealth of info about this into:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=4216

Basically these do look like "random" segfaults happening any point on the system and having a variety of effects. The double fault is particularly worrying as it suggests the problem is kernel or qemu. I've put a script which tends to reproduce the issue into the bugzilla.

When you say you've seen ld.so faults on real hardware, have you any more details? Are they happening often? Just ld.so?

I need to know if this is a qemu issue or on real hardware too since the latter is a release blocker and extremely serious.

Cheers,

Richard