Skip to contentSkip to conversation

QEMU rcu stalls on autobuilder - not what they seem?

OE-core Richard Purdie 2 participants 3 messages May 11, 2021 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 24 lines · Richard Purdie via lists.openembedded.org

We have patches in master-next where we test enabling SMP for x86 and arm. This has resulted in a few build failures where qemu crashes with an RCU stall. The SMP output is different to our non-SMP case. For ages we've been blaming the autobuilder load/contention. The SMP case does appear to have it happen more regularly but it was happening before.

As an example, this morning I awoke to:

https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2123/steps/14/logs/stdio

and I've pasted the full boot log below. It is easy to blame the autobuilder load as usual but I'd note that the kernel messages stop after about 1s after the tsc clocksource message.

I'd argue that the RCU stall message is just pointing out the kernel is mostly dead, its also interesting the RCU stall handler crashes before it can print the process traces.

I read that the tsc clocksource message hang is notorious as it is where the kernel starts initing other subsystems so the issue may not be in the tsc clocksource but elsewhere. I'd also note it is using the kvm-clocksource and just inited the tsc clocksource, it isn't using it. The TSC clocksource appears to be know it init slowly.

Quoted 3 lines

I think we may have some other kernel issue going on here and blaming the autobuilder load isn't the right path forward. Is there anyone who understands what may be going on here or a way we could debug it further?

To follow up, some qemu literate friends looked at this and asked why on earth we were using a machine type from 1996 and why such an old cpu which didn't support SMP and had a broken TSC. I think those are good questions. The current options we're using aren't tested by others. They also suggested some qemu boot options which if nothing else could speed things up.

It is time we changed the x86 qemu types so I've put a patch into master-next for testing moving to IvyBridge and the cpu options. I think these are changes we should be making anyway but I'd be very happy if the help the AB-INT issues which have been plaguing us.

Cheers,

Richard

#2 lore raw

A quick question: are the AB machines using nested kvm, or are they baremetal?

We've seen similar errors here with qemu SMP that were only resolved after moving to baremetal builds for qemu targets.

Alex

Quoted 68 lines · Richard Purdie
Quoted 1 line · Richard Purdie via lists.openembedded.org

We have patches in master-next where we test enabling SMP for x86 and

arm.

Quoted 8 lines

This has resulted in a few build failures where qemu crashes with an RCU stall. The SMP output is different to our non-SMP case. For ages we've been blaming the autobuilder load/contention. The SMP case does appear to have it happen more regularly but it was happening before.

As an example, this morning I awoke to:

https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2123/steps/14/logs/stdio

Quoted 2 lines

and I've pasted the full boot log below. It is easy to blame the

autobuilder

Quoted 1 line

load as usual but I'd note that the kernel messages stop after about 1s

after

Quoted 3 lines

the tsc clocksource message.

I'd argue that the RCU stall message is just pointing out the kernel is

mostly

Quoted 1 line

dead, its also interesting the RCU stall handler crashes before it can

print

Quoted 3 lines

the process traces.

I read that the tsc clocksource message hang is notorious as it is where

the

Quoted 1 line

kernel starts initing other subsystems so the issue may not be in the

tsc

Quoted 1 line

clocksource but elsewhere. I'd also note it is using the kvm-clocksource

and just

Quoted 1 line

inited the tsc clocksource, it isn't using it. The TSC clocksource

appears to be

Quoted 3 lines

know it init slowly.

I think we may have some other kernel issue going on here and blaming

the

Quoted 1 line

autobuilder load isn't the right path forward. Is there anyone who

understands

Quoted 1 line

what may be going on here or a way we could debug it further?

To follow up, some qemu literate friends looked at this and asked why on earth we were using a machine type from 1996 and why such an old cpu which didn't
support SMP and had a broken TSC. I think those are good questions. The current
options we're using aren't tested by others. They also suggested some qemu boot
options which if nothing else could speed things up.

It is time we changed the x86 qemu types so I've put a patch into master-next
for testing moving to IvyBridge and the cpu options. I think these are changes
we should be making anyway but I'd be very happy if the help the AB-INT issues
which have been plaguing us.

Cheers,

Richard

#3 lore raw
Quoted 5 lines · Alexander Kanavin

A quick question: are the AB machines using nested kvm, or are they baremetal?

We've seen similar errors here with qemu SMP that were only resolved after moving to baremetal builds for qemu targets.

Baremetal so that is one thing we can rule out! :)

Cheers,

Richard