QEMU rcu stalls on autobuilder - not what they seem?
Quoted 24 lines · Richard Purdie via lists.openembedded.org
We have patches in master-next where we test enabling SMP for x86 and arm. This has resulted in a few build failures where qemu crashes with an RCU stall. The SMP output is different to our non-SMP case. For ages we've been blaming the autobuilder load/contention. The SMP case does appear to have it happen more regularly but it was happening before.
As an example, this morning I awoke to:
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2123/steps/14/logs/stdio
and I've pasted the full boot log below. It is easy to blame the autobuilder load as usual but I'd note that the kernel messages stop after about 1s after the tsc clocksource message.
I'd argue that the RCU stall message is just pointing out the kernel is mostly dead, its also interesting the RCU stall handler crashes before it can print the process traces.
I read that the tsc clocksource message hang is notorious as it is where the kernel starts initing other subsystems so the issue may not be in the tsc clocksource but elsewhere. I'd also note it is using the kvm-clocksource and just inited the tsc clocksource, it isn't using it. The TSC clocksource appears to be know it init slowly.
Quoted 3 lines
I think we may have some other kernel issue going on here and blaming the autobuilder load isn't the right path forward. Is there anyone who understands what may be going on here or a way we could debug it further?
To follow up, some qemu literate friends looked at this and asked why on earth we were using a machine type from 1996 and why such an old cpu which didn't support SMP and had a broken TSC. I think those are good questions. The current options we're using aren't tested by others. They also suggested some qemu boot options which if nothing else could speed things up.
It is time we changed the x86 qemu types so I've put a patch into master-next for testing moving to IvyBridge and the cpu options. I think these are changes we should be making anyway but I'd be very happy if the help the AB-INT issues which have been plaguing us.
Cheers,
Richard
A quick question: are the AB machines using nested kvm, or are they baremetal?
We've seen similar errors here with qemu SMP that were only resolved after moving to baremetal builds for qemu targets.
Alex
Quoted 68 lines · Richard Purdie
Quoted 1 line · Richard Purdie via lists.openembedded.org
We have patches in master-next where we test enabling SMP for x86 and
arm.
Quoted 8 lines
This has resulted in a few build failures where qemu crashes with an RCU stall. The SMP output is different to our non-SMP case. For ages we've been blaming the autobuilder load/contention. The SMP case does appear to have it happen more regularly but it was happening before.
As an example, this morning I awoke to:
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2123/steps/14/logs/stdio
Quoted 2 lines
and I've pasted the full boot log below. It is easy to blame the
autobuilder
Quoted 1 line
load as usual but I'd note that the kernel messages stop after about 1s
after
Quoted 3 lines
the tsc clocksource message.
I'd argue that the RCU stall message is just pointing out the kernel is
mostly
Quoted 1 line
dead, its also interesting the RCU stall handler crashes before it can
Quoted 3 lines
the process traces.
I read that the tsc clocksource message hang is notorious as it is where
the
Quoted 1 line
kernel starts initing other subsystems so the issue may not be in the
tsc
Quoted 1 line
clocksource but elsewhere. I'd also note it is using the kvm-clocksource
and just
Quoted 1 line
inited the tsc clocksource, it isn't using it. The TSC clocksource
appears to be
Quoted 3 lines
know it init slowly.
I think we may have some other kernel issue going on here and blaming
the
Quoted 1 line
autobuilder load isn't the right path forward. Is there anyone who
understands
Quoted 1 line
what may be going on here or a way we could debug it further?
To follow up, some qemu literate friends looked at this and asked why on earth we were using a machine type from 1996 and why such an old cpu which didn't
support SMP and had a broken TSC. I think those are good questions. The current
options we're using aren't tested by others. They also suggested some qemu boot
options which if nothing else could speed things up.It is time we changed the x86 qemu types so I've put a patch into master-next
for testing moving to IvyBridge and the cpu options. I think these are changes
we should be making anyway but I'd be very happy if the help the AB-INT issues
which have been plaguing us.Cheers,
Richard
Quoted 5 lines · Alexander Kanavin
A quick question: are the AB machines using nested kvm, or are they baremetal?
We've seen similar errors here with qemu SMP that were only resolved after moving to baremetal builds for qemu targets.
Baremetal so that is one thing we can rule out! :)
Cheers,
Richard