Skip to contentSkip to conversation

RCU stall with high number of KVM vcpus

DISCUSSION Marc Zyngier 4 participants 11 messages Nov 13, 2017 – Nov 14, 2017 1 message not in the archive KVM

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): 20171113131000.GA10546@hc

#1 lore raw
Quoted 2 lines · Jan Glauber

I'm seeing RCU stalls in the host with 4.14 when I run KVM on ARM64 (ThunderX2) with a high number of vcpus (60). I only use one guest that does kernel compiles in

Is that only reproducible on 4.14? With or without VHE? Can you reproduce this on another implementation (such as ThunderX-1)?

Quoted 8 lines

a loop. After some hours (less likely the more debugging options are enabled, more likely with more vcpus) RCU stalls are happening in both host & guest.

Both host & guest recover after some time, until the issue is triggered again.

Stack traces in the guest are next to useless, everything is messed up there. The host seems to stave on kvm->mmu_lock spin lock, the lock_stat

Please elaborate. Messed in what way? Corrupted? The guest crashing? Or is that a tooling issue?

Quoted 18 lines

numbers don't look good, see waittime-max:

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
class name con-bounces contentions waittime-min waittime-max waittime-total waittime-avg acq-bounces acquisitions holdtime-min holdtime-max holdtime-total holdtime-avg
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

                &(&kvm->mmu_lock)->rlock:      99346764       99406604           0.14  1321260806.59 710654434972.0        7148.97      154228320      225122857           0.13   917688890.60  3705916481.39          16.46
                ------------------------
                &(&kvm->mmu_lock)->rlock       99365598          [<ffff0000080b43b8>] kvm_handle_guest_abort+0x4c0/0x950
                &(&kvm->mmu_lock)->rlock          25164          [<ffff0000080a4e30>] kvm_mmu_notifier_invalidate_range_start+0x70/0xe8
                &(&kvm->mmu_lock)->rlock          14934          [<ffff0000080a7eec>] kvm_mmu_notifier_invalidate_range_end+0x24/0x68
                &(&kvm->mmu_lock)->rlock            908          [<ffff00000810a1f0>] __cond_resched_lock+0x68/0xb8
                ------------------------
                &(&kvm->mmu_lock)->rlock              3          [<ffff0000080b34c8>] stage2_flush_vm+0x60/0xd8
                &(&kvm->mmu_lock)->rlock       99186296          [<ffff0000080b43b8>] kvm_handle_guest_abort+0x4c0/0x950
                &(&kvm->mmu_lock)->rlock         179238          [<ffff0000080a4e30>] kvm_mmu_notifier_invalidate_range_start+0x70/0xe8
                &(&kvm->mmu_lock)->rlock          19181          [<ffff0000080a7eec>] kvm_mmu_notifier_invalidate_range_end+0x24/0x68
Quoted 1 line

.............................................................................................................................................................................................................................

[slots of stuff]

Well, the mmu_lock is clearly contended. Is the box in a state where you are swapping? There seem to be as many faults as contentions, which is a bit surprising...

Also, we recently moved arm64 to qrwlocks, which may have an impact. Care to give this[1] a go and report the result?

Thanks,

	M.

[1]: https://lkml.org/lkml/2017/10/12/266

Signature · 1 line
Jazz is not dead. It just smells funny...
#2 lore raw
Quoted 6 lines · Marc Zyngier
Quoted 2 lines · Jan Glauber

I'm seeing RCU stalls in the host with 4.14 when I run KVM on ARM64 (ThunderX2) with a high number of vcpus (60). I only use one guest that does kernel compiles in

Is that only reproducible on 4.14? With or without VHE? Can you reproduce this on another implementation (such as ThunderX-1)?

I've reproduced it on a distro 4.13 and several vanilla 4.14 rc's and tip/locking. VHE is enabled. I've not yet tried to reproduce it with older kernels or ThunderX-1. I can check if it happens also on ThunderX-1.

Quoted 11 lines
Quoted 8 lines

a loop. After some hours (less likely the more debugging options are enabled, more likely with more vcpus) RCU stalls are happening in both host & guest.

Both host & guest recover after some time, until the issue is triggered again.

Stack traces in the guest are next to useless, everything is messed up there. The host seems to stave on kvm->mmu_lock spin lock, the lock_stat

Please elaborate. Messed in what way? Corrupted? The guest crashing? Or is that a tooling issue?

Every vcpu that oopses prints one line in parallel, so I get blocks like:

[58880.179814] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179834] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179847] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179873] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179893] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179911] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179917] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180288] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180303] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180336] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180363] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180384] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180415] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180461] [<ffff000008084b98>] ret_from_fork+0x10/0x18

I can send the full log if you want to have a look.

Quoted 18 lines
Quoted 18 lines

numbers don't look good, see waittime-max:

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
class name con-bounces contentions waittime-min waittime-max waittime-total waittime-avg acq-bounces acquisitions holdtime-min holdtime-max holdtime-total holdtime-avg
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

                &(&kvm->mmu_lock)->rlock:      99346764       99406604           0.14  1321260806.59 710654434972.0        7148.97      154228320      225122857           0.13   917688890.60  3705916481.39          16.46
                ------------------------
                &(&kvm->mmu_lock)->rlock       99365598          [<ffff0000080b43b8>] kvm_handle_guest_abort+0x4c0/0x950
                &(&kvm->mmu_lock)->rlock          25164          [<ffff0000080a4e30>] kvm_mmu_notifier_invalidate_range_start+0x70/0xe8
                &(&kvm->mmu_lock)->rlock          14934          [<ffff0000080a7eec>] kvm_mmu_notifier_invalidate_range_end+0x24/0x68
                &(&kvm->mmu_lock)->rlock            908          [<ffff00000810a1f0>] __cond_resched_lock+0x68/0xb8
                ------------------------
                &(&kvm->mmu_lock)->rlock              3          [<ffff0000080b34c8>] stage2_flush_vm+0x60/0xd8
                &(&kvm->mmu_lock)->rlock       99186296          [<ffff0000080b43b8>] kvm_handle_guest_abort+0x4c0/0x950
                &(&kvm->mmu_lock)->rlock         179238          [<ffff0000080a4e30>] kvm_mmu_notifier_invalidate_range_start+0x70/0xe8
                &(&kvm->mmu_lock)->rlock          19181          [<ffff0000080a7eec>] kvm_mmu_notifier_invalidate_range_end+0x24/0x68
Quoted 6 lines
Quoted 1 line

.............................................................................................................................................................................................................................

[slots of stuff]

Well, the mmu_lock is clearly contended. Is the box in a state where you are swapping? There seem to be as many faults as contentions, which is a bit surprising...

I don't think it is swapping but need to double check.

Quoted 2 lines

Also, we recently moved arm64 to qrwlocks, which may have an impact. Care to give this[1] a go and report the result?

Sure, I that was my first suspision but I can reproduce the issue with and without the qrwlock patches, so these are not to blame. Also, the starving lock is a spinlock and not a qrwlock. So maybe the spinlocks have fairness issues too?

--Jan

Quoted 7 lines

Thanks,

	M.

[1]: https://lkml.org/lkml/2017/10/12/266

Signature · 1 line
Jazz is not dead. It just smells funny...
#3 lore raw
Quoted 11 lines · Jan Glauber
Quoted 6 lines · Marc Zyngier
Quoted 2 lines · Jan Glauber

I'm seeing RCU stalls in the host with 4.14 when I run KVM on ARM64 (ThunderX2) with a high number of vcpus (60). I only use one guest that does kernel compiles in

Is that only reproducible on 4.14? With or without VHE? Can you reproduce this on another implementation (such as ThunderX-1)?

I've reproduced it on a distro 4.13 and several vanilla 4.14 rc's and tip/locking. VHE is enabled. I've not yet tried to reproduce it with older kernels or ThunderX-1. I can check if it happens also on ThunderX-1.

That'd be interesting. Also, what happens if you turn off any form of multi-threading (as I've been told this system has 4 threads per core)? Disabling VHE would be interesting too (after all, this is the only box I know of that has the feature...).

Quoted 28 lines
Quoted 11 lines
Quoted 8 lines

a loop. After some hours (less likely the more debugging options are enabled, more likely with more vcpus) RCU stalls are happening in both host & guest.

Both host & guest recover after some time, until the issue is triggered again.

Stack traces in the guest are next to useless, everything is messed up there. The host seems to stave on kvm->mmu_lock spin lock, the lock_stat

Please elaborate. Messed in what way? Corrupted? The guest crashing? Or is that a tooling issue?

Every vcpu that oopses prints one line in parallel, so I get blocks like:

[58880.179814] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179834] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179847] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179873] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179893] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179911] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179917] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180288] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180303] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180336] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180363] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180384] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180415] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180461] [<ffff000008084b98>] ret_from_fork+0x10/0x18
Quoted 1 line

I can send the full log if you want to have a look.

Sure, send that over (maybe not over email though).

Quoted 19 lines
Quoted 19 lines
Quoted 19 lines

numbers don't look good, see waittime-max:

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
class name con-bounces contentions waittime-min waittime-max waittime-total waittime-avg acq-bounces acquisitions holdtime-min holdtime-max holdtime-total holdtime-avg
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

                &(&kvm->mmu_lock)->rlock:      99346764       99406604           0.14  1321260806.59 710654434972.0        7148.97      154228320      225122857           0.13   917688890.60  3705916481.39          16.46
                ------------------------
                &(&kvm->mmu_lock)->rlock       99365598          [<ffff0000080b43b8>] kvm_handle_guest_abort+0x4c0/0x950
                &(&kvm->mmu_lock)->rlock          25164          [<ffff0000080a4e30>] kvm_mmu_notifier_invalidate_range_start+0x70/0xe8
                &(&kvm->mmu_lock)->rlock          14934          [<ffff0000080a7eec>] kvm_mmu_notifier_invalidate_range_end+0x24/0x68
                &(&kvm->mmu_lock)->rlock            908          [<ffff00000810a1f0>] __cond_resched_lock+0x68/0xb8
                ------------------------
                &(&kvm->mmu_lock)->rlock              3          [<ffff0000080b34c8>] stage2_flush_vm+0x60/0xd8
                &(&kvm->mmu_lock)->rlock       99186296          [<ffff0000080b43b8>] kvm_handle_guest_abort+0x4c0/0x950
                &(&kvm->mmu_lock)->rlock         179238          [<ffff0000080a4e30>] kvm_mmu_notifier_invalidate_range_start+0x70/0xe8
                &(&kvm->mmu_lock)->rlock          19181          [<ffff0000080a7eec>] kvm_mmu_notifier_invalidate_range_end+0x24/0x68
Quoted 8 lines
Quoted 6 lines
Quoted 1 line

.............................................................................................................................................................................................................................

[slots of stuff]

Well, the mmu_lock is clearly contended. Is the box in a state where you are swapping? There seem to be as many faults as contentions, which is a bit surprising...

I don't think it is swapping but need to double check.

It is the number of aborts that is staggering. And each one of them leads to the mmu_lock being contended. So something seems to be taking its sweet time holding the damned lock.

Quoted 8 lines
Quoted 3 lines

Also, we recently moved arm64 to qrwlocks, which may have an impact. Care to give this[1] a go and report the result?

Sure, I that was my first suspision but I can reproduce the issue with and without the qrwlock patches, so these are not to blame. Also, the starving lock is a spinlock and not a qrwlock. So maybe the spinlocks have fairness issues too?

We use ticket locks, so they should be fine on the host (it is a different story altogether in a guest).

Can you send me your kernel configuration as well? I wonder if we're not seeing interactions with things like KSM and the like...

Thanks,

	M.
Signature · 1 line
Jazz is not dead. It just smells funny...
#4 lore raw
Quoted 2 lines · Marc Zyngier
Quoted 1 line · Jan Glauber

On Mon, Nov 13, 2017 at 01:47:38PM +0000, Marc Zyngier wrote:

[...]

Quoted 19 lines
Quoted 19 lines
Quoted 2 lines

Please elaborate. Messed in what way? Corrupted? The guest crashing? Or is that a tooling issue?

Every vcpu that oopses prints one line in parallel, so I get blocks like:

[58880.179814] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179834] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179847] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179873] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179893] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179911] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179917] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180288] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180303] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180336] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180363] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180384] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180415] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180461] [<ffff000008084b98>] ret_from_fork+0x10/0x18
Quoted 3 lines
Quoted 1 line

I can send the full log if you want to have a look.

Sure, send that over (maybe not over email though).

Here is the guest dmesg:
http://paste.ubuntu.com/25955682/

And the host dmesg as it might have been too big for the lists:
http://paste.ubuntu.com/25955699/

Quoted 2 lines

Can you send me your kernel configuration as well? I wonder if we're not seeing interactions with things like KSM and the like...

It was plain arm64 defconfig but I disabled HAVE_KVM_CPU_RELAX_INTERCEPT. Not that it had an impact, just tried.

http://paste.ubuntu.com/25955726/

--Jan

#5 lore raw
Quoted 25 lines · Jan Glauber
Quoted 2 lines · Marc Zyngier
Quoted 1 line · Jan Glauber

On Mon, Nov 13, 2017 at 01:47:38PM +0000, Marc Zyngier wrote:

[...]

Quoted 19 lines
Quoted 19 lines
Quoted 2 lines

Please elaborate. Messed in what way? Corrupted? The guest crashing? Or is that a tooling issue?

Every vcpu that oopses prints one line in parallel, so I get blocks like:

[58880.179814] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179834] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179847] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179873] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179893] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179911] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179917] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180288] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180303] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180336] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180363] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180384] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180415] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180461] [<ffff000008084b98>] ret_from_fork+0x10/0x18
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

I can send the full log if you want to have a look.

Sure, send that over (maybe not over email though).

Here is the guest dmesg:
http://paste.ubuntu.com/25955682/

Yeah, that's because all the vcpus are getting starved at the same time, and spitting out interleaved traces... Not very useful anyway, as I think this is only a consequence of what's happening on the host.

Quoted 3 lines

And the host dmesg as it might have been too big for the lists:
http://paste.ubuntu.com/25955699/

And that one doesn't show much either, apart from indicating that something is keeping the lock for itself. Drat.

We need to narrow down the problem, or make it appear on more common HW. Let me know if you've managed to reproduce it with non-VHE and/or on TX-1.

Thanks,

	M.
Signature · 1 line
Jazz is not dead. It just smells funny...
#6 lore raw
Quoted 38 lines · Marc Zyngier
Quoted 25 lines · Jan Glauber
Quoted 2 lines · Marc Zyngier
Quoted 1 line · Jan Glauber

On Mon, Nov 13, 2017 at 01:47:38PM +0000, Marc Zyngier wrote:

[...]

Quoted 19 lines
Quoted 19 lines
Quoted 2 lines

Please elaborate. Messed in what way? Corrupted? The guest crashing? Or is that a tooling issue?

Every vcpu that oopses prints one line in parallel, so I get blocks like:

[58880.179814] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179834] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179847] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179873] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179893] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179911] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179917] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180288] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180303] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180336] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180363] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180384] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180415] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180461] [<ffff000008084b98>] ret_from_fork+0x10/0x18
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

I can send the full log if you want to have a look.

Sure, send that over (maybe not over email though).

Here is the guest dmesg:
http://paste.ubuntu.com/25955682/

Yeah, that's because all the vcpus are getting starved at the same time, and spitting out interleaved traces... Not very useful anyway, as I think this is only a consequence of what's happening on the host.

Quoted 1 line
Quoted 8 lines
Quoted 2 lines

And the host dmesg as it might have been too big for the lists:
http://paste.ubuntu.com/25955699/

And that one doesn't show much either, apart from indicating that something is keeping the lock for itself. Drat.

We need to narrow down the problem, or make it appear on more common HW. Let me know if you've managed to reproduce it with non-VHE and/or on TX-1.

It also shows up when I disable VHE (CONFIG_ARM64_VHE). I'll try enabling some tracepoints next.

--Jan

#7 lore raw
In reply to Marc Zyngier
Quoted 1 line · Marc Zyngier

On 13/11/17 17:35, Jan Glauber wrote:

[...]

Quoted 25 lines
Quoted 25 lines
Quoted 24 lines
Quoted 18 lines

numbers don't look good, see waittime-max:

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
class name con-bounces contentions waittime-min waittime-max waittime-total waittime-avg acq-bounces acquisitions holdtime-min holdtime-max holdtime-total holdtime-avg
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

                &(&kvm->mmu_lock)->rlock:      99346764       99406604           0.14  1321260806.59 710654434972.0        7148.97      154228320      225122857           0.13   917688890.60  3705916481.39          16.46
                ------------------------
                &(&kvm->mmu_lock)->rlock       99365598          [<ffff0000080b43b8>] kvm_handle_guest_abort+0x4c0/0x950
                &(&kvm->mmu_lock)->rlock          25164          [<ffff0000080a4e30>] kvm_mmu_notifier_invalidate_range_start+0x70/0xe8
                &(&kvm->mmu_lock)->rlock          14934          [<ffff0000080a7eec>] kvm_mmu_notifier_invalidate_range_end+0x24/0x68
                &(&kvm->mmu_lock)->rlock            908          [<ffff00000810a1f0>] __cond_resched_lock+0x68/0xb8
                ------------------------
                &(&kvm->mmu_lock)->rlock              3          [<ffff0000080b34c8>] stage2_flush_vm+0x60/0xd8
                &(&kvm->mmu_lock)->rlock       99186296          [<ffff0000080b43b8>] kvm_handle_guest_abort+0x4c0/0x950
                &(&kvm->mmu_lock)->rlock         179238          [<ffff0000080a4e30>] kvm_mmu_notifier_invalidate_range_start+0x70/0xe8
                &(&kvm->mmu_lock)->rlock          19181          [<ffff0000080a7eec>] kvm_mmu_notifier_invalidate_range_end+0x24/0x68
Quoted 1 line

.............................................................................................................................................................................................................................

[slots of stuff]

Well, the mmu_lock is clearly contended. Is the box in a state where you are swapping? There seem to be as many faults as contentions, which is a bit surprising...

Quoted 5 lines
Quoted 1 line

I don't think it is swapping but need to double check.

It is the number of aborts that is staggering. And each one of them leads to the mmu_lock being contended. So something seems to be taking its sweet time holding the damned lock.

Can you elaborate on the aborts, I'm not familiar with KVM but from a first look I thought kvm_handle_guest_abort() is in the normal path when a vcpu is stopped. Is that wrong?

--Jan

#8 lore raw
Quoted 32 lines · Jan Glauber
Quoted 1 line · Marc Zyngier

On 13/11/17 17:35, Jan Glauber wrote:

[...]

Quoted 27 lines
Quoted 26 lines
Quoted 18 lines
Quoted 18 lines

numbers don't look good, see waittime-max:

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
class name con-bounces contentions waittime-min waittime-max waittime-total waittime-avg acq-bounces acquisitions holdtime-min holdtime-max holdtime-total holdtime-avg
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

                &(&kvm->mmu_lock)->rlock:      99346764       99406604           0.14  1321260806.59 710654434972.0        7148.97      154228320      225122857           0.13   917688890.60  3705916481.39          16.46
                ------------------------
                &(&kvm->mmu_lock)->rlock       99365598          [<ffff0000080b43b8>] kvm_handle_guest_abort+0x4c0/0x950
                &(&kvm->mmu_lock)->rlock          25164          [<ffff0000080a4e30>] kvm_mmu_notifier_invalidate_range_start+0x70/0xe8
                &(&kvm->mmu_lock)->rlock          14934          [<ffff0000080a7eec>] kvm_mmu_notifier_invalidate_range_end+0x24/0x68
                &(&kvm->mmu_lock)->rlock            908          [<ffff00000810a1f0>] __cond_resched_lock+0x68/0xb8
                ------------------------
                &(&kvm->mmu_lock)->rlock              3          [<ffff0000080b34c8>] stage2_flush_vm+0x60/0xd8
                &(&kvm->mmu_lock)->rlock       99186296          [<ffff0000080b43b8>] kvm_handle_guest_abort+0x4c0/0x950
                &(&kvm->mmu_lock)->rlock         179238          [<ffff0000080a4e30>] kvm_mmu_notifier_invalidate_range_start+0x70/0xe8
                &(&kvm->mmu_lock)->rlock          19181          [<ffff0000080a7eec>] kvm_mmu_notifier_invalidate_range_end+0x24/0x68
Quoted 6 lines
Quoted 1 line

.............................................................................................................................................................................................................................

[slots of stuff]

Well, the mmu_lock is clearly contended. Is the box in a state where you are swapping? There seem to be as many faults as contentions, which is a bit surprising...

I don't think it is swapping but need to double check.

Quoted 7 lines
Quoted 3 lines

It is the number of aborts that is staggering. And each one of them leads to the mmu_lock being contended. So something seems to be taking its sweet time holding the damned lock.

Can you elaborate on the aborts, I'm not familiar with KVM but from a first look I thought kvm_handle_guest_abort() is in the normal path when a vcpu is stopped. Is that wrong?

kvm_handle_guest_abort() is the entry point for our page fault handling (hence the mmu_lock being taken). On its own, the number of faults is irrelevant. What worries me is that in almost all the cases the lock was contended, we were handling a page fault.

What would be interesting is to find out *who* is holding the lock when we're being blocked in kvm_handle_guest_abort...

	M.
Signature · 1 line
Jazz is not dead. It just smells funny...
#9 lore raw
Quoted 41 lines · Marc Zyngier
Quoted 32 lines · Jan Glauber
Quoted 1 line · Marc Zyngier

On 13/11/17 17:35, Jan Glauber wrote:

[...]

Quoted 27 lines
Quoted 26 lines
Quoted 18 lines
Quoted 18 lines

numbers don't look good, see waittime-max:

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
class name con-bounces contentions waittime-min waittime-max waittime-total waittime-avg acq-bounces acquisitions holdtime-min holdtime-max holdtime-total holdtime-avg
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

                 &(&kvm->mmu_lock)->rlock:      99346764       99406604           0.14  1321260806.59 710654434972.0        7148.97      154228320      225122857           0.13   917688890.60  3705916481.39          16.46
                 ------------------------
                 &(&kvm->mmu_lock)->rlock       99365598          [<ffff0000080b43b8>] kvm_handle_guest_abort+0x4c0/0x950
                 &(&kvm->mmu_lock)->rlock          25164          [<ffff0000080a4e30>] kvm_mmu_notifier_invalidate_range_start+0x70/0xe8
                 &(&kvm->mmu_lock)->rlock          14934          [<ffff0000080a7eec>] kvm_mmu_notifier_invalidate_range_end+0x24/0x68
                 &(&kvm->mmu_lock)->rlock            908          [<ffff00000810a1f0>] __cond_resched_lock+0x68/0xb8
                 ------------------------
                 &(&kvm->mmu_lock)->rlock              3          [<ffff0000080b34c8>] stage2_flush_vm+0x60/0xd8
                 &(&kvm->mmu_lock)->rlock       99186296          [<ffff0000080b43b8>] kvm_handle_guest_abort+0x4c0/0x950
                 &(&kvm->mmu_lock)->rlock         179238          [<ffff0000080a4e30>] kvm_mmu_notifier_invalidate_range_start+0x70/0xe8
                 &(&kvm->mmu_lock)->rlock          19181          [<ffff0000080a7eec>] kvm_mmu_notifier_invalidate_range_end+0x24/0x68
Quoted 6 lines
Quoted 1 line

.............................................................................................................................................................................................................................

[slots of stuff]

Well, the mmu_lock is clearly contended. Is the box in a state where you are swapping? There seem to be as many faults as contentions, which is a bit surprising...

I don't think it is swapping but need to double check.

Quoted 7 lines
Quoted 3 lines

It is the number of aborts that is staggering. And each one of them leads to the mmu_lock being contended. So something seems to be taking its sweet time holding the damned lock.

Can you elaborate on the aborts, I'm not familiar with KVM but from a first look I thought kvm_handle_guest_abort() is in the normal path when a vcpu is stopped. Is that wrong?

Quoted 7 lines

kvm_handle_guest_abort() is the entry point for our page fault handling (hence the mmu_lock being taken). On its own, the number of faults is irrelevant. What worries me is that in almost all the cases the lock was contended, we were handling a page fault.

What would be interesting is to find out *who* is holding the lock when we're being blocked in kvm_handle_guest_abort...

Just a thought, turning on the tracepoints for kvm_hva_* might help to get some more data on what we are doing with the HVA ranges.

Cheers
Suzuki

#10 lore raw
In reply to Jan Glauber
Quoted 82 lines

-----Original Message-----
From: linux-arm-kernel [mailto:linux-arm-kernel-bounces at lists.infradead.org] On Behalf Of Jan Glauber
Sent: Monday, November 13, 2017 5:36 PM
To: Marc Zyngier <[email protected]>
Cc: linux-arm-kernel at lists.infradead.org; Paolo Bonzini <[email protected]>; Christoffer Dall <[email protected]>; kvm at vger.kernel.org; Radim Kr?m?? <[email protected]>
Subject: Re: RCU stall with high number of KVM vcpus

Quoted 2 lines · Marc Zyngier
Quoted 1 line · Jan Glauber

I'm seeing RCU stalls in the host with 4.14 when I run KVM on ARM64

(ThunderX2) with a high

Quoted 4 lines
Quoted 1 line

number of vcpus (60). I only use one guest that does kernel compiles in

Is that only reproducible on 4.14? With or without VHE? Can you reproduce this on another implementation (such as ThunderX-1)?

I've reproduced it on a distro 4.13 and several vanilla 4.14 rc's and tip/locking. VHE is enabled. I've not yet tried to reproduce it with older kernels or ThunderX-1. I can check if it happens also on ThunderX-1.

Quoted 2 lines
Quoted 2 lines

a loop. After some hours (less likely the more debugging options are enabled, more likely with more vcpus) RCU stalls are happening in both

host & guest.

Quoted 9 lines
Quoted 6 lines

Both host & guest recover after some time, until the issue is triggered again.

Stack traces in the guest are next to useless, everything is messed up there. The host seems to stave on kvm->mmu_lock spin lock, the lock_stat

Please elaborate. Messed in what way? Corrupted? The guest crashing? Or is that a tooling issue?

Every vcpu that oopses prints one line in parallel, so I get blocks like:

[58880.179814] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179834] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179847] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179873] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179893] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179911] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.179917] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180288] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180303] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180336] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180363] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180384] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180415] [<ffff000008084b98>] ret_from_fork+0x10/0x18
[58880.180461] [<ffff000008084b98>] ret_from_fork+0x10/0x18

I can send the full log if you want to have a look.

Quoted 3 lines
Quoted 3 lines

numbers don't look good, see waittime-max:

---------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------------------------
-------------------

Quoted 1 line
Quoted 1 line

class name con-bounces contentions waittime-min

waittime-max waittime-total waittime-avg acq-bounces acquisitions
holdtime-min holdtime-max holdtime-total holdtime-avg

Quoted 1 line
Quoted 1 line

---------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------------------------
-------------------

Quoted 2 lines
Quoted 2 lines
                &(&kvm->mmu_lock)->rlock:      99346764       99406604

0.14 1321260806.59 710654434972.0 7148.97 154228320

225122857           0.13   917688890.60  3705916481.39          16.46
> >                 ------------------------
> >                 &(&kvm->mmu_lock)->rlock       99365598

[<ffff0000080b43b8>] kvm_handle_guest_abort+0x4c0/0x950

Quoted 1 line
Quoted 1 line
                &(&kvm->mmu_lock)->rlock          25164

[<ffff0000080a4e30>] kvm_mmu_notifier_invalidate_range_start+0x70/0xe8

Quoted 1 line
Quoted 1 line
                &(&kvm->mmu_lock)->rlock          14934

[<ffff0000080a7eec>] kvm_mmu_notifier_invalidate_range_end+0x24/0x68

Quoted 1 line
Quoted 1 line
                &(&kvm->mmu_lock)->rlock            908

[<ffff00000810a1f0>] __cond_resched_lock+0x68/0xb8

Quoted 2 lines
Quoted 2 lines
                ------------------------
                &(&kvm->mmu_lock)->rlock              3          [<ffff0000080b34c8>]
Quoted 7 lines

stage2_flush_vm+0x60/0xd8

Quoted 1 line
Quoted 1 line
                &(&kvm->mmu_lock)->rlock       99186296

[<ffff0000080b43b8>] kvm_handle_guest_abort+0x4c0/0x950

Quoted 1 line
Quoted 1 line
                &(&kvm->mmu_lock)->rlock         179238

[<ffff0000080a4e30>] kvm_mmu_notifier_invalidate_range_start+0x70/0xe8

Quoted 1 line
Quoted 1 line
                &(&kvm->mmu_lock)->rlock          19181

[<ffff0000080a7eec>] kvm_mmu_notifier_invalidate_range_end+0x24/0x68

That looks like something similar we had on our hip07 platform when multiple VMs were launched. The issue was tracked down to CONFIG_NUMA set with memory_less nodes. This results in lot of individual 4K pages and unmap_stage2_ptes() takes a good amount of time coupled with some HW cache flush latencies. I am not sure you are seeing the same thing, but may be worth checking.

Thanks,
Shameer

#11 lore raw

On Mon, Nov 13, 2017 at 06:13:08PM +0000, Shameerali Kolothum Thodi wrote:

[...]

Quoted 31 lines
Quoted 31 lines
Quoted 3 lines
Quoted 3 lines

numbers don't look good, see waittime-max:

---------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------------------------
-------------------

Quoted 1 line
Quoted 1 line

class name con-bounces contentions waittime-min

waittime-max waittime-total waittime-avg acq-bounces acquisitions
holdtime-min holdtime-max holdtime-total holdtime-avg

Quoted 1 line
Quoted 1 line

---------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------------------------
-------------------

Quoted 2 lines
Quoted 2 lines
                &(&kvm->mmu_lock)->rlock:      99346764       99406604

0.14 1321260806.59 710654434972.0 7148.97 154228320

225122857           0.13   917688890.60  3705916481.39          16.46
> >                 ------------------------
> >                 &(&kvm->mmu_lock)->rlock       99365598

[<ffff0000080b43b8>] kvm_handle_guest_abort+0x4c0/0x950

Quoted 1 line
Quoted 1 line
                &(&kvm->mmu_lock)->rlock          25164

[<ffff0000080a4e30>] kvm_mmu_notifier_invalidate_range_start+0x70/0xe8

Quoted 1 line
Quoted 1 line
                &(&kvm->mmu_lock)->rlock          14934

[<ffff0000080a7eec>] kvm_mmu_notifier_invalidate_range_end+0x24/0x68

Quoted 1 line
Quoted 1 line
                &(&kvm->mmu_lock)->rlock            908

[<ffff00000810a1f0>] __cond_resched_lock+0x68/0xb8

Quoted 2 lines
Quoted 2 lines
                ------------------------
                &(&kvm->mmu_lock)->rlock              3          [<ffff0000080b34c8>]

stage2_flush_vm+0x60/0xd8

Quoted 1 line
Quoted 1 line
                &(&kvm->mmu_lock)->rlock       99186296

[<ffff0000080b43b8>] kvm_handle_guest_abort+0x4c0/0x950

Quoted 1 line
Quoted 1 line
                &(&kvm->mmu_lock)->rlock         179238

[<ffff0000080a4e30>] kvm_mmu_notifier_invalidate_range_start+0x70/0xe8

Quoted 8 lines
Quoted 2 lines
Quoted 1 line
Quoted 1 line
                &(&kvm->mmu_lock)->rlock          19181

[<ffff0000080a7eec>] kvm_mmu_notifier_invalidate_range_end+0x24/0x68

That looks like something similar we had on our hip07 platform when multiple VMs were launched. The issue was tracked down to CONFIG_NUMA set with memory_less nodes. This results in lot of individual 4K pages and unmap_stage2_ptes() takes a good amount of time coupled with some HW cache flush latencies. I am not sure you are seeing the same thing, but may be worth checking.

Hi Shameer,

thanks for the tip. We don't have memory-less nodes but it might me related to NUMA. I've tried putting the guest onto one node but that did not help.

PID Node 0 Node 1 Total
----------------------- --------------- --------------- ---------------
56753 (qemu-nbd) 4.48 11.16 15.64
56813 (qemu-system-aar) 2.02 1685.72 1687.75
----------------------- --------------- --------------- ---------------
Total 6.51 1696.88 1703.39

I'll try switching to 64K pages in the host next.

thanks,
Jan