RCU stall with high number of KVM vcpus
Quoted 2 lines · Jan Glauber
I'm seeing RCU stalls in the host with 4.14 when I run KVM on ARM64 (ThunderX2) with a high number of vcpus (60). I only use one guest that does kernel compiles in
Is that only reproducible on 4.14? With or without VHE? Can you reproduce this on another implementation (such as ThunderX-1)?
Quoted 8 lines
a loop. After some hours (less likely the more debugging options are enabled, more likely with more vcpus) RCU stalls are happening in both host & guest.
Both host & guest recover after some time, until the issue is triggered again.
Stack traces in the guest are next to useless, everything is messed up there. The host seems to stave on
kvm->mmu_lockspin lock, thelock_stat
Please elaborate. Messed in what way? Corrupted? The guest crashing? Or is that a tooling issue?
Quoted 18 lines
numbers don't look good, see waittime-max:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
class name con-bounces contentions waittime-min waittime-max waittime-total waittime-avg acq-bounces acquisitions holdtime-min holdtime-max holdtime-total holdtime-avg
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------&(&kvm->mmu_lock)->rlock: 99346764 99406604 0.14 1321260806.59 710654434972.0 7148.97 154228320 225122857 0.13 917688890.60 3705916481.39 16.46 ------------------------ &(&kvm->mmu_lock)->rlock 99365598 [<ffff0000080b43b8>] kvm_handle_guest_abort+0x4c0/0x950 &(&kvm->mmu_lock)->rlock 25164 [<ffff0000080a4e30>] kvm_mmu_notifier_invalidate_range_start+0x70/0xe8 &(&kvm->mmu_lock)->rlock 14934 [<ffff0000080a7eec>] kvm_mmu_notifier_invalidate_range_end+0x24/0x68 &(&kvm->mmu_lock)->rlock 908 [<ffff00000810a1f0>] __cond_resched_lock+0x68/0xb8 ------------------------ &(&kvm->mmu_lock)->rlock 3 [<ffff0000080b34c8>] stage2_flush_vm+0x60/0xd8 &(&kvm->mmu_lock)->rlock 99186296 [<ffff0000080b43b8>] kvm_handle_guest_abort+0x4c0/0x950 &(&kvm->mmu_lock)->rlock 179238 [<ffff0000080a4e30>] kvm_mmu_notifier_invalidate_range_start+0x70/0xe8 &(&kvm->mmu_lock)->rlock 19181 [<ffff0000080a7eec>] kvm_mmu_notifier_invalidate_range_end+0x24/0x68
Quoted 1 line
.............................................................................................................................................................................................................................
[slots of stuff]
Well, the mmu_lock is clearly contended. Is the box in a state where you are swapping? There seem to be as many faults as contentions, which is a bit surprising...
Also, we recently moved arm64 to qrwlocks, which may have an impact. Care to give this[1] a go and report the result?
Thanks,
M.
[1]: https://lkml.org/lkml/2017/10/12/266
Signature · 1 line
Jazz is not dead. It just smells funny...
Quoted 6 lines · Marc Zyngier
Quoted 2 lines · Jan Glauber
I'm seeing RCU stalls in the host with 4.14 when I run KVM on ARM64 (ThunderX2) with a high number of vcpus (60). I only use one guest that does kernel compiles in
Is that only reproducible on 4.14? With or without VHE? Can you reproduce this on another implementation (such as ThunderX-1)?
I've reproduced it on a distro 4.13 and several vanilla 4.14 rc's and tip/locking. VHE is enabled. I've not yet tried to reproduce it with older kernels or ThunderX-1. I can check if it happens also on ThunderX-1.
Quoted 11 lines
Quoted 8 lines
a loop. After some hours (less likely the more debugging options are enabled, more likely with more vcpus) RCU stalls are happening in both host & guest.
Both host & guest recover after some time, until the issue is triggered again.
Stack traces in the guest are next to useless, everything is messed up there. The host seems to stave on
kvm->mmu_lockspin lock, thelock_statPlease elaborate. Messed in what way? Corrupted? The guest crashing? Or is that a tooling issue?
Every vcpu that oopses prints one line in parallel, so I get blocks like:
[58880.179814] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179834] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179847] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179873] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179893] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179911] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179917] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180288] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180303] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180336] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180363] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180384] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180415] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180461] [<ffff000008084b98>] ret_from_fork+0x10/0x18
I can send the full log if you want to have a look.
Quoted 18 lines
Quoted 18 lines
numbers don't look good, see waittime-max:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
class name con-bounces contentions waittime-min waittime-max waittime-total waittime-avg acq-bounces acquisitions holdtime-min holdtime-max holdtime-total holdtime-avg
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------&(&kvm->mmu_lock)->rlock: 99346764 99406604 0.14 1321260806.59 710654434972.0 7148.97 154228320 225122857 0.13 917688890.60 3705916481.39 16.46 ------------------------ &(&kvm->mmu_lock)->rlock 99365598 [<ffff0000080b43b8>] kvm_handle_guest_abort+0x4c0/0x950 &(&kvm->mmu_lock)->rlock 25164 [<ffff0000080a4e30>] kvm_mmu_notifier_invalidate_range_start+0x70/0xe8 &(&kvm->mmu_lock)->rlock 14934 [<ffff0000080a7eec>] kvm_mmu_notifier_invalidate_range_end+0x24/0x68 &(&kvm->mmu_lock)->rlock 908 [<ffff00000810a1f0>] __cond_resched_lock+0x68/0xb8 ------------------------ &(&kvm->mmu_lock)->rlock 3 [<ffff0000080b34c8>] stage2_flush_vm+0x60/0xd8 &(&kvm->mmu_lock)->rlock 99186296 [<ffff0000080b43b8>] kvm_handle_guest_abort+0x4c0/0x950 &(&kvm->mmu_lock)->rlock 179238 [<ffff0000080a4e30>] kvm_mmu_notifier_invalidate_range_start+0x70/0xe8 &(&kvm->mmu_lock)->rlock 19181 [<ffff0000080a7eec>] kvm_mmu_notifier_invalidate_range_end+0x24/0x68
Quoted 6 lines
Quoted 1 line
.............................................................................................................................................................................................................................
[slots of stuff]
Well, the
mmu_lockis clearly contended. Is the box in a state where you are swapping? There seem to be as many faults as contentions, which is a bit surprising...
I don't think it is swapping but need to double check.
Quoted 2 lines
Also, we recently moved arm64 to qrwlocks, which may have an impact. Care to give this[1] a go and report the result?
Sure, I that was my first suspision but I can reproduce the issue with and without the qrwlock patches, so these are not to blame. Also, the starving lock is a spinlock and not a qrwlock. So maybe the spinlocks have fairness issues too?
--Jan
Quoted 7 lines
Thanks,
M.[1]: https://lkml.org/lkml/2017/10/12/266
Signature · 1 line
Jazz is not dead. It just smells funny...
Quoted 11 lines · Jan Glauber
Quoted 6 lines · Marc Zyngier
Quoted 2 lines · Jan Glauber
I'm seeing RCU stalls in the host with 4.14 when I run KVM on ARM64 (ThunderX2) with a high number of vcpus (60). I only use one guest that does kernel compiles in
Is that only reproducible on 4.14? With or without VHE? Can you reproduce this on another implementation (such as ThunderX-1)?
I've reproduced it on a distro 4.13 and several vanilla 4.14 rc's and tip/locking. VHE is enabled. I've not yet tried to reproduce it with older kernels or ThunderX-1. I can check if it happens also on ThunderX-1.
That'd be interesting. Also, what happens if you turn off any form of multi-threading (as I've been told this system has 4 threads per core)? Disabling VHE would be interesting too (after all, this is the only box I know of that has the feature...).
Quoted 28 lines
Quoted 11 lines
Quoted 8 lines
a loop. After some hours (less likely the more debugging options are enabled, more likely with more vcpus) RCU stalls are happening in both host & guest.
Both host & guest recover after some time, until the issue is triggered again.
Stack traces in the guest are next to useless, everything is messed up there. The host seems to stave on
kvm->mmu_lockspin lock, thelock_statPlease elaborate. Messed in what way? Corrupted? The guest crashing? Or is that a tooling issue?
Every vcpu that oopses prints one line in parallel, so I get blocks like:
[58880.179814] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179834] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179847] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179873] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179893] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179911] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179917] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180288] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180303] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180336] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180363] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180384] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180415] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180461] [<ffff000008084b98>] ret_from_fork+0x10/0x18
Quoted 1 line
I can send the full log if you want to have a look.
Sure, send that over (maybe not over email though).
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
numbers don't look good, see waittime-max:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
class name con-bounces contentions waittime-min waittime-max waittime-total waittime-avg acq-bounces acquisitions holdtime-min holdtime-max holdtime-total holdtime-avg
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------&(&kvm->mmu_lock)->rlock: 99346764 99406604 0.14 1321260806.59 710654434972.0 7148.97 154228320 225122857 0.13 917688890.60 3705916481.39 16.46 ------------------------ &(&kvm->mmu_lock)->rlock 99365598 [<ffff0000080b43b8>] kvm_handle_guest_abort+0x4c0/0x950 &(&kvm->mmu_lock)->rlock 25164 [<ffff0000080a4e30>] kvm_mmu_notifier_invalidate_range_start+0x70/0xe8 &(&kvm->mmu_lock)->rlock 14934 [<ffff0000080a7eec>] kvm_mmu_notifier_invalidate_range_end+0x24/0x68 &(&kvm->mmu_lock)->rlock 908 [<ffff00000810a1f0>] __cond_resched_lock+0x68/0xb8 ------------------------ &(&kvm->mmu_lock)->rlock 3 [<ffff0000080b34c8>] stage2_flush_vm+0x60/0xd8 &(&kvm->mmu_lock)->rlock 99186296 [<ffff0000080b43b8>] kvm_handle_guest_abort+0x4c0/0x950 &(&kvm->mmu_lock)->rlock 179238 [<ffff0000080a4e30>] kvm_mmu_notifier_invalidate_range_start+0x70/0xe8 &(&kvm->mmu_lock)->rlock 19181 [<ffff0000080a7eec>] kvm_mmu_notifier_invalidate_range_end+0x24/0x68
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
.............................................................................................................................................................................................................................
[slots of stuff]
Well, the
mmu_lockis clearly contended. Is the box in a state where you are swapping? There seem to be as many faults as contentions, which is a bit surprising...I don't think it is swapping but need to double check.
It is the number of aborts that is staggering. And each one of them leads to the mmu_lock being contended. So something seems to be taking its sweet time holding the damned lock.
Quoted 8 lines
Quoted 3 lines
Also, we recently moved arm64 to qrwlocks, which may have an impact. Care to give this[1] a go and report the result?
Sure, I that was my first suspision but I can reproduce the issue with and without the qrwlock patches, so these are not to blame. Also, the starving lock is a spinlock and not a qrwlock. So maybe the spinlocks have fairness issues too?
We use ticket locks, so they should be fine on the host (it is a different story altogether in a guest).
Can you send me your kernel configuration as well? I wonder if we're not seeing interactions with things like KSM and the like...
Thanks,
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
Quoted 82 lines
-----Original Message-----
From: linux-arm-kernel [mailto:linux-arm-kernel-bounces at lists.infradead.org] On Behalf Of Jan Glauber
Sent: Monday, November 13, 2017 5:36 PM
To: Marc Zyngier <[email protected]>
Cc: linux-arm-kernel at lists.infradead.org; Paolo Bonzini <[email protected]>; Christoffer Dall <[email protected]>; kvm at vger.kernel.org; Radim Kr?m?? <[email protected]>
Subject: Re: RCU stall with high number of KVM vcpusQuoted 2 lines · Marc Zyngier
Quoted 1 line · Jan Glauber
I'm seeing RCU stalls in the host with 4.14 when I run KVM on ARM64
(ThunderX2) with a high
Quoted 4 lines
Quoted 1 line
number of vcpus (60). I only use one guest that does kernel compiles in
Is that only reproducible on 4.14? With or without VHE? Can you reproduce this on another implementation (such as ThunderX-1)?
I've reproduced it on a distro 4.13 and several vanilla 4.14 rc's and tip/locking. VHE is enabled. I've not yet tried to reproduce it with older kernels or ThunderX-1. I can check if it happens also on ThunderX-1.
Quoted 2 lines
Quoted 2 lines
a loop. After some hours (less likely the more debugging options are enabled, more likely with more vcpus) RCU stalls are happening in both
host & guest.
Quoted 9 lines
Quoted 6 lines
Both host & guest recover after some time, until the issue is triggered again.
Stack traces in the guest are next to useless, everything is messed up there. The host seems to stave on
kvm->mmu_lockspin lock, thelock_statPlease elaborate. Messed in what way? Corrupted? The guest crashing? Or is that a tooling issue?
Every vcpu that oopses prints one line in parallel, so I get blocks like:
[58880.179814] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179834] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179847] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179873] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179893] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179911] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179917] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180288] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180303] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180336] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180363] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180384] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180415] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180461] [<ffff000008084b98>] ret_from_fork+0x10/0x18I can send the full log if you want to have a look.
Quoted 3 lines
Quoted 3 lines
numbers don't look good, see waittime-max:
---------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------
-------------------Quoted 1 line
Quoted 1 line
class name con-bounces contentions waittime-min
waittime-max waittime-total waittime-avg acq-bounces acquisitions
holdtime-min holdtime-max holdtime-total holdtime-avgQuoted 1 line
Quoted 1 line
---------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------
-------------------Quoted 2 lines
Quoted 2 lines
&(&kvm->mmu_lock)->rlock: 99346764 994066040.14 1321260806.59 710654434972.0 7148.97 154228320
225122857 0.13 917688890.60 3705916481.39 16.46 > > ------------------------ > > &(&kvm->mmu_lock)->rlock 99365598[<
ffff0000080b43b8>]kvm_handle_guest_abort+0x4c0/0x950Quoted 1 line
Quoted 1 line
&(&kvm->mmu_lock)->rlock 25164[<
ffff0000080a4e30>]kvm_mmu_notifier_invalidate_range_start+0x70/0xe8Quoted 1 line
Quoted 1 line
&(&kvm->mmu_lock)->rlock 14934[<
ffff0000080a7eec>]kvm_mmu_notifier_invalidate_range_end+0x24/0x68Quoted 1 line
Quoted 1 line
&(&kvm->mmu_lock)->rlock 908[<
ffff00000810a1f0>]__cond_resched_lock+0x68/0xb8Quoted 2 lines
Quoted 2 lines
------------------------ &(&kvm->mmu_lock)->rlock 3 [<ffff0000080b34c8>]
Quoted 7 lines
stage2_flush_vm+0x60/0xd8Quoted 1 line
Quoted 1 line
&(&kvm->mmu_lock)->rlock 99186296[<
ffff0000080b43b8>]kvm_handle_guest_abort+0x4c0/0x950Quoted 1 line
Quoted 1 line
&(&kvm->mmu_lock)->rlock 179238[<
ffff0000080a4e30>]kvm_mmu_notifier_invalidate_range_start+0x70/0xe8Quoted 1 line
Quoted 1 line
&(&kvm->mmu_lock)->rlock 19181[<
ffff0000080a7eec>]kvm_mmu_notifier_invalidate_range_end+0x24/0x68
That looks like something similar we had on our hip07 platform when multiple VMs were launched. The issue was tracked down to CONFIG_NUMA set with memory_less nodes. This results in lot of individual 4K pages and unmap_stage2_ptes() takes a good amount of time coupled with some HW cache flush latencies. I am not sure you are seeing the same thing, but may be worth checking.
Thanks,
Shameer
Quoted 2 lines · Marc Zyngier
Quoted 1 line · Jan Glauber
On Mon, Nov 13, 2017 at 01:47:38PM +0000, Marc Zyngier wrote:
[...]
Quoted 19 lines
Quoted 19 lines
Quoted 2 lines
Please elaborate. Messed in what way? Corrupted? The guest crashing? Or is that a tooling issue?
Every vcpu that oopses prints one line in parallel, so I get blocks like:
[58880.179814] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179834] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179847] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179873] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179893] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179911] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179917] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180288] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180303] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180336] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180363] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180384] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180415] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180461] [<ffff000008084b98>] ret_from_fork+0x10/0x18
Quoted 3 lines
Quoted 1 line
I can send the full log if you want to have a look.
Sure, send that over (maybe not over email though).
Here is the guest dmesg:
http://paste.ubuntu.com/25955682/
And the host dmesg as it might have been too big for the lists:
http://paste.ubuntu.com/25955699/
Quoted 2 lines
Can you send me your kernel configuration as well? I wonder if we're not seeing interactions with things like KSM and the like...
It was plain arm64 defconfig but I disabled HAVE_KVM_CPU_RELAX_INTERCEPT. Not that it had an impact, just tried.
http://paste.ubuntu.com/25955726/
--Jan
On Mon, Nov 13, 2017 at 06:13:08PM +0000, Shameerali Kolothum Thodi wrote:
[...]
Quoted 31 lines
Quoted 31 lines
Quoted 3 lines
Quoted 3 lines
numbers don't look good, see waittime-max:
---------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------
-------------------Quoted 1 line
Quoted 1 line
class name con-bounces contentions waittime-min
waittime-max waittime-total waittime-avg acq-bounces acquisitions
holdtime-min holdtime-max holdtime-total holdtime-avgQuoted 1 line
Quoted 1 line
---------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------
-------------------Quoted 2 lines
Quoted 2 lines
&(&kvm->mmu_lock)->rlock: 99346764 994066040.14 1321260806.59 710654434972.0 7148.97 154228320
225122857 0.13 917688890.60 3705916481.39 16.46 > > ------------------------ > > &(&kvm->mmu_lock)->rlock 99365598[<
ffff0000080b43b8>]kvm_handle_guest_abort+0x4c0/0x950Quoted 1 line
Quoted 1 line
&(&kvm->mmu_lock)->rlock 25164[<
ffff0000080a4e30>]kvm_mmu_notifier_invalidate_range_start+0x70/0xe8Quoted 1 line
Quoted 1 line
&(&kvm->mmu_lock)->rlock 14934[<
ffff0000080a7eec>]kvm_mmu_notifier_invalidate_range_end+0x24/0x68Quoted 1 line
Quoted 1 line
&(&kvm->mmu_lock)->rlock 908[<
ffff00000810a1f0>]__cond_resched_lock+0x68/0xb8Quoted 2 lines
Quoted 2 lines
------------------------ &(&kvm->mmu_lock)->rlock 3 [<ffff0000080b34c8>]
stage2_flush_vm+0x60/0xd8Quoted 1 line
Quoted 1 line
&(&kvm->mmu_lock)->rlock 99186296[<
ffff0000080b43b8>]kvm_handle_guest_abort+0x4c0/0x950Quoted 1 line
Quoted 1 line
&(&kvm->mmu_lock)->rlock 179238[<
ffff0000080a4e30>]kvm_mmu_notifier_invalidate_range_start+0x70/0xe8
Quoted 8 lines
Quoted 2 lines
Quoted 1 line
Quoted 1 line
&(&kvm->mmu_lock)->rlock 19181[<
ffff0000080a7eec>]kvm_mmu_notifier_invalidate_range_end+0x24/0x68That looks like something similar we had on our hip07 platform when multiple VMs were launched. The issue was tracked down to
CONFIG_NUMAset withmemory_lessnodes. This results in lot of individual 4K pages andunmap_stage2_ptes()takes a good amount of time coupled with some HW cache flush latencies. I am not sure you are seeing the same thing, but may be worth checking.
Hi Shameer,
thanks for the tip. We don't have memory-less nodes but it might me related to NUMA. I've tried putting the guest onto one node but that did not help.
PID Node 0 Node 1 Total
----------------------- --------------- --------------- ---------------
56753 (qemu-nbd) 4.48 11.16 15.64
56813 (qemu-system-aar) 2.02 1685.72 1687.75
----------------------- --------------- --------------- ---------------
Total 6.51 1696.88 1703.39
I'll try switching to 64K pages in the host next.
thanks,
Jan
Quoted 1 line · Marc Zyngier
On 13/11/17 17:35, Jan Glauber wrote:
[...]
Quoted 25 lines
Quoted 25 lines
Quoted 24 lines
Quoted 18 lines
numbers don't look good, see waittime-max:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
class name con-bounces contentions waittime-min waittime-max waittime-total waittime-avg acq-bounces acquisitions holdtime-min holdtime-max holdtime-total holdtime-avg
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------&(&kvm->mmu_lock)->rlock: 99346764 99406604 0.14 1321260806.59 710654434972.0 7148.97 154228320 225122857 0.13 917688890.60 3705916481.39 16.46 ------------------------ &(&kvm->mmu_lock)->rlock 99365598 [<ffff0000080b43b8>] kvm_handle_guest_abort+0x4c0/0x950 &(&kvm->mmu_lock)->rlock 25164 [<ffff0000080a4e30>] kvm_mmu_notifier_invalidate_range_start+0x70/0xe8 &(&kvm->mmu_lock)->rlock 14934 [<ffff0000080a7eec>] kvm_mmu_notifier_invalidate_range_end+0x24/0x68 &(&kvm->mmu_lock)->rlock 908 [<ffff00000810a1f0>] __cond_resched_lock+0x68/0xb8 ------------------------ &(&kvm->mmu_lock)->rlock 3 [<ffff0000080b34c8>] stage2_flush_vm+0x60/0xd8 &(&kvm->mmu_lock)->rlock 99186296 [<ffff0000080b43b8>] kvm_handle_guest_abort+0x4c0/0x950 &(&kvm->mmu_lock)->rlock 179238 [<ffff0000080a4e30>] kvm_mmu_notifier_invalidate_range_start+0x70/0xe8 &(&kvm->mmu_lock)->rlock 19181 [<ffff0000080a7eec>] kvm_mmu_notifier_invalidate_range_end+0x24/0x68Quoted 1 line
.............................................................................................................................................................................................................................
[slots of stuff]
Well, the
mmu_lockis clearly contended. Is the box in a state where you are swapping? There seem to be as many faults as contentions, which is a bit surprising...
Quoted 5 lines
Quoted 1 line
I don't think it is swapping but need to double check.
It is the number of aborts that is staggering. And each one of them leads to the
mmu_lockbeing contended. So something seems to be taking its sweet time holding the damned lock.
Can you elaborate on the aborts, I'm not familiar with KVM but from a first look I thought kvm_handle_guest_abort() is in the normal path when a vcpu is stopped. Is that wrong?
--Jan
Quoted 32 lines · Jan Glauber
Quoted 1 line · Marc Zyngier
On 13/11/17 17:35, Jan Glauber wrote:
[...]
Quoted 27 lines
Quoted 26 lines
Quoted 18 lines
Quoted 18 lines
numbers don't look good, see waittime-max:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
class name con-bounces contentions waittime-min waittime-max waittime-total waittime-avg acq-bounces acquisitions holdtime-min holdtime-max holdtime-total holdtime-avg
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------&(&kvm->mmu_lock)->rlock: 99346764 99406604 0.14 1321260806.59 710654434972.0 7148.97 154228320 225122857 0.13 917688890.60 3705916481.39 16.46 ------------------------ &(&kvm->mmu_lock)->rlock 99365598 [<ffff0000080b43b8>] kvm_handle_guest_abort+0x4c0/0x950 &(&kvm->mmu_lock)->rlock 25164 [<ffff0000080a4e30>] kvm_mmu_notifier_invalidate_range_start+0x70/0xe8 &(&kvm->mmu_lock)->rlock 14934 [<ffff0000080a7eec>] kvm_mmu_notifier_invalidate_range_end+0x24/0x68 &(&kvm->mmu_lock)->rlock 908 [<ffff00000810a1f0>] __cond_resched_lock+0x68/0xb8 ------------------------ &(&kvm->mmu_lock)->rlock 3 [<ffff0000080b34c8>] stage2_flush_vm+0x60/0xd8 &(&kvm->mmu_lock)->rlock 99186296 [<ffff0000080b43b8>] kvm_handle_guest_abort+0x4c0/0x950 &(&kvm->mmu_lock)->rlock 179238 [<ffff0000080a4e30>] kvm_mmu_notifier_invalidate_range_start+0x70/0xe8 &(&kvm->mmu_lock)->rlock 19181 [<ffff0000080a7eec>] kvm_mmu_notifier_invalidate_range_end+0x24/0x68Quoted 6 lines
Quoted 1 line
.............................................................................................................................................................................................................................
[slots of stuff]
Well, the
mmu_lockis clearly contended. Is the box in a state where you are swapping? There seem to be as many faults as contentions, which is a bit surprising...I don't think it is swapping but need to double check.
Quoted 7 lines
Quoted 3 lines
It is the number of aborts that is staggering. And each one of them leads to the
mmu_lockbeing contended. So something seems to be taking its sweet time holding the damned lock.Can you elaborate on the aborts, I'm not familiar with KVM but from a first look I thought
kvm_handle_guest_abort()is in the normal path when a vcpu is stopped. Is that wrong?
kvm_handle_guest_abort() is the entry point for our page fault handling (hence the mmu_lock being taken). On its own, the number of faults is irrelevant. What worries me is that in almost all the cases the lock was contended, we were handling a page fault.
What would be interesting is to find out *who* is holding the lock when we're being blocked in kvm_handle_guest_abort...
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
Quoted 41 lines · Marc Zyngier
Quoted 32 lines · Jan Glauber
Quoted 1 line · Marc Zyngier
On 13/11/17 17:35, Jan Glauber wrote:
[...]
Quoted 27 lines
Quoted 26 lines
Quoted 18 lines
Quoted 18 lines
numbers don't look good, see waittime-max:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
class name con-bounces contentions waittime-min waittime-max waittime-total waittime-avg acq-bounces acquisitions holdtime-min holdtime-max holdtime-total holdtime-avg
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------&(&kvm->mmu_lock)->rlock: 99346764 99406604 0.14 1321260806.59 710654434972.0 7148.97 154228320 225122857 0.13 917688890.60 3705916481.39 16.46 ------------------------ &(&kvm->mmu_lock)->rlock 99365598 [<ffff0000080b43b8>] kvm_handle_guest_abort+0x4c0/0x950 &(&kvm->mmu_lock)->rlock 25164 [<ffff0000080a4e30>] kvm_mmu_notifier_invalidate_range_start+0x70/0xe8 &(&kvm->mmu_lock)->rlock 14934 [<ffff0000080a7eec>] kvm_mmu_notifier_invalidate_range_end+0x24/0x68 &(&kvm->mmu_lock)->rlock 908 [<ffff00000810a1f0>] __cond_resched_lock+0x68/0xb8 ------------------------ &(&kvm->mmu_lock)->rlock 3 [<ffff0000080b34c8>] stage2_flush_vm+0x60/0xd8 &(&kvm->mmu_lock)->rlock 99186296 [<ffff0000080b43b8>] kvm_handle_guest_abort+0x4c0/0x950 &(&kvm->mmu_lock)->rlock 179238 [<ffff0000080a4e30>] kvm_mmu_notifier_invalidate_range_start+0x70/0xe8 &(&kvm->mmu_lock)->rlock 19181 [<ffff0000080a7eec>] kvm_mmu_notifier_invalidate_range_end+0x24/0x68Quoted 6 lines
Quoted 1 line
.............................................................................................................................................................................................................................
[slots of stuff]
Well, the
mmu_lockis clearly contended. Is the box in a state where you are swapping? There seem to be as many faults as contentions, which is a bit surprising...I don't think it is swapping but need to double check.
Quoted 7 lines
Quoted 3 lines
It is the number of aborts that is staggering. And each one of them leads to the
mmu_lockbeing contended. So something seems to be taking its sweet time holding the damned lock.Can you elaborate on the aborts, I'm not familiar with KVM but from a first look I thought
kvm_handle_guest_abort()is in the normal path when a vcpu is stopped. Is that wrong?
Quoted 7 lines
kvm_handle_guest_abort()is the entry point for our page fault handling (hence themmu_lockbeing taken). On its own, the number of faults is irrelevant. What worries me is that in almost all the cases the lock was contended, we were handling a page fault.What would be interesting is to find out *who* is holding the lock when we're being blocked in
kvm_handle_guest_abort...
Just a thought, turning on the tracepoints for kvm_hva_* might help to get some more data on what we are doing with the HVA ranges.
Cheers
Suzuki
Quoted 25 lines · Jan Glauber
Quoted 2 lines · Marc Zyngier
Quoted 1 line · Jan Glauber
On Mon, Nov 13, 2017 at 01:47:38PM +0000, Marc Zyngier wrote:
[...]
Quoted 19 lines
Quoted 19 lines
Quoted 2 lines
Please elaborate. Messed in what way? Corrupted? The guest crashing? Or is that a tooling issue?
Every vcpu that oopses prints one line in parallel, so I get blocks like:
[58880.179814] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179834] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179847] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179873] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179893] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179911] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179917] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180288] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180303] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180336] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180363] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180384] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180415] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180461] [<ffff000008084b98>] ret_from_fork+0x10/0x18
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
I can send the full log if you want to have a look.
Sure, send that over (maybe not over email though).
Here is the guest dmesg:
http://paste.ubuntu.com/25955682/
Yeah, that's because all the vcpus are getting starved at the same time, and spitting out interleaved traces... Not very useful anyway, as I think this is only a consequence of what's happening on the host.
Quoted 3 lines
And the host dmesg as it might have been too big for the lists:
http://paste.ubuntu.com/25955699/
And that one doesn't show much either, apart from indicating that something is keeping the lock for itself. Drat.
We need to narrow down the problem, or make it appear on more common HW. Let me know if you've managed to reproduce it with non-VHE and/or on TX-1.
Thanks,
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
Quoted 38 lines · Marc Zyngier
Quoted 25 lines · Jan Glauber
Quoted 2 lines · Marc Zyngier
Quoted 1 line · Jan Glauber
On Mon, Nov 13, 2017 at 01:47:38PM +0000, Marc Zyngier wrote:
[...]
Quoted 19 lines
Quoted 19 lines
Quoted 2 lines
Please elaborate. Messed in what way? Corrupted? The guest crashing? Or is that a tooling issue?
Every vcpu that oopses prints one line in parallel, so I get blocks like:
[58880.179814] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179834] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179847] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179873] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179893] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179911] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.179917] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180288] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180303] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180336] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180363] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180384] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180415] [<ffff000008084b98>] ret_from_fork+0x10/0x18 [58880.180461] [<ffff000008084b98>] ret_from_fork+0x10/0x18Quoted 6 lines
Quoted 3 lines
Quoted 1 line
I can send the full log if you want to have a look.
Sure, send that over (maybe not over email though).
Here is the guest dmesg:
http://paste.ubuntu.com/25955682/Yeah, that's because all the vcpus are getting starved at the same time, and spitting out interleaved traces... Not very useful anyway, as I think this is only a consequence of what's happening on the host.
Quoted 1 line
Quoted 8 lines
Quoted 2 lines
And the host dmesg as it might have been too big for the lists:
http://paste.ubuntu.com/25955699/And that one doesn't show much either, apart from indicating that something is keeping the lock for itself. Drat.
We need to narrow down the problem, or make it appear on more common HW. Let me know if you've managed to reproduce it with non-VHE and/or on TX-1.
It also shows up when I disable VHE (CONFIG_ARM64_VHE). I'll try enabling some tracepoints next.
--Jan