powerpc: kvm: fix rare but potential deadlock scene
Subsystems (3)
PATCH RESEND v4 1/1 powerpc: kvm: fix rare but potential deadlock scene
Since kvmppc_hv_find_lock_hpte() is called from both virtmode and realmode, so it can trigger the deadlock.
Suppose the following scene:
Two physical cpuM, cpuN, two VM instances A, B, each VM has a group of vcpus.
If on cpuM, vcpu_A_1 holds bitlock X (HPTE_V_HVLOCK), then is switched out, and on cpuN, vcpu_A_2 try to lock X in realmode, then cpuN will be caught in realmode for a long time.
What makes things even worse if the following happens,
On cpuM, bitlockX is hold, on cpuN, Y is hold. vcpu_B_2 try to lock Y on cpuM in realmode vcpu_A_2 try to lock X on cpuN in realmode
Oops! deadlock happens
--- arch/powerpc/kvm/book3s_64_mmu_hv.c | 6 +++++- arch/powerpc/kvm/book3s_hv_rm_mmu.c | 4 ++++ 2 files changed, 9 insertions(+), 1 deletion(-)
arch/powerpc/kvm/book3s_64_mmu_hv.c +5 -1
@@ -473,11 +473,14 @@ static int kvmppc_mmu_book3s_64_hv_xlate(struct kvm_vcpu *vcpu, gva_t eaddr, slb_v = vcpu->kvm->arch.vrma_slb_v; } +preempt_disable ();/* Find the HPTE in the hash table */ index =kvmppc_hv_find_lock_hpte (kvm, eaddr, slb_v,HPTE_V_VALID |HPTE_V_ABSENT ); -if (index <0 ) +if (index <0 ) { +preempt_enable ();return -ENOENT ; + } hptep = (unsigned long *)(kvm->arch.hpt_virt + (index <<4 )); v = hptep[0 ] & ~HPTE_V_HVLOCK ; gr = kvm->arch.revmap[index].guest_rpte;
@@ -485,6 +488,7 @@ static int kvmppc_mmu_book3s_64_hv_xlate(struct kvm_vcpu *vcpu, gva_t eaddr,/* Unlock the HPTE */ asmvolatile ("lwsync" : : :"memory" ); hptep[0 ] = v; +preempt_enable (); gpte->eaddr = eaddr; gpte->vpage = ((v &HPTE_V_AVPN ) <<4 ) | ((eaddr >>12 ) &0xfff );
arch/powerpc/kvm/book3s_hv_rm_mmu.c +4 -0
@@ -749,6 +749,10 @@ static int slb_base_page_shift[4] = { 20, /* 1M, unsupported */ }; +/* When called from virtmode, this func should be protected by +* preempt_disable(), otherwise, the holding of HPTE_V_HVLOCK +* can trigger deadlock issue. +*/ long kvmppc_hv_find_lock_hpte (struct kvm *kvm,gva_t eaddr,unsigned long slb_v,unsigned long valid) {
Signature · 1 line
1.8.1.4
PATCH v3 1/1 powerpc: kvm: optimize "sc 1" as fast return
In some scene, e.g openstack CI, PR guest can trigger "sc 1" frequently, this patch optimizes the path by directly delivering BOOK3S_INTERRUPT_SYSCALL to HV guest, so powernv can return to HV guest without heavy exit, i.e, no need to swap TLB, HTAB,.. etc
---
v3: add some document
--- arch/powerpc/kvm/book3s_hv.c | 10 ++++------ arch/powerpc/kvm/book3s_hv_rmhandlers.S | 19 ++++++++++++++++++- 2 files changed, 22 insertions(+), 7 deletions(-)
arch/powerpc/kvm/book3s_hv.c +4 -6
@@ -628,12 +628,10 @@ static int kvmppc_handle_exit(struct kvm_run *run, struct kvm_vcpu *vcpu,/* hcall - punt to userspace */ int i; -if (vcpu->arch.shregs.msr &MSR_PR ) { -/* sc 1 from userspace - reflect to guest syscall */ -kvmppc_book3s_queue_irqprio (vcpu,BOOK3S_INTERRUPT_SYSCALL ); - r =RESUME_GUEST ; -break ; - } +/* hypercall with MSR_PR has already been handled in rmode, +* and never reaches here. +*/ + run->papr_hcall.nr =kvmppc_get_gpr (vcpu,3 );for (i =0 ; i <9 ; ++i) run->papr_hcall.args[i] =kvmppc_get_gpr (vcpu,4 + i);
arch/powerpc/kvm/book3s_hv_rmhandlers.S +18 -1
@@ -534,6 +534,13 @@ END_FTR_SECTION_IFSET(CPU_FTR_ARCH_206) 5: mtspr SPRN_SRR0, r6 mtspr SPRN_SRR1, r7 +/* + * Required state: + * R4 = vcpu + * R10: value for HSRR0 + * R11: value for HSRR1 + * R13 = PACA + */ fast_guest_return: li r0,0 stb r0,VCPU_CEDED(r4) /* cancel cede */
@@ -1388,7 +1395,8 @@ kvmppc_hisi: hcall_try_real_mode: ld r3,VCPU_GPR(R3)(r9) andi. r0,r11,MSR_PR - bne guest_exit_cont + /* sc 1 from userspace - reflect to guest syscall */ + bne sc_1_fast_return clrrdi r3,r3,2 cmpldi r3,hcall_real_table_end - hcall_real_table bge guest_exit_cont
@@ -1409,6 +1417,15 @@ hcall_try_real_mode: ld r11,VCPU_MSR(r4) b fast_guest_return +sc_1_fast_return: + mtspr SPRN_SRR0,r10 + mtspr SPRN_SRR1,r11 + li r10, BOOK3S_INTERRUPT_SYSCALL + li r11, (MSR_ME << 1) | 1 /* synthesize MSR_SF | MSR_ME */ + rotldi r11, r11, 63 + mr r4,r9 + b fast_guest_return + /* We've attempted a real mode hcall, but it's punted it back * to userspace. We need to restore some clobbered volatiles * before resuming the pass-it-to-qemu path */
Signature · 1 line
1.8.1.4
Alexander Graf
AppliedRe: [PATCH v3] powerpc: kvm: optimize "sc 1" as fast return
Quoted 8 lines · Liu Ping Fan
In some scene, e.g openstack CI, PR guest can trigger "sc 1" frequently, this patch optimizes the path by directly delivering
BOOK3S_INTERRUPT_SYSCALLto HV guest, so powernv can return to HV guest without heavy exit, i.e, no need to swap TLB, HTAB,.. etcSigned-off-by Liu Ping Fan <[email protected]>---
v3: add some document
This is v4 I think
Quoted 12 lines
---
arch/powerpc/kvm/book3s_hv.c| 10 ++++------arch/powerpc/kvm/book3s_hv_rmhandlers.S| 19 ++++++++++++++++++-2 files changed, 22 insertions(+), 7 deletions(-)arch/powerpc/kvm/book3s_hv.c +0
-0@@ -628,12 +628,10 @@ static int kvmppc_handle_exit(struct kvm_run *run, struct kvm_vcpu *vcpu,/* hcall - punt to userspace */ int i;
arch/powerpc/kvm/book3s_hv.c static int kvmppc_handle_exit(struct kvm_run *run, struct kvm_vcpu *vcpu, · line 633
Quoted 10 lines
arch/powerpc/kvm/book3s_hv.c +3
-6-if (vcpu->arch.shregs.msr &MSR_PR ) { -/* sc 1 from userspace - reflect to guest syscall */ -kvmppc_book3s_queue_irqprio (vcpu,BOOK3S_INTERRUPT_SYSCALL ); - r =RESUME_GUEST ; -break ; - } +/* hypercall with MSR_PR has already been handled in rmode, +* and never reaches here. +*/
It would've been nice to also mention the real mode hypercall handling, but I can just post a follow-up patch for that one.
Thanks, applied to kvm-ppc-queue.
Alex
Quoted 12 lines · Liu Ping Fan
Since
kvmppc_hv_find_lock_hpte()is called from both virtmode and realmode, so it can trigger the deadlock.Suppose the following scene:
Two physical cpuM, cpuN, two VM instances A, B, each VM has a group of vcpus.
If on cpuM,
vcpu_A_1holds bitlock X (HPTE_V_HVLOCK), then is switched out, and on cpuN,vcpu_A_2try to lock X in realmode, then cpuN will be caught in realmode for a long time.
Quoted 8 lines
What makes things even worse if the following happens,
On cpuM, bitlockX is hold, on cpuN, Y is hold.vcpu_B_2try to lock Y on cpuM in realmodevcpu_A_2try to lock X on cpuN in realmodeOops! deadlock happens
Signed-off-by Liu Ping Fan <[email protected]>
Any particular reason for the resend? The patch is already applied, no?
Alex
Quoted 20 lines · Alexander Graf
Quoted 17 lines · Liu Ping Fan
Since
kvmppc_hv_find_lock_hpte()is called from both virtmode and realmode, so it can trigger the deadlock.Suppose the following scene:
Two physical cpuM, cpuN, two VM instances A, B, each VM has a group of vcpus.
If on cpuM,
vcpu_A_1holds bitlock X (HPTE_V_HVLOCK), then is switched out, and on cpuN,vcpu_A_2try to lock X in realmode, then cpuN will be caught in realmode for a long time.What makes things even worse if the following happens,
On cpuM, bitlockX is hold, on cpuN, Y is hold.vcpu_B_2try to lock Y on cpuM in realmodevcpu_A_2try to lock X on cpuN in realmode
Quoted 6 lines
Quoted 3 lines
Oops! deadlock happens
Signed-off-by Liu Ping Fan <[email protected]>Any particular reason for the resend? The patch is already applied, no?
Oh, seems that I misunderstood your meaning. You said "Actually, I've changed my mind and moved the patch to the for-3.13 branch instead. Please make sure to CC kvm@vger on all patches you submit though". So I think it is necessary to resend with cc kvm@vger
Regards,
Pingfan
Quoted 27 lines
Am 20.11.2013 um 03:42 schrieb Liu ping fan <[email protected]>:
Quoted 23 lines · Alexander Graf
Quoted 22 lines · Liu Ping Fan
Since
kvmppc_hv_find_lock_hpte()is called from both virtmode and realmode, so it can trigger the deadlock.Suppose the following scene:
Two physical cpuM, cpuN, two VM instances A, B, each VM has a group of vcpus.
If on cpuM,
vcpu_A_1holds bitlock X (HPTE_V_HVLOCK), then is switched out, and on cpuN,vcpu_A_2try to lock X in realmode, then cpuN will be caught in realmode for a long time.What makes things even worse if the following happens, On cpuM, bitlockX is hold, on cpuN, Y is hold.
vcpu_B_2try to lock Y on cpuM in realmodevcpu_A_2try to lock X on cpuN in realmodeOops! deadlock happens
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Signed-off-by Liu Ping Fan <[email protected]>Any particular reason for the resend? The patch is already applied, no?
Oh, seems that I misunderstood your meaning. You said "Actually, I've changed my mind and moved the patch to the for-3.13 branch instead. Please make sure to CC kvm@vger on all patches you submit though". So I think it is necessary to resend with cc kvm@vger
Oh. I meant "for next time" :). But now that it happened for this patch already it's even better.
Thanks a lot!
Alex
Quoted 3 lines
Regards,
Pingfan