powerpc/kvm: simplify the entering logic for secondary thread
Subsystems (2)
PATCH 1/3 powerpc/kvm: simplify the entering logic for secondary thread
After the primary vcpu changes vcore_state to VCORE_RUNNING, there is very little chance to schedule to secondary vcpu. So if we change the code sequence around set vcore_state to VCORE_RUNNING and disable preemption, we lost little. But we simplify the entering logi, based on the fact that if primary vcpu runs, the secondary vcpu can not be scheduled.
--- arch/powerpc/kvm/book3s_hv.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-)
arch/powerpc/kvm/book3s_hv.c +2 -7
@@ -1222,8 +1222,8 @@ static void kvmppc_run_core(struct kvmppc_vcore *vc)kvmppc_create_dtl_entry (vcpu, vc); } - vc->vcore_state =VCORE_RUNNING ;preempt_disable (); + vc->vcore_state =VCORE_RUNNING ;spin_unlock (&vc->lock);kvm_guest_enter ();
@@ -1351,12 +1351,7 @@ static int kvmppc_run_vcpu(struct kvm_run *kvm_run, struct kvm_vcpu *vcpu)* this thread straight away and have it join in. */ if (!signal_pending (current)) { -if (vc->vcore_state ==VCORE_RUNNING && -VCORE_EXIT_COUNT (vc) ==0 ) { - vcpu->arch.ptid = vc->n_runnable -1 ; -kvmppc_create_dtl_entry (vcpu, vc); -kvmppc_start_thread (vcpu); - }else if (vc->vcore_state ==VCORE_SLEEPING ) { +if (vc->vcore_state ==VCORE_SLEEPING ) {wake_up (&vc->wq); }
Signature · 1 line
1.8.1.4
PATCH 2/3 powerpc/kvm: fix rare but potential deadlock scene
Since kvmppc_hv_find_lock_hpte() is called from both virtmode and realmode, so it can trigger the deadlock.
Suppose the following scene:
Two physical cpuM, cpuN, two VM instances A, B, each VM has a group of vcpus.
If on cpuM, vcpu_A_1 holds bitlock X (HPTE_V_HVLOCK), then is switched out, and on cpuN, vcpu_A_2 try to lock X in realmode, then cpuN will be caught in realmode for a long time.
What makes things even worse if the following happens,
On cpuM, bitlockX is hold, on cpuN, Y is hold. vcpu_B_2 try to lock Y on cpuM in realmode vcpu_A_2 try to lock X on cpuN in realmode
Oops! deadlock happens
--- arch/powerpc/kvm/book3s_64_mmu_hv.c | 2 ++ 1 file changed, 2 insertions(+)
arch/powerpc/kvm/book3s_64_mmu_hv.c +2 -0
@@ -474,8 +474,10 @@ static int kvmppc_mmu_book3s_64_hv_xlate(struct kvm_vcpu *vcpu, gva_t eaddr, }/* Find the HPTE in the hash table */ +preempt_disable (); index =kvmppc_hv_find_lock_hpte (kvm, eaddr, slb_v,HPTE_V_VALID |HPTE_V_ABSENT ); +preempt_enable ();if (index <0 )return -ENOENT ; hptep = (unsigned long *)(kvm->arch.hpt_virt + (index <<4 ));
Signature · 1 line
1.8.1.4
Re: [PATCH 2/3] powerpc/kvm: fix rare but potential deadlock scene
Quoted 2 lines · Liu Ping Fan
Since
kvmppc_hv_find_lock_hpte()is called from both virtmode and realmode, so it can trigger the deadlock.
Good catch, we should have preemption disabled while ever we have a HPTE locked.
arch/powerpc/kvm/book3s_64_mmu_hv.c static int kvmppc_mmu_book3s_64_hv_xlate(struct kvm_vcpu *vcpu, gva_t eaddr, · line 480
Quoted 8 lines
arch/powerpc/kvm/book3s_64_mmu_hv.c +2
-0@@ -474,8 +474,10 @@ static int kvmppc_mmu_book3s_64_hv_xlate(struct kvm_vcpu *vcpu, gva_t eaddr, }/* Find the HPTE in the hash table */ +preempt_disable (); index =kvmppc_hv_find_lock_hpte (kvm, eaddr, slb_v,HPTE_V_VALID |HPTE_V_ABSENT ); +preempt_enable ();
Which means we need to add the preempt_enable after unlocking the HPTE, not here.
Regards,
Paul.
Re: [PATCH 2/3] powerpc/kvm: fix rare but potential deadlock scene
Quoted 10 lines · Paul Mackerras
Quoted 2 lines · Liu Ping Fan
Since
kvmppc_hv_find_lock_hpte()is called from both virtmode and realmode, so it can trigger the deadlock.Good catch, we should have preemption disabled while ever we have a HPTE locked.
Quoted 3 lines
(quoted diff) +0
-0@@ -474,8 +474,10 @@ static int kvmppc_mmu_book3s_64_hv_xlate(struct kvm_vcpu *vcpu, gva_t eaddr, }
Quoted 9 lines
arch/powerpc/kvm/book3s_64_mmu_hv.c static int kvmppc_mmu_book3s_64_hv_xlate(struct kvm_vcpu *vcpu, gva_t eaddr, · line 480
Quoted 5 lines
arch/powerpc/kvm/book3s_64_mmu_hv.c +2
-0/* Find the HPTE in the hash table */ +preempt_disable (); index =kvmppc_hv_find_lock_hpte (kvm, eaddr, slb_v,HPTE_V_VALID |HPTE_V_ABSENT ); +preempt_enable ();Which means we need to add the
preempt_enableafter unlocking the HPTE, not here.
Yes. Sorry, but I am not sure about whether we can call preempt_disable/enable() in realmode. I think since thread_info is allocated with linear address, so we can use preempt_disable/enable() inside kvmppc_hv_find_lock_hpte(), right?
Thanks and regards,
Pingfan
Quoted 2 lines
Regards,
Paul.
Re: [PATCH 2/3] powerpc/kvm: fix rare but potential deadlock scene
Quoted 20 lines · Liu ping fan
Quoted 10 lines · Paul Mackerras
Quoted 2 lines · Liu Ping Fan
Since
kvmppc_hv_find_lock_hpte()is called from both virtmode and realmode, so it can trigger the deadlock.Good catch, we should have preemption disabled while ever we have a HPTE locked.
Quoted 3 lines
(quoted diff) +0
-0@@ -474,8 +474,10 @@ static int kvmppc_mmu_book3s_64_hv_xlate(struct kvm_vcpu *vcpu, gva_t eaddr, }Quoted 9 lines
arch/powerpc/kvm/book3s_64_mmu_hv.c static int kvmppc_mmu_book3s_64_hv_xlate(struct kvm_vcpu *vcpu, gva_t eaddr, · line 480
Quoted 5 lines
arch/powerpc/kvm/book3s_64_mmu_hv.c +2
-0/* Find the HPTE in the hash table */ +preempt_disable (); index =kvmppc_hv_find_lock_hpte (kvm, eaddr, slb_v,HPTE_V_VALID |HPTE_V_ABSENT ); +preempt_enable ();Which means we need to add the
preempt_enableafter unlocking the HPTE, not here.
Quoted 4 lines
Yes. Sorry, but I am not sure about whether we can call preempt_disable/enable() in realmode. I think since
thread_infois allocated with linear address, so we can use preempt_disable/enable() insidekvmppc_hv_find_lock_hpte(), right?
Your analysis correctly pointed out that we can get a deadlock if we can be preempted while holding a lock on a HPTE. That means that we have to disable preemption before taking an HPTE lock and keep it disabled until after we unlock the HPTE. Since the point of kvmppc_hv_find_lock_hpte() is to lock the HPTE and return with it locked, we can't have the preempt_enable() inside it. The preempt_enable() has to come after we have unlocked the HPTE. That is also why we can't have the preempt_enable() where your patch put it; it needs to be about 9 lines further down, after the statement hptep[0] = v. (We also need to make sure to re-enable preemption in the index < 0 case.)
Regards,
Paul.
Re: [PATCH 2/3] powerpc/kvm: fix rare but potential deadlock scene
Quoted 26 lines · Paul Mackerras
Quoted 20 lines · Liu ping fan
Quoted 10 lines · Paul Mackerras
Quoted 2 lines · Liu Ping Fan
Since
kvmppc_hv_find_lock_hpte()is called from both virtmode and realmode, so it can trigger the deadlock.Good catch, we should have preemption disabled while ever we have a HPTE locked.
Quoted 3 lines
(quoted diff) +0
-0@@ -474,8 +474,10 @@ static int kvmppc_mmu_book3s_64_hv_xlate(struct kvm_vcpu *vcpu, gva_t eaddr, }Quoted 9 lines
arch/powerpc/kvm/book3s_64_mmu_hv.c static int kvmppc_mmu_book3s_64_hv_xlate(struct kvm_vcpu *vcpu, gva_t eaddr, · line 480
Quoted 5 lines
arch/powerpc/kvm/book3s_64_mmu_hv.c +2
-0/* Find the HPTE in the hash table */ +preempt_disable (); index =kvmppc_hv_find_lock_hpte (kvm, eaddr, slb_v,HPTE_V_VALID |HPTE_V_ABSENT ); +preempt_enable ();Which means we need to add the
preempt_enableafter unlocking the HPTE, not here.Quoted 4 lines
Yes. Sorry, but I am not sure about whether we can call preempt_disable/enable() in realmode. I think since
thread_infois allocated with linear address, so we can use preempt_disable/enable() insidekvmppc_hv_find_lock_hpte(), right?
Quoted 12 lines
Your analysis correctly pointed out that we can get a deadlock if we can be preempted while holding a lock on a HPTE. That means that we have to disable preemption before taking an HPTE lock and keep it disabled until after we unlock the HPTE. Since the point of
kvmppc_hv_find_lock_hpte()is to lock the HPTE and return with it locked, we can't have thepreempt_enable()inside it. Thepreempt_enable()has to come after we have unlocked the HPTE. That is also why we can't have thepreempt_enable()where your patch put it; it needs to be about 9 lines further down, after the statement hptep[0] = v. (We also need to make sure to re-enable preemption in the index < 0 case.)
Oh, yes, will fix like what you said. My attention is attracted by the trick of calling kernel func in realmode, and miss the exact point where the lock is released.
Thanks and regards,
Pingfan
PATCH 3/3 powerpc/kvm: remove redundant assignment
In reply to Liu Ping Fan--- arch/powerpc/kvm/book3s_64_mmu_hv.c | 1 - 1 file changed, 1 deletion(-)
arch/powerpc/kvm/book3s_64_mmu_hv.c +0 -1
@@ -731,7 +731,6 @@ int kvmppc_book3s_hv_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu,lock_rmap (rmap);/* Check if we might have been invalidated; let the guest retry if so */ - ret =RESUME_GUEST ;if (mmu_notifier_retry (vcpu->kvm, mmu_seq)) {unlock_rmap (rmap); goto out_unlock;
Signature · 1 line
1.8.1.4
Re: [PATCH 3/3] powerpc/kvm: remove redundant assignment
Quoted 10 lines · Liu Ping Fan
Signed-off-by Liu Ping Fan <[email protected]>--- arch/powerpc/kvm/book3s_64_mmu_hv.c | 1 - 1 file changed, 1 deletion(-)arch/powerpc/kvm/book3s_64_mmu_hv.c +0
-0
arch/powerpc/kvm/book3s_64_mmu_hv.c int kvmppc_book3s_hv_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu, · line 736
Quoted 7 lines
arch/powerpc/kvm/book3s_64_mmu_hv.c +0
-1@@ -731,7 +731,6 @@ int kvmppc_book3s_hv_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu,lock_rmap (rmap);/* Check if we might have been invalidated; let the guest retry if so */ - ret =RESUME_GUEST ;if (mmu_notifier_retry (vcpu->kvm, mmu_seq)) {unlock_rmap (rmap); goto out_unlock;
Re: [PATCH 3/3] powerpc/kvm: remove redundant assignment
In reply to Liu Ping FanQuoted 1 line · Liu Ping Fan
Signed-off-by Liu Ping Fan <[email protected]>
Patch description missing.
Please add Paul's ack in the next revision of this patch :).
Alex
Quoted 23 lines
---
arch/powerpc/kvm/book3s_64_mmu_hv.c| 1 -1 file changed, 1 deletion(-)arch/powerpc/kvm/book3s_64_mmu_hv.c +0
-1@@ -731,7 +731,6 @@ int kvmppc_book3s_hv_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu,lock_rmap (rmap);/* Check if we might have been invalidated; let the guest retry if so */ - ret =RESUME_GUEST ;if (mmu_notifier_retry (vcpu->kvm, mmu_seq)) {unlock_rmap (rmap); goto out_unlock;Signature · 6 lines
1.8.1.4 -- To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 3/3] powerpc/kvm: remove redundant assignment
Quoted 5 lines · Alexander Graf
Quoted 1 line · Liu Ping Fan
Signed-off-by Liu Ping Fan <[email protected]>Patch description missing.
Do you really need a description for trivial one-lines whose subject is a perfectly complete description already ?
Quoted 28 lines
Please add Paul's ack in the next revision of this patch :).
Alex
Quoted 23 lines
---
arch/powerpc/kvm/book3s_64_mmu_hv.c| 1 -1 file changed, 1 deletion(-)arch/powerpc/kvm/book3s_64_mmu_hv.c +0
-1@@ -731,7 +731,6 @@ int kvmppc_book3s_hv_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu,lock_rmap (rmap);/* Check if we might have been invalidated; let the guest retry if so */ - ret =RESUME_GUEST ;if (mmu_notifier_retry (vcpu->kvm, mmu_seq)) {unlock_rmap (rmap); goto out_unlock;Signature · 6 lines
1.8.1.4 -- To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 3/3] powerpc/kvm: remove redundant assignment
Am 06.11.2013 um 20:58 schrieb Benjamin Herrenschmidt <[email protected]>:
Quoted 9 lines · Alexander Graf
Quoted 5 lines · Liu Ping Fan
Quoted 2 lines
Signed-off-by Liu Ping Fan <[email protected]>Patch description missing.
Do you really need a description for trivial one-lines whose subject is a perfectly complete description already ?
Would I ask for it otherwise? It's also not 100% obvious that the assignment is redundant.
Alex
Re: [PATCH 3/3] powerpc/kvm: remove redundant assignment
Quoted 5 lines · Alexander Graf
Am 06.11.2013 um 20:58 schrieb Benjamin Herrenschmidt <[email protected]>:
Quoted 3 lines · Alexander Graf
Quoted 2 lines · Liu Ping Fan
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Signed-off-by Liu Ping Fan <[email protected]>Patch description missing.
Do you really need a description for trivial one-lines whose subject is a perfectly complete description already ?
Would I ask for it otherwise? It's also not 100% obvious that the assignment is redundant.
And ? An explanation isn't going to be clearer than the code in that case ...
Ben.
Re: [PATCH 3/3] powerpc/kvm: remove redundant assignment
Am 07.11.2013 um 08:55 schrieb Benjamin Herrenschmidt <[email protected]>:
Quoted 10 lines · Alexander Graf
Quoted 9 lines
Am 06.11.2013 um 20:58 schrieb Benjamin Herrenschmidt <[email protected]>:
Quoted 7 lines · Alexander Graf
Quoted 5 lines · Liu Ping Fan
Quoted 2 lines
Signed-off-by Liu Ping Fan <[email protected]>Patch description missing.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Do you really need a description for trivial one-lines whose subject is a perfectly complete description already ?
Would I ask for it otherwise? It's also not 100% obvious that the assignment is redundant.
And ? An explanation isn't going to be clearer than the code in that case ...
It's pretty non-obvious when you do a git show on that patch in 1 year from now, as the redundancy is out of scope of what the diff shows.
Alex
Re: [PATCH 3/3] powerpc/kvm: remove redundant assignment
Quoted 5 lines · Alexander Graf
Quoted 2 lines
And ? An explanation isn't going to be clearer than the code in that case ...
It's pretty non-obvious when you do a git show on that patch in 1 year from now, as the redundancy is out of scope of what the diff shows.
And ? How would an explanation help ?
Either it's redundant or it's not ... but only look at the code can prove it. An explanation won't because if the patch is wrong, so will be the explanation.
Cheers,
Ben.
Quoted 2 lines · Liu Ping Fan
After the primary vcpu changes
vcore_statetoVCORE_RUNNING, there is very little chance to schedule to secondary vcpu. So if we change the
Why do you say there is very little chance to run the secondary vcpu?
Quoted 3 lines
code sequence around set
vcore_statetoVCORE_RUNNINGand disable preemption, we lost little. But we simplify the entering logi, based on the fact that if primary vcpu runs, the secondary vcpu can not be scheduled.
If a vcpu does a hypercall or something else that requires the host (kernel or userspace) to do something, that can happen in the context of the vcpu task for that vcpu. That vcpu task can run on another core (unless it has been pinned). When it is finished we would like the vcpu to continue executing in the guest as soon as possible. The code that you remove in this patch enables that to happen without having to wait until the other threads exit the guest. So I don't think it is a good idea to remove this code.
Regards,
Paul.
Quoted 10 lines · Paul Mackerras
Quoted 2 lines · Liu Ping Fan
After the primary vcpu changes
vcore_statetoVCORE_RUNNING, there is very little chance to schedule to secondary vcpu. So if we change theWhy do you say there is very little chance to run the secondary vcpu?
Quoted 3 lines
code sequence around set
vcore_statetoVCORE_RUNNINGand disable preemption, we lost little. But we simplify the entering logi, based on the fact that if primary vcpu runs, the secondary vcpu can not be scheduled.
Quoted 9 lines
If a vcpu does a hypercall or something else that requires the host (kernel or userspace) to do something, that can happen in the context of the vcpu task for that vcpu. That vcpu task can run on another core (unless it has been pinned). When it is finished we would like the vcpu to continue executing in the guest as soon as possible. The code that you remove in this patch enables that to happen without having to wait until the other threads exit the guest. So I don't think it is a good idea to remove this code.
Yes, you are right.
Thanks and regards,
Pingfan
Quoted 2 lines
Regards,
Paul.