Skip to contentSkip to conversation

powerpc/kvm: simplify the entering logic for secondary thread

PATCH In review Liu Ping Fan 4 participants 16 messages Nov 5, 2013 – Nov 7, 2013

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/3 powerpc/kvm: simplify the entering logic for secondary thread

After the primary vcpu changes vcore_state to VCORE_RUNNING, there is very little chance to schedule to secondary vcpu. So if we change the code sequence around set vcore_state to VCORE_RUNNING and disable preemption, we lost little. But we simplify the entering logi, based on the fact that if primary vcpu runs, the secondary vcpu can not be scheduled.

---
 arch/powerpc/kvm/book3s_hv.c | 9 ++-------
 1 file changed, 2 insertions(+), 7 deletions(-)
arch/powerpc/kvm/book3s_hv.c +2 -7
index 62a2b5a..38b1fc0 100644
@@ -1222,8 +1222,8 @@ static void kvmppc_run_core(struct kvmppc_vcore *vc)
 		kvmppc_create_dtl_entry(vcpu, vc);
 	}
 
-	vc->vcore_state = VCORE_RUNNING;
 	preempt_disable();
+	vc->vcore_state = VCORE_RUNNING;
 	spin_unlock(&vc->lock);
 
 	kvm_guest_enter();
@@ -1351,12 +1351,7 @@ static int kvmppc_run_vcpu(struct kvm_run *kvm_run, struct kvm_vcpu *vcpu)
 	 * this thread straight away and have it join in.
 	 */
 	if (!signal_pending(current)) {
-		if (vc->vcore_state == VCORE_RUNNING &&
-		    VCORE_EXIT_COUNT(vc) == 0) {
-			vcpu->arch.ptid = vc->n_runnable - 1;
-			kvmppc_create_dtl_entry(vcpu, vc);
-			kvmppc_start_thread(vcpu);
-		} else if (vc->vcore_state == VCORE_SLEEPING) {
+		if (vc->vcore_state == VCORE_SLEEPING) {
 			wake_up(&vc->wq);
 		}
 
Signature · 1 line
1.8.1.4
#1 lore raw

PATCH 2/3 powerpc/kvm: fix rare but potential deadlock scene

Since kvmppc_hv_find_lock_hpte() is called from both virtmode and realmode, so it can trigger the deadlock.

Suppose the following scene:

Two physical cpuM, cpuN, two VM instances A, B, each VM has a group of vcpus.

If on cpuM, vcpu_A_1 holds bitlock X (HPTE_V_HVLOCK), then is switched out, and on cpuN, vcpu_A_2 try to lock X in realmode, then cpuN will be caught in realmode for a long time.

What makes things even worse if the following happens,
On cpuM, bitlockX is hold, on cpuN, Y is hold. vcpu_B_2 try to lock Y on cpuM in realmode vcpu_A_2 try to lock X on cpuN in realmode

Oops! deadlock happens

---
 arch/powerpc/kvm/book3s_64_mmu_hv.c | 2 ++
 1 file changed, 2 insertions(+)
arch/powerpc/kvm/book3s_64_mmu_hv.c +2 -0
index 043eec8..28160ac 100644
@@ -474,8 +474,10 @@ static int kvmppc_mmu_book3s_64_hv_xlate(struct kvm_vcpu *vcpu, gva_t eaddr,
 	}
 
 	/* Find the HPTE in the hash table */
+	preempt_disable();
 	index = kvmppc_hv_find_lock_hpte(kvm, eaddr, slb_v,
 					 HPTE_V_VALID | HPTE_V_ABSENT);
+	preempt_enable();
 	if (index < 0)
 		return -ENOENT;
 	hptep = (unsigned long *)(kvm->arch.hpt_virt + (index << 4));
Signature · 1 line
1.8.1.4
#2 lore raw

Re: [PATCH 2/3] powerpc/kvm: fix rare but potential deadlock scene

Quoted 2 lines · Liu Ping Fan

Since kvmppc_hv_find_lock_hpte() is called from both virtmode and realmode, so it can trigger the deadlock.

Good catch, we should have preemption disabled while ever we have a HPTE locked.

arch/powerpc/kvm/book3s_64_mmu_hv.c static int kvmppc_mmu_book3s_64_hv_xlate(struct kvm_vcpu *vcpu, gva_t eaddr, · line 480

Quoted 8 lines
arch/powerpc/kvm/book3s_64_mmu_hv.c +2 -0
@@ -474,8 +474,10 @@ static int kvmppc_mmu_book3s_64_hv_xlate(struct kvm_vcpu *vcpu, gva_t eaddr,
 	}
 
 	/* Find the HPTE in the hash table */
+	preempt_disable();
 	index = kvmppc_hv_find_lock_hpte(kvm, eaddr, slb_v,
 					 HPTE_V_VALID | HPTE_V_ABSENT);
+	preempt_enable();

Which means we need to add the preempt_enable after unlocking the HPTE, not here.

Regards,
Paul.

#3 lore raw

Re: [PATCH 2/3] powerpc/kvm: fix rare but potential deadlock scene

Quoted 10 lines · Paul Mackerras
Quoted 2 lines · Liu Ping Fan

Since kvmppc_hv_find_lock_hpte() is called from both virtmode and realmode, so it can trigger the deadlock.

Good catch, we should have preemption disabled while ever we have a HPTE locked.

Quoted 3 lines
(quoted diff) +0 -0
@@ -474,8 +474,10 @@ static int kvmppc_mmu_book3s_64_hv_xlate(struct kvm_vcpu *vcpu, gva_t eaddr,
      }
 
Quoted 9 lines

arch/powerpc/kvm/book3s_64_mmu_hv.c static int kvmppc_mmu_book3s_64_hv_xlate(struct kvm_vcpu *vcpu, gva_t eaddr, · line 480

Quoted 5 lines
arch/powerpc/kvm/book3s_64_mmu_hv.c +2 -0
      /* Find the HPTE in the hash table */
+     preempt_disable();
      index = kvmppc_hv_find_lock_hpte(kvm, eaddr, slb_v,
                                       HPTE_V_VALID | HPTE_V_ABSENT);
+     preempt_enable();

Which means we need to add the preempt_enable after unlocking the HPTE, not here.

Yes. Sorry, but I am not sure about whether we can call preempt_disable/enable() in realmode. I think since thread_info is allocated with linear address, so we can use preempt_disable/enable() inside kvmppc_hv_find_lock_hpte(), right?

Thanks and regards,
Pingfan

Quoted 2 lines

Regards,
Paul.

#4 lore raw

Re: [PATCH 2/3] powerpc/kvm: fix rare but potential deadlock scene

Quoted 20 lines · Liu ping fan
Quoted 10 lines · Paul Mackerras
Quoted 2 lines · Liu Ping Fan

Since kvmppc_hv_find_lock_hpte() is called from both virtmode and realmode, so it can trigger the deadlock.

Good catch, we should have preemption disabled while ever we have a HPTE locked.

Quoted 3 lines
(quoted diff) +0 -0
@@ -474,8 +474,10 @@ static int kvmppc_mmu_book3s_64_hv_xlate(struct kvm_vcpu *vcpu, gva_t eaddr,
      }
 
Quoted 9 lines

arch/powerpc/kvm/book3s_64_mmu_hv.c static int kvmppc_mmu_book3s_64_hv_xlate(struct kvm_vcpu *vcpu, gva_t eaddr, · line 480

Quoted 5 lines
arch/powerpc/kvm/book3s_64_mmu_hv.c +2 -0
      /* Find the HPTE in the hash table */
+     preempt_disable();
      index = kvmppc_hv_find_lock_hpte(kvm, eaddr, slb_v,
                                       HPTE_V_VALID | HPTE_V_ABSENT);
+     preempt_enable();

Which means we need to add the preempt_enable after unlocking the HPTE, not here.

Quoted 4 lines

Yes. Sorry, but I am not sure about whether we can call preempt_disable/enable() in realmode. I think since thread_info is allocated with linear address, so we can use preempt_disable/enable() inside kvmppc_hv_find_lock_hpte(), right?

Your analysis correctly pointed out that we can get a deadlock if we can be preempted while holding a lock on a HPTE. That means that we have to disable preemption before taking an HPTE lock and keep it disabled until after we unlock the HPTE. Since the point of kvmppc_hv_find_lock_hpte() is to lock the HPTE and return with it locked, we can't have the preempt_enable() inside it. The preempt_enable() has to come after we have unlocked the HPTE. That is also why we can't have the preempt_enable() where your patch put it; it needs to be about 9 lines further down, after the statement hptep[0] = v. (We also need to make sure to re-enable preemption in the index < 0 case.)

Regards,
Paul.

#5 lore raw

Re: [PATCH 2/3] powerpc/kvm: fix rare but potential deadlock scene

Quoted 26 lines · Paul Mackerras
Quoted 20 lines · Liu ping fan
Quoted 10 lines · Paul Mackerras
Quoted 2 lines · Liu Ping Fan

Since kvmppc_hv_find_lock_hpte() is called from both virtmode and realmode, so it can trigger the deadlock.

Good catch, we should have preemption disabled while ever we have a HPTE locked.

Quoted 3 lines
(quoted diff) +0 -0
@@ -474,8 +474,10 @@ static int kvmppc_mmu_book3s_64_hv_xlate(struct kvm_vcpu *vcpu, gva_t eaddr,
      }
 
Quoted 9 lines

arch/powerpc/kvm/book3s_64_mmu_hv.c static int kvmppc_mmu_book3s_64_hv_xlate(struct kvm_vcpu *vcpu, gva_t eaddr, · line 480

Quoted 5 lines
arch/powerpc/kvm/book3s_64_mmu_hv.c +2 -0
      /* Find the HPTE in the hash table */
+     preempt_disable();
      index = kvmppc_hv_find_lock_hpte(kvm, eaddr, slb_v,
                                       HPTE_V_VALID | HPTE_V_ABSENT);
+     preempt_enable();

Which means we need to add the preempt_enable after unlocking the HPTE, not here.

Quoted 4 lines

Yes. Sorry, but I am not sure about whether we can call preempt_disable/enable() in realmode. I think since thread_info is allocated with linear address, so we can use preempt_disable/enable() inside kvmppc_hv_find_lock_hpte(), right?

Quoted 12 lines

Your analysis correctly pointed out that we can get a deadlock if we can be preempted while holding a lock on a HPTE. That means that we have to disable preemption before taking an HPTE lock and keep it disabled until after we unlock the HPTE. Since the point of kvmppc_hv_find_lock_hpte() is to lock the HPTE and return with it locked, we can't have the preempt_enable() inside it. The preempt_enable() has to come after we have unlocked the HPTE. That is also why we can't have the preempt_enable() where your patch put it; it needs to be about 9 lines further down, after the statement hptep[0] = v. (We also need to make sure to re-enable preemption in the index < 0 case.)

Oh, yes, will fix like what you said. My attention is attracted by the trick of calling kernel func in realmode, and miss the exact point where the lock is released.

Thanks and regards,
Pingfan

#6 lore raw

PATCH 3/3 powerpc/kvm: remove redundant assignment

In reply to Liu Ping Fan
---
 arch/powerpc/kvm/book3s_64_mmu_hv.c | 1 -
 1 file changed, 1 deletion(-)
arch/powerpc/kvm/book3s_64_mmu_hv.c +0 -1
index 28160ac..7682837 100644
@@ -731,7 +731,6 @@ int kvmppc_book3s_hv_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu,
 	lock_rmap(rmap);
 
 	/* Check if we might have been invalidated; let the guest retry if so */
-	ret = RESUME_GUEST;
 	if (mmu_notifier_retry(vcpu->kvm, mmu_seq)) {
 		unlock_rmap(rmap);
 		goto out_unlock;
Signature · 1 line
1.8.1.4
Acked-by on v1 3/3
#7 lore raw

Re: [PATCH 3/3] powerpc/kvm: remove redundant assignment

Quoted 10 lines · Liu Ping Fan
---
 arch/powerpc/kvm/book3s_64_mmu_hv.c | 1 -
 1 file changed, 1 deletion(-)
arch/powerpc/kvm/book3s_64_mmu_hv.c +0 -0
index 28160ac..7682837 100644

arch/powerpc/kvm/book3s_64_mmu_hv.c int kvmppc_book3s_hv_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu, · line 736

Quoted 7 lines
arch/powerpc/kvm/book3s_64_mmu_hv.c +0 -1
@@ -731,7 +731,6 @@ int kvmppc_book3s_hv_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu,
 	lock_rmap(rmap);
 
 	/* Check if we might have been invalidated; let the guest retry if so */
-	ret = RESUME_GUEST;
 	if (mmu_notifier_retry(vcpu->kvm, mmu_seq)) {
 		unlock_rmap(rmap);
 		goto out_unlock;
#8 lore raw

Re: [PATCH 3/3] powerpc/kvm: remove redundant assignment

In reply to Liu Ping Fan
Quoted 1 line · Liu Ping Fan

Patch description missing.

Please add Paul's ack in the next revision of this patch :).

Alex

Quoted 23 lines

---
arch/powerpc/kvm/book3s_64_mmu_hv.c | 1 -

1 file changed, 1 deletion(-)
arch/powerpc/kvm/book3s_64_mmu_hv.c +0 -1
index 28160ac..7682837 100644
@@ -731,7 +731,6 @@ int kvmppc_book3s_hv_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu,
 	lock_rmap(rmap);
 
 	/* Check if we might have been invalidated; let the guest retry if so */
-	ret = RESUME_GUEST;
 	if (mmu_notifier_retry(vcpu->kvm, mmu_seq)) {
 		unlock_rmap(rmap);
 		goto out_unlock;
Signature · 6 lines
1.8.1.4

--
To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#9 lore raw

Re: [PATCH 3/3] powerpc/kvm: remove redundant assignment

Quoted 5 lines · Alexander Graf
Quoted 1 line · Liu Ping Fan

Patch description missing.

Do you really need a description for trivial one-lines whose subject is a perfectly complete description already ?

Quoted 28 lines

Please add Paul's ack in the next revision of this patch :).

Alex

Quoted 23 lines

---
arch/powerpc/kvm/book3s_64_mmu_hv.c | 1 -

1 file changed, 1 deletion(-)
arch/powerpc/kvm/book3s_64_mmu_hv.c +0 -1
index 28160ac..7682837 100644
@@ -731,7 +731,6 @@ int kvmppc_book3s_hv_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu,
 	lock_rmap(rmap);
 
 	/* Check if we might have been invalidated; let the guest retry if so */
-	ret = RESUME_GUEST;
 	if (mmu_notifier_retry(vcpu->kvm, mmu_seq)) {
 		unlock_rmap(rmap);
 		goto out_unlock;
Signature · 6 lines
1.8.1.4

--
To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#10 lore raw

Re: [PATCH 3/3] powerpc/kvm: remove redundant assignment

Am 06.11.2013 um 20:58 schrieb Benjamin Herrenschmidt <[email protected]>:

Quoted 9 lines · Alexander Graf
Quoted 5 lines · Liu Ping Fan
Quoted 2 lines

Patch description missing.

Do you really need a description for trivial one-lines whose subject is a perfectly complete description already ?

Would I ask for it otherwise? It's also not 100% obvious that the assignment is redundant.

Alex

#11 lore raw

Re: [PATCH 3/3] powerpc/kvm: remove redundant assignment

Quoted 5 lines · Alexander Graf

Am 06.11.2013 um 20:58 schrieb Benjamin Herrenschmidt <[email protected]>:

Quoted 3 lines · Alexander Graf
Quoted 2 lines · Liu Ping Fan
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Patch description missing.

Do you really need a description for trivial one-lines whose subject is a perfectly complete description already ?

Would I ask for it otherwise? It's also not 100% obvious that the assignment is redundant.

And ? An explanation isn't going to be clearer than the code in that case ...

Ben.

#12 lore raw

Re: [PATCH 3/3] powerpc/kvm: remove redundant assignment

Am 07.11.2013 um 08:55 schrieb Benjamin Herrenschmidt <[email protected]>:

Quoted 10 lines · Alexander Graf
Quoted 9 lines

Am 06.11.2013 um 20:58 schrieb Benjamin Herrenschmidt <[email protected]>:

Quoted 7 lines · Alexander Graf
Quoted 5 lines · Liu Ping Fan
Quoted 2 lines

Patch description missing.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Do you really need a description for trivial one-lines whose subject is a perfectly complete description already ?

Would I ask for it otherwise? It's also not 100% obvious that the assignment is redundant.

And ? An explanation isn't going to be clearer than the code in that case ...

It's pretty non-obvious when you do a git show on that patch in 1 year from now, as the redundancy is out of scope of what the diff shows.

Alex

#13 lore raw

Re: [PATCH 3/3] powerpc/kvm: remove redundant assignment

Quoted 5 lines · Alexander Graf
Quoted 2 lines

And ? An explanation isn't going to be clearer than the code in that case ...

It's pretty non-obvious when you do a git show on that patch in 1 year from now, as the redundancy is out of scope of what the diff shows.

And ? How would an explanation help ?

Either it's redundant or it's not ... but only look at the code can prove it. An explanation won't because if the patch is wrong, so will be the explanation.

Cheers,
Ben.

#14 lore raw
In reply to Liu Ping Fan
Quoted 2 lines · Liu Ping Fan

After the primary vcpu changes vcore_state to VCORE_RUNNING, there is very little chance to schedule to secondary vcpu. So if we change the

Why do you say there is very little chance to run the secondary vcpu?

Quoted 3 lines

code sequence around set vcore_state to VCORE_RUNNING and disable preemption, we lost little. But we simplify the entering logi, based on the fact that if primary vcpu runs, the secondary vcpu can not be scheduled.

If a vcpu does a hypercall or something else that requires the host (kernel or userspace) to do something, that can happen in the context of the vcpu task for that vcpu. That vcpu task can run on another core (unless it has been pinned). When it is finished we would like the vcpu to continue executing in the guest as soon as possible. The code that you remove in this patch enables that to happen without having to wait until the other threads exit the guest. So I don't think it is a good idea to remove this code.

Regards,
Paul.

#15 lore raw
Quoted 10 lines · Paul Mackerras
Quoted 2 lines · Liu Ping Fan

After the primary vcpu changes vcore_state to VCORE_RUNNING, there is very little chance to schedule to secondary vcpu. So if we change the

Why do you say there is very little chance to run the secondary vcpu?

Quoted 3 lines

code sequence around set vcore_state to VCORE_RUNNING and disable preemption, we lost little. But we simplify the entering logi, based on the fact that if primary vcpu runs, the secondary vcpu can not be scheduled.

Quoted 9 lines

If a vcpu does a hypercall or something else that requires the host (kernel or userspace) to do something, that can happen in the context of the vcpu task for that vcpu. That vcpu task can run on another core (unless it has been pinned). When it is finished we would like the vcpu to continue executing in the guest as soon as possible. The code that you remove in this patch enables that to happen without having to wait until the other threads exit the guest. So I don't think it is a good idea to remove this code.

Yes, you are right.

Thanks and regards,
Pingfan

Quoted 2 lines

Regards,
Paul.