Skip to contentSkip to conversation

KVM: PPC: booke: Improve timer register emulation

PATCH In review Scott Wood 4 participants 22 messages Aug 26, 2011 – Sep 27, 2011

Subsystems (2)
lore
Share & tools
lore

Scott Wood

author
#0 lore raw

PATCH 5/5 KVM: PPC: booke: Improve timer register emulation

From: Liu Yu <[email protected]>

Decrementers are now properly driven by TCR/TSR, and the guest has full read/write access to these registers.

The decrementer keeps ticking (and setting the TSR bit) regardless of whether the interrupts are enabled with TCR.

The decrementer stops at zero, rather than going negative.

[scott: added dequeue in kvmppc_booke_irqprio_deliver, and dec stop-at-zero]
---
 arch/powerpc/include/asm/kvm_host.h |    2 +-
 arch/powerpc/include/asm/kvm_ppc.h  |   11 +++++
 arch/powerpc/kvm/book3s.c           |    8 ++++
 arch/powerpc/kvm/booke.c            |   80 +++++++++++++++++++++++++++--------
 arch/powerpc/kvm/booke.h            |    4 ++
 arch/powerpc/kvm/booke_emulate.c    |   11 ++++-
 arch/powerpc/kvm/emulate.c          |   45 ++++++++-----------
 arch/powerpc/kvm/powerpc.c          |   20 +--------
 8 files changed, 114 insertions(+), 67 deletions(-)
arch/powerpc/include/asm/kvm_host.h +1 -1
index 3305af4..ea08c79 100644
@@ -334,7 +334,7 @@ struct kvm_vcpu_arch {
 	u32 tbl;
 	u32 tbu;
 	u32 tcr;
-	u32 tsr;
+	ulong tsr; /* we need to perform set/clr_bits() which requires ulong */
 	u32 ivor[64];
 	ulong ivpr;
 	u32 pvr;
arch/powerpc/include/asm/kvm_ppc.h +11 -0
index bdaa6c8..ddaa615 100644
@@ -66,6 +66,7 @@ extern int kvmppc_emulate_instruction(struct kvm_run *run,
 extern int kvmppc_emulate_mmio(struct kvm_run *run, struct kvm_vcpu *vcpu);
 extern void kvmppc_emulate_dec(struct kvm_vcpu *vcpu);
 extern u32 kvmppc_get_dec(struct kvm_vcpu *vcpu, u64 tb);
+extern void kvmppc_decrementer_func(unsigned long data);
 
 /* Core-specific hooks */
 
@@ -197,4 +198,14 @@ int kvm_vcpu_ioctl_config_tlb(struct kvm_vcpu *vcpu,
 int kvm_vcpu_ioctl_dirty_tlb(struct kvm_vcpu *vcpu,
 			     struct kvm_dirty_tlb *cfg);
 
+static inline void kvmppc_wakeup_vcpu(struct kvm_vcpu *vcpu)
+{
+	if (waitqueue_active(&vcpu->wq)) {
+		wake_up_interruptible(&vcpu->wq);
+		vcpu->stat.halt_wakeup++;
+	} else if (vcpu->cpu != -1) {
+		smp_send_reschedule(vcpu->cpu);
+	}
+}
+
 #endif /* __POWERPC_KVM_PPC_H__ */
arch/powerpc/kvm/book3s.c +8 -0
index f68a34d..b057856 100644
@@ -514,3 +514,11 @@ out:
 	mutex_unlock(&kvm->slots_lock);
 	return r;
 }
+
+void kvmppc_decrementer_func(unsigned long data)
+{
+	struct kvm_vcpu *vcpu = (struct kvm_vcpu *)data;
+
+	kvmppc_core_queue_dec(vcpu);
+	kvmppc_wakeup_vcpu(vcpu);
+}
arch/powerpc/kvm/booke.c +62 -18

122 lines not on this page · raw message

arch/powerpc/kvm/booke.h +4 -0
index 8e1fe33..2fe2027 100644
@@ -55,6 +55,10 @@ extern unsigned long kvmppc_booke_handlers;
 void kvmppc_set_msr(struct kvm_vcpu *vcpu, u32 new_msr);
 void kvmppc_mmu_msr_notify(struct kvm_vcpu *vcpu, u32 old_msr);
 
+void kvmppc_set_tcr(struct kvm_vcpu *vcpu, u32 new_tcr);
+void kvmppc_set_tsr_bits(struct kvm_vcpu *vcpu, u32 tsr_bits);
+void kvmppc_clr_tsr_bits(struct kvm_vcpu *vcpu, u32 tsr_bits);
+
 int kvmppc_booke_emulate_op(struct kvm_run *run, struct kvm_vcpu *vcpu,
                             unsigned int inst, int *advance);
 int kvmppc_booke_emulate_mfspr(struct kvm_vcpu *vcpu, int sprn, int rt);
arch/powerpc/kvm/booke_emulate.c +8 -3
index 162b7ac..dee534e 100644
@@ -13,6 +13,7 @@
  * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
  *
  * Copyright IBM Corp. 2008
+ * Copyright 2011 Freescale Semiconductor, Inc.
  *
  * Authors: Hollis Blanchard <[email protected]>
  */
@@ -115,10 +116,10 @@ int kvmppc_booke_emulate_mtspr(struct kvm_vcpu *vcpu, int sprn, int rs)
 	case SPRN_DBSR:
 		vcpu->arch.dbsr &= ~spr_val; break;
 	case SPRN_TSR:
-		vcpu->arch.tsr &= ~spr_val; break;
+		kvmppc_clr_tsr_bits(vcpu, spr_val);
+		break;
 	case SPRN_TCR:
-		vcpu->arch.tcr = spr_val;
-		kvmppc_emulate_dec(vcpu);
+		kvmppc_set_tcr(vcpu, spr_val);
 		break;
 
 	/* Note: SPRG4-7 are user-readable. These values are
@@ -209,6 +210,10 @@ int kvmppc_booke_emulate_mfspr(struct kvm_vcpu *vcpu, int sprn, int rt)
 		kvmppc_set_gpr(vcpu, rt, vcpu->arch.dbcr1); break;
 	case SPRN_DBSR:
 		kvmppc_set_gpr(vcpu, rt, vcpu->arch.dbsr); break;
+	case SPRN_TSR:
+		kvmppc_set_gpr(vcpu, rt, vcpu->arch.tsr); break;
+	case SPRN_TCR:
+		kvmppc_set_gpr(vcpu, rt, vcpu->arch.tcr); break;
 
 	case SPRN_IVOR0:
 		kvmppc_set_gpr(vcpu, rt, vcpu->arch.ivor[BOOKE_IRQPRIO_CRITICAL]);
arch/powerpc/kvm/emulate.c +20 -27

70 lines not on this page · raw message

arch/powerpc/kvm/powerpc.c +1 -19
index 0ee0b80..05f8dc4 100644
@@ -286,18 +286,6 @@ int kvm_cpu_has_pending_timer(struct kvm_vcpu *vcpu)
 	return kvmppc_core_pending_dec(vcpu);
 }
 
-static void kvmppc_decrementer_func(unsigned long data)
-{
-	struct kvm_vcpu *vcpu = (struct kvm_vcpu *)data;
-
-	kvmppc_core_queue_dec(vcpu);
-
-	if (waitqueue_active(&vcpu->wq)) {
-		wake_up_interruptible(&vcpu->wq);
-		vcpu->stat.halt_wakeup++;
-	}
-}
-
 /*
  * low level hrtimer wake routine. Because this runs in hardirq context
  * we schedule a tasklet to do the real work.
@@ -551,13 +539,7 @@ int kvm_vcpu_ioctl_interrupt(struct kvm_vcpu *vcpu, struct kvm_interrupt *irq)
 	else
 		kvmppc_core_queue_external(vcpu, irq);
 
-	if (waitqueue_active(&vcpu->wq)) {
-		wake_up_interruptible(&vcpu->wq);
-		vcpu->stat.halt_wakeup++;
-	} else if (vcpu->cpu != -1) {
-		smp_send_reschedule(vcpu->cpu);
-	}
-
+	kvmppc_wakeup_vcpu(vcpu);
 	return 0;
 }
 
Signature · 1 line
1.7.6
#1 lore raw
Quoted 175 lines · Scott Wood

From: Liu Yu <[email protected]>

Decrementers are now properly driven by TCR/TSR, and the guest has full read/write access to these registers.

The decrementer keeps ticking (and setting the TSR bit) regardless of whether the interrupts are enabled with TCR.

The decrementer stops at zero, rather than going negative.

[scott: added dequeue in kvmppc_booke_irqprio_deliver, and dec stop-at-zero]

---

arch/powerpc/include/asm/kvm_host.h |    2 +-
arch/powerpc/include/asm/kvm_ppc.h  |   11 +++++
arch/powerpc/kvm/book3s.c           |    8 ++++
arch/powerpc/kvm/booke.c            |   80 +++++++++++++++++++++++++++--------
arch/powerpc/kvm/booke.h            |    4 ++
arch/powerpc/kvm/booke_emulate.c    |   11 ++++-
arch/powerpc/kvm/emulate.c          |   45 ++++++++-----------
arch/powerpc/kvm/powerpc.c          |   20 +--------
8 files changed, 114 insertions(+), 67 deletions(-)
arch/powerpc/include/asm/kvm_host.h +1 -1
index 3305af4..ea08c79 100644
@@ -334,7 +334,7 @@ struct kvm_vcpu_arch {
 	u32 tbl;
 	u32 tbu;
 	u32 tcr;
-	u32 tsr;
+	ulong tsr; /* we need to perform set/clr_bits() which requires ulong */
 	u32 ivor[64];
 	ulong ivpr;
 	u32 pvr;
arch/powerpc/include/asm/kvm_ppc.h +11 -0
index bdaa6c8..ddaa615 100644
@@ -66,6 +66,7 @@ extern int kvmppc_emulate_instruction(struct kvm_run *run,
 extern int kvmppc_emulate_mmio(struct kvm_run *run, struct kvm_vcpu *vcpu);
 extern void kvmppc_emulate_dec(struct kvm_vcpu *vcpu);
 extern u32 kvmppc_get_dec(struct kvm_vcpu *vcpu, u64 tb);
+extern void kvmppc_decrementer_func(unsigned long data);
 
 /* Core-specific hooks */
 
@@ -197,4 +198,14 @@ int kvm_vcpu_ioctl_config_tlb(struct kvm_vcpu *vcpu,
 int kvm_vcpu_ioctl_dirty_tlb(struct kvm_vcpu *vcpu,
 			     struct kvm_dirty_tlb *cfg);
 
+static inline void kvmppc_wakeup_vcpu(struct kvm_vcpu *vcpu)
+{
+	if (waitqueue_active(&vcpu->wq)) {
+		wake_up_interruptible(&vcpu->wq);
+		vcpu->stat.halt_wakeup++;
+	} else if (vcpu->cpu != -1) {
+		smp_send_reschedule(vcpu->cpu);
+	}
+}
+
 #endif /* __POWERPC_KVM_PPC_H__ */
arch/powerpc/kvm/book3s.c +8 -0
index f68a34d..b057856 100644
@@ -514,3 +514,11 @@ out:
 	mutex_unlock(&kvm->slots_lock);
 	return r;
 }
+
+void kvmppc_decrementer_func(unsigned long data)
+{
+	struct kvm_vcpu *vcpu = (struct kvm_vcpu *)data;
+
+	kvmppc_core_queue_dec(vcpu);
+	kvmppc_wakeup_vcpu(vcpu);
+}
arch/powerpc/kvm/booke.c +28 -18
index 0ed62c1..502f9ff 100644
@@ -205,7 +205,8 @@ void kvmppc_core_dequeue_external(struct kvm_vcpu *vcpu,
 static int kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu,
                                        unsigned int priority)
 {
-	int allowed = 0;
+	int allowed = 1;
+	int dequeue = 0;
 	ulong uninitialized_var(msr_mask);
 	bool update_esr = false, update_dear = false;
 	ulong crit_raw = vcpu->arch.shared->critical;
@@ -258,10 +259,15 @@ static int kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu,
 		allowed = vcpu->arch.shared->msr & MSR_ME;
 		msr_mask = 0;
 		break;
-	case BOOKE_IRQPRIO_EXTERNAL:
 	case BOOKE_IRQPRIO_DECREMENTER:
+		if (!(vcpu->arch.tcr & TCR_DIE)) {
+			allowed = 0;
+			dequeue = 1;
+		}
+		/* fall through */
+	case BOOKE_IRQPRIO_EXTERNAL:
 	case BOOKE_IRQPRIO_FIT:
-		allowed = vcpu->arch.shared->msr & MSR_EE;
+		allowed = allowed && vcpu->arch.shared->msr & MSR_EE;
 		allowed = allowed && !crit;
 		msr_mask = MSR_CE|MSR_ME|MSR_DE;
 		break;
@@ -269,6 +275,8 @@ static int kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu,
 		allowed = vcpu->arch.shared->msr & MSR_DE;
 		msr_mask = MSR_ME;
 		break;
+	default:
+		allowed = 0;
 	}
 
 	if (allowed) {
@@ -283,6 +291,9 @@ static int kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu,
 
 		if (!keep_irq)
 			clear_bit(priority, &vcpu->arch.pending_exceptions);
+	} else if (dequeue) {
+		/* Should only happen due to races with masking */
+		clear_bit(priority, &vcpu->arch.pending_exceptions);
 	}
 
 	return allowed;
@@ -721,25 +732,16 @@ static int set_sregs_base(struct kvm_vcpu *vcpu,
 	vcpu->arch.shared->esr = sregs->u.e.esr;
 	vcpu->arch.shared->dar = sregs->u.e.dear;
 	vcpu->arch.vrsave = sregs->u.e.vrsave;
-	vcpu->arch.tcr = sregs->u.e.tcr;
+	kvmppc_set_tcr(vcpu, sregs->u.e.tcr);
 
-	if (sregs->u.e.update_special & KVM_SREGS_E_UPDATE_DEC)
+	if (sregs->u.e.update_special & KVM_SREGS_E_UPDATE_DEC) {
 		vcpu->arch.dec = sregs->u.e.dec;
-
-	kvmppc_emulate_dec(vcpu);
+		kvmppc_emulate_dec(vcpu);
+	}
 
 	if (sregs->u.e.update_special & KVM_SREGS_E_UPDATE_TSR) {
-		/*
-		 * FIXME: existing KVM timer handling is incomplete.
-		 * TSR cannot be read by the guest, and its value in
-		 * vcpu->arch is always zero.  For now, just handle
-		 * the case where the caller is trying to inject a
-		 * decrementer interrupt.
-		 */
-
-		if ((sregs->u.e.tsr & TSR_DIS) &&
-		    (vcpu->arch.tcr & TCR_DIE))
-			kvmppc_core_queue_dec(vcpu);
+		kvmppc_clr_tsr_bits(vcpu, ~0);
+		kvmppc_set_tsr_bits(vcpu, sregs->u.e.tsr);
 	}
 
 	return 0;
@@ -895,6 +897,48 @@ void kvmppc_core_destroy_vm(struct kvm *kvm)
 {
 }
 
+void kvmppc_set_tcr(struct kvm_vcpu *vcpu, u32 new_tcr)
+{
+	vcpu->arch.tcr = new_tcr;
+	smp_wmb();
+
+	/*
+	 * Since TCR changed, we need to check
+	 * if blocked interrupts are deliverable.

arch/powerpc/kvm/booke.c void kvmppc_core_destroy_vm(struct kvm *kvm) · line 917

Quoted 10 lines
arch/powerpc/kvm/booke.c +10 -0
+	 */
+	if ((new_tcr & TCR_DIE) && (vcpu->arch.tsr & TSR_DIS)) {
+		kvmppc_core_queue_dec(vcpu);
+		kvmppc_wakeup_vcpu(vcpu);
+	}
+}
+
+void kvmppc_set_tsr_bits(struct kvm_vcpu *vcpu, u32 tsr_bits)
+{
+	set_bits(tsr_bits, &vcpu->arch.tsr);

I'm still missing the point why you need set_bits/clear_bits. Can't you just use |= and &= ~ like everyone else? :)

arch/powerpc/kvm/booke.c void kvmppc_core_destroy_vm(struct kvm *kvm) · line 923

Quoted 6 lines
arch/powerpc/kvm/booke.c +6 -0
+	smp_wmb();
+
+	if ((tsr_bits & TSR_DIS) && (vcpu->arch.tcr & TCR_DIE)) {
+		kvmppc_core_queue_dec(vcpu);
+		kvmppc_wakeup_vcpu(vcpu);
+	}

Please combine into a function and call that for updating tsr and tcr.

Alex

#2 lore raw
In reply to Scott Wood
Quoted 209 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Alexander Graf
Sent: Tuesday, September 06, 2011 6:45 AM
To: Wood Scott-B07421
Cc: [email protected]
Subject: Re: [PATCH 5/5] KVM: PPC: booke: Improve timer register emulation

Quoted 6 lines · Scott Wood

From: Liu Yu <[email protected]>

Decrementers are now properly driven by TCR/TSR, and the guest has full read/write access to these registers.

The decrementer keeps ticking (and setting the TSR bit)

regardless of

Quoted 6 lines

whether the interrupts are enabled with TCR.

The decrementer stops at zero, rather than going negative.

[scott: added dequeue in kvmppc_booke_irqprio_deliver, and

dec stop-at-zero]

Quoted 6 lines

---

arch/powerpc/include/asm/kvm_host.h |    2 +-
arch/powerpc/include/asm/kvm_ppc.h  |   11 +++++
arch/powerpc/kvm/book3s.c           |    8 ++++
arch/powerpc/kvm/booke.c            |   80 

+++++++++++++++++++++++++++--------

Review

Quoted 7 lines
arch/powerpc/kvm/booke.h            |    4 ++
arch/powerpc/kvm/booke_emulate.c    |   11 ++++-
arch/powerpc/kvm/emulate.c          |   45 ++++++++-----------
arch/powerpc/kvm/powerpc.c          |   20 +--------
8 files changed, 114 insertions(+), 67 deletions(-)
(quoted diff) +0 -0

b/arch/powerpc/include/asm/kvm_host.h

arch/powerpc/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 337

Quoted 9 lines

index 3305af4..ea08c79 100644

arch/powerpc/include/asm/kvm_host.h +1 -1
@@ -334,7 +334,7 @@ struct kvm_vcpu_arch {
 	u32 tbl;
 	u32 tbu;
 	u32 tcr;
-	u32 tsr;
+	ulong tsr; /* we need to perform set/clr_bits() which 

requires ulong */

Review

Quoted 4 lines
arch/powerpc/include/asm/kvm_host.h +0 -0
 	u32 ivor[64];
 	ulong ivpr;
 	u32 pvr;
(quoted diff) +0 -0

b/arch/powerpc/include/asm/kvm_ppc.h

arch/powerpc/include/asm/kvm_ppc.h extern int

Quoted 4 lines

index bdaa6c8..ddaa615 100644

arch/powerpc/include/asm/kvm_ppc.h +0 -0
@@ -66,6 +66,7 @@ extern int 

kvmppc_emulate_instruction(struct kvm_run *run,

Quoted 1 line

extern int kvmppc_emulate_mmio(struct kvm_run *run, struct

kvm_vcpu *vcpu);

arch/powerpc/include/asm/kvm_ppc.h int kvm_vcpu_ioctl_config_tlb(struct

Quoted 7 lines

extern void kvmppc_emulate_dec(struct kvm_vcpu *vcpu); extern u32 kvmppc_get_dec(struct kvm_vcpu *vcpu, u64 tb); +extern void kvmppc_decrementer_func(unsigned long data);

/* Core-specific hooks */

arch/powerpc/include/asm/kvm_ppc.h +0 -0
@@ -197,4 +198,14 @@ int kvm_vcpu_ioctl_config_tlb(struct 

kvm_vcpu *vcpu,

arch/powerpc/kvm/booke.c void

Quoted 35 lines
arch/powerpc/include/asm/kvm_ppc.h +10 -0
 int kvm_vcpu_ioctl_dirty_tlb(struct kvm_vcpu *vcpu,
 			     struct kvm_dirty_tlb *cfg);
 
+static inline void kvmppc_wakeup_vcpu(struct kvm_vcpu *vcpu)
+{
+	if (waitqueue_active(&vcpu->wq)) {
+		wake_up_interruptible(&vcpu->wq);
+		vcpu->stat.halt_wakeup++;
+	} else if (vcpu->cpu != -1) {
+		smp_send_reschedule(vcpu->cpu);
+	}
+}
+
 #endif /* __POWERPC_KVM_PPC_H__ */
arch/powerpc/kvm/book3s.c +8 -0
index f68a34d..b057856 100644
@@ -514,3 +514,11 @@ out:
 	mutex_unlock(&kvm->slots_lock);
 	return r;
 }
+
+void kvmppc_decrementer_func(unsigned long data)
+{
+	struct kvm_vcpu *vcpu = (struct kvm_vcpu *)data;
+
+	kvmppc_core_queue_dec(vcpu);
+	kvmppc_wakeup_vcpu(vcpu);
+}
arch/powerpc/kvm/booke.c +0 -0
index 0ed62c1..502f9ff 100644
@@ -205,7 +205,8 @@ void 

kvmppc_core_dequeue_external(struct kvm_vcpu *vcpu,

arch/powerpc/kvm/booke.c static int

Quoted 10 lines
arch/powerpc/kvm/booke.c +2 -1
 static int kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu,
                                        unsigned int priority)
 {
-	int allowed = 0;
+	int allowed = 1;
+	int dequeue = 0;
 	ulong uninitialized_var(msr_mask);
 	bool update_esr = false, update_dear = false;
 	ulong crit_raw = vcpu->arch.shared->critical;
@@ -258,10 +259,15 @@ static int 

kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu,

arch/powerpc/kvm/booke.c static int

Quoted 18 lines
arch/powerpc/kvm/booke.c +7 -2
 		allowed = vcpu->arch.shared->msr & MSR_ME;
 		msr_mask = 0;
 		break;
-	case BOOKE_IRQPRIO_EXTERNAL:
 	case BOOKE_IRQPRIO_DECREMENTER:
+		if (!(vcpu->arch.tcr & TCR_DIE)) {
+			allowed = 0;
+			dequeue = 1;
+		}
+		/* fall through */
+	case BOOKE_IRQPRIO_EXTERNAL:
 	case BOOKE_IRQPRIO_FIT:
-		allowed = vcpu->arch.shared->msr & MSR_EE;
+		allowed = allowed && vcpu->arch.shared->msr & MSR_EE;
 		allowed = allowed && !crit;
 		msr_mask = MSR_CE|MSR_ME|MSR_DE;
 		break;
@@ -269,6 +275,8 @@ static int 

kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu,

arch/powerpc/kvm/booke.c static int

Quoted 9 lines
arch/powerpc/kvm/booke.c +2 -0
 		allowed = vcpu->arch.shared->msr & MSR_DE;
 		msr_mask = MSR_ME;
 		break;
+	default:
+		allowed = 0;
 	}
 
 	if (allowed) {
@@ -283,6 +291,9 @@ static int 

kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu,

arch/powerpc/kvm/booke.c static int · line 292

Quoted 3 lines
arch/powerpc/kvm/booke.c +0 -0
 		if (!keep_irq)
 			clear_bit(priority, 

&vcpu->arch.pending_exceptions);

arch/powerpc/kvm/booke.c static int set_sregs_base(struct

Quoted 7 lines
arch/powerpc/kvm/booke.c +3 -0
+	} else if (dequeue) {
+		/* Should only happen due to races with masking */
+		clear_bit(priority, &vcpu->arch.pending_exceptions);
 	}
 
 	return allowed;
@@ -721,25 +732,16 @@ static int set_sregs_base(struct 

kvm_vcpu *vcpu,

Quoted 48 lines
arch/powerpc/kvm/booke.c +19 -15
 	vcpu->arch.shared->esr = sregs->u.e.esr;
 	vcpu->arch.shared->dar = sregs->u.e.dear;
 	vcpu->arch.vrsave = sregs->u.e.vrsave;
-	vcpu->arch.tcr = sregs->u.e.tcr;
+	kvmppc_set_tcr(vcpu, sregs->u.e.tcr);
 
-	if (sregs->u.e.update_special & KVM_SREGS_E_UPDATE_DEC)
+	if (sregs->u.e.update_special & KVM_SREGS_E_UPDATE_DEC) {
 		vcpu->arch.dec = sregs->u.e.dec;
-
-	kvmppc_emulate_dec(vcpu);
+		kvmppc_emulate_dec(vcpu);
+	}
 
 	if (sregs->u.e.update_special & KVM_SREGS_E_UPDATE_TSR) {
-		/*
-		 * FIXME: existing KVM timer handling is incomplete.
-		 * TSR cannot be read by the guest, and its value in
-		 * vcpu->arch is always zero.  For now, just handle
-		 * the case where the caller is trying to inject a
-		 * decrementer interrupt.
-		 */
-
-		if ((sregs->u.e.tsr & TSR_DIS) &&
-		    (vcpu->arch.tcr & TCR_DIE))
-			kvmppc_core_queue_dec(vcpu);
+		kvmppc_clr_tsr_bits(vcpu, ~0);
+		kvmppc_set_tsr_bits(vcpu, sregs->u.e.tsr);
 	}
 
 	return 0;
@@ -895,6 +897,48 @@ void kvmppc_core_destroy_vm(struct kvm *kvm)
 {
 }
 
+void kvmppc_set_tcr(struct kvm_vcpu *vcpu, u32 new_tcr)
+{
+	vcpu->arch.tcr = new_tcr;
+	smp_wmb();
+
+	/*
+	 * Since TCR changed, we need to check
+	 * if blocked interrupts are deliverable.
+	 */
+	if ((new_tcr & TCR_DIE) && (vcpu->arch.tsr & TSR_DIS)) {
+		kvmppc_core_queue_dec(vcpu);
+		kvmppc_wakeup_vcpu(vcpu);
+	}
Quoted 9 lines

arch/powerpc/kvm/booke.c void kvmppc_core_destroy_vm(struct kvm *kvm) · line 917

Quoted 5 lines
arch/powerpc/kvm/booke.c +5 -0
+}
+
+void kvmppc_set_tsr_bits(struct kvm_vcpu *vcpu, u32 tsr_bits)
+{
+	set_bits(tsr_bits, &vcpu->arch.tsr);

I'm still missing the point why you need set_bits/clear_bits. Can't you just use |= and &= ~ like everyone else? :)

Because set_bits/clear_bits is atomic.
set/clr_tsr_bits() may be called in different threads so that on different cores. We need to make sure modification to be atomic to prevent missing changes.

Thanks,
Yu

#3 lore raw
In reply to Scott Wood
Quoted 219 lines · On 06.09.2011, at 05:17, Liu Yu-B13201 wrote:
Quoted 209 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Alexander Graf
Sent: Tuesday, September 06, 2011 6:45 AM
To: Wood Scott-B07421
Cc: [email protected]
Subject: Re: [PATCH 5/5] KVM: PPC: booke: Improve timer register emulation

Quoted 6 lines · Scott Wood

From: Liu Yu <[email protected]>

Decrementers are now properly driven by TCR/TSR, and the guest has full read/write access to these registers.

The decrementer keeps ticking (and setting the TSR bit)

regardless of

Quoted 6 lines

whether the interrupts are enabled with TCR.

The decrementer stops at zero, rather than going negative.

[scott: added dequeue in kvmppc_booke_irqprio_deliver, and

dec stop-at-zero]

Quoted 6 lines

---

arch/powerpc/include/asm/kvm_host.h |    2 +-
arch/powerpc/include/asm/kvm_ppc.h  |   11 +++++
arch/powerpc/kvm/book3s.c           |    8 ++++
arch/powerpc/kvm/booke.c            |   80 

+++++++++++++++++++++++++++--------

Review

Quoted 7 lines
arch/powerpc/kvm/booke.h            |    4 ++
arch/powerpc/kvm/booke_emulate.c    |   11 ++++-
arch/powerpc/kvm/emulate.c          |   45 ++++++++-----------
arch/powerpc/kvm/powerpc.c          |   20 +--------
8 files changed, 114 insertions(+), 67 deletions(-)
(quoted diff) +0 -0

b/arch/powerpc/include/asm/kvm_host.h

arch/powerpc/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 337

Quoted 9 lines

index 3305af4..ea08c79 100644

arch/powerpc/include/asm/kvm_host.h +1 -1
@@ -334,7 +334,7 @@ struct kvm_vcpu_arch {
 	u32 tbl;
 	u32 tbu;
 	u32 tcr;
-	u32 tsr;
+	ulong tsr; /* we need to perform set/clr_bits() which 

requires ulong */

Review

Quoted 4 lines
arch/powerpc/include/asm/kvm_host.h +0 -0
 	u32 ivor[64];
 	ulong ivpr;
 	u32 pvr;
(quoted diff) +0 -0

b/arch/powerpc/include/asm/kvm_ppc.h

arch/powerpc/include/asm/kvm_ppc.h extern int

Quoted 4 lines

index bdaa6c8..ddaa615 100644

arch/powerpc/include/asm/kvm_ppc.h +0 -0
@@ -66,6 +66,7 @@ extern int 

kvmppc_emulate_instruction(struct kvm_run *run,

Quoted 1 line

extern int kvmppc_emulate_mmio(struct kvm_run *run, struct

kvm_vcpu *vcpu);

arch/powerpc/include/asm/kvm_ppc.h int kvm_vcpu_ioctl_config_tlb(struct

Quoted 7 lines

extern void kvmppc_emulate_dec(struct kvm_vcpu *vcpu); extern u32 kvmppc_get_dec(struct kvm_vcpu *vcpu, u64 tb); +extern void kvmppc_decrementer_func(unsigned long data);

/* Core-specific hooks */

arch/powerpc/include/asm/kvm_ppc.h +0 -0
@@ -197,4 +198,14 @@ int kvm_vcpu_ioctl_config_tlb(struct 

kvm_vcpu *vcpu,

arch/powerpc/kvm/booke.c void

Quoted 35 lines
arch/powerpc/include/asm/kvm_ppc.h +10 -0
 int kvm_vcpu_ioctl_dirty_tlb(struct kvm_vcpu *vcpu,
 			     struct kvm_dirty_tlb *cfg);
 
+static inline void kvmppc_wakeup_vcpu(struct kvm_vcpu *vcpu)
+{
+	if (waitqueue_active(&vcpu->wq)) {
+		wake_up_interruptible(&vcpu->wq);
+		vcpu->stat.halt_wakeup++;
+	} else if (vcpu->cpu != -1) {
+		smp_send_reschedule(vcpu->cpu);
+	}
+}
+
 #endif /* __POWERPC_KVM_PPC_H__ */
arch/powerpc/kvm/book3s.c +8 -0
index f68a34d..b057856 100644
@@ -514,3 +514,11 @@ out:
 	mutex_unlock(&kvm->slots_lock);
 	return r;
 }
+
+void kvmppc_decrementer_func(unsigned long data)
+{
+	struct kvm_vcpu *vcpu = (struct kvm_vcpu *)data;
+
+	kvmppc_core_queue_dec(vcpu);
+	kvmppc_wakeup_vcpu(vcpu);
+}
arch/powerpc/kvm/booke.c +0 -0
index 0ed62c1..502f9ff 100644
@@ -205,7 +205,8 @@ void 

kvmppc_core_dequeue_external(struct kvm_vcpu *vcpu,

arch/powerpc/kvm/booke.c static int

Quoted 10 lines
arch/powerpc/kvm/booke.c +2 -1
 static int kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu,
                                       unsigned int priority)
 {
-	int allowed = 0;
+	int allowed = 1;
+	int dequeue = 0;
 	ulong uninitialized_var(msr_mask);
 	bool update_esr = false, update_dear = false;
 	ulong crit_raw = vcpu->arch.shared->critical;
@@ -258,10 +259,15 @@ static int 

kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu,

arch/powerpc/kvm/booke.c static int

Quoted 18 lines
arch/powerpc/kvm/booke.c +7 -2
 		allowed = vcpu->arch.shared->msr & MSR_ME;
 		msr_mask = 0;
 		break;
-	case BOOKE_IRQPRIO_EXTERNAL:
 	case BOOKE_IRQPRIO_DECREMENTER:
+		if (!(vcpu->arch.tcr & TCR_DIE)) {
+			allowed = 0;
+			dequeue = 1;
+		}
+		/* fall through */
+	case BOOKE_IRQPRIO_EXTERNAL:
 	case BOOKE_IRQPRIO_FIT:
-		allowed = vcpu->arch.shared->msr & MSR_EE;
+		allowed = allowed && vcpu->arch.shared->msr & MSR_EE;
 		allowed = allowed && !crit;
 		msr_mask = MSR_CE|MSR_ME|MSR_DE;
 		break;
@@ -269,6 +275,8 @@ static int 

kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu,

arch/powerpc/kvm/booke.c static int

Quoted 9 lines
arch/powerpc/kvm/booke.c +2 -0
 		allowed = vcpu->arch.shared->msr & MSR_DE;
 		msr_mask = MSR_ME;
 		break;
+	default:
+		allowed = 0;
 	}
 
 	if (allowed) {
@@ -283,6 +291,9 @@ static int 

kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu,

arch/powerpc/kvm/booke.c static int · line 292

Quoted 3 lines
arch/powerpc/kvm/booke.c +0 -0
 		if (!keep_irq)
 			clear_bit(priority, 

&vcpu->arch.pending_exceptions);

arch/powerpc/kvm/booke.c static int set_sregs_base(struct

Quoted 7 lines
arch/powerpc/kvm/booke.c +3 -0
+	} else if (dequeue) {
+		/* Should only happen due to races with masking */
+		clear_bit(priority, &vcpu->arch.pending_exceptions);
 	}
 
 	return allowed;
@@ -721,25 +732,16 @@ static int set_sregs_base(struct 

kvm_vcpu *vcpu,

Quoted 48 lines
arch/powerpc/kvm/booke.c +19 -15
 	vcpu->arch.shared->esr = sregs->u.e.esr;
 	vcpu->arch.shared->dar = sregs->u.e.dear;
 	vcpu->arch.vrsave = sregs->u.e.vrsave;
-	vcpu->arch.tcr = sregs->u.e.tcr;
+	kvmppc_set_tcr(vcpu, sregs->u.e.tcr);
 
-	if (sregs->u.e.update_special & KVM_SREGS_E_UPDATE_DEC)
+	if (sregs->u.e.update_special & KVM_SREGS_E_UPDATE_DEC) {
 		vcpu->arch.dec = sregs->u.e.dec;
-
-	kvmppc_emulate_dec(vcpu);
+		kvmppc_emulate_dec(vcpu);
+	}
 
 	if (sregs->u.e.update_special & KVM_SREGS_E_UPDATE_TSR) {
-		/*
-		 * FIXME: existing KVM timer handling is incomplete.
-		 * TSR cannot be read by the guest, and its value in
-		 * vcpu->arch is always zero.  For now, just handle
-		 * the case where the caller is trying to inject a
-		 * decrementer interrupt.
-		 */
-
-		if ((sregs->u.e.tsr & TSR_DIS) &&
-		    (vcpu->arch.tcr & TCR_DIE))
-			kvmppc_core_queue_dec(vcpu);
+		kvmppc_clr_tsr_bits(vcpu, ~0);
+		kvmppc_set_tsr_bits(vcpu, sregs->u.e.tsr);
 	}
 
 	return 0;
@@ -895,6 +897,48 @@ void kvmppc_core_destroy_vm(struct kvm *kvm)
 {
 }
 
+void kvmppc_set_tcr(struct kvm_vcpu *vcpu, u32 new_tcr)
+{
+	vcpu->arch.tcr = new_tcr;
+	smp_wmb();
+
+	/*
+	 * Since TCR changed, we need to check
+	 * if blocked interrupts are deliverable.
+	 */
+	if ((new_tcr & TCR_DIE) && (vcpu->arch.tsr & TSR_DIS)) {
+		kvmppc_core_queue_dec(vcpu);
+		kvmppc_wakeup_vcpu(vcpu);
+	}
Quoted 9 lines

arch/powerpc/kvm/booke.c void kvmppc_core_destroy_vm(struct kvm *kvm) · line 917

Quoted 5 lines
arch/powerpc/kvm/booke.c +5 -0
+}
+
+void kvmppc_set_tsr_bits(struct kvm_vcpu *vcpu, u32 tsr_bits)
+{
+	set_bits(tsr_bits, &vcpu->arch.tsr);

I'm still missing the point why you need set_bits/clear_bits. Can't you just use |= and &= ~ like everyone else? :)

Quoted 3 lines

Because set_bits/clear_bits is atomic.
set/clr_tsr_bits() may be called in different threads so that on different cores. We need to make sure modification to be atomic to prevent missing changes.

Ah, so you can set the TSR bit in the decrementer timer callback? Hrm. Please add a comment there saying so. All the other cases should be under vcpu_load() scope. If they're not, we have bugs in our code :). So they shouldn't need atomic operations (except for now they do because there's one user that doesn't do vcpu_load).

Alex

#4 lore raw
In reply to Scott Wood

Am 06.09.2011 um 10:04 schrieb Alexander Graf <[email protected]>:

Quoted 222 lines · On 06.09.2011, at 05:17, Liu Yu-B13201 wrote:
Quoted 220 lines
Quoted 219 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Alexander Graf
Sent: Tuesday, September 06, 2011 6:45 AM
To: Wood Scott-B07421
Cc: [email protected]
Subject: Re: [PATCH 5/5] KVM: PPC: booke: Improve timer register emulation

Quoted 6 lines · Scott Wood

From: Liu Yu <[email protected]>

Decrementers are now properly driven by TCR/TSR, and the guest has full read/write access to these registers.

The decrementer keeps ticking (and setting the TSR bit)

regardless of

Quoted 6 lines

whether the interrupts are enabled with TCR.

The decrementer stops at zero, rather than going negative.

[scott: added dequeue in kvmppc_booke_irqprio_deliver, and

dec stop-at-zero]

Quoted 6 lines

---

arch/powerpc/include/asm/kvm_host.h |    2 +-
arch/powerpc/include/asm/kvm_ppc.h  |   11 +++++
arch/powerpc/kvm/book3s.c           |    8 ++++
arch/powerpc/kvm/booke.c            |   80 

+++++++++++++++++++++++++++--------

Review

Quoted 7 lines
arch/powerpc/kvm/booke.h            |    4 ++
arch/powerpc/kvm/booke_emulate.c    |   11 ++++-
arch/powerpc/kvm/emulate.c          |   45 ++++++++-----------
arch/powerpc/kvm/powerpc.c          |   20 +--------
8 files changed, 114 insertions(+), 67 deletions(-)
(quoted diff) +0 -0

b/arch/powerpc/include/asm/kvm_host.h

arch/powerpc/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 337

Quoted 9 lines

index 3305af4..ea08c79 100644

arch/powerpc/include/asm/kvm_host.h +1 -1
@@ -334,7 +334,7 @@ struct kvm_vcpu_arch {
   u32 tbl;
   u32 tbu;
   u32 tcr;
-    u32 tsr;
+    ulong tsr; /* we need to perform set/clr_bits() which 

requires ulong */

Review

Quoted 4 lines
arch/powerpc/include/asm/kvm_host.h +0 -0
   u32 ivor[64];
   ulong ivpr;
   u32 pvr;
(quoted diff) +0 -0

b/arch/powerpc/include/asm/kvm_ppc.h

arch/powerpc/include/asm/kvm_ppc.h extern int

Quoted 4 lines

index bdaa6c8..ddaa615 100644

arch/powerpc/include/asm/kvm_ppc.h +0 -0
@@ -66,6 +66,7 @@ extern int 

kvmppc_emulate_instruction(struct kvm_run *run,

Quoted 1 line

extern int kvmppc_emulate_mmio(struct kvm_run *run, struct

kvm_vcpu *vcpu);

arch/powerpc/include/asm/kvm_ppc.h int kvm_vcpu_ioctl_config_tlb(struct

Quoted 7 lines

extern void kvmppc_emulate_dec(struct kvm_vcpu *vcpu); extern u32 kvmppc_get_dec(struct kvm_vcpu *vcpu, u64 tb); +extern void kvmppc_decrementer_func(unsigned long data);

/* Core-specific hooks */

arch/powerpc/include/asm/kvm_ppc.h +0 -0
@@ -197,4 +198,14 @@ int kvm_vcpu_ioctl_config_tlb(struct 

kvm_vcpu *vcpu,

arch/powerpc/kvm/booke.c void

Quoted 35 lines
arch/powerpc/include/asm/kvm_ppc.h +10 -0
 int kvm_vcpu_ioctl_dirty_tlb(struct kvm_vcpu *vcpu,
                struct kvm_dirty_tlb *cfg);
 
+static inline void kvmppc_wakeup_vcpu(struct kvm_vcpu *vcpu)
+{
+    if (waitqueue_active(&vcpu->wq)) {
+        wake_up_interruptible(&vcpu->wq);
+        vcpu->stat.halt_wakeup++;
+    } else if (vcpu->cpu != -1) {
+        smp_send_reschedule(vcpu->cpu);
+    }
+}
+
 #endif /* __POWERPC_KVM_PPC_H__ */
arch/powerpc/kvm/book3s.c +8 -0
index f68a34d..b057856 100644
@@ -514,3 +514,11 @@ out:
   mutex_unlock(&kvm->slots_lock);
   return r;
 }
+
+void kvmppc_decrementer_func(unsigned long data)
+{
+    struct kvm_vcpu *vcpu = (struct kvm_vcpu *)data;
+
+    kvmppc_core_queue_dec(vcpu);
+    kvmppc_wakeup_vcpu(vcpu);
+}
arch/powerpc/kvm/booke.c +0 -0
index 0ed62c1..502f9ff 100644
@@ -205,7 +205,8 @@ void 

kvmppc_core_dequeue_external(struct kvm_vcpu *vcpu,

arch/powerpc/kvm/booke.c static int

Quoted 10 lines
arch/powerpc/kvm/booke.c +2 -1
 static int kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu,
                                      unsigned int priority)
 {
-    int allowed = 0;
+    int allowed = 1;
+    int dequeue = 0;
   ulong uninitialized_var(msr_mask);
   bool update_esr = false, update_dear = false;
   ulong crit_raw = vcpu->arch.shared->critical;
@@ -258,10 +259,15 @@ static int 

kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu,

arch/powerpc/kvm/booke.c static int

Quoted 18 lines
arch/powerpc/kvm/booke.c +7 -2
       allowed = vcpu->arch.shared->msr & MSR_ME;
       msr_mask = 0;
       break;
-    case BOOKE_IRQPRIO_EXTERNAL:
   case BOOKE_IRQPRIO_DECREMENTER:
+        if (!(vcpu->arch.tcr & TCR_DIE)) {
+            allowed = 0;
+            dequeue = 1;
+        }
+        /* fall through */
+    case BOOKE_IRQPRIO_EXTERNAL:
   case BOOKE_IRQPRIO_FIT:
-        allowed = vcpu->arch.shared->msr & MSR_EE;
+        allowed = allowed && vcpu->arch.shared->msr & MSR_EE;
       allowed = allowed && !crit;
       msr_mask = MSR_CE|MSR_ME|MSR_DE;
       break;
@@ -269,6 +275,8 @@ static int 

kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu,

arch/powerpc/kvm/booke.c static int

Quoted 9 lines
arch/powerpc/kvm/booke.c +2 -0
       allowed = vcpu->arch.shared->msr & MSR_DE;
       msr_mask = MSR_ME;
       break;
+    default:
+        allowed = 0;
   }
 
   if (allowed) {
@@ -283,6 +291,9 @@ static int 

kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu,

arch/powerpc/kvm/booke.c static int · line 292

Quoted 3 lines
arch/powerpc/kvm/booke.c +0 -0
       if (!keep_irq)
           clear_bit(priority, 

&vcpu->arch.pending_exceptions);

arch/powerpc/kvm/booke.c static int set_sregs_base(struct

Quoted 7 lines
arch/powerpc/kvm/booke.c +3 -0
+    } else if (dequeue) {
+        /* Should only happen due to races with masking */
+        clear_bit(priority, &vcpu->arch.pending_exceptions);
   }
 
   return allowed;
@@ -721,25 +732,16 @@ static int set_sregs_base(struct 

kvm_vcpu *vcpu,

Quoted 43 lines
arch/powerpc/kvm/booke.c +14 -15
   vcpu->arch.shared->esr = sregs->u.e.esr;
   vcpu->arch.shared->dar = sregs->u.e.dear;
   vcpu->arch.vrsave = sregs->u.e.vrsave;
-    vcpu->arch.tcr = sregs->u.e.tcr;
+    kvmppc_set_tcr(vcpu, sregs->u.e.tcr);
 
-    if (sregs->u.e.update_special & KVM_SREGS_E_UPDATE_DEC)
+    if (sregs->u.e.update_special & KVM_SREGS_E_UPDATE_DEC) {
       vcpu->arch.dec = sregs->u.e.dec;
-
-    kvmppc_emulate_dec(vcpu);
+        kvmppc_emulate_dec(vcpu);
+    }
 
   if (sregs->u.e.update_special & KVM_SREGS_E_UPDATE_TSR) {
-        /*
-         * FIXME: existing KVM timer handling is incomplete.
-         * TSR cannot be read by the guest, and its value in
-         * vcpu->arch is always zero.  For now, just handle
-         * the case where the caller is trying to inject a
-         * decrementer interrupt.
-         */
-
-        if ((sregs->u.e.tsr & TSR_DIS) &&
-            (vcpu->arch.tcr & TCR_DIE))
-            kvmppc_core_queue_dec(vcpu);
+        kvmppc_clr_tsr_bits(vcpu, ~0);
+        kvmppc_set_tsr_bits(vcpu, sregs->u.e.tsr);
   }
 
   return 0;
@@ -895,6 +897,48 @@ void kvmppc_core_destroy_vm(struct kvm *kvm)
 {
 }
 
+void kvmppc_set_tcr(struct kvm_vcpu *vcpu, u32 new_tcr)
+{
+    vcpu->arch.tcr = new_tcr;
+    smp_wmb();
+
+    /*
+     * Since TCR changed, we need to check
+     * if blocked interrupts are deliverable.

arch/powerpc/kvm/booke.c void kvmppc_core_destroy_vm(struct kvm *kvm) · line 917

Quoted 10 lines
arch/powerpc/kvm/booke.c +10 -0
+     */
+    if ((new_tcr & TCR_DIE) && (vcpu->arch.tsr & TSR_DIS)) {
+        kvmppc_core_queue_dec(vcpu);
+        kvmppc_wakeup_vcpu(vcpu);
+    }
+}
+
+void kvmppc_set_tsr_bits(struct kvm_vcpu *vcpu, u32 tsr_bits)
+{
+    set_bits(tsr_bits, &vcpu->arch.tsr);

I'm still missing the point why you need set_bits/clear_bits. Can't you just use |= and &= ~ like everyone else? :)

Quoted 5 lines
Quoted 3 lines

Because set_bits/clear_bits is atomic.
set/clr_tsr_bits() may be called in different threads so that on different cores. We need to make sure modification to be atomic to prevent missing changes.

Ah, so you can set the TSR bit in the decrementer timer callback? Hrm. Please add a comment there saying so. All the other cases should be under vcpu_load() scope. If they're not, we have bugs in our code :). So they shouldn't need atomic operations (except for now they do because there's one user that doesn't do vcpu_load).

Thinking about this for a bit more, shouldn't we be able to share the TSR with the guest so that it can clear the DEC interrupt without exiting the guest? Then we can't do ulong because that would make the pv abi 32/64 specific.

Overall, I'm also not very sure it's too useful to set TSR from non-vcpu context. After all, in most cases the guest should have interrupts enabled at that point and probably wants to receive the interrupt in that moment, not some seconds later when the next intercept occurs. So we still want to kick the vcpu thread on dec interrupts. And then we don't need anything to be atomic.

Alex

Quoted 1 line
#5 lore raw
In reply to Scott Wood
Quoted 256 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Alexander Graf
Sent: Wednesday, September 07, 2011 3:06 AM
To: Liu Yu-B13201
Cc: Wood Scott-B07421; [email protected]
Subject: Re: [PATCH 5/5] KVM: PPC: booke: Improve timer register emulation

Am 06.09.2011 um 10:04 schrieb Alexander Graf <[email protected]>:

Quoted 79 lines · On 06.09.2011, at 05:17, Liu Yu-B13201 wrote:
Quoted 77 lines
Quoted 77 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Alexander Graf
Sent: Tuesday, September 06, 2011 6:45 AM
To: Wood Scott-B07421
Cc: [email protected]
Subject: Re: [PATCH 5/5] KVM: PPC: booke: Improve timer register emulation

Quoted 6 lines · Scott Wood

From: Liu Yu <[email protected]>

Decrementers are now properly driven by TCR/TSR, and the guest has full read/write access to these registers.

The decrementer keeps ticking (and setting the TSR bit)

regardless of

Quoted 6 lines

whether the interrupts are enabled with TCR.

The decrementer stops at zero, rather than going negative.

[scott: added dequeue in kvmppc_booke_irqprio_deliver, and

dec stop-at-zero]

Quoted 6 lines

---

arch/powerpc/include/asm/kvm_host.h |    2 +-
arch/powerpc/include/asm/kvm_ppc.h  |   11 +++++
arch/powerpc/kvm/book3s.c           |    8 ++++
arch/powerpc/kvm/booke.c            |   80 

+++++++++++++++++++++++++++--------

Review

Quoted 7 lines
arch/powerpc/kvm/booke.h            |    4 ++
arch/powerpc/kvm/booke_emulate.c    |   11 ++++-
arch/powerpc/kvm/emulate.c          |   45 ++++++++-----------
arch/powerpc/kvm/powerpc.c          |   20 +--------
8 files changed, 114 insertions(+), 67 deletions(-)
(quoted diff) +0 -0

b/arch/powerpc/include/asm/kvm_host.h

arch/powerpc/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 337

Quoted 9 lines

index 3305af4..ea08c79 100644

arch/powerpc/include/asm/kvm_host.h +1 -1
@@ -334,7 +334,7 @@ struct kvm_vcpu_arch {
   u32 tbl;
   u32 tbu;
   u32 tcr;
-    u32 tsr;
+    ulong tsr; /* we need to perform set/clr_bits() which 

requires ulong */

Review

Quoted 4 lines
arch/powerpc/include/asm/kvm_host.h +0 -0
   u32 ivor[64];
   ulong ivpr;
   u32 pvr;
(quoted diff) +0 -0

b/arch/powerpc/include/asm/kvm_ppc.h

arch/powerpc/include/asm/kvm_ppc.h extern int

Quoted 4 lines

index bdaa6c8..ddaa615 100644

arch/powerpc/include/asm/kvm_ppc.h +0 -0
@@ -66,6 +66,7 @@ extern int 

kvmppc_emulate_instruction(struct kvm_run *run,

Quoted 1 line

extern int kvmppc_emulate_mmio(struct kvm_run *run, struct

kvm_vcpu *vcpu);

arch/powerpc/include/asm/kvm_ppc.h int kvm_vcpu_ioctl_config_tlb(struct

Quoted 7 lines

extern void kvmppc_emulate_dec(struct kvm_vcpu *vcpu); extern u32 kvmppc_get_dec(struct kvm_vcpu *vcpu, u64 tb); +extern void kvmppc_decrementer_func(unsigned long data);

/* Core-specific hooks */

arch/powerpc/include/asm/kvm_ppc.h +0 -0
@@ -197,4 +198,14 @@ int kvm_vcpu_ioctl_config_tlb(struct 

kvm_vcpu *vcpu,

Quoted 3 lines
arch/powerpc/include/asm/kvm_ppc.h +0 -0
 int kvm_vcpu_ioctl_dirty_tlb(struct kvm_vcpu *vcpu,
                struct kvm_dirty_tlb *cfg);
 
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
arch/powerpc/include/asm/kvm_ppc.h +4 -0
+static inline void kvmppc_wakeup_vcpu(struct kvm_vcpu *vcpu)
+{
+    if (waitqueue_active(&vcpu->wq)) {
+        wake_up_interruptible(&vcpu->wq);
Quoted 8 lines
Quoted 8 lines

Review

Quoted 8 lines
arch/powerpc/include/asm/kvm_ppc.h +6 -0
+        vcpu->stat.halt_wakeup++;
+    } else if (vcpu->cpu != -1) {
+        smp_send_reschedule(vcpu->cpu);
+    }
+}
+
 #endif /* __POWERPC_KVM_PPC_H__ */
(quoted diff) +0 -0

b/arch/powerpc/kvm/book3s.c

Quoted 131 lines
Quoted 131 lines
Quoted 130 lines

arch/powerpc/kvm/booke.c void

Quoted 20 lines

index f68a34d..b057856 100644

arch/powerpc/kvm/book3s.c +8 -0
@@ -514,3 +514,11 @@ out:
   mutex_unlock(&kvm->slots_lock);
   return r;
 }
+
+void kvmppc_decrementer_func(unsigned long data)
+{
+    struct kvm_vcpu *vcpu = (struct kvm_vcpu *)data;
+
+    kvmppc_core_queue_dec(vcpu);
+    kvmppc_wakeup_vcpu(vcpu);
+}
arch/powerpc/kvm/booke.c +0 -0
index 0ed62c1..502f9ff 100644
@@ -205,7 +205,8 @@ void 

kvmppc_core_dequeue_external(struct kvm_vcpu *vcpu,

arch/powerpc/kvm/booke.c static int

Quoted 10 lines
arch/powerpc/kvm/booke.c +2 -1
 static int kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu,
                                      unsigned int priority)
 {
-    int allowed = 0;
+    int allowed = 1;
+    int dequeue = 0;
   ulong uninitialized_var(msr_mask);
   bool update_esr = false, update_dear = false;
   ulong crit_raw = vcpu->arch.shared->critical;
@@ -258,10 +259,15 @@ static int 

kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu,

arch/powerpc/kvm/booke.c static int

Quoted 18 lines
arch/powerpc/kvm/booke.c +7 -2
       allowed = vcpu->arch.shared->msr & MSR_ME;
       msr_mask = 0;
       break;
-    case BOOKE_IRQPRIO_EXTERNAL:
   case BOOKE_IRQPRIO_DECREMENTER:
+        if (!(vcpu->arch.tcr & TCR_DIE)) {
+            allowed = 0;
+            dequeue = 1;
+        }
+        /* fall through */
+    case BOOKE_IRQPRIO_EXTERNAL:
   case BOOKE_IRQPRIO_FIT:
-        allowed = vcpu->arch.shared->msr & MSR_EE;
+        allowed = allowed && vcpu->arch.shared->msr & MSR_EE;
       allowed = allowed && !crit;
       msr_mask = MSR_CE|MSR_ME|MSR_DE;
       break;
@@ -269,6 +275,8 @@ static int 

kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu,

arch/powerpc/kvm/booke.c static int

Quoted 9 lines
arch/powerpc/kvm/booke.c +2 -0
       allowed = vcpu->arch.shared->msr & MSR_DE;
       msr_mask = MSR_ME;
       break;
+    default:
+        allowed = 0;
   }
 
   if (allowed) {
@@ -283,6 +291,9 @@ static int 

kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu,

arch/powerpc/kvm/booke.c static int · line 292

Quoted 3 lines
arch/powerpc/kvm/booke.c +0 -0
       if (!keep_irq)
           clear_bit(priority, 

&vcpu->arch.pending_exceptions);

arch/powerpc/kvm/booke.c static int set_sregs_base(struct

Quoted 7 lines
arch/powerpc/kvm/booke.c +3 -0
+    } else if (dequeue) {
+        /* Should only happen due to races with masking */
+        clear_bit(priority, &vcpu->arch.pending_exceptions);
   }
 
   return allowed;
@@ -721,25 +732,16 @@ static int set_sregs_base(struct 

kvm_vcpu *vcpu,

Quoted 43 lines
arch/powerpc/kvm/booke.c +14 -15
   vcpu->arch.shared->esr = sregs->u.e.esr;
   vcpu->arch.shared->dar = sregs->u.e.dear;
   vcpu->arch.vrsave = sregs->u.e.vrsave;
-    vcpu->arch.tcr = sregs->u.e.tcr;
+    kvmppc_set_tcr(vcpu, sregs->u.e.tcr);
 
-    if (sregs->u.e.update_special & KVM_SREGS_E_UPDATE_DEC)
+    if (sregs->u.e.update_special & KVM_SREGS_E_UPDATE_DEC) {
       vcpu->arch.dec = sregs->u.e.dec;
-
-    kvmppc_emulate_dec(vcpu);
+        kvmppc_emulate_dec(vcpu);
+    }
 
   if (sregs->u.e.update_special & KVM_SREGS_E_UPDATE_TSR) {
-        /*
-         * FIXME: existing KVM timer handling is incomplete.
-         * TSR cannot be read by the guest, and its value in
-         * vcpu->arch is always zero.  For now, just handle
-         * the case where the caller is trying to inject a
-         * decrementer interrupt.
-         */
-
-        if ((sregs->u.e.tsr & TSR_DIS) &&
-            (vcpu->arch.tcr & TCR_DIE))
-            kvmppc_core_queue_dec(vcpu);
+        kvmppc_clr_tsr_bits(vcpu, ~0);
+        kvmppc_set_tsr_bits(vcpu, sregs->u.e.tsr);
   }
 
   return 0;
@@ -895,6 +897,48 @@ void kvmppc_core_destroy_vm(struct kvm *kvm)
 {
 }
 
+void kvmppc_set_tcr(struct kvm_vcpu *vcpu, u32 new_tcr)
+{
+    vcpu->arch.tcr = new_tcr;
+    smp_wmb();
+
+    /*
+     * Since TCR changed, we need to check
+     * if blocked interrupts are deliverable.

arch/powerpc/kvm/booke.c void kvmppc_core_destroy_vm(struct kvm *kvm) · line 917

Quoted 10 lines
arch/powerpc/kvm/booke.c +10 -0
+     */
+    if ((new_tcr & TCR_DIE) && (vcpu->arch.tsr & TSR_DIS)) {
+        kvmppc_core_queue_dec(vcpu);
+        kvmppc_wakeup_vcpu(vcpu);
+    }
+}
+
+void kvmppc_set_tsr_bits(struct kvm_vcpu *vcpu, u32 tsr_bits)
+{
+    set_bits(tsr_bits, &vcpu->arch.tsr);

I'm still missing the point why you need set_bits/clear_bits. Can't you just use |= and &= ~ like everyone else? :)

Quoted 2 lines
Quoted 2 lines

Because set_bits/clear_bits is atomic. set/clr_tsr_bits() may be called in different threads so

that on different cores.

Quoted 1 line
Quoted 1 line

We need to make sure modification to be atomic to prevent

missing changes.

Quoted 2 lines

Ah, so you can set the TSR bit in the decrementer timer

callback? Hrm. Please add a comment there saying so. All the other cases should be under vcpu_load() scope. If they're not, we have bugs in our code :). So they shouldn't need atomic operations (except for now they do because there's one user that doesn't do vcpu_load).

Thinking about this for a bit more, shouldn't we be able to share the TSR with the guest so that it can clear the DEC interrupt without exiting the guest? Then we can't do ulong because that would make the pv abi 32/64 specific.

Quoted 8 lines

Overall, I'm also not very sure it's too useful to set TSR from non-vcpu context. After all, in most cases the guest should have interrupts enabled at that point and probably wants to receive the interrupt in that moment, not some seconds later when the next intercept occurs. So we still want to kick the vcpu thread on dec interrupts. And then we don't need anything to be atomic.

We don't only set TSR for dec interrupt case.
In future, we would send out watchdog stuff which also set/clear TSR in non-vcpu context. The watchdog bit of TSR is not only related to watchdog interrupt but maintains a simple statemachine. Not set those states may confuse guest OS.

Thanks,
Yu

#6 lore raw
In reply to Scott Wood
Quoted 265 lines · On 07.09.2011, at 12:16, Liu Yu-B13201 wrote:
Quoted 256 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Alexander Graf
Sent: Wednesday, September 07, 2011 3:06 AM
To: Liu Yu-B13201
Cc: Wood Scott-B07421; [email protected]
Subject: Re: [PATCH 5/5] KVM: PPC: booke: Improve timer register emulation

Am 06.09.2011 um 10:04 schrieb Alexander Graf <[email protected]>:

Quoted 79 lines · On 06.09.2011, at 05:17, Liu Yu-B13201 wrote:
Quoted 77 lines
Quoted 77 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Alexander Graf
Sent: Tuesday, September 06, 2011 6:45 AM
To: Wood Scott-B07421
Cc: [email protected]
Subject: Re: [PATCH 5/5] KVM: PPC: booke: Improve timer register emulation

Quoted 6 lines · Scott Wood

From: Liu Yu <[email protected]>

Decrementers are now properly driven by TCR/TSR, and the guest has full read/write access to these registers.

The decrementer keeps ticking (and setting the TSR bit)

regardless of

Quoted 6 lines

whether the interrupts are enabled with TCR.

The decrementer stops at zero, rather than going negative.

[scott: added dequeue in kvmppc_booke_irqprio_deliver, and

dec stop-at-zero]

Quoted 6 lines

---

arch/powerpc/include/asm/kvm_host.h |    2 +-
arch/powerpc/include/asm/kvm_ppc.h  |   11 +++++
arch/powerpc/kvm/book3s.c           |    8 ++++
arch/powerpc/kvm/booke.c            |   80 

+++++++++++++++++++++++++++--------

Review

Quoted 7 lines
arch/powerpc/kvm/booke.h            |    4 ++
arch/powerpc/kvm/booke_emulate.c    |   11 ++++-
arch/powerpc/kvm/emulate.c          |   45 ++++++++-----------
arch/powerpc/kvm/powerpc.c          |   20 +--------
8 files changed, 114 insertions(+), 67 deletions(-)
(quoted diff) +0 -0

b/arch/powerpc/include/asm/kvm_host.h

arch/powerpc/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 337

Quoted 9 lines

index 3305af4..ea08c79 100644

arch/powerpc/include/asm/kvm_host.h +1 -1
@@ -334,7 +334,7 @@ struct kvm_vcpu_arch {
 u32 tbl;
 u32 tbu;
 u32 tcr;
-    u32 tsr;
+    ulong tsr; /* we need to perform set/clr_bits() which 

requires ulong */

Review

Quoted 4 lines
arch/powerpc/include/asm/kvm_host.h +0 -0
 u32 ivor[64];
 ulong ivpr;
 u32 pvr;
(quoted diff) +0 -0

b/arch/powerpc/include/asm/kvm_ppc.h

arch/powerpc/include/asm/kvm_ppc.h extern int

Quoted 4 lines

index bdaa6c8..ddaa615 100644

arch/powerpc/include/asm/kvm_ppc.h +0 -0
@@ -66,6 +66,7 @@ extern int 

kvmppc_emulate_instruction(struct kvm_run *run,

Quoted 1 line

extern int kvmppc_emulate_mmio(struct kvm_run *run, struct

kvm_vcpu *vcpu);

arch/powerpc/include/asm/kvm_ppc.h int kvm_vcpu_ioctl_config_tlb(struct

Quoted 7 lines

extern void kvmppc_emulate_dec(struct kvm_vcpu *vcpu); extern u32 kvmppc_get_dec(struct kvm_vcpu *vcpu, u64 tb); +extern void kvmppc_decrementer_func(unsigned long data);

/* Core-specific hooks */

arch/powerpc/include/asm/kvm_ppc.h +0 -0
@@ -197,4 +198,14 @@ int kvm_vcpu_ioctl_config_tlb(struct 

kvm_vcpu *vcpu,

Quoted 3 lines
arch/powerpc/include/asm/kvm_ppc.h +0 -0
 int kvm_vcpu_ioctl_dirty_tlb(struct kvm_vcpu *vcpu,
              struct kvm_dirty_tlb *cfg);
 
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
arch/powerpc/include/asm/kvm_ppc.h +4 -0
+static inline void kvmppc_wakeup_vcpu(struct kvm_vcpu *vcpu)
+{
+    if (waitqueue_active(&vcpu->wq)) {
+        wake_up_interruptible(&vcpu->wq);
Quoted 8 lines
Quoted 8 lines

Review

Quoted 8 lines
arch/powerpc/include/asm/kvm_ppc.h +6 -0
+        vcpu->stat.halt_wakeup++;
+    } else if (vcpu->cpu != -1) {
+        smp_send_reschedule(vcpu->cpu);
+    }
+}
+
 #endif /* __POWERPC_KVM_PPC_H__ */
(quoted diff) +0 -0

b/arch/powerpc/kvm/book3s.c

Quoted 131 lines
Quoted 131 lines
Quoted 130 lines

arch/powerpc/kvm/booke.c void

Quoted 20 lines

index f68a34d..b057856 100644

arch/powerpc/kvm/book3s.c +8 -0
@@ -514,3 +514,11 @@ out:
 mutex_unlock(&kvm->slots_lock);
 return r;
 }
+
+void kvmppc_decrementer_func(unsigned long data)
+{
+    struct kvm_vcpu *vcpu = (struct kvm_vcpu *)data;
+
+    kvmppc_core_queue_dec(vcpu);
+    kvmppc_wakeup_vcpu(vcpu);
+}
arch/powerpc/kvm/booke.c +0 -0
index 0ed62c1..502f9ff 100644
@@ -205,7 +205,8 @@ void 

kvmppc_core_dequeue_external(struct kvm_vcpu *vcpu,

arch/powerpc/kvm/booke.c static int

Quoted 10 lines
arch/powerpc/kvm/booke.c +2 -1
 static int kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu,
                                    unsigned int priority)
 {
-    int allowed = 0;
+    int allowed = 1;
+    int dequeue = 0;
 ulong uninitialized_var(msr_mask);
 bool update_esr = false, update_dear = false;
 ulong crit_raw = vcpu->arch.shared->critical;
@@ -258,10 +259,15 @@ static int 

kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu,

arch/powerpc/kvm/booke.c static int

Quoted 18 lines
arch/powerpc/kvm/booke.c +7 -2
     allowed = vcpu->arch.shared->msr & MSR_ME;
     msr_mask = 0;
     break;
-    case BOOKE_IRQPRIO_EXTERNAL:
 case BOOKE_IRQPRIO_DECREMENTER:
+        if (!(vcpu->arch.tcr & TCR_DIE)) {
+            allowed = 0;
+            dequeue = 1;
+        }
+        /* fall through */
+    case BOOKE_IRQPRIO_EXTERNAL:
 case BOOKE_IRQPRIO_FIT:
-        allowed = vcpu->arch.shared->msr & MSR_EE;
+        allowed = allowed && vcpu->arch.shared->msr & MSR_EE;
     allowed = allowed && !crit;
     msr_mask = MSR_CE|MSR_ME|MSR_DE;
     break;
@@ -269,6 +275,8 @@ static int 

kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu,

arch/powerpc/kvm/booke.c static int

Quoted 9 lines
arch/powerpc/kvm/booke.c +2 -0
     allowed = vcpu->arch.shared->msr & MSR_DE;
     msr_mask = MSR_ME;
     break;
+    default:
+        allowed = 0;
 }
 
 if (allowed) {
@@ -283,6 +291,9 @@ static int 

kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu,

arch/powerpc/kvm/booke.c static int · line 292

Quoted 3 lines
arch/powerpc/kvm/booke.c +0 -0
     if (!keep_irq)
         clear_bit(priority, 

&vcpu->arch.pending_exceptions);

arch/powerpc/kvm/booke.c static int set_sregs_base(struct

Quoted 7 lines
arch/powerpc/kvm/booke.c +3 -0
+    } else if (dequeue) {
+        /* Should only happen due to races with masking */
+        clear_bit(priority, &vcpu->arch.pending_exceptions);
 }
 
 return allowed;
@@ -721,25 +732,16 @@ static int set_sregs_base(struct 

kvm_vcpu *vcpu,

Quoted 43 lines
arch/powerpc/kvm/booke.c +14 -15
 vcpu->arch.shared->esr = sregs->u.e.esr;
 vcpu->arch.shared->dar = sregs->u.e.dear;
 vcpu->arch.vrsave = sregs->u.e.vrsave;
-    vcpu->arch.tcr = sregs->u.e.tcr;
+    kvmppc_set_tcr(vcpu, sregs->u.e.tcr);
 
-    if (sregs->u.e.update_special & KVM_SREGS_E_UPDATE_DEC)
+    if (sregs->u.e.update_special & KVM_SREGS_E_UPDATE_DEC) {
     vcpu->arch.dec = sregs->u.e.dec;
-
-    kvmppc_emulate_dec(vcpu);
+        kvmppc_emulate_dec(vcpu);
+    }
 
 if (sregs->u.e.update_special & KVM_SREGS_E_UPDATE_TSR) {
-        /*
-         * FIXME: existing KVM timer handling is incomplete.
-         * TSR cannot be read by the guest, and its value in
-         * vcpu->arch is always zero.  For now, just handle
-         * the case where the caller is trying to inject a
-         * decrementer interrupt.
-         */
-
-        if ((sregs->u.e.tsr & TSR_DIS) &&
-            (vcpu->arch.tcr & TCR_DIE))
-            kvmppc_core_queue_dec(vcpu);
+        kvmppc_clr_tsr_bits(vcpu, ~0);
+        kvmppc_set_tsr_bits(vcpu, sregs->u.e.tsr);
 }
 
 return 0;
@@ -895,6 +897,48 @@ void kvmppc_core_destroy_vm(struct kvm *kvm)
 {
 }
 
+void kvmppc_set_tcr(struct kvm_vcpu *vcpu, u32 new_tcr)
+{
+    vcpu->arch.tcr = new_tcr;
+    smp_wmb();
+
+    /*
+     * Since TCR changed, we need to check
+     * if blocked interrupts are deliverable.

arch/powerpc/kvm/booke.c void kvmppc_core_destroy_vm(struct kvm *kvm) · line 917

Quoted 10 lines
arch/powerpc/kvm/booke.c +10 -0
+     */
+    if ((new_tcr & TCR_DIE) && (vcpu->arch.tsr & TSR_DIS)) {
+        kvmppc_core_queue_dec(vcpu);
+        kvmppc_wakeup_vcpu(vcpu);
+    }
+}
+
+void kvmppc_set_tsr_bits(struct kvm_vcpu *vcpu, u32 tsr_bits)
+{
+    set_bits(tsr_bits, &vcpu->arch.tsr);

I'm still missing the point why you need set_bits/clear_bits. Can't you just use |= and &= ~ like everyone else? :)

Quoted 2 lines
Quoted 2 lines

Because set_bits/clear_bits is atomic. set/clr_tsr_bits() may be called in different threads so

that on different cores.

Quoted 1 line
Quoted 1 line

We need to make sure modification to be atomic to prevent

missing changes.

Quoted 2 lines

Ah, so you can set the TSR bit in the decrementer timer

callback? Hrm. Please add a comment there saying so. All the other cases should be under vcpu_load() scope. If they're not, we have bugs in our code :). So they shouldn't need atomic operations (except for now they do because there's one user that doesn't do vcpu_load).

Thinking about this for a bit more, shouldn't we be able to share the TSR with the guest so that it can clear the DEC interrupt without exiting the guest? Then we can't do ulong because that would make the pv abi 32/64 specific.

Quoted 8 lines

Overall, I'm also not very sure it's too useful to set TSR from non-vcpu context. After all, in most cases the guest should have interrupts enabled at that point and probably wants to receive the interrupt in that moment, not some seconds later when the next intercept occurs. So we still want to kick the vcpu thread on dec interrupts. And then we don't need anything to be atomic.

Quoted 4 lines

We don't only set TSR for dec interrupt case.
In future, we would send out watchdog stuff which also set/clear TSR in non-vcpu context. The watchdog bit of TSR is not only related to watchdog interrupt but maintains a simple statemachine. Not set those states may confuse guest OS.

Yes, but why can't we do this in the vcpu thread's context so we only ever have a single instance accessing the vcpu struct? It makes a lot of things a lot easier. And for the watchdog, we don't have to set the bit unless it's clean, right? So we don't add any overhead to the vcpu unless it's actually cleaning the watchdog TSR bit.

Alex

#7 lore raw
In reply to Scott Wood
Quoted 286 lines

-----Original Message-----
From: Alexander Graf [mailto:[email protected]]
Sent: Wednesday, September 07, 2011 6:42 PM
To: Liu Yu-B13201
Cc: Wood Scott-B07421; [email protected]
Subject: Re: [PATCH 5/5] KVM: PPC: booke: Improve timer register emulation

Quoted 16 lines · On 07.09.2011, at 12:16, Liu Yu-B13201 wrote:
Quoted 16 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Alexander Graf
Sent: Wednesday, September 07, 2011 3:06 AM
To: Liu Yu-B13201
Cc: Wood Scott-B07421; [email protected]
Subject: Re: [PATCH 5/5] KVM: PPC: booke: Improve timer register emulation

Am 06.09.2011 um 10:04 schrieb Alexander Graf <[email protected]>:

Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines · On 06.09.2011, at 05:17, Liu Yu-B13201 wrote:
Quoted 6 lines
Quoted 6 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of

Alexander Graf

Quoted 84 lines
Quoted 84 lines
Quoted 71 lines
Quoted 71 lines
Quoted 71 lines

Sent: Tuesday, September 06, 2011 6:45 AM
To: Wood Scott-B07421
Cc: [email protected]
Subject: Re: [PATCH 5/5] KVM: PPC: booke: Improve timer register emulation

Quoted 6 lines · Scott Wood

From: Liu Yu <[email protected]>

Decrementers are now properly driven by TCR/TSR, and the guest has full read/write access to these registers.

The decrementer keeps ticking (and setting the TSR bit)

regardless of

Quoted 6 lines

whether the interrupts are enabled with TCR.

The decrementer stops at zero, rather than going negative.

[scott: added dequeue in kvmppc_booke_irqprio_deliver, and

dec stop-at-zero]

Quoted 6 lines

---

arch/powerpc/include/asm/kvm_host.h |    2 +-
arch/powerpc/include/asm/kvm_ppc.h  |   11 +++++
arch/powerpc/kvm/book3s.c           |    8 ++++
arch/powerpc/kvm/booke.c            |   80 

+++++++++++++++++++++++++++--------

Review

Quoted 7 lines
arch/powerpc/kvm/booke.h            |    4 ++
arch/powerpc/kvm/booke_emulate.c    |   11 ++++-
arch/powerpc/kvm/emulate.c          |   45 ++++++++-----------
arch/powerpc/kvm/powerpc.c          |   20 +--------
8 files changed, 114 insertions(+), 67 deletions(-)
(quoted diff) +0 -0

b/arch/powerpc/include/asm/kvm_host.h

arch/powerpc/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 337

Quoted 9 lines

index 3305af4..ea08c79 100644

arch/powerpc/include/asm/kvm_host.h +1 -1
@@ -334,7 +334,7 @@ struct kvm_vcpu_arch {
 u32 tbl;
 u32 tbu;
 u32 tcr;
-    u32 tsr;
+    ulong tsr; /* we need to perform set/clr_bits() which 

requires ulong */

Review

Quoted 4 lines
arch/powerpc/include/asm/kvm_host.h +0 -0
 u32 ivor[64];
 ulong ivpr;
 u32 pvr;
(quoted diff) +0 -0

b/arch/powerpc/include/asm/kvm_ppc.h

arch/powerpc/include/asm/kvm_ppc.h extern int

Quoted 4 lines

index bdaa6c8..ddaa615 100644

arch/powerpc/include/asm/kvm_ppc.h +0 -0
@@ -66,6 +66,7 @@ extern int 

kvmppc_emulate_instruction(struct kvm_run *run,

Quoted 1 line

extern int kvmppc_emulate_mmio(struct kvm_run *run, struct

kvm_vcpu *vcpu);

arch/powerpc/include/asm/kvm_ppc.h int kvm_vcpu_ioctl_config_tlb(struct

Quoted 7 lines

extern void kvmppc_emulate_dec(struct kvm_vcpu *vcpu); extern u32 kvmppc_get_dec(struct kvm_vcpu *vcpu, u64 tb); +extern void kvmppc_decrementer_func(unsigned long data);

/* Core-specific hooks */

arch/powerpc/include/asm/kvm_ppc.h +0 -0
@@ -197,4 +198,14 @@ int kvm_vcpu_ioctl_config_tlb(struct 

kvm_vcpu *vcpu,

Quoted 3 lines
arch/powerpc/include/asm/kvm_ppc.h +0 -0
 int kvm_vcpu_ioctl_dirty_tlb(struct kvm_vcpu *vcpu,
              struct kvm_dirty_tlb *cfg);
 
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
arch/powerpc/include/asm/kvm_ppc.h +4 -0
+static inline void kvmppc_wakeup_vcpu(struct kvm_vcpu *vcpu)
+{
+    if (waitqueue_active(&vcpu->wq)) {
+        wake_up_interruptible(&vcpu->wq);
Quoted 8 lines
Quoted 8 lines

Review

Quoted 8 lines
arch/powerpc/include/asm/kvm_ppc.h +6 -0
+        vcpu->stat.halt_wakeup++;
+    } else if (vcpu->cpu != -1) {
+        smp_send_reschedule(vcpu->cpu);
+    }
+}
+
 #endif /* __POWERPC_KVM_PPC_H__ */
(quoted diff) +0 -0

b/arch/powerpc/kvm/book3s.c

Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

index f68a34d..b057856 100644

arch/powerpc/kvm/book3s.c +1 -0
@@ -514,3 +514,11 @@ out:
 mutex_unlock(&kvm->slots_lock);
 return r;
 }
+
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Review

Quoted 8 lines
arch/powerpc/kvm/book3s.c +7 -0
+void kvmppc_decrementer_func(unsigned long data)
+{
+    struct kvm_vcpu *vcpu = (struct kvm_vcpu *)data;
+
+    kvmppc_core_queue_dec(vcpu);
+    kvmppc_wakeup_vcpu(vcpu);
+}
(quoted diff) +0 -0

b/arch/powerpc/kvm/booke.c

Quoted 87 lines
Quoted 87 lines
Quoted 87 lines
Quoted 87 lines
Quoted 87 lines

arch/powerpc/kvm/booke.c void

Quoted 4 lines

index 0ed62c1..502f9ff 100644

arch/powerpc/kvm/booke.c +0 -0
@@ -205,7 +205,8 @@ void 

kvmppc_core_dequeue_external(struct kvm_vcpu *vcpu,

arch/powerpc/kvm/booke.c static int

Quoted 10 lines
arch/powerpc/kvm/booke.c +2 -1
 static int kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu,
                                    unsigned int priority)
 {
-    int allowed = 0;
+    int allowed = 1;
+    int dequeue = 0;
 ulong uninitialized_var(msr_mask);
 bool update_esr = false, update_dear = false;
 ulong crit_raw = vcpu->arch.shared->critical;
@@ -258,10 +259,15 @@ static int 

kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu,

arch/powerpc/kvm/booke.c static int

Quoted 18 lines
arch/powerpc/kvm/booke.c +7 -2
     allowed = vcpu->arch.shared->msr & MSR_ME;
     msr_mask = 0;
     break;
-    case BOOKE_IRQPRIO_EXTERNAL:
 case BOOKE_IRQPRIO_DECREMENTER:
+        if (!(vcpu->arch.tcr & TCR_DIE)) {
+            allowed = 0;
+            dequeue = 1;
+        }
+        /* fall through */
+    case BOOKE_IRQPRIO_EXTERNAL:
 case BOOKE_IRQPRIO_FIT:
-        allowed = vcpu->arch.shared->msr & MSR_EE;
+        allowed = allowed && vcpu->arch.shared->msr & MSR_EE;
     allowed = allowed && !crit;
     msr_mask = MSR_CE|MSR_ME|MSR_DE;
     break;
@@ -269,6 +275,8 @@ static int 

kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu,

arch/powerpc/kvm/booke.c static int

Quoted 9 lines
arch/powerpc/kvm/booke.c +2 -0
     allowed = vcpu->arch.shared->msr & MSR_DE;
     msr_mask = MSR_ME;
     break;
+    default:
+        allowed = 0;
 }
 
 if (allowed) {
@@ -283,6 +291,9 @@ static int 

kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu,

arch/powerpc/kvm/booke.c static int · line 292

Quoted 3 lines
arch/powerpc/kvm/booke.c +0 -0
     if (!keep_irq)
         clear_bit(priority, 

&vcpu->arch.pending_exceptions);

arch/powerpc/kvm/booke.c static int set_sregs_base(struct

Quoted 7 lines
arch/powerpc/kvm/booke.c +3 -0
+    } else if (dequeue) {
+        /* Should only happen due to races with masking */
+        clear_bit(priority, &vcpu->arch.pending_exceptions);
 }
 
 return allowed;
@@ -721,25 +732,16 @@ static int set_sregs_base(struct 

kvm_vcpu *vcpu,

Quoted 30 lines
arch/powerpc/kvm/booke.c +6 -15
 vcpu->arch.shared->esr = sregs->u.e.esr;
 vcpu->arch.shared->dar = sregs->u.e.dear;
 vcpu->arch.vrsave = sregs->u.e.vrsave;
-    vcpu->arch.tcr = sregs->u.e.tcr;
+    kvmppc_set_tcr(vcpu, sregs->u.e.tcr);
 
-    if (sregs->u.e.update_special & KVM_SREGS_E_UPDATE_DEC)
+    if (sregs->u.e.update_special & KVM_SREGS_E_UPDATE_DEC) {
     vcpu->arch.dec = sregs->u.e.dec;
-
-    kvmppc_emulate_dec(vcpu);
+        kvmppc_emulate_dec(vcpu);
+    }
 
 if (sregs->u.e.update_special & KVM_SREGS_E_UPDATE_TSR) {
-        /*
-         * FIXME: existing KVM timer handling is incomplete.
-         * TSR cannot be read by the guest, and its value in
-         * vcpu->arch is always zero.  For now, just handle
-         * the case where the caller is trying to inject a
-         * decrementer interrupt.
-         */
-
-        if ((sregs->u.e.tsr & TSR_DIS) &&
-            (vcpu->arch.tcr & TCR_DIE))
-            kvmppc_core_queue_dec(vcpu);
+        kvmppc_clr_tsr_bits(vcpu, ~0);
+        kvmppc_set_tsr_bits(vcpu, sregs->u.e.tsr);
 }
 
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

arch/powerpc/kvm/booke.c void

Quoted 2 lines
arch/powerpc/kvm/booke.c +0 -0
 return 0;
@@ -895,6 +897,48 @@ void 

kvmppc_core_destroy_vm(struct kvm *kvm)

Quoted 53 lines
Quoted 44 lines
Quoted 26 lines
Quoted 26 lines
Quoted 25 lines
Quoted 11 lines
arch/powerpc/kvm/booke.c +8 -0
 {
 }
 
+void kvmppc_set_tcr(struct kvm_vcpu *vcpu, u32 new_tcr)
+{
+    vcpu->arch.tcr = new_tcr;
+    smp_wmb();
+
+    /*
+     * Since TCR changed, we need to check
+     * if blocked interrupts are deliverable.

arch/powerpc/kvm/booke.c void kvmppc_core_destroy_vm(struct kvm *kvm) · line 917

Quoted 10 lines
arch/powerpc/kvm/booke.c +10 -0
+     */
+    if ((new_tcr & TCR_DIE) && (vcpu->arch.tsr & TSR_DIS)) {
+        kvmppc_core_queue_dec(vcpu);
+        kvmppc_wakeup_vcpu(vcpu);
+    }
+}
+
+void kvmppc_set_tsr_bits(struct kvm_vcpu *vcpu, u32 tsr_bits)
+{
+    set_bits(tsr_bits, &vcpu->arch.tsr);

I'm still missing the point why you need set_bits/clear_bits. Can't you just use |= and &= ~ like everyone else? :)

Quoted 2 lines
Quoted 2 lines

Because set_bits/clear_bits is atomic. set/clr_tsr_bits() may be called in different threads so

that on different cores.

Quoted 1 line
Quoted 1 line

We need to make sure modification to be atomic to prevent

missing changes.

Quoted 2 lines

Ah, so you can set the TSR bit in the decrementer timer

callback? Hrm. Please add a comment there saying so. All the other cases should be under vcpu_load() scope. If they're not, we have bugs in our code :). So they shouldn't need atomic operations (except for now they do because there's one user that doesn't do vcpu_load).

Thinking about this for a bit more, shouldn't we be able to share the TSR with the guest so that it can clear the DEC interrupt without exiting the guest? Then we can't do ulong because that would make the pv abi 32/64 specific.

Quoted 8 lines

Overall, I'm also not very sure it's too useful to set TSR from non-vcpu context. After all, in most cases the guest should have interrupts enabled at that point and probably wants to receive the interrupt in that moment, not some seconds later when the next intercept occurs. So we still want to kick the vcpu thread on dec interrupts. And then we don't need anything to be atomic.

Quoted 2 lines

We don't only set TSR for dec interrupt case. In future, we would send out watchdog stuff which also

set/clear TSR in non-vcpu context.

Quoted 1 line

The watchdog bit of TSR is not only related to watchdog

interrupt but maintains a simple statemachine.

Quoted 9 lines
Quoted 1 line

Not set those states may confuse guest OS.

Yes, but why can't we do this in the vcpu thread's context so we only ever have a single instance accessing the vcpu struct? It makes a lot of things a lot easier. And for the watchdog, we don't have to set the bit unless it's clean, right? So we don't add any overhead to the vcpu unless it's actually cleaning the watchdog TSR bit.

Yes. We can try lazy update. :)

Thanks,
Yu

#8 lore raw
In reply to Scott Wood
Quoted 286 lines

-----Original Message-----
From: Alexander Graf [mailto:[email protected]]
Sent: Wednesday, September 07, 2011 6:42 PM
To: Liu Yu-B13201
Cc: Wood Scott-B07421; [email protected]
Subject: Re: [PATCH 5/5] KVM: PPC: booke: Improve timer register emulation

Quoted 16 lines · On 07.09.2011, at 12:16, Liu Yu-B13201 wrote:
Quoted 16 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Alexander Graf
Sent: Wednesday, September 07, 2011 3:06 AM
To: Liu Yu-B13201
Cc: Wood Scott-B07421; [email protected]
Subject: Re: [PATCH 5/5] KVM: PPC: booke: Improve timer register emulation

Am 06.09.2011 um 10:04 schrieb Alexander Graf <[email protected]>:

Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines · On 06.09.2011, at 05:17, Liu Yu-B13201 wrote:
Quoted 6 lines
Quoted 6 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of

Alexander Graf

Quoted 84 lines
Quoted 84 lines
Quoted 71 lines
Quoted 71 lines
Quoted 71 lines

Sent: Tuesday, September 06, 2011 6:45 AM
To: Wood Scott-B07421
Cc: [email protected]
Subject: Re: [PATCH 5/5] KVM: PPC: booke: Improve timer register emulation

Quoted 6 lines · Scott Wood

From: Liu Yu <[email protected]>

Decrementers are now properly driven by TCR/TSR, and the guest has full read/write access to these registers.

The decrementer keeps ticking (and setting the TSR bit)

regardless of

Quoted 6 lines

whether the interrupts are enabled with TCR.

The decrementer stops at zero, rather than going negative.

[scott: added dequeue in kvmppc_booke_irqprio_deliver, and

dec stop-at-zero]

Quoted 6 lines

---

arch/powerpc/include/asm/kvm_host.h |    2 +-
arch/powerpc/include/asm/kvm_ppc.h  |   11 +++++
arch/powerpc/kvm/book3s.c           |    8 ++++
arch/powerpc/kvm/booke.c            |   80 

+++++++++++++++++++++++++++--------

Review

Quoted 7 lines
arch/powerpc/kvm/booke.h            |    4 ++
arch/powerpc/kvm/booke_emulate.c    |   11 ++++-
arch/powerpc/kvm/emulate.c          |   45 ++++++++-----------
arch/powerpc/kvm/powerpc.c          |   20 +--------
8 files changed, 114 insertions(+), 67 deletions(-)
(quoted diff) +0 -0

b/arch/powerpc/include/asm/kvm_host.h

arch/powerpc/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 337

Quoted 9 lines

index 3305af4..ea08c79 100644

arch/powerpc/include/asm/kvm_host.h +1 -1
@@ -334,7 +334,7 @@ struct kvm_vcpu_arch {
 u32 tbl;
 u32 tbu;
 u32 tcr;
-    u32 tsr;
+    ulong tsr; /* we need to perform set/clr_bits() which 

requires ulong */

Review

Quoted 4 lines
arch/powerpc/include/asm/kvm_host.h +0 -0
 u32 ivor[64];
 ulong ivpr;
 u32 pvr;
(quoted diff) +0 -0

b/arch/powerpc/include/asm/kvm_ppc.h

arch/powerpc/include/asm/kvm_ppc.h extern int

Quoted 4 lines

index bdaa6c8..ddaa615 100644

arch/powerpc/include/asm/kvm_ppc.h +0 -0
@@ -66,6 +66,7 @@ extern int 

kvmppc_emulate_instruction(struct kvm_run *run,

Quoted 1 line

extern int kvmppc_emulate_mmio(struct kvm_run *run, struct

kvm_vcpu *vcpu);

arch/powerpc/include/asm/kvm_ppc.h int kvm_vcpu_ioctl_config_tlb(struct

Quoted 7 lines

extern void kvmppc_emulate_dec(struct kvm_vcpu *vcpu); extern u32 kvmppc_get_dec(struct kvm_vcpu *vcpu, u64 tb); +extern void kvmppc_decrementer_func(unsigned long data);

/* Core-specific hooks */

arch/powerpc/include/asm/kvm_ppc.h +0 -0
@@ -197,4 +198,14 @@ int kvm_vcpu_ioctl_config_tlb(struct 

kvm_vcpu *vcpu,

Quoted 3 lines
arch/powerpc/include/asm/kvm_ppc.h +0 -0
 int kvm_vcpu_ioctl_dirty_tlb(struct kvm_vcpu *vcpu,
              struct kvm_dirty_tlb *cfg);
 
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
arch/powerpc/include/asm/kvm_ppc.h +4 -0
+static inline void kvmppc_wakeup_vcpu(struct kvm_vcpu *vcpu)
+{
+    if (waitqueue_active(&vcpu->wq)) {
+        wake_up_interruptible(&vcpu->wq);
Quoted 8 lines
Quoted 8 lines

Review

Quoted 8 lines
arch/powerpc/include/asm/kvm_ppc.h +6 -0
+        vcpu->stat.halt_wakeup++;
+    } else if (vcpu->cpu != -1) {
+        smp_send_reschedule(vcpu->cpu);
+    }
+}
+
 #endif /* __POWERPC_KVM_PPC_H__ */
(quoted diff) +0 -0

b/arch/powerpc/kvm/book3s.c

Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

index f68a34d..b057856 100644

arch/powerpc/kvm/book3s.c +1 -0
@@ -514,3 +514,11 @@ out:
 mutex_unlock(&kvm->slots_lock);
 return r;
 }
+
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Review

Quoted 8 lines
arch/powerpc/kvm/book3s.c +7 -0
+void kvmppc_decrementer_func(unsigned long data)
+{
+    struct kvm_vcpu *vcpu = (struct kvm_vcpu *)data;
+
+    kvmppc_core_queue_dec(vcpu);
+    kvmppc_wakeup_vcpu(vcpu);
+}
(quoted diff) +0 -0

b/arch/powerpc/kvm/booke.c

Quoted 87 lines
Quoted 87 lines
Quoted 87 lines
Quoted 87 lines
Quoted 87 lines

arch/powerpc/kvm/booke.c void

Quoted 4 lines

index 0ed62c1..502f9ff 100644

arch/powerpc/kvm/booke.c +0 -0
@@ -205,7 +205,8 @@ void 

kvmppc_core_dequeue_external(struct kvm_vcpu *vcpu,

arch/powerpc/kvm/booke.c static int

Quoted 10 lines
arch/powerpc/kvm/booke.c +2 -1
 static int kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu,
                                    unsigned int priority)
 {
-    int allowed = 0;
+    int allowed = 1;
+    int dequeue = 0;
 ulong uninitialized_var(msr_mask);
 bool update_esr = false, update_dear = false;
 ulong crit_raw = vcpu->arch.shared->critical;
@@ -258,10 +259,15 @@ static int 

kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu,

arch/powerpc/kvm/booke.c static int

Quoted 18 lines
arch/powerpc/kvm/booke.c +7 -2
     allowed = vcpu->arch.shared->msr & MSR_ME;
     msr_mask = 0;
     break;
-    case BOOKE_IRQPRIO_EXTERNAL:
 case BOOKE_IRQPRIO_DECREMENTER:
+        if (!(vcpu->arch.tcr & TCR_DIE)) {
+            allowed = 0;
+            dequeue = 1;
+        }
+        /* fall through */
+    case BOOKE_IRQPRIO_EXTERNAL:
 case BOOKE_IRQPRIO_FIT:
-        allowed = vcpu->arch.shared->msr & MSR_EE;
+        allowed = allowed && vcpu->arch.shared->msr & MSR_EE;
     allowed = allowed && !crit;
     msr_mask = MSR_CE|MSR_ME|MSR_DE;
     break;
@@ -269,6 +275,8 @@ static int 

kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu,

arch/powerpc/kvm/booke.c static int

Quoted 9 lines
arch/powerpc/kvm/booke.c +2 -0
     allowed = vcpu->arch.shared->msr & MSR_DE;
     msr_mask = MSR_ME;
     break;
+    default:
+        allowed = 0;
 }
 
 if (allowed) {
@@ -283,6 +291,9 @@ static int 

kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu,

arch/powerpc/kvm/booke.c static int · line 292

Quoted 3 lines
arch/powerpc/kvm/booke.c +0 -0
     if (!keep_irq)
         clear_bit(priority, 

&vcpu->arch.pending_exceptions);

arch/powerpc/kvm/booke.c static int set_sregs_base(struct

Quoted 7 lines
arch/powerpc/kvm/booke.c +3 -0
+    } else if (dequeue) {
+        /* Should only happen due to races with masking */
+        clear_bit(priority, &vcpu->arch.pending_exceptions);
 }
 
 return allowed;
@@ -721,25 +732,16 @@ static int set_sregs_base(struct 

kvm_vcpu *vcpu,

Quoted 30 lines
arch/powerpc/kvm/booke.c +6 -15
 vcpu->arch.shared->esr = sregs->u.e.esr;
 vcpu->arch.shared->dar = sregs->u.e.dear;
 vcpu->arch.vrsave = sregs->u.e.vrsave;
-    vcpu->arch.tcr = sregs->u.e.tcr;
+    kvmppc_set_tcr(vcpu, sregs->u.e.tcr);
 
-    if (sregs->u.e.update_special & KVM_SREGS_E_UPDATE_DEC)
+    if (sregs->u.e.update_special & KVM_SREGS_E_UPDATE_DEC) {
     vcpu->arch.dec = sregs->u.e.dec;
-
-    kvmppc_emulate_dec(vcpu);
+        kvmppc_emulate_dec(vcpu);
+    }
 
 if (sregs->u.e.update_special & KVM_SREGS_E_UPDATE_TSR) {
-        /*
-         * FIXME: existing KVM timer handling is incomplete.
-         * TSR cannot be read by the guest, and its value in
-         * vcpu->arch is always zero.  For now, just handle
-         * the case where the caller is trying to inject a
-         * decrementer interrupt.
-         */
-
-        if ((sregs->u.e.tsr & TSR_DIS) &&
-            (vcpu->arch.tcr & TCR_DIE))
-            kvmppc_core_queue_dec(vcpu);
+        kvmppc_clr_tsr_bits(vcpu, ~0);
+        kvmppc_set_tsr_bits(vcpu, sregs->u.e.tsr);
 }
 
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

arch/powerpc/kvm/booke.c void

Quoted 2 lines
arch/powerpc/kvm/booke.c +0 -0
 return 0;
@@ -895,6 +897,48 @@ void 

kvmppc_core_destroy_vm(struct kvm *kvm)

Quoted 53 lines
Quoted 44 lines
Quoted 26 lines
Quoted 26 lines
Quoted 25 lines
Quoted 11 lines
arch/powerpc/kvm/booke.c +8 -0
 {
 }
 
+void kvmppc_set_tcr(struct kvm_vcpu *vcpu, u32 new_tcr)
+{
+    vcpu->arch.tcr = new_tcr;
+    smp_wmb();
+
+    /*
+     * Since TCR changed, we need to check
+     * if blocked interrupts are deliverable.

arch/powerpc/kvm/booke.c void kvmppc_core_destroy_vm(struct kvm *kvm) · line 917

Quoted 10 lines
arch/powerpc/kvm/booke.c +10 -0
+     */
+    if ((new_tcr & TCR_DIE) && (vcpu->arch.tsr & TSR_DIS)) {
+        kvmppc_core_queue_dec(vcpu);
+        kvmppc_wakeup_vcpu(vcpu);
+    }
+}
+
+void kvmppc_set_tsr_bits(struct kvm_vcpu *vcpu, u32 tsr_bits)
+{
+    set_bits(tsr_bits, &vcpu->arch.tsr);

I'm still missing the point why you need set_bits/clear_bits. Can't you just use |= and &= ~ like everyone else? :)

Quoted 2 lines
Quoted 2 lines

Because set_bits/clear_bits is atomic. set/clr_tsr_bits() may be called in different threads so

that on different cores.

Quoted 1 line
Quoted 1 line

We need to make sure modification to be atomic to prevent

missing changes.

Quoted 2 lines

Ah, so you can set the TSR bit in the decrementer timer

callback? Hrm. Please add a comment there saying so. All the other cases should be under vcpu_load() scope. If they're not, we have bugs in our code :). So they shouldn't need atomic operations (except for now they do because there's one user that doesn't do vcpu_load).

Thinking about this for a bit more, shouldn't we be able to share the TSR with the guest so that it can clear the DEC interrupt without exiting the guest? Then we can't do ulong because that would make the pv abi 32/64 specific.

Quoted 8 lines

Overall, I'm also not very sure it's too useful to set TSR from non-vcpu context. After all, in most cases the guest should have interrupts enabled at that point and probably wants to receive the interrupt in that moment, not some seconds later when the next intercept occurs. So we still want to kick the vcpu thread on dec interrupts. And then we don't need anything to be atomic.

Quoted 2 lines

We don't only set TSR for dec interrupt case. In future, we would send out watchdog stuff which also

set/clear TSR in non-vcpu context.

Quoted 1 line

The watchdog bit of TSR is not only related to watchdog

interrupt but maintains a simple statemachine.

Quoted 9 lines
Quoted 1 line

Not set those states may confuse guest OS.

Yes, but why can't we do this in the vcpu thread's context so we only ever have a single instance accessing the vcpu struct? It makes a lot of things a lot easier. And for the watchdog, we don't have to set the bit unless it's clean, right? So we don't add any overhead to the vcpu unless it's actually cleaning the watchdog TSR bit.

Hmm, just want to make it clear.
If we want to do the TSR updates in vcpu thread, we need to record all the TSR update requests And when guest access TSR, we update TSR according to those requests.
Doesn't this sound somewhat more complicated and harder to read than atomic way?

Thanks,
Yu

author
#9 lore raw
In reply to Scott Wood
Quoted 3 lines · Alexander Graf

Yes, but why can't we do this in the vcpu thread's context so we only ever have a single instance accessing the vcpu struct? It makes a lot of things a lot easier.

Why? We don't do it for external interrupts. It would complicate locking, not simplify it -- we'd need to defer the execution to some context which can block, and then kick the guest out of guest mode, and make sure it doesn't reenter before we get the mutex...

Quoted 2 lines

And for the watchdog, we don't have to set the bit unless it's clean, right?

Not sure what you mean by "clean". We always set the bit when the timer expires.

Likewise for the decrementer.

-Scott

#10 lore raw
In reply to Scott Wood
Quoted 9 lines · Scott Wood
Quoted 3 lines · Alexander Graf

Yes, but why can't we do this in the vcpu thread's context so we only ever have a single instance accessing the vcpu struct? It makes a lot of things a lot easier.

Why? We don't do it for external interrupts. It would complicate locking, not simplify it -- we'd need to defer the execution to some context which can block, and then kick the guest out of guest mode, and make sure it doesn't reenter before we get the mutex...

We need to kick the vcpu thread out of context anyways because we're otherwise delaying delivery of the decr interrupt. So we can just as well handle it all there then. Being lazy is good, but please not at the cost of latency.

Quoted 8 lines
Quoted 3 lines

And for the watchdog, we don't have to set the bit unless it's clean, right?

Not sure what you mean by "clean". We always set the bit when the timer expires.

Likewise for the decrementer.

What I mean is that we don't want to have a timer periodically triggering in the background for watchdog when the guest never clears the watchdog delivered bit.

Alex

author
#11 lore raw
In reply to Scott Wood
Quoted 13 lines · Alexander Graf
Quoted 9 lines · Scott Wood
Quoted 3 lines · Alexander Graf

Yes, but why can't we do this in the vcpu thread's context so we only ever have a single instance accessing the vcpu struct? It makes a lot of things a lot easier.

Why? We don't do it for external interrupts. It would complicate locking, not simplify it -- we'd need to defer the execution to some context which can block, and then kick the guest out of guest mode, and make sure it doesn't reenter before we get the mutex...

Quoted 4 lines

We need to kick the vcpu thread out of context anyways because we're otherwise delaying delivery of the decr interrupt. So we can just as well handle it all there then. Being lazy is good, but please not at the cost of latency.

That means we need a way to tell the thread to do this. Instead of using set_bit(), you want dedicated variables for pending_decr, pending_fit, pending_watchdog? And then we check them on every exit?

Besides avoiding the overhead of an atomic operation (premature optimization), what does this buy us? The ability to move the dequeue-after-mask from kvmppc_booke_irqprio_deliver() to kvmppc_set_tcr() (only to be replaced by the check for pending_foo)?

How is using set_bits() for TSR any different from the set_bit() we use in kvmppc_booke_queue_irqprio()?

Quoted 9 lines
Quoted 7 lines
Quoted 2 lines

And for the watchdog, we don't have to set the bit unless it's clean, right?

Not sure what you mean by "clean". We always set the bit when the timer expires.

Likewise for the decrementer.

What I mean is that we don't want to have a timer periodically triggering in the background for watchdog when the guest never clears the watchdog delivered bit.

I suppose we could stop the timer if the watchdog has hit final expiration with no action configured, and restart it if the bits get cleared. It's not really a case that warrants optimization (in the default case where the guest doesn't use the watchdog, the period will be so high that the timer will never expire), but we do want to avoid a situation where the guest sets a period that is too short and ends up timer-storming the host.

-Scott

#12 lore raw
In reply to Scott Wood
Quoted 14 lines · Scott Wood
Quoted 13 lines · Alexander Graf
Quoted 9 lines · Scott Wood
Quoted 3 lines · Alexander Graf

Yes, but why can't we do this in the vcpu thread's context so we only ever have a single instance accessing the vcpu struct? It makes a lot of things a lot easier.

Why? We don't do it for external interrupts. It would complicate locking, not simplify it -- we'd need to defer the execution to some context which can block, and then kick the guest out of guest mode, and make sure it doesn't reenter before we get the mutex...

Quoted 8 lines
Quoted 4 lines

We need to kick the vcpu thread out of context anyways because we're otherwise delaying delivery of the decr interrupt. So we can just as well handle it all there then. Being lazy is good, but please not at the cost of latency.

That means we need a way to tell the thread to do this. Instead of using set_bit(), you want dedicated variables for pending_decr, pending_fit, pending_watchdog? And then we check them on every exit?

Well, whatever we do we have to kvm_vcpu_kick() the receiving vcpu out of context. So we can just as well tell it to inject its own interrupt and don't have to deal with atomic operations in most cases then, no?

Quoted 8 lines

Besides avoiding the overhead of an atomic operation (premature optimization), what does this buy us? The ability to move the dequeue-after-mask from kvmppc_booke_irqprio_deliver() to kvmppc_set_tcr() (only to be replaced by the check for pending_foo)?

How is using set_bits() for TSR any different from the set_bit() we use in kvmppc_booke_queue_irqprio()?

It keeps guest visible registers vcpu private. I want the vcpu thread to own all vcpu registers - and TSR is a vcpu register.

Quoted 11 lines
Quoted 10 lines
Quoted 8 lines
Quoted 3 lines

And for the watchdog, we don't have to set the bit unless it's clean, right?

Not sure what you mean by "clean". We always set the bit when the timer expires.

Likewise for the decrementer.

What I mean is that we don't want to have a timer periodically triggering in the background for watchdog when the guest never clears the watchdog delivered bit.

Quoted 7 lines

I suppose we could stop the timer if the watchdog has hit final expiration with no action configured, and restart it if the bits get cleared. It's not really a case that warrants optimization (in the default case where the guest doesn't use the watchdog, the period will be so high that the timer will never expire), but we do want to avoid a situation where the guest sets a period that is too short and ends up timer-storming the host.

or where the guest simply has the watchdog only used once, then keeps the irq masked and we still happily burn CPU by triggering a timer when it's not even affecting state. Yup :)

Alex

author
#13 lore raw
In reply to Scott Wood
Quoted 22 lines · Alexander Graf
Quoted 19 lines · Scott Wood
Quoted 17 lines · Alexander Graf
Quoted 9 lines · Scott Wood
Quoted 3 lines · Alexander Graf

Yes, but why can't we do this in the vcpu thread's context so we only ever have a single instance accessing the vcpu struct? It makes a lot of things a lot easier.

Why? We don't do it for external interrupts. It would complicate locking, not simplify it -- we'd need to defer the execution to some context which can block, and then kick the guest out of guest mode, and make sure it doesn't reenter before we get the mutex...

We need to kick the vcpu thread out of context anyways because we're otherwise delaying delivery of the decr interrupt. So we can just as well handle it all there then. Being lazy is good, but please not at the cost of latency.

Quoted 6 lines
Quoted 3 lines

That means we need a way to tell the thread to do this. Instead of using set_bit(), you want dedicated variables for pending_decr, pending_fit, pending_watchdog? And then we check them on every exit?

Well, whatever we do we have to kvm_vcpu_kick() the receiving vcpu out of context.

Yes, the point is how the information about the event is communicated.

Quoted 2 lines

So we can just as well tell it to inject its own interrupt and don't have to deal with atomic operations in most cases then, no?

Either we use atomics, or we split each event into its own variable. We already use atomics for injecting other interrupts.

It seems simpler to just use the architected format rather than invent something new. And if we do something non-architected, that makes it more likely that injection of the interrupt triggered by qemu setting the bit with sregs is broken (untested special case).

Quoted 9 lines
Quoted 7 lines

Besides avoiding the overhead of an atomic operation (premature optimization), what does this buy us? The ability to move the dequeue-after-mask from kvmppc_booke_irqprio_deliver() to kvmppc_set_tcr() (only to be replaced by the check for pending_foo)?

How is using set_bits() for TSR any different from the set_bit() we use in kvmppc_booke_queue_irqprio()?

It keeps guest visible registers vcpu private. I want the vcpu thread to own all vcpu registers - and TSR is a vcpu register.

It is private to the vcpu implementation. I don't get why it should be private to the thread just because it's an architected register -- would it be any different if we called it "not_tsr" (or "pending_timers") whose field "not_dis" (or a bit called BOOKE_TIMER_DECR) happened to be at the right location, and we happen to be able to very easily turn it into tsr when the guest wants to read it? :-)

-Scott

#14 lore raw
In reply to Scott Wood
Quoted 51 lines · Scott Wood
Quoted 22 lines · Alexander Graf
Quoted 19 lines · Scott Wood
Quoted 17 lines · Alexander Graf
Quoted 9 lines · Scott Wood
Quoted 3 lines · Alexander Graf

Yes, but why can't we do this in the vcpu thread's context so we only ever have a single instance accessing the vcpu struct? It makes a lot of things a lot easier.

Why? We don't do it for external interrupts. It would complicate locking, not simplify it -- we'd need to defer the execution to some context which can block, and then kick the guest out of guest mode, and make sure it doesn't reenter before we get the mutex...

We need to kick the vcpu thread out of context anyways because we're otherwise delaying delivery of the decr interrupt. So we can just as well handle it all there then. Being lazy is good, but please not at the cost of latency.

Quoted 6 lines
Quoted 3 lines

That means we need a way to tell the thread to do this. Instead of using set_bit(), you want dedicated variables for pending_decr, pending_fit, pending_watchdog? And then we check them on every exit?

Well, whatever we do we have to kvm_vcpu_kick() the receiving vcpu out of context.

Yes, the point is how the information about the event is communicated.

Quoted 2 lines

So we can just as well tell it to inject its own interrupt and don't have to deal with atomic operations in most cases then, no?

Either we use atomics, or we split each event into its own variable. We already use atomics for injecting other interrupts.

It seems simpler to just use the architected format rather than invent something new. And if we do something non-architected, that makes it more likely that injection of the interrupt triggered by qemu setting the bit with sregs is broken (untested special case).

Quoted 8 lines
Quoted 7 lines

Besides avoiding the overhead of an atomic operation (premature optimization), what does this buy us? The ability to move the dequeue-after-mask from kvmppc_booke_irqprio_deliver() to kvmppc_set_tcr() (only to be replaced by the check for pending_foo)?

How is using set_bits() for TSR any different from the set_bit() we use in kvmppc_booke_queue_irqprio()?

Quoted 8 lines
Quoted 1 line

It keeps guest visible registers vcpu private. I want the vcpu thread to own all vcpu registers - and TSR is a vcpu register.

It is private to the vcpu implementation. I don't get why it should be private to the thread just because it's an architected register -- would it be any different if we called it "not_tsr" (or "pending_timers") whose field "not_dis" (or a bit called BOOKE_TIMER_DECR) happened to be at the right location, and we happen to be able to very easily turn it into tsr when the guest wants to read it? :-)

I think I'm getting your point. So what we want is:

in timer handler:

  set_bit(TSR_DIS, vcpu->arch.tsr);
  kvm_make_request(KVM_REQ_PPC_TSR_UPDATE, vcpu);
  kvm_vcpu_kick(vcpu);

in vcpu entry code:

  if (vcpu->requests)
     if (kvm_check_request(KVM_REQ_PPC_TSR_UPDATE, vcpu))
       kvmppc_update_tsr(vcpu);

  void kvmppc_update_tsr(struct kvm_vcpu *vcpu)
  {
    if (vcpu->arch.tsr & TSR_DIS &&
        vcpu->arch.tcr & TCR_DIE) {
      kvmppc_core_queue_dec(vcpu);
    }
    // XXX also implement dequeue!
  }

in emulate code:

  case SPR_TSR:
    vcpu->arch.tsr &= ~TSR_DIS;
    kvmppc_update_tsr(vcpu);

Right?

Alex

author
#15 lore raw
In reply to Scott Wood
Quoted 10 lines · Alexander Graf

I think I'm getting your point. So what we want is:

in timer handler:

  set_bit(TSR_DIS, vcpu->arch.tsr);
  kvm_make_request(KVM_REQ_PPC_TSR_UPDATE, vcpu);
  kvm_vcpu_kick(vcpu);

in vcpu entry code:

Quoted 12 lines
  if (vcpu->requests)
     if (kvm_check_request(KVM_REQ_PPC_TSR_UPDATE, vcpu))
       kvmppc_update_tsr(vcpu);

  void kvmppc_update_tsr(struct kvm_vcpu *vcpu)
  {
    if (vcpu->arch.tsr & TSR_DIS &&
        vcpu->arch.tcr & TCR_DIE) {
      kvmppc_core_queue_dec(vcpu);
    }
    // XXX also implement dequeue!
  }

OK, this avoids the possibility for the guest to clear DIS/DIE after enqueue but before delivery (which in the current patch is dealt with by an extra check at delivery), and we only have to check one thing in the common nothing-requested case (even if more things use this mechanism, as long as we don't run out of bits).

If we convert external IRQs to using this as well, and expand the timer case to include non-booke (maybe just use KVM_REQ_PENDING_TIMER?), we should be able to de-atomicize pending_exceptions (which should be a bigger win than de-atomicizing arch.tsr).

Quoted 3 lines
  case SPR_TSR:
    vcpu->arch.tsr &= ~TSR_DIS;
    kvmppc_update_tsr(vcpu);

Well, we should probably replace TSR_DIS here with the actual value that the guest wrote. :-)

-Scott

#16 lore raw
In reply to Scott Wood
Quoted 35 lines · Scott Wood
Quoted 14 lines · Alexander Graf

I think I'm getting your point. So what we want is:

in timer handler:

set_bit(TSR_DIS, vcpu->arch.tsr);
kvm_make_request(KVM_REQ_PPC_TSR_UPDATE, vcpu);
kvm_vcpu_kick(vcpu);

in vcpu entry code:

if (vcpu->requests)
if (kvm_check_request(KVM_REQ_PPC_TSR_UPDATE, vcpu))
kvmppc_update_tsr(vcpu);

Quoted 8 lines

void kvmppc_update_tsr(struct kvm_vcpu *vcpu)

 {
   if (vcpu->arch.tsr & TSR_DIS &&
       vcpu->arch.tcr & TCR_DIE) {
     kvmppc_core_queue_dec(vcpu);
   }
   // XXX also implement dequeue!

}

OK, this avoids the possibility for the guest to clear DIS/DIE after enqueue but before delivery (which in the current patch is dealt with by an extra check at delivery), and we only have to check one thing in the common nothing-requested case (even if more things use this mechanism, as long as we don't run out of bits).

If we convert external IRQs to using this as well, and expand the timer case to include non-booke (maybe just use KVM_REQ_PENDING_TIMER?), we should be able to de-atomicize pending_exceptions (which should be a bigger win than de-atomicizing arch.tsr).

Quoted 6 lines
Quoted 3 lines

case SPR_TSR:

   vcpu->arch.tsr &= ~TSR_DIS;
   kvmppc_update_tsr(vcpu);

Well, we should probably replace TSR_DIS here with the actual value that the guest wrote. :-)

Yes, and use the atomic operation. This is all just pseudo-code.

I like the idea of moving pending_exceptions off of atomic. This really is a private field that no other vcpu should have access to. We only have to think hard about how to implement SET_INTERRUPT for another vcpu then. But I think this could just be another kvm_make_request() type.

Alex

#17 lore raw
In reply to Scott Wood
Quoted 12 lines

-----Original Message-----
From: [email protected] [mailto:kvm-ppc- [email protected]] On Behalf Of Alexander Graf
Sent: Tuesday, September 27, 2011 1:45 PM
To: Wood Scott-B07421
Cc: Liu Yu-B13201; Wood Scott-B07421; [email protected]
Subject: Re: [PATCH 5/5] KVM: PPC: booke: Improve timer register emulation

Quoted 10 lines
Quoted 8 lines · Scott Wood
Quoted 7 lines · Alexander Graf

I think I'm getting your point. So what we want is:

in timer handler:

set_bit(TSR_DIS, vcpu->arch.tsr);
kvm_make_request(KVM_REQ_PPC_TSR_UPDATE, vcpu);
kvm_vcpu_kick(vcpu);

Still there can be a case where first request not handled and another event happens? Or we would like to pause till first request is handled by vcpu?

Thanks
-Bharat

Quoted 48 lines
Quoted 21 lines
Quoted 7 lines

in vcpu entry code:

if (vcpu->requests)
if (kvm_check_request(KVM_REQ_PPC_TSR_UPDATE, vcpu))
kvmppc_update_tsr(vcpu);

Quoted 7 lines
 void kvmppc_update_tsr(struct kvm_vcpu *vcpu)  {
   if (vcpu->arch.tsr & TSR_DIS &&
       vcpu->arch.tcr & TCR_DIE) {
     kvmppc_core_queue_dec(vcpu);
   }
   // XXX also implement dequeue!

}

OK, this avoids the possibility for the guest to clear DIS/DIE after enqueue but before delivery (which in the current patch is dealt with by an extra check at delivery), and we only have to check one thing in the common nothing-requested case (even if more things use this mechanism, as long as we don't run out of bits).

Quoted 4 lines

If we convert external IRQs to using this as well, and expand the timer case to include non-booke (maybe just use KVM_REQ_PENDING_TIMER?), we should be able to de-atomicize pending_exceptions (which should be a bigger win than de-atomicizing

arch.tsr).

Quoted 7 lines
Quoted 4 lines

case SPR_TSR:

   vcpu->arch.tsr &= ~TSR_DIS;
   kvmppc_update_tsr(vcpu);

Well, we should probably replace TSR_DIS here with the actual value that the guest wrote. :-)

Yes, and use the atomic operation. This is all just pseudo-code.

I like the idea of moving pending_exceptions off of atomic. This really is a private field that no other vcpu should have access to. We only have to think hard about how to implement SET_INTERRUPT for another vcpu then. But I think this could just be another kvm_make_request() type.

Alex

--
To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html

#18 lore raw
In reply to Scott Wood
Quoted 17 lines · On 27.09.2011, at 18:01, Bhushan Bharat-R65777 wrote:
Quoted 17 lines

-----Original Message-----
From: [email protected] [mailto:kvm-ppc- [email protected]] On Behalf Of Alexander Graf
Sent: Tuesday, September 27, 2011 1:45 PM
To: Wood Scott-B07421
Cc: Liu Yu-B13201; Wood Scott-B07421; [email protected]
Subject: Re: [PATCH 5/5] KVM: PPC: booke: Improve timer register emulation

Quoted 3 lines · Scott Wood
Quoted 2 lines · Alexander Graf

I think I'm getting your point. So what we want is:

Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

in timer handler:

set_bit(TSR_DIS, vcpu->arch.tsr);
kvm_make_request(KVM_REQ_PPC_TSR_UPDATE, vcpu);
kvm_vcpu_kick(vcpu);

Still there can be a case where first request not handled and another event happens? Or we would like to pause till first request is handled by vcpu?

If the first DIS request is not handled and another event happens, the interrupts get coalesced (like on real hardware). If there is another TSR bit set still, int_pending should still be active and the guest exits the next time it sets MSR_EE, making us inject the next interrupt.

Alex

#19 lore raw
In reply to Scott Wood
Quoted 37 lines

-----Original Message-----
From: [email protected] [mailto:kvm-ppc- [email protected]] On Behalf Of Alexander Graf
Sent: Tuesday, September 27, 2011 9:39 PM
To: Bhushan Bharat-R65777
Cc: Wood Scott-B07421; Liu Yu-B13201; [email protected]
Subject: Re: [PATCH 5/5] KVM: PPC: booke: Improve timer register emulation

Quoted 23 lines · On 27.09.2011, at 18:01, Bhushan Bharat-R65777 wrote:
Quoted 22 lines

-----Original Message-----
From: [email protected] [mailto:kvm-ppc- [email protected]] On Behalf Of Alexander Graf
Sent: Tuesday, September 27, 2011 1:45 PM
To: Wood Scott-B07421
Cc: Liu Yu-B13201; Wood Scott-B07421; [email protected]
Subject: Re: [PATCH 5/5] KVM: PPC: booke: Improve timer register emulation

Quoted 8 lines · Scott Wood
Quoted 7 lines · Alexander Graf

I think I'm getting your point. So what we want is:

in timer handler:

set_bit(TSR_DIS, vcpu->arch.tsr);
kvm_make_request(KVM_REQ_PPC_TSR_UPDATE, vcpu);
kvm_vcpu_kick(vcpu);

Quoted 9 lines
Quoted 1 line

Still there can be a case where first request not handled and another

event happens? Or we would like to pause till first request is handled by vcpu?

If the first DIS request is not handled and another event happens, the interrupts get coalesced (like on real hardware). If there is another TSR bit set still, int_pending should still be active and the guest exits the next time it sets MSR_EE, making us inject the next interrupt.

This may not work for watchdog, where every event causes a state change.

Thanks
-Bharat

#20 lore raw
In reply to Scott Wood
Quoted 41 lines · On 27.09.2011, at 18:21, Bhushan Bharat-R65777 wrote:
Quoted 41 lines

-----Original Message-----
From: [email protected] [mailto:kvm-ppc- [email protected]] On Behalf Of Alexander Graf
Sent: Tuesday, September 27, 2011 9:39 PM
To: Bhushan Bharat-R65777
Cc: Wood Scott-B07421; Liu Yu-B13201; [email protected]
Subject: Re: [PATCH 5/5] KVM: PPC: booke: Improve timer register emulation

Quoted 17 lines · On 27.09.2011, at 18:01, Bhushan Bharat-R65777 wrote:
Quoted 17 lines

-----Original Message-----
From: [email protected] [mailto:kvm-ppc- [email protected]] On Behalf Of Alexander Graf
Sent: Tuesday, September 27, 2011 1:45 PM
To: Wood Scott-B07421
Cc: Liu Yu-B13201; Wood Scott-B07421; [email protected]
Subject: Re: [PATCH 5/5] KVM: PPC: booke: Improve timer register emulation

Quoted 3 lines · Scott Wood
Quoted 2 lines · Alexander Graf

I think I'm getting your point. So what we want is:

Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

in timer handler:

set_bit(TSR_DIS, vcpu->arch.tsr);
kvm_make_request(KVM_REQ_PPC_TSR_UPDATE, vcpu);
kvm_vcpu_kick(vcpu);

Still there can be a case where first request not handled and another

event happens? Or we would like to pause till first request is handled by vcpu?

Quoted 7 lines
Quoted 5 lines

If the first DIS request is not handled and another event happens, the interrupts get coalesced (like on real hardware). If there is another TSR bit set still, int_pending should still be active and the guest exits the next time it sets MSR_EE, making us inject the next interrupt.

This may not work for watchdog, where every event causes a state change.

Not sure I understand. What state does it change except for the TSR bit?

Alex

author
#21 lore raw
In reply to Scott Wood
Quoted 41 lines · On 09/27/2011 11:21 AM, Bhushan Bharat-R65777 wrote:
Quoted 41 lines

-----Original Message-----
From: [email protected] [mailto:kvm-ppc- [email protected]] On Behalf Of Alexander Graf
Sent: Tuesday, September 27, 2011 9:39 PM
To: Bhushan Bharat-R65777
Cc: Wood Scott-B07421; Liu Yu-B13201; [email protected]
Subject: Re: [PATCH 5/5] KVM: PPC: booke: Improve timer register emulation

Quoted 17 lines · On 27.09.2011, at 18:01, Bhushan Bharat-R65777 wrote:
Quoted 17 lines

-----Original Message-----
From: [email protected] [mailto:kvm-ppc- [email protected]] On Behalf Of Alexander Graf
Sent: Tuesday, September 27, 2011 1:45 PM
To: Wood Scott-B07421
Cc: Liu Yu-B13201; Wood Scott-B07421; [email protected]
Subject: Re: [PATCH 5/5] KVM: PPC: booke: Improve timer register emulation

Quoted 3 lines · Scott Wood
Quoted 2 lines · Alexander Graf

I think I'm getting your point. So what we want is:

Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

in timer handler:

set_bit(TSR_DIS, vcpu->arch.tsr);
kvm_make_request(KVM_REQ_PPC_TSR_UPDATE, vcpu);
kvm_vcpu_kick(vcpu);

Still there can be a case where first request not handled and another

event happens? Or we would like to pause till first request is handled by vcpu?

Quoted 7 lines
Quoted 5 lines

If the first DIS request is not handled and another event happens, the interrupts get coalesced (like on real hardware). If there is another TSR bit set still, int_pending should still be active and the guest exits the next time it sets MSR_EE, making us inject the next interrupt.

This may not work for watchdog, where every event causes a state change.

For watchdog, this would only be used for delivering the interrupt (or a final expiration exit to qemu, which would be a separate request bit). ENW/WIS would be handled from timer context, like DIS is here.

The watchdog timer could use a compare and swap to avoid this sort of race:

vcpu thread watchdog timer
----------- --------------
*expires*
set ENW

*expires*
see ENW set
clear ENW+WIS
set WIS

sees WIS but not ENW

...if we care. It's something that shouldn't be possible in hardware, but not of much consequence (the guest is unlikely to clear ENW+WIS one time but only ENW the next). Not hard to avoid either, though.

Other races are possible, but (unless I'm missing one) they don't produce results that couldn't have been produced by the vcpu thread being a few cycles slower.

-Scott