Skip to contentSkip to conversation

KVM: arm/arm64: fix emulated ptimer irq injection

PATCH In review Andrew Jones 3 participants 13 messages May 27, 2019 – Jun 13, 2019 KVM arm64

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 KVM: arm/arm64: fix emulated ptimer irq injection

The emulated ptimer needs to track the level changes, otherwise the the interrupt will never get deasserted, resulting in the guest getting stuck in an interrupt storm if it enables ptimer interrupts. This was found with kvm-unit-tests; the ptimer tests hung as soon as interrupts were enabled. Typical Linux guests don't have a problem as they prefer using the virtual timer.

Fixes bee038a674875 ("KVM: arm/arm64: Rework the timer code to use a timer_map")
---
 virt/kvm/arm/arch_timer.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
virt/kvm/arm/arch_timer.c +6 -1
index 7fc272ecae16..9f5d8cc8b5e5 100644
@@ -324,10 +324,15 @@ static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level,
 static void timer_emulate(struct arch_timer_context *ctx)
 {
 	bool should_fire = kvm_timer_should_fire(ctx);
+	struct timer_map map;
+
+	get_timer_map(ctx->vcpu, &map);
 
 	trace_kvm_timer_emulate(ctx, should_fire);
 
-	if (should_fire) {
+	if (ctx == map.emul_ptimer && should_fire != ctx->irq.level) {
+		kvm_timer_update_irq(ctx->vcpu, !ctx->irq.level, ctx);
+	} else if (should_fire) {
 		kvm_timer_update_irq(ctx->vcpu, true, ctx);
 		return;
 	}
Signature · 6 lines
2.18.1

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#1 lore raw
Quoted 24 lines · Andrew Jones

The emulated ptimer needs to track the level changes, otherwise the the interrupt will never get deasserted, resulting in the guest getting stuck in an interrupt storm if it enables ptimer interrupts. This was found with kvm-unit-tests; the ptimer tests hung as soon as interrupts were enabled. Typical Linux guests don't have a problem as they prefer using the virtual timer.

Fixes bee038a674875 ("KVM: arm/arm64: Rework the timer code to use a timer_map")
---
 virt/kvm/arm/arch_timer.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
virt/kvm/arm/arch_timer.c +3 -0
index 7fc272ecae16..9f5d8cc8b5e5 100644
@@ -324,10 +324,15 @@ static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level,
 static void timer_emulate(struct arch_timer_context *ctx)
 {
 	bool should_fire = kvm_timer_should_fire(ctx);
+	struct timer_map map;
+
+	get_timer_map(ctx->vcpu, &map);

virt/kvm/arm/arch_timer.c static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level, · line 338

Quoted 10 lines
virt/kvm/arm/arch_timer.c +3 -1
 
 	trace_kvm_timer_emulate(ctx, should_fire);
 
-	if (should_fire) {
+	if (ctx == map.emul_ptimer && should_fire != ctx->irq.level) {
+		kvm_timer_update_irq(ctx->vcpu, !ctx->irq.level, ctx);
+	} else if (should_fire) {
 		kvm_timer_update_irq(ctx->vcpu, true, ctx);
 		return;
 	}

Hmm, this doesn't feel completely right.

Lowering the line of an emulated timer should only ever happen when the guest (or user space) writes to one of the system registers for that timer, which should be trapped and that should cause an update of the line.

Are we missing a call to kvm_timer_update_irq() from kvm_arm_timer_set_reg() ?

Thanks,

Christoffer
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

#2 lore raw
Quoted 38 lines · Christoffer Dall
Quoted 24 lines · Andrew Jones

The emulated ptimer needs to track the level changes, otherwise the the interrupt will never get deasserted, resulting in the guest getting stuck in an interrupt storm if it enables ptimer interrupts. This was found with kvm-unit-tests; the ptimer tests hung as soon as interrupts were enabled. Typical Linux guests don't have a problem as they prefer using the virtual timer.

Fixes bee038a674875 ("KVM: arm/arm64: Rework the timer code to use a timer_map")
---
 virt/kvm/arm/arch_timer.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
virt/kvm/arm/arch_timer.c +3 -0
index 7fc272ecae16..9f5d8cc8b5e5 100644
@@ -324,10 +324,15 @@ static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level,
 static void timer_emulate(struct arch_timer_context *ctx)
 {
 	bool should_fire = kvm_timer_should_fire(ctx);
+	struct timer_map map;
+
+	get_timer_map(ctx->vcpu, &map);

virt/kvm/arm/arch_timer.c static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level, · line 338

Quoted 10 lines
virt/kvm/arm/arch_timer.c +3 -1
 
 	trace_kvm_timer_emulate(ctx, should_fire);
 
-	if (should_fire) {
+	if (ctx == map.emul_ptimer && should_fire != ctx->irq.level) {
+		kvm_timer_update_irq(ctx->vcpu, !ctx->irq.level, ctx);
+	} else if (should_fire) {
 		kvm_timer_update_irq(ctx->vcpu, true, ctx);
 		return;
 	}

Hmm, this doesn't feel completely right.

Quoted 7 lines

Lowering the line of an emulated timer should only ever happen when the guest (or user space) writes to one of the system registers for that timer, which should be trapped and that should cause an update of the line.

Are we missing a call to kvm_timer_update_irq() from kvm_arm_timer_set_reg() ?

Which is exactly what we removed in 6bc210003dff, for good reasons.

Looking at kvm_arm_timer_write_sysreg(), we end-up calling kvm_timer_vcpu_load, but not updating the irq status.

How about something like this instead (untested):

virt/kvm/arm/arch_timer.c +5 -1
index 7fc272ecae16..6a418dcc5433 100644
@@ -882,10 +882,14 @@ void kvm_arm_timer_write_sysreg(struct kvm_vcpu *vcpu,
 				enum kvm_arch_timer_regs treg,
 				u64 val)
 {
+	struct arch_timer_context *timer;
+
 	preempt_disable();
 	kvm_timer_vcpu_put(vcpu);
 
-	kvm_arm_timer_write(vcpu, vcpu_get_timer(vcpu, tmr), treg, val);
+	timer = vcpu_get_timer(vcpu, tmr);
+	kvm_arm_timer_write(vcpu, timer, treg, val);
+	kvm_timer_update_irq(vcpu, kvm_timer_should_fire(timer), timer);
 
 	kvm_timer_vcpu_load(vcpu);
 	preempt_enable();

Thanks,

	M.
Signature · 5 lines
Jazz is not dead. It just smells funny...
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#3 lore raw
Quoted 44 lines · Marc Zyngier
Quoted 43 lines · Christoffer Dall
Quoted 24 lines · Andrew Jones

The emulated ptimer needs to track the level changes, otherwise the the interrupt will never get deasserted, resulting in the guest getting stuck in an interrupt storm if it enables ptimer interrupts. This was found with kvm-unit-tests; the ptimer tests hung as soon as interrupts were enabled. Typical Linux guests don't have a problem as they prefer using the virtual timer.

Fixes bee038a674875 ("KVM: arm/arm64: Rework the timer code to use a timer_map")
---
 virt/kvm/arm/arch_timer.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
virt/kvm/arm/arch_timer.c +3 -0
index 7fc272ecae16..9f5d8cc8b5e5 100644
@@ -324,10 +324,15 @@ static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level,
 static void timer_emulate(struct arch_timer_context *ctx)
 {
 	bool should_fire = kvm_timer_should_fire(ctx);
+	struct timer_map map;
+
+	get_timer_map(ctx->vcpu, &map);

virt/kvm/arm/arch_timer.c static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level, · line 338

Quoted 10 lines
virt/kvm/arm/arch_timer.c +3 -1
 
 	trace_kvm_timer_emulate(ctx, should_fire);
 
-	if (should_fire) {
+	if (ctx == map.emul_ptimer && should_fire != ctx->irq.level) {
+		kvm_timer_update_irq(ctx->vcpu, !ctx->irq.level, ctx);
+	} else if (should_fire) {
 		kvm_timer_update_irq(ctx->vcpu, true, ctx);
 		return;
 	}

Hmm, this doesn't feel completely right.

Lowering the line of an emulated timer should only ever happen when the guest (or user space) writes to one of the system registers for that timer, which should be trapped and that should cause an update of the line.

Quoted 5 lines
Quoted 2 lines

Are we missing a call to kvm_timer_update_irq() from kvm_arm_timer_set_reg() ?

Which is exactly what we removed in 6bc210003dff, for good reasons.

Ah well, I can be wrong twice. Or even three times.

Quoted 15 lines

Looking at kvm_arm_timer_write_sysreg(), we end-up calling kvm_timer_vcpu_load, but not updating the irq status.

How about something like this instead (untested):

virt/kvm/arm/arch_timer.c +2 -0
index 7fc272ecae16..6a418dcc5433 100644
@@ -882,10 +882,14 @@ void kvm_arm_timer_write_sysreg(struct kvm_vcpu *vcpu,
 				enum kvm_arch_timer_regs treg,
 				u64 val)
 {
+	struct arch_timer_context *timer;
+

virt/kvm/arm/arch_timer.c void kvm_arm_timer_write_sysreg(struct kvm_vcpu *vcpu, · line 895

Quoted 10 lines
virt/kvm/arm/arch_timer.c +3 -1
 	preempt_disable();
 	kvm_timer_vcpu_put(vcpu);
 
-	kvm_arm_timer_write(vcpu, vcpu_get_timer(vcpu, tmr), treg, val);
+	timer = vcpu_get_timer(vcpu, tmr);
+	kvm_arm_timer_write(vcpu, timer, treg, val);
+	kvm_timer_update_irq(vcpu, kvm_timer_should_fire(timer), timer);
 
 	kvm_timer_vcpu_load(vcpu);
 	preempt_enable();

Yes, that looks reasonable. Basically, in 6bc210003dff we should have only removed the call to timer_emulate, and not messed around with kvm_timer_update_irq()?

After this patch, we'll have moved the call to kvm_timer_update_irq() from kvm_arm_timer_set_reg() to kvm_arm_timer_write_sysreg(). I can't seem to decide if clearly belongs in one place or the other.

Thanks,

Christoffer
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

#4 lore raw
Quoted 30 lines · Christoffer Dall
Quoted 29 lines · Marc Zyngier
Quoted 28 lines · Christoffer Dall
Quoted 27 lines · Andrew Jones

The emulated ptimer needs to track the level changes, otherwise the the interrupt will never get deasserted, resulting in the guest getting stuck in an interrupt storm if it enables ptimer interrupts. This was found with kvm-unit-tests; the ptimer tests hung as soon as interrupts were enabled. Typical Linux guests don't have a problem as they prefer using the virtual timer.

Fixes bee038a674875 ("KVM: arm/arm64: Rework the timer code to use a timer_map")
---
 virt/kvm/arm/arch_timer.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
virt/kvm/arm/arch_timer.c +3 -0
index 7fc272ecae16..9f5d8cc8b5e5 100644
@@ -324,10 +324,15 @@ static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level,
 static void timer_emulate(struct arch_timer_context *ctx)
 {
 	bool should_fire = kvm_timer_should_fire(ctx);
+	struct timer_map map;
+
+	get_timer_map(ctx->vcpu, &map);
 
 	trace_kvm_timer_emulate(ctx, should_fire);
 
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

virt/kvm/arm/arch_timer.c static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level, · line 338

Quoted 7 lines
virt/kvm/arm/arch_timer.c +3 -1
-	if (should_fire) {
+	if (ctx == map.emul_ptimer && should_fire != ctx->irq.level) {
+		kvm_timer_update_irq(ctx->vcpu, !ctx->irq.level, ctx);
+	} else if (should_fire) {
 		kvm_timer_update_irq(ctx->vcpu, true, ctx);
 		return;
 	}

Hmm, this doesn't feel completely right.

I won't try to argue that this is the right fix, as I haven't fully grasped how all this code works, but, afaict, this is how it worked prior to bee038a6.

Quoted 31 lines
Quoted 11 lines
Quoted 8 lines

Lowering the line of an emulated timer should only ever happen when the guest (or user space) writes to one of the system registers for that timer, which should be trapped and that should cause an update of the line.

Are we missing a call to kvm_timer_update_irq() from kvm_arm_timer_set_reg() ?

Which is exactly what we removed in 6bc210003dff, for good reasons.

Ah well, I can be wrong twice. Or even three times.

Quoted 17 lines

Looking at kvm_arm_timer_write_sysreg(), we end-up calling kvm_timer_vcpu_load, but not updating the irq status.

How about something like this instead (untested):

virt/kvm/arm/arch_timer.c +2 -0
index 7fc272ecae16..6a418dcc5433 100644
@@ -882,10 +882,14 @@ void kvm_arm_timer_write_sysreg(struct kvm_vcpu *vcpu,
 				enum kvm_arch_timer_regs treg,
 				u64 val)
 {
+	struct arch_timer_context *timer;
+
 	preempt_disable();
 	kvm_timer_vcpu_put(vcpu);
 
Quoted 8 lines

virt/kvm/arm/arch_timer.c void kvm_arm_timer_write_sysreg(struct kvm_vcpu *vcpu, · line 895

Quoted 8 lines
virt/kvm/arm/arch_timer.c +3 -1
-	kvm_arm_timer_write(vcpu, vcpu_get_timer(vcpu, tmr), treg, val);
+	timer = vcpu_get_timer(vcpu, tmr);
+	kvm_arm_timer_write(vcpu, timer, treg, val);
+	kvm_timer_update_irq(vcpu, kvm_timer_should_fire(timer), timer);
 
 	kvm_timer_vcpu_load(vcpu);
 	preempt_enable();

Marc, I've tested this and it resolves the issue for me. If/when you post it you can add a t-b from me if you like.

Quoted 9 lines

Yes, that looks reasonable. Basically, in 6bc210003dff we should have only removed the call to timer_emulate, and not messed around with kvm_timer_update_irq()?

After this patch, we'll have moved the call to kvm_timer_update_irq() from kvm_arm_timer_set_reg() to kvm_arm_timer_write_sysreg(). I can't seem to decide if clearly belongs in one place or the other.

Isn't kvm_arm_timer_set_reg() only for userspace setting of the register? In this test case I don't think userspace is involved at that point.

Thanks,
drew
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

#5 lore raw
Quoted 93 lines · Andrew Jones
Quoted 30 lines · Christoffer Dall
Quoted 29 lines · Marc Zyngier
Quoted 28 lines · Christoffer Dall
Quoted 27 lines · Andrew Jones

The emulated ptimer needs to track the level changes, otherwise the the interrupt will never get deasserted, resulting in the guest getting stuck in an interrupt storm if it enables ptimer interrupts. This was found with kvm-unit-tests; the ptimer tests hung as soon as interrupts were enabled. Typical Linux guests don't have a problem as they prefer using the virtual timer.

Fixes bee038a674875 ("KVM: arm/arm64: Rework the timer code to use a timer_map")
---
 virt/kvm/arm/arch_timer.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
virt/kvm/arm/arch_timer.c +3 -0
index 7fc272ecae16..9f5d8cc8b5e5 100644
@@ -324,10 +324,15 @@ static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level,
 static void timer_emulate(struct arch_timer_context *ctx)
 {
 	bool should_fire = kvm_timer_should_fire(ctx);
+	struct timer_map map;
+
+	get_timer_map(ctx->vcpu, &map);
 
 	trace_kvm_timer_emulate(ctx, should_fire);
 
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

virt/kvm/arm/arch_timer.c static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level, · line 338

Quoted 7 lines
virt/kvm/arm/arch_timer.c +3 -1
-	if (should_fire) {
+	if (ctx == map.emul_ptimer && should_fire != ctx->irq.level) {
+		kvm_timer_update_irq(ctx->vcpu, !ctx->irq.level, ctx);
+	} else if (should_fire) {
 		kvm_timer_update_irq(ctx->vcpu, true, ctx);
 		return;
 	}

Hmm, this doesn't feel completely right.

I won't try to argue that this is the right fix, as I haven't fully grasped how all this code works, but, afaict, this is how it worked prior to bee038a6.

Quoted 31 lines
Quoted 11 lines
Quoted 8 lines

Lowering the line of an emulated timer should only ever happen when the guest (or user space) writes to one of the system registers for that timer, which should be trapped and that should cause an update of the line.

Are we missing a call to kvm_timer_update_irq() from kvm_arm_timer_set_reg() ?

Which is exactly what we removed in 6bc210003dff, for good reasons.

Ah well, I can be wrong twice. Or even three times.

Quoted 17 lines

Looking at kvm_arm_timer_write_sysreg(), we end-up calling kvm_timer_vcpu_load, but not updating the irq status.

How about something like this instead (untested):

virt/kvm/arm/arch_timer.c +2 -0
index 7fc272ecae16..6a418dcc5433 100644
@@ -882,10 +882,14 @@ void kvm_arm_timer_write_sysreg(struct kvm_vcpu *vcpu,
 				enum kvm_arch_timer_regs treg,
 				u64 val)
 {
+	struct arch_timer_context *timer;
+
 	preempt_disable();
 	kvm_timer_vcpu_put(vcpu);
 
Quoted 8 lines

virt/kvm/arm/arch_timer.c void kvm_arm_timer_write_sysreg(struct kvm_vcpu *vcpu, · line 895

Quoted 8 lines
virt/kvm/arm/arch_timer.c +3 -1
-	kvm_arm_timer_write(vcpu, vcpu_get_timer(vcpu, tmr), treg, val);
+	timer = vcpu_get_timer(vcpu, tmr);
+	kvm_arm_timer_write(vcpu, timer, treg, val);
+	kvm_timer_update_irq(vcpu, kvm_timer_should_fire(timer), timer);
 
 	kvm_timer_vcpu_load(vcpu);
 	preempt_enable();

Marc, I've tested this and it resolves the issue for me. If/when you post it you can add a t-b from me if you like.

Quoted 5 lines

Yes, that looks reasonable. Basically, in 6bc210003dff we should have only removed the call to timer_emulate, and not messed around with kvm_timer_update_irq()?

Quoted 7 lines
Quoted 4 lines

After this patch, we'll have moved the call to kvm_timer_update_irq() from kvm_arm_timer_set_reg() to kvm_arm_timer_write_sysreg(). I can't seem to decide if clearly belongs in one place or the other.

Isn't kvm_arm_timer_set_reg() only for userspace setting of the register? In this test case I don't think userspace is involved at that point.

It still remains that userspace writing to any of the registers has an effect on the interrupt line. Or rather that it should.

And the more I look at this, the more I have the feeling this thing should happen on kvm_timer_vcpu_load(), wherever the writes comes from. It'd have slightly more overhead than doing it from every register access path, but at least it'd be clearer... Untested, again.

virt/kvm/arm/arch_timer.c +5 -1
index 7fc272ecae16..8244e40af196 100644
@@ -557,8 +557,12 @@ void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu)
 	if (map.direct_ptimer)
 		timer_restore_state(map.direct_ptimer);
 
-	if (map.emul_ptimer)
+	if (map.emul_ptimer) {
+		kvm_timer_update_irq(vcpu,
+				     kvm_timer_should_fire(map.emul_ptimer),
+				     map.emul_ptimer);
 		timer_emulate(map.emul_ptimer);
+	}
 }
 
 bool kvm_timer_should_notify_user(struct kvm_vcpu *vcpu)

Thanks,

	M.
Signature · 5 lines
Jazz is not dead. It just smells funny...
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#6 lore raw
Quoted 119 lines · Marc Zyngier
Quoted 93 lines · Andrew Jones
Quoted 30 lines · Christoffer Dall
Quoted 29 lines · Marc Zyngier
Quoted 28 lines · Christoffer Dall
Quoted 27 lines · Andrew Jones

The emulated ptimer needs to track the level changes, otherwise the the interrupt will never get deasserted, resulting in the guest getting stuck in an interrupt storm if it enables ptimer interrupts. This was found with kvm-unit-tests; the ptimer tests hung as soon as interrupts were enabled. Typical Linux guests don't have a problem as they prefer using the virtual timer.

Fixes bee038a674875 ("KVM: arm/arm64: Rework the timer code to use a timer_map")
---
 virt/kvm/arm/arch_timer.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
virt/kvm/arm/arch_timer.c +3 -0
index 7fc272ecae16..9f5d8cc8b5e5 100644
@@ -324,10 +324,15 @@ static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level,
 static void timer_emulate(struct arch_timer_context *ctx)
 {
 	bool should_fire = kvm_timer_should_fire(ctx);
+	struct timer_map map;
+
+	get_timer_map(ctx->vcpu, &map);
 
 	trace_kvm_timer_emulate(ctx, should_fire);
 
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

virt/kvm/arm/arch_timer.c static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level, · line 338

Quoted 7 lines
virt/kvm/arm/arch_timer.c +3 -1
-	if (should_fire) {
+	if (ctx == map.emul_ptimer && should_fire != ctx->irq.level) {
+		kvm_timer_update_irq(ctx->vcpu, !ctx->irq.level, ctx);
+	} else if (should_fire) {
 		kvm_timer_update_irq(ctx->vcpu, true, ctx);
 		return;
 	}

Hmm, this doesn't feel completely right.

I won't try to argue that this is the right fix, as I haven't fully grasped how all this code works, but, afaict, this is how it worked prior to bee038a6.

Quoted 31 lines
Quoted 11 lines
Quoted 8 lines

Lowering the line of an emulated timer should only ever happen when the guest (or user space) writes to one of the system registers for that timer, which should be trapped and that should cause an update of the line.

Are we missing a call to kvm_timer_update_irq() from kvm_arm_timer_set_reg() ?

Which is exactly what we removed in 6bc210003dff, for good reasons.

Ah well, I can be wrong twice. Or even three times.

Quoted 17 lines

Looking at kvm_arm_timer_write_sysreg(), we end-up calling kvm_timer_vcpu_load, but not updating the irq status.

How about something like this instead (untested):

virt/kvm/arm/arch_timer.c +2 -0
index 7fc272ecae16..6a418dcc5433 100644
@@ -882,10 +882,14 @@ void kvm_arm_timer_write_sysreg(struct kvm_vcpu *vcpu,
 				enum kvm_arch_timer_regs treg,
 				u64 val)
 {
+	struct arch_timer_context *timer;
+
 	preempt_disable();
 	kvm_timer_vcpu_put(vcpu);
 
Quoted 8 lines

virt/kvm/arm/arch_timer.c void kvm_arm_timer_write_sysreg(struct kvm_vcpu *vcpu, · line 895

Quoted 8 lines
virt/kvm/arm/arch_timer.c +3 -1
-	kvm_arm_timer_write(vcpu, vcpu_get_timer(vcpu, tmr), treg, val);
+	timer = vcpu_get_timer(vcpu, tmr);
+	kvm_arm_timer_write(vcpu, timer, treg, val);
+	kvm_timer_update_irq(vcpu, kvm_timer_should_fire(timer), timer);
 
 	kvm_timer_vcpu_load(vcpu);
 	preempt_enable();

Marc, I've tested this and it resolves the issue for me. If/when you post it you can add a t-b from me if you like.

Quoted 5 lines

Yes, that looks reasonable. Basically, in 6bc210003dff we should have only removed the call to timer_emulate, and not messed around with kvm_timer_update_irq()?

Quoted 7 lines
Quoted 4 lines

After this patch, we'll have moved the call to kvm_timer_update_irq() from kvm_arm_timer_set_reg() to kvm_arm_timer_write_sysreg(). I can't seem to decide if clearly belongs in one place or the other.

Isn't kvm_arm_timer_set_reg() only for userspace setting of the register? In this test case I don't think userspace is involved at that point.

It still remains that userspace writing to any of the registers has an effect on the interrupt line. Or rather that it should.

And the more I look at this, the more I have the feeling this thing should happen on kvm_timer_vcpu_load(), wherever the writes comes from. It'd have slightly more overhead than doing it from every register access path, but at least it'd be clearer... Untested, again.

virt/kvm/arm/arch_timer.c +0 -0
index 7fc272ecae16..8244e40af196 100644
@@ -557,8 +557,12 @@ void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu)
 	if (map.direct_ptimer)
 		timer_restore_state(map.direct_ptimer);
 

virt/kvm/arm/arch_timer.c void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu) · line 568

Quoted 10 lines
virt/kvm/arm/arch_timer.c +5 -1
-	if (map.emul_ptimer)
+	if (map.emul_ptimer) {
+		kvm_timer_update_irq(vcpu,
+				     kvm_timer_should_fire(map.emul_ptimer),
+				     map.emul_ptimer);
 		timer_emulate(map.emul_ptimer);
+	}
 }
 
 bool kvm_timer_should_notify_user(struct kvm_vcpu *vcpu)

This also resolves the issue with kvm-unit-tests.

Thanks,
drew
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

#7 lore raw
In reply to Marc Zyngier
Quoted 119 lines · Marc Zyngier
Quoted 93 lines · Andrew Jones
Quoted 30 lines · Christoffer Dall
Quoted 29 lines · Marc Zyngier
Quoted 28 lines · Christoffer Dall
Quoted 27 lines · Andrew Jones

The emulated ptimer needs to track the level changes, otherwise the the interrupt will never get deasserted, resulting in the guest getting stuck in an interrupt storm if it enables ptimer interrupts. This was found with kvm-unit-tests; the ptimer tests hung as soon as interrupts were enabled. Typical Linux guests don't have a problem as they prefer using the virtual timer.

Fixes bee038a674875 ("KVM: arm/arm64: Rework the timer code to use a timer_map")
---
 virt/kvm/arm/arch_timer.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
virt/kvm/arm/arch_timer.c +3 -0
index 7fc272ecae16..9f5d8cc8b5e5 100644
@@ -324,10 +324,15 @@ static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level,
 static void timer_emulate(struct arch_timer_context *ctx)
 {
 	bool should_fire = kvm_timer_should_fire(ctx);
+	struct timer_map map;
+
+	get_timer_map(ctx->vcpu, &map);
 
 	trace_kvm_timer_emulate(ctx, should_fire);
 
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

virt/kvm/arm/arch_timer.c static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level, · line 338

Quoted 7 lines
virt/kvm/arm/arch_timer.c +3 -1
-	if (should_fire) {
+	if (ctx == map.emul_ptimer && should_fire != ctx->irq.level) {
+		kvm_timer_update_irq(ctx->vcpu, !ctx->irq.level, ctx);
+	} else if (should_fire) {
 		kvm_timer_update_irq(ctx->vcpu, true, ctx);
 		return;
 	}

Hmm, this doesn't feel completely right.

I won't try to argue that this is the right fix, as I haven't fully grasped how all this code works, but, afaict, this is how it worked prior to bee038a6.

Quoted 31 lines
Quoted 11 lines
Quoted 8 lines

Lowering the line of an emulated timer should only ever happen when the guest (or user space) writes to one of the system registers for that timer, which should be trapped and that should cause an update of the line.

Are we missing a call to kvm_timer_update_irq() from kvm_arm_timer_set_reg() ?

Which is exactly what we removed in 6bc210003dff, for good reasons.

Ah well, I can be wrong twice. Or even three times.

Quoted 17 lines

Looking at kvm_arm_timer_write_sysreg(), we end-up calling kvm_timer_vcpu_load, but not updating the irq status.

How about something like this instead (untested):

virt/kvm/arm/arch_timer.c +2 -0
index 7fc272ecae16..6a418dcc5433 100644
@@ -882,10 +882,14 @@ void kvm_arm_timer_write_sysreg(struct kvm_vcpu *vcpu,
 				enum kvm_arch_timer_regs treg,
 				u64 val)
 {
+	struct arch_timer_context *timer;
+
 	preempt_disable();
 	kvm_timer_vcpu_put(vcpu);
 
Quoted 8 lines

virt/kvm/arm/arch_timer.c void kvm_arm_timer_write_sysreg(struct kvm_vcpu *vcpu, · line 895

Quoted 8 lines
virt/kvm/arm/arch_timer.c +3 -1
-	kvm_arm_timer_write(vcpu, vcpu_get_timer(vcpu, tmr), treg, val);
+	timer = vcpu_get_timer(vcpu, tmr);
+	kvm_arm_timer_write(vcpu, timer, treg, val);
+	kvm_timer_update_irq(vcpu, kvm_timer_should_fire(timer), timer);
 
 	kvm_timer_vcpu_load(vcpu);
 	preempt_enable();

Marc, I've tested this and it resolves the issue for me. If/when you post it you can add a t-b from me if you like.

Quoted 5 lines

Yes, that looks reasonable. Basically, in 6bc210003dff we should have only removed the call to timer_emulate, and not messed around with kvm_timer_update_irq()?

Quoted 7 lines
Quoted 4 lines

After this patch, we'll have moved the call to kvm_timer_update_irq() from kvm_arm_timer_set_reg() to kvm_arm_timer_write_sysreg(). I can't seem to decide if clearly belongs in one place or the other.

Isn't kvm_arm_timer_set_reg() only for userspace setting of the register? In this test case I don't think userspace is involved at that point.

It still remains that userspace writing to any of the registers has an effect on the interrupt line. Or rather that it should.

And the more I look at this, the more I have the feeling this thing should happen on kvm_timer_vcpu_load(), wherever the writes comes from. It'd have slightly more overhead than doing it from every register access path, but at least it'd be clearer... Untested, again.

virt/kvm/arm/arch_timer.c +0 -0
index 7fc272ecae16..8244e40af196 100644
@@ -557,8 +557,12 @@ void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu)
 	if (map.direct_ptimer)
 		timer_restore_state(map.direct_ptimer);
 

virt/kvm/arm/arch_timer.c void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu) · line 568

Quoted 10 lines
virt/kvm/arm/arch_timer.c +5 -1
-	if (map.emul_ptimer)
+	if (map.emul_ptimer) {
+		kvm_timer_update_irq(vcpu,
+				     kvm_timer_should_fire(map.emul_ptimer),
+				     map.emul_ptimer);
 		timer_emulate(map.emul_ptimer);
+	}
 }
 
 bool kvm_timer_should_notify_user(struct kvm_vcpu *vcpu)

But do we do the put/load dance when we trap a write to a register from the VM ?

Thanks,

Christoffer
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

#8 lore raw
Quoted 128 lines · Christoffer Dall
Quoted 127 lines · Marc Zyngier
Quoted 93 lines · Andrew Jones
Quoted 30 lines · Christoffer Dall
Quoted 29 lines · Marc Zyngier
Quoted 28 lines · Christoffer Dall
Quoted 27 lines · Andrew Jones

The emulated ptimer needs to track the level changes, otherwise the the interrupt will never get deasserted, resulting in the guest getting stuck in an interrupt storm if it enables ptimer interrupts. This was found with kvm-unit-tests; the ptimer tests hung as soon as interrupts were enabled. Typical Linux guests don't have a problem as they prefer using the virtual timer.

Fixes bee038a674875 ("KVM: arm/arm64: Rework the timer code to use a timer_map")
---
 virt/kvm/arm/arch_timer.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
virt/kvm/arm/arch_timer.c +3 -0
index 7fc272ecae16..9f5d8cc8b5e5 100644
@@ -324,10 +324,15 @@ static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level,
 static void timer_emulate(struct arch_timer_context *ctx)
 {
 	bool should_fire = kvm_timer_should_fire(ctx);
+	struct timer_map map;
+
+	get_timer_map(ctx->vcpu, &map);
 
 	trace_kvm_timer_emulate(ctx, should_fire);
 
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

virt/kvm/arm/arch_timer.c static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level, · line 338

Quoted 7 lines
virt/kvm/arm/arch_timer.c +3 -1
-	if (should_fire) {
+	if (ctx == map.emul_ptimer && should_fire != ctx->irq.level) {
+		kvm_timer_update_irq(ctx->vcpu, !ctx->irq.level, ctx);
+	} else if (should_fire) {
 		kvm_timer_update_irq(ctx->vcpu, true, ctx);
 		return;
 	}

Hmm, this doesn't feel completely right.

I won't try to argue that this is the right fix, as I haven't fully grasped how all this code works, but, afaict, this is how it worked prior to bee038a6.

Quoted 31 lines
Quoted 11 lines
Quoted 8 lines

Lowering the line of an emulated timer should only ever happen when the guest (or user space) writes to one of the system registers for that timer, which should be trapped and that should cause an update of the line.

Are we missing a call to kvm_timer_update_irq() from kvm_arm_timer_set_reg() ?

Which is exactly what we removed in 6bc210003dff, for good reasons.

Ah well, I can be wrong twice. Or even three times.

Quoted 17 lines

Looking at kvm_arm_timer_write_sysreg(), we end-up calling kvm_timer_vcpu_load, but not updating the irq status.

How about something like this instead (untested):

virt/kvm/arm/arch_timer.c +2 -0
index 7fc272ecae16..6a418dcc5433 100644
@@ -882,10 +882,14 @@ void kvm_arm_timer_write_sysreg(struct kvm_vcpu *vcpu,
 				enum kvm_arch_timer_regs treg,
 				u64 val)
 {
+	struct arch_timer_context *timer;
+
 	preempt_disable();
 	kvm_timer_vcpu_put(vcpu);
 
Quoted 8 lines

virt/kvm/arm/arch_timer.c void kvm_arm_timer_write_sysreg(struct kvm_vcpu *vcpu, · line 895

Quoted 8 lines
virt/kvm/arm/arch_timer.c +3 -1
-	kvm_arm_timer_write(vcpu, vcpu_get_timer(vcpu, tmr), treg, val);
+	timer = vcpu_get_timer(vcpu, tmr);
+	kvm_arm_timer_write(vcpu, timer, treg, val);
+	kvm_timer_update_irq(vcpu, kvm_timer_should_fire(timer), timer);
 
 	kvm_timer_vcpu_load(vcpu);
 	preempt_enable();

Marc, I've tested this and it resolves the issue for me. If/when you post it you can add a t-b from me if you like.

Quoted 5 lines

Yes, that looks reasonable. Basically, in 6bc210003dff we should have only removed the call to timer_emulate, and not messed around with kvm_timer_update_irq()?

Quoted 7 lines
Quoted 4 lines

After this patch, we'll have moved the call to kvm_timer_update_irq() from kvm_arm_timer_set_reg() to kvm_arm_timer_write_sysreg(). I can't seem to decide if clearly belongs in one place or the other.

Isn't kvm_arm_timer_set_reg() only for userspace setting of the register? In this test case I don't think userspace is involved at that point.

It still remains that userspace writing to any of the registers has an effect on the interrupt line. Or rather that it should.

And the more I look at this, the more I have the feeling this thing should happen on kvm_timer_vcpu_load(), wherever the writes comes from. It'd have slightly more overhead than doing it from every register access path, but at least it'd be clearer... Untested, again.

virt/kvm/arm/arch_timer.c +5 -1
index 7fc272ecae16..8244e40af196 100644
@@ -557,8 +557,12 @@ void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu)
 	if (map.direct_ptimer)
 		timer_restore_state(map.direct_ptimer);
 
-	if (map.emul_ptimer)
+	if (map.emul_ptimer) {
+		kvm_timer_update_irq(vcpu,
+				     kvm_timer_should_fire(map.emul_ptimer),
+				     map.emul_ptimer);
 		timer_emulate(map.emul_ptimer);
+	}
 }
 
Quoted 5 lines

virt/kvm/arm/arch_timer.c void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu) · line 568

Quoted 2 lines
virt/kvm/arm/arch_timer.c +0 -0
 bool kvm_timer_should_notify_user(struct kvm_vcpu *vcpu)

But do we do the put/load dance when we trap a write to a register from the VM ?

Yup, that's what kvm_arm_timer_write_sysreg() does:

	preempt_disable();
	kvm_timer_vcpu_put(vcpu);

	kvm_arm_timer_write(vcpu, vcpu_get_timer(vcpu, tmr), treg, val);

	kvm_timer_vcpu_load(vcpu);
	preempt_enable();

Thanks,

	M.
Signature · 5 lines
Jazz is not dead. It just smells funny...
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#9 lore raw
Quoted 137 lines · Marc Zyngier
Quoted 128 lines · Christoffer Dall
Quoted 127 lines · Marc Zyngier
Quoted 93 lines · Andrew Jones
Quoted 30 lines · Christoffer Dall
Quoted 29 lines · Marc Zyngier
Quoted 28 lines · Christoffer Dall
Quoted 27 lines · Andrew Jones

The emulated ptimer needs to track the level changes, otherwise the the interrupt will never get deasserted, resulting in the guest getting stuck in an interrupt storm if it enables ptimer interrupts. This was found with kvm-unit-tests; the ptimer tests hung as soon as interrupts were enabled. Typical Linux guests don't have a problem as they prefer using the virtual timer.

Fixes bee038a674875 ("KVM: arm/arm64: Rework the timer code to use a timer_map")
---
 virt/kvm/arm/arch_timer.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
virt/kvm/arm/arch_timer.c +3 -0
index 7fc272ecae16..9f5d8cc8b5e5 100644
@@ -324,10 +324,15 @@ static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level,
 static void timer_emulate(struct arch_timer_context *ctx)
 {
 	bool should_fire = kvm_timer_should_fire(ctx);
+	struct timer_map map;
+
+	get_timer_map(ctx->vcpu, &map);
 
 	trace_kvm_timer_emulate(ctx, should_fire);
 
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

virt/kvm/arm/arch_timer.c static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level, · line 338

Quoted 7 lines
virt/kvm/arm/arch_timer.c +3 -1
-	if (should_fire) {
+	if (ctx == map.emul_ptimer && should_fire != ctx->irq.level) {
+		kvm_timer_update_irq(ctx->vcpu, !ctx->irq.level, ctx);
+	} else if (should_fire) {
 		kvm_timer_update_irq(ctx->vcpu, true, ctx);
 		return;
 	}

Hmm, this doesn't feel completely right.

I won't try to argue that this is the right fix, as I haven't fully grasped how all this code works, but, afaict, this is how it worked prior to bee038a6.

Quoted 31 lines
Quoted 11 lines
Quoted 8 lines

Lowering the line of an emulated timer should only ever happen when the guest (or user space) writes to one of the system registers for that timer, which should be trapped and that should cause an update of the line.

Are we missing a call to kvm_timer_update_irq() from kvm_arm_timer_set_reg() ?

Which is exactly what we removed in 6bc210003dff, for good reasons.

Ah well, I can be wrong twice. Or even three times.

Quoted 17 lines

Looking at kvm_arm_timer_write_sysreg(), we end-up calling kvm_timer_vcpu_load, but not updating the irq status.

How about something like this instead (untested):

virt/kvm/arm/arch_timer.c +2 -0
index 7fc272ecae16..6a418dcc5433 100644
@@ -882,10 +882,14 @@ void kvm_arm_timer_write_sysreg(struct kvm_vcpu *vcpu,
 				enum kvm_arch_timer_regs treg,
 				u64 val)
 {
+	struct arch_timer_context *timer;
+
 	preempt_disable();
 	kvm_timer_vcpu_put(vcpu);
 
Quoted 8 lines

virt/kvm/arm/arch_timer.c void kvm_arm_timer_write_sysreg(struct kvm_vcpu *vcpu, · line 895

Quoted 8 lines
virt/kvm/arm/arch_timer.c +3 -1
-	kvm_arm_timer_write(vcpu, vcpu_get_timer(vcpu, tmr), treg, val);
+	timer = vcpu_get_timer(vcpu, tmr);
+	kvm_arm_timer_write(vcpu, timer, treg, val);
+	kvm_timer_update_irq(vcpu, kvm_timer_should_fire(timer), timer);
 
 	kvm_timer_vcpu_load(vcpu);
 	preempt_enable();

Marc, I've tested this and it resolves the issue for me. If/when you post it you can add a t-b from me if you like.

Quoted 5 lines

Yes, that looks reasonable. Basically, in 6bc210003dff we should have only removed the call to timer_emulate, and not messed around with kvm_timer_update_irq()?

Quoted 7 lines
Quoted 4 lines

After this patch, we'll have moved the call to kvm_timer_update_irq() from kvm_arm_timer_set_reg() to kvm_arm_timer_write_sysreg(). I can't seem to decide if clearly belongs in one place or the other.

Isn't kvm_arm_timer_set_reg() only for userspace setting of the register? In this test case I don't think userspace is involved at that point.

It still remains that userspace writing to any of the registers has an effect on the interrupt line. Or rather that it should.

And the more I look at this, the more I have the feeling this thing should happen on kvm_timer_vcpu_load(), wherever the writes comes from. It'd have slightly more overhead than doing it from every register access path, but at least it'd be clearer... Untested, again.

virt/kvm/arm/arch_timer.c +5 -1
index 7fc272ecae16..8244e40af196 100644
@@ -557,8 +557,12 @@ void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu)
 	if (map.direct_ptimer)
 		timer_restore_state(map.direct_ptimer);
 
-	if (map.emul_ptimer)
+	if (map.emul_ptimer) {
+		kvm_timer_update_irq(vcpu,
+				     kvm_timer_should_fire(map.emul_ptimer),
+				     map.emul_ptimer);
 		timer_emulate(map.emul_ptimer);
+	}
 }
 
Quoted 5 lines

virt/kvm/arm/arch_timer.c void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu) · line 568

Quoted 2 lines
virt/kvm/arm/arch_timer.c +0 -0
 bool kvm_timer_should_notify_user(struct kvm_vcpu *vcpu)

But do we do the put/load dance when we trap a write to a register from the VM ?

Yup, that's what kvm_arm_timer_write_sysreg() does:

Quoted 8 lines
	preempt_disable();
	kvm_timer_vcpu_put(vcpu);

	kvm_arm_timer_write(vcpu, vcpu_get_timer(vcpu, tmr), treg, val);

	kvm_timer_vcpu_load(vcpu);
	preempt_enable();

Ah, I missed that. In that case, fair enough. The only question then is if we should unconditionally do this in timer_emulate (almost Drew's original patch) or do it here in vcpu_load ?

I don't remember how the nesting code looks like, but when it will start to use emul_vtimer, we now need to do this for both, which would be an argument for doing it in timer_emulate, I believe.

Also, a nice comment in there why this is necessary (i.e. for handling proper emulation when trapping sysreg changes) would probably be worthwhile.

Thanks,

Christoffer
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

#10 lore raw
Quoted 151 lines · Christoffer Dall
Quoted 137 lines · Marc Zyngier
Quoted 128 lines · Christoffer Dall
Quoted 127 lines · Marc Zyngier
Quoted 93 lines · Andrew Jones
Quoted 30 lines · Christoffer Dall
Quoted 29 lines · Marc Zyngier
Quoted 28 lines · Christoffer Dall
Quoted 27 lines · Andrew Jones

The emulated ptimer needs to track the level changes, otherwise the the interrupt will never get deasserted, resulting in the guest getting stuck in an interrupt storm if it enables ptimer interrupts. This was found with kvm-unit-tests; the ptimer tests hung as soon as interrupts were enabled. Typical Linux guests don't have a problem as they prefer using the virtual timer.

Fixes bee038a674875 ("KVM: arm/arm64: Rework the timer code to use a timer_map")
---
 virt/kvm/arm/arch_timer.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
virt/kvm/arm/arch_timer.c +3 -0
index 7fc272ecae16..9f5d8cc8b5e5 100644
@@ -324,10 +324,15 @@ static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level,
 static void timer_emulate(struct arch_timer_context *ctx)
 {
 	bool should_fire = kvm_timer_should_fire(ctx);
+	struct timer_map map;
+
+	get_timer_map(ctx->vcpu, &map);
 
 	trace_kvm_timer_emulate(ctx, should_fire);
 
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

virt/kvm/arm/arch_timer.c static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level, · line 338

Quoted 7 lines
virt/kvm/arm/arch_timer.c +3 -1
-	if (should_fire) {
+	if (ctx == map.emul_ptimer && should_fire != ctx->irq.level) {
+		kvm_timer_update_irq(ctx->vcpu, !ctx->irq.level, ctx);
+	} else if (should_fire) {
 		kvm_timer_update_irq(ctx->vcpu, true, ctx);
 		return;
 	}

Hmm, this doesn't feel completely right.

I won't try to argue that this is the right fix, as I haven't fully grasped how all this code works, but, afaict, this is how it worked prior to bee038a6.

Quoted 31 lines
Quoted 11 lines
Quoted 8 lines

Lowering the line of an emulated timer should only ever happen when the guest (or user space) writes to one of the system registers for that timer, which should be trapped and that should cause an update of the line.

Are we missing a call to kvm_timer_update_irq() from kvm_arm_timer_set_reg() ?

Which is exactly what we removed in 6bc210003dff, for good reasons.

Ah well, I can be wrong twice. Or even three times.

Quoted 17 lines

Looking at kvm_arm_timer_write_sysreg(), we end-up calling kvm_timer_vcpu_load, but not updating the irq status.

How about something like this instead (untested):

virt/kvm/arm/arch_timer.c +2 -0
index 7fc272ecae16..6a418dcc5433 100644
@@ -882,10 +882,14 @@ void kvm_arm_timer_write_sysreg(struct kvm_vcpu *vcpu,
 				enum kvm_arch_timer_regs treg,
 				u64 val)
 {
+	struct arch_timer_context *timer;
+
 	preempt_disable();
 	kvm_timer_vcpu_put(vcpu);
 
Quoted 8 lines

virt/kvm/arm/arch_timer.c void kvm_arm_timer_write_sysreg(struct kvm_vcpu *vcpu, · line 895

Quoted 8 lines
virt/kvm/arm/arch_timer.c +3 -1
-	kvm_arm_timer_write(vcpu, vcpu_get_timer(vcpu, tmr), treg, val);
+	timer = vcpu_get_timer(vcpu, tmr);
+	kvm_arm_timer_write(vcpu, timer, treg, val);
+	kvm_timer_update_irq(vcpu, kvm_timer_should_fire(timer), timer);
 
 	kvm_timer_vcpu_load(vcpu);
 	preempt_enable();

Marc, I've tested this and it resolves the issue for me. If/when you post it you can add a t-b from me if you like.

Quoted 5 lines

Yes, that looks reasonable. Basically, in 6bc210003dff we should have only removed the call to timer_emulate, and not messed around with kvm_timer_update_irq()?

Quoted 7 lines
Quoted 4 lines

After this patch, we'll have moved the call to kvm_timer_update_irq() from kvm_arm_timer_set_reg() to kvm_arm_timer_write_sysreg(). I can't seem to decide if clearly belongs in one place or the other.

Isn't kvm_arm_timer_set_reg() only for userspace setting of the register? In this test case I don't think userspace is involved at that point.

It still remains that userspace writing to any of the registers has an effect on the interrupt line. Or rather that it should.

And the more I look at this, the more I have the feeling this thing should happen on kvm_timer_vcpu_load(), wherever the writes comes from. It'd have slightly more overhead than doing it from every register access path, but at least it'd be clearer... Untested, again.

virt/kvm/arm/arch_timer.c +5 -1
index 7fc272ecae16..8244e40af196 100644
@@ -557,8 +557,12 @@ void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu)
 	if (map.direct_ptimer)
 		timer_restore_state(map.direct_ptimer);
 
-	if (map.emul_ptimer)
+	if (map.emul_ptimer) {
+		kvm_timer_update_irq(vcpu,
+				     kvm_timer_should_fire(map.emul_ptimer),
+				     map.emul_ptimer);
 		timer_emulate(map.emul_ptimer);
+	}
 }
 
Quoted 5 lines

virt/kvm/arm/arch_timer.c void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu) · line 568

Quoted 2 lines
virt/kvm/arm/arch_timer.c +0 -0
 bool kvm_timer_should_notify_user(struct kvm_vcpu *vcpu)

But do we do the put/load dance when we trap a write to a register from the VM ?

Yup, that's what kvm_arm_timer_write_sysreg() does:

Quoted 8 lines
	preempt_disable();
	kvm_timer_vcpu_put(vcpu);

	kvm_arm_timer_write(vcpu, vcpu_get_timer(vcpu, tmr), treg, val);

	kvm_timer_vcpu_load(vcpu);
	preempt_enable();

Ah, I missed that. In that case, fair enough. The only question then is if we should unconditionally do this in timer_emulate (almost Drew's original patch) or do it here in vcpu_load ?

Quoted 8 lines

I don't remember how the nesting code looks like, but when it will start to use emul_vtimer, we now need to do this for both, which would be an argument for doing it in timer_emulate, I believe.

Also, a nice comment in there why this is necessary (i.e. for handling proper emulation when trapping sysreg changes) would probably be worthwhile.

Any more thoughts on how to proceed with this? FWIW, I found a commit[*] that indicates kvm_timer_vcpu_load() was at least once the correct place.

[*] 245715cbe83c ("KVM: arm/arm64: Fix lost IRQs from emulated physcial timer when blocked", 2018-07-25)

Thanks,
drew
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

#11 lore raw
Quoted 162 lines · Andrew Jones
Quoted 151 lines · Christoffer Dall
Quoted 137 lines · Marc Zyngier
Quoted 128 lines · Christoffer Dall
Quoted 127 lines · Marc Zyngier
Quoted 93 lines · Andrew Jones
Quoted 30 lines · Christoffer Dall
Quoted 29 lines · Marc Zyngier
Quoted 28 lines · Christoffer Dall
Quoted 27 lines · Andrew Jones

The emulated ptimer needs to track the level changes, otherwise the the interrupt will never get deasserted, resulting in the guest getting stuck in an interrupt storm if it enables ptimer interrupts. This was found with kvm-unit-tests; the ptimer tests hung as soon as interrupts were enabled. Typical Linux guests don't have a problem as they prefer using the virtual timer.

Fixes bee038a674875 ("KVM: arm/arm64: Rework the timer code to use a timer_map")
---
 virt/kvm/arm/arch_timer.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
virt/kvm/arm/arch_timer.c +3 -0
index 7fc272ecae16..9f5d8cc8b5e5 100644
@@ -324,10 +324,15 @@ static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level,
 static void timer_emulate(struct arch_timer_context *ctx)
 {
 	bool should_fire = kvm_timer_should_fire(ctx);
+	struct timer_map map;
+
+	get_timer_map(ctx->vcpu, &map);
 
 	trace_kvm_timer_emulate(ctx, should_fire);
 
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

virt/kvm/arm/arch_timer.c static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level, · line 338

Quoted 7 lines
virt/kvm/arm/arch_timer.c +3 -1
-	if (should_fire) {
+	if (ctx == map.emul_ptimer && should_fire != ctx->irq.level) {
+		kvm_timer_update_irq(ctx->vcpu, !ctx->irq.level, ctx);
+	} else if (should_fire) {
 		kvm_timer_update_irq(ctx->vcpu, true, ctx);
 		return;
 	}

Hmm, this doesn't feel completely right.

I won't try to argue that this is the right fix, as I haven't fully grasped how all this code works, but, afaict, this is how it worked prior to bee038a6.

Quoted 31 lines
Quoted 11 lines
Quoted 8 lines

Lowering the line of an emulated timer should only ever happen when the guest (or user space) writes to one of the system registers for that timer, which should be trapped and that should cause an update of the line.

Are we missing a call to kvm_timer_update_irq() from kvm_arm_timer_set_reg() ?

Which is exactly what we removed in 6bc210003dff, for good reasons.

Ah well, I can be wrong twice. Or even three times.

Quoted 17 lines

Looking at kvm_arm_timer_write_sysreg(), we end-up calling kvm_timer_vcpu_load, but not updating the irq status.

How about something like this instead (untested):

virt/kvm/arm/arch_timer.c +2 -0
index 7fc272ecae16..6a418dcc5433 100644
@@ -882,10 +882,14 @@ void kvm_arm_timer_write_sysreg(struct kvm_vcpu *vcpu,
 				enum kvm_arch_timer_regs treg,
 				u64 val)
 {
+	struct arch_timer_context *timer;
+
 	preempt_disable();
 	kvm_timer_vcpu_put(vcpu);
 
Quoted 8 lines

virt/kvm/arm/arch_timer.c void kvm_arm_timer_write_sysreg(struct kvm_vcpu *vcpu, · line 895

Quoted 8 lines
virt/kvm/arm/arch_timer.c +3 -1
-	kvm_arm_timer_write(vcpu, vcpu_get_timer(vcpu, tmr), treg, val);
+	timer = vcpu_get_timer(vcpu, tmr);
+	kvm_arm_timer_write(vcpu, timer, treg, val);
+	kvm_timer_update_irq(vcpu, kvm_timer_should_fire(timer), timer);
 
 	kvm_timer_vcpu_load(vcpu);
 	preempt_enable();

Marc, I've tested this and it resolves the issue for me. If/when you post it you can add a t-b from me if you like.

Quoted 5 lines

Yes, that looks reasonable. Basically, in 6bc210003dff we should have only removed the call to timer_emulate, and not messed around with kvm_timer_update_irq()?

Quoted 7 lines
Quoted 4 lines

After this patch, we'll have moved the call to kvm_timer_update_irq() from kvm_arm_timer_set_reg() to kvm_arm_timer_write_sysreg(). I can't seem to decide if clearly belongs in one place or the other.

Isn't kvm_arm_timer_set_reg() only for userspace setting of the register? In this test case I don't think userspace is involved at that point.

It still remains that userspace writing to any of the registers has an effect on the interrupt line. Or rather that it should.

And the more I look at this, the more I have the feeling this thing should happen on kvm_timer_vcpu_load(), wherever the writes comes from. It'd have slightly more overhead than doing it from every register access path, but at least it'd be clearer... Untested, again.

virt/kvm/arm/arch_timer.c +5 -1
index 7fc272ecae16..8244e40af196 100644
@@ -557,8 +557,12 @@ void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu)
 	if (map.direct_ptimer)
 		timer_restore_state(map.direct_ptimer);
 
-	if (map.emul_ptimer)
+	if (map.emul_ptimer) {
+		kvm_timer_update_irq(vcpu,
+				     kvm_timer_should_fire(map.emul_ptimer),
+				     map.emul_ptimer);
 		timer_emulate(map.emul_ptimer);
+	}
 }
 
Quoted 5 lines

virt/kvm/arm/arch_timer.c void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu) · line 568

Quoted 2 lines
virt/kvm/arm/arch_timer.c +0 -0
 bool kvm_timer_should_notify_user(struct kvm_vcpu *vcpu)

But do we do the put/load dance when we trap a write to a register from the VM ?

Yup, that's what kvm_arm_timer_write_sysreg() does:

Quoted 8 lines
	preempt_disable();
	kvm_timer_vcpu_put(vcpu);

	kvm_arm_timer_write(vcpu, vcpu_get_timer(vcpu, tmr), treg, val);

	kvm_timer_vcpu_load(vcpu);
	preempt_enable();

Ah, I missed that. In that case, fair enough. The only question then is if we should unconditionally do this in timer_emulate (almost Drew's original patch) or do it here in vcpu_load ?

Quoted 8 lines

I don't remember how the nesting code looks like, but when it will start to use emul_vtimer, we now need to do this for both, which would be an argument for doing it in timer_emulate, I believe.

Also, a nice comment in there why this is necessary (i.e. for handling proper emulation when trapping sysreg changes) would probably be worthwhile.

Quoted 5 lines

Any more thoughts on how to proceed with this? FWIW, I found a commit[*] that indicates kvm_timer_vcpu_load() was at least once the correct place.

[*] 245715cbe83c ("KVM: arm/arm64: Fix lost IRQs from emulated physcial timer when blocked", 2018-07-25)

Coming back to this: I wonder if the simplest fix isn't a small variation on your initial patch:

virt/kvm/arm/arch_timer.c +3 -2
index 7fc272ecae16..1b1c449ceaf4 100644
@@ -321,14 +321,15 @@ static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level,
 	}
 }
 
+/* Only called for a fully emulated timer */
 static void timer_emulate(struct arch_timer_context *ctx)
 {
 	bool should_fire = kvm_timer_should_fire(ctx);
 
 	trace_kvm_timer_emulate(ctx, should_fire);
 
-	if (should_fire) {
-		kvm_timer_update_irq(ctx->vcpu, true, ctx);
+	if (should_fire != ctx->irq.level) {
+		kvm_timer_update_irq(ctx->vcpu, should_fire, ctx);
 		return;
 	}
 

It fixes the emulated ptimer for me (just gave KVM unit tests a go). The rational is that we only come here for the emulated ptimer already (from vcpu_load), so the whole test can be simplified.

Christoffer, am I missing anything with respect to cancelling the timer by always returning early?

Thanks,

	M.
Signature · 5 lines
Jazz is not dead, it just smells funny.
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#12 lore raw
Quoted 195 lines · Marc Zyngier
Quoted 162 lines · Andrew Jones
Quoted 151 lines · Christoffer Dall
Quoted 137 lines · Marc Zyngier
Quoted 128 lines · Christoffer Dall
Quoted 127 lines · Marc Zyngier
Quoted 93 lines · Andrew Jones
Quoted 30 lines · Christoffer Dall
Quoted 29 lines · Marc Zyngier
Quoted 28 lines · Christoffer Dall
Quoted 27 lines · Andrew Jones

The emulated ptimer needs to track the level changes, otherwise the the interrupt will never get deasserted, resulting in the guest getting stuck in an interrupt storm if it enables ptimer interrupts. This was found with kvm-unit-tests; the ptimer tests hung as soon as interrupts were enabled. Typical Linux guests don't have a problem as they prefer using the virtual timer.

Fixes bee038a674875 ("KVM: arm/arm64: Rework the timer code to use a timer_map")
---
 virt/kvm/arm/arch_timer.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
virt/kvm/arm/arch_timer.c +3 -0
index 7fc272ecae16..9f5d8cc8b5e5 100644
@@ -324,10 +324,15 @@ static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level,
 static void timer_emulate(struct arch_timer_context *ctx)
 {
 	bool should_fire = kvm_timer_should_fire(ctx);
+	struct timer_map map;
+
+	get_timer_map(ctx->vcpu, &map);
 
 	trace_kvm_timer_emulate(ctx, should_fire);
 
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

virt/kvm/arm/arch_timer.c static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level, · line 338

Quoted 7 lines
virt/kvm/arm/arch_timer.c +3 -1
-	if (should_fire) {
+	if (ctx == map.emul_ptimer && should_fire != ctx->irq.level) {
+		kvm_timer_update_irq(ctx->vcpu, !ctx->irq.level, ctx);
+	} else if (should_fire) {
 		kvm_timer_update_irq(ctx->vcpu, true, ctx);
 		return;
 	}

Hmm, this doesn't feel completely right.

I won't try to argue that this is the right fix, as I haven't fully grasped how all this code works, but, afaict, this is how it worked prior to bee038a6.

Quoted 31 lines
Quoted 11 lines
Quoted 8 lines

Lowering the line of an emulated timer should only ever happen when the guest (or user space) writes to one of the system registers for that timer, which should be trapped and that should cause an update of the line.

Are we missing a call to kvm_timer_update_irq() from kvm_arm_timer_set_reg() ?

Which is exactly what we removed in 6bc210003dff, for good reasons.

Ah well, I can be wrong twice. Or even three times.

Quoted 17 lines

Looking at kvm_arm_timer_write_sysreg(), we end-up calling kvm_timer_vcpu_load, but not updating the irq status.

How about something like this instead (untested):

virt/kvm/arm/arch_timer.c +2 -0
index 7fc272ecae16..6a418dcc5433 100644
@@ -882,10 +882,14 @@ void kvm_arm_timer_write_sysreg(struct kvm_vcpu *vcpu,
 				enum kvm_arch_timer_regs treg,
 				u64 val)
 {
+	struct arch_timer_context *timer;
+
 	preempt_disable();
 	kvm_timer_vcpu_put(vcpu);
 
Quoted 8 lines

virt/kvm/arm/arch_timer.c void kvm_arm_timer_write_sysreg(struct kvm_vcpu *vcpu, · line 895

Quoted 8 lines
virt/kvm/arm/arch_timer.c +3 -1
-	kvm_arm_timer_write(vcpu, vcpu_get_timer(vcpu, tmr), treg, val);
+	timer = vcpu_get_timer(vcpu, tmr);
+	kvm_arm_timer_write(vcpu, timer, treg, val);
+	kvm_timer_update_irq(vcpu, kvm_timer_should_fire(timer), timer);
 
 	kvm_timer_vcpu_load(vcpu);
 	preempt_enable();

Marc, I've tested this and it resolves the issue for me. If/when you post it you can add a t-b from me if you like.

Quoted 5 lines

Yes, that looks reasonable. Basically, in 6bc210003dff we should have only removed the call to timer_emulate, and not messed around with kvm_timer_update_irq()?

Quoted 7 lines
Quoted 4 lines

After this patch, we'll have moved the call to kvm_timer_update_irq() from kvm_arm_timer_set_reg() to kvm_arm_timer_write_sysreg(). I can't seem to decide if clearly belongs in one place or the other.

Isn't kvm_arm_timer_set_reg() only for userspace setting of the register? In this test case I don't think userspace is involved at that point.

It still remains that userspace writing to any of the registers has an effect on the interrupt line. Or rather that it should.

And the more I look at this, the more I have the feeling this thing should happen on kvm_timer_vcpu_load(), wherever the writes comes from. It'd have slightly more overhead than doing it from every register access path, but at least it'd be clearer... Untested, again.

virt/kvm/arm/arch_timer.c +5 -1
index 7fc272ecae16..8244e40af196 100644
@@ -557,8 +557,12 @@ void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu)
 	if (map.direct_ptimer)
 		timer_restore_state(map.direct_ptimer);
 
-	if (map.emul_ptimer)
+	if (map.emul_ptimer) {
+		kvm_timer_update_irq(vcpu,
+				     kvm_timer_should_fire(map.emul_ptimer),
+				     map.emul_ptimer);
 		timer_emulate(map.emul_ptimer);
+	}
 }
 
Quoted 5 lines

virt/kvm/arm/arch_timer.c void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu) · line 568

Quoted 2 lines
virt/kvm/arm/arch_timer.c +0 -0
 bool kvm_timer_should_notify_user(struct kvm_vcpu *vcpu)

But do we do the put/load dance when we trap a write to a register from the VM ?

Yup, that's what kvm_arm_timer_write_sysreg() does:

Quoted 8 lines
	preempt_disable();
	kvm_timer_vcpu_put(vcpu);

	kvm_arm_timer_write(vcpu, vcpu_get_timer(vcpu, tmr), treg, val);

	kvm_timer_vcpu_load(vcpu);
	preempt_enable();

Ah, I missed that. In that case, fair enough. The only question then is if we should unconditionally do this in timer_emulate (almost Drew's original patch) or do it here in vcpu_load ?

Quoted 8 lines

I don't remember how the nesting code looks like, but when it will start to use emul_vtimer, we now need to do this for both, which would be an argument for doing it in timer_emulate, I believe.

Also, a nice comment in there why this is necessary (i.e. for handling proper emulation when trapping sysreg changes) would probably be worthwhile.

Quoted 5 lines

Any more thoughts on how to proceed with this? FWIW, I found a commit[*] that indicates kvm_timer_vcpu_load() was at least once the correct place.

[*] 245715cbe83c ("KVM: arm/arm64: Fix lost IRQs from emulated physcial timer when blocked", 2018-07-25)

Coming back to this: I wonder if the simplest fix isn't a small variation on your initial patch:

virt/kvm/arm/arch_timer.c +3 -2
index 7fc272ecae16..1b1c449ceaf4 100644
@@ -321,14 +321,15 @@ static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level,
 	}
 }
 
+/* Only called for a fully emulated timer */
 static void timer_emulate(struct arch_timer_context *ctx)
 {
 	bool should_fire = kvm_timer_should_fire(ctx);
 
 	trace_kvm_timer_emulate(ctx, should_fire);
 
-	if (should_fire) {
-		kvm_timer_update_irq(ctx->vcpu, true, ctx);
+	if (should_fire != ctx->irq.level) {
+		kvm_timer_update_irq(ctx->vcpu, should_fire, ctx);
 		return;
 	}
 
Quoted 3 lines

It fixes the emulated ptimer for me (just gave KVM unit tests a go). The rational is that we only come here for the emulated ptimer already (from vcpu_load), so the whole test can be simplified.

This is what I had in mind as well.

Quoted 4 lines

Christoffer, am I missing anything with respect to cancelling the timer by always returning early?

Always returning early? Not sure I understand.

Thanks,

Christoffer
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm