Skip to contentSkip to conversation

KVM: Yield CPU when vcpu executes a WFE

PATCH Applied Marc Zyngier 4 participants 11 messages Oct 8, 2013 – Oct 16, 2013

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

v2 cover KVM: Yield CPU when vcpu executes a WFE

This is a respin of a patch I posted a long while ago, this time with numbers that I hope to be convincing enough.

The basic idea is that spinning on WFE in a guest is a waste of resource, and that we're better of running another vcpu instead. This specially shows when the system is oversubscribed. The guest vcpus can be seen spinning, waiting for a lock to be released while the lock holder is nowhere near a physical CPU.

This patch series just enables WFE trapping on both ARM and arm64, and calls kvm_vcpu_on_spin(). This is enough to boost other vcpus, and dramatically reduce the overhead.

Branch available at:
git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git kvm-arm64/wfe-trap

Changes from v1:
- Added CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT, as it seems to give slightly better results (Thanks to Raghavendra K T)
- Updated commit message with results of 8x configuration

Marc Zyngier (2):
ARM: KVM: Yield CPU when vcpu executes a WFE
arm64: KVM: Yield CPU when vcpu executes a WFE

 arch/arm/include/asm/kvm_arm.h   |  4 +++-
 arch/arm/kvm/Kconfig             |  1 +
 arch/arm/kvm/handle_exit.c       |  6 +++++-
 arch/arm64/include/asm/kvm_arm.h |  8 ++++++--
 arch/arm64/kvm/Kconfig           |  1 +
 arch/arm64/kvm/handle_exit.c     | 18 +++++++++++++-----
 6 files changed, 29 insertions(+), 9 deletions(-)
Signature · 1 line
1.8.2.3
#1 lore raw

PATCH v2 1/2 ARM: KVM: Yield CPU when vcpu executes a WFE

On an (even slightly) oversubscribed system, spinlocks are quickly becoming a bottleneck, as some vcpus are spinning, waiting for a lock to be released, while the vcpu holding the lock may not be running at all.

This creates contention, and the observed slowdown is 40x for hackbench. No, this isn't a typo.

The solution is to trap blocking WFEs and tell KVM that we're now spinning. This ensures that other vpus will get a scheduling boost, allowing the lock to be released more quickly. Also, using CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT slightly improves the performance when the VM is severely overcommited.

Quick test to estimate the performance: hackbench 1 process 1000

2xA15 host (baseline): 1.843s

2xA15 guest w/o patch: 2.083s
4xA15 guest w/o patch: 80.212s
8xA15 guest w/o patch: Could not be bothered to find out

2xA15 guest w/ patch: 2.102s
4xA15 guest w/ patch: 3.205s
8xA15 guest w/ patch: 6.887s

So we go from a 40x degradation to 1.5x in the 2x overcommit case, which is vaguely more acceptable.

---
 arch/arm/include/asm/kvm_arm.h | 4 +++-
 arch/arm/kvm/Kconfig           | 1 +
 arch/arm/kvm/handle_exit.c     | 6 +++++-
 3 files changed, 9 insertions(+), 2 deletions(-)
arch/arm/include/asm/kvm_arm.h +3 -1
index 64e9696..693d5b2 100644
@@ -67,7 +67,7 @@
  */
 #define HCR_GUEST_MASK (HCR_TSC | HCR_TSW | HCR_TWI | HCR_VM | HCR_BSU_IS | \
 			HCR_FB | HCR_TAC | HCR_AMO | HCR_IMO | HCR_FMO | \
-			HCR_SWIO | HCR_TIDCP)
+			HCR_TWE | HCR_SWIO | HCR_TIDCP)
 #define HCR_VIRT_EXCP_MASK (HCR_VA | HCR_VI | HCR_VF)
 
 /* System Control Register (SCTLR) bits */
@@ -208,6 +208,8 @@
 #define HSR_EC_DABT	(0x24)
 #define HSR_EC_DABT_HYP	(0x25)
 
+#define HSR_WFI_IS_WFE		(1U << 0)
+
 #define HSR_HVC_IMM_MASK	((1UL << 16) - 1)
 
 #define HSR_DABT_S1PTW		(1U << 7)
arch/arm/kvm/Kconfig +1 -0
index ebf5015..466bd29 100644
@@ -20,6 +20,7 @@ config KVM
 	bool "Kernel-based Virtual Machine (KVM) support"
 	select PREEMPT_NOTIFIERS
 	select ANON_INODES
+	select HAVE_KVM_CPU_RELAX_INTERCEPT
 	select KVM_MMIO
 	select KVM_ARM_HOST
 	depends on ARM_VIRT_EXT && ARM_LPAE
arch/arm/kvm/handle_exit.c +5 -1
index df4c82d..c4c496f 100644
@@ -84,7 +84,11 @@ static int handle_dabt_hyp(struct kvm_vcpu *vcpu, struct kvm_run *run)
 static int kvm_handle_wfi(struct kvm_vcpu *vcpu, struct kvm_run *run)
 {
 	trace_kvm_wfi(*vcpu_pc(vcpu));
-	kvm_vcpu_block(vcpu);
+	if (kvm_vcpu_get_hsr(vcpu) & HSR_WFI_IS_WFE)
+		kvm_vcpu_on_spin(vcpu);
+	else
+		kvm_vcpu_block(vcpu);
+
 	return 1;
 }
 
Signature · 1 line
1.8.2.3
#2 lore raw

PATCH 1/1 KVM: ARM: Update comments for kvm_handle_wfi

Update comments to reflect what is really going on and add the TWE bit to the comments in kvm_arm.h.

Also renames the function to kvm_handle_wfx like is done on arm64 for consistency and uber-correctness.

---
 arch/arm/include/asm/kvm_arm.h |    1 +
 arch/arm/kvm/handle_exit.c     |   14 ++++++++------
 2 files changed, 9 insertions(+), 6 deletions(-)
arch/arm/include/asm/kvm_arm.h +1 -0
index fe395b7..1d3153c 100644
@@ -57,6 +57,7 @@
  * TSC:		Trap SMC
  * TSW:		Trap cache operations by set/way
  * TWI:		Trap WFI
+ * TWE:		Trap WFE
  * TIDCP:	Trap L2CTLR/L2ECTLR
  * BSU_IS:	Upgrade barriers to the inner shareable domain
  * FB:		Force broadcast of all maintainance operations
arch/arm/kvm/handle_exit.c +8 -6
index c4c496f..a920790 100644
@@ -73,15 +73,17 @@ static int handle_dabt_hyp(struct kvm_vcpu *vcpu, struct kvm_run *run)
 }
 
 /**
- * kvm_handle_wfi - handle a wait-for-interrupts instruction executed by a guest
+ * kvm_handle_wfx - handle a WFI or WFE instructions trapped in guests
  * @vcpu:	the vcpu pointer
  * @run:	the kvm_run structure pointer
  *
- * Simply sets the wait_for_interrupts flag on the vcpu structure, which will
- * halt execution of world-switches and schedule other host processes until
- * there is an incoming IRQ or FIQ to the VM.
+ * WFE: Yield the CPU and come back to this vcpu when the scheduler
+ * decides to.
+ * WFI: Simply call kvm_vcpu_block(), which will halt execution of
+ * world-switches and schedule other host processes until there is an
+ * incoming IRQ or FIQ to the VM.
  */
-static int kvm_handle_wfi(struct kvm_vcpu *vcpu, struct kvm_run *run)
+static int kvm_handle_wfx(struct kvm_vcpu *vcpu, struct kvm_run *run)
 {
 	trace_kvm_wfi(*vcpu_pc(vcpu));
 	if (kvm_vcpu_get_hsr(vcpu) & HSR_WFI_IS_WFE)
@@ -93,7 +95,7 @@ static int kvm_handle_wfi(struct kvm_vcpu *vcpu, struct kvm_run *run)
 }
 
 static exit_handle_fn arm_exit_handlers[] = {
-	[HSR_EC_WFI]		= kvm_handle_wfi,
+	[HSR_EC_WFI]		= kvm_handle_wfx,
 	[HSR_EC_CP15_32]	= kvm_handle_cp15_32,
 	[HSR_EC_CP15_64]	= kvm_handle_cp15_64,
 	[HSR_EC_CP14_MR]	= kvm_handle_cp14_access,
Signature · 1 line
1.7.10.4
#3 lore raw

PATCH 1/1 KVM: ARM: Update comments for kvm_handle_wfi

Quoted 9 lines

-----Original Message-----
From: Christoffer Dall [mailto:christoffer.dall at linaro.org]
Sent: Wednesday, October 16, 2013 6:43 AM
To: Marc Zyngier
Cc: kvmarm at lists.cs.columbia.edu; linux-arm-kernel at lists.infradead.org
Subject: [PATCH] KVM: ARM: Update comments for kvm_handle_wfi

Quoted 5 lines

Update comments to reflect what is really going on and add the TWE bit to the comments in kvm_arm.h.

Also renames the function to kvm_handle_wfx like is done on arm64 for consistency and uber-correctness.

s/uber/user

-Bharat

Quoted 62 lines
---
 arch/arm/include/asm/kvm_arm.h |    1 +
 arch/arm/kvm/handle_exit.c     |   14 ++++++++------
 2 files changed, 9 insertions(+), 6 deletions(-)
arch/arm/include/asm/kvm_arm.h +1 -0
index fe395b7..1d3153c 100644
@@ -57,6 +57,7 @@
  * TSC:		Trap SMC
  * TSW:		Trap cache operations by set/way
  * TWI:		Trap WFI
+ * TWE:		Trap WFE
  * TIDCP:	Trap L2CTLR/L2ECTLR
  * BSU_IS:	Upgrade barriers to the inner shareable domain
  * FB:		Force broadcast of all maintainance operations
arch/arm/kvm/handle_exit.c index renamed from arch/arm/kvm/handle_exit.c +0 -0

c4c496f..a920790 100644

arch/arm/kvm/handle_exit.c +0 -1
@@ -73,15 +73,17 @@ static int handle_dabt_hyp(struct kvm_vcpu *vcpu, struct
 kvm_run *run)  }
 
 /**
- * kvm_handle_wfi - handle a wait-for-interrupts instruction executed by a

guest
+ * kvm_handle_wfx - handle a WFI or WFE instructions trapped in guests
* @vcpu: the vcpu pointer
* @run: the kvm_run structure pointer
*
- * Simply sets the wait_for_interrupts flag on the vcpu structure, which will
- * halt execution of world-switches and schedule other host processes until
- * there is an incoming IRQ or FIQ to the VM.
+ * WFE: Yield the CPU and come back to this vcpu when the scheduler
+ * decides to.
+ * WFI: Simply call kvm_vcpu_block(), which will halt execution of
+ * world-switches and schedule other host processes until there is an
+ * incoming IRQ or FIQ to the VM.
*/

arch/arm/kvm/handle_exit.c +1 -1
-static int kvm_handle_wfi(struct kvm_vcpu *vcpu, struct kvm_run *run)
+static int kvm_handle_wfx(struct kvm_vcpu *vcpu, struct kvm_run *run)
 {
 	trace_kvm_wfi(*vcpu_pc(vcpu));
 	if (kvm_vcpu_get_hsr(vcpu) & HSR_WFI_IS_WFE) @@ -93,7 +95,7 @@ static int

kvm_handle_wfi(struct kvm_vcpu *vcpu, struct kvm_run *run) }

static exit_handle_fn arm_exit_handlers[] = {

arch/arm/kvm/handle_exit.c +1 -2
-	[HSR_EC_WFI]		= kvm_handle_wfi,
+	[HSR_EC_WFI]		= kvm_handle_wfx,
 	[HSR_EC_CP15_32]	= kvm_handle_cp15_32,
 	[HSR_EC_CP15_64]	= kvm_handle_cp15_64,
 	[HSR_EC_CP14_MR]	= kvm_handle_cp14_access,
--

1.7.10.4

_______________________________________________
kvmarm mailing list
kvmarm at lists.cs.columbia.edu
https://lists.cs.columbia.edu/cucslists/listinfo/kvmarm

#4 lore raw

PATCH 1/1 KVM: ARM: Update comments for kvm_handle_wfi

Quoted 9 lines · On Wed, Oct 16, 2013 at 04:19:45AM +0000, Bhushan Bharat-R65777 wrote:
Quoted 9 lines

-----Original Message-----
From: Christoffer Dall [mailto:christoffer.dall at linaro.org]
Sent: Wednesday, October 16, 2013 6:43 AM
To: Marc Zyngier
Cc: kvmarm at lists.cs.columbia.edu; linux-arm-kernel at lists.infradead.org
Subject: [PATCH] KVM: ARM: Update comments for kvm_handle_wfi

Quoted 8 lines
Quoted 5 lines

Update comments to reflect what is really going on and add the TWE bit to the comments in kvm_arm.h.

Also renames the function to kvm_handle_wfx like is done on arm64 for consistency and uber-correctness.

s/uber/user

No, that was actually the German word "uber" (with the dots, but I didn't want to put them), with the meaning very very correct. My attempt at humor failed here miserably I see :)

-Christoffer

#5 lore raw

Re: [PATCH v2 1/2] ARM: KVM: Yield CPU when vcpu executes a WFE

In reply to Marc Zyngier
Quoted 22 lines · Marc Zyngier

On an (even slightly) oversubscribed system, spinlocks are quickly becoming a bottleneck, as some vcpus are spinning, waiting for a lock to be released, while the vcpu holding the lock may not be running at all.

This creates contention, and the observed slowdown is 40x for hackbench. No, this isn't a typo.

The solution is to trap blocking WFEs and tell KVM that we're now spinning. This ensures that other vpus will get a scheduling boost, allowing the lock to be released more quickly. Also, using CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT slightly improves the performance when the VM is severely overcommited.

Quick test to estimate the performance: hackbench 1 process 1000

2xA15 host (baseline): 1.843s

2xA15 guest w/o patch: 2.083s
4xA15 guest w/o patch: 80.212s
8xA15 guest w/o patch: Could not be bothered to find out

Quoted 7 lines

2xA15 guest w/ patch: 2.102s
4xA15 guest w/ patch: 3.205s
8xA15 guest w/ patch: 6.887s

So we go from a 40x degradation to 1.5x in the 2x overcommit case, which is vaguely more acceptable.

Patch looks good, I can just apply it and add the other one I just send as a reply if there are no objections.

Sorry for the long turn-around on this one.

-Christoffer

#6 lore raw

Re: [PATCH v2 1/2] ARM: KVM: Yield CPU when vcpu executes a WFE

Quoted 28 lines · Christoffer Dall
Quoted 27 lines · Marc Zyngier

On an (even slightly) oversubscribed system, spinlocks are quickly becoming a bottleneck, as some vcpus are spinning, waiting for a lock to be released, while the vcpu holding the lock may not be running at all.

This creates contention, and the observed slowdown is 40x for hackbench. No, this isn't a typo.

The solution is to trap blocking WFEs and tell KVM that we're now spinning. This ensures that other vpus will get a scheduling boost, allowing the lock to be released more quickly. Also, using CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT slightly improves the performance
when the VM is severely overcommited.

Quick test to estimate the performance: hackbench 1 process 1000

2xA15 host (baseline): 1.843s

2xA15 guest w/o patch: 2.083s
4xA15 guest w/o patch: 80.212s
8xA15 guest w/o patch: Could not be bothered to find out

2xA15 guest w/ patch: 2.102s
4xA15 guest w/ patch: 3.205s
8xA15 guest w/ patch: 6.887s

Quoted 6 lines
Quoted 3 lines

So we go from a 40x degradation to 1.5x in the 2x overcommit case, which is vaguely more acceptable.

Patch looks good, I can just apply it and add the other one I just send
as a reply if there are no objections.

Yeah, I missed the updated comments on this one, thanks for taking care of it.

Quoted 1 line

Sorry for the long turn-around on this one.

No worries. As long as it goes in, I'm happy. It makes such a difference on my box, it is absolutely mind boggling.

Thanks,

          M.
Signature · 1 line
Fast, cheap, reliable. Pick two.
#7 lore raw

Re: [PATCH v2 1/2] ARM: KVM: Yield CPU when vcpu executes a WFE

Quoted 33 lines · Marc Zyngier
Quoted 32 lines · Christoffer Dall
Quoted 30 lines · Marc Zyngier

On an (even slightly) oversubscribed system, spinlocks are quickly becoming a bottleneck, as some vcpus are spinning, waiting for a lock to be released, while the vcpu holding the lock may not be running at all.

This creates contention, and the observed slowdown is 40x for hackbench. No, this isn't a typo.

The solution is to trap blocking WFEs and tell KVM that we're now spinning. This ensures that other vpus will get a scheduling boost, allowing the lock to be released more quickly. Also, using CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT slightly improves the performance when the VM is severely overcommited.

Quick test to estimate the performance: hackbench 1 process 1000

2xA15 host (baseline): 1.843s

2xA15 guest w/o patch:  2.083s
4xA15 guest w/o patch:  80.212s
8xA15 guest w/o patch:  Could not be bothered to find out
2xA15 guest w/ patch:   2.102s
4xA15 guest w/ patch:   3.205s
8xA15 guest w/ patch:   6.887s

So we go from a 40x degradation to 1.5x in the 2x overcommit case, which is vaguely more acceptable.

Quoted 7 lines
Quoted 2 lines

Patch looks good, I can just apply it and add the other one I just send as a reply if there are no objections.

Yeah, I missed the updated comments on this one, thanks for taking care of it.

np.

Quoted 7 lines
Quoted 2 lines

Sorry for the long turn-around on this one.

No worries. As long as it goes in, I'm happy. It makes such a difference on my box, it is absolutely mind boggling.

Applied to kvm-arm-next.

-Christoffer

#8 lore raw

PATCH v2 2/2 arm64: KVM: Yield CPU when vcpu executes a WFE

In reply to Marc Zyngier

On an (even slightly) oversubscribed system, spinlocks are quickly becoming a bottleneck, as some vcpus are spinning, waiting for a lock to be released, while the vcpu holding the lock may not be running at all.

The solution is to trap blocking WFEs and tell KVM that we're now spinning. This ensures that other vpus will get a scheduling boost, allowing the lock to be released more quickly. Also, using CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT slightly improves the performance when the VM is severely overcommited.

---
 arch/arm64/include/asm/kvm_arm.h |  8 ++++++--
 arch/arm64/kvm/Kconfig           |  1 +
 arch/arm64/kvm/handle_exit.c     | 18 +++++++++++++-----
 3 files changed, 20 insertions(+), 7 deletions(-)
arch/arm64/include/asm/kvm_arm.h +6 -2
index a5f28e2..c98ef47 100644
@@ -63,6 +63,7 @@
  * TAC:		Trap ACTLR
  * TSC:		Trap SMC
  * TSW:		Trap cache operations by set/way
+ * TWE:		Trap WFE
  * TWI:		Trap WFI
  * TIDCP:	Trap L2CTLR/L2ECTLR
  * BSU_IS:	Upgrade barriers to the inner shareable domain
@@ -72,8 +73,9 @@
  * FMO:		Override CPSR.F and enable signaling with VF
  * SWIO:	Turn set/way invalidates into set/way clean+invalidate
  */
-#define HCR_GUEST_FLAGS (HCR_TSC | HCR_TSW | HCR_TWI | HCR_VM | HCR_BSU_IS | \
-			 HCR_FB | HCR_TAC | HCR_AMO | HCR_IMO | HCR_FMO | \
+#define HCR_GUEST_FLAGS (HCR_TSC | HCR_TSW | HCR_TWE | HCR_TWI | HCR_VM | \
+			 HCR_BSU_IS | HCR_FB | HCR_TAC | \
+			 HCR_AMO | HCR_IMO | HCR_FMO | \
 			 HCR_SWIO | HCR_TIDCP | HCR_RW)
 #define HCR_VIRT_EXCP_MASK (HCR_VA | HCR_VI | HCR_VF)
 
@@ -242,4 +244,6 @@
 
 #define ESR_EL2_EC_xABT_xFSR_EXTABT	0x10
 
+#define ESR_EL2_EC_WFI_ISS_WFE	(1 << 0)
+
 #endif /* __ARM64_KVM_ARM_H__ */
arch/arm64/kvm/Kconfig +1 -0
index 21e9082..4480ab3 100644
@@ -21,6 +21,7 @@ config KVM
 	select MMU_NOTIFIER
 	select PREEMPT_NOTIFIERS
 	select ANON_INODES
+	select HAVE_KVM_CPU_RELAX_INTERCEPT
 	select KVM_MMIO
 	select KVM_ARM_HOST
 	select KVM_ARM_VGIC
arch/arm64/kvm/handle_exit.c +13 -5
index 9beaca03..8da5606 100644
@@ -47,21 +47,29 @@ static int handle_smc(struct kvm_vcpu *vcpu, struct kvm_run *run)
 }
 
 /**
- * kvm_handle_wfi - handle a wait-for-interrupts instruction executed by a guest
+ * kvm_handle_wfx - handle a wait-for-interrupts or wait-for-event
+ *		    instruction executed by a guest
+ *
  * @vcpu:	the vcpu pointer
  *
- * Simply call kvm_vcpu_block(), which will halt execution of
+ * WFE: Yield the CPU and come back to this vcpu when the scheduler
+ * decides to.
+ * WFI: Simply call kvm_vcpu_block(), which will halt execution of
  * world-switches and schedule other host processes until there is an
  * incoming IRQ or FIQ to the VM.
  */
-static int kvm_handle_wfi(struct kvm_vcpu *vcpu, struct kvm_run *run)
+static int kvm_handle_wfx(struct kvm_vcpu *vcpu, struct kvm_run *run)
 {
-	kvm_vcpu_block(vcpu);
+	if (kvm_vcpu_get_hsr(vcpu) & ESR_EL2_EC_WFI_ISS_WFE)
+		kvm_vcpu_on_spin(vcpu);
+	else
+		kvm_vcpu_block(vcpu);
+
 	return 1;
 }
 
 static exit_handle_fn arm_exit_handlers[] = {
-	[ESR_EL2_EC_WFI]	= kvm_handle_wfi,
+	[ESR_EL2_EC_WFI]	= kvm_handle_wfx,
 	[ESR_EL2_EC_CP15_32]	= kvm_handle_cp15_32,
 	[ESR_EL2_EC_CP15_64]	= kvm_handle_cp15_64,
 	[ESR_EL2_EC_CP14_MR]	= kvm_handle_cp14_access,
Signature · 1 line
1.8.2.3
#9 lore raw

Re: [PATCH v2 2/2] arm64: KVM: Yield CPU when vcpu executes a WFE

Quoted 103 lines · Marc Zyngier

On an (even slightly) oversubscribed system, spinlocks are quickly becoming a bottleneck, as some vcpus are spinning, waiting for a lock to be released, while the vcpu holding the lock may not be running at all.

The solution is to trap blocking WFEs and tell KVM that we're now spinning. This ensures that other vpus will get a scheduling boost, allowing the lock to be released more quickly. Also, using CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT slightly improves the performance when the VM is severely overcommited.

---
 arch/arm64/include/asm/kvm_arm.h |  8 ++++++--
 arch/arm64/kvm/Kconfig           |  1 +
 arch/arm64/kvm/handle_exit.c     | 18 +++++++++++++-----
 3 files changed, 20 insertions(+), 7 deletions(-)
arch/arm64/include/asm/kvm_arm.h +6 -2
index a5f28e2..c98ef47 100644
@@ -63,6 +63,7 @@
  * TAC:		Trap ACTLR
  * TSC:		Trap SMC
  * TSW:		Trap cache operations by set/way
+ * TWE:		Trap WFE
  * TWI:		Trap WFI
  * TIDCP:	Trap L2CTLR/L2ECTLR
  * BSU_IS:	Upgrade barriers to the inner shareable domain
@@ -72,8 +73,9 @@
  * FMO:		Override CPSR.F and enable signaling with VF
  * SWIO:	Turn set/way invalidates into set/way clean+invalidate
  */
-#define HCR_GUEST_FLAGS (HCR_TSC | HCR_TSW | HCR_TWI | HCR_VM | HCR_BSU_IS | \
-			 HCR_FB | HCR_TAC | HCR_AMO | HCR_IMO | HCR_FMO | \
+#define HCR_GUEST_FLAGS (HCR_TSC | HCR_TSW | HCR_TWE | HCR_TWI | HCR_VM | \
+			 HCR_BSU_IS | HCR_FB | HCR_TAC | \
+			 HCR_AMO | HCR_IMO | HCR_FMO | \
 			 HCR_SWIO | HCR_TIDCP | HCR_RW)
 #define HCR_VIRT_EXCP_MASK (HCR_VA | HCR_VI | HCR_VF)
 
@@ -242,4 +244,6 @@
 
 #define ESR_EL2_EC_xABT_xFSR_EXTABT	0x10
 
+#define ESR_EL2_EC_WFI_ISS_WFE	(1 << 0)
+
 #endif /* __ARM64_KVM_ARM_H__ */
arch/arm64/kvm/Kconfig +1 -0
index 21e9082..4480ab3 100644
@@ -21,6 +21,7 @@ config KVM
 	select MMU_NOTIFIER
 	select PREEMPT_NOTIFIERS
 	select ANON_INODES
+	select HAVE_KVM_CPU_RELAX_INTERCEPT
 	select KVM_MMIO
 	select KVM_ARM_HOST
 	select KVM_ARM_VGIC
arch/arm64/kvm/handle_exit.c +13 -5
index 9beaca03..8da5606 100644
@@ -47,21 +47,29 @@ static int handle_smc(struct kvm_vcpu *vcpu, struct kvm_run *run)
 }
 
 /**
- * kvm_handle_wfi - handle a wait-for-interrupts instruction executed by a guest
+ * kvm_handle_wfx - handle a wait-for-interrupts or wait-for-event
+ *		    instruction executed by a guest
+ *
  * @vcpu:	the vcpu pointer
  *
- * Simply call kvm_vcpu_block(), which will halt execution of
+ * WFE: Yield the CPU and come back to this vcpu when the scheduler
+ * decides to.
+ * WFI: Simply call kvm_vcpu_block(), which will halt execution of
  * world-switches and schedule other host processes until there is an
  * incoming IRQ or FIQ to the VM.
  */
-static int kvm_handle_wfi(struct kvm_vcpu *vcpu, struct kvm_run *run)
+static int kvm_handle_wfx(struct kvm_vcpu *vcpu, struct kvm_run *run)
 {
-	kvm_vcpu_block(vcpu);
+	if (kvm_vcpu_get_hsr(vcpu) & ESR_EL2_EC_WFI_ISS_WFE)
+		kvm_vcpu_on_spin(vcpu);
+	else
+		kvm_vcpu_block(vcpu);
+
 	return 1;
 }
 
 static exit_handle_fn arm_exit_handlers[] = {
-	[ESR_EL2_EC_WFI]	= kvm_handle_wfi,
+	[ESR_EL2_EC_WFI]	= kvm_handle_wfx,
 	[ESR_EL2_EC_CP15_32]	= kvm_handle_cp15_32,
 	[ESR_EL2_EC_CP15_64]	= kvm_handle_cp15_64,
 	[ESR_EL2_EC_CP14_MR]	= kvm_handle_cp14_access,
Signature · 1 line
1.8.2.3

ack

#10 lore raw
In reply to Marc Zyngier
Quoted 16 lines · Marc Zyngier

This is a respin of a patch I posted a long while ago, this time with numbers that I hope to be convincing enough.

The basic idea is that spinning on WFE in a guest is a waste of resource, and that we're better of running another vcpu instead. This specially shows when the system is oversubscribed. The guest vcpus can be seen spinning, waiting for a lock to be released while the lock holder is nowhere near a physical CPU.

This patch series just enables WFE trapping on both ARM and arm64, and calls kvm_vcpu_on_spin(). This is enough to boost other vcpus, and dramatically reduce the overhead.

Branch available at:
git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git kvm-arm64/wfe-trap

Quoted 8 lines

Changes from v1:
- Added CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT, as it seems to give slightly better results (Thanks to Raghavendra K T)
- Updated commit message with results of 8x configuration

Marc Zyngier (2):
ARM: KVM: Yield CPU when vcpu executes a WFE
arm64: KVM: Yield CPU when vcpu executes a WFE

Using PLE handler and enabling CPU_RELAX_INTERCEPT part of the patches looks fine.