Skip to contentSkip to conversation

Add support for ARMv8.6 TWED feature

RFC PATCH In review Jingyi Wang 2 participants 11 messages Sep 29, 2020 – Nov 26, 2020

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover Add support for ARMv8.6 TWED feature

TWE Delay is an optional feature in ARMv8.6 Extentions. There is a performance benefit in waiting for a period of time for an event to arrive before taking the trap as it is common that event will arrive “quite soon” after executing the WFE instruction.

This series adds support for TWED feature and implements TWE delay value dynamic adjustment.

Thanks for Shameer's advice on this series. The function of this patch has been tested on TWED supported hardware and the performance of it is still on test, any advice will be welcomed.

Jingyi Wang (2):
KVM: arm64: Make use of TWED feature
KVM: arm64: Use dynamic TWE Delay value

Zengruan Ye (2):
arm64: cpufeature: TWED support detection
KVM: arm64: Add trace for TWED update

 arch/arm64/Kconfig                   | 10 +++++
 arch/arm64/include/asm/cpucaps.h     |  3 +-
 arch/arm64/include/asm/kvm_arm.h     |  5 +++
 arch/arm64/include/asm/kvm_emulate.h | 38 ++++++++++++++++++
 arch/arm64/include/asm/kvm_host.h    | 19 ++++++++-
 arch/arm64/include/asm/virt.h        |  8 ++++
 arch/arm64/kernel/cpufeature.c       | 12 ++++++
 arch/arm64/kvm/arm.c                 | 58 ++++++++++++++++++++++++++++
 arch/arm64/kvm/handle_exit.c         |  2 +
 arch/arm64/kvm/trace_arm.h           | 21 ++++++++++
 10 files changed, 174 insertions(+), 2 deletions(-)
Signature · 1 line
2.19.1
#1 lore raw

RFC PATCH 1/4 arm64: cpufeature: TWED support detection

From: Zengruan Ye <[email protected]>

TWE Delay is an optional feature in ARMv8.6 Extentions. This patch detect this feature.

---
 arch/arm64/Kconfig               | 10 ++++++++++
 arch/arm64/include/asm/cpucaps.h |  3 ++-
 arch/arm64/kernel/cpufeature.c   | 12 ++++++++++++
 3 files changed, 24 insertions(+), 1 deletion(-)
arch/arm64/Kconfig +10 -0
index 6d232837cbee..fe66f4fc5f49 100644
@@ -1666,6 +1666,16 @@ config ARCH_RANDOM
 
 endmenu
 
+menu "ARMv8.6 architectural features"
+
+config ARM64_TWED
+	bool "Enable suppot for delayed trapping of WFE"
+	default y
+	help
+	  Delayed Trapping of WFE (part of the ARMv8.6 Extensions)
+
+endmenu
+
 config ARM64_SVE
 	bool "ARM Scalable Vector Extension support"
 	default y
arch/arm64/include/asm/cpucaps.h +2 -1
index 07b643a70710..7b8f018d142b 100644
@@ -64,7 +64,8 @@
 #define ARM64_BTI				54
 #define ARM64_HAS_ARMv8_4_TTL			55
 #define ARM64_HAS_TLB_RANGE			56
+#define ARM64_HAS_TWED				57
 
-#define ARM64_NCAPS				57
+#define ARM64_NCAPS				58
 
 #endif /* __ASM_CPUCAPS_H */
arch/arm64/kernel/cpufeature.c +12 -0
index 6424584be01e..d042d32a1144 100644
@@ -2120,6 +2120,18 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
 		.min_field_value = ID_AA64PFR1_BT_BTI,
 		.sign = FTR_UNSIGNED,
 	},
+#endif
+#ifdef CONFIG_ARM64_TWED
+	{
+		.desc = "Delayed Trapping of WFE",
+		.capability = ARM64_HAS_TWED,
+		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
+		.matches = has_cpuid_feature,
+		.sys_reg = SYS_ID_AA64MMFR1_EL1,
+		.field_pos = ID_AA64MMFR1_TWED_SHIFT,
+		.sign = FTR_UNSIGNED,
+		.min_field_value = 1,
+	},
 #endif
 	{},
 };
Signature · 1 line
2.19.1
#2 lore raw

RFC PATCH 2/4 KVM: arm64: Make use of TWED feature

In reply to Jingyi Wang

For HCR_EL2, TWEDEn(bit[59]) decides whether TWED is enabled, and when the configurable delay is enabled, TWEDEL (bits[63:60]) encodes the minimum delay in taking a trap of WFE caused by the TWE bit in this register as 2^(TWEDEL + 8) cycles.

We use two kernel parameters "twed_enable" and "twed" to configure the register.

---
 arch/arm64/include/asm/kvm_arm.h     |  5 +++++
 arch/arm64/include/asm/kvm_emulate.h | 32 ++++++++++++++++++++++++++++
 arch/arm64/include/asm/kvm_host.h    | 13 +++++++++++
 arch/arm64/include/asm/virt.h        |  8 +++++++
 arch/arm64/kvm/arm.c                 | 22 +++++++++++++++++++
 5 files changed, 80 insertions(+)
arch/arm64/include/asm/kvm_arm.h +5 -0
index 1da8e3dc4455..03afb2921f1c 100644
@@ -12,6 +12,11 @@
 #include <asm/types.h>
 
 /* Hyp Configuration Register (HCR) bits */
+#define HCR_TWEDEL_SHIFT	60
+#define HCR_TWEDEL_MAX		(UL(0xf))
+#define HCR_TWEDEL_MASK		(HCR_TWEDEL_MAX << HCR_TWEDEL_SHIFT)
+#define HCR_TWEDEL	(UL(1) << HCR_TWEDEL_SHIFT)
+#define HCR_TWEDEN	(UL(1) << 59)
 #define HCR_FWB		(UL(1) << 46)
 #define HCR_API		(UL(1) << 41)
 #define HCR_APK		(UL(1) << 40)
arch/arm64/include/asm/kvm_emulate.h +32 -0
index 1cc5f5f72d0b..1a9cce836170 100644
@@ -102,6 +102,38 @@ static inline void vcpu_set_wfx_traps(struct kvm_vcpu *vcpu)
 	vcpu->arch.hcr_el2 |= HCR_TWI;
 }
 
+#ifdef CONFIG_ARM64_TWED
+static inline void vcpu_twed_enable(struct kvm_vcpu *vcpu)
+{
+	vcpu->arch.hcr_el2 |= HCR_TWEDEN;
+}
+
+static inline void vcpu_twed_disable(struct kvm_vcpu *vcpu)
+{
+	vcpu->arch.hcr_el2 &= ~HCR_TWEDEN;
+}
+
+static inline void vcpu_twed_init(struct kvm_vcpu *vcpu)
+{
+	vcpu->arch.twed = (u64)twed;
+}
+
+static inline void vcpu_set_twed(struct kvm_vcpu *vcpu)
+{
+	u64 delay = vcpu->arch.twed;
+	if (delay > HCR_TWEDEL_MAX)
+		delay = HCR_TWEDEL_MAX;
+
+	vcpu->arch.hcr_el2 &= ~HCR_TWEDEL_MASK;
+	vcpu->arch.hcr_el2 |= (delay << HCR_TWEDEL_SHIFT);
+}
+#else
+static inline void vcpu_twed_enable(struct kvm_vcpu *vcpu) {};
+static inline void vcpu_twed_disable(struct kvm_vcpu *vcpu) {};
+static inline void vcpu_twed_init(struct kvm_vcpu *vcpu) {};
+static inline void vcpu_set_twed(struct kvm_vcpu *vcpu) {};
+#endif
+
 static inline void vcpu_ptrauth_enable(struct kvm_vcpu *vcpu)
 {
 	vcpu->arch.hcr_el2 |= (HCR_API | HCR_APK);
arch/arm64/include/asm/kvm_host.h +13 -0
index 905c2b87e05a..380cd9c8ad0f 100644
@@ -371,6 +371,11 @@ struct kvm_vcpu_arch {
 		u64 last_steal;
 		gpa_t base;
 	} steal;
+
+#ifdef CONFIG_ARM64_TWED
+	/* WFE trap delay */
+	u64 twed;
+#endif
 };
 
 /* Pointer to the vcpu's SVE FFR for sve_{save,load}_state() */
@@ -688,4 +693,12 @@ bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu);
 #define kvm_arm_vcpu_sve_finalized(vcpu) \
 	((vcpu)->arch.flags & KVM_ARM64_VCPU_SVE_FINALIZED)
 
+#ifdef CONFIG_ARM64_TWED
+#define use_twed() (has_twed() && twed_enable)
+extern bool twed_enable;
+extern unsigned int twed;
+#else
+#define use_twed() false
+#endif
+
 #endif /* __ARM64_KVM_HOST_H__ */
arch/arm64/include/asm/virt.h +8 -0
index 09977acc007d..1a9e437e5bf6 100644
@@ -98,6 +98,14 @@ static __always_inline bool has_vhe(void)
 		return cpus_have_final_cap(ARM64_HAS_VIRT_HOST_EXTN);
 }
 
+static __always_inline bool has_twed(void)
+{
+	if (cpus_have_const_cap(ARM64_HAS_TWED))
+		return true;
+
+	return false;
+}
+
 #endif /* __ASSEMBLY__ */
 
 #endif /* ! __ASM__VIRT_H */
arch/arm64/kvm/arm.c +22 -0
index b588c3b5c2f0..5b5e8b14dcd5 100644
@@ -59,6 +59,14 @@ static bool vgic_present;
 static DEFINE_PER_CPU(unsigned char, kvm_arm_hardware_enabled);
 DEFINE_STATIC_KEY_FALSE(userspace_irqchip_in_use);
 
+#ifdef CONFIG_ARM64_TWED
+bool twed_enable = false;
+module_param(twed_enable, bool, S_IRUGO | S_IWUSR);
+
+unsigned int twed = 0;
+module_param(twed, uint, S_IRUGO | S_IWUSR);
+#endif
+
 int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu)
 {
 	return kvm_vcpu_exiting_guest_mode(vcpu) == IN_GUEST_MODE;
@@ -270,6 +278,13 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
 
 	vcpu->arch.hw_mmu = &vcpu->kvm->arch.mmu;
 
+	if (use_twed()) {
+		vcpu_twed_enable(vcpu);
+		vcpu_twed_init(vcpu);
+	} else {
+		vcpu_twed_disable(vcpu);
+	}
+
 	err = kvm_vgic_vcpu_init(vcpu);
 	if (err)
 		return err;
@@ -736,6 +751,13 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
 
 		kvm_arm_setup_debug(vcpu);
 
+		if (use_twed()) {
+			vcpu_twed_enable(vcpu);
+			vcpu_set_twed(vcpu);
+		} else {
+			vcpu_twed_disable(vcpu);
+		}
+
 		/**************************************************************
 		 * Enter the guest
 		 */
Signature · 1 line
2.19.1
#3 lore raw

RFC PATCH 3/4 KVM: arm64: Use dynamic TWE Delay value

In reply to Jingyi Wang

We implement two new kernel parameters for changing the delay adaptively:
twed_grow and twed_shrink
twed_grow affects the delay on WFE trap and twed_shrink does it on sched_in; depending on their value, the delay is modifier like this:

twed_shrink/ |
twed_grow | WFE trap exit | sched_in
--------------+---------------+------------
< 1 | = twed | = twed
otherwise | = twed + 1 | = twed - 1

---
 arch/arm64/include/asm/kvm_emulate.h |  6 ++++++
 arch/arm64/include/asm/kvm_host.h    |  6 +++++-
 arch/arm64/kvm/arm.c                 | 32 ++++++++++++++++++++++++++++
 arch/arm64/kvm/handle_exit.c         |  2 ++
 4 files changed, 45 insertions(+), 1 deletion(-)
arch/arm64/include/asm/kvm_emulate.h +6 -0
index 1a9cce836170..546d10b3b534 100644
@@ -116,16 +116,22 @@ static inline void vcpu_twed_disable(struct kvm_vcpu *vcpu)
 static inline void vcpu_twed_init(struct kvm_vcpu *vcpu)
 {
 	vcpu->arch.twed = (u64)twed;
+	vcpu->arch.twed_dirty = true;
 }
 
 static inline void vcpu_set_twed(struct kvm_vcpu *vcpu)
 {
 	u64 delay = vcpu->arch.twed;
+
+	if (!vcpu->arch.twed_dirty)
+		return;
+
 	if (delay > HCR_TWEDEL_MAX)
 		delay = HCR_TWEDEL_MAX;
 
 	vcpu->arch.hcr_el2 &= ~HCR_TWEDEL_MASK;
 	vcpu->arch.hcr_el2 |= (delay << HCR_TWEDEL_SHIFT);
+	vcpu->arch.twed_dirty = false;
 }
 #else
 static inline void vcpu_twed_enable(struct kvm_vcpu *vcpu) {};
arch/arm64/include/asm/kvm_host.h +5 -1
index 380cd9c8ad0f..35d1953d9d35 100644
@@ -375,6 +375,7 @@ struct kvm_vcpu_arch {
 #ifdef CONFIG_ARM64_TWED
 	/* WFE trap delay */
 	u64 twed;
+	bool twed_dirty;
 #endif
 };
 
@@ -595,7 +596,6 @@ void kvm_arm_vcpu_ptrauth_trap(struct kvm_vcpu *vcpu);
 
 static inline void kvm_arch_hardware_unsetup(void) {}
 static inline void kvm_arch_sync_events(struct kvm *kvm) {}
-static inline void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu) {}
 static inline void kvm_arch_vcpu_block_finish(struct kvm_vcpu *vcpu) {}
 
 void kvm_arm_init_debug(void);
@@ -697,8 +697,12 @@ bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu);
 #define use_twed() (has_twed() && twed_enable)
 extern bool twed_enable;
 extern unsigned int twed;
+void grow_twed(struct kvm_vcpu *vcpu);
+void shrink_twed(struct kvm_vcpu *vcpu);
 #else
 #define use_twed() false
+static inline void grow_twed(struct kvm_vcpu *vcpu) {};
+static inline void shrink_twed(struct kvm_vcpu *vcpu) {};
 #endif
 
 #endif /* __ARM64_KVM_HOST_H__ */
arch/arm64/kvm/arm.c +32 -0
index 5b5e8b14dcd5..989bffdcb3e9 100644
@@ -65,6 +65,32 @@ module_param(twed_enable, bool, S_IRUGO | S_IWUSR);
 
 unsigned int twed = 0;
 module_param(twed, uint, S_IRUGO | S_IWUSR);
+
+static unsigned int twed_grow = 0;
+module_param(twed_grow, uint, S_IRUGO | S_IWUSR);
+
+static unsigned int twed_shrink = 0;
+module_param(twed_shrink, uint, S_IRUGO | S_IWUSR);
+
+void grow_twed(struct kvm_vcpu *vcpu)
+{
+	u64 old = vcpu->arch.twed;
+
+	if (old < HCR_TWEDEL_MAX && twed_grow) {
+		vcpu->arch.twed += 1;
+		vcpu->arch.twed_dirty = true;
+	}
+}
+
+void shrink_twed(struct kvm_vcpu *vcpu)
+{
+	u64 old = vcpu->arch.twed;
+
+	if (old > 0 && twed_shrink) {
+		vcpu->arch.twed -= 1;
+		vcpu->arch.twed_dirty = true;
+	}
+}
 #endif
 
 int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu)
@@ -108,6 +134,12 @@ static int kvm_arm_default_max_vcpus(void)
 	return vgic_present ? kvm_vgic_get_max_vcpus() : KVM_MAX_VCPUS;
 }
 
+void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu)
+{
+	if (use_twed())
+		shrink_twed(vcpu);
+}
+
 /**
  * kvm_arch_init_vm - initializes a VM data structure
  * @kvm:	pointer to the KVM struct
arch/arm64/kvm/handle_exit.c +2 -0
index 5d690d60ccad..2ad72defa3c9 100644
@@ -91,6 +91,8 @@ static int kvm_handle_wfx(struct kvm_vcpu *vcpu)
 {
 	if (kvm_vcpu_get_esr(vcpu) & ESR_ELx_WFx_ISS_WFE) {
 		trace_kvm_wfx_arm64(*vcpu_pc(vcpu), true);
+		if (use_twed())
+			grow_twed(vcpu);
 		vcpu->stat.wfe_exit_stat++;
 		kvm_vcpu_on_spin(vcpu, vcpu_mode_priv(vcpu));
 	} else {
Signature · 6 lines
2.19.1

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#4 lore raw

RFC PATCH 4/4 KVM: arm64: Add trace for TWED update

In reply to Jingyi Wang

From: Zengruan Ye <[email protected]>

Add tracepoints for TWE delay value update

---
 arch/arm64/kvm/arm.c       |  4 ++++
 arch/arm64/kvm/trace_arm.h | 21 +++++++++++++++++++++
 2 files changed, 25 insertions(+)
arch/arm64/kvm/arm.c +4 -0
index 989bffdcb3e9..c3d7a326bf1b 100644
@@ -80,6 +80,8 @@ void grow_twed(struct kvm_vcpu *vcpu)
 		vcpu->arch.twed += 1;
 		vcpu->arch.twed_dirty = true;
 	}
+
+	trace_kvm_twed_update(vcpu->vcpu_id, vcpu->arch.twed, old);
 }
 
 void shrink_twed(struct kvm_vcpu *vcpu)
@@ -90,6 +92,8 @@ void shrink_twed(struct kvm_vcpu *vcpu)
 		vcpu->arch.twed -= 1;
 		vcpu->arch.twed_dirty = true;
 	}
+
+	trace_kvm_twed_update(vcpu->vcpu_id, vcpu->arch.twed, old);
 }
 #endif
 
arch/arm64/kvm/trace_arm.h +21 -0
index ff0444352bba..5081266399e8 100644
@@ -367,6 +367,27 @@ TRACE_EVENT(kvm_timer_emulate,
 		  __entry->timer_idx, __entry->should_fire)
 );
 
+TRACE_EVENT(kvm_twed_update,
+	TP_PROTO(unsigned int vcpu_id, unsigned int new, unsigned int old),
+	TP_ARGS(vcpu_id, new, old),
+
+	TP_STRUCT__entry(
+		__field(	unsigned int,	vcpu_id		)
+		__field(	unsigned int,	new		)
+		__field(	unsigned int,	old		)
+	),
+
+	TP_fast_assign(
+		__entry->vcpu_id	= vcpu_id;
+		__entry->new		= new;
+		__entry->old		= old;
+	),
+
+	TP_printk("vcpu %u old %u new %u (%s)",
+		  __entry->vcpu_id, __entry->old, __entry->new,
+		  __entry->old < __entry->new ? "growed" : "shrinked")
+);
+
 #endif /* _TRACE_ARM_ARM64_KVM_H */
 
 #undef TRACE_INCLUDE_PATH
Signature · 1 line
2.19.1
maintainer
#5 lore raw
In reply to Jingyi Wang
Quoted 4 lines · Jingyi Wang

TWE Delay is an optional feature in ARMv8.6 Extentions. There is a performance benefit in waiting for a period of time for an event to arrive before taking the trap as it is common that event will arrive “quite soon” after executing the WFE instruction.

Define "quite soon". Quantify "performance benefits". Which are the workloads that actually benefit from this imitation of the x86 PLE?

I was opposed to this when the spec was drafted, and I still am given that there is zero supporting evidence that it bring any gain over immediate trapping in an oversubscribed environment (which is the only case where it matters).

Thanks,

        M.
Quoted 27 lines

This series adds support for TWED feature and implements TWE delay value dynamic adjustment.

Thanks for Shameer's advice on this series. The function of this patch has been tested on TWED supported hardware and the performance of it is still on test, any advice will be welcomed.

Jingyi Wang (2):
KVM: arm64: Make use of TWED feature
KVM: arm64: Use dynamic TWE Delay value

Zengruan Ye (2):
arm64: cpufeature: TWED support detection
KVM: arm64: Add trace for TWED update

 arch/arm64/Kconfig                   | 10 +++++
 arch/arm64/include/asm/cpucaps.h     |  3 +-
 arch/arm64/include/asm/kvm_arm.h     |  5 +++
 arch/arm64/include/asm/kvm_emulate.h | 38 ++++++++++++++++++
 arch/arm64/include/asm/kvm_host.h    | 19 ++++++++-
 arch/arm64/include/asm/virt.h        |  8 ++++
 arch/arm64/kernel/cpufeature.c       | 12 ++++++
 arch/arm64/kvm/arm.c                 | 58 ++++++++++++++++++++++++++++
 arch/arm64/kvm/handle_exit.c         |  2 +
 arch/arm64/kvm/trace_arm.h           | 21 ++++++++++
 10 files changed, 174 insertions(+), 2 deletions(-)
Signature · 1 line
Jazz is not dead. It just smells funny...
#6 lore raw

Hi Marc,

Quoted 9 lines · Marc Zyngier
Quoted 4 lines · Jingyi Wang

TWE Delay is an optional feature in ARMv8.6 Extentions. There is a performance benefit in waiting for a period of time for an event to arrive before taking the trap as it is common that event will arrive “quite soon” after executing the WFE instruction.

Define "quite soon". Quantify "performance benefits". Which are the workloads that actually benefit from this imitation of the x86 PLE?

Quoted 8 lines

I was opposed to this when the spec was drafted, and I still am given that there is zero supporting evidence that it bring any gain over immediate trapping in an oversubscribed environment (which is the only case where it matters).

Thanks,

        M.

Sure, I will do more performance tests and post the results as soon as possible.

Thanks,
Jingyi

#7 lore raw
In reply to Jingyi Wang

Hi all,

Sorry for the delay. I have been testing the TWED feature performance
lately. We select unixbench as the benchmark for some items of it is lock-intensive(fstime/fsbuffer/fsdisk). We run unixbench on a 4-VCPU VM, and bind every two VCPUs on one PCPU. Fixed TWED value is used and here is the result.

     twed_value   | fstime        | fsbuffer   | fsdisk
    --------------+---------------+------------+------------
     disable      | 16.0          | 14.1       | 18.0
     0            | 16.3          | 13.5       | 17.2
     1            | 17.5          | 14.7       | 17.4
     2            | 17.3          | 15.3       | 18.0
     3            | 17.7          | 15.2       | 18.9
     4            | 17.9          | 14.3       | 18.2
     5            | 17.2          | 14.1       | 19.0
     6            | 5.8           | 4.2        | 5.7
     7            | 6.2           | 5.6        | 12.8

Note:
fstime: File Copy 1024 bufsize 2000 maxblocks
fsbuffer: File Copy 256 bufsize 500 maxblocks
fsdisk: File Copy 4096 bufsize 8000 maxblocks
The index of unixbench, higher is better.

It is shown that, compared to the circumstance that TWED is disabled, lock-intensive testing items have better performance if an appropriate TWED value is set(up to 5.6%~11.9%). Meanwhile, the complete unixbench test is run to prove that other testing items are not sensitive to this parameter.

Thanks
Jingyi

Quoted 32 lines · Jingyi Wang

TWE Delay is an optional feature in ARMv8.6 Extentions. There is a performance benefit in waiting for a period of time for an event to arrive before taking the trap as it is common that event will arrive “quite soon” after executing the WFE instruction.

This series adds support for TWED feature and implements TWE delay value dynamic adjustment.

Thanks for Shameer's advice on this series. The function of this patch has been tested on TWED supported hardware and the performance of it is still on test, any advice will be welcomed.

Jingyi Wang (2):
KVM: arm64: Make use of TWED feature
KVM: arm64: Use dynamic TWE Delay value

Zengruan Ye (2):
arm64: cpufeature: TWED support detection
KVM: arm64: Add trace for TWED update

  arch/arm64/Kconfig                   | 10 +++++
  arch/arm64/include/asm/cpucaps.h     |  3 +-
  arch/arm64/include/asm/kvm_arm.h     |  5 +++
  arch/arm64/include/asm/kvm_emulate.h | 38 ++++++++++++++++++
  arch/arm64/include/asm/kvm_host.h    | 19 ++++++++-
  arch/arm64/include/asm/virt.h        |  8 ++++
  arch/arm64/kernel/cpufeature.c       | 12 ++++++
  arch/arm64/kvm/arm.c                 | 58 ++++++++++++++++++++++++++++
  arch/arm64/kvm/handle_exit.c         |  2 +
  arch/arm64/kvm/trace_arm.h           | 21 ++++++++++
  10 files changed, 174 insertions(+), 2 deletions(-)
#8 lore raw

Hi Marc,

Gentle ping, could you please give some comments on this patch or the current test results? Thanks in advance.

Thanks,
Jingyi

Quoted 30 lines · Jingyi Wang

Hi all,

Sorry for the delay. I have been testing the TWED feature performance
lately. We select unixbench as the benchmark for some items of it is lock-intensive(fstime/fsbuffer/fsdisk). We run unixbench on a 4-VCPU VM, and bind every two VCPUs on one PCPU. Fixed TWED value is used and here is the result.

     twed_value | fstime    | fsbuffer  | fsdisk
    --------------+---------------+------------+------------

      disable   | 16.0      | 14.1    | 18.0
      0      | 16.3      | 13.5    | 17.2
      1      | 17.5      | 14.7    | 17.4
      2      | 17.3      | 15.3    | 18.0
      3      | 17.7      | 15.2    | 18.9
      4      | 17.9      | 14.3    | 18.2
      5      | 17.2      | 14.1    | 19.0
      6      | 5.8       | 4.2     | 5.7
      7      | 6.2      | 5.6     | 12.8

Note:
fstime: File Copy 1024 bufsize 2000 maxblocks
fsbuffer: File Copy 256 bufsize 500 maxblocks
fsdisk: File Copy 4096 bufsize 8000 maxblocks
The index of unixbench, higher is better.

It is shown that, compared to the circumstance that TWED is disabled, lock-intensive testing items have better performance if an appropriate TWED value is set(up to 5.6%~11.9%). Meanwhile, the complete unixbench test is run to prove that other testing items are not sensitive to this parameter.

Quoted 36 lines

Thanks
Jingyi

Quoted 32 lines · Jingyi Wang

TWE Delay is an optional feature in ARMv8.6 Extentions. There is a performance benefit in waiting for a period of time for an event to arrive before taking the trap as it is common that event will arrive “quite soon” after executing the WFE instruction.

This series adds support for TWED feature and implements TWE delay value dynamic adjustment.

Thanks for Shameer's advice on this series. The function of this patch has been tested on TWED supported hardware and the performance of it is still on test, any advice will be welcomed.

Jingyi Wang (2):
   KVM: arm64: Make use of TWED feature
   KVM: arm64: Use dynamic TWE Delay value

Zengruan Ye (2):
   arm64: cpufeature: TWED support detection
   KVM: arm64: Add trace for TWED update

  arch/arm64/Kconfig                   | 10 +++++ arch/arm64/include/asm/cpucaps.h     |  3 +- arch/arm64/include/asm/kvm_arm.h     |  5 +++ arch/arm64/include/asm/kvm_emulate.h | 38 ++++++++++++++++++ arch/arm64/include/asm/kvm_host.h    | 19 ++++++++- arch/arm64/include/asm/virt.h        |  8 ++++ arch/arm64/kernel/cpufeature.c       | 12 ++++++ arch/arm64/kvm/arm.c                 | 58 ++++++++++++++++++++++++++++ arch/arm64/kvm/handle_exit.c         |  2 + arch/arm64/kvm/trace_arm.h           | 21 ++++++++++

  10 files changed, 174 insertions(+), 2 deletions(-)

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

maintainer
#9 lore raw
In reply to Jingyi Wang
Quoted 7 lines · Jingyi Wang

Hi all,

Sorry for the delay. I have been testing the TWED feature performance lately. We select unixbench as the benchmark for some items of it is lock-intensive(fstime/fsbuffer/fsdisk). We run unixbench on a 4-VCPU VM, and bind every two VCPUs on one PCPU. Fixed TWED value is used and here is the result.

How representative is this?

TBH, I only know of two real world configurations: one where the vCPUs are pinned to different physical CPUs (and in this case your patch has absolutely no effect as long as there is no concurrent tasks), and one where there is oversubscription, and the scheduler moves things around as it sees fit, depending on the load.

Having two vCPUs pinned per CPU feels like a test that has been picked to give the result you wanted. I'd like to see the full picture, including the case that matters for current use cases. I'm specially interested in the cases where the system is oversubscribed, because TWED is definitely going to screw with the scheduler latency.

Thanks,

        M.
Signature · 5 lines
Jazz is not dead. It just smells funny...
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#10 lore raw

Hi Marc,

I will consider more circumstances in the later test. Thanks for the advice.

Thanks,
Jingyi

Quoted 25 lines · Marc Zyngier
Quoted 7 lines · Jingyi Wang

Hi all,

Sorry for the delay. I have been testing the TWED feature performance lately. We select unixbench as the benchmark for some items of it is lock-intensive(fstime/fsbuffer/fsdisk). We run unixbench on a 4-VCPU VM, and bind every two VCPUs on one PCPU. Fixed TWED value is used and here is the result.

How representative is this?

TBH, I only know of two real world configurations: one where the vCPUs are pinned to different physical CPUs (and in this case your patch has absolutely no effect as long as there is no concurrent tasks), and one where there is oversubscription, and the scheduler moves things around as it sees fit, depending on the load.

Having two vCPUs pinned per CPU feels like a test that has been picked to give the result you wanted. I'd like to see the full picture, including the case that matters for current use cases. I'm specially interested in the cases where the system is oversubscribed, because TWED is definitely going to screw with the scheduler latency.

Quoted 3 lines

Thanks,

        M.

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm