arm64: Add initial support for FEAT_WFxT
Subsystems (4)
Marc Zyngier
maintainerv2 cover arm64: Add initial support for FEAT_WFxT
The ARMv8.7 WFxT feature is a new take on the good old WFI/WFE instructions as they behave the same way, only taking an extra timeout parameter.
This small series aims at adding the minimal support for this feature, enabling it for both the kernel and KVM.
A potential addition to this series would be to remove the event generation from the counters, and rely on the timeout where it matters (spinlocks?). Feedback welcome.
Patches on top of 5.18-rc2, tested of the FVP AEM.
* From v1 [1]:
- Properly generate traces even if the deadline has already expired
- Collect RBs, with thanks.
[1] https://lore.kernel.org/r/[email protected]
Marc Zyngier (10):
arm64: Expand ESR_ELx_WFx_ISS_TI to match its ARMv8.7 definition
arm64: Add RV and RN fields for ESR_ELx_WFx_ISS
KVM: arm64: Simplify kvm_cpu_has_pending_timer()
KVM: arm64: Introduce kvm_counter_compute_delta() helper
KVM: arm64: Handle blocking WFIT instruction
KVM: arm64: Offer early resume for non-blocking WFxT instructions
KVM: arm64: Expose the WFXT feature to guests
arm64: Add HWCAP advertising FEAT_WFXT
arm64: Add wfet()/wfit() helpers
arm64: Use WFxT for __delay() when possible
Documentation/arm64/cpu-feature-registers.rst | 2 + Documentation/arm64/elf_hwcaps.rst | 4 ++ arch/arm64/include/asm/barrier.h | 4 ++ arch/arm64/include/asm/esr.h | 8 +++- arch/arm64/include/asm/hwcap.h | 1 + arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/include/uapi/asm/hwcap.h | 1 + arch/arm64/kernel/cpufeature.c | 13 +++++ arch/arm64/kernel/cpuinfo.c | 1 + arch/arm64/kvm/arch_timer.c | 47 ++++++++++++------- arch/arm64/kvm/arm.c | 6 +-- arch/arm64/kvm/handle_exit.c | 35 ++++++++++++-- arch/arm64/kvm/sys_regs.c | 2 + arch/arm64/lib/delay.c | 12 ++++- arch/arm64/tools/cpucaps | 1 + include/kvm/arm_arch_timer.h | 2 - 16 files changed, 110 insertions(+), 30 deletions(-)
Signature · 6 lines
2.34.1 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v2 1/10 arm64: Expand ESR_ELx_WFx_ISS_TI to match its ARMv8.7 definition
Starting with FEAT_WFXT in ARMv8.7, the TI field in the ISS that is reported on a WFx trap is expanded by one bit to allow the description of WFET and WFIT.
Special care is taken to exclude the WFxT bit from the mask used to match WFI so that it also matches WFIT when trapped from EL0.
--- arch/arm64/include/asm/esr.h | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
arch/arm64/include/asm/esr.h +4 -2
@@ -133,7 +133,8 @@#define ESR_ELx_CV (UL(1) << 24)#define ESR_ELx_COND_SHIFT (20)#define ESR_ELx_COND_MASK (UL(0xF) << ESR_ELx_COND_SHIFT) -#define ESR_ELx_WFx_ISS_TI (UL(1) << 0) +#define ESR_ELx_WFx_ISS_TI (UL(3) << 0) +#define ESR_ELx_WFx_ISS_WFxT (UL(2) << 0)#define ESR_ELx_WFx_ISS_WFI (UL(0) << 0)#define ESR_ELx_WFx_ISS_WFE (UL(1) << 0)#define ESR_ELx_xVC_IMM_MASK ((1UL << 16) - 1)
@@ -146,7 +147,8 @@#define DISR_EL1_ESR_MASK (ESR_ELx_AET | ESR_ELx_EA | ESR_ELx_FSC)/* ESR value templates for specific events */ -#define ESR_ELx_WFx_MASK (ESR_ELx_EC_MASK | ESR_ELx_WFx_ISS_TI) +#define ESR_ELx_WFx_MASK (ESR_ELx_EC_MASK | \ + (ESR_ELx_WFx_ISS_TI & ~ESR_ELx_WFx_ISS_WFxT))#define ESR_ELx_WFx_WFI_VAL ((ESR_ELx_EC_WFx << ESR_ELx_EC_SHIFT) | \ ESR_ELx_WFx_ISS_WFI)
Signature · 6 lines
2.34.1 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v2 01/10] arm64: Expand ESR_ELx_WFx_ISS_TI to match its ARMv8.7 definition
Quoted 10 lines · Marc Zyngier
Starting with
FEAT_WFXTin ARMv8.7, the TI field in the ISS that is reported on a WFx trap is expanded by one bit to allow the description of WFET and WFIT.Special care is taken to exclude the WFxT bit from the mask used to match WFI so that it also matches WFIT when trapped from EL0.
Reviewed-by Joey Gouly <[email protected]>Signed-off-by Marc Zyngier <[email protected]>
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v2 2/10 arm64: Add RV and RN fields for ESR_ELx_WFx_ISS
In reply to Marc ZyngierThe ISS field exposed by ESR_ELx contain two additional subfields with FEAT_WFxT:
- RN, the register number containing the timeout
- RV, indicating if the register number is valid
Describe these two fields according to the arch spec.
No functional change.
--- arch/arm64/include/asm/esr.h | 2 ++ 1 file changed, 2 insertions(+)
arch/arm64/include/asm/esr.h +2 -0
@@ -133,6 +133,8 @@#define ESR_ELx_CV (UL(1) << 24)#define ESR_ELx_COND_SHIFT (20)#define ESR_ELx_COND_MASK (UL(0xF) << ESR_ELx_COND_SHIFT) +#define ESR_ELx_WFx_ISS_RN (UL(0x1F) << 5) +#define ESR_ELx_WFx_ISS_RV (UL(1) << 2)#define ESR_ELx_WFx_ISS_TI (UL(3) << 0)#define ESR_ELx_WFx_ISS_WFxT (UL(2) << 0)#define ESR_ELx_WFx_ISS_WFI (UL(0) << 0)
Signature · 6 lines
2.34.1 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v2 02/10] arm64: Add RV and RN fields for ESR_ELx_WFx_ISS
Quoted 6 lines · Marc Zyngier
The ISS field exposed by
ESR_ELxcontain two additional subfields withFEAT_WFxT:- RN, the register number containing the timeout
- RV, indicating if the register number is valid
Quoted 6 lines
Describe these two fields according to the arch spec.
No functional change.
Reviewed-by Joey Gouly <[email protected]>Signed-off-by Marc Zyngier <[email protected]>
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v2 3/10 KVM: arm64: Simplify kvm_cpu_has_pending_timer()
In reply to Marc Zyngierkvm_cpu_has_pending_timer() ends up checking all the possible timers for a wake-up cause. However, we already check for pending interrupts whenever we try to wake-up a vcpu, including the timer interrupts.
Obviously, doing the same work twice is once too many. Reduce this helper to almost nothing, but keep it around, as we are going to make use of it soon.
--- arch/arm64/kvm/arch_timer.c | 10 ++-------- arch/arm64/kvm/arm.c | 5 ----- include/kvm/arm_arch_timer.h | 2 -- 3 files changed, 2 insertions(+), 15 deletions(-)
arch/arm64/kvm/arch_timer.c +2 -8
@@ -350,15 +350,9 @@ static bool kvm_timer_should_fire(struct arch_timer_context *timer_ctx)return cval <= now; } -bool kvm_timer_is_pending (struct kvm_vcpu *vcpu) +int kvm_cpu_has_pending_timer (struct kvm_vcpu *vcpu) { -struct timer_map map; - -get_timer_map (vcpu, &map); - -return kvm_timer_should_fire (map.direct_vtimer) || -kvm_timer_should_fire (map.direct_ptimer) || -kvm_timer_should_fire (map.emul_ptimer); +return 0 ; }/*
arch/arm64/kvm/arm.c +0 -5
@@ -356,11 +356,6 @@ void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu)kvm_arm_vcpu_destroy (vcpu); } -int kvm_cpu_has_pending_timer (struct kvm_vcpu *vcpu) -{ -return kvm_timer_is_pending (vcpu); -} -void kvm_arch_vcpu_blocking (struct kvm_vcpu *vcpu) {
include/kvm/arm_arch_timer.h +0 -2
@@ -76,8 +76,6 @@ int kvm_arm_timer_set_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr);int kvm_arm_timer_get_attr (struct kvm_vcpu *vcpu,struct kvm_device_attr *attr);int kvm_arm_timer_has_attr (struct kvm_vcpu *vcpu,struct kvm_device_attr *attr); -bool kvm_timer_is_pending (struct kvm_vcpu *vcpu); -u64 kvm_phys_timer_read (void );void kvm_timer_vcpu_load (struct kvm_vcpu *vcpu);
Signature · 6 lines
2.34.1 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v2 4/10 KVM: arm64: Introduce kvm_counter_compute_delta() helper
In reply to Marc ZyngierRefactor kvm_timer_compute_delta() and extract a helper that compute the delta (in ns) between a given timer and an arbitrary value.
No functional change expected.
--- arch/arm64/kvm/arch_timer.c | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-)
arch/arm64/kvm/arch_timer.c +10 -7
@@ -208,18 +208,16 @@ static irqreturn_t kvm_arch_timer_handler(int irq, void *dev_id)return IRQ_HANDLED ; } -static u64 kvm_timer_compute_delta (struct arch_timer_context *timer_ctx) +static u64 kvm_counter_compute_delta (struct arch_timer_context *timer_ctx, +u64 val) { -u64 cval, now; - - cval =timer_get_cval (timer_ctx); - now =kvm_phys_timer_read () -timer_get_offset (timer_ctx); +u64 now =kvm_phys_timer_read () -timer_get_offset (timer_ctx); -if (now < cval) { +if (now < val) {u64 ns; ns =cyclecounter_cyc2ns (timecounter->cc, - cval - now, + val - now, timecounter->mask, &timecounter->frac);return ns;
@@ -228,6 +226,11 @@ static u64 kvm_timer_compute_delta(struct arch_timer_context *timer_ctx)return 0 ; } +static u64 kvm_timer_compute_delta (struct arch_timer_context *timer_ctx) +{ +return kvm_counter_compute_delta (timer_ctx,timer_get_cval (timer_ctx)); +} +static bool kvm_timer_irq_can_fire (struct arch_timer_context *timer_ctx) {WARN_ON (timer_ctx && timer_ctx->loaded);
Signature · 6 lines
2.34.1 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v2 5/10 KVM: arm64: Handle blocking WFIT instruction
In reply to Marc ZyngierWhen trapping a blocking WFIT instruction, take it into account when computing the deadline of the background timer.
The state is tracked with a new vcpu flag, and is gated by a new CPU capability, which isn't currently enabled.
--- arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/kvm/arch_timer.c | 22 ++++++++++++++++++++-- arch/arm64/kvm/arm.c | 1 + arch/arm64/kvm/handle_exit.c | 7 ++++++- arch/arm64/tools/cpucaps | 1 + 5 files changed, 29 insertions(+), 3 deletions(-)
arch/arm64/include/asm/kvm_host.h +1 -0
@@ -441,6 +441,7 @@ struct kvm_vcpu_arch {#define KVM_ARM64_DEBUG_STATE_SAVE_TRBE (1 << 13)/* Save TRBE context if active */ #define KVM_ARM64_FP_FOREIGN_FPSTATE (1 << 14)#define KVM_ARM64_ON_UNSUPPORTED_CPU (1 << 15)/* Physical CPU not in supported_cpus */ +#define KVM_ARM64_WFIT (1 << 16)/* WFIT instruction trapped */ #define KVM_GUESTDBG_VALID_MASK (KVM_GUESTDBG_ENABLE | \ KVM_GUESTDBG_USE_SW_BP | \
arch/arm64/kvm/arch_timer.c +20 -2
@@ -239,6 +239,20 @@ static bool kvm_timer_irq_can_fire(struct arch_timer_context *timer_ctx) (ARCH_TIMER_CTRL_IT_MASK | ARCH_TIMER_CTRL_ENABLE)) == ARCH_TIMER_CTRL_ENABLE); } +static bool vcpu_has_wfit_active (struct kvm_vcpu *vcpu) +{ +return (cpus_have_final_cap (ARM64_HAS_WFXT ) && + (vcpu->arch.flags &KVM_ARM64_WFIT )); +} + +static u64 wfit_delay_ns (struct kvm_vcpu *vcpu) +{ +struct arch_timer_context *ctx =vcpu_vtimer (vcpu); +u64 val =vcpu_get_reg (vcpu,kvm_vcpu_sys_get_rt (vcpu)); + +return kvm_counter_compute_delta (ctx, val); +} +/* * Returns the earliest expiration time in ns among guest timers. * Note that it will return 0 if none of timers can fire.
@@ -256,6 +270,9 @@ static u64 kvm_timer_earliest_exp(struct kvm_vcpu *vcpu) min_delta =min (min_delta,kvm_timer_compute_delta (ctx)); } +if (vcpu_has_wfit_active (vcpu)) + min_delta =min (min_delta,wfit_delay_ns (vcpu)); +/* If none of timers can fire, then return 0 */ if (min_delta ==ULLONG_MAX )return 0 ;
@@ -355,7 +372,7 @@ static bool kvm_timer_should_fire(struct arch_timer_context *timer_ctx)int kvm_cpu_has_pending_timer (struct kvm_vcpu *vcpu) { -return 0 ; +return vcpu_has_wfit_active (vcpu) &&wfit_delay_ns (vcpu) ==0 ; }/*
@@ -481,7 +498,8 @@ static void kvm_timer_blocking(struct kvm_vcpu *vcpu)*/ if (!kvm_timer_irq_can_fire (map.direct_vtimer) && !kvm_timer_irq_can_fire (map.direct_ptimer) && - !kvm_timer_irq_can_fire (map.emul_ptimer)) + !kvm_timer_irq_can_fire (map.emul_ptimer) && + !vcpu_has_wfit_active (vcpu))return ;/*
arch/arm64/kvm/arm.c +1 -0
@@ -634,6 +634,7 @@ void kvm_vcpu_wfi(struct kvm_vcpu *vcpu)preempt_enable ();kvm_vcpu_halt (vcpu); + vcpu->arch.flags &= ~KVM_ARM64_WFIT ;kvm_clear_request (KVM_REQ_UNHALT , vcpu);preempt_disable ();
arch/arm64/kvm/handle_exit.c +6 -1
@@ -85,16 +85,21 @@ static int handle_no_fpsimd(struct kvm_vcpu *vcpu)* WFI: Simply call kvm_vcpu_halt(), which will halt execution of * world-switches and schedule other host processes until there is an * incoming IRQ or FIQ to the VM. +* WFIT: Same as WFI, with a timed wakeup implemented as a background timer */ static int kvm_handle_wfx (struct kvm_vcpu *vcpu) { -if (kvm_vcpu_get_esr (vcpu) & ESR_ELx_WFx_ISS_WFE) { +u64 esr =kvm_vcpu_get_esr (vcpu); + +if (esr & ESR_ELx_WFx_ISS_WFE) {trace_kvm_wfx_arm64 (*vcpu_pc (vcpu), true); vcpu->stat.wfe_exit_stat++;kvm_vcpu_on_spin (vcpu,vcpu_mode_priv (vcpu)); }else {trace_kvm_wfx_arm64 (*vcpu_pc (vcpu), false); vcpu->stat.wfi_exit_stat++; +if ((esr & (ESR_ELx_WFx_ISS_RV | ESR_ELx_WFx_ISS_WFxT)) == (ESR_ELx_WFx_ISS_RV | ESR_ELx_WFx_ISS_WFxT)) + vcpu->arch.flags |=KVM_ARM64_WFIT ;kvm_vcpu_wfi (vcpu); }
arch/arm64/tools/cpucaps +1 -0
@@ -38,6 +38,7 @@ HAS_STAGE2_FWB HAS_SYSREG_GIC_CPUIF HAS_TLB_RANGE HAS_VIRT_HOST_EXTN +HAS_WFXT HW_DBM KVM_PROTECTED_MODE MISMATCHED_CACHE_TYPE
Signature · 6 lines
2.34.1 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v2 6/10 KVM: arm64: Offer early resume for non-blocking WFxT instructions
In reply to Marc ZyngierFor WFxT instructions used with very small delays, it is not unlikely that the deadline is already expired by the time we reach the WFx handling code.
Check for this condition as soon as possible, and return to the guest immediately if we can.
--- arch/arm64/kvm/handle_exit.c | 30 ++++++++++++++++++++++++++---- 1 file changed, 26 insertions(+), 4 deletions(-)
arch/arm64/kvm/handle_exit.c +26 -4
@@ -80,12 +80,14 @@ static int handle_no_fpsimd(struct kvm_vcpu *vcpu)* * @vcpu: the vcpu pointer * -* WFE: Yield the CPU and come back to this vcpu when the scheduler +* WFE[T]: Yield the CPU and come back to this vcpu when the scheduler * decides to. * WFI: Simply call kvm_vcpu_halt(), which will halt execution of * world-switches and schedule other host processes until there is an * incoming IRQ or FIQ to the VM. * WFIT: Same as WFI, with a timed wakeup implemented as a background timer +* +* WF{I,E}T can immediately return if the deadline has already expired. */ static int kvm_handle_wfx (struct kvm_vcpu *vcpu) {
@@ -94,15 +96,35 @@ static int kvm_handle_wfx(struct kvm_vcpu *vcpu)if (esr & ESR_ELx_WFx_ISS_WFE) {trace_kvm_wfx_arm64 (*vcpu_pc (vcpu), true); vcpu->stat.wfe_exit_stat++; -kvm_vcpu_on_spin (vcpu,vcpu_mode_priv (vcpu)); }else {trace_kvm_wfx_arm64 (*vcpu_pc (vcpu), false); vcpu->stat.wfi_exit_stat++; -if ((esr & (ESR_ELx_WFx_ISS_RV | ESR_ELx_WFx_ISS_WFxT)) == (ESR_ELx_WFx_ISS_RV | ESR_ELx_WFx_ISS_WFxT)) + } + +if (esr & ESR_ELx_WFx_ISS_WFxT) { +if (esr & ESR_ELx_WFx_ISS_RV) { +u64 val, now; + + now =kvm_arm_timer_get_reg (vcpu,KVM_REG_ARM_TIMER_CNT ); + val =vcpu_get_reg (vcpu,kvm_vcpu_sys_get_rt (vcpu)); + +if (now >= val) + goto out; + }else { +/* Treat WFxT as WFx if RN is invalid */ + esr &= ~ESR_ELx_WFx_ISS_WFxT; + } + } + +if (esr & ESR_ELx_WFx_ISS_WFE) { +kvm_vcpu_on_spin (vcpu,vcpu_mode_priv (vcpu)); + }else { +if (esr & ESR_ELx_WFx_ISS_WFxT) vcpu->arch.flags |=KVM_ARM64_WFIT ; +kvm_vcpu_wfi (vcpu); } - +out:kvm_incr_pc (vcpu);return 1 ;
Signature · 6 lines
2.34.1 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v2 7/10 KVM: arm64: Expose the WFXT feature to guests
In reply to Marc ZyngierPlumb in the capability, and expose WFxT to guests when available.
--- arch/arm64/kernel/cpufeature.c | 12 ++++++++++++ arch/arm64/kvm/sys_regs.c | 2 ++ 2 files changed, 14 insertions(+)
arch/arm64/kernel/cpufeature.c +12 -0
@@ -237,6 +237,7 @@ static const struct arm64_ftr_bits ftr_id_aa64isar2[] = {ARM64_FTR_BITS (FTR_VISIBLE_IF_IS_ENABLED (CONFIG_ARM64_PTR_AUTH ),FTR_STRICT ,FTR_LOWER_SAFE ,ID_AA64ISAR2_GPA3_SHIFT ,4 ,0 ),ARM64_FTR_BITS (FTR_VISIBLE ,FTR_NONSTRICT ,FTR_LOWER_SAFE ,ID_AA64ISAR2_RPRES_SHIFT ,4 ,0 ), +ARM64_FTR_BITS (FTR_HIDDEN ,FTR_NONSTRICT ,FTR_LOWER_SAFE ,ID_AA64ISAR2_WFXT_SHIFT ,4 ,0 ), ARM64_FTR_END, };
@@ -2442,6 +2443,17 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .matches = has_cpuid_feature, .min_field_value = 1, }, + { + .desc = "WFx with timeout", + .capability = ARM64_HAS_WFXT, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .sys_reg = SYS_ID_AA64ISAR2_EL1, + .sign = FTR_UNSIGNED, + .field_pos = ID_AA64ISAR2_WFXT_SHIFT, + .field_width = 4, + .matches = has_cpuid_feature, + .min_field_value = ID_AA64ISAR2_WFXT_SUPPORTED, + }, {}, };
arch/arm64/kvm/sys_regs.c +2 -0
@@ -1144,6 +1144,8 @@ static u64 read_id_reg(const struct kvm_vcpu *vcpu,if (!vcpu_has_ptrauth (vcpu)) val &= ~(ARM64_FEATURE_MASK (ID_AA64ISAR2_APA3 ) |ARM64_FEATURE_MASK (ID_AA64ISAR2_GPA3 )); +if (!cpus_have_final_cap (ARM64_HAS_WFXT )) + val &= ~ARM64_FEATURE_MASK (ID_AA64ISAR2_WFXT );break ;case SYS_ID_AA64DFR0_EL1 :/* Limit debug to ARMv8.0 */
Signature · 6 lines
2.34.1 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v2 07/10] KVM: arm64: Expose the WFXT feature to guests
Quoted 3 lines · Marc Zyngier
Plumb in the capability, and expose WFxT to guests when available.
Signed-off-by Marc Zyngier <[email protected]>
Marc Zyngier
maintainerPATCH v2 8/10 arm64: Add HWCAP advertising FEAT_WFXT
In reply to Marc ZyngierIn order to allow userspace to enjoy WFET, add a new HWCAP that advertises it when available.
--- Documentation/arm64/cpu-feature-registers.rst | 2 ++ Documentation/arm64/elf_hwcaps.rst | 4 ++++ arch/arm64/include/asm/hwcap.h | 1 + arch/arm64/include/uapi/asm/hwcap.h | 1 + arch/arm64/kernel/cpufeature.c | 3 ++- arch/arm64/kernel/cpuinfo.c | 1 + 6 files changed, 11 insertions(+), 1 deletion(-)
Documentation/arm64/cpu-feature-registers.rst +2 -0
@@ -290,6 +290,8 @@ infrastructure: +------------------------------+---------+---------+ | RPRES | [7-4] | y | +------------------------------+---------+---------+ + | WFXT | [3-0] | y | + +------------------------------+---------+---------+ Appendix I: Example
Documentation/arm64/elf_hwcaps.rst +4 -0
@@ -264,6 +264,10 @@ HWCAP2_MTE3 Functionality implied by ID_AA64PFR1_EL1.MTE == 0b0011, as described by Documentation/arm64/memory-tagging-extension.rst. +HWCAP2_WFXT + + Functionality implied by ID_AA64ISAR2_EL1.WFXT == 0b0010. + 4. Unused AT_HWCAP bits -----------------------
arch/arm64/include/asm/hwcap.h +1 -0
@@ -109,6 +109,7 @@#define KERNEL_HWCAP_AFP __khwcap2_feature(AFP)#define KERNEL_HWCAP_RPRES __khwcap2_feature(RPRES)#define KERNEL_HWCAP_MTE3 __khwcap2_feature(MTE3) +#define KERNEL_HWCAP_WFXT __khwcap2_feature(WFXT)/* * This yields a mask that user programs can use to figure out what
arch/arm64/include/uapi/asm/hwcap.h +1 -0
@@ -79,5 +79,6 @@#define HWCAP2_AFP (1 << 20)#define HWCAP2_RPRES (1 << 21)#define HWCAP2_MTE3 (1 << 22) +#define HWCAP2_WFXT (1 << 23)#endif /* _UAPI__ASM_HWCAP_H */
arch/arm64/kernel/cpufeature.c +2 -1
@@ -237,7 +237,7 @@ static const struct arm64_ftr_bits ftr_id_aa64isar2[] = {ARM64_FTR_BITS (FTR_VISIBLE_IF_IS_ENABLED (CONFIG_ARM64_PTR_AUTH ),FTR_STRICT ,FTR_LOWER_SAFE ,ID_AA64ISAR2_GPA3_SHIFT ,4 ,0 ),ARM64_FTR_BITS (FTR_VISIBLE ,FTR_NONSTRICT ,FTR_LOWER_SAFE ,ID_AA64ISAR2_RPRES_SHIFT ,4 ,0 ), -ARM64_FTR_BITS (FTR_HIDDEN ,FTR_NONSTRICT ,FTR_LOWER_SAFE ,ID_AA64ISAR2_WFXT_SHIFT ,4 ,0 ), +ARM64_FTR_BITS (FTR_VISIBLE ,FTR_NONSTRICT ,FTR_LOWER_SAFE ,ID_AA64ISAR2_WFXT_SHIFT ,4 ,0 ), ARM64_FTR_END, };
@@ -2587,6 +2587,7 @@ static const struct arm64_cpu_capabilities arm64_elf_hwcaps[] = {HWCAP_CAP (SYS_ID_AA64MMFR0_EL1 ,ID_AA64MMFR0_ECV_SHIFT ,4 ,FTR_UNSIGNED ,1 ,CAP_HWCAP ,KERNEL_HWCAP_ECV ),HWCAP_CAP (SYS_ID_AA64MMFR1_EL1 ,ID_AA64MMFR1_AFP_SHIFT ,4 ,FTR_UNSIGNED ,1 ,CAP_HWCAP ,KERNEL_HWCAP_AFP ),HWCAP_CAP (SYS_ID_AA64ISAR2_EL1 ,ID_AA64ISAR2_RPRES_SHIFT ,4 ,FTR_UNSIGNED ,1 ,CAP_HWCAP ,KERNEL_HWCAP_RPRES ), + HWCAP_CAP(SYS_ID_AA64ISAR2_EL1, ID_AA64ISAR2_WFXT_SHIFT, 4, FTR_UNSIGNED, ID_AA64ISAR2_WFXT_SUPPORTED, CAP_HWCAP, KERNEL_HWCAP_WFXT), {}, };
arch/arm64/kernel/cpuinfo.c +1 -0
@@ -98,6 +98,7 @@ static const char *const hwcap_str[] = { [KERNEL_HWCAP_AFP] = "afp", [KERNEL_HWCAP_RPRES] = "rpres", [KERNEL_HWCAP_MTE3] = "mte3", + [KERNEL_HWCAP_WFXT] = "wfxt", };#ifdef CONFIG_COMPAT
Signature · 6 lines
2.34.1 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v2 08/10] arm64: Add HWCAP advertising FEAT_WFXT
Quoted 4 lines · Marc Zyngier
In order to allow userspace to enjoy WFET, add a new HWCAP that advertises it when available.
Signed-off-by Marc Zyngier <[email protected]>
I assume this series will go in via the kvm tree? Could we have the HWCAP changes at the beginning, I'm sure they'll conflict with other series (I plan to queue the SME patches).
Signature · 1 line
Catalin
Marc Zyngier
maintainerRe: [PATCH v2 08/10] arm64: Add HWCAP advertising FEAT_WFXT
Quoted 5 lines · Catalin Marinas
Quoted 3 lines · Marc Zyngier
In order to allow userspace to enjoy WFET, add a new HWCAP that advertises it when available.
Quoted 7 lines
Quoted 1 line
Signed-off-by Marc Zyngier <[email protected]>Acked-by Catalin Marinas <[email protected]>I assume this series will go in via the kvm tree? Could we have the HWCAP changes at the beginning, I'm sure they'll conflict with other series (I plan to queue the SME patches).
No problem, I can rejig the series to fit that plan. I'll push something tomorrow.
Thanks,
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Marc Zyngier
maintainerPATCH v2 9/10 arm64: Add wfet()/wfit() helpers
In reply to Marc ZyngierJust like we have helpers for WFI and WFE, add the WFxT versions. Note that the encoding is that reported by objdump, as no currrent toolchain knows about these instructions yet.
--- arch/arm64/include/asm/barrier.h | 4 ++++ 1 file changed, 4 insertions(+)
arch/arm64/include/asm/barrier.h +4 -0
@@ -16,7 +16,11 @@#define sev () asm volatile("sev" : : : "memory")#define wfe () asm volatile("wfe" : : : "memory") +#define wfet (val) asm volatile("msr s0_3_c1_c0_0, %0" \ + : : "r" (val) : "memory")#define wfi () asm volatile("wfi" : : : "memory") +#define wfit (val) asm volatile("msr s0_3_c1_c0_1, %0" \ + : : "r" (val) : "memory")#define isb () asm volatile("isb" : : : "memory")#define dmb (opt) asm volatile("dmb " #opt : : : "memory")
Signature · 6 lines
2.34.1 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v2 09/10] arm64: Add wfet()/wfit() helpers
Quoted 5 lines · Marc Zyngier
Just like we have helpers for WFI and WFE, add the WFxT versions. Note that the encoding is that reported by objdump, as no currrent toolchain knows about these instructions yet.
Signed-off-by Marc Zyngier <[email protected]>
Marc Zyngier
maintainerPATCH v2 10/10 arm64: Use WFxT for __delay() when possible
In reply to Marc ZyngierMarginally optimise __delay() by using a WFIT/WFET sequence. It probably is a win if no interrupt fires during the delay.
--- arch/arm64/lib/delay.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
arch/arm64/lib/delay.c +11 -1
@@ -27,7 +27,17 @@ void __delay(unsigned long cycles) {cycles_t start =get_cycles (); -if (arch_timer_evtstrm_available ()) { +if (cpus_have_const_cap (ARM64_HAS_WFXT )) { +u64 end = start + cycles; + +/* +* Start with WFIT. If an interrupt makes us resume +* early, use a WFET loop to complete the delay. +*/ +wfit (end); +while ((get_cycles () - start) < cycles) +wfet (end); + }else if (arch_timer_evtstrm_available ()) {const cycles_t timer_evt_period =USECS_TO_CYCLES (ARCH_TIMER_EVT_STREAM_PERIOD_US );
Signature · 6 lines
2.34.1 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v2 10/10] arm64: Use WFxT for __delay() when possible
Quoted 17 lines · Marc Zyngier
Marginally optimise
__delay()by using a WFIT/WFET sequence. It probably is a win if no interrupt fires during the delay.Signed-off-by Marc Zyngier <[email protected]>--- arch/arm64/lib/delay.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)arch/arm64/lib/delay.c +0
-1@@ -27,7 +27,17 @@ void __delay(unsigned long cycles) {cycles_t start =get_cycles (); -if (arch_timer_evtstrm_available ()) {
arch/arm64/lib/delay.c void __delay(unsigned long cycles) · line 39
Quoted 10 lines
arch/arm64/lib/delay.c +10
-0+if (cpus_have_const_cap (ARM64_HAS_WFXT )) { +u64 end = start + cycles; + +/* +* Start with WFIT. If an interrupt makes us resume +* early, use a WFET loop to complete the delay. +*/ +wfit (end); +while ((get_cycles () - start) < cycles) +wfet (end);
Do you use WFET here as a pending interrupt would cause WFIT to complete immediately?
arch/arm64/lib/delay.c void __delay(unsigned long cycles) · line 40
Quoted 1 line
arch/arm64/lib/delay.c +1
-0+ }else if (arch_timer_evtstrm_available ()) {
Nit: two spaces between else and if ;).
Marc Zyngier
maintainerRe: [PATCH v2 10/10] arm64: Use WFxT for __delay() when possible
Quoted 18 lines · Catalin Marinas
Quoted 16 lines · Marc Zyngier
Marginally optimise
__delay()by using a WFIT/WFET sequence. It probably is a win if no interrupt fires during the delay.Signed-off-by Marc Zyngier <[email protected]>--- arch/arm64/lib/delay.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)arch/arm64/lib/delay.c +0
-0@@ -27,7 +27,17 @@ void __delay(unsigned long cycles) {cycles_t start =get_cycles ();
Quoted 14 lines
arch/arm64/lib/delay.c void __delay(unsigned long cycles) · line 39
Quoted 11 lines
arch/arm64/lib/delay.c +10
-1-if (arch_timer_evtstrm_available ()) { +if (cpus_have_const_cap (ARM64_HAS_WFXT )) { +u64 end = start + cycles; + +/* +* Start with WFIT. If an interrupt makes us resume +* early, use a WFET loop to complete the delay. +*/ +wfit (end); +while ((get_cycles () - start) < cycles) +wfet (end);Do you use WFET here as a pending interrupt would cause WFIT to complete immediately?
Yes, that's the idea. Even if a pending interrupt is not immediately present, it could come halfway through, shortening the delay, and making WFIT useless until the interrupt is acknowledged.
I would have loved for WFIT to return a status indicating whether the wakeup was for a pending interrupt or for another reason (such as reaching the timeout), but apparently it was too much to ask... Maybe in ARMv11! ;-)
Quoted 4 lines
arch/arm64/lib/delay.c void __delay(unsigned long cycles) · line 40
Quoted 2 lines
arch/arm64/lib/delay.c +1
-0+ }else if (arch_timer_evtstrm_available ()) {Nit: two spaces between else and if ;).
I'll make sure to fix this! ;-)
Quoted 3 lines
Acked-by Catalin Marinas <[email protected]>
Thanks,
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Catalin Marinas
maintainerQuoted 3 lines · Marc Zyngier
A potential addition to this series would be to remove the event generation from the counters, and rely on the timeout where it matters (spinlocks?). Feedback welcome.
I think we still need to keep the event generation around, at least for hardware bugs we don't know about. I don't think user-space rely on it though, people tend to come up with weird delays like isb ;). But yes, the WFET should be handy when it turns up in hardware.
Signature · 5 lines
Catalin _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerQuoted 10 lines · Catalin Marinas
Quoted 3 lines · Marc Zyngier
A potential addition to this series would be to remove the event generation from the counters, and rely on the timeout where it matters (spinlocks?). Feedback welcome.
I think we still need to keep the event generation around, at least for hardware bugs we don't know about. I don't think user-space rely on it though, people tend to come up with weird delays like isb ;). But yes, the WFET should be handy when it turns up in hardware.
My hope was that the trick of using the event generation to work around systems failing to broadcast events could become a thing of the past when WFET is present in the HW. After all, they serve the same purpose (generate a local event to un-wedge the CPU).
But the more I look at it, the more I hate the potential solution. One of the issues is that WFxT takes an absolute deadline, rather than a relative one. So you end up with things like:
ISB MRS x0, CNTVCT_EL0 ADD x0, x0, #some_small_value WFET x0
which is really heavy handed for the slow path of an atomic operation. Even if you have ECV and CNTVCTSS_EL0 (which allows you to get rid of the ISB), it is a royal pain.
It would be much better if there was a *relative* version of WFET that would directly take a timeout relative to the current virtual count, but I can sense HW designers calling me names already, so I'll shut up.
M.
Signature · 5 lines
Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm