KVM/arm fixes for 5.6
Subsystems (5)
Marc Zyngier
maintainerPaolo,
This is a small update containing a number of fixes, the most important ones making sure we force the inlining of any helper that gets used by the EL2 code (James identified that some bad things happen with CLang and the Shadow Call Stack extention).
Please pull,
M.
The following changes since commit 4a267aa707953a9a73d1f5dc7f894dd9024a92be:
KVM: arm64: Treat emulated TVAL TimerValue as a signed 32-bit integer (2020-01-28 13:09:31 +0000)
are available in the Git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git tags/kvmarm-fixes-5.6-1
for you to fetch changes up to e43f1331e2ef913b8c566920c9af75e0ccdd1d3f:
arm64: Ask the compiler to __always_inline functions used by KVM at HYP (2020-02-22 11:01:47 +0000)
----------------------------------------------------------------
KVM/arm fixes for 5.6, take #1
- Fix compilation on 32bit
- Move VHE guest entry/exit into the VHE-specific entry code
- Make sure all functions called by the non-VHE HYP code is tagged as __always_inline
----------------------------------------------------------------
James Morse (3):
KVM: arm64: Ask the compiler to __always_inline functions used at HYP
KVM: arm64: Define our own swab32() to avoid a uapi static inline
arm64: Ask the compiler to __always_inline functions used by KVM at HYP
Jeremy Cline (1):
KVM: arm/arm64: Fix up includes for trace.h
Mark Rutland (1):
kvm: arm/arm64: Fold VHE entry/exit work into kvm_vcpu_run_vhe()
arch/arm/include/asm/kvm_host.h | 3 -- arch/arm64/include/asm/arch_gicv3.h | 2 +- arch/arm64/include/asm/cache.h | 2 +- arch/arm64/include/asm/cacheflush.h | 2 +- arch/arm64/include/asm/cpufeature.h | 10 +++---- arch/arm64/include/asm/io.h | 4 +-- arch/arm64/include/asm/kvm_emulate.h | 48 ++++++++++++++++---------------- arch/arm64/include/asm/kvm_host.h | 32 --------------------- arch/arm64/include/asm/kvm_hyp.h | 7 +++++ arch/arm64/include/asm/kvm_mmu.h | 3 +- arch/arm64/include/asm/virt.h | 2 +- arch/arm64/kvm/hyp/switch.c | 39 ++++++++++++++++++++++++-- arch/arm64/kvm/hyp/vgic-v2-cpuif-proxy.c | 4 +-- virt/kvm/arm/arm.c | 2 -- virt/kvm/arm/trace.h | 1 + 15 files changed, 84 insertions(+), 77 deletions(-)
Marc Zyngier
maintainerPATCH 1/5 KVM: arm/arm64: Fix up includes for trace.h
From: Jeremy Cline <[email protected]>
Fedora kernel builds on armv7hl began failing recently because kvm_arm_exception_type and kvm_arm_exception_class were undeclared in trace.h. Add the missing include.
--- virt/kvm/arm/trace.h | 1 + 1 file changed, 1 insertion(+)
virt/kvm/arm/trace.h +1 -0
@@ -4,6 +4,7 @@#include <kvm/arm_arch_timer.h> #include <linux/tracepoint.h> +#include <asm/kvm_arm.h> #undef TRACE_SYSTEM#define TRACE_SYSTEM kvm
Signature · 1 line
2.20.1
Marc Zyngier
maintainerPATCH 2/5 kvm: arm/arm64: Fold VHE entry/exit work into kvm_vcpu_run_vhe()
In reply to Marc ZyngierFrom: Mark Rutland <[email protected]>
With VHE, running a vCPU always requires the sequence:
1. kvm_arm_vhe_guest_enter();
2. kvm_vcpu_run_vhe();
3. kvm_arm_vhe_guest_exit()
... and as we invoke this from the shared arm/arm64 KVM code, 32-bit arm has to provide stubs for all three functions.
To simplify the common code, and make it easier to make further modifications to the arm64-specific portions in the near future, let's fold kvm_arm_vhe_guest_enter() and kvm_arm_vhe_guest_exit() into kvm_vcpu_run_vhe().
The 32-bit stubs for kvm_arm_vhe_guest_enter() and kvm_arm_vhe_guest_exit() are removed, as they are no longer used. The 32-bit stub for kvm_vcpu_run_vhe() is left as-is.
There should be no functional change as a result of this patch.
--- arch/arm/include/asm/kvm_host.h | 3 --- arch/arm64/include/asm/kvm_host.h | 32 ------------------------- arch/arm64/kvm/hyp/switch.c | 39 +++++++++++++++++++++++++++++-- virt/kvm/arm/arm.c | 2 -- 4 files changed, 37 insertions(+), 39 deletions(-)
arch/arm/include/asm/kvm_host.h +0 -3
@@ -394,9 +394,6 @@ static inline void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu) {}static inline void kvm_vcpu_pmu_restore_guest (struct kvm_vcpu *vcpu) {}static inline void kvm_vcpu_pmu_restore_host (struct kvm_vcpu *vcpu) {} -static inline void kvm_arm_vhe_guest_enter (void ) {} -static inline void kvm_arm_vhe_guest_exit (void ) {} -#define KVM_BP_HARDEN_UNKNOWN -1#define KVM_BP_HARDEN_WA_NEEDED 0#define KVM_BP_HARDEN_NOT_REQUIRED 1
arch/arm64/include/asm/kvm_host.h +0 -32
@@ -628,38 +628,6 @@ static inline void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr) {}static inline void kvm_clr_pmu_events (u32 clr) {}#endif -static inline void kvm_arm_vhe_guest_enter (void ) -{ -local_daif_mask (); - -/* -* Having IRQs masked via PMR when entering the guest means the GIC -* will not signal the CPU of interrupts of lower priority, and the -* only way to get out will be via guest exceptions. -* Naturally, we want to avoid this. -* -* local_daif_mask() already sets GIC_PRIO_PSR_I_SET, we just need a -* dsb to ensure the redistributor is forwards EL2 IRQs to the CPU. -*/ -pmr_sync (); -} - -static inline void kvm_arm_vhe_guest_exit (void ) -{ -/* -* local_daif_restore() takes care to properly restore PSTATE.DAIF -* and the GIC PMR if the host is using IRQ priorities. -*/ -local_daif_restore (DAIF_PROCCTX_NOIRQ ); - -/* -* When we exit from the guest we change a number of CPU configuration -* parameters, such as traps. Make sure these changes take effect -* before running the host or additional guests. -*/ -isb (); -} -#define KVM_BP_HARDEN_UNKNOWN -1#define KVM_BP_HARDEN_WA_NEEDED 0#define KVM_BP_HARDEN_NOT_REQUIRED 1
arch/arm64/kvm/hyp/switch.c +37 -2
@@ -617,7 +617,7 @@ static void __hyp_text __pmu_switch_to_host(struct kvm_cpu_context *host_ctxt) }/* Switch to the guest for VHE systems running in EL2 */ -int kvm_vcpu_run_vhe (struct kvm_vcpu *vcpu) +static int __kvm_vcpu_run_vhe (struct kvm_vcpu *vcpu) {struct kvm_cpu_context *host_ctxt;struct kvm_cpu_context *guest_ctxt;
@@ -670,7 +670,42 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)return exit_code; } -NOKPROBE_SYMBOL (kvm_vcpu_run_vhe); +NOKPROBE_SYMBOL (__kvm_vcpu_run_vhe); + +int kvm_vcpu_run_vhe (struct kvm_vcpu *vcpu) +{ +int ret; + +local_daif_mask (); + +/* +* Having IRQs masked via PMR when entering the guest means the GIC +* will not signal the CPU of interrupts of lower priority, and the +* only way to get out will be via guest exceptions. +* Naturally, we want to avoid this. +* +* local_daif_mask() already sets GIC_PRIO_PSR_I_SET, we just need a +* dsb to ensure the redistributor is forwards EL2 IRQs to the CPU. +*/ +pmr_sync (); + + ret =__kvm_vcpu_run_vhe (vcpu); + +/* +* local_daif_restore() takes care to properly restore PSTATE.DAIF +* and the GIC PMR if the host is using IRQ priorities. +*/ +local_daif_restore (DAIF_PROCCTX_NOIRQ ); + +/* +* When we exit from the guest we change a number of CPU configuration +* parameters, such as traps. Make sure these changes take effect +* before running the host or additional guests. +*/ +isb (); + +return ret; +}/* Switch to the guest for legacy non-VHE systems */ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
virt/kvm/arm/arm.c +0 -2
@@ -797,9 +797,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)guest_enter_irqoff ();if (has_vhe ()) { -kvm_arm_vhe_guest_enter (); ret =kvm_vcpu_run_vhe (vcpu); -kvm_arm_vhe_guest_exit (); }else { ret =kvm_call_hyp_ret (__kvm_vcpu_run_nvhe, vcpu); }
Signature · 1 line
2.20.1
Marc Zyngier
maintainerPATCH 3/5 KVM: arm64: Ask the compiler to __always_inline functions used at HYP
In reply to Marc ZyngierFrom: James Morse <[email protected]>
On non VHE CPUs, KVM's __hyp_text contains code run at EL2 while the rest of the kernel runs at EL1. This code lives in its own section with start and end markers so we can map it to EL2.
The compiler may decide not to inline static-inline functions from the header file. It may also decide not to put these out-of-line functions in the same section, meaning they aren't mapped when called at EL2.
Clang-9 does exactly this with __kern_hyp_va() and a few others when x18 is reserved for the shadow call stack. Add the additional __always_ hint to all the static-inlines that are called from a hyp file.
----kvm_get_hyp_vector() pulls in all the regular per-cpu accessors and this_cpu_has_cap(), fortunately its only called for VHE.
--- arch/arm64/include/asm/arch_gicv3.h | 2 +- arch/arm64/include/asm/cpufeature.h | 2 +- arch/arm64/include/asm/kvm_emulate.h | 48 ++++++++++++++-------------- arch/arm64/include/asm/kvm_mmu.h | 3 +- arch/arm64/include/asm/virt.h | 2 +- 5 files changed, 29 insertions(+), 28 deletions(-)
arch/arm64/include/asm/arch_gicv3.h +1 -1
@@ -32,7 +32,7 @@ static inline void gic_write_eoir(u32 irq)isb (); } -static inline void gic_write_dir (u32 irq) +static __always_inline void gic_write_dir(u32 irq) {write_sysreg_s (irq,SYS_ICC_DIR_EL1 );isb ();
arch/arm64/include/asm/cpufeature.h +1 -1
@@ -581,7 +581,7 @@ static inline bool system_supports_sve(void)cpus_have_const_cap (ARM64_SVE ); } -static inline bool system_supports_cnp (void ) +static __always_inline bool system_supports_cnp(void) {return IS_ENABLED (CONFIG_ARM64_CNP ) &&cpus_have_const_cap (ARM64_HAS_CNP );
arch/arm64/include/asm/kvm_emulate.h +24 -24
172 lines not on this page · raw message
arch/arm64/include/asm/kvm_mmu.h +2 -1
@@ -93,7 +93,7 @@ void kvm_update_va_mask(struct alt_instr *alt, __le32 *origptr, __le32 *updptr, int nr_inst);void kvm_compute_layout (void ); -static inline unsigned long __kern_hyp_va (unsigned long v) +static __always_inline unsigned long __kern_hyp_va (unsigned long v) { asm volatile(ALTERNATIVE_CB("and %0, %0, #1\n" "ror %0, %0, #1\n"
@@ -473,6 +473,7 @@ static inline int kvm_write_guest_lock(struct kvm *kvm, gpa_t gpa,extern void *__kvm_bp_vect_base;extern int __kvm_harden_el2_vector_slot; +/* This is only called on a VHE system */ static inline void *kvm_get_hyp_vector (void ) {struct bp_hardening_data *data =arm64_get_bp_hardening_data ();
arch/arm64/include/asm/virt.h +1 -1
@@ -83,7 +83,7 @@ static inline bool is_kernel_in_hyp_mode(void)return read_sysreg (CurrentEL) == CurrentEL_EL2; } -static inline bool has_vhe (void ) +static __always_inline bool has_vhe(void) {if (cpus_have_const_cap (ARM64_HAS_VIRT_HOST_EXTN ))return true;
Signature · 1 line
2.20.1
Marc Zyngier
maintainerPATCH 4/5 KVM: arm64: Define our own swab32() to avoid a uapi static inline
In reply to Marc ZyngierFrom: James Morse <[email protected]>
KVM uses swab32() when mediating GIC MMIO accesses if the GICV is badly aligned, and the host and guest differ in endianness.
arm64 doesn't provide a __arch_swab32(), so __fswab32() is always backed by the macro implementation that the compiler reduces to a single instruction. But the static-inline causes problems for KVM if the compiler chooses not to inline this function, it may not be located in the __hyp_text where __vgic_v2_perform_cpuif_access() needs it.
Create our own __kvm_swab32() macro that calls ___constant_swab32() directly. This way we know it will always be inlined.
--- arch/arm64/include/asm/kvm_hyp.h | 7 +++++++ arch/arm64/kvm/hyp/vgic-v2-cpuif-proxy.c | 4 ++-- 2 files changed, 9 insertions(+), 2 deletions(-)
arch/arm64/include/asm/kvm_hyp.h +7 -0
@@ -47,6 +47,13 @@#define read_sysreg_el2 (r) read_sysreg_elx(r, _EL2, _EL1)#define write_sysreg_el2 (v,r) write_sysreg_elx(v, r, _EL2, _EL1) +/* +* Without an __arch_swab32(), we fall back to ___constant_swab32(), but the +* static inline can allow the compiler to out-of-line this. KVM always wants +* the macro version as its always inlined. +*/ +#define __kvm_swab32 (x) ___constant_swab32(x) +int __vgic_v2_perform_cpuif_access (struct kvm_vcpu *vcpu);void __vgic_v3_save_state (struct kvm_vcpu *vcpu);
arch/arm64/kvm/hyp/vgic-v2-cpuif-proxy.c +2 -2
@@ -69,14 +69,14 @@ int __hyp_text __vgic_v2_perform_cpuif_access(struct kvm_vcpu *vcpu)u32 data =vcpu_get_reg (vcpu, rd);if (__is_be (vcpu)) {/* guest pre-swabbed data, undo this for writel() */ - data =swab32 (data); + data =__kvm_swab32 (data); }writel_relaxed (data, addr); }else {u32 data =readl_relaxed (addr);if (__is_be (vcpu)) {/* guest expects swabbed data */ - data =swab32 (data); + data =__kvm_swab32 (data); }vcpu_set_reg (vcpu, rd, data); }
Signature · 1 line
2.20.1
Marc Zyngier
maintainerPATCH 5/5 arm64: Ask the compiler to __always_inline functions used by KVM at HYP
In reply to Marc ZyngierFrom: James Morse <[email protected]>
KVM uses some of the static-inline helpers like icache_is_vipt() from its HYP code. This assumes the function is inlined so that the code is mapped to EL2. The compiler may decide not to inline these, and the out-of-line version may not be in the __hyp_text section.
Add the additional __always_ hint to these static-inlines that are used by KVM.
--- arch/arm64/include/asm/cache.h | 2 +- arch/arm64/include/asm/cacheflush.h | 2 +- arch/arm64/include/asm/cpufeature.h | 8 ++++---- arch/arm64/include/asm/io.h | 4 ++-- 4 files changed, 8 insertions(+), 8 deletions(-)
arch/arm64/include/asm/cache.h +1 -1
@@ -69,7 +69,7 @@ static inline int icache_is_aliasing(void)return test_bit (ICACHEF_ALIASING , &__icache_flags); } -static inline int icache_is_vpipt (void ) +static __always_inline int icache_is_vpipt(void) {return test_bit (ICACHEF_VPIPT , &__icache_flags); }
arch/arm64/include/asm/cacheflush.h +1 -1
@@ -145,7 +145,7 @@ extern void copy_to_user_page(struct vm_area_struct *, struct page *,#define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1extern void flush_dcache_page (struct page *); -static inline void __flush_icache_all (void ) +static __always_inline void __flush_icache_all(void) {if (cpus_have_const_cap (ARM64_HAS_CACHE_DIC ))return ;
arch/arm64/include/asm/cpufeature.h +4 -4
@@ -435,13 +435,13 @@ cpuid_feature_extract_signed_field(u64 features, int field)return cpuid_feature_extract_signed_field_width (features, field,4 ); } -static inline unsigned int __attribute_const__ +static __always_inline unsigned int __attribute_const__cpuid_feature_extract_unsigned_field_width (u64 features,int field,int width) {return (u64)(features << (64 - width - field)) >> (64 - width); } -static inline unsigned int __attribute_const__ +static __always_inline unsigned int __attribute_const__cpuid_feature_extract_unsigned_field (u64 features,int field) {return cpuid_feature_extract_unsigned_field_width (features, field,4 );
@@ -564,7 +564,7 @@ static inline bool system_supports_mixed_endian(void)return val ==0x1 ; } -static inline bool system_supports_fpsimd (void ) +static __always_inline bool system_supports_fpsimd(void) {return !cpus_have_const_cap (ARM64_HAS_NO_FPSIMD ); }
@@ -575,7 +575,7 @@ static inline bool system_uses_ttbr0_pan(void) !cpus_have_const_cap (ARM64_HAS_PAN ); } -static inline bool system_supports_sve (void ) +static __always_inline bool system_supports_sve(void) {return IS_ENABLED (CONFIG_ARM64_SVE ) &&cpus_have_const_cap (ARM64_SVE );
arch/arm64/include/asm/io.h +2 -2
@@ -34,7 +34,7 @@ static inline void __raw_writew(u16 val, volatile void __iomem *addr) }#define __raw_writel __raw_writel -static inline void __raw_writel (u32 val,volatile void __iomem *addr) +static __always_inline void __raw_writel(u32 val, volatile void __iomem *addr) { asmvolatile ("str %w0, [%1]" : :"rZ" (val),"r" (addr)); }
@@ -69,7 +69,7 @@ static inline u16 __raw_readw(const volatile void __iomem *addr) }#define __raw_readl __raw_readl -static inline u32 __raw_readl (const volatile void __iomem *addr) +static __always_inline u32 __raw_readl(const volatile void __iomem *addr) {u32 val; asm volatile(ALTERNATIVE("ldr %w0, [%1]",
Signature · 1 line
2.20.1
Paolo Bonzini
maintainerQuoted 42 lines · Marc Zyngier
Paolo,
This is a small update containing a number of fixes, the most important ones making sure we force the inlining of any helper that gets used by the EL2 code (James identified that some bad things happen with CLang and the Shadow Call Stack extention).
Please pull,
M.The following changes since commit
4a267aa707953a9a73d1f5dc7f894dd9024a92be:KVM: arm64: Treat emulated TVAL TimerValue as a signed 32-bit integer (2020-01-28 13:09:31 +0000)
are available in the Git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.gittags/kvmarm-fixes-5.6-1for you to fetch changes up to
e43f1331e2ef913b8c566920c9af75e0ccdd1d3f:arm64: Ask the compiler to
__always_inlinefunctions used by KVM at HYP (2020-02-22 11:01:47 +0000)----------------------------------------------------------------
KVM/arm fixes for 5.6, take #1- Fix compilation on 32bit
- Move VHE guest entry/exit into the VHE-specific entry code
- Make sure all functions called by the non-VHE HYP code is tagged as__always_inline----------------------------------------------------------------
James Morse (3):
KVM: arm64: Ask the compiler to__always_inlinefunctions used at HYP
KVM: arm64: Define our ownswab32()to avoid a uapi static inline
arm64: Ask the compiler to__always_inlinefunctions used by KVM at HYPJeremy Cline (1):
KVM: arm/arm64: Fix up includes fortrace.hMark Rutland (1):
kvm: arm/arm64: Fold VHE entry/exit work intokvm_vcpu_run_vhe()
Quoted 17 lines
arch/arm/include/asm/kvm_host.h | 3 -- arch/arm64/include/asm/arch_gicv3.h | 2 +- arch/arm64/include/asm/cache.h | 2 +- arch/arm64/include/asm/cacheflush.h | 2 +- arch/arm64/include/asm/cpufeature.h | 10 +++---- arch/arm64/include/asm/io.h | 4 +-- arch/arm64/include/asm/kvm_emulate.h | 48 ++++++++++++++++---------------- arch/arm64/include/asm/kvm_host.h | 32 --------------------- arch/arm64/include/asm/kvm_hyp.h | 7 +++++ arch/arm64/include/asm/kvm_mmu.h | 3 +- arch/arm64/include/asm/virt.h | 2 +- arch/arm64/kvm/hyp/switch.c | 39 ++++++++++++++++++++++++-- arch/arm64/kvm/hyp/vgic-v2-cpuif-proxy.c | 4 +-- virt/kvm/arm/arm.c | 2 -- virt/kvm/arm/trace.h | 1 + 15 files changed, 84 insertions(+), 77 deletions(-)
Pulled, thanks.
Paolo