arm64: KVM: add SPE profiling support
Subsystems (7)
v2 cover arm64: KVM: add SPE profiling support
This series implements support for allowing KVM guests to use the Arm Statistical Profiling Extension (SPE).
It has been tested on a model to ensure that both host and guest can simultaneously use SPE with valid data. E.g.
$ perf record -e arm_spe/ts_enable=1,pa_enable=1,pct_enable=1/ \
dd if=/dev/zero of=/dev/null count=1000
$ perf report --dump-raw-trace > spe_buf.txtAs we save and restore the SPE context, the guest can access the SPE registers directly, thus in this version of the series we remove the trapping and emulation.
In the previous series of this support, when KVM SPE isn't supported (e.g. via CONFIG_KVM_ARM_SPE) we were able to return a value of 0 to all reads of the SPE registers - as we can no longer do this there isn't a mechanism to prevent the guest from using SPE - thus I'm keen for feedback on the best way of resolving this.
It appears necessary to pin the entire guest memory in order to provide guest SPE access - otherwise it is possible for the guest to receive Stage-2 faults.
The last two extra patches are for the kvmtool if someone wants to play with it.
Changes since v2:
- Rebased on v5.5-rc2
- Renamed kvm_spe structure 'irq' member to 'irq_num'
- Added irq_level to kvm_spe structure
- Clear PMBSR service bit on save to avoid spurious interrupts
- Update kvmtool headers to 5.4
- Enabled SPE in KVM init features
- No longer trap and emulate
- Add support for guest/host exclusion flags
- Fix virq support for SPE
- Adjusted sysreg_elx_s macros with merged clang build support
Andrew Murray (4):
KVM: arm64: don't trap Statistical Profiling controls to EL2
perf: arm_spe: Add KVM structure for obtaining IRQ info
KVM: arm64: spe: Provide guest virtual interrupts for SPE
perf: arm_spe: Handle guest/host exclusion flags
Sudeep Holla (12):
dt-bindings: ARM SPE: highlight the need for PPI partitions on
heterogeneous systems
arm64: KVM: reset E2PB correctly in MDCR_EL2 when exiting the
guest(VHE)
arm64: KVM: define SPE data structure for each vcpu
arm64: KVM: add SPE system registers to sys_reg_descs
arm64: KVM/VHE: enable the use PMSCR_EL12 on VHE systems
arm64: KVM: split debug save restore across vm/traps activation
arm64: KVM/debug: drop pmscr_el1 and use sys_regs[PMSCR_EL1] in
kvm_cpu_context
arm64: KVM: add support to save/restore SPE profiling buffer controls
arm64: KVM: enable conditional save/restore full SPE profiling buffer
controls
arm64: KVM/debug: use EL1&0 stage 1 translation regime
KVM: arm64: add a new vcpu device control group for SPEv1
KVM: arm64: enable SPE support
KVMTOOL: update_headers: Sync kvm UAPI headers with linux v5.5-rc2
KVMTOOL: kvm: add a vcpu feature for SPEv1 support
.../devicetree/bindings/arm/spe-pmu.txt | 5 +- Documentation/virt/kvm/devices/vcpu.txt | 28 +++ arch/arm64/include/asm/kvm_host.h | 18 +- arch/arm64/include/asm/kvm_hyp.h | 6 +- arch/arm64/include/asm/sysreg.h | 1 + arch/arm64/include/uapi/asm/kvm.h | 4 + arch/arm64/kvm/Kconfig | 7 + arch/arm64/kvm/Makefile | 1 + arch/arm64/kvm/debug.c | 2 - arch/arm64/kvm/guest.c | 6 + arch/arm64/kvm/hyp/debug-sr.c | 105 +++++--- arch/arm64/kvm/hyp/switch.c | 18 +- arch/arm64/kvm/reset.c | 3 + arch/arm64/kvm/sys_regs.c | 11 + drivers/perf/arm_spe_pmu.c | 26 ++ include/kvm/arm_spe.h | 82 ++++++ include/uapi/linux/kvm.h | 1 + virt/kvm/arm/arm.c | 10 +- virt/kvm/arm/spe.c | 234 ++++++++++++++++++ 19 files changed, 521 insertions(+), 47 deletions(-) create mode 100644 include/kvm/arm_spe.h create mode 100644 virt/kvm/arm/spe.c
Signature · 1 line
2.21.0
PATCH v2 1/18 dt-bindings: ARM SPE: highlight the need for PPI partitions on heterogeneous systems
From: Sudeep Holla <[email protected]>
It's not entirely clear for the binding document that the only way to express ARM SPE affined to a subset of CPUs on a heterogeneous systems is through the use of PPI partitions available in the interrupt controller bindings.
Let's make it clear.
--- Documentation/devicetree/bindings/arm/spe-pmu.txt | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
Documentation/devicetree/bindings/arm/spe-pmu.txt +3 -2
@@ -9,8 +9,9 @@ performance sample data using an in-memory trace buffer. "arm,statistical-profiling-extension-v1" - interrupts : Exactly 1 PPI must be listed. For heterogeneous systems where - SPE is only supported on a subset of the CPUs, please consult - the arm,gic-v3 binding for details on describing a PPI partition. + SPE is only supported on a subset of the CPUs, a PPI partition + described in the arm,gic-v3 binding must be used to describe + the set of CPUs this interrupt is affine to. ** Example:
Signature · 6 lines
2.21.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v2 2/18 arm64: KVM: reset E2PB correctly in MDCR_EL2 when exiting the guest(VHE)
In reply to Andrew MurrayFrom: Sudeep Holla <[email protected]>
On VHE systems, the reset value for MDCR_EL2.E2PB=b00 which defaults to profiling buffer using the EL2 stage 1 translations. However if the guest are allowed to use profiling buffers changing E2PB settings, we need to ensure we resume back MDCR_EL2.E2PB=b00. Currently we just do bitwise '&' with MDCR_EL2_E2PB_MASK which will retain the value.
So fix it by clearing all the bits in E2PB.
--- arch/arm64/kvm/hyp/switch.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
arch/arm64/kvm/hyp/switch.c +1 -3
@@ -228,9 +228,7 @@ void deactivate_traps_vhe_put(void) {u64 mdcr_el2 =read_sysreg (mdcr_el2); - mdcr_el2 &=MDCR_EL2_HPMN_MASK | -MDCR_EL2_E2PB_MASK <<MDCR_EL2_E2PB_SHIFT | -MDCR_EL2_TPMS ; + mdcr_el2 &=MDCR_EL2_HPMN_MASK |MDCR_EL2_TPMS ;write_sysreg (mdcr_el2, mdcr_el2);
Signature · 6 lines
2.21.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerRe: [PATCH v2 02/18] arm64: KVM: reset E2PB correctly in MDCR_EL2 when exiting the guest(VHE)
Quoted 4 lines · On Fri, 20 Dec 2019 14:30:09 +0000 Andrew Murray <[email protected]> wrote:
From: Sudeep Holla <[email protected]>
On VHE systems, the reset value for MDCR_EL2.E2PB=b00 which defaults to profiling buffer using the EL2 stage 1 translations.
Does the reset value actually matter here? I don't see it being specific to VHE systems, and all we're trying to achieve is to restore the SPE configuration to a state where it can be used by the host.
Quoted 2 lines
However if the
guest are allowed to use profiling buffers changing E2PB settings, we
How can the guest be allowed to change E2PB settings? Or do you mean here that allowing the guest to use SPE will mandate changes of the E2PB settings, and that we'd better restore the hypervisor state once we exit?
Quoted 16 lines
need to ensure we resume back MDCR_EL2.E2PB=b00. Currently we just do bitwise '&' with
MDCR_EL2_E2PB_MASKwhich will retain the value.So fix it by clearing all the bits in E2PB.
Signed-off-by Sudeep Holla <[email protected]>Signed-off-by Andrew Murray <[email protected]>--- arch/arm64/kvm/hyp/switch.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)arch/arm64/kvm/hyp/switch.c +0
-0
arch/arm64/kvm/hyp/switch.c void deactivate_traps_vhe_put(void) · line 234
Quoted 10 lines
arch/arm64/kvm/hyp/switch.c +1
-3@@ -228,9 +228,7 @@ void deactivate_traps_vhe_put(void) {u64 mdcr_el2 =read_sysreg (mdcr_el2); - mdcr_el2 &=MDCR_EL2_HPMN_MASK | -MDCR_EL2_E2PB_MASK <<MDCR_EL2_E2PB_SHIFT | -MDCR_EL2_TPMS ; + mdcr_el2 &=MDCR_EL2_HPMN_MASK |MDCR_EL2_TPMS ;write_sysreg (mdcr_el2, mdcr_el2);
I'm OK with this change, but I believe the commit message could use some tidying up.
Thanks,
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
Re: [PATCH v2 02/18] arm64: KVM: reset E2PB correctly in MDCR_EL2 when exiting the guest(VHE)
Quoted 44 lines · Marc Zyngier
Quoted 4 lines · On Fri, 20 Dec 2019 14:30:09 +0000 Andrew Murray <[email protected]> wrote:
From: Sudeep Holla <[email protected]>
On VHE systems, the reset value for MDCR_EL2.E2PB=b00 which defaults to profiling buffer using the EL2 stage 1 translations.
Does the reset value actually matter here? I don't see it being specific to VHE systems, and all we're trying to achieve is to restore the SPE configuration to a state where it can be used by the host.
Quoted 2 lines
However if the
guest are allowed to use profiling buffers changing E2PB settings, weHow can the guest be allowed to change E2PB settings? Or do you mean here that allowing the guest to use SPE will mandate changes of the E2PB settings, and that we'd better restore the hypervisor state once we exit?
Quoted 24 lines
need to ensure we resume back MDCR_EL2.E2PB=b00. Currently we just do bitwise '&' with
MDCR_EL2_E2PB_MASKwhich will retain the value.So fix it by clearing all the bits in E2PB.
Signed-off-by Sudeep Holla <[email protected]>Signed-off-by Andrew Murray <[email protected]>--- arch/arm64/kvm/hyp/switch.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)arch/arm64/kvm/hyp/switch.c +1
-3@@ -228,9 +228,7 @@ void deactivate_traps_vhe_put(void) {u64 mdcr_el2 =read_sysreg (mdcr_el2); - mdcr_el2 &=MDCR_EL2_HPMN_MASK | -MDCR_EL2_E2PB_MASK <<MDCR_EL2_E2PB_SHIFT | -MDCR_EL2_TPMS ; + mdcr_el2 &=MDCR_EL2_HPMN_MASK |MDCR_EL2_TPMS ;
Quoted 5 lines
arch/arm64/kvm/hyp/switch.c void deactivate_traps_vhe_put(void) · line 234
Quoted 2 lines
arch/arm64/kvm/hyp/switch.c +0
-0write_sysreg (mdcr_el2, mdcr_el2);I'm OK with this change, but I believe the commit message could use some tidying up.
No problem, I'll update the commit message.
Thanks,
Andrew Murray
Quoted 6 lines
Thanks,
M.Signature · 1 line
Jazz is not dead. It just smells funny...
Re: [PATCH v2 02/18] arm64: KVM: reset E2PB correctly in MDCR_EL2 when exiting the guest(VHE)
Quoted 45 lines · Andrew Murray
Quoted 44 lines · Marc Zyngier
Quoted 4 lines · On Fri, 20 Dec 2019 14:30:09 +0000 Andrew Murray <[email protected]> wrote:
From: Sudeep Holla <[email protected]>
On VHE systems, the reset value for MDCR_EL2.E2PB=b00 which defaults to profiling buffer using the EL2 stage 1 translations.
Does the reset value actually matter here? I don't see it being specific to VHE systems, and all we're trying to achieve is to restore the SPE configuration to a state where it can be used by the host.
Quoted 2 lines
However if the
guest are allowed to use profiling buffers changing E2PB settings, weHow can the guest be allowed to change E2PB settings? Or do you mean here that allowing the guest to use SPE will mandate changes of the E2PB settings, and that we'd better restore the hypervisor state once we exit?
Quoted 24 lines
need to ensure we resume back MDCR_EL2.E2PB=b00. Currently we just do bitwise '&' with
MDCR_EL2_E2PB_MASKwhich will retain the value.So fix it by clearing all the bits in E2PB.
Signed-off-by Sudeep Holla <[email protected]>Signed-off-by Andrew Murray <[email protected]>--- arch/arm64/kvm/hyp/switch.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)arch/arm64/kvm/hyp/switch.c +1
-3@@ -228,9 +228,7 @@ void deactivate_traps_vhe_put(void) {u64 mdcr_el2 =read_sysreg (mdcr_el2); - mdcr_el2 &=MDCR_EL2_HPMN_MASK | -MDCR_EL2_E2PB_MASK <<MDCR_EL2_E2PB_SHIFT | -MDCR_EL2_TPMS ; + mdcr_el2 &=MDCR_EL2_HPMN_MASK |MDCR_EL2_TPMS ;
Quoted 7 lines
Quoted 5 lines
arch/arm64/kvm/hyp/switch.c void deactivate_traps_vhe_put(void) · line 234
Quoted 2 lines
arch/arm64/kvm/hyp/switch.c +0
-0write_sysreg (mdcr_el2, mdcr_el2);I'm OK with this change, but I believe the commit message could use some tidying up.
No problem, I'll update the commit message.
This is my new description:
arm64: KVM: reset E2PB correctly in MDCR_EL2 when exiting the guest (VHE)
Upon leaving the guest on VHE systems we currently preserve the value of MDCR_EL2.E2PB. This register determines if the SPE profiling buffer controls are trapped and which translation regime they use.
In order to permit guest access to SPE we may use a different translation regime whilst the vCPU is scheduled - therefore let's ensure that upon leaving the guest we set E2PB back to the value expected by the host (b00).
For nVHE systems we already explictly set E2PB back to the expected value of 0b11 in __deactivate_traps_nvhe.
Thanks,
Andrew Murray
Quoted 15 lines
Thanks,
Andrew Murray
Quoted 6 lines
Thanks,
M.Signature · 1 line
Jazz is not dead. It just smells funny..._______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v2 3/18 arm64: KVM: define SPE data structure for each vcpu
In reply to Andrew MurrayFrom: Sudeep Holla <[email protected]>
In order to support virtual SPE for guest, so define some basic structs. This features depends on host having hardware with SPE support.
Since we can support this only on ARM64, add a separate config symbol for the same.
--- arch/arm64/include/asm/kvm_host.h | 2 ++ arch/arm64/kvm/Kconfig | 7 +++++++ include/kvm/arm_spe.h | 19 +++++++++++++++++++ 3 files changed, 28 insertions(+) create mode 100644 include/kvm/arm_spe.h
arch/arm64/include/asm/kvm_host.h +2 -0
@@ -35,6 +35,7 @@#include <kvm/arm_vgic.h> #include <kvm/arm_arch_timer.h> #include <kvm/arm_pmu.h> +#include <kvm/arm_spe.h> #define KVM_MAX_VCPUS VGIC_V3_MAX_CPUS
@@ -302,6 +303,7 @@ struct kvm_vcpu_arch {struct vgic_cpu vgic_cpu;struct arch_timer_cpu timer_cpu;struct kvm_pmu pmu; +struct kvm_spe spe;/* * Anything that is not used directly from assembly code goes
arch/arm64/kvm/Kconfig +7 -0
@@ -35,6 +35,7 @@ config KVM select HAVE_KVM_EVENTFD select HAVE_KVM_IRQFD select KVM_ARM_PMU if HW_PERF_EVENTS + select KVM_ARM_SPE if (HW_PERF_EVENTS && ARM_SPE_PMU) select HAVE_KVM_MSI select HAVE_KVM_IRQCHIP select HAVE_KVM_IRQ_ROUTING
@@ -61,6 +62,12 @@ config KVM_ARM_PMU Adds support for a virtual Performance Monitoring Unit (PMU) in virtual machines. +config KVM_ARM_SPE + bool + ---help--- + Adds support for a virtual Statistical Profiling Extension(SPE) in + virtual machines. + config KVM_INDIRECT_VECTORS def_bool KVM && (HARDEN_BRANCH_PREDICTOR || HARDEN_EL2_VECTORS)
include/kvm/arm_spe.h new file +19 -0
@@ -0,0 +1,19 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Copyright (C) 2019 ARM Ltd. +*/ + +#ifndef __ASM_ARM_KVM_SPE_H +#define __ASM_ARM_KVM_SPE_H + +#include <uapi/linux/kvm.h> +#include <linux/kvm_host.h> + +struct kvm_spe { +int irq_num; +bool ready;/* indicates that SPE KVM instance is ready for use */ +bool created;/* SPE KVM instance is created, may not be ready yet */ +bool irq_level; +}; + +#endif /* __ASM_ARM_KVM_SPE_H */
Signature · 6 lines
2.21.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerRe: [PATCH v2 03/18] arm64: KVM: define SPE data structure for each vcpu
Quoted 69 lines · On Fri, 20 Dec 2019 14:30:10 +0000 Andrew Murray <[email protected]> wrote:
From: Sudeep Holla <[email protected]>
In order to support virtual SPE for guest, so define some basic structs. This features depends on host having hardware with SPE support.
Since we can support this only on ARM64, add a separate config symbol for the same.
Signed-off-by Sudeep Holla <[email protected]>[ Add irq_level, rename irq to irq_num for kvm_spe ]Signed-off-by Andrew Murray <[email protected]>--- arch/arm64/include/asm/kvm_host.h | 2 ++ arch/arm64/kvm/Kconfig | 7 +++++++ include/kvm/arm_spe.h | 19 +++++++++++++++++++ 3 files changed, 28 insertions(+) create mode 100644 include/kvm/arm_spe.harch/arm64/include/asm/kvm_host.h +2
-0@@ -35,6 +35,7 @@#include <kvm/arm_vgic.h> #include <kvm/arm_arch_timer.h> #include <kvm/arm_pmu.h> +#include <kvm/arm_spe.h> #define KVM_MAX_VCPUS VGIC_V3_MAX_CPUS@@ -302,6 +303,7 @@ struct kvm_vcpu_arch {struct vgic_cpu vgic_cpu;struct arch_timer_cpu timer_cpu;struct kvm_pmu pmu; +struct kvm_spe spe;/* * Anything that is not used directly from assembly code goes arch/arm64/kvm/Kconfig +7
-0@@ -35,6 +35,7 @@ config KVM select HAVE_KVM_EVENTFD select HAVE_KVM_IRQFD select KVM_ARM_PMU if HW_PERF_EVENTS + select KVM_ARM_SPE if (HW_PERF_EVENTS && ARM_SPE_PMU) select HAVE_KVM_MSI select HAVE_KVM_IRQCHIP select HAVE_KVM_IRQ_ROUTING@@ -61,6 +62,12 @@ config KVM_ARM_PMU Adds support for a virtual Performance Monitoring Unit (PMU) in virtual machines. +config KVM_ARM_SPE + bool + ---help--- + Adds support for a virtual Statistical Profiling Extension(SPE) in + virtual machines. + config KVM_INDIRECT_VECTORS def_bool KVM && (HARDEN_BRANCH_PREDICTOR || HARDEN_EL2_VECTORS)include/kvm/arm_spe.h new file +0
-0
include/kvm/arm_spe.h line 10
Quoted 10 lines
include/kvm/arm_spe.h new file +10
-0@@ -0,0 +1,19 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Copyright (C) 2019 ARM Ltd. +*/ + +#ifndef __ASM_ARM_KVM_SPE_H +#define __ASM_ARM_KVM_SPE_H + +#include <uapi/linux/kvm.h> +#include <linux/kvm_host.h>
I don't believe these are required at this stage.
include/kvm/arm_spe.h line 13
Quoted 3 lines
include/kvm/arm_spe.h new file +3
-0+ +struct kvm_spe { +int irq_num;
'irq' was the right name *if* this represents a Linux irq. If this instead represents a guest PPI, then it should be named 'intid'.
In either case, please document what this represents.
include/kvm/arm_spe.h line 16
Quoted 3 lines
include/kvm/arm_spe.h new file +3
-0+bool ready;/* indicates that SPE KVM instance is ready for use */ +bool created;/* SPE KVM instance is created, may not be ready yet */ +bool irq_level;
What does this represent? The state of the interrupt on the host? The guest? Something else? Also, please consider grouping related fields together.
include/kvm/arm_spe.h line 17
Quoted 1 line
include/kvm/arm_spe.h new file +1
-0+};
If you've added a config option that controls the selection of the SPE feature, why doesn't this result in an empty structure when it isn't selected?
include/kvm/arm_spe.h line 19
Quoted 2 lines
include/kvm/arm_spe.h new file +2
-0+ +#endif /* __ASM_ARM_KVM_SPE_H */
Thanks,
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
Re: [PATCH v2 03/18] arm64: KVM: define SPE data structure for each vcpu
Quoted 94 lines · Marc Zyngier
Quoted 69 lines · On Fri, 20 Dec 2019 14:30:10 +0000 Andrew Murray <[email protected]> wrote:
From: Sudeep Holla <[email protected]>
In order to support virtual SPE for guest, so define some basic structs. This features depends on host having hardware with SPE support.
Since we can support this only on ARM64, add a separate config symbol for the same.
Signed-off-by Sudeep Holla <[email protected]>[ Add irq_level, rename irq to irq_num for kvm_spe ]Signed-off-by Andrew Murray <[email protected]>--- arch/arm64/include/asm/kvm_host.h | 2 ++ arch/arm64/kvm/Kconfig | 7 +++++++ include/kvm/arm_spe.h | 19 +++++++++++++++++++ 3 files changed, 28 insertions(+) create mode 100644 include/kvm/arm_spe.harch/arm64/include/asm/kvm_host.h +2
-0@@ -35,6 +35,7 @@#include <kvm/arm_vgic.h> #include <kvm/arm_arch_timer.h> #include <kvm/arm_pmu.h> +#include <kvm/arm_spe.h> #define KVM_MAX_VCPUS VGIC_V3_MAX_CPUS@@ -302,6 +303,7 @@ struct kvm_vcpu_arch {struct vgic_cpu vgic_cpu;struct arch_timer_cpu timer_cpu;struct kvm_pmu pmu; +struct kvm_spe spe;/* * Anything that is not used directly from assembly code goes arch/arm64/kvm/Kconfig +7
-0@@ -35,6 +35,7 @@ config KVM select HAVE_KVM_EVENTFD select HAVE_KVM_IRQFD select KVM_ARM_PMU if HW_PERF_EVENTS + select KVM_ARM_SPE if (HW_PERF_EVENTS && ARM_SPE_PMU) select HAVE_KVM_MSI select HAVE_KVM_IRQCHIP select HAVE_KVM_IRQ_ROUTING@@ -61,6 +62,12 @@ config KVM_ARM_PMU Adds support for a virtual Performance Monitoring Unit (PMU) in virtual machines. +config KVM_ARM_SPE + bool + ---help--- + Adds support for a virtual Statistical Profiling Extension(SPE) in + virtual machines. + config KVM_INDIRECT_VECTORS def_bool KVM && (HARDEN_BRANCH_PREDICTOR || HARDEN_EL2_VECTORS)include/kvm/arm_spe.h new file +0
-0include/kvm/arm_spe.h line 10
Quoted 10 lines
include/kvm/arm_spe.h new file +10
-0@@ -0,0 +1,19 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Copyright (C) 2019 ARM Ltd. +*/ + +#ifndef __ASM_ARM_KVM_SPE_H +#define __ASM_ARM_KVM_SPE_H + +#include <uapi/linux/kvm.h> +#include <linux/kvm_host.h> I don't believe these are required at this stage.
include/kvm/arm_spe.h line 13
Quoted 3 lines
include/kvm/arm_spe.h new file +3
-0+ +struct kvm_spe { +int irq_num;'irq' was the right name *if* this represents a Linux irq. If this instead represents a guest PPI, then it should be named 'intid'.
In either case, please document what this represents.
Quoted 7 lines
include/kvm/arm_spe.h line 16
Quoted 3 lines
include/kvm/arm_spe.h new file +3
-0+bool ready;/* indicates that SPE KVM instance is ready for use */ +bool created;/* SPE KVM instance is created, may not be ready yet */ +bool irq_level;What does this represent? The state of the interrupt on the host? The guest? Something else? Also, please consider grouping related fields together.
It should be the state of the interrupt on the guest.
Quoted 6 lines
include/kvm/arm_spe.h line 17
Quoted 2 lines
include/kvm/arm_spe.h new file +1
-0+};If you've added a config option that controls the selection of the SPE feature, why doesn't this result in an empty structure when it isn't selected?
OK, all noted.
Andrew Murray
Quoted 9 lines
include/kvm/arm_spe.h line 19
Quoted 3 lines
include/kvm/arm_spe.h new file +2
-0+ +#endif /* __ASM_ARM_KVM_SPE_H */ Thanks,
M.Signature · 1 line
Jazz is not dead. It just smells funny...
PATCH v2 4/18 arm64: KVM: add SPE system registers to sys_reg_descs
In reply to Andrew MurrayFrom: Sudeep Holla <[email protected]>
Add the Statistical Profiling Extension(SPE) Profiling Buffer controls registers such that we can provide initial register values and use the sys_regs structure as a store for our SPE context.
--- arch/arm64/include/asm/kvm_host.h | 12 ++++++++++++ arch/arm64/kvm/sys_regs.c | 11 +++++++++++ 2 files changed, 23 insertions(+)
arch/arm64/include/asm/kvm_host.h +12 -0
@@ -145,6 +145,18 @@ enum vcpu_sysreg {MDCCINT_EL1 ,/* Monitor Debug Comms Channel Interrupt Enable Reg */ DISR_EL1 ,/* Deferred Interrupt Status Register */ +/* Statistical Profiling Extension Registers */ +PMSCR_EL1 , +PMSICR_EL1 , +PMSIRR_EL1 , +PMSFCR_EL1 , +PMSEVFR_EL1 , +PMSLATFR_EL1 , +PMSIDR_EL1 , +PMBLIMITR_EL1 , +PMBPTR_EL1 , +PMBSR_EL1 , +/* Performance Monitors Registers */ PMCR_EL0 ,/* Control Register */ PMSELR_EL0, /* Event Counter Selection Register */
arch/arm64/kvm/sys_regs.c +11 -0
@@ -1506,6 +1506,17 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_FAR_EL1), access_vm_reg, reset_unknown, FAR_EL1 }, { SYS_DESC(SYS_PAR_EL1), NULL, reset_unknown, PAR_EL1 }, + { SYS_DESC(SYS_PMSCR_EL1), NULL, reset_val, PMSCR_EL1, 0 }, + { SYS_DESC(SYS_PMSICR_EL1), NULL, reset_val, PMSICR_EL1, 0 }, + { SYS_DESC(SYS_PMSIRR_EL1), NULL, reset_val, PMSIRR_EL1, 0 }, + { SYS_DESC(SYS_PMSFCR_EL1), NULL, reset_val, PMSFCR_EL1, 0 }, + { SYS_DESC(SYS_PMSEVFR_EL1), NULL, reset_val, PMSEVFR_EL1, 0 }, + { SYS_DESC(SYS_PMSLATFR_EL1), NULL, reset_val, PMSLATFR_EL1, 0 }, + { SYS_DESC(SYS_PMSIDR_EL1), NULL, reset_val, PMSIDR_EL1, 0 }, + { SYS_DESC(SYS_PMBLIMITR_EL1), NULL, reset_val, PMBLIMITR_EL1, 0 }, + { SYS_DESC(SYS_PMBPTR_EL1), NULL, reset_val, PMBPTR_EL1, 0 }, + { SYS_DESC(SYS_PMBSR_EL1), NULL, reset_val, PMBSR_EL1, 0 }, + { SYS_DESC(SYS_PMINTENSET_EL1), access_pminten, reset_unknown, PMINTENSET_EL1 }, { SYS_DESC(SYS_PMINTENCLR_EL1), access_pminten, NULL, PMINTENSET_EL1 },
Signature · 6 lines
2.21.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v2 5/18 arm64: KVM/VHE: enable the use PMSCR_EL12 on VHE systems
In reply to Andrew MurrayFrom: Sudeep Holla <[email protected]>
Currently, we are just using PMSCR_EL1 in the host for non VHE systems. We already have the {read,write}_sysreg_el*() accessors for accessing particular ELs' sysregs in the presence of VHE.
Lets just define PMSCR_EL12 and start making use of it here which will access the right register on both VHE and non VHE systems. This change is required to add SPE guest support on VHE systems.
--- arch/arm64/include/asm/sysreg.h | 1 + arch/arm64/kvm/hyp/debug-sr.c | 6 +++--- 2 files changed, 4 insertions(+), 3 deletions(-)
arch/arm64/include/asm/sysreg.h +1 -0
@@ -468,6 +468,7 @@#define SYS_AFSR1_EL12 sys_reg(3, 5, 5, 1, 1)#define SYS_ESR_EL12 sys_reg(3, 5, 5, 2, 0)#define SYS_FAR_EL12 sys_reg(3, 5, 6, 0, 0) +#define SYS_PMSCR_EL12 sys_reg(3, 5, 9, 9, 0)#define SYS_MAIR_EL12 sys_reg(3, 5, 10, 2, 0)#define SYS_AMAIR_EL12 sys_reg(3, 5, 10, 3, 0)#define SYS_VBAR_EL12 sys_reg(3, 5, 12, 0, 0)
arch/arm64/kvm/hyp/debug-sr.c +3 -3
@@ -108,8 +108,8 @@ static void __hyp_text __debug_save_spe_nvhe(u64 *pmscr_el1)return ;/* Yes; save the control register and disable data generation */ - *pmscr_el1 =read_sysreg_s (SYS_PMSCR_EL1 ); -write_sysreg_s (0 ,SYS_PMSCR_EL1 ); + *pmscr_el1 =read_sysreg_el1 (SYS_PMSCR ); +write_sysreg_el1 (0 ,SYS_PMSCR );isb ();/* Now drain all buffered data to memory */
@@ -126,7 +126,7 @@ static void __hyp_text __debug_restore_spe_nvhe(u64 pmscr_el1)isb ();/* Re-enable data generation */ -write_sysreg_s (pmscr_el1,SYS_PMSCR_EL1 ); +write_sysreg_el1 (pmscr_el1,SYS_PMSCR ); } static void __hyp_text __debug_save_state(struct kvm_vcpu *vcpu,
Signature · 1 line
2.21.0
PATCH v2 6/18 arm64: KVM: split debug save restore across vm/traps activation
In reply to Andrew MurrayFrom: Sudeep Holla <[email protected]>
If we enable profiling buffer controls at EL1 generate a trap exception to EL2, it also changes profiling buffer to use EL1&0 stage 1 translation regime in case of VHE. To support SPE both in the guest and host, we need to first stop profiling and flush the profiling buffers before we activate/switch vm or enable/disable the traps.
In prepartion to do that, lets split the debug save restore functionality into 4 steps:
1. debug_save_host_context - saves the host context
2. debug_restore_guest_context - restore the guest context
3. debug_save_guest_context - saves the guest context
4. debug_restore_host_context - restores the host context
Lets rename existing __debug_switch_to_{host,guest} to make sure it's aligned to the above and just add the place holders for new ones getting added here as we need them to support SPE in guests.
--- arch/arm64/include/asm/kvm_hyp.h | 6 ++++-- arch/arm64/kvm/hyp/debug-sr.c | 25 ++++++++++++++++--------- arch/arm64/kvm/hyp/switch.c | 12 ++++++++---- 3 files changed, 28 insertions(+), 15 deletions(-)
arch/arm64/include/asm/kvm_hyp.h +4 -2
@@ -69,8 +69,10 @@ void sysreg_restore_guest_state_vhe(struct kvm_cpu_context *ctxt);void __sysreg32_save_state (struct kvm_vcpu *vcpu);void __sysreg32_restore_state (struct kvm_vcpu *vcpu); -void __debug_switch_to_guest (struct kvm_vcpu *vcpu); -void __debug_switch_to_host (struct kvm_vcpu *vcpu); +void __debug_save_host_context (struct kvm_vcpu *vcpu); +void __debug_restore_guest_context (struct kvm_vcpu *vcpu); +void __debug_save_guest_context (struct kvm_vcpu *vcpu); +void __debug_restore_host_context (struct kvm_vcpu *vcpu);void __fpsimd_save_state (struct user_fpsimd_state *fp_regs);void __fpsimd_restore_state (struct user_fpsimd_state *fp_regs);
arch/arm64/kvm/hyp/debug-sr.c +16 -9
@@ -168,20 +168,13 @@ static void __hyp_text __debug_restore_state(struct kvm_vcpu *vcpu,write_sysreg (ctxt->sys_regs[MDCCINT_EL1 ], mdccint_el1); } -void __hyp_text __debug_switch_to_guest(struct kvm_vcpu *vcpu) +void __hyp_text __debug_restore_guest_context(struct kvm_vcpu *vcpu) {struct kvm_cpu_context *host_ctxt;struct kvm_cpu_context *guest_ctxt;struct kvm_guest_debug_arch *host_dbg;struct kvm_guest_debug_arch *guest_dbg; -/* -* Non-VHE: Disable and flush SPE data generation -* VHE: The vcpu can run, but it can't hide. -*/ -if (!has_vhe ()) -__debug_save_spe_nvhe (&vcpu->arch.host_debug_state.pmscr_el1); -if (!(vcpu->arch.flags &KVM_ARM64_DEBUG_DIRTY ))return ;
@@ -194,7 +187,7 @@ void __hyp_text __debug_switch_to_guest(struct kvm_vcpu *vcpu)__debug_restore_state (vcpu, guest_dbg, guest_ctxt); } -void __hyp_text __debug_switch_to_host(struct kvm_vcpu *vcpu) +void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) {struct kvm_cpu_context *host_ctxt;struct kvm_cpu_context *guest_ctxt;
@@ -218,6 +211,20 @@ void __hyp_text __debug_switch_to_host(struct kvm_vcpu *vcpu) vcpu->arch.flags &= ~KVM_ARM64_DEBUG_DIRTY ; } +void __hyp_text __debug_save_host_context(struct kvm_vcpu *vcpu) +{ +/* +* Non-VHE: Disable and flush SPE data generation +* VHE: The vcpu can run, but it can't hide. +*/ +if (!has_vhe ()) +__debug_save_spe_nvhe (&vcpu->arch.host_debug_state.pmscr_el1); +} + +void __hyp_text __debug_save_guest_context(struct kvm_vcpu *vcpu) +{ +} + u32 __hyp_text __kvm_get_mdcr_el2(void) {return read_sysreg (mdcr_el2);
arch/arm64/kvm/hyp/switch.c +8 -4
@@ -626,6 +626,7 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) guest_ctxt = &vcpu->arch.ctxt;sysreg_save_host_state_vhe (host_ctxt); +__debug_save_host_context (vcpu);/* * ARM erratum 1165522 requires us to configure both stage 1 and
@@ -642,7 +643,7 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)__activate_traps (vcpu);sysreg_restore_guest_state_vhe (guest_ctxt); -__debug_switch_to_guest (vcpu); +__debug_restore_guest_context (vcpu);__set_guest_arch_workaround_state (vcpu);
@@ -656,6 +657,7 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)__set_host_arch_workaround_state (vcpu);sysreg_save_guest_state_vhe (guest_ctxt); +__debug_save_guest_context (vcpu);__deactivate_traps (vcpu);
@@ -664,7 +666,7 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)if (vcpu->arch.flags &KVM_ARM64_FP_ENABLED )__fpsimd_save_fpexc32 (vcpu); -__debug_switch_to_host (vcpu); +__debug_restore_host_context (vcpu);return exit_code; }
@@ -698,6 +700,7 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu) pmu_switch_needed =__pmu_switch_to_guest (host_ctxt);__sysreg_save_state_nvhe (host_ctxt); +__debug_save_host_context (vcpu);/* * We must restore the 32-bit state before the sysregs, thanks
@@ -716,7 +719,7 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)__hyp_vgic_restore_state (vcpu);__timer_enable_traps (vcpu); -__debug_switch_to_guest (vcpu); +__debug_restore_guest_context (vcpu);__set_guest_arch_workaround_state (vcpu);
@@ -730,6 +733,7 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)__set_host_arch_workaround_state (vcpu);__sysreg_save_state_nvhe (guest_ctxt); +__debug_save_guest_context (vcpu);__sysreg32_save_state (vcpu);__timer_disable_traps (vcpu);__hyp_vgic_save_state (vcpu);
@@ -746,7 +750,7 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)* This must come after restoring the host sysregs, since a non-VHE * system may enable SPE here and make use of the TTBRs. */ -__debug_switch_to_host (vcpu); +__debug_restore_host_context (vcpu);if (pmu_switch_needed)__pmu_switch_to_host (host_ctxt);
Signature · 1 line
2.21.0
PATCH v2 7/18 arm64: KVM/debug: drop pmscr_el1 and use sys_regs[PMSCR_EL1] in kvm_cpu_context
In reply to Andrew MurrayFrom: Sudeep Holla <[email protected]>
kvm_cpu_context now has support to stash the complete SPE buffer control context. We no longer need the pmscr_el1 kvm_vcpu_arch and it can be dropped.
--- arch/arm64/include/asm/kvm_host.h | 2 -- arch/arm64/kvm/hyp/debug-sr.c | 26 +++++++++++++++----------- 2 files changed, 15 insertions(+), 13 deletions(-)
arch/arm64/include/asm/kvm_host.h +0 -2
@@ -307,8 +307,6 @@ struct kvm_vcpu_arch {struct {/* {Break,watch}point registers */ struct kvm_guest_debug_arch regs; -/* Statistical profiling extension */ -u64 pmscr_el1; } host_debug_state;/* VGIC state */
arch/arm64/kvm/hyp/debug-sr.c +15 -11
@@ -85,19 +85,19 @@default :write_debug (ptr[0 ], reg,0 ); \ } -static void __hyp_text __debug_save_spe_nvhe(u64 *pmscr_el1) +static void __hyp_text __debug_save_spe_nvhe(struct kvm_cpu_context *ctxt) {u64 reg;/* Clear pmscr in case of early return */ - *pmscr_el1 =0 ; + ctxt->sys_regs[PMSCR_EL1 ] =0 ;/* SPE present on this CPU? */ if (!cpuid_feature_extract_unsigned_field (read_sysreg (id_aa64dfr0_el1),ID_AA64DFR0_PMSVER_SHIFT ))return ; -/* Yes; is it owned by EL3? */ +/* Yes; is it owned by higher EL? */ reg =read_sysreg_s (SYS_PMBIDR_EL1 );if (reg &BIT (SYS_PMBIDR_EL1_P_SHIFT ))return ;
@@ -108,7 +108,7 @@ static void __hyp_text __debug_save_spe_nvhe(u64 *pmscr_el1)return ;/* Yes; save the control register and disable data generation */ - *pmscr_el1 =read_sysreg_el1 (SYS_PMSCR ); + ctxt->sys_regs[PMSCR_EL1 ] =read_sysreg_el1 (SYS_PMSCR );write_sysreg_el1 (0 ,SYS_PMSCR );isb ();
@@ -117,16 +117,16 @@ static void __hyp_text __debug_save_spe_nvhe(u64 *pmscr_el1)dsb (nsh); } -static void __hyp_text __debug_restore_spe_nvhe(u64 pmscr_el1) +static void __hyp_text __debug_restore_spe_nvhe(struct kvm_cpu_context *ctxt) { -if (!pmscr_el1) +if (!ctxt->sys_regs[PMSCR_EL1 ])return ;/* The host page table is installed, but not yet synchronised */ isb ();/* Re-enable data generation */ -write_sysreg_el1 (pmscr_el1,SYS_PMSCR ); +write_sysreg_el1 (ctxt->sys_regs[PMSCR_EL1 ],SYS_PMSCR ); } static void __hyp_text __debug_save_state(struct kvm_vcpu *vcpu,
@@ -194,14 +194,15 @@ void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu)struct kvm_guest_debug_arch *host_dbg;struct kvm_guest_debug_arch *guest_dbg; + host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); + guest_ctxt = &vcpu->arch.ctxt; +if (!has_vhe ()) -__debug_restore_spe_nvhe (vcpu->arch.host_debug_state.pmscr_el1); +__debug_restore_spe_nvhe (host_ctxt);if (!(vcpu->arch.flags &KVM_ARM64_DEBUG_DIRTY ))return ; - host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); - guest_ctxt = &vcpu->arch.ctxt; host_dbg = &vcpu->arch.host_debug_state.regs; guest_dbg =kern_hyp_va (vcpu->arch.debug_ptr);
@@ -217,8 +218,11 @@ void __hyp_text __debug_save_host_context(struct kvm_vcpu *vcpu)* Non-VHE: Disable and flush SPE data generation * VHE: The vcpu can run, but it can't hide. */ +struct kvm_cpu_context *host_ctxt; + + host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context);if (!has_vhe ()) -__debug_save_spe_nvhe (&vcpu->arch.host_debug_state.pmscr_el1); +__debug_save_spe_nvhe (host_ctxt); } void __hyp_text __debug_save_guest_context(struct kvm_vcpu *vcpu)
Signature · 6 lines
2.21.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v2 8/18 arm64: KVM: add support to save/restore SPE profiling buffer controls
In reply to Andrew MurrayFrom: Sudeep Holla <[email protected]>
Currently since we don't support profiling using SPE in the guests, we just save the PMSCR_EL1, flush the profiling buffers and disable sampling. However in order to support simultaneous sampling both in the host and guests, we need to save and reatore the complete SPE profiling buffer controls' context.
Let's add the support for the same and keep it disabled for now. We can enable it conditionally only if guests are allowed to use SPE.
--- arch/arm64/kvm/hyp/debug-sr.c | 51 +++++++++++++++++++++++++++++------ 1 file changed, 43 insertions(+), 8 deletions(-)
arch/arm64/kvm/hyp/debug-sr.c +43 -8
@@ -85,7 +85,8 @@default :write_debug (ptr[0 ], reg,0 ); \ } -static void __hyp_text __debug_save_spe_nvhe(struct kvm_cpu_context *ctxt) +static void __hyp_text +__debug_save_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt) {u64 reg;
@@ -102,22 +103,46 @@ static void __hyp_text __debug_save_spe_nvhe(struct kvm_cpu_context *ctxt)if (reg &BIT (SYS_PMBIDR_EL1_P_SHIFT ))return ; -/* No; is the host actually using the thing? */ - reg =read_sysreg_s (SYS_PMBLIMITR_EL1 ); -if (!(reg &BIT (SYS_PMBLIMITR_EL1_E_SHIFT ))) +/* Save the control register and disable data generation */ + ctxt->sys_regs[PMSCR_EL1 ] =read_sysreg_el1 (SYS_PMSCR ); + +if (!ctxt->sys_regs[PMSCR_EL1 ])return ;/* Yes; save the control register and disable data generation */ - ctxt->sys_regs[PMSCR_EL1 ] =read_sysreg_el1 (SYS_PMSCR );write_sysreg_el1 (0 ,SYS_PMSCR );isb ();/* Now drain all buffered data to memory */ psb_csync ();dsb (nsh); + +if (!full_ctxt) +return ; + + ctxt->sys_regs[PMBLIMITR_EL1 ] =read_sysreg_s (SYS_PMBLIMITR_EL1 ); +write_sysreg_s (0 ,SYS_PMBLIMITR_EL1 ); + +/* +* As PMBSR is conditionally restored when returning to the host we +* must ensure the service bit is unset here to prevent a spurious +* host SPE interrupt from being raised. +*/ + ctxt->sys_regs[PMBSR_EL1 ] =read_sysreg_s (SYS_PMBSR_EL1 ); +write_sysreg_s (0 ,SYS_PMBSR_EL1 ); + +isb (); + + ctxt->sys_regs[PMSICR_EL1 ] =read_sysreg_s (SYS_PMSICR_EL1 ); + ctxt->sys_regs[PMSIRR_EL1 ] =read_sysreg_s (SYS_PMSIRR_EL1 ); + ctxt->sys_regs[PMSFCR_EL1 ] =read_sysreg_s (SYS_PMSFCR_EL1 ); + ctxt->sys_regs[PMSEVFR_EL1 ] =read_sysreg_s (SYS_PMSEVFR_EL1 ); + ctxt->sys_regs[PMSLATFR_EL1 ] =read_sysreg_s (SYS_PMSLATFR_EL1 ); + ctxt->sys_regs[PMBPTR_EL1 ] =read_sysreg_s (SYS_PMBPTR_EL1 ); } -static void __hyp_text __debug_restore_spe_nvhe(struct kvm_cpu_context *ctxt) +static void __hyp_text +__debug_restore_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt) {if (!ctxt->sys_regs[PMSCR_EL1 ])return ;
@@ -126,6 +151,16 @@ static void __hyp_text __debug_restore_spe_nvhe(struct kvm_cpu_context *ctxt)isb ();/* Re-enable data generation */ +if (full_ctxt) { +write_sysreg_s (ctxt->sys_regs[PMBPTR_EL1 ],SYS_PMBPTR_EL1 ); +write_sysreg_s (ctxt->sys_regs[PMBLIMITR_EL1 ],SYS_PMBLIMITR_EL1 ); +write_sysreg_s (ctxt->sys_regs[PMSFCR_EL1 ],SYS_PMSFCR_EL1 ); +write_sysreg_s (ctxt->sys_regs[PMSEVFR_EL1 ],SYS_PMSEVFR_EL1 ); +write_sysreg_s (ctxt->sys_regs[PMSLATFR_EL1 ],SYS_PMSLATFR_EL1 ); +write_sysreg_s (ctxt->sys_regs[PMSIRR_EL1 ],SYS_PMSIRR_EL1 ); +write_sysreg_s (ctxt->sys_regs[PMSICR_EL1 ],SYS_PMSICR_EL1 ); +write_sysreg_s (ctxt->sys_regs[PMBSR_EL1 ],SYS_PMBSR_EL1 ); + }write_sysreg_el1 (ctxt->sys_regs[PMSCR_EL1 ],SYS_PMSCR ); }
@@ -198,7 +233,7 @@ void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) guest_ctxt = &vcpu->arch.ctxt;if (!has_vhe ()) -__debug_restore_spe_nvhe (host_ctxt); +__debug_restore_spe_nvhe (host_ctxt, false);if (!(vcpu->arch.flags &KVM_ARM64_DEBUG_DIRTY ))return ;
@@ -222,7 +257,7 @@ void __hyp_text __debug_save_host_context(struct kvm_vcpu *vcpu) host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context);if (!has_vhe ()) -__debug_save_spe_nvhe (host_ctxt); +__debug_save_spe_nvhe (host_ctxt, false); } void __hyp_text __debug_save_guest_context(struct kvm_vcpu *vcpu)
Signature · 6 lines
2.21.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerRe: [PATCH v2 08/18] arm64: KVM: add support to save/restore SPE profiling buffer controls
Quoted 5 lines · On Fri, 20 Dec 2019 14:30:15 +0000 Andrew Murray <[email protected]> wrote:
From: Sudeep Holla <[email protected]>
Currently since we don't support profiling using SPE in the guests, we just save the
PMSCR_EL1, flush the profiling buffers and disable sampling. However in order to support simultaneous sampling both in
Is the sampling actually simultaneous? I don't believe so (the whole series would be much simpler if it was).
Quoted 1 line
the host and guests, we need to save and reatore the complete SPE
s/reatore/restore/
Quoted 18 lines
profiling buffer controls' context.
Let's add the support for the same and keep it disabled for now. We can enable it conditionally only if guests are allowed to use SPE.
Signed-off-by Sudeep Holla <[email protected]>[ Clear PMBSR bit when saving state to prevent spurious interrupts ]Signed-off-by Andrew Murray <[email protected]>--- arch/arm64/kvm/hyp/debug-sr.c | 51 +++++++++++++++++++++++++++++------ 1 file changed, 43 insertions(+), 8 deletions(-)arch/arm64/kvm/hyp/debug-sr.c +0
-0
arch/arm64/kvm/hyp/debug-sr.c line 89
Quoted 6 lines
arch/arm64/kvm/hyp/debug-sr.c +2
-1@@ -85,7 +85,8 @@default :write_debug (ptr[0 ], reg,0 ); \ } -static void __hyp_text __debug_save_spe_nvhe(struct kvm_cpu_context *ctxt) +static void __hyp_text +__debug_save_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt)
nit: don't split lines like this if you can avoid it. You can put the full_ctxt parameter on a separate line instead.
Quoted 4 lines
arch/arm64/kvm/hyp/debug-sr.c +0
-0{u64 reg;
arch/arm64/kvm/hyp/debug-sr.c static void __hyp_text __debug_save_spe_nvhe(struct kvm_cpu_context *ctxt) · line 109
Quoted 10 lines
arch/arm64/kvm/hyp/debug-sr.c +4
-3@@ -102,22 +103,46 @@ static void __hyp_text __debug_save_spe_nvhe(struct kvm_cpu_context *ctxt)if (reg &BIT (SYS_PMBIDR_EL1_P_SHIFT ))return ; -/* No; is the host actually using the thing? */ - reg =read_sysreg_s (SYS_PMBLIMITR_EL1 ); -if (!(reg &BIT (SYS_PMBLIMITR_EL1_E_SHIFT ))) +/* Save the control register and disable data generation */ + ctxt->sys_regs[PMSCR_EL1 ] =read_sysreg_el1 (SYS_PMSCR ); + +if (!ctxt->sys_regs[PMSCR_EL1 ])
Shouldn't you check the enable bits instead of relying on the whole thing being zero?
arch/arm64/kvm/hyp/debug-sr.c static void __hyp_text __debug_save_spe_nvhe(struct kvm_cpu_context *ctxt) · line 111
Quoted 4 lines
arch/arm64/kvm/hyp/debug-sr.c +0
-1return ;/* Yes; save the control register and disable data generation */ - ctxt->sys_regs[PMSCR_EL1 ] =read_sysreg_el1 (SYS_PMSCR );
You've already saved the control register...
Quoted 64 lines
arch/arm64/kvm/hyp/debug-sr.c +36
-2write_sysreg_el1 (0 ,SYS_PMSCR );isb ();/* Now drain all buffered data to memory */ psb_csync ();dsb (nsh); + +if (!full_ctxt) +return ; + + ctxt->sys_regs[PMBLIMITR_EL1 ] =read_sysreg_s (SYS_PMBLIMITR_EL1 ); +write_sysreg_s (0 ,SYS_PMBLIMITR_EL1 ); + +/* +* As PMBSR is conditionally restored when returning to the host we +* must ensure the service bit is unset here to prevent a spurious +* host SPE interrupt from being raised. +*/ + ctxt->sys_regs[PMBSR_EL1 ] =read_sysreg_s (SYS_PMBSR_EL1 ); +write_sysreg_s (0 ,SYS_PMBSR_EL1 ); + +isb (); + + ctxt->sys_regs[PMSICR_EL1 ] =read_sysreg_s (SYS_PMSICR_EL1 ); + ctxt->sys_regs[PMSIRR_EL1 ] =read_sysreg_s (SYS_PMSIRR_EL1 ); + ctxt->sys_regs[PMSFCR_EL1 ] =read_sysreg_s (SYS_PMSFCR_EL1 ); + ctxt->sys_regs[PMSEVFR_EL1 ] =read_sysreg_s (SYS_PMSEVFR_EL1 ); + ctxt->sys_regs[PMSLATFR_EL1 ] =read_sysreg_s (SYS_PMSLATFR_EL1 ); + ctxt->sys_regs[PMBPTR_EL1 ] =read_sysreg_s (SYS_PMBPTR_EL1 ); } -static void __hyp_text __debug_restore_spe_nvhe(struct kvm_cpu_context *ctxt) +static void __hyp_text +__debug_restore_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt) {if (!ctxt->sys_regs[PMSCR_EL1 ])return ;@@ -126,6 +151,16 @@ static void __hyp_text __debug_restore_spe_nvhe(struct kvm_cpu_context *ctxt)isb ();/* Re-enable data generation */ +if (full_ctxt) { +write_sysreg_s (ctxt->sys_regs[PMBPTR_EL1 ],SYS_PMBPTR_EL1 ); +write_sysreg_s (ctxt->sys_regs[PMBLIMITR_EL1 ],SYS_PMBLIMITR_EL1 ); +write_sysreg_s (ctxt->sys_regs[PMSFCR_EL1 ],SYS_PMSFCR_EL1 ); +write_sysreg_s (ctxt->sys_regs[PMSEVFR_EL1 ],SYS_PMSEVFR_EL1 ); +write_sysreg_s (ctxt->sys_regs[PMSLATFR_EL1 ],SYS_PMSLATFR_EL1 ); +write_sysreg_s (ctxt->sys_regs[PMSIRR_EL1 ],SYS_PMSIRR_EL1 ); +write_sysreg_s (ctxt->sys_regs[PMSICR_EL1 ],SYS_PMSICR_EL1 ); +write_sysreg_s (ctxt->sys_regs[PMBSR_EL1 ],SYS_PMBSR_EL1 ); + }write_sysreg_el1 (ctxt->sys_regs[PMSCR_EL1 ],SYS_PMSCR ); }@@ -198,7 +233,7 @@ void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) guest_ctxt = &vcpu->arch.ctxt;if (!has_vhe ()) -__debug_restore_spe_nvhe (host_ctxt); +__debug_restore_spe_nvhe (host_ctxt, false);if (!(vcpu->arch.flags &KVM_ARM64_DEBUG_DIRTY ))return ;
arch/arm64/kvm/hyp/debug-sr.c void __hyp_text __debug_save_host_context(struct kvm_vcpu *vcpu) · line 263
Quoted 8 lines
arch/arm64/kvm/hyp/debug-sr.c +1
-1@@ -222,7 +257,7 @@ void __hyp_text __debug_save_host_context(struct kvm_vcpu *vcpu) host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context);if (!has_vhe ()) -__debug_save_spe_nvhe (host_ctxt); +__debug_save_spe_nvhe (host_ctxt, false); } void __hyp_text __debug_save_guest_context(struct kvm_vcpu *vcpu)
So all of this is for non-VHE. What happens in the VHE case?
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
Re: [PATCH v2 08/18] arm64: KVM: add support to save/restore SPE profiling buffer controls
Quoted 11 lines · Marc Zyngier
Quoted 5 lines · On Fri, 20 Dec 2019 14:30:15 +0000 Andrew Murray <[email protected]> wrote:
From: Sudeep Holla <[email protected]>
Currently since we don't support profiling using SPE in the guests, we just save the
PMSCR_EL1, flush the profiling buffers and disable sampling. However in order to support simultaneous sampling both inIs the sampling actually simultaneous? I don't believe so (the whole series would be much simpler if it was).
No the SPE is used by either the guest or host at any one time. I guess the term simultaneous was used to refer to illusion given to both guest and host that they are able to use it whenever they like. I'll update the commit message to drop the magic.
Quoted 4 lines
Quoted 2 lines
the host and guests, we need to save and reatore the complete SPE
s/reatore/restore/
Noted.
Quoted 22 lines
Quoted 22 lines
profiling buffer controls' context.
Let's add the support for the same and keep it disabled for now. We can enable it conditionally only if guests are allowed to use SPE.
Signed-off-by Sudeep Holla <[email protected]>[ Clear PMBSR bit when saving state to prevent spurious interrupts ]Signed-off-by Andrew Murray <[email protected]>--- arch/arm64/kvm/hyp/debug-sr.c | 51 +++++++++++++++++++++++++++++------ 1 file changed, 43 insertions(+), 8 deletions(-)arch/arm64/kvm/hyp/debug-sr.c +0
-0@@ -85,7 +85,8 @@default :write_debug (ptr[0 ], reg,0 ); \ }
Quoted 6 lines
arch/arm64/kvm/hyp/debug-sr.c line 89
Quoted 3 lines
arch/arm64/kvm/hyp/debug-sr.c +2
-1-static void __hyp_text __debug_save_spe_nvhe(struct kvm_cpu_context *ctxt) +static void __hyp_text +__debug_save_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt)nit: don't split lines like this if you can avoid it. You can put the
full_ctxtparameter on a separate line instead.
Yes understood.
Quoted 8 lines
Quoted 8 lines
arch/arm64/kvm/hyp/debug-sr.c +0
-0{u64 reg;@@ -102,22 +103,46 @@ static void __hyp_text __debug_save_spe_nvhe(struct kvm_cpu_context *ctxt)if (reg &BIT (SYS_PMBIDR_EL1_P_SHIFT ))return ;
Quoted 10 lines
arch/arm64/kvm/hyp/debug-sr.c static void __hyp_text __debug_save_spe_nvhe(struct kvm_cpu_context *ctxt) · line 109
Quoted 7 lines
arch/arm64/kvm/hyp/debug-sr.c +4
-3-/* No; is the host actually using the thing? */ - reg =read_sysreg_s (SYS_PMBLIMITR_EL1 ); -if (!(reg &BIT (SYS_PMBLIMITR_EL1_E_SHIFT ))) +/* Save the control register and disable data generation */ + ctxt->sys_regs[PMSCR_EL1 ] =read_sysreg_el1 (SYS_PMSCR ); + +if (!ctxt->sys_regs[PMSCR_EL1 ])Shouldn't you check the enable bits instead of relying on the whole thing being zero?
Yes that would make more sense (E1SPE and E0SPE).
I feel that this check makes an assumption about the guest/host SPE driver... What happens if the SPE driver writes to some SPE registers but doesn't enable PMSCR? If the guest is also using SPE then those writes will be lost, when the host returns and the SPE driver enables SPE it won't work.
With a quick look at the SPE driver I'm not sure this will happen, but even so it makes me nervous relying on these assumptions. I wonder if this risk is present in other devices?
Quoted 7 lines
arch/arm64/kvm/hyp/debug-sr.c static void __hyp_text __debug_save_spe_nvhe(struct kvm_cpu_context *ctxt) · line 111
Quoted 5 lines
arch/arm64/kvm/hyp/debug-sr.c +0
-1return ;/* Yes; save the control register and disable data generation */ - ctxt->sys_regs[PMSCR_EL1 ] =read_sysreg_el1 (SYS_PMSCR );You've already saved the control register...
I'll remove that.
Quoted 72 lines
Quoted 72 lines
arch/arm64/kvm/hyp/debug-sr.c +37
-3write_sysreg_el1 (0 ,SYS_PMSCR );isb ();/* Now drain all buffered data to memory */ psb_csync ();dsb (nsh); + +if (!full_ctxt) +return ; + + ctxt->sys_regs[PMBLIMITR_EL1 ] =read_sysreg_s (SYS_PMBLIMITR_EL1 ); +write_sysreg_s (0 ,SYS_PMBLIMITR_EL1 ); + +/* +* As PMBSR is conditionally restored when returning to the host we +* must ensure the service bit is unset here to prevent a spurious +* host SPE interrupt from being raised. +*/ + ctxt->sys_regs[PMBSR_EL1 ] =read_sysreg_s (SYS_PMBSR_EL1 ); +write_sysreg_s (0 ,SYS_PMBSR_EL1 ); + +isb (); + + ctxt->sys_regs[PMSICR_EL1 ] =read_sysreg_s (SYS_PMSICR_EL1 ); + ctxt->sys_regs[PMSIRR_EL1 ] =read_sysreg_s (SYS_PMSIRR_EL1 ); + ctxt->sys_regs[PMSFCR_EL1 ] =read_sysreg_s (SYS_PMSFCR_EL1 ); + ctxt->sys_regs[PMSEVFR_EL1 ] =read_sysreg_s (SYS_PMSEVFR_EL1 ); + ctxt->sys_regs[PMSLATFR_EL1 ] =read_sysreg_s (SYS_PMSLATFR_EL1 ); + ctxt->sys_regs[PMBPTR_EL1 ] =read_sysreg_s (SYS_PMBPTR_EL1 ); } -static void __hyp_text __debug_restore_spe_nvhe(struct kvm_cpu_context *ctxt) +static void __hyp_text +__debug_restore_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt) {if (!ctxt->sys_regs[PMSCR_EL1 ])return ;@@ -126,6 +151,16 @@ static void __hyp_text __debug_restore_spe_nvhe(struct kvm_cpu_context *ctxt)isb ();/* Re-enable data generation */ +if (full_ctxt) { +write_sysreg_s (ctxt->sys_regs[PMBPTR_EL1 ],SYS_PMBPTR_EL1 ); +write_sysreg_s (ctxt->sys_regs[PMBLIMITR_EL1 ],SYS_PMBLIMITR_EL1 ); +write_sysreg_s (ctxt->sys_regs[PMSFCR_EL1 ],SYS_PMSFCR_EL1 ); +write_sysreg_s (ctxt->sys_regs[PMSEVFR_EL1 ],SYS_PMSEVFR_EL1 ); +write_sysreg_s (ctxt->sys_regs[PMSLATFR_EL1 ],SYS_PMSLATFR_EL1 ); +write_sysreg_s (ctxt->sys_regs[PMSIRR_EL1 ],SYS_PMSIRR_EL1 ); +write_sysreg_s (ctxt->sys_regs[PMSICR_EL1 ],SYS_PMSICR_EL1 ); +write_sysreg_s (ctxt->sys_regs[PMBSR_EL1 ],SYS_PMBSR_EL1 ); + }write_sysreg_el1 (ctxt->sys_regs[PMSCR_EL1 ],SYS_PMSCR ); }@@ -198,7 +233,7 @@ void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) guest_ctxt = &vcpu->arch.ctxt;if (!has_vhe ()) -__debug_restore_spe_nvhe (host_ctxt); +__debug_restore_spe_nvhe (host_ctxt, false);if (!(vcpu->arch.flags &KVM_ARM64_DEBUG_DIRTY ))return ;@@ -222,7 +257,7 @@ void __hyp_text __debug_save_host_context(struct kvm_vcpu *vcpu) host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context);if (!has_vhe ()) -__debug_save_spe_nvhe (host_ctxt); +__debug_save_spe_nvhe (host_ctxt, false); }
Quoted 3 lines
arch/arm64/kvm/hyp/debug-sr.c void __hyp_text __debug_save_host_context(struct kvm_vcpu *vcpu) · line 263
Quoted 1 line
arch/arm64/kvm/hyp/debug-sr.c +0
-0void __hyp_text __debug_save_guest_context(struct kvm_vcpu *vcpu)So all of this is for non-VHE. What happens in the VHE case?
By the end of the series this ends up in __debug_save_host_context which is called for both VHE/nVHE - on the re-spin I'll make it not look so confusing.
Thanks,
Andrew Murray
Quoted 4 lines
M.Signature · 1 line
Jazz is not dead. It just smells funny...
Re: [PATCH v2 08/18] arm64: KVM: add support to save/restore SPE profiling buffer controls
Quoted 86 lines · Andrew Murray
Quoted 11 lines · Marc Zyngier
Quoted 5 lines · On Fri, 20 Dec 2019 14:30:15 +0000 Andrew Murray <[email protected]> wrote:
From: Sudeep Holla <[email protected]>
Currently since we don't support profiling using SPE in the guests, we just save the
PMSCR_EL1, flush the profiling buffers and disable sampling. However in order to support simultaneous sampling both inIs the sampling actually simultaneous? I don't believe so (the whole series would be much simpler if it was).
No the SPE is used by either the guest or host at any one time. I guess the term simultaneous was used to refer to illusion given to both guest and host that they are able to use it whenever they like. I'll update the commit message to drop the magic.
Quoted 4 lines
Quoted 2 lines
the host and guests, we need to save and reatore the complete SPE
s/reatore/restore/
Noted.
Quoted 22 lines
Quoted 22 lines
profiling buffer controls' context.
Let's add the support for the same and keep it disabled for now. We can enable it conditionally only if guests are allowed to use SPE.
Signed-off-by Sudeep Holla <[email protected]>[ Clear PMBSR bit when saving state to prevent spurious interrupts ]Signed-off-by Andrew Murray <[email protected]>--- arch/arm64/kvm/hyp/debug-sr.c | 51 +++++++++++++++++++++++++++++------ 1 file changed, 43 insertions(+), 8 deletions(-)arch/arm64/kvm/hyp/debug-sr.c +0
-0@@ -85,7 +85,8 @@default :write_debug (ptr[0 ], reg,0 ); \ }Quoted 6 lines
arch/arm64/kvm/hyp/debug-sr.c line 89
Quoted 3 lines
arch/arm64/kvm/hyp/debug-sr.c +2
-1-static void __hyp_text __debug_save_spe_nvhe(struct kvm_cpu_context *ctxt) +static void __hyp_text +__debug_save_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt)nit: don't split lines like this if you can avoid it. You can put the
full_ctxtparameter on a separate line instead.Yes understood.
Quoted 8 lines
Quoted 8 lines
arch/arm64/kvm/hyp/debug-sr.c +0
-0{u64 reg;@@ -102,22 +103,46 @@ static void __hyp_text __debug_save_spe_nvhe(struct kvm_cpu_context *ctxt)if (reg &BIT (SYS_PMBIDR_EL1_P_SHIFT ))return ;Quoted 10 lines
arch/arm64/kvm/hyp/debug-sr.c static void __hyp_text __debug_save_spe_nvhe(struct kvm_cpu_context *ctxt) · line 109
Quoted 7 lines
arch/arm64/kvm/hyp/debug-sr.c +4
-3-/* No; is the host actually using the thing? */ - reg =read_sysreg_s (SYS_PMBLIMITR_EL1 ); -if (!(reg &BIT (SYS_PMBLIMITR_EL1_E_SHIFT ))) +/* Save the control register and disable data generation */ + ctxt->sys_regs[PMSCR_EL1 ] =read_sysreg_el1 (SYS_PMSCR ); + +if (!ctxt->sys_regs[PMSCR_EL1 ])Shouldn't you check the enable bits instead of relying on the whole thing being zero?
Yes that would make more sense (E1SPE and E0SPE).
I feel that this check makes an assumption about the guest/host SPE driver... What happens if the SPE driver writes to some SPE registers but doesn't enable PMSCR? If the guest is also using SPE then those writes will be lost, when the host returns and the SPE driver enables SPE it won't work.
Quoted 3 lines
With a quick look at the SPE driver I'm not sure this will happen, but even so it makes me nervous relying on these assumptions. I wonder if this risk is present in other devices?
In fact, this may be a good reason to trap the SPE registers - this would allow you to conditionally save/restore based on a dirty bit. It would also allow you to re-evaluate the SPE interrupt (for example when the guest clears the status register) and thus potentially reduce any black hole.
Thanks,
Andrew Murray
Quoted 104 lines
Quoted 9 lines
arch/arm64/kvm/hyp/debug-sr.c static void __hyp_text __debug_save_spe_nvhe(struct kvm_cpu_context *ctxt) · line 111
Quoted 7 lines
arch/arm64/kvm/hyp/debug-sr.c +0
-1return ;/* Yes; save the control register and disable data generation */ - ctxt->sys_regs[PMSCR_EL1 ] =read_sysreg_el1 (SYS_PMSCR );You've already saved the control register...
I'll remove that.
Quoted 72 lines
Quoted 72 lines
arch/arm64/kvm/hyp/debug-sr.c +37
-3write_sysreg_el1 (0 ,SYS_PMSCR );isb ();/* Now drain all buffered data to memory */ psb_csync ();dsb (nsh); + +if (!full_ctxt) +return ; + + ctxt->sys_regs[PMBLIMITR_EL1 ] =read_sysreg_s (SYS_PMBLIMITR_EL1 ); +write_sysreg_s (0 ,SYS_PMBLIMITR_EL1 ); + +/* +* As PMBSR is conditionally restored when returning to the host we +* must ensure the service bit is unset here to prevent a spurious +* host SPE interrupt from being raised. +*/ + ctxt->sys_regs[PMBSR_EL1 ] =read_sysreg_s (SYS_PMBSR_EL1 ); +write_sysreg_s (0 ,SYS_PMBSR_EL1 ); + +isb (); + + ctxt->sys_regs[PMSICR_EL1 ] =read_sysreg_s (SYS_PMSICR_EL1 ); + ctxt->sys_regs[PMSIRR_EL1 ] =read_sysreg_s (SYS_PMSIRR_EL1 ); + ctxt->sys_regs[PMSFCR_EL1 ] =read_sysreg_s (SYS_PMSFCR_EL1 ); + ctxt->sys_regs[PMSEVFR_EL1 ] =read_sysreg_s (SYS_PMSEVFR_EL1 ); + ctxt->sys_regs[PMSLATFR_EL1 ] =read_sysreg_s (SYS_PMSLATFR_EL1 ); + ctxt->sys_regs[PMBPTR_EL1 ] =read_sysreg_s (SYS_PMBPTR_EL1 ); } -static void __hyp_text __debug_restore_spe_nvhe(struct kvm_cpu_context *ctxt) +static void __hyp_text +__debug_restore_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt) {if (!ctxt->sys_regs[PMSCR_EL1 ])return ;@@ -126,6 +151,16 @@ static void __hyp_text __debug_restore_spe_nvhe(struct kvm_cpu_context *ctxt)isb ();/* Re-enable data generation */ +if (full_ctxt) { +write_sysreg_s (ctxt->sys_regs[PMBPTR_EL1 ],SYS_PMBPTR_EL1 ); +write_sysreg_s (ctxt->sys_regs[PMBLIMITR_EL1 ],SYS_PMBLIMITR_EL1 ); +write_sysreg_s (ctxt->sys_regs[PMSFCR_EL1 ],SYS_PMSFCR_EL1 ); +write_sysreg_s (ctxt->sys_regs[PMSEVFR_EL1 ],SYS_PMSEVFR_EL1 ); +write_sysreg_s (ctxt->sys_regs[PMSLATFR_EL1 ],SYS_PMSLATFR_EL1 ); +write_sysreg_s (ctxt->sys_regs[PMSIRR_EL1 ],SYS_PMSIRR_EL1 ); +write_sysreg_s (ctxt->sys_regs[PMSICR_EL1 ],SYS_PMSICR_EL1 ); +write_sysreg_s (ctxt->sys_regs[PMBSR_EL1 ],SYS_PMBSR_EL1 ); + }write_sysreg_el1 (ctxt->sys_regs[PMSCR_EL1 ],SYS_PMSCR ); }@@ -198,7 +233,7 @@ void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) guest_ctxt = &vcpu->arch.ctxt;if (!has_vhe ()) -__debug_restore_spe_nvhe (host_ctxt); +__debug_restore_spe_nvhe (host_ctxt, false);if (!(vcpu->arch.flags &KVM_ARM64_DEBUG_DIRTY ))return ;@@ -222,7 +257,7 @@ void __hyp_text __debug_save_host_context(struct kvm_vcpu *vcpu) host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context);if (!has_vhe ()) -__debug_save_spe_nvhe (host_ctxt); +__debug_save_spe_nvhe (host_ctxt, false); }Quoted 3 lines
arch/arm64/kvm/hyp/debug-sr.c void __hyp_text __debug_save_host_context(struct kvm_vcpu *vcpu) · line 263
Quoted 1 line
arch/arm64/kvm/hyp/debug-sr.c +0
-0void __hyp_text __debug_save_guest_context(struct kvm_vcpu *vcpu)So all of this is for non-VHE. What happens in the VHE case?
By the end of the series this ends up in
__debug_save_host_contextwhich is called for both VHE/nVHE - on the re-spin I'll make it not look so confusing.Thanks,
Andrew Murray
Quoted 4 lines
M.Signature · 1 line
Jazz is not dead. It just smells funny..._______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerRe: [PATCH v2 08/18] arm64: KVM: add support to save/restore SPE profiling buffer controls
Quoted 88 lines · Andrew Murray
Quoted 86 lines · Andrew Murray
Quoted 11 lines · Marc Zyngier
Quoted 5 lines · On Fri, 20 Dec 2019 14:30:15 +0000 Andrew Murray <[email protected]> wrote:
From: Sudeep Holla <[email protected]>
Currently since we don't support profiling using SPE in the guests, we just save the
PMSCR_EL1, flush the profiling buffers and disable sampling. However in order to support simultaneous sampling both inIs the sampling actually simultaneous? I don't believe so (the whole series would be much simpler if it was).
No the SPE is used by either the guest or host at any one time. I guess the term simultaneous was used to refer to illusion given to both guest and host that they are able to use it whenever they like. I'll update the commit message to drop the magic.
Quoted 4 lines
Quoted 2 lines
the host and guests, we need to save and reatore the complete SPE
s/reatore/restore/
Noted.
Quoted 22 lines
Quoted 22 lines
profiling buffer controls' context.
Let's add the support for the same and keep it disabled for now. We can enable it conditionally only if guests are allowed to use SPE.
Signed-off-by Sudeep Holla <[email protected]>[ Clear PMBSR bit when saving state to prevent spurious interrupts ]Signed-off-by Andrew Murray <[email protected]>--- arch/arm64/kvm/hyp/debug-sr.c | 51 +++++++++++++++++++++++++++++------ 1 file changed, 43 insertions(+), 8 deletions(-)arch/arm64/kvm/hyp/debug-sr.c +0
-0@@ -85,7 +85,8 @@default :write_debug (ptr[0 ], reg,0 ); \ }Quoted 6 lines
arch/arm64/kvm/hyp/debug-sr.c line 89
Quoted 3 lines
arch/arm64/kvm/hyp/debug-sr.c +2
-1-static void __hyp_text __debug_save_spe_nvhe(struct kvm_cpu_context *ctxt) +static void __hyp_text +__debug_save_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt)nit: don't split lines like this if you can avoid it. You can put the
full_ctxtparameter on a separate line instead.Yes understood.
Quoted 8 lines
Quoted 8 lines
arch/arm64/kvm/hyp/debug-sr.c +0
-0{u64 reg;@@ -102,22 +103,46 @@ static void __hyp_text __debug_save_spe_nvhe(struct kvm_cpu_context *ctxt)if (reg &BIT (SYS_PMBIDR_EL1_P_SHIFT ))return ;Quoted 10 lines
arch/arm64/kvm/hyp/debug-sr.c static void __hyp_text __debug_save_spe_nvhe(struct kvm_cpu_context *ctxt) · line 109
Quoted 7 lines
arch/arm64/kvm/hyp/debug-sr.c +4
-3-/* No; is the host actually using the thing? */ - reg =read_sysreg_s (SYS_PMBLIMITR_EL1 ); -if (!(reg &BIT (SYS_PMBLIMITR_EL1_E_SHIFT ))) +/* Save the control register and disable data generation */ + ctxt->sys_regs[PMSCR_EL1 ] =read_sysreg_el1 (SYS_PMSCR ); + +if (!ctxt->sys_regs[PMSCR_EL1 ])Shouldn't you check the enable bits instead of relying on the whole thing being zero?
Yes that would make more sense (E1SPE and E0SPE).
I feel that this check makes an assumption about the guest/host SPE driver... What happens if the SPE driver writes to some SPE registers but doesn't enable PMSCR? If the guest is also using SPE then those writes will be lost, when the host returns and the SPE driver enables SPE it won't work.
Quoted 3 lines
Quoted 3 lines
With a quick look at the SPE driver I'm not sure this will happen, but even so it makes me nervous relying on these assumptions. I wonder if this risk is present in other devices?
As a rule of thumb, you should always save whatever you're about to overwrite if the registers are not under exclusive control of KVM. No exception.
So if the guest is willing to use SPE *and* that it isn't enabled on the host, these registers have to be saved on vcpu_load() and restored on vcpu_put().
If SPE is enabled on the host, then trapping has to be enabled, and no tracing occurs in the guest for this time slice.
Quoted 4 lines
In fact, this may be a good reason to trap the SPE registers - this would allow you to conditionally save/restore based on a dirty bit. It would also allow you to re-evaluate the SPE interrupt (for example when the guest clears the status register) and thus potentially reduce any black hole.
I don't see what trapping buys you in the expected case (where the guest is tracing and the host isn't). To clear PMBSR_EL1.S, you first need to know that an interrupt has fired. So this brings you exactly nothing in this particular case, and just adds overhead for everything else. The whole point of the architecture is that in the non-contended case, we can give SPE to the guest and mostly forget about it.
I strongly suggest that you start with the simplest possible, non broken implementation. It doesn't matter if the black holes last for seconds for now. Once you have something that looks reasonable, we can evaluate how to improve on it by throwing actual HW and workloads at it.
M.
Signature · 1 line
Jazz is not dead, it just smells funny.
PATCH v2 9/18 arm64: KVM: enable conditional save/restore full SPE profiling buffer controls
In reply to Andrew MurrayFrom: Sudeep Holla <[email protected]>
Now that we can save/restore the full SPE controls, we can enable it if SPE is setup and ready to use in KVM. It's supported in KVM only if all the CPUs in the system supports SPE.
However to support heterogenous systems, we need to move the check if host supports SPE and do a partial save/restore.
--- arch/arm64/kvm/hyp/debug-sr.c | 33 ++++++++++++++++----------------- include/kvm/arm_spe.h | 6 ++++++ 2 files changed, 22 insertions(+), 17 deletions(-)
arch/arm64/kvm/hyp/debug-sr.c +16 -17
@@ -86,18 +86,13 @@ } static void __hyp_text -__debug_save_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt) +__debug_save_spe_context (struct kvm_cpu_context *ctxt,bool full_ctxt) {u64 reg;/* Clear pmscr in case of early return */ ctxt->sys_regs[PMSCR_EL1 ] =0 ; -/* SPE present on this CPU? */ -if (!cpuid_feature_extract_unsigned_field (read_sysreg (id_aa64dfr0_el1), -ID_AA64DFR0_PMSVER_SHIFT )) -return ; -/* Yes; is it owned by higher EL? */ reg =read_sysreg_s (SYS_PMBIDR_EL1 );if (reg &BIT (SYS_PMBIDR_EL1_P_SHIFT ))
@@ -142,7 +137,7 @@ __debug_save_spe_nvhe(struct kvm_cpu_context *ctxt, bool full_ctxt) } static void __hyp_text -__debug_restore_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt) +__debug_restore_spe_context (struct kvm_cpu_context *ctxt,bool full_ctxt) {if (!ctxt->sys_regs[PMSCR_EL1 ])return ;
@@ -210,11 +205,14 @@ void __hyp_text __debug_restore_guest_context(struct kvm_vcpu *vcpu)struct kvm_guest_debug_arch *host_dbg;struct kvm_guest_debug_arch *guest_dbg; + host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); + guest_ctxt = &vcpu->arch.ctxt; + +__debug_restore_spe_context (guest_ctxt,kvm_arm_spe_v1_ready (vcpu)); +if (!(vcpu->arch.flags &KVM_ARM64_DEBUG_DIRTY ))return ; - host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); - guest_ctxt = &vcpu->arch.ctxt; host_dbg = &vcpu->arch.host_debug_state.regs; guest_dbg =kern_hyp_va (vcpu->arch.debug_ptr);
@@ -232,8 +230,7 @@ void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); guest_ctxt = &vcpu->arch.ctxt; -if (!has_vhe ()) -__debug_restore_spe_nvhe (host_ctxt, false); +__debug_restore_spe_context (host_ctxt,kvm_arm_spe_v1_ready (vcpu));if (!(vcpu->arch.flags &KVM_ARM64_DEBUG_DIRTY ))return ;
@@ -249,19 +246,21 @@ void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) void __hyp_text __debug_save_host_context(struct kvm_vcpu *vcpu) { -/* -* Non-VHE: Disable and flush SPE data generation -* VHE: The vcpu can run, but it can't hide. -*/ struct kvm_cpu_context *host_ctxt; host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); -if (!has_vhe ()) -__debug_save_spe_nvhe (host_ctxt, false); +if (cpuid_feature_extract_unsigned_field (read_sysreg (id_aa64dfr0_el1), +ID_AA64DFR0_PMSVER_SHIFT )) +__debug_save_spe_context (host_ctxt,kvm_arm_spe_v1_ready (vcpu)); } void __hyp_text __debug_save_guest_context(struct kvm_vcpu *vcpu) { +bool kvm_spe_ready =kvm_arm_spe_v1_ready (vcpu); + +/* SPE present on this vCPU? */ +if (kvm_spe_ready) +__debug_save_spe_context (&vcpu->arch.ctxt, kvm_spe_ready); } u32 __hyp_text __kvm_get_mdcr_el2(void)
include/kvm/arm_spe.h +6 -0
@@ -16,4 +16,10 @@ struct kvm_spe { bool irq_level; }; +#ifdef CONFIG_KVM_ARM_SPE +#define kvm_arm_spe_v1_ready (v) ((v)->arch.spe.ready) +#else +#define kvm_arm_spe_v1_ready (v) (false) +#endif /* CONFIG_KVM_ARM_SPE */ +#endif /* __ASM_ARM_KVM_SPE_H */
Signature · 6 lines
2.21.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Mark Rutland
maintainerRe: [PATCH v2 09/18] arm64: KVM: enable conditional save/restore full SPE profiling buffer controls
Quoted 8 lines · Andrew Murray
From: Sudeep Holla <[email protected]>
Now that we can save/restore the full SPE controls, we can enable it if SPE is setup and ready to use in KVM. It's supported in KVM only if all the CPUs in the system supports SPE.
However to support heterogenous systems, we need to move the check if host supports SPE and do a partial save/restore.
I don't think that it makes sense to support this for heterogeneous systems, given their SPE capabilities and IMP DEF details will differ.
Is there some way we can limit this to homogeneous systems?
Thanks,
Mark.
Quoted 114 lines
Signed-off-by Sudeep Holla <[email protected]>Signed-off-by Andrew Murray <[email protected]>--- arch/arm64/kvm/hyp/debug-sr.c | 33 ++++++++++++++++----------------- include/kvm/arm_spe.h | 6 ++++++ 2 files changed, 22 insertions(+), 17 deletions(-)arch/arm64/kvm/hyp/debug-sr.c +16
-17@@ -86,18 +86,13 @@ } static void __hyp_text -__debug_save_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt) +__debug_save_spe_context (struct kvm_cpu_context *ctxt,bool full_ctxt) {u64 reg;/* Clear pmscr in case of early return */ ctxt->sys_regs[PMSCR_EL1 ] =0 ; -/* SPE present on this CPU? */ -if (!cpuid_feature_extract_unsigned_field (read_sysreg (id_aa64dfr0_el1), -ID_AA64DFR0_PMSVER_SHIFT )) -return ; -/* Yes; is it owned by higher EL? */ reg =read_sysreg_s (SYS_PMBIDR_EL1 );if (reg &BIT (SYS_PMBIDR_EL1_P_SHIFT ))@@ -142,7 +137,7 @@ __debug_save_spe_nvhe(struct kvm_cpu_context *ctxt, bool full_ctxt) } static void __hyp_text -__debug_restore_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt) +__debug_restore_spe_context (struct kvm_cpu_context *ctxt,bool full_ctxt) {if (!ctxt->sys_regs[PMSCR_EL1 ])return ;@@ -210,11 +205,14 @@ void __hyp_text __debug_restore_guest_context(struct kvm_vcpu *vcpu)struct kvm_guest_debug_arch *host_dbg;struct kvm_guest_debug_arch *guest_dbg; + host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); + guest_ctxt = &vcpu->arch.ctxt; + +__debug_restore_spe_context (guest_ctxt,kvm_arm_spe_v1_ready (vcpu)); +if (!(vcpu->arch.flags &KVM_ARM64_DEBUG_DIRTY ))return ; - host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); - guest_ctxt = &vcpu->arch.ctxt; host_dbg = &vcpu->arch.host_debug_state.regs; guest_dbg =kern_hyp_va (vcpu->arch.debug_ptr);@@ -232,8 +230,7 @@ void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); guest_ctxt = &vcpu->arch.ctxt; -if (!has_vhe ()) -__debug_restore_spe_nvhe (host_ctxt, false); +__debug_restore_spe_context (host_ctxt,kvm_arm_spe_v1_ready (vcpu));if (!(vcpu->arch.flags &KVM_ARM64_DEBUG_DIRTY ))return ;@@ -249,19 +246,21 @@ void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) void __hyp_text __debug_save_host_context(struct kvm_vcpu *vcpu) { -/* -* Non-VHE: Disable and flush SPE data generation -* VHE: The vcpu can run, but it can't hide. -*/ struct kvm_cpu_context *host_ctxt; host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); -if (!has_vhe ()) -__debug_save_spe_nvhe (host_ctxt, false); +if (cpuid_feature_extract_unsigned_field (read_sysreg (id_aa64dfr0_el1), +ID_AA64DFR0_PMSVER_SHIFT )) +__debug_save_spe_context (host_ctxt,kvm_arm_spe_v1_ready (vcpu)); } void __hyp_text __debug_save_guest_context(struct kvm_vcpu *vcpu) { +bool kvm_spe_ready =kvm_arm_spe_v1_ready (vcpu); + +/* SPE present on this vCPU? */ +if (kvm_spe_ready) +__debug_save_spe_context (&vcpu->arch.ctxt, kvm_spe_ready); } u32 __hyp_text __kvm_get_mdcr_el2(void)include/kvm/arm_spe.h +6
-0@@ -16,4 +16,10 @@ struct kvm_spe { bool irq_level; }; +#ifdef CONFIG_KVM_ARM_SPE +#define kvm_arm_spe_v1_ready (v) ((v)->arch.spe.ready) +#else +#define kvm_arm_spe_v1_ready (v) (false) +#endif /* CONFIG_KVM_ARM_SPE */ +#endif /* __ASM_ARM_KVM_SPE_H */ Signature · 1 line
2.21.0
Re: [PATCH v2 09/18] arm64: KVM: enable conditional save/restore full SPE profiling buffer controls
Quoted 7 lines · Mark Rutland
Quoted 6 lines · Andrew Murray
From: Sudeep Holla <[email protected]>
Now that we can save/restore the full SPE controls, we can enable it if SPE is setup and ready to use in KVM. It's supported in KVM only if all the CPUs in the system supports SPE.
Quoted 7 lines
Quoted 2 lines
However to support heterogenous systems, we need to move the check if host supports SPE and do a partial save/restore.
I don't think that it makes sense to support this for heterogeneous systems, given their SPE capabilities and IMP DEF details will differ.
Is there some way we can limit this to homogeneous systems?
No problem, I'll see how to limit this.
Thanks,
Andrew Murray
Quoted 118 lines
Thanks,
Mark.Quoted 114 lines
Signed-off-by Sudeep Holla <[email protected]>Signed-off-by Andrew Murray <[email protected]>--- arch/arm64/kvm/hyp/debug-sr.c | 33 ++++++++++++++++----------------- include/kvm/arm_spe.h | 6 ++++++ 2 files changed, 22 insertions(+), 17 deletions(-)arch/arm64/kvm/hyp/debug-sr.c +16
-17@@ -86,18 +86,13 @@ } static void __hyp_text -__debug_save_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt) +__debug_save_spe_context (struct kvm_cpu_context *ctxt,bool full_ctxt) {u64 reg;/* Clear pmscr in case of early return */ ctxt->sys_regs[PMSCR_EL1 ] =0 ; -/* SPE present on this CPU? */ -if (!cpuid_feature_extract_unsigned_field (read_sysreg (id_aa64dfr0_el1), -ID_AA64DFR0_PMSVER_SHIFT )) -return ; -/* Yes; is it owned by higher EL? */ reg =read_sysreg_s (SYS_PMBIDR_EL1 );if (reg &BIT (SYS_PMBIDR_EL1_P_SHIFT ))@@ -142,7 +137,7 @@ __debug_save_spe_nvhe(struct kvm_cpu_context *ctxt, bool full_ctxt) } static void __hyp_text -__debug_restore_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt) +__debug_restore_spe_context (struct kvm_cpu_context *ctxt,bool full_ctxt) {if (!ctxt->sys_regs[PMSCR_EL1 ])return ;@@ -210,11 +205,14 @@ void __hyp_text __debug_restore_guest_context(struct kvm_vcpu *vcpu)struct kvm_guest_debug_arch *host_dbg;struct kvm_guest_debug_arch *guest_dbg; + host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); + guest_ctxt = &vcpu->arch.ctxt; + +__debug_restore_spe_context (guest_ctxt,kvm_arm_spe_v1_ready (vcpu)); +if (!(vcpu->arch.flags &KVM_ARM64_DEBUG_DIRTY ))return ; - host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); - guest_ctxt = &vcpu->arch.ctxt; host_dbg = &vcpu->arch.host_debug_state.regs; guest_dbg =kern_hyp_va (vcpu->arch.debug_ptr);@@ -232,8 +230,7 @@ void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); guest_ctxt = &vcpu->arch.ctxt; -if (!has_vhe ()) -__debug_restore_spe_nvhe (host_ctxt, false); +__debug_restore_spe_context (host_ctxt,kvm_arm_spe_v1_ready (vcpu));if (!(vcpu->arch.flags &KVM_ARM64_DEBUG_DIRTY ))return ;@@ -249,19 +246,21 @@ void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) void __hyp_text __debug_save_host_context(struct kvm_vcpu *vcpu) { -/* -* Non-VHE: Disable and flush SPE data generation -* VHE: The vcpu can run, but it can't hide. -*/ struct kvm_cpu_context *host_ctxt; host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); -if (!has_vhe ()) -__debug_save_spe_nvhe (host_ctxt, false); +if (cpuid_feature_extract_unsigned_field (read_sysreg (id_aa64dfr0_el1), +ID_AA64DFR0_PMSVER_SHIFT )) +__debug_save_spe_context (host_ctxt,kvm_arm_spe_v1_ready (vcpu)); } void __hyp_text __debug_save_guest_context(struct kvm_vcpu *vcpu) { +bool kvm_spe_ready =kvm_arm_spe_v1_ready (vcpu); + +/* SPE present on this vCPU? */ +if (kvm_spe_ready) +__debug_save_spe_context (&vcpu->arch.ctxt, kvm_spe_ready); } u32 __hyp_text __kvm_get_mdcr_el2(void)include/kvm/arm_spe.h +6
-0@@ -16,4 +16,10 @@ struct kvm_spe { bool irq_level; }; +#ifdef CONFIG_KVM_ARM_SPE +#define kvm_arm_spe_v1_ready (v) ((v)->arch.spe.ready) +#else +#define kvm_arm_spe_v1_ready (v) (false) +#endif /* CONFIG_KVM_ARM_SPE */ +#endif /* __ASM_ARM_KVM_SPE_H */ Signature · 1 line
2.21.0
Marc Zyngier
maintainerRe: [PATCH v2 09/18] arm64: KVM: enable conditional save/restore full SPE profiling buffer controls
In reply to Andrew MurrayOn Fri, 20 Dec 2019 14:30:16 +0000
Andrew Murray <[email protected]> wrote:
[somehow managed not to do a reply all, re-sending]
Quoted 8 lines
From: Sudeep Holla <[email protected]>
Now that we can save/restore the full SPE controls, we can enable it if SPE is setup and ready to use in KVM. It's supported in KVM only if all the CPUs in the system supports SPE.
However to support heterogenous systems, we need to move the check if host supports SPE and do a partial save/restore.
No. Let's just not go down that path. For now, KVM on heterogeneous systems do not get SPE. If SPE has been enabled on a guest and a CPU comes up without SPE, this CPU should fail to boot (same as exposing a feature to userspace).
Quoted 59 lines
Signed-off-by Sudeep Holla <[email protected]>Signed-off-by Andrew Murray <[email protected]>--- arch/arm64/kvm/hyp/debug-sr.c | 33 ++++++++++++++++----------------- include/kvm/arm_spe.h | 6 ++++++ 2 files changed, 22 insertions(+), 17 deletions(-)arch/arm64/kvm/hyp/debug-sr.c +7
-9@@ -86,18 +86,13 @@ } static void __hyp_text -__debug_save_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt) +__debug_save_spe_context (struct kvm_cpu_context *ctxt,bool full_ctxt) {u64 reg;/* Clear pmscr in case of early return */ ctxt->sys_regs[PMSCR_EL1 ] =0 ; -/* SPE present on this CPU? */ -if (!cpuid_feature_extract_unsigned_field (read_sysreg (id_aa64dfr0_el1), -ID_AA64DFR0_PMSVER_SHIFT )) -return ; -/* Yes; is it owned by higher EL? */ reg =read_sysreg_s (SYS_PMBIDR_EL1 );if (reg &BIT (SYS_PMBIDR_EL1_P_SHIFT ))@@ -142,7 +137,7 @@ __debug_save_spe_nvhe(struct kvm_cpu_context *ctxt, bool full_ctxt) } static void __hyp_text -__debug_restore_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt) +__debug_restore_spe_context (struct kvm_cpu_context *ctxt,bool full_ctxt) {if (!ctxt->sys_regs[PMSCR_EL1 ])return ;@@ -210,11 +205,14 @@ void __hyp_text __debug_restore_guest_context(struct kvm_vcpu *vcpu)struct kvm_guest_debug_arch *host_dbg;struct kvm_guest_debug_arch *guest_dbg; + host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); + guest_ctxt = &vcpu->arch.ctxt; + +__debug_restore_spe_context (guest_ctxt,kvm_arm_spe_v1_ready (vcpu)); +if (!(vcpu->arch.flags &KVM_ARM64_DEBUG_DIRTY ))return ; - host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); - guest_ctxt = &vcpu->arch.ctxt; host_dbg = &vcpu->arch.host_debug_state.regs; guest_dbg =kern_hyp_va (vcpu->arch.debug_ptr);
arch/arm64/kvm/hyp/debug-sr.c void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) · line 233
Quoted 6 lines
arch/arm64/kvm/hyp/debug-sr.c +1
-2@@ -232,8 +230,7 @@ void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); guest_ctxt = &vcpu->arch.ctxt; -if (!has_vhe ()) -__debug_restore_spe_nvhe (host_ctxt, false); +__debug_restore_spe_context (host_ctxt,kvm_arm_spe_v1_ready (vcpu));
So you now do an unconditional save/restore on the exit path for VHE as well? Even if the host isn't using the SPE HW? That's not acceptable as, in most cases, only the host /or/ the guest will use SPE. Here, you put a measurable overhead on each exit.
If the host is not using SPE, then the restore/save should happen in vcpu_load/vcpu_put. Only if the host is using SPE should you do something in the run loop. Of course, this only applies to VHE and non-VHE must switch eagerly.
Quoted 36 lines
arch/arm64/kvm/hyp/debug-sr.c +8
-6if (!(vcpu->arch.flags &KVM_ARM64_DEBUG_DIRTY ))return ;@@ -249,19 +246,21 @@ void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) void __hyp_text __debug_save_host_context(struct kvm_vcpu *vcpu) { -/* -* Non-VHE: Disable and flush SPE data generation -* VHE: The vcpu can run, but it can't hide. -*/ struct kvm_cpu_context *host_ctxt; host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); -if (!has_vhe ()) -__debug_save_spe_nvhe (host_ctxt, false); +if (cpuid_feature_extract_unsigned_field (read_sysreg (id_aa64dfr0_el1), +ID_AA64DFR0_PMSVER_SHIFT )) +__debug_save_spe_context (host_ctxt,kvm_arm_spe_v1_ready (vcpu)); } void __hyp_text __debug_save_guest_context(struct kvm_vcpu *vcpu) { +bool kvm_spe_ready =kvm_arm_spe_v1_ready (vcpu); + +/* SPE present on this vCPU? */ +if (kvm_spe_ready) +__debug_save_spe_context (&vcpu->arch.ctxt, kvm_spe_ready); } u32 __hyp_text __kvm_get_mdcr_el2(void)include/kvm/arm_spe.h +0
-0
include/kvm/arm_spe.h struct kvm_spe { · line 25
Quoted 10 lines
include/kvm/arm_spe.h +6
-0@@ -16,4 +16,10 @@ struct kvm_spe { bool irq_level; }; +#ifdef CONFIG_KVM_ARM_SPE +#define kvm_arm_spe_v1_ready (v) ((v)->arch.spe.ready) +#else +#define kvm_arm_spe_v1_ready (v) (false) +#endif /* CONFIG_KVM_ARM_SPE */ +#endif /* __ASM_ARM_KVM_SPE_H */
Thanks,
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
Re: [PATCH v2 09/18] arm64: KVM: enable conditional save/restore full SPE profiling buffer controls
Quoted 11 lines · Marc Zyngier
On Fri, 20 Dec 2019 14:30:16 +0000
Andrew Murray <[email protected]> wrote:[somehow managed not to do a reply all, re-sending]
Quoted 6 lines
From: Sudeep Holla <[email protected]>
Now that we can save/restore the full SPE controls, we can enable it if SPE is setup and ready to use in KVM. It's supported in KVM only if all the CPUs in the system supports SPE.
Quoted 5 lines
Quoted 2 lines
However to support heterogenous systems, we need to move the check if host supports SPE and do a partial save/restore.
No. Let's just not go down that path. For now, KVM on heterogeneous systems do not get SPE.
At present these patches only offer the SPE feature to VCPU's where the sanitised AA64DFR0 register indicates that all CPUs have this support (kvm_arm_support_spe_v1) at the time of setting the attribute (KVM_SET_DEVICE_ATTR).
Therefore if a new CPU comes online without SPE support, and an existing VCPU is scheduled onto it, then bad things happen - which I guess must have been the intention behind this patch.
Quoted 3 lines
If SPE has been enabled on a guest and a CPU
comes up without SPE, this CPU should fail to boot (same as exposing a feature to userspace).
I'm unclear as how to prevent this. We can set the FTR_STRICT flag on the sanitised register - thus tainting the kernel if such a non-SPE CPU comes online - thought that doesn't prevent KVM from blowing up. Though I don't believe we can prevent a CPU coming up. At the moment this is my preferred approach.
Looking at the vcpu_load and related code, I don't see a way of saying 'don't schedule this VCPU on this CPU' or bailing in any way.
One solution could be to allow scheduling onto non-SPE VCPUs but wrap the SPE save/restore code in a macro (much like kvm_arm_spe_v1_ready) that reads the non-sanitised feature register. Therefore we don't go bang, but we also increase the size of any black-holes in SPE capturing. Though this feels like something that will cause grief down the line.
Is there something else that can be done?
Thanks,
Andrew Murray
Quoted 129 lines
Quoted 60 lines
Signed-off-by Sudeep Holla <[email protected]>Signed-off-by Andrew Murray <[email protected]>--- arch/arm64/kvm/hyp/debug-sr.c | 33 ++++++++++++++++----------------- include/kvm/arm_spe.h | 6 ++++++ 2 files changed, 22 insertions(+), 17 deletions(-)arch/arm64/kvm/hyp/debug-sr.c +7
-9@@ -86,18 +86,13 @@ } static void __hyp_text -__debug_save_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt) +__debug_save_spe_context (struct kvm_cpu_context *ctxt,bool full_ctxt) {u64 reg;/* Clear pmscr in case of early return */ ctxt->sys_regs[PMSCR_EL1 ] =0 ; -/* SPE present on this CPU? */ -if (!cpuid_feature_extract_unsigned_field (read_sysreg (id_aa64dfr0_el1), -ID_AA64DFR0_PMSVER_SHIFT )) -return ; -/* Yes; is it owned by higher EL? */ reg =read_sysreg_s (SYS_PMBIDR_EL1 );if (reg &BIT (SYS_PMBIDR_EL1_P_SHIFT ))@@ -142,7 +137,7 @@ __debug_save_spe_nvhe(struct kvm_cpu_context *ctxt, bool full_ctxt) } static void __hyp_text -__debug_restore_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt) +__debug_restore_spe_context (struct kvm_cpu_context *ctxt,bool full_ctxt) {if (!ctxt->sys_regs[PMSCR_EL1 ])return ;@@ -210,11 +205,14 @@ void __hyp_text __debug_restore_guest_context(struct kvm_vcpu *vcpu)struct kvm_guest_debug_arch *host_dbg;struct kvm_guest_debug_arch *guest_dbg; + host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); + guest_ctxt = &vcpu->arch.ctxt; + +__debug_restore_spe_context (guest_ctxt,kvm_arm_spe_v1_ready (vcpu)); +if (!(vcpu->arch.flags &KVM_ARM64_DEBUG_DIRTY ))return ; - host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); - guest_ctxt = &vcpu->arch.ctxt; host_dbg = &vcpu->arch.host_debug_state.regs; guest_dbg =kern_hyp_va (vcpu->arch.debug_ptr);arch/arm64/kvm/hyp/debug-sr.c void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) · line 233
Quoted 6 lines
arch/arm64/kvm/hyp/debug-sr.c +1
-2@@ -232,8 +230,7 @@ void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); guest_ctxt = &vcpu->arch.ctxt; -if (!has_vhe ()) -__debug_restore_spe_nvhe (host_ctxt, false); +__debug_restore_spe_context (host_ctxt,kvm_arm_spe_v1_ready (vcpu));So you now do an unconditional save/restore on the exit path for VHE as well? Even if the host isn't using the SPE HW? That's not acceptable as, in most cases, only the host /or/ the guest will use SPE. Here, you put a measurable overhead on each exit.
If the host is not using SPE, then the restore/save should happen in vcpu_load/vcpu_put. Only if the host is using SPE should you do something in the run loop. Of course, this only applies to VHE and non-VHE must switch eagerly.
Quoted 36 lines
arch/arm64/kvm/hyp/debug-sr.c +8
-6if (!(vcpu->arch.flags &KVM_ARM64_DEBUG_DIRTY ))return ;@@ -249,19 +246,21 @@ void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) void __hyp_text __debug_save_host_context(struct kvm_vcpu *vcpu) { -/* -* Non-VHE: Disable and flush SPE data generation -* VHE: The vcpu can run, but it can't hide. -*/ struct kvm_cpu_context *host_ctxt; host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); -if (!has_vhe ()) -__debug_save_spe_nvhe (host_ctxt, false); +if (cpuid_feature_extract_unsigned_field (read_sysreg (id_aa64dfr0_el1), +ID_AA64DFR0_PMSVER_SHIFT )) +__debug_save_spe_context (host_ctxt,kvm_arm_spe_v1_ready (vcpu)); } void __hyp_text __debug_save_guest_context(struct kvm_vcpu *vcpu) { +bool kvm_spe_ready =kvm_arm_spe_v1_ready (vcpu); + +/* SPE present on this vCPU? */ +if (kvm_spe_ready) +__debug_save_spe_context (&vcpu->arch.ctxt, kvm_spe_ready); } u32 __hyp_text __kvm_get_mdcr_el2(void)include/kvm/arm_spe.h +0
-0include/kvm/arm_spe.h struct kvm_spe { · line 25
Quoted 10 lines
include/kvm/arm_spe.h +6
-0@@ -16,4 +16,10 @@ struct kvm_spe { bool irq_level; }; +#ifdef CONFIG_KVM_ARM_SPE +#define kvm_arm_spe_v1_ready (v) ((v)->arch.spe.ready) +#else +#define kvm_arm_spe_v1_ready (v) (false) +#endif /* CONFIG_KVM_ARM_SPE */ +#endif /* __ASM_ARM_KVM_SPE_H */ Thanks,
M.Signature · 1 line
Jazz is not dead. It just smells funny...
Marc Zyngier
maintainerRe: [PATCH v2 09/18] arm64: KVM: enable conditional save/restore full SPE profiling buffer controls
Quoted 18 lines · Andrew Murray
Quoted 11 lines · Marc Zyngier
On Fri, 20 Dec 2019 14:30:16 +0000
Andrew Murray <[email protected]> wrote:[somehow managed not to do a reply all, re-sending]
Quoted 6 lines
From: Sudeep Holla <[email protected]>
Now that we can save/restore the full SPE controls, we can enable it if SPE is setup and ready to use in KVM. It's supported in KVM only if all the CPUs in the system supports SPE.
Quoted 5 lines
Quoted 2 lines
However to support heterogenous systems, we need to move the check if host supports SPE and do a partial save/restore.
No. Let's just not go down that path. For now, KVM on heterogeneous systems do not get SPE.
Quoted 8 lines
At present these patches only offer the SPE feature to VCPU's where the sanitised AA64DFR0 register indicates that all CPUs have this support (
kvm_arm_support_spe_v1) at the time of setting the attribute (KVM_SET_DEVICE_ATTR).Therefore if a new CPU comes online without SPE support, and an existing VCPU is scheduled onto it, then bad things happen - which I guess must have been the intention behind this patch.
I guess that was the intent.
Quoted 9 lines
Quoted 3 lines
If SPE has been enabled on a guest and a CPU
comes up without SPE, this CPU should fail to boot (same as exposing a feature to userspace).I'm unclear as how to prevent this. We can set the
FTR_STRICTflag on the sanitised register - thus tainting the kernel if such a non-SPE CPU comes online - thought that doesn't prevent KVM from blowing up. Though I don't believe we can prevent a CPU coming up. At the moment this is my preferred approach.
I'd be OK with this as a stop-gap measure. Do we know of any existing design where only half of the CPUs have SPE?
Quoted 2 lines
Looking at the
vcpu_loadand related code, I don't see a way of saying 'don't schedule this VCPU on this CPU' or bailing in any way.
That would actually be pretty easy to implement. In vcpu_load(), check that that the CPU physical has SPE. If not, raise a request for that vcpu. In the run loop, check for that request and abort if raised, returning to userspace.
Userspace can always check /sys/devices/arm_spe_0/cpumask and work out where to run that particular vcpu.
Quoted 8 lines
One solution could be to allow scheduling onto non-SPE VCPUs but wrap the SPE save/restore code in a macro (much like
kvm_arm_spe_v1_ready) that reads the non-sanitised feature register. Therefore we don't go bang, but we also increase the size of any black-holes in SPE capturing. Though this feels like something that will cause grief down the line.Is there something else that can be done?
How does userspace deal with this? When SPE is only available on half of the CPUs, how does perf work in these conditions?
Thanks,
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
Will Deacon
maintainerRe: [PATCH v2 09/18] arm64: KVM: enable conditional save/restore full SPE profiling buffer controls
Quoted 35 lines · Marc Zyngier
Quoted 23 lines · Andrew Murray
Quoted 11 lines · Marc Zyngier
On Fri, 20 Dec 2019 14:30:16 +0000
Andrew Murray <[email protected]> wrote:[somehow managed not to do a reply all, re-sending]
Quoted 6 lines
From: Sudeep Holla <[email protected]>
Now that we can save/restore the full SPE controls, we can enable it if SPE is setup and ready to use in KVM. It's supported in KVM only if all the CPUs in the system supports SPE.
Quoted 5 lines
Quoted 2 lines
However to support heterogenous systems, we need to move the check if host supports SPE and do a partial save/restore.
No. Let's just not go down that path. For now, KVM on heterogeneous systems do not get SPE.
At present these patches only offer the SPE feature to VCPU's where the sanitised AA64DFR0 register indicates that all CPUs have this support (
kvm_arm_support_spe_v1) at the time of setting the attribute (KVM_SET_DEVICE_ATTR).Quoted 4 lines
Therefore if a new CPU comes online without SPE support, and an existing VCPU is scheduled onto it, then bad things happen - which I guess
must have been the intention behind this patch.I guess that was the intent.
Quoted 4 lines
Quoted 3 lines
If SPE has been enabled on a guest and a CPU
comes up without SPE, this CPU should fail to boot (same as exposing a feature to userspace).
Quoted 8 lines
Quoted 5 lines
I'm unclear as how to prevent this. We can set the
FTR_STRICTflag on the sanitised register - thus tainting the kernel if such a non-SPE CPU comes online - thought that doesn't prevent KVM from blowing up. Though I don't believe we can prevent a CPU coming up. At the moment this is my preferred approach.I'd be OK with this as a stop-gap measure. Do we know of any existing design where only half of the CPUs have SPE?
No, but given how few CPUs implement SPE I'd say that this configuration is inevitable. I certainly went out of my way to support it in the driver.
Quoted 10 lines
Quoted 2 lines
Looking at the
vcpu_loadand related code, I don't see a way of saying 'don't schedule this VCPU on this CPU' or bailing in any way.That would actually be pretty easy to implement. In
vcpu_load(), check that that the CPU physical has SPE. If not, raise a request for that vcpu. In the run loop, check for that request and abort if raised, returning to userspace.Userspace can always check
/sys/devices/arm_spe_0/cpumaskand work out where to run that particular vcpu.
It's also worth considering systems where there are multiple implementations of SPE in play. Assuming we don't want to expose this to a guest, then the right interface here is probably for userspace to pick one SPE implementation and expose that to the guest. That fits with your idea above, where you basically get an immediate exit if we try to schedule a vCPU onto a CPU that isn't part of the SPE mask.
Quoted 9 lines
Quoted 9 lines
One solution could be to allow scheduling onto non-SPE VCPUs but wrap the
SPE save/restore code in a macro (much likekvm_arm_spe_v1_ready) that reads the non-sanitised feature register. Therefore we don't go bang, but
we also increase the size of any black-holes in SPE capturing. Though this
feels like something that will cause grief down the line.
Quoted 4 lines
Quoted 1 line
Is there something else that can be done?
How does userspace deal with this? When SPE is only available on half of the CPUs, how does perf work in these conditions?
Not sure about userspace, but the kernel driver works by instantiating an SPE PMU instance only for the CPUs that have it and then that instance profiles for only those CPUs. You also need to do something similar if you had two CPU types with SPE, since the SPE configuration is likely to be different between them.
Will
Marc Zyngier
maintainerRe: [PATCH v2 09/18] arm64: KVM: enable conditional save/restore full SPE profiling buffer controls
Quoted 59 lines · Will Deacon
Quoted 35 lines · Marc Zyngier
Quoted 23 lines · Andrew Murray
Quoted 11 lines · Marc Zyngier
On Fri, 20 Dec 2019 14:30:16 +0000
Andrew Murray <[email protected]> wrote:[somehow managed not to do a reply all, re-sending]
Quoted 6 lines
From: Sudeep Holla <[email protected]>
Now that we can save/restore the full SPE controls, we can enable it if SPE is setup and ready to use in KVM. It's supported in KVM only if all the CPUs in the system supports SPE.
Quoted 5 lines
Quoted 2 lines
However to support heterogenous systems, we need to move the check if host supports SPE and do a partial save/restore.
No. Let's just not go down that path. For now, KVM on heterogeneous systems do not get SPE.
At present these patches only offer the SPE feature to VCPU's where the sanitised AA64DFR0 register indicates that all CPUs have this support (
kvm_arm_support_spe_v1) at the time of setting the attribute (KVM_SET_DEVICE_ATTR).Quoted 4 lines
Therefore if a new CPU comes online without SPE support, and an existing VCPU is scheduled onto it, then bad things happen - which I guess
must have been the intention behind this patch.I guess that was the intent.
Quoted 4 lines
Quoted 3 lines
If SPE has been enabled on a guest and a CPU
comes up without SPE, this CPU should fail to boot (same as exposing a feature to userspace).Quoted 8 lines
Quoted 5 lines
I'm unclear as how to prevent this. We can set the
FTR_STRICTflag on the sanitised register - thus tainting the kernel if such a non-SPE CPU comes online - thought that doesn't prevent KVM from blowing up. Though I don't believe we can prevent a CPU coming up. At the moment this is my preferred approach.I'd be OK with this as a stop-gap measure. Do we know of any existing design where only half of the CPUs have SPE?
No, but given how few CPUs implement SPE I'd say that this configuration is inevitable. I certainly went out of my way to support it in the driver.
Quoted 10 lines
Quoted 2 lines
Looking at the
vcpu_loadand related code, I don't see a way of saying 'don't schedule this VCPU on this CPU' or bailing in any way.That would actually be pretty easy to implement. In
vcpu_load(), check that that the CPU physical has SPE. If not, raise a request for that vcpu. In the run loop, check for that request and abort if raised, returning to userspace.Userspace can always check
/sys/devices/arm_spe_0/cpumaskand work out where to run that particular vcpu.
Quoted 6 lines
It's also worth considering systems where there are multiple implementations of SPE in play. Assuming we don't want to expose this to a guest, then the right interface here is probably for userspace to pick one SPE implementation and expose that to the guest. That fits with your idea above, where you basically get an immediate exit if we try to schedule a vCPU onto a CPU that isn't part of the SPE mask.
Then it means that the VM should be configured with a mask indicating which CPUs it is intended to run on, and setting such a mask is mandatory for SPE.
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
One solution could be to allow scheduling onto non-SPE VCPUs but wrap the
SPE save/restore code in a macro (much likekvm_arm_spe_v1_ready) that reads the non-sanitised feature register. Therefore we don't go bang, but
we also increase the size of any black-holes in SPE capturing. Though this
feels like something that will cause grief down the line.Is there something else that can be done?
Quoted 8 lines
Quoted 2 lines
How does userspace deal with this? When SPE is only available on half of the CPUs, how does perf work in these conditions?
Not sure about userspace, but the kernel driver works by instantiating an SPE PMU instance only for the CPUs that have it and then that instance profiles for only those CPUs. You also need to do something similar if you had two CPU types with SPE, since the SPE configuration is likely to be different between them.
So that's closer to what Andrew was suggesting above (running a guest on a non-SPE CPU creates a profiling black hole). Except that we can't really run a SPE-enabled guest on a non-SPE CPU, as the SPE sysregs will UNDEF at EL1.
Conclusion: we need a mix of a cpumask to indicate which CPUs we want to run on (generic, not-SPE related), and a check for SPE-capable CPUs. If any of these condition is not satisfied, the vcpu exits for userspace to sort out the affinity.
I hate heterogeneous systems.
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
Will Deacon
maintainerRe: [PATCH v2 09/18] arm64: KVM: enable conditional save/restore full SPE profiling buffer controls
Quoted 26 lines · Marc Zyngier
Quoted 14 lines · Will Deacon
Quoted 4 lines · Marc Zyngier
Quoted 2 lines · Andrew Murray
Looking at the
vcpu_loadand related code, I don't see a way of saying 'don't schedule this VCPU on this CPU' or bailing in any way.Quoted 8 lines
That would actually be pretty easy to implement. In
vcpu_load(), check that that the CPU physical has SPE. If not, raise a request for that vcpu.
In the run loop, check for that request and abort if raised, returning to userspace.Userspace can always check
/sys/devices/arm_spe_0/cpumaskand work out where to run that particular vcpu.Quoted 10 lines
It's also worth considering systems where there are multiple implementations
of SPE in play. Assuming we don't want to expose this to a guest, then the
right interface here is probably for userspace to pick one SPE implementation and expose that to the guest. That fits with your idea above,
where you basically get an immediate exit if we try to schedule a vCPU onto
a CPU that isn't part of the SPE mask.
Quoted 3 lines
Then it means that the VM should be configured with a mask indicating which CPUs it is intended to run on, and setting such a mask is mandatory for SPE.
Yeah, and this could probably all be wrapped up by userspace so you just pass the SPE PMU name or something and it grabs the corresponding cpumask for you.
Quoted 23 lines
Quoted 15 lines
Quoted 9 lines
Quoted 9 lines
One solution could be to allow scheduling onto non-SPE VCPUs but wrap the
SPE save/restore code in a macro (much likekvm_arm_spe_v1_ready) that reads the non-sanitised feature register. Therefore we don't go bang, but
we also increase the size of any black-holes in SPE capturing. Though this
feels like something that will cause grief down the line.Quoted 5 lines
Quoted 1 line
Is there something else that can be done?
How does userspace deal with this? When SPE is only available on half of
the CPUs, how does perf work in these conditions?Quoted 7 lines
Not sure about userspace, but the kernel driver works by instantiating an
SPE PMU instance only for the CPUs that have it and then that instance profiles for only those CPUs. You also need to do something similar if you had two CPU types with SPE, since the SPE configuration is likely to be
different between them.
Quoted 4 lines
So that's closer to what Andrew was suggesting above (running a guest on a non-SPE CPU creates a profiling black hole). Except that we can't really run a SPE-enabled guest on a non-SPE CPU, as the SPE sysregs will UNDEF at EL1.
Right. I wouldn't suggest the "black hole" approach for VMs, but it works for userspace so that's why the driver does it that way.
Quoted 6 lines
Conclusion: we need a mix of a cpumask to indicate which CPUs we want to run on (generic, not-SPE related), and a check for SPE-capable CPUs. If any of these condition is not satisfied, the vcpu exits for userspace to sort out the affinity.
I hate heterogeneous systems.
They hate you too ;)
Will
Re: [PATCH v2 09/18] arm64: KVM: enable conditional save/restore full SPE profiling buffer controls
Quoted 4 lines · Will Deacon
Quoted 3 lines · Marc Zyngier
Quoted 2 lines · Will Deacon
Quoted 1 line · Marc Zyngier
On 2020-01-07 15:13, Andrew Murray wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Looking at the
vcpu_loadand related code, I don't see a way of saying 'don't schedule this VCPU on this CPU' or bailing in any way.That would actually be pretty easy to implement. In
vcpu_load(), check that that the CPU physical has SPE. If not, raise a request for that vcpu.
In the run loop, check for that request and abort if raised, returning to userspace.
I hadn't really noticed the kvm_make_request mechanism - however it's now clear how this could be implemented.
This approach gives responsibility for which CPUs should be used to userspace and if userspace gets it wrong then the KVM_RUN ioctl won't do very much.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 3 lines
Userspace can always check
/sys/devices/arm_spe_0/cpumaskand work out where to run that particular vcpu.It's also worth considering systems where there are multiple implementations
of SPE in play. Assuming we don't want to expose this to a guest, then the
right interface here is probably for userspace to pick one SPE implementation and expose that to the guest.
If I understand correctly then this implies the following:
- If the host userspace indicates it wants support for SPE in the guest (via KVM_SET_DEVICE_ATTR at start of day) - then we should check in vcpu_load that the minimum version of SPE is present on the current CPU. 'minimum' because we don't know why userspace has selected the given cpumask.
- Userspace can get it wrong, i.e. it can create a CPU mask with CPUs that have SPE with differing versions. If it does, and all CPUs have some form of SPE then errors may occur in the guest. Perhaps this is OK and userspace shouldn't get it wrong?
Quoted 37 lines
Quoted 9 lines
Quoted 5 lines
That fits with your idea
above,
where you basically get an immediate exit if we try to schedule a vCPU onto
a CPU that isn't part of the SPE mask.Then it means that the VM should be configured with a mask indicating which CPUs it is intended to run on, and setting such a mask is mandatory for SPE.
Yeah, and this could probably all be wrapped up by userspace so you just pass the SPE PMU name or something and it grabs the corresponding cpumask for you.
Quoted 23 lines
Quoted 22 lines
Quoted 9 lines
Quoted 9 lines
One solution could be to allow scheduling onto non-SPE VCPUs but wrap the
SPE save/restore code in a macro (much likekvm_arm_spe_v1_ready) that reads the non-sanitised feature register. Therefore we don't go bang, but
we also increase the size of any black-holes in SPE capturing. Though this
feels like something that will cause grief down the line.Quoted 5 lines
Quoted 1 line
Is there something else that can be done?
How does userspace deal with this? When SPE is only available on half of
the CPUs, how does perf work in these conditions?Not sure about userspace, but the kernel driver works by instantiating an
SPE PMU instance only for the CPUs that have it and then that instance profiles for only those CPUs. You also need to do something similar if you had two CPU types with SPE, since the SPE configuration is likely to be
different between them.
Quoted 10 lines
Quoted 4 lines
So that's closer to what Andrew was suggesting above (running a guest on a non-SPE CPU creates a profiling black hole). Except that we can't really run a SPE-enabled guest on a non-SPE CPU, as the SPE sysregs will UNDEF at EL1.
Right. I wouldn't suggest the "black hole" approach for VMs, but it works for userspace so that's why the driver does it that way.
Quoted 2 lines
Conclusion: we need a mix of a cpumask to indicate which CPUs we want to run on (generic, not-SPE related),
If I understand correctly this mask isn't exposed to KVM (in the kernel) and KVM (in the kernel) is unware of how the CPUs that have KVM_RUN called are selected.
Thus this implies the cpumask is a feature of KVM tool or QEMU that would need to be added there. (E.g. kvm_cmd_run_work would set some affinity when creating pthreads - based on a CPU mask triggered by setting the --spe flag)?
Thanks,
Andrew Murray
Quoted 9 lines
and a check for SPE-capable CPUs.
Quoted 4 lines
If any of these condition is not satisfied, the vcpu exits for userspace to sort out the affinity.
I hate heterogeneous systems.
They hate you too ;)
Will
Re: [PATCH v2 09/18] arm64: KVM: enable conditional save/restore full SPE profiling buffer controls
Quoted 39 lines · Andrew Murray
Quoted 4 lines · Will Deacon
Quoted 3 lines · Marc Zyngier
Quoted 2 lines · Will Deacon
Quoted 1 line · Marc Zyngier
On 2020-01-07 15:13, Andrew Murray wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Looking at the
vcpu_loadand related code, I don't see a way of saying 'don't schedule this VCPU on this CPU' or bailing in any way.That would actually be pretty easy to implement. In
vcpu_load(), check that that the CPU physical has SPE. If not, raise a request for that vcpu.
In the run loop, check for that request and abort if raised, returning to userspace.I hadn't really noticed the
kvm_make_requestmechanism - however it's now clear how this could be implemented.This approach gives responsibility for which CPUs should be used to userspace and if userspace gets it wrong then the
KVM_RUNioctl won't do very much.Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 3 lines
Userspace can always check
/sys/devices/arm_spe_0/cpumaskand work out where to run that particular vcpu.It's also worth considering systems where there are multiple implementations
of SPE in play. Assuming we don't want to expose this to a guest, then the
right interface here is probably for userspace to pick one SPE implementation and expose that to the guest.If I understand correctly then this implies the following:
- If the host userspace indicates it wants support for SPE in the guest (via
KVM_SET_DEVICE_ATTRat start of day) - then we should check invcpu_loadthat the minimum version of SPE is present on the current CPU. 'minimum' because we don't know why userspace has selected the given cpumask.
Quoted 4 lines
- Userspace can get it wrong, i.e. it can create a CPU mask with CPUs that have SPE with differing versions. If it does, and all CPUs have some form of SPE then errors may occur in the guest. Perhaps this is OK and userspace shouldn't get it wrong?
Actually this could be guarded against by emulating the ID_AA64DFR0_EL1 such to cap the version to the minimum SPE version - if absolutely required.
Thanks,
Andrew Murray
Quoted 75 lines
Quoted 39 lines
Quoted 11 lines
Quoted 7 lines
That fits with your idea
above,
where you basically get an immediate exit if we try to schedule a vCPU onto
a CPU that isn't part of the SPE mask.Then it means that the VM should be configured with a mask indicating which CPUs it is intended to run on, and setting such a mask is mandatory for SPE.
Yeah, and this could probably all be wrapped up by userspace so you just pass the SPE PMU name or something and it grabs the corresponding cpumask for you.
Quoted 23 lines
Quoted 22 lines
Quoted 9 lines
Quoted 9 lines
One solution could be to allow scheduling onto non-SPE VCPUs but wrap the
SPE save/restore code in a macro (much likekvm_arm_spe_v1_ready) that reads the non-sanitised feature register. Therefore we don't go bang, but
we also increase the size of any black-holes in SPE capturing. Though this
feels like something that will cause grief down the line.Quoted 5 lines
Quoted 1 line
Is there something else that can be done?
How does userspace deal with this? When SPE is only available on half of
the CPUs, how does perf work in these conditions?Not sure about userspace, but the kernel driver works by instantiating an
SPE PMU instance only for the CPUs that have it and then that instance profiles for only those CPUs. You also need to do something similar if you had two CPU types with SPE, since the SPE configuration is likely to be
different between them.Quoted 10 lines
Quoted 4 lines
So that's closer to what Andrew was suggesting above (running a guest on a non-SPE CPU creates a profiling black hole). Except that we can't really run a SPE-enabled guest on a non-SPE CPU, as the SPE sysregs will UNDEF at EL1.
Right. I wouldn't suggest the "black hole" approach for VMs, but it works for userspace so that's why the driver does it that way.
Quoted 2 lines
Conclusion: we need a mix of a cpumask to indicate which CPUs we want to run on (generic, not-SPE related),
If I understand correctly this mask isn't exposed to KVM (in the kernel) and KVM (in the kernel) is unware of how the CPUs that have
KVM_RUNcalled are selected.Thus this implies the cpumask is a feature of KVM tool or QEMU that would need to be added there. (E.g.
kvm_cmd_run_workwould set some affinity when creating pthreads - based on a CPU mask triggered by setting the --spe flag)?Thanks,
Andrew Murray
Quoted 9 lines
and a check for SPE-capable CPUs.
Quoted 4 lines
If any of these condition is not satisfied, the vcpu exits for userspace to sort out the affinity.
I hate heterogeneous systems.
They hate you too ;)
Will
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Will Deacon
maintainerRe: [PATCH v2 09/18] arm64: KVM: enable conditional save/restore full SPE profiling buffer controls
Quoted 40 lines · Andrew Murray
Quoted 39 lines · Andrew Murray
Quoted 4 lines · Will Deacon
Quoted 3 lines · Marc Zyngier
Quoted 2 lines · Will Deacon
Quoted 1 line · Marc Zyngier
On 2020-01-07 15:13, Andrew Murray wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Looking at the
vcpu_loadand related code, I don't see a way of saying 'don't schedule this VCPU on this CPU' or bailing in any way.That would actually be pretty easy to implement. In
vcpu_load(), check that that the CPU physical has SPE. If not, raise a request for that vcpu.
In the run loop, check for that request and abort if raised, returning to userspace.I hadn't really noticed the
kvm_make_requestmechanism - however it's now clear how this could be implemented.This approach gives responsibility for which CPUs should be used to userspace and if userspace gets it wrong then the
KVM_RUNioctl won't do very much.Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 3 lines
Userspace can always check
/sys/devices/arm_spe_0/cpumaskand work out where to run that particular vcpu.It's also worth considering systems where there are multiple implementations
of SPE in play. Assuming we don't want to expose this to a guest, then the
right interface here is probably for userspace to pick one SPE implementation and expose that to the guest.If I understand correctly then this implies the following:
- If the host userspace indicates it wants support for SPE in the guest (via
KVM_SET_DEVICE_ATTRat start of day) - then we should check invcpu_loadthat the minimum version of SPE is present on the current CPU. 'minimum' because we don't know why userspace has selected the given cpumask.
Quoted 7 lines
Quoted 4 lines
- Userspace can get it wrong, i.e. it can create a CPU mask with CPUs that have SPE with differing versions. If it does, and all CPUs have some form of SPE then errors may occur in the guest. Perhaps this is OK and userspace shouldn't get it wrong?
Actually this could be guarded against by emulating the
ID_AA64DFR0_EL1such to cap the version to the minimum SPE version - if absolutely required.
The problem is, it's not as simple as checking a version field. Instead, you'd have to look at all of the ID registers for SPE so that you don't end up with funny differences such as minimum sampling interval, or hardware RNG support. Ultimately though, *much* of the trace is going to be describing IMP DEF stuff because it's so micro-architectural, and there's very little you can do to hide that.
Will
Re: [PATCH v2 09/18] arm64: KVM: enable conditional save/restore full SPE profiling buffer controls
In reply to Marc ZyngierQuoted 90 lines · Marc Zyngier
On Fri, 20 Dec 2019 14:30:16 +0000
Andrew Murray <[email protected]> wrote:[somehow managed not to do a reply all, re-sending]
Quoted 8 lines
From: Sudeep Holla <[email protected]>
Now that we can save/restore the full SPE controls, we can enable it if SPE is setup and ready to use in KVM. It's supported in KVM only if all the CPUs in the system supports SPE.
However to support heterogenous systems, we need to move the check if host supports SPE and do a partial save/restore.
No. Let's just not go down that path. For now, KVM on heterogeneous systems do not get SPE. If SPE has been enabled on a guest and a CPU comes up without SPE, this CPU should fail to boot (same as exposing a feature to userspace).
Quoted 59 lines
Signed-off-by Sudeep Holla <[email protected]>Signed-off-by Andrew Murray <[email protected]>--- arch/arm64/kvm/hyp/debug-sr.c | 33 ++++++++++++++++----------------- include/kvm/arm_spe.h | 6 ++++++ 2 files changed, 22 insertions(+), 17 deletions(-)arch/arm64/kvm/hyp/debug-sr.c +7
-9@@ -86,18 +86,13 @@ } static void __hyp_text -__debug_save_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt) +__debug_save_spe_context (struct kvm_cpu_context *ctxt,bool full_ctxt) {u64 reg;/* Clear pmscr in case of early return */ ctxt->sys_regs[PMSCR_EL1 ] =0 ; -/* SPE present on this CPU? */ -if (!cpuid_feature_extract_unsigned_field (read_sysreg (id_aa64dfr0_el1), -ID_AA64DFR0_PMSVER_SHIFT )) -return ; -/* Yes; is it owned by higher EL? */ reg =read_sysreg_s (SYS_PMBIDR_EL1 );if (reg &BIT (SYS_PMBIDR_EL1_P_SHIFT ))@@ -142,7 +137,7 @@ __debug_save_spe_nvhe(struct kvm_cpu_context *ctxt, bool full_ctxt) } static void __hyp_text -__debug_restore_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt) +__debug_restore_spe_context (struct kvm_cpu_context *ctxt,bool full_ctxt) {if (!ctxt->sys_regs[PMSCR_EL1 ])return ;@@ -210,11 +205,14 @@ void __hyp_text __debug_restore_guest_context(struct kvm_vcpu *vcpu)struct kvm_guest_debug_arch *host_dbg;struct kvm_guest_debug_arch *guest_dbg; + host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); + guest_ctxt = &vcpu->arch.ctxt; + +__debug_restore_spe_context (guest_ctxt,kvm_arm_spe_v1_ready (vcpu)); +if (!(vcpu->arch.flags &KVM_ARM64_DEBUG_DIRTY ))return ; - host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); - guest_ctxt = &vcpu->arch.ctxt; host_dbg = &vcpu->arch.host_debug_state.regs; guest_dbg =kern_hyp_va (vcpu->arch.debug_ptr);arch/arm64/kvm/hyp/debug-sr.c void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) · line 233
Quoted 6 lines
arch/arm64/kvm/hyp/debug-sr.c +1
-2@@ -232,8 +230,7 @@ void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); guest_ctxt = &vcpu->arch.ctxt; -if (!has_vhe ()) -__debug_restore_spe_nvhe (host_ctxt, false); +__debug_restore_spe_context (host_ctxt,kvm_arm_spe_v1_ready (vcpu));So you now do an unconditional save/restore on the exit path for VHE as well? Even if the host isn't using the SPE HW? That's not acceptable as, in most cases, only the host /or/ the guest will use SPE. Here, you put a measurable overhead on each exit.
Quoted 5 lines
If the host is not using SPE, then the restore/save should happen in vcpu_load/vcpu_put. Only if the host is using SPE should you do something in the run loop. Of course, this only applies to VHE and non-VHE must switch eagerly.
On VHE where SPE is used in the guest only - we save/restore in vcpu_load/put.
On VHE where SPE is used in the host only - we save/restore in the run loop.
On VHE where SPE is used in guest and host - we save/restore in the run loop.
As the guest can't trace EL2 it doesn't matter if we restore guest SPE early in the vcpu_load/put functions. (I assume it doesn't matter that we restore an EL0/EL1 profiling buffer address at this point and enable tracing given that there is nothing to trace until entering the guest).
However the reason for moving save/restore to vcpu_load/put when the host is using SPE is to minimise the host EL2 black-out window.
On nVHE we always save/restore in the run loop. For the SPE guest-use-only use-case we can't save/restore in vcpu_load/put - because the guest runs at the same ELx level as the host - and thus doing so would result in the guest tracing part of the host.
Though if we determine that (for nVHE systems) the guest SPE is profiling only EL0 - then we could also save/restore in vcpu_load/put where SPE is only being used in the guest.
Does that make sense, are my reasons correct?
Thanks,
Andrew Murray
Quoted 52 lines
Quoted 36 lines
arch/arm64/kvm/hyp/debug-sr.c +8
-6if (!(vcpu->arch.flags &KVM_ARM64_DEBUG_DIRTY ))return ;@@ -249,19 +246,21 @@ void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) void __hyp_text __debug_save_host_context(struct kvm_vcpu *vcpu) { -/* -* Non-VHE: Disable and flush SPE data generation -* VHE: The vcpu can run, but it can't hide. -*/ struct kvm_cpu_context *host_ctxt; host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); -if (!has_vhe ()) -__debug_save_spe_nvhe (host_ctxt, false); +if (cpuid_feature_extract_unsigned_field (read_sysreg (id_aa64dfr0_el1), +ID_AA64DFR0_PMSVER_SHIFT )) +__debug_save_spe_context (host_ctxt,kvm_arm_spe_v1_ready (vcpu)); } void __hyp_text __debug_save_guest_context(struct kvm_vcpu *vcpu) { +bool kvm_spe_ready =kvm_arm_spe_v1_ready (vcpu); + +/* SPE present on this vCPU? */ +if (kvm_spe_ready) +__debug_save_spe_context (&vcpu->arch.ctxt, kvm_spe_ready); } u32 __hyp_text __kvm_get_mdcr_el2(void)include/kvm/arm_spe.h +0
-0include/kvm/arm_spe.h struct kvm_spe { · line 25
Quoted 10 lines
include/kvm/arm_spe.h +6
-0@@ -16,4 +16,10 @@ struct kvm_spe { bool irq_level; }; +#ifdef CONFIG_KVM_ARM_SPE +#define kvm_arm_spe_v1_ready (v) ((v)->arch.spe.ready) +#else +#define kvm_arm_spe_v1_ready (v) (false) +#endif /* CONFIG_KVM_ARM_SPE */ +#endif /* __ASM_ARM_KVM_SPE_H */ Thanks,
M.Signature · 1 line
Jazz is not dead. It just smells funny...
Re: [PATCH v2 09/18] arm64: KVM: enable conditional save/restore full SPE profiling buffer controls
Quoted 117 lines · Andrew Murray
Quoted 90 lines · Marc Zyngier
On Fri, 20 Dec 2019 14:30:16 +0000
Andrew Murray <[email protected]> wrote:[somehow managed not to do a reply all, re-sending]
Quoted 8 lines
From: Sudeep Holla <[email protected]>
Now that we can save/restore the full SPE controls, we can enable it if SPE is setup and ready to use in KVM. It's supported in KVM only if all the CPUs in the system supports SPE.
However to support heterogenous systems, we need to move the check if host supports SPE and do a partial save/restore.
No. Let's just not go down that path. For now, KVM on heterogeneous systems do not get SPE. If SPE has been enabled on a guest and a CPU comes up without SPE, this CPU should fail to boot (same as exposing a feature to userspace).
Quoted 59 lines
Signed-off-by Sudeep Holla <[email protected]>Signed-off-by Andrew Murray <[email protected]>--- arch/arm64/kvm/hyp/debug-sr.c | 33 ++++++++++++++++----------------- include/kvm/arm_spe.h | 6 ++++++ 2 files changed, 22 insertions(+), 17 deletions(-)arch/arm64/kvm/hyp/debug-sr.c +7
-9@@ -86,18 +86,13 @@ } static void __hyp_text -__debug_save_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt) +__debug_save_spe_context (struct kvm_cpu_context *ctxt,bool full_ctxt) {u64 reg;/* Clear pmscr in case of early return */ ctxt->sys_regs[PMSCR_EL1 ] =0 ; -/* SPE present on this CPU? */ -if (!cpuid_feature_extract_unsigned_field (read_sysreg (id_aa64dfr0_el1), -ID_AA64DFR0_PMSVER_SHIFT )) -return ; -/* Yes; is it owned by higher EL? */ reg =read_sysreg_s (SYS_PMBIDR_EL1 );if (reg &BIT (SYS_PMBIDR_EL1_P_SHIFT ))@@ -142,7 +137,7 @@ __debug_save_spe_nvhe(struct kvm_cpu_context *ctxt, bool full_ctxt) } static void __hyp_text -__debug_restore_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt) +__debug_restore_spe_context (struct kvm_cpu_context *ctxt,bool full_ctxt) {if (!ctxt->sys_regs[PMSCR_EL1 ])return ;@@ -210,11 +205,14 @@ void __hyp_text __debug_restore_guest_context(struct kvm_vcpu *vcpu)struct kvm_guest_debug_arch *host_dbg;struct kvm_guest_debug_arch *guest_dbg; + host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); + guest_ctxt = &vcpu->arch.ctxt; + +__debug_restore_spe_context (guest_ctxt,kvm_arm_spe_v1_ready (vcpu)); +if (!(vcpu->arch.flags &KVM_ARM64_DEBUG_DIRTY ))return ; - host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); - guest_ctxt = &vcpu->arch.ctxt; host_dbg = &vcpu->arch.host_debug_state.regs; guest_dbg =kern_hyp_va (vcpu->arch.debug_ptr);arch/arm64/kvm/hyp/debug-sr.c void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) · line 233
Quoted 6 lines
arch/arm64/kvm/hyp/debug-sr.c +1
-2@@ -232,8 +230,7 @@ void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); guest_ctxt = &vcpu->arch.ctxt; -if (!has_vhe ()) -__debug_restore_spe_nvhe (host_ctxt, false); +__debug_restore_spe_context (host_ctxt,kvm_arm_spe_v1_ready (vcpu));So you now do an unconditional save/restore on the exit path for VHE as well? Even if the host isn't using the SPE HW? That's not acceptable as, in most cases, only the host /or/ the guest will use SPE. Here, you put a measurable overhead on each exit.
Quoted 5 lines
If the host is not using SPE, then the restore/save should happen in vcpu_load/vcpu_put. Only if the host is using SPE should you do something in the run loop. Of course, this only applies to VHE and non-VHE must switch eagerly.
On VHE where SPE is used in the guest only - we save/restore in vcpu_load/put.
On VHE where SPE is used in the host only - we save/restore in the run loop.
On VHE where SPE is used in guest and host - we save/restore in the run loop.
As the guest can't trace EL2 it doesn't matter if we restore guest SPE early in the vcpu_load/put functions. (I assume it doesn't matter that we restore an EL0/EL1 profiling buffer address at this point and enable tracing given that there is nothing to trace until entering the guest).
However the reason for moving save/restore to vcpu_load/put when the host is using SPE is to minimise the host EL2 black-out window.
On nVHE we always save/restore in the run loop. For the SPE guest-use-only use-case we can't save/restore in vcpu_load/put - because the guest runs at the same ELx level as the host - and thus doing so would result in the guest tracing part of the host.
Quoted 5 lines
Though if we determine that (for nVHE systems) the guest SPE is profiling only EL0 - then we could also save/restore in vcpu_load/put where SPE is only being used in the guest.
Does that make sense, are my reasons correct?
Also I'm making the following assumptions:
- We determine if the host or guest are using SPE by seeing if profiling (e.g. PMSCR_EL1) is enabled. That should determine *when* we restore as per my previous email.
- I'm less sure on this: We should determine *what* we restore based on the availability of the SPE feature and not if it is being used - so for guest this is if the guest has the feature on the vcpu. For host this is based on the CPU feature registers.
The downshot of this is that if you have SPE support present on guest and host and they aren't being used, then you still save/restore upon entering/ leaving a guest. The reason I feel this is needed is to prevent the issue where the host starts programming the SPE registers, but is preempted by KVM entering a guest, before it could enable host SPE. Thus when we enter the guest we don't save all the registers, we return to the host and the host SPE carries on from where it left of and enables it - yet because we didn't restore all the programmed registers it doesn't work.
Thanks,
Andrew Murray
Quoted 62 lines
Thanks,
Andrew Murray
Quoted 49 lines
Quoted 36 lines
arch/arm64/kvm/hyp/debug-sr.c +8
-6if (!(vcpu->arch.flags &KVM_ARM64_DEBUG_DIRTY ))return ;@@ -249,19 +246,21 @@ void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) void __hyp_text __debug_save_host_context(struct kvm_vcpu *vcpu) { -/* -* Non-VHE: Disable and flush SPE data generation -* VHE: The vcpu can run, but it can't hide. -*/ struct kvm_cpu_context *host_ctxt; host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); -if (!has_vhe ()) -__debug_save_spe_nvhe (host_ctxt, false); +if (cpuid_feature_extract_unsigned_field (read_sysreg (id_aa64dfr0_el1), +ID_AA64DFR0_PMSVER_SHIFT )) +__debug_save_spe_context (host_ctxt,kvm_arm_spe_v1_ready (vcpu)); } void __hyp_text __debug_save_guest_context(struct kvm_vcpu *vcpu) { +bool kvm_spe_ready =kvm_arm_spe_v1_ready (vcpu); + +/* SPE present on this vCPU? */ +if (kvm_spe_ready) +__debug_save_spe_context (&vcpu->arch.ctxt, kvm_spe_ready); } u32 __hyp_text __kvm_get_mdcr_el2(void)include/kvm/arm_spe.h +0
-0include/kvm/arm_spe.h struct kvm_spe { · line 25
Quoted 10 lines
include/kvm/arm_spe.h +6
-0@@ -16,4 +16,10 @@ struct kvm_spe { bool irq_level; }; +#ifdef CONFIG_KVM_ARM_SPE +#define kvm_arm_spe_v1_ready (v) ((v)->arch.spe.ready) +#else +#define kvm_arm_spe_v1_ready (v) (false) +#endif /* CONFIG_KVM_ARM_SPE */ +#endif /* __ASM_ARM_KVM_SPE_H */ Thanks,
Quoted 3 lines
M.Signature · 1 line
Jazz is not dead. It just smells funny..._______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerRe: [PATCH v2 09/18] arm64: KVM: enable conditional save/restore full SPE profiling buffer controls
Quoted 124 lines · Andrew Murray
Quoted 117 lines · Andrew Murray
Quoted 90 lines · Marc Zyngier
On Fri, 20 Dec 2019 14:30:16 +0000
Andrew Murray <[email protected]> wrote:[somehow managed not to do a reply all, re-sending]
Quoted 8 lines
From: Sudeep Holla <[email protected]>
Now that we can save/restore the full SPE controls, we can enable it if SPE is setup and ready to use in KVM. It's supported in KVM only if all the CPUs in the system supports SPE.
However to support heterogenous systems, we need to move the check if host supports SPE and do a partial save/restore.
No. Let's just not go down that path. For now, KVM on heterogeneous systems do not get SPE. If SPE has been enabled on a guest and a CPU comes up without SPE, this CPU should fail to boot (same as exposing a feature to userspace).
Quoted 59 lines
Signed-off-by Sudeep Holla <[email protected]>Signed-off-by Andrew Murray <[email protected]>--- arch/arm64/kvm/hyp/debug-sr.c | 33 ++++++++++++++++----------------- include/kvm/arm_spe.h | 6 ++++++ 2 files changed, 22 insertions(+), 17 deletions(-)arch/arm64/kvm/hyp/debug-sr.c +7
-9@@ -86,18 +86,13 @@ } static void __hyp_text -__debug_save_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt) +__debug_save_spe_context (struct kvm_cpu_context *ctxt,bool full_ctxt) {u64 reg;/* Clear pmscr in case of early return */ ctxt->sys_regs[PMSCR_EL1 ] =0 ; -/* SPE present on this CPU? */ -if (!cpuid_feature_extract_unsigned_field (read_sysreg (id_aa64dfr0_el1), -ID_AA64DFR0_PMSVER_SHIFT )) -return ; -/* Yes; is it owned by higher EL? */ reg =read_sysreg_s (SYS_PMBIDR_EL1 );if (reg &BIT (SYS_PMBIDR_EL1_P_SHIFT ))@@ -142,7 +137,7 @@ __debug_save_spe_nvhe(struct kvm_cpu_context *ctxt, bool full_ctxt) } static void __hyp_text -__debug_restore_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt) +__debug_restore_spe_context (struct kvm_cpu_context *ctxt,bool full_ctxt) {if (!ctxt->sys_regs[PMSCR_EL1 ])return ;@@ -210,11 +205,14 @@ void __hyp_text __debug_restore_guest_context(struct kvm_vcpu *vcpu)struct kvm_guest_debug_arch *host_dbg;struct kvm_guest_debug_arch *guest_dbg; + host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); + guest_ctxt = &vcpu->arch.ctxt; + +__debug_restore_spe_context (guest_ctxt,kvm_arm_spe_v1_ready (vcpu)); +if (!(vcpu->arch.flags &KVM_ARM64_DEBUG_DIRTY ))return ; - host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); - guest_ctxt = &vcpu->arch.ctxt; host_dbg = &vcpu->arch.host_debug_state.regs; guest_dbg =kern_hyp_va (vcpu->arch.debug_ptr);arch/arm64/kvm/hyp/debug-sr.c void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) · line 233
Quoted 6 lines
arch/arm64/kvm/hyp/debug-sr.c +1
-2@@ -232,8 +230,7 @@ void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); guest_ctxt = &vcpu->arch.ctxt; -if (!has_vhe ()) -__debug_restore_spe_nvhe (host_ctxt, false); +__debug_restore_spe_context (host_ctxt,kvm_arm_spe_v1_ready (vcpu));So you now do an unconditional save/restore on the exit path for VHE as well? Even if the host isn't using the SPE HW? That's not acceptable as, in most cases, only the host /or/ the guest will use SPE. Here, you put a measurable overhead on each exit.
Quoted 5 lines
If the host is not using SPE, then the restore/save should happen in vcpu_load/vcpu_put. Only if the host is using SPE should you do something in the run loop. Of course, this only applies to VHE and non-VHE must switch eagerly.
On VHE where SPE is used in the guest only - we save/restore in vcpu_load/put.
On VHE where SPE is used in the host only - we save/restore in the run loop.
On VHE where SPE is used in guest and host - we save/restore in the run loop.
As the guest can't trace EL2 it doesn't matter if we restore guest SPE early in the vcpu_load/put functions. (I assume it doesn't matter that we restore an EL0/EL1 profiling buffer address at this point and enable tracing given that there is nothing to trace until entering the guest).
However the reason for moving save/restore to vcpu_load/put when the host is using SPE is to minimise the host EL2 black-out window.
On nVHE we always save/restore in the run loop. For the SPE guest-use-only use-case we can't save/restore in vcpu_load/put - because the guest runs at the same ELx level as the host - and thus doing so would result in the guest tracing part of the host.
Quoted 5 lines
Though if we determine that (for nVHE systems) the guest SPE is profiling only EL0 - then we could also save/restore in vcpu_load/put where SPE is only being used in the guest.
Does that make sense, are my reasons correct?
Quoted 7 lines
Also I'm making the following assumptions:
- We determine if the host or guest are using SPE by seeing if
profiling
(e.g.PMSCR_EL1) is enabled. That should determine *when* we restore
as per
my previous email.
Yes.
Quoted 7 lines
- I'm less sure on this: We should determine *what* we restore based
on the
availability of the SPE feature and not if it is being used - so for
guest
this is if the guest has the feature on the vcpu. For host this is
based on
the CPU feature registers.
As long as the guest's feature is conditionned on the HW being present *and* that you're running on a CPU that has the HW.
Quoted 7 lines
The downshot of this is that if you have SPE support present on
guest and
host and they aren't being used, then you still save/restore upon
entering/
leaving a guest. The reason I feel this is needed is to prevent the
issue
where the host starts programming the SPE registers, but is
Quoted 8 lines
preempted by
KVM entering a guest, before it could enable host SPE. Thus when we
enter the
guest we don't save all the registers, we return to the host and the
host
SPE carries on from where it left of and enables it - yet because we
didn't
restore all the programmed registers it doesn't work.
Saving the host registers is never optional if they are shared with the guest.
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
Re: [PATCH v2 09/18] arm64: KVM: enable conditional save/restore full SPE profiling buffer controls
Quoted 155 lines · Marc Zyngier
Quoted 136 lines · Andrew Murray
Quoted 127 lines · Andrew Murray
Quoted 90 lines · Marc Zyngier
On Fri, 20 Dec 2019 14:30:16 +0000
Andrew Murray <[email protected]> wrote:[somehow managed not to do a reply all, re-sending]
Quoted 8 lines
From: Sudeep Holla <[email protected]>
Now that we can save/restore the full SPE controls, we can enable it if SPE is setup and ready to use in KVM. It's supported in KVM only if all the CPUs in the system supports SPE.
However to support heterogenous systems, we need to move the check if host supports SPE and do a partial save/restore.
No. Let's just not go down that path. For now, KVM on heterogeneous systems do not get SPE. If SPE has been enabled on a guest and a CPU comes up without SPE, this CPU should fail to boot (same as exposing a feature to userspace).
Quoted 59 lines
Signed-off-by Sudeep Holla <[email protected]>Signed-off-by Andrew Murray <[email protected]>--- arch/arm64/kvm/hyp/debug-sr.c | 33 ++++++++++++++++----------------- include/kvm/arm_spe.h | 6 ++++++ 2 files changed, 22 insertions(+), 17 deletions(-)arch/arm64/kvm/hyp/debug-sr.c +7
-9@@ -86,18 +86,13 @@ } static void __hyp_text -__debug_save_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt) +__debug_save_spe_context (struct kvm_cpu_context *ctxt,bool full_ctxt) {u64 reg;/* Clear pmscr in case of early return */ ctxt->sys_regs[PMSCR_EL1 ] =0 ; -/* SPE present on this CPU? */ -if (!cpuid_feature_extract_unsigned_field (read_sysreg (id_aa64dfr0_el1), -ID_AA64DFR0_PMSVER_SHIFT )) -return ; -/* Yes; is it owned by higher EL? */ reg =read_sysreg_s (SYS_PMBIDR_EL1 );if (reg &BIT (SYS_PMBIDR_EL1_P_SHIFT ))@@ -142,7 +137,7 @@ __debug_save_spe_nvhe(struct kvm_cpu_context *ctxt, bool full_ctxt) } static void __hyp_text -__debug_restore_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt) +__debug_restore_spe_context (struct kvm_cpu_context *ctxt,bool full_ctxt) {if (!ctxt->sys_regs[PMSCR_EL1 ])return ;@@ -210,11 +205,14 @@ void __hyp_text __debug_restore_guest_context(struct kvm_vcpu *vcpu)struct kvm_guest_debug_arch *host_dbg;struct kvm_guest_debug_arch *guest_dbg; + host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); + guest_ctxt = &vcpu->arch.ctxt; + +__debug_restore_spe_context (guest_ctxt,kvm_arm_spe_v1_ready (vcpu)); +if (!(vcpu->arch.flags &KVM_ARM64_DEBUG_DIRTY ))return ; - host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); - guest_ctxt = &vcpu->arch.ctxt; host_dbg = &vcpu->arch.host_debug_state.regs; guest_dbg =kern_hyp_va (vcpu->arch.debug_ptr);arch/arm64/kvm/hyp/debug-sr.c void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) · line 233
Quoted 6 lines
arch/arm64/kvm/hyp/debug-sr.c +1
-2@@ -232,8 +230,7 @@ void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); guest_ctxt = &vcpu->arch.ctxt; -if (!has_vhe ()) -__debug_restore_spe_nvhe (host_ctxt, false); +__debug_restore_spe_context (host_ctxt,kvm_arm_spe_v1_ready (vcpu));So you now do an unconditional save/restore on the exit path for VHE as well? Even if the host isn't using the SPE HW? That's not acceptable as, in most cases, only the host /or/ the guest will use SPE. Here, you put a measurable overhead on each exit.
Quoted 5 lines
If the host is not using SPE, then the restore/save should happen in vcpu_load/vcpu_put. Only if the host is using SPE should you do something in the run loop. Of course, this only applies to VHE and non-VHE must switch eagerly.
On VHE where SPE is used in the guest only - we save/restore in vcpu_load/put.
On VHE where SPE is used in the host only - we save/restore in the run loop.
On VHE where SPE is used in guest and host - we save/restore in the run loop.
As the guest can't trace EL2 it doesn't matter if we restore guest SPE early
in the vcpu_load/put functions. (I assume it doesn't matter that we restore
an EL0/EL1 profiling buffer address at this point and enable tracing given
that there is nothing to trace until entering the guest).However the reason for moving save/restore to vcpu_load/put when the host is
using SPE is to minimise the host EL2 black-out window.On nVHE we always save/restore in the run loop. For the SPE guest-use-only
use-case we can't save/restore in vcpu_load/put - because the guest runs at
the same ELx level as the host - and thus doing so would result in the guest
tracing part of the host.Quoted 7 lines
Though if we determine that (for nVHE systems) the guest SPE is profiling only
EL0 - then we could also save/restore in vcpu_load/put where SPE is only being
used in the guest.Does that make sense, are my reasons correct?
Quoted 7 lines
Also I'm making the following assumptions:
- We determine if the host or guest are using SPE by seeing if
profiling
(e.g.PMSCR_EL1) is enabled. That should determine *when* we restore
as per
my previous email.Yes.
Quoted 7 lines
- I'm less sure on this: We should determine *what* we restore based on
the
availability of the SPE feature and not if it is being used - so for
guest
this is if the guest has the feature on the vcpu. For host this is
based on
the CPU feature registers.
Quoted 2 lines
As long as the guest's feature is conditionned on the HW being present *and* that you're running on a CPU that has the HW.
Yes that makes sense.
Quoted 17 lines
Quoted 16 lines
The downshot of this is that if you have SPE support present on guest
and
host and they aren't being used, then you still save/restore upon
entering/
leaving a guest. The reason I feel this is needed is to prevent the
issue
where the host starts programming the SPE registers, but is preempted
by
KVM entering a guest, before it could enable host SPE. Thus when we
enter the
guest we don't save all the registers, we return to the host and the
host
SPE carries on from where it left of and enables it - yet because we
didn't
restore all the programmed registers it doesn't work.
Quoted 2 lines
Saving the host registers is never optional if they are shared with the guest.
That make me feel better :)
Thanks,
Andrew Murray
Quoted 4 lines
M.Signature · 1 line
Jazz is not dead. It just smells funny...
Marc Zyngier
maintainerRe: [PATCH v2 09/18] arm64: KVM: enable conditional save/restore full SPE profiling buffer controls
In reply to Andrew MurrayQuoted 91 lines · Andrew Murray
Quoted 90 lines · Marc Zyngier
On Fri, 20 Dec 2019 14:30:16 +0000
Andrew Murray <[email protected]> wrote:[somehow managed not to do a reply all, re-sending]
Quoted 8 lines
From: Sudeep Holla <[email protected]>
Now that we can save/restore the full SPE controls, we can enable it if SPE is setup and ready to use in KVM. It's supported in KVM only if all the CPUs in the system supports SPE.
However to support heterogenous systems, we need to move the check if host supports SPE and do a partial save/restore.
No. Let's just not go down that path. For now, KVM on heterogeneous systems do not get SPE. If SPE has been enabled on a guest and a CPU comes up without SPE, this CPU should fail to boot (same as exposing a feature to userspace).
Quoted 59 lines
Signed-off-by Sudeep Holla <[email protected]>Signed-off-by Andrew Murray <[email protected]>--- arch/arm64/kvm/hyp/debug-sr.c | 33 ++++++++++++++++----------------- include/kvm/arm_spe.h | 6 ++++++ 2 files changed, 22 insertions(+), 17 deletions(-)arch/arm64/kvm/hyp/debug-sr.c +7
-9@@ -86,18 +86,13 @@ } static void __hyp_text -__debug_save_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt) +__debug_save_spe_context (struct kvm_cpu_context *ctxt,bool full_ctxt) {u64 reg;/* Clear pmscr in case of early return */ ctxt->sys_regs[PMSCR_EL1 ] =0 ; -/* SPE present on this CPU? */ -if (!cpuid_feature_extract_unsigned_field (read_sysreg (id_aa64dfr0_el1), -ID_AA64DFR0_PMSVER_SHIFT )) -return ; -/* Yes; is it owned by higher EL? */ reg =read_sysreg_s (SYS_PMBIDR_EL1 );if (reg &BIT (SYS_PMBIDR_EL1_P_SHIFT ))@@ -142,7 +137,7 @@ __debug_save_spe_nvhe(struct kvm_cpu_context *ctxt, bool full_ctxt) } static void __hyp_text -__debug_restore_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt) +__debug_restore_spe_context (struct kvm_cpu_context *ctxt,bool full_ctxt) {if (!ctxt->sys_regs[PMSCR_EL1 ])return ;@@ -210,11 +205,14 @@ void __hyp_text __debug_restore_guest_context(struct kvm_vcpu *vcpu)struct kvm_guest_debug_arch *host_dbg;struct kvm_guest_debug_arch *guest_dbg; + host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); + guest_ctxt = &vcpu->arch.ctxt; + +__debug_restore_spe_context (guest_ctxt,kvm_arm_spe_v1_ready (vcpu)); +if (!(vcpu->arch.flags &KVM_ARM64_DEBUG_DIRTY ))return ; - host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); - guest_ctxt = &vcpu->arch.ctxt; host_dbg = &vcpu->arch.host_debug_state.regs; guest_dbg =kern_hyp_va (vcpu->arch.debug_ptr);arch/arm64/kvm/hyp/debug-sr.c void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) · line 233
Quoted 6 lines
arch/arm64/kvm/hyp/debug-sr.c +1
-2@@ -232,8 +230,7 @@ void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); guest_ctxt = &vcpu->arch.ctxt; -if (!has_vhe ()) -__debug_restore_spe_nvhe (host_ctxt, false); +__debug_restore_spe_context (host_ctxt,kvm_arm_spe_v1_ready (vcpu));So you now do an unconditional save/restore on the exit path for VHE as well? Even if the host isn't using the SPE HW? That's not acceptable as, in most cases, only the host /or/ the guest will use SPE. Here, you put a measurable overhead on each exit.
Quoted 7 lines
Quoted 5 lines
If the host is not using SPE, then the restore/save should happen in vcpu_load/vcpu_put. Only if the host is using SPE should you do something in the run loop. Of course, this only applies to VHE and non-VHE must switch eagerly.
On VHE where SPE is used in the guest only - we save/restore in vcpu_load/put.
Yes.
Quoted 1 line
On VHE where SPE is used in the host only - we save/restore in the run loop.
Why? If only the host is using SPE, why should we do *anything at all*?
Quoted 6 lines
On VHE where SPE is used in guest and host - we save/restore in the run loop.
As the guest can't trace EL2 it doesn't matter if we restore guest SPE early in the vcpu_load/put functions. (I assume it doesn't matter that we restore an EL0/EL1 profiling buffer address at this point and enable tracing given that there is nothing to trace until entering the guest).
As long as you do it after the EL1 sysregs have need restored so that the SPE HW has a valid context, we should be fine. Don't restore it before that point though (you have no idea whether the SPE HW can do speculative memory accesses that would use the wrong page tables).
Quoted 2 lines
However the reason for moving save/restore to vcpu_load/put when the host is using SPE is to minimise the host EL2 black-out window.
You should move it to *the run loop* when both host and guest are using SPE.
Quoted 4 lines
On nVHE we always save/restore in the run loop. For the SPE guest-use-only use-case we can't save/restore in vcpu_load/put - because the guest runs at the same ELx level as the host - and thus doing so would result in the guest tracing part of the host.
Not only. It would actively corrupt memory in the host by using the wrong page tables.
Quoted 3 lines
Though if we determine that (for nVHE systems) the guest SPE is profiling only EL0 - then we could also save/restore in vcpu_load/put where SPE is only being used in the guest.
Same as above: wrong MM context, speculation, potential memory corruption.
Quoted 1 line
Does that make sense, are my reasons correct?
Not entirely. I think you should use the following table:
VHE | Host-SPE | Guest-SPE | Switch location 0 | 0 | 0 | none 0 | 0 | 1 | run loop 0 | 1 | 0 | run loop 0 | 1 | 1 | run loop 1 | 0 | 0 | none 1 | 0 | 1 | load/put 1 | 1 | 0 | none 1 | 1 | 1 | run loop
Thanks,
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
Re: [PATCH v2 09/18] arm64: KVM: enable conditional save/restore full SPE profiling buffer controls
Quoted 103 lines · Marc Zyngier
Quoted 93 lines · Andrew Murray
Quoted 92 lines · Marc Zyngier
On Fri, 20 Dec 2019 14:30:16 +0000
Andrew Murray <[email protected]> wrote:[somehow managed not to do a reply all, re-sending]
Quoted 8 lines
From: Sudeep Holla <[email protected]>
Now that we can save/restore the full SPE controls, we can enable it if SPE is setup and ready to use in KVM. It's supported in KVM only if all the CPUs in the system supports SPE.
However to support heterogenous systems, we need to move the check if host supports SPE and do a partial save/restore.
No. Let's just not go down that path. For now, KVM on heterogeneous systems do not get SPE. If SPE has been enabled on a guest and a CPU comes up without SPE, this CPU should fail to boot (same as exposing a feature to userspace).
Quoted 59 lines
Signed-off-by Sudeep Holla <[email protected]>Signed-off-by Andrew Murray <[email protected]>--- arch/arm64/kvm/hyp/debug-sr.c | 33 ++++++++++++++++----------------- include/kvm/arm_spe.h | 6 ++++++ 2 files changed, 22 insertions(+), 17 deletions(-)arch/arm64/kvm/hyp/debug-sr.c +7
-9@@ -86,18 +86,13 @@ } static void __hyp_text -__debug_save_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt) +__debug_save_spe_context (struct kvm_cpu_context *ctxt,bool full_ctxt) {u64 reg;/* Clear pmscr in case of early return */ ctxt->sys_regs[PMSCR_EL1 ] =0 ; -/* SPE present on this CPU? */ -if (!cpuid_feature_extract_unsigned_field (read_sysreg (id_aa64dfr0_el1), -ID_AA64DFR0_PMSVER_SHIFT )) -return ; -/* Yes; is it owned by higher EL? */ reg =read_sysreg_s (SYS_PMBIDR_EL1 );if (reg &BIT (SYS_PMBIDR_EL1_P_SHIFT ))@@ -142,7 +137,7 @@ __debug_save_spe_nvhe(struct kvm_cpu_context *ctxt, bool full_ctxt) } static void __hyp_text -__debug_restore_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt) +__debug_restore_spe_context (struct kvm_cpu_context *ctxt,bool full_ctxt) {if (!ctxt->sys_regs[PMSCR_EL1 ])return ;@@ -210,11 +205,14 @@ void __hyp_text __debug_restore_guest_context(struct kvm_vcpu *vcpu)struct kvm_guest_debug_arch *host_dbg;struct kvm_guest_debug_arch *guest_dbg; + host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); + guest_ctxt = &vcpu->arch.ctxt; + +__debug_restore_spe_context (guest_ctxt,kvm_arm_spe_v1_ready (vcpu)); +if (!(vcpu->arch.flags &KVM_ARM64_DEBUG_DIRTY ))return ; - host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); - guest_ctxt = &vcpu->arch.ctxt; host_dbg = &vcpu->arch.host_debug_state.regs; guest_dbg =kern_hyp_va (vcpu->arch.debug_ptr);arch/arm64/kvm/hyp/debug-sr.c void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) · line 233
Quoted 6 lines
arch/arm64/kvm/hyp/debug-sr.c +1
-2@@ -232,8 +230,7 @@ void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); guest_ctxt = &vcpu->arch.ctxt; -if (!has_vhe ()) -__debug_restore_spe_nvhe (host_ctxt, false); +__debug_restore_spe_context (host_ctxt,kvm_arm_spe_v1_ready (vcpu));So you now do an unconditional save/restore on the exit path for VHE as
well? Even if the host isn't using the SPE HW? That's not acceptable as, in most cases, only the host /or/ the guest will use SPE. Here, you
put a measurable overhead on each exit.Quoted 8 lines
Quoted 5 lines
If the host is not using SPE, then the restore/save should happen in vcpu_load/vcpu_put. Only if the host is using SPE should you do something in the run loop. Of course, this only applies to VHE and non-VHE must switch eagerly.
On VHE where SPE is used in the guest only - we save/restore in vcpu_load/put.
Quoted 6 lines
Yes.
Quoted 2 lines
On VHE where SPE is used in the host only - we save/restore in the run loop.
Why? If only the host is using SPE, why should we do *anything at all*?
Oh yeah of course, we trap them in this case.
(Do I understand correctly that we don't/can't trap them for nVHE? - and so we should save/restore them for this use-case in nVHE)
Quoted 12 lines
Quoted 11 lines
On VHE where SPE is used in guest and host - we save/restore in the run loop.
As the guest can't trace EL2 it doesn't matter if we restore guest SPE early
in the vcpu_load/put functions. (I assume it doesn't matter that we restore
an EL0/EL1 profiling buffer address at this point and enable tracing given
that there is nothing to trace until entering the guest).
Quoted 7 lines
As long as you do it after the EL1 sysregs have need restored so that the SPE
HW has a valid context, we should be fine. Don't restore it before that point
though (you have no idea whether the SPE HW can do speculative memory accesses
that would use the wrong page tables).
Right, so don't enable tracing until SPE has a valid context. I understand that to mean at least the SPE buffer address registers (PMBPTR, PMBLIMITR) in the right context with respect to the E2PB bits (translation regime) and having those tables mapped in (which I think relate to the __activateX, __sysreg_restore_guest_stateX type of calls in kvm_vcpu_run_X right?).
I think that means we can restore the registers no earler than vcpu_load/put but we can't re-enable the tracing (PMSCR) until no earlier than just before __set_guest_arch_workaround_state. I think that applies to both VHE and nVHE?
Quoted 30 lines
Quoted 4 lines
However the reason for moving save/restore to vcpu_load/put when the host is
using SPE is to minimise the host EL2 black-out window.You should move it to *the run loop* when both host and guest are using SPE.
Quoted 7 lines
On nVHE we always save/restore in the run loop. For the SPE guest-use-only
use-case we can't save/restore in vcpu_load/put - because the guest runs at
the same ELx level as the host - and thus doing so would result in the guest
tracing part of the host.Not only. It would actively corrupt memory in the host by using the wrong page tables.
Quoted 5 lines
Though if we determine that (for nVHE systems) the guest SPE is profiling only
EL0 - then we could also save/restore in vcpu_load/put where SPE is only being
used in the guest.Same as above: wrong MM context, speculation, potential memory corruption.
Quoted 1 line
Does that make sense, are my reasons correct?
Not entirely. I think you should use the following table:
Quoted 9 lines
VHE | Host-SPE | Guest-SPE | Switch location 0 | 0 | 0 | none 0 | 0 | 1 | run loop 0 | 1 | 0 | run loop 0 | 1 | 1 | run loop 1 | 0 | 0 | none 1 | 0 | 1 | load/put 1 | 1 | 0 | none 1 | 1 | 1 | run loop
Thanks,
Andrew Murray
Quoted 6 lines
Thanks,
M.Signature · 1 line
Jazz is not dead. It just smells funny...
Marc Zyngier
maintainerRe: [PATCH v2 09/18] arm64: KVM: enable conditional save/restore full SPE profiling buffer controls
Quoted 109 lines · Andrew Murray
Quoted 108 lines · Marc Zyngier
Quoted 93 lines · Andrew Murray
Quoted 92 lines · Marc Zyngier
On Fri, 20 Dec 2019 14:30:16 +0000
Andrew Murray <[email protected]> wrote:[somehow managed not to do a reply all, re-sending]
Quoted 8 lines
From: Sudeep Holla <[email protected]>
Now that we can save/restore the full SPE controls, we can enable it if SPE is setup and ready to use in KVM. It's supported in KVM only if all the CPUs in the system supports SPE.
However to support heterogenous systems, we need to move the check if host supports SPE and do a partial save/restore.
No. Let's just not go down that path. For now, KVM on heterogeneous systems do not get SPE. If SPE has been enabled on a guest and a CPU comes up without SPE, this CPU should fail to boot (same as exposing a feature to userspace).
Quoted 59 lines
Signed-off-by Sudeep Holla <[email protected]>Signed-off-by Andrew Murray <[email protected]>--- arch/arm64/kvm/hyp/debug-sr.c | 33 ++++++++++++++++----------------- include/kvm/arm_spe.h | 6 ++++++ 2 files changed, 22 insertions(+), 17 deletions(-)arch/arm64/kvm/hyp/debug-sr.c +7
-9@@ -86,18 +86,13 @@ } static void __hyp_text -__debug_save_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt) +__debug_save_spe_context (struct kvm_cpu_context *ctxt,bool full_ctxt) {u64 reg;/* Clear pmscr in case of early return */ ctxt->sys_regs[PMSCR_EL1 ] =0 ; -/* SPE present on this CPU? */ -if (!cpuid_feature_extract_unsigned_field (read_sysreg (id_aa64dfr0_el1), -ID_AA64DFR0_PMSVER_SHIFT )) -return ; -/* Yes; is it owned by higher EL? */ reg =read_sysreg_s (SYS_PMBIDR_EL1 );if (reg &BIT (SYS_PMBIDR_EL1_P_SHIFT ))@@ -142,7 +137,7 @@ __debug_save_spe_nvhe(struct kvm_cpu_context *ctxt, bool full_ctxt) } static void __hyp_text -__debug_restore_spe_nvhe (struct kvm_cpu_context *ctxt,bool full_ctxt) +__debug_restore_spe_context (struct kvm_cpu_context *ctxt,bool full_ctxt) {if (!ctxt->sys_regs[PMSCR_EL1 ])return ;@@ -210,11 +205,14 @@ void __hyp_text __debug_restore_guest_context(struct kvm_vcpu *vcpu)struct kvm_guest_debug_arch *host_dbg;struct kvm_guest_debug_arch *guest_dbg; + host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); + guest_ctxt = &vcpu->arch.ctxt; + +__debug_restore_spe_context (guest_ctxt,kvm_arm_spe_v1_ready (vcpu)); +if (!(vcpu->arch.flags &KVM_ARM64_DEBUG_DIRTY ))return ; - host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); - guest_ctxt = &vcpu->arch.ctxt; host_dbg = &vcpu->arch.host_debug_state.regs; guest_dbg =kern_hyp_va (vcpu->arch.debug_ptr);arch/arm64/kvm/hyp/debug-sr.c void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) · line 233
Quoted 6 lines
arch/arm64/kvm/hyp/debug-sr.c +1
-2@@ -232,8 +230,7 @@ void __hyp_text __debug_restore_host_context(struct kvm_vcpu *vcpu) host_ctxt =kern_hyp_va (vcpu->arch.host_cpu_context); guest_ctxt = &vcpu->arch.ctxt; -if (!has_vhe ()) -__debug_restore_spe_nvhe (host_ctxt, false); +__debug_restore_spe_context (host_ctxt,kvm_arm_spe_v1_ready (vcpu));So you now do an unconditional save/restore on the exit path for VHE as
well? Even if the host isn't using the SPE HW? That's not acceptable as, in most cases, only the host /or/ the guest will use SPE. Here, you
put a measurable overhead on each exit.Quoted 8 lines
Quoted 5 lines
If the host is not using SPE, then the restore/save should happen in vcpu_load/vcpu_put. Only if the host is using SPE should you do something in the run loop. Of course, this only applies to VHE and non-VHE must switch eagerly.
On VHE where SPE is used in the guest only - we save/restore in vcpu_load/put.
Yes.
Quoted 2 lines
On VHE where SPE is used in the host only - we save/restore in the run loop.
Quoted 6 lines
Quoted 1 line
Why? If only the host is using SPE, why should we do *anything at all*?
Oh yeah of course, we trap them in this case.
(Do I understand correctly that we don't/can't trap them for nVHE? - and so we should save/restore them for this use-case in nVHE)
We can always trap. Otherwise we wouldn't be able to hide the feature from the guest.
Quoted 19 lines
Quoted 11 lines
Quoted 10 lines
On VHE where SPE is used in guest and host - we save/restore in the run loop.
As the guest can't trace EL2 it doesn't matter if we restore guest SPE early
in the vcpu_load/put functions. (I assume it doesn't matter that we restore
an EL0/EL1 profiling buffer address at this point and enable tracing given
that there is nothing to trace until entering the guest).Quoted 7 lines
As long as you do it after the EL1 sysregs have need restored so that the SPE
HW has a valid context, we should be fine. Don't restore it before that point
though (you have no idea whether the SPE HW can do speculative memory accesses
that would use the wrong page tables).
Quoted 5 lines
Right, so don't enable tracing until SPE has a valid context. I understand that to mean at least the SPE buffer address registers (PMBPTR, PMBLIMITR) in the right context with respect to the E2PB bits (translation regime) and having those tables mapped in (which I think relate to the
__activateX,__sysreg_restore_guest_stateXtype of calls inkvm_vcpu_run_Xright?).
The full MM context has to be in place before you can do anything. This means at least TTBR*_EL1, TCR_EL1 and co. But maybe this note in the SPE architecture document would allow us to relax things:
"The Statistical Profiling Extension is always disabled if the owning Exception level is a lower Exception level than the current Exception level."
So as long as you restore the guest state from EL2, SPE should be disabled.
Quoted 3 lines
I think that means we can restore the registers no earler than vcpu_load/put but we can't re-enable the tracing (PMSCR) until no earlier than just before
__set_guest_arch_workaround_state. I think that applies to both VHE and nVHE?
I'm sorry, but I don't understand what you mean.
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
PATCH v2 10/18 arm64: KVM/debug: use EL1&0 stage 1 translation regime
In reply to Andrew MurrayFrom: Sudeep Holla <[email protected]>
Now that we have all the save/restore mechanism in place, lets enable the translation regime used by buffer from EL2 stage 1 to EL1 stage 1 on VHE systems.
--- arch/arm64/kvm/hyp/switch.c | 2 ++ 1 file changed, 2 insertions(+)
arch/arm64/kvm/hyp/switch.c +2 -0
@@ -100,6 +100,7 @@ static void activate_traps_vhe(struct kvm_vcpu *vcpu)write_sysreg (val, cpacr_el1); +write_sysreg (vcpu->arch.mdcr_el2 |3 <<MDCR_EL2_E2PB_SHIFT , mdcr_el2);write_sysreg (kvm_get_hyp_vector (), vbar_el1); }NOKPROBE_SYMBOL (activate_traps_vhe);
@@ -117,6 +118,7 @@ static void __hyp_text __activate_traps_nvhe(struct kvm_vcpu *vcpu)__activate_traps_fpsimd32 (vcpu); } +write_sysreg (vcpu->arch.mdcr_el2 |3 <<MDCR_EL2_E2PB_SHIFT , mdcr_el2);write_sysreg (val, cptr_el2);if (cpus_have_const_cap (ARM64_WORKAROUND_1319367 )) {
Signature · 6 lines
2.21.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerRe: [PATCH v2 10/18] arm64: KVM/debug: use EL1&0 stage 1 translation regime
Quoted 9 lines · Andrew Murray
From: Sudeep Holla <[email protected]>
Now that we have all the save/restore mechanism in place, lets enable the translation regime used by buffer from EL2 stage 1 to EL1 stage 1 on VHE systems.
Signed-off-by Sudeep Holla <[email protected]>[ Reword commit, don't trap to EL2 ]
Not trapping to EL2 for the case where we don't allow SPE in the guest is not acceptable.
Quoted 18 lines
Signed-off-by Andrew Murray <[email protected]>--- arch/arm64/kvm/hyp/switch.c | 2 ++ 1 file changed, 2 insertions(+)arch/arm64/kvm/hyp/switch.c +1
-0@@ -100,6 +100,7 @@ static void activate_traps_vhe(struct kvm_vcpu *vcpu)write_sysreg (val, cpacr_el1); +write_sysreg (vcpu->arch.mdcr_el2 |3 <<MDCR_EL2_E2PB_SHIFT , mdcr_el2);write_sysreg (kvm_get_hyp_vector (), vbar_el1); }NOKPROBE_SYMBOL (activate_traps_vhe);
arch/arm64/kvm/hyp/switch.c static void __hyp_text __activate_traps_nvhe(struct kvm_vcpu *vcpu) · line 121
Quoted 4 lines
arch/arm64/kvm/hyp/switch.c +1
-0@@ -117,6 +118,7 @@ static void __hyp_text __activate_traps_nvhe(struct kvm_vcpu *vcpu)__activate_traps_fpsimd32 (vcpu); } +write_sysreg (vcpu->arch.mdcr_el2 |3 <<MDCR_EL2_E2PB_SHIFT , mdcr_el2);
There is a _MASK macro that can replace this '3', and is in keeping with the rest of the code.
It still remains that it looks like the wrong place to do this, and vcpu_load seems much better. Why should you write to mdcr_el2 on each entry to the guest, since you know whether it has SPE enabled at the point where it gets scheduled?
M.
Signature · 1 line
Jazz is not dead, it just smells funny.
Re: [PATCH v2 10/18] arm64: KVM/debug: use EL1&0 stage 1 translation regime
Quoted 9 lines · Marc Zyngier
Quoted 7 lines · Andrew Murray
From: Sudeep Holla <[email protected]>
Now that we have all the save/restore mechanism in place, lets enable the translation regime used by buffer from EL2 stage 1 to EL1 stage 1 on VHE systems.
Quoted 5 lines
Quoted 2 lines
Signed-off-by Sudeep Holla <[email protected]>[ Reword commit, don't trap to EL2 ]Not trapping to EL2 for the case where we don't allow SPE in the guest is not acceptable.
Yes understood (because of this I had meant to send the series as RFC btw).
Quoted 22 lines
Quoted 22 lines
Signed-off-by Andrew Murray <[email protected]>--- arch/arm64/kvm/hyp/switch.c | 2 ++ 1 file changed, 2 insertions(+)arch/arm64/kvm/hyp/switch.c +1
-0@@ -100,6 +100,7 @@ static void activate_traps_vhe(struct kvm_vcpu *vcpu)write_sysreg (val, cpacr_el1); +write_sysreg (vcpu->arch.mdcr_el2 |3 <<MDCR_EL2_E2PB_SHIFT , mdcr_el2);write_sysreg (kvm_get_hyp_vector (), vbar_el1); }NOKPROBE_SYMBOL (activate_traps_vhe);@@ -117,6 +118,7 @@ static void __hyp_text __activate_traps_nvhe(struct kvm_vcpu *vcpu)__activate_traps_fpsimd32 (vcpu); }
Quoted 4 lines
arch/arm64/kvm/hyp/switch.c static void __hyp_text __activate_traps_nvhe(struct kvm_vcpu *vcpu) · line 121
Quoted 1 line
arch/arm64/kvm/hyp/switch.c +1
-0+write_sysreg (vcpu->arch.mdcr_el2 |3 <<MDCR_EL2_E2PB_SHIFT , mdcr_el2);There is a
_MASKmacro that can replace this '3', and is in keeping with the rest of the code.
OK.
Quoted 5 lines
It still remains that it looks like the wrong place to do this, and
vcpu_loadseems much better. Why should you write tomdcr_el2on each entry to the guest, since you know whether it has SPE enabled at the point where it gets scheduled?
Yes OK, I'll move what I can to vcpu_load.
Thanks,
Andrew Murray
Quoted 5 lines
M.Signature · 1 line
Jazz is not dead, it just smells funny.
Re: [PATCH v2 10/18] arm64: KVM/debug: use EL1&0 stage 1 translation regime
In reply to Marc ZyngierQuoted 41 lines · Marc Zyngier
Quoted 9 lines · Andrew Murray
From: Sudeep Holla <[email protected]>
Now that we have all the save/restore mechanism in place, lets enable the translation regime used by buffer from EL2 stage 1 to EL1 stage 1 on VHE systems.
Signed-off-by Sudeep Holla <[email protected]>[ Reword commit, don't trap to EL2 ]Not trapping to EL2 for the case where we don't allow SPE in the guest is not acceptable.
Quoted 18 lines
Signed-off-by Andrew Murray <[email protected]>--- arch/arm64/kvm/hyp/switch.c | 2 ++ 1 file changed, 2 insertions(+)arch/arm64/kvm/hyp/switch.c +1
-0@@ -100,6 +100,7 @@ static void activate_traps_vhe(struct kvm_vcpu *vcpu)write_sysreg (val, cpacr_el1); +write_sysreg (vcpu->arch.mdcr_el2 |3 <<MDCR_EL2_E2PB_SHIFT , mdcr_el2);write_sysreg (kvm_get_hyp_vector (), vbar_el1); }NOKPROBE_SYMBOL (activate_traps_vhe);arch/arm64/kvm/hyp/switch.c static void __hyp_text __activate_traps_nvhe(struct kvm_vcpu *vcpu) · line 121
Quoted 4 lines
arch/arm64/kvm/hyp/switch.c +1
-0@@ -117,6 +118,7 @@ static void __hyp_text __activate_traps_nvhe(struct kvm_vcpu *vcpu)__activate_traps_fpsimd32 (vcpu); } +write_sysreg (vcpu->arch.mdcr_el2 |3 <<MDCR_EL2_E2PB_SHIFT , mdcr_el2);There is a
_MASKmacro that can replace this '3', and is in keeping with the rest of the code.
Quoted 4 lines
It still remains that it looks like the wrong place to do this, and
vcpu_loadseems much better. Why should you write tomdcr_el2on each entry to the guest, since you know whether it has SPE enabled at the point where it gets scheduled?
For nVHE, the only reason we'd want to change E2PB on entry/exit of guest would be if the host is also using SPE. If the host is using SPE whilst the vcpu is 'loaded' but we're not in the guest, then host SPE could raise an interrupt - we need the E2PB bits to allow access from EL1 (host).
Thanks,
Andrew Murray
Quoted 5 lines
M.Signature · 1 line
Jazz is not dead, it just smells funny.
Marc Zyngier
maintainerRe: [PATCH v2 10/18] arm64: KVM/debug: use EL1&0 stage 1 translation regime
Quoted 47 lines · Andrew Murray
Quoted 41 lines · Marc Zyngier
Quoted 9 lines · Andrew Murray
From: Sudeep Holla <[email protected]>
Now that we have all the save/restore mechanism in place, lets enable the translation regime used by buffer from EL2 stage 1 to EL1 stage 1 on VHE systems.
Signed-off-by Sudeep Holla <[email protected]>[ Reword commit, don't trap to EL2 ]Not trapping to EL2 for the case where we don't allow SPE in the guest is not acceptable.
Quoted 18 lines
Signed-off-by Andrew Murray <[email protected]>--- arch/arm64/kvm/hyp/switch.c | 2 ++ 1 file changed, 2 insertions(+)arch/arm64/kvm/hyp/switch.c +1
-0@@ -100,6 +100,7 @@ static void activate_traps_vhe(struct kvm_vcpu *vcpu)write_sysreg (val, cpacr_el1); +write_sysreg (vcpu->arch.mdcr_el2 |3 <<MDCR_EL2_E2PB_SHIFT , mdcr_el2);write_sysreg (kvm_get_hyp_vector (), vbar_el1); }NOKPROBE_SYMBOL (activate_traps_vhe);arch/arm64/kvm/hyp/switch.c static void __hyp_text __activate_traps_nvhe(struct kvm_vcpu *vcpu) · line 121
Quoted 4 lines
arch/arm64/kvm/hyp/switch.c +1
-0@@ -117,6 +118,7 @@ static void __hyp_text __activate_traps_nvhe(struct kvm_vcpu *vcpu)__activate_traps_fpsimd32 (vcpu); } +write_sysreg (vcpu->arch.mdcr_el2 |3 <<MDCR_EL2_E2PB_SHIFT , mdcr_el2);There is a
_MASKmacro that can replace this '3', and is in keeping with the rest of the code.Quoted 4 lines
It still remains that it looks like the wrong place to do this, and
vcpu_loadseems much better. Why should you write tomdcr_el2on each entry to the guest, since you know whether it has SPE enabled at the point where it gets scheduled?
Quoted 4 lines
For nVHE, the only reason we'd want to change E2PB on entry/exit of guest would be if the host is also using SPE. If the host is using SPE whilst the vcpu is 'loaded' but we're not in the guest, then host SPE could raise an interrupt - we need the E2PB bits to allow access from EL1 (host).
My comment was of course for VHE. nVHE hardly makes use of load/put at all, for obvious reasons.
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
PATCH v2 11/18 KVM: arm64: don't trap Statistical Profiling controls to EL2
In reply to Andrew MurrayAs we now save/restore the profiler state there is no need to trap accesses to the statistical profiling controls. Let's unset the _TPMS bit.
--- arch/arm64/kvm/debug.c | 2 -- 1 file changed, 2 deletions(-)
arch/arm64/kvm/debug.c +0 -2
@@ -88,7 +88,6 @@ void kvm_arm_reset_debug_ptr(struct kvm_vcpu *vcpu)* - Performance monitors (MDCR_EL2_TPM/MDCR_EL2_TPMCR) * - Debug ROM Address (MDCR_EL2_TDRA) * - OS related registers (MDCR_EL2_TDOSA) -* - Statistical profiler (MDCR_EL2_TPMS/MDCR_EL2_E2PB) * * Additionally, KVM only traps guest accesses to the debug registers if * the guest is not actively using them (see the KVM_ARM64_DEBUG_DIRTY
@@ -111,7 +110,6 @@ void kvm_arm_setup_debug(struct kvm_vcpu *vcpu)*/ vcpu->arch.mdcr_el2 =__this_cpu_read (mdcr_el2) &MDCR_EL2_HPMN_MASK ; vcpu->arch.mdcr_el2 |= (MDCR_EL2_TPM | -MDCR_EL2_TPMS |MDCR_EL2_TPMCR |MDCR_EL2_TDRA |MDCR_EL2_TDOSA );
Signature · 6 lines
2.21.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Mark Rutland
maintainerRe: [PATCH v2 11/18] KVM: arm64: don't trap Statistical Profiling controls to EL2
Quoted 22 lines · Andrew Murray
As we now save/restore the profiler state there is no need to trap accesses to the statistical profiling controls. Let's unset the
_TPMSbit.Signed-off-by Andrew Murray <[email protected]>--- arch/arm64/kvm/debug.c | 2 -- 1 file changed, 2 deletions(-)arch/arm64/kvm/debug.c +0
-1@@ -88,7 +88,6 @@ void kvm_arm_reset_debug_ptr(struct kvm_vcpu *vcpu)* - Performance monitors (MDCR_EL2_TPM/MDCR_EL2_TPMCR) * - Debug ROM Address (MDCR_EL2_TDRA) * - OS related registers (MDCR_EL2_TDOSA) -* - Statistical profiler (MDCR_EL2_TPMS/MDCR_EL2_E2PB) * * Additionally, KVM only traps guest accesses to the debug registers if * the guest is not actively using them (see the KVM_ARM64_DEBUG_DIRTY
arch/arm64/kvm/debug.c void kvm_arm_setup_debug(struct kvm_vcpu *vcpu) · line 115
Quoted 7 lines
arch/arm64/kvm/debug.c +0
-1@@ -111,7 +110,6 @@ void kvm_arm_setup_debug(struct kvm_vcpu *vcpu)*/ vcpu->arch.mdcr_el2 =__this_cpu_read (mdcr_el2) &MDCR_EL2_HPMN_MASK ; vcpu->arch.mdcr_el2 |= (MDCR_EL2_TPM | -MDCR_EL2_TPMS |MDCR_EL2_TPMCR |MDCR_EL2_TDRA |MDCR_EL2_TDOSA );
I think that this should be conditional on some vcpu feature flag.
If nothing else, this could break existing migration cases otherwise.
Thanks,
Mark.
Marc Zyngier
maintainerRe: [PATCH v2 11/18] KVM: arm64: don't trap Statistical Profiling controls to EL2
In reply to Andrew MurrayQuoted 23 lines · Andrew Murray
As we now save/restore the profiler state there is no need to trap accesses to the statistical profiling controls. Let's unset the
_TPMSbit.Signed-off-by Andrew Murray <[email protected]>--- arch/arm64/kvm/debug.c | 2 -- 1 file changed, 2 deletions(-)arch/arm64/kvm/debug.c +0
-1@@ -88,7 +88,6 @@ void kvm_arm_reset_debug_ptr(struct kvm_vcpu *vcpu)* - Performance monitors (MDCR_EL2_TPM/MDCR_EL2_TPMCR) * - Debug ROM Address (MDCR_EL2_TDRA) * - OS related registers (MDCR_EL2_TDOSA) -* - Statistical profiler (MDCR_EL2_TPMS/MDCR_EL2_E2PB) * * Additionally, KVM only traps guest accesses to the debug registers if * the guest is not actively using them (see the KVM_ARM64_DEBUG_DIRTY
arch/arm64/kvm/debug.c void kvm_arm_setup_debug(struct kvm_vcpu *vcpu) · line 114
Quoted 4 lines
arch/arm64/kvm/debug.c +0
-1@@ -111,7 +110,6 @@ void kvm_arm_setup_debug(struct kvm_vcpu *vcpu)*/ vcpu->arch.mdcr_el2 =__this_cpu_read (mdcr_el2) &MDCR_EL2_HPMN_MASK ; vcpu->arch.mdcr_el2 |= (MDCR_EL2_TPM | - MDCR_EL2_TPMS |
No. This is an *optional* feature (the guest could not be presented with the SPE feature, or the the support simply not be compiled in).
If the guest is not allowed to see the feature, for whichever reason, the traps *must* be enabled and handled.
M.
Signature · 1 line
Jazz is not dead, it just smells funny.
Re: [PATCH v2 11/18] KVM: arm64: don't trap Statistical Profiling controls to EL2
Quoted 33 lines · Marc Zyngier
Quoted 23 lines · Andrew Murray
As we now save/restore the profiler state there is no need to trap accesses to the statistical profiling controls. Let's unset the
_TPMSbit.Signed-off-by Andrew Murray <[email protected]>--- arch/arm64/kvm/debug.c | 2 -- 1 file changed, 2 deletions(-)arch/arm64/kvm/debug.c +0
-1@@ -88,7 +88,6 @@ void kvm_arm_reset_debug_ptr(struct kvm_vcpu *vcpu)* - Performance monitors (MDCR_EL2_TPM/MDCR_EL2_TPMCR) * - Debug ROM Address (MDCR_EL2_TDRA) * - OS related registers (MDCR_EL2_TDOSA) -* - Statistical profiler (MDCR_EL2_TPMS/MDCR_EL2_E2PB) * * Additionally, KVM only traps guest accesses to the debug registers if * the guest is not actively using them (see the KVM_ARM64_DEBUG_DIRTY arch/arm64/kvm/debug.c void kvm_arm_setup_debug(struct kvm_vcpu *vcpu) · line 114
Quoted 4 lines
arch/arm64/kvm/debug.c +0
-1@@ -111,7 +110,6 @@ void kvm_arm_setup_debug(struct kvm_vcpu *vcpu)*/ vcpu->arch.mdcr_el2 =__this_cpu_read (mdcr_el2) &MDCR_EL2_HPMN_MASK ; vcpu->arch.mdcr_el2 |= (MDCR_EL2_TPM | - MDCR_EL2_TPMS |No. This is an *optional* feature (the guest could not be presented with the SPE feature, or the the support simply not be compiled in).
Quoted 2 lines
If the guest is not allowed to see the feature, for whichever reason, the traps *must* be enabled and handled.
I'll update this (and similar) to trap such registers when we don't support SPE in the guest.
My original concern in the cover letter was in how to prevent the guest from attempting to use these registers in the first place - I think the solution I was looking for is to trap-and-emulate ID_AA64DFR0_EL1 such that the PMSVer bits indicate that SPE is not emulated.
Thanks,
Andrew Murray
Quoted 5 lines
M.Signature · 1 line
Jazz is not dead, it just smells funny.
Marc Zyngier
maintainerRe: [PATCH v2 11/18] KVM: arm64: don't trap Statistical Profiling controls to EL2
Quoted 42 lines · Andrew Murray
Quoted 32 lines · Marc Zyngier
arch/arm64/kvm/debug.c void kvm_arm_reset_debug_ptr(struct kvm_vcpu
Quoted 15 lines · Andrew Murray
As we now save/restore the profiler state there is no need to trap accesses to the statistical profiling controls. Let's unset the
_TPMSbit.Signed-off-by Andrew Murray <[email protected]>--- arch/arm64/kvm/debug.c | 2 -- 1 file changed, 2 deletions(-)arch/arm64/kvm/debug.c +0
-0@@ -88,7 +88,6 @@ void kvm_arm_reset_debug_ptr(struct kvm_vcpu*vcpu)
arch/arm64/kvm/debug.c void kvm_arm_reset_debug_ptr(struct kvm_vcpu *vcpu)
Quoted 6 lines
arch/arm64/kvm/debug.c +0
-1* - Performance monitors (MDCR_EL2_TPM/MDCR_EL2_TPMCR) * - Debug ROM Address (MDCR_EL2_TDRA) * - OS related registers (MDCR_EL2_TDOSA) -* - Statistical profiler (MDCR_EL2_TPMS/MDCR_EL2_E2PB) * * Additionally, KVM only traps guest accesses to the debug registers if
arch/arm64/kvm/debug.c void kvm_arm_reset_debug_ptr(struct kvm_vcpu · line 93
Quoted 1 line
arch/arm64/kvm/debug.c +0
-0* the guest is not actively using them (see the
KVM_ARM64_DEBUG_DIRTYarch/arm64/kvm/debug.c void kvm_arm_setup_debug(struct kvm_vcpu
Quoted 1 line
arch/arm64/kvm/debug.c +0
-0@@ -111,7 +110,6 @@ void kvm_arm_setup_debug(struct kvm_vcpu*vcpu)
arch/arm64/kvm/debug.c void kvm_arm_setup_debug(struct kvm_vcpu · line 111
Quoted 2 lines
arch/arm64/kvm/debug.c +0
-0*/ vcpu->arch.mdcr_el2 = __this_cpu_read(mdcr_el2) &
MDCR_EL2_HPMN_MASK;Quoted 8 lines
arch/arm64/kvm/debug.c void kvm_arm_setup_debug(struct kvm_vcpu *vcpu) · line 114
Quoted 2 lines
arch/arm64/kvm/debug.c +0
-1vcpu->arch.mdcr_el2 |= (MDCR_EL2_TPM | - MDCR_EL2_TPMS |No. This is an *optional* feature (the guest could not be presented with the SPE feature, or the the support simply not be compiled in).
If the guest is not allowed to see the feature, for whichever reason, the traps *must* be enabled and handled.
Quoted 7 lines
I'll update this (and similar) to trap such registers when we don't support SPE in the guest.
My original concern in the cover letter was in how to prevent the guest from attempting to use these registers in the first place - I think the solution I was looking for is to trap-and-emulate
ID_AA64DFR0_EL1such that the PMSVer bits indicate that SPE is not emulated.
That, and active trapping of the SPE system registers resulting in injection of an UNDEF into the offending guest.
Thanks,
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
Re: [PATCH v2 11/18] KVM: arm64: don't trap Statistical Profiling controls to EL2
Quoted 48 lines · Marc Zyngier
Quoted 47 lines · Andrew Murray
Quoted 38 lines · Marc Zyngier
arch/arm64/kvm/debug.c void kvm_arm_reset_debug_ptr(struct kvm_vcpu
Quoted 15 lines · Andrew Murray
As we now save/restore the profiler state there is no need to trap accesses to the statistical profiling controls. Let's unset the
_TPMSbit.Signed-off-by Andrew Murray <[email protected]>--- arch/arm64/kvm/debug.c | 2 -- 1 file changed, 2 deletions(-)arch/arm64/kvm/debug.c +0
-0@@ -88,7 +88,6 @@ void kvm_arm_reset_debug_ptr(struct kvm_vcpu*vcpu)
arch/arm64/kvm/debug.c void kvm_arm_reset_debug_ptr(struct kvm_vcpu *vcpu)
Quoted 6 lines
arch/arm64/kvm/debug.c +0
-1* - Performance monitors (MDCR_EL2_TPM/MDCR_EL2_TPMCR) * - Debug ROM Address (MDCR_EL2_TDRA) * - OS related registers (MDCR_EL2_TDOSA) -* - Statistical profiler (MDCR_EL2_TPMS/MDCR_EL2_E2PB) * * Additionally, KVM only traps guest accesses to the debug registers if
arch/arm64/kvm/debug.c void kvm_arm_reset_debug_ptr(struct kvm_vcpu · line 93
Quoted 1 line
arch/arm64/kvm/debug.c +0
-0* the guest is not actively using them (see the
KVM_ARM64_DEBUG_DIRTYarch/arm64/kvm/debug.c void kvm_arm_setup_debug(struct kvm_vcpu
Quoted 1 line
arch/arm64/kvm/debug.c +0
-0@@ -111,7 +110,6 @@ void kvm_arm_setup_debug(struct kvm_vcpu*vcpu)
arch/arm64/kvm/debug.c void kvm_arm_setup_debug(struct kvm_vcpu · line 111
Quoted 2 lines
arch/arm64/kvm/debug.c +0
-0*/ vcpu->arch.mdcr_el2 = __this_cpu_read(mdcr_el2) &
MDCR_EL2_HPMN_MASK;arch/arm64/kvm/debug.c void kvm_arm_setup_debug(struct kvm_vcpu *vcpu) · line 114
Quoted 2 lines
arch/arm64/kvm/debug.c +0
-1vcpu->arch.mdcr_el2 |= (MDCR_EL2_TPM | - MDCR_EL2_TPMS |No. This is an *optional* feature (the guest could not be presented with the SPE feature, or the the support simply not be compiled in).
Quoted 3 lines
If the guest is not allowed to see the feature, for whichever reason,
the traps *must* be enabled and handled.I'll update this (and similar) to trap such registers when we don't support
SPE in the guest.
Quoted 8 lines
Quoted 5 lines
My original concern in the cover letter was in how to prevent the guest from attempting to use these registers in the first place - I think the solution I was looking for is to trap-and-emulate
ID_AA64DFR0_EL1such that
the PMSVer bits indicate that SPE is not emulated.That, and active trapping of the SPE system registers resulting in injection of an UNDEF into the offending guest.
Yes that's no problem.
Thanks,
Andrew Murray
Quoted 6 lines
Thanks,
M.Signature · 1 line
Jazz is not dead. It just smells funny...
Re: [PATCH v2 11/18] KVM: arm64: don't trap Statistical Profiling controls to EL2
Quoted 55 lines · Andrew Murray
Quoted 54 lines · Marc Zyngier
Quoted 52 lines · Andrew Murray
Quoted 38 lines · Marc Zyngier
arch/arm64/kvm/debug.c void kvm_arm_reset_debug_ptr(struct kvm_vcpu
Quoted 15 lines · Andrew Murray
As we now save/restore the profiler state there is no need to trap accesses to the statistical profiling controls. Let's unset the
_TPMSbit.Signed-off-by Andrew Murray <[email protected]>--- arch/arm64/kvm/debug.c | 2 -- 1 file changed, 2 deletions(-)arch/arm64/kvm/debug.c +0
-0@@ -88,7 +88,6 @@ void kvm_arm_reset_debug_ptr(struct kvm_vcpu*vcpu)
arch/arm64/kvm/debug.c void kvm_arm_reset_debug_ptr(struct kvm_vcpu *vcpu)
Quoted 6 lines
arch/arm64/kvm/debug.c +0
-1* - Performance monitors (MDCR_EL2_TPM/MDCR_EL2_TPMCR) * - Debug ROM Address (MDCR_EL2_TDRA) * - OS related registers (MDCR_EL2_TDOSA) -* - Statistical profiler (MDCR_EL2_TPMS/MDCR_EL2_E2PB) * * Additionally, KVM only traps guest accesses to the debug registers if
arch/arm64/kvm/debug.c void kvm_arm_reset_debug_ptr(struct kvm_vcpu · line 93
Quoted 1 line
arch/arm64/kvm/debug.c +0
-0* the guest is not actively using them (see the
KVM_ARM64_DEBUG_DIRTYarch/arm64/kvm/debug.c void kvm_arm_setup_debug(struct kvm_vcpu
Quoted 1 line
arch/arm64/kvm/debug.c +0
-0@@ -111,7 +110,6 @@ void kvm_arm_setup_debug(struct kvm_vcpu*vcpu)
arch/arm64/kvm/debug.c void kvm_arm_setup_debug(struct kvm_vcpu · line 111
Quoted 2 lines
arch/arm64/kvm/debug.c +0
-0*/ vcpu->arch.mdcr_el2 = __this_cpu_read(mdcr_el2) &
MDCR_EL2_HPMN_MASK;arch/arm64/kvm/debug.c void kvm_arm_setup_debug(struct kvm_vcpu *vcpu) · line 114
Quoted 2 lines
arch/arm64/kvm/debug.c +0
-1vcpu->arch.mdcr_el2 |= (MDCR_EL2_TPM | - MDCR_EL2_TPMS |No. This is an *optional* feature (the guest could not be presented with the SPE feature, or the the support simply not be compiled in).
Quoted 3 lines
If the guest is not allowed to see the feature, for whichever reason,
the traps *must* be enabled and handled.I'll update this (and similar) to trap such registers when we don't support
SPE in the guest.My original concern in the cover letter was in how to prevent the guest from attempting to use these registers in the first place - I think the solution I was looking for is to trap-and-emulate
ID_AA64DFR0_EL1such that
the PMSVer bits indicate that SPE is not emulated.
Quoted 4 lines
Quoted 2 lines
That, and active trapping of the SPE system registers resulting in injection of an UNDEF into the offending guest.
Yes that's no problem.
The spec says that 'direct access to [these registers] are UNDEFINED' - is it not more correct to handle this with trap_raz_wi than an undefined instruction?
Thanks,
Andrew Murray
Quoted 15 lines
Thanks,
Andrew Murray
Quoted 6 lines
Thanks,
M.Signature · 1 line
Jazz is not dead. It just smells funny..._______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Mark Rutland
maintainerRe: [PATCH v2 11/18] KVM: arm64: don't trap Statistical Profiling controls to EL2
Hi Andrew,
Quoted 9 lines · Andrew Murray
Quoted 8 lines · Andrew Murray
Quoted 7 lines · Marc Zyngier
Quoted 5 lines · Andrew Murray
My original concern in the cover letter was in how to prevent the guest from attempting to use these registers in the first place - I think the solution I was looking for is to trap-and-emulate
ID_AA64DFR0_EL1such that the PMSVer bits indicate that SPE is not emulated.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
That, and active trapping of the SPE system registers resulting in injection of an UNDEF into the offending guest.
Yes that's no problem.
The spec says that 'direct access to [these registers] are UNDEFINED' - is it not more correct to handle this with
trap_raz_withan an undefined instruction?
The term UNDEFINED specifically means treated as an undefined instruction. The Glossary in ARM DDI 0487E.a says for UNDEFINED:
| Indicates cases where an attempt to execute a particular encoding bit | pattern generates an exception, that is taken to the current Exception | level, or to the default Exception level for taking exceptions if the | UNDEFINED encoding was executed at EL0. This applies to: | | * Any encoding that is not allocated to any instruction. | | * Any encoding that is defined as never accessible at the current | Exception level. | | * Some cases where an enable, disable, or trap control means an | encoding is not accessible at the current Exception level.
So these should trigger an UNDEFINED exception rather than behaving as RAZ/WI.
Thanks,
Mark.
Re: [PATCH v2 11/18] KVM: arm64: don't trap Statistical Profiling controls to EL2
Quoted 36 lines · Mark Rutland
Hi Andrew,
Quoted 9 lines · Andrew Murray
Quoted 8 lines · Andrew Murray
Quoted 7 lines · Marc Zyngier
Quoted 5 lines · Andrew Murray
My original concern in the cover letter was in how to prevent the guest from attempting to use these registers in the first place - I think the solution I was looking for is to trap-and-emulate
ID_AA64DFR0_EL1such that the PMSVer bits indicate that SPE is not emulated.Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
That, and active trapping of the SPE system registers resulting in injection of an UNDEF into the offending guest.
Yes that's no problem.
The spec says that 'direct access to [these registers] are UNDEFINED' - is it not more correct to handle this with
trap_raz_withan an undefined instruction?The term UNDEFINED specifically means treated as an undefined instruction. The Glossary in ARM DDI 0487E.a says for UNDEFINED:
| Indicates cases where an attempt to execute a particular encoding bit | pattern generates an exception, that is taken to the current Exception | level, or to the default Exception level for taking exceptions if the | UNDEFINED encoding was executed at EL0. This applies to: | | * Any encoding that is not allocated to any instruction. | | * Any encoding that is defined as never accessible at the current | Exception level. | | * Some cases where an enable, disable, or trap control means an | encoding is not accessible at the current Exception level.
Quoted 2 lines
So these should trigger an UNDEFINED exception rather than behaving as RAZ/WI.
OK thanks for the clarification - I'll leave it as an undefined instruction.
Thanks,
Andrew Murray
Quoted 3 lines
Thanks,
Mark.
PATCH v2 12/18 KVM: arm64: add a new vcpu device control group for SPEv1
In reply to Andrew MurrayFrom: Sudeep Holla <[email protected]>
To configure the virtual SPEv1 overflow interrupt number, we use the vcpu kvm_device ioctl, encapsulating the KVM_ARM_VCPU_SPE_V1_IRQ attribute within the KVM_ARM_VCPU_SPE_V1_CTRL group.
After configuring the SPEv1, call the vcpu ioctl with attribute KVM_ARM_VCPU_SPE_V1_INIT to initialize the SPEv1.
--- Documentation/virt/kvm/devices/vcpu.txt | 28 ++++ arch/arm64/include/asm/kvm_host.h | 2 +- arch/arm64/include/uapi/asm/kvm.h | 4 + arch/arm64/kvm/Makefile | 1 + arch/arm64/kvm/guest.c | 6 + arch/arm64/kvm/reset.c | 3 + include/kvm/arm_spe.h | 45 +++++++ include/uapi/linux/kvm.h | 1 + virt/kvm/arm/arm.c | 1 + virt/kvm/arm/spe.c | 163 ++++++++++++++++++++++++ 10 files changed, 253 insertions(+), 1 deletion(-) create mode 100644 virt/kvm/arm/spe.c
Documentation/virt/kvm/devices/vcpu.txt +28 -0
@@ -74,3 +74,31 @@ Specifies the base address of the stolen time structure for this VCPU. The base address must be 64 byte aligned and exist within a valid guest memory region. See Documentation/virt/kvm/arm/pvtime.txt for more information including the layout of the stolen time structure. + +4. GROUP: KVM_ARM_VCPU_SPE_V1_CTRL +Architectures: ARM64 + +4.1. ATTRIBUTE: KVM_ARM_VCPU_SPE_V1_IRQ +Parameters: in kvm_device_attr.addr the address for SPE buffer overflow interrupt + is a pointer to an int +Returns: -EBUSY: The SPE overflow interrupt is already set + -ENXIO: The overflow interrupt not set when attempting to get it + -ENODEV: SPEv1 not supported + -EINVAL: Invalid SPE overflow interrupt number supplied or + trying to set the IRQ number without using an in-kernel + irqchip. + +A value describing the SPEv1 (Statistical Profiling Extension v1) overflow +interrupt number for this vcpu. This interrupt should be PPI and the interrupt +type and number must be same for each vcpu. + +4.2 ATTRIBUTE: KVM_ARM_VCPU_SPE_V1_INIT +Parameters: no additional parameter in kvm_device_attr.addr +Returns: -ENODEV: SPEv1 not supported or GIC not initialized + -ENXIO: SPEv1 not properly configured or in-kernel irqchip not + configured as required prior to calling this attribute + -EBUSY: SPEv1 already initialized + +Request the initialization of the SPEv1. If using the SPEv1 with an in-kernel +virtual GIC implementation, this must be done after initializing the in-kernel +irqchip.
arch/arm64/include/asm/kvm_host.h +1 -1
@@ -39,7 +39,7 @@#define KVM_MAX_VCPUS VGIC_V3_MAX_CPUS -#define KVM_VCPU_MAX_FEATURES 7 +#define KVM_VCPU_MAX_FEATURES 8#define KVM_REQ_SLEEP \ KVM_ARCH_REQ_FLAGS(0, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
arch/arm64/include/uapi/asm/kvm.h +4 -0
@@ -106,6 +106,7 @@ struct kvm_regs {#define KVM_ARM_VCPU_SVE 4/* enable SVE for this CPU */ #define KVM_ARM_VCPU_PTRAUTH_ADDRESS 5/* VCPU uses address authentication */ #define KVM_ARM_VCPU_PTRAUTH_GENERIC 6/* VCPU uses generic authentication */ +#define KVM_ARM_VCPU_SPE_V1 7/* Support guest SPEv1 */ struct kvm_vcpu_init {__u32 target;
@@ -326,6 +327,9 @@ struct kvm_vcpu_events {#define KVM_ARM_VCPU_TIMER_IRQ_PTIMER 1#define KVM_ARM_VCPU_PVTIME_CTRL 2#define KVM_ARM_VCPU_PVTIME_IPA 0 +#define KVM_ARM_VCPU_SPE_V1_CTRL 3 +#define KVM_ARM_VCPU_SPE_V1_IRQ 0 +#define KVM_ARM_VCPU_SPE_V1_INIT 1/* KVM_IRQ_LINE irq field index values */ #define KVM_ARM_IRQ_VCPU2_SHIFT 28
arch/arm64/kvm/Makefile +1 -0
@@ -37,3 +37,4 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-debug.o kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/irqchip.o kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/arch_timer.o kvm-$(CONFIG_KVM_ARM_PMU) += $(KVM)/arm/pmu.o +kvm-$(CONFIG_KVM_ARM_SPE) += $(KVM)/arm/spe.o
arch/arm64/kvm/guest.c +6 -0
@@ -874,6 +874,8 @@ int kvm_arm_vcpu_arch_set_attr(struct kvm_vcpu *vcpu,break ;case KVM_ARM_VCPU_PVTIME_CTRL : ret =kvm_arm_pvtime_set_attr (vcpu, attr); +case KVM_ARM_VCPU_SPE_V1_CTRL : + ret =kvm_arm_spe_v1_set_attr (vcpu, attr);break ;default : ret = -ENXIO ;
@@ -897,6 +899,8 @@ int kvm_arm_vcpu_arch_get_attr(struct kvm_vcpu *vcpu,break ;case KVM_ARM_VCPU_PVTIME_CTRL : ret =kvm_arm_pvtime_get_attr (vcpu, attr); +case KVM_ARM_VCPU_SPE_V1_CTRL : + ret =kvm_arm_spe_v1_get_attr (vcpu, attr);break ;default : ret = -ENXIO ;
@@ -920,6 +924,8 @@ int kvm_arm_vcpu_arch_has_attr(struct kvm_vcpu *vcpu,break ;case KVM_ARM_VCPU_PVTIME_CTRL : ret =kvm_arm_pvtime_has_attr (vcpu, attr); +case KVM_ARM_VCPU_SPE_V1_CTRL : + ret =kvm_arm_spe_v1_has_attr (vcpu, attr);break ;default : ret = -ENXIO ;
arch/arm64/kvm/reset.c +3 -0
@@ -80,6 +80,9 @@ int kvm_arch_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_ARM_INJECT_SERROR_ESR : r =cpus_have_const_cap (ARM64_HAS_RAS_EXTN );break ; +case KVM_CAP_ARM_SPE_V1 : + r =kvm_arm_support_spe_v1 (); +break ;case KVM_CAP_SET_GUEST_DEBUG :case KVM_CAP_VCPU_ATTRIBUTES : r =1 ;
include/kvm/arm_spe.h +45 -0
59 lines not on this page · raw message
include/uapi/linux/kvm.h +1 -0
@@ -1009,6 +1009,7 @@ struct kvm_ppc_resize_hpt {#define KVM_CAP_PPC_GUEST_DEBUG_SSTEP 176#define KVM_CAP_ARM_NISV_TO_USER 177#define KVM_CAP_ARM_INJECT_EXT_DABT 178 +#define KVM_CAP_ARM_SPE_V1 179#ifdef KVM_CAP_IRQ_ROUTING
virt/kvm/arm/arm.c +1 -0
@@ -22,6 +22,7 @@#include <trace/events/kvm.h> #include <kvm/arm_pmu.h> #include <kvm/arm_psci.h> +#include <kvm/arm_spe.h> #define CREATE_TRACE_POINTS #include "trace.h"
virt/kvm/arm/spe.c new file +163 -0
163 lines not on this page · raw message
Signature · 1 line
2.21.0
Marc Zyngier
maintainerRe: [PATCH v2 12/18] KVM: arm64: add a new vcpu device control group for SPEv1
Quoted 164 lines · Andrew Murray
From: Sudeep Holla <[email protected]>
To configure the virtual SPEv1 overflow interrupt number, we use the vcpu
kvm_deviceioctl, encapsulating theKVM_ARM_VCPU_SPE_V1_IRQattribute within theKVM_ARM_VCPU_SPE_V1_CTRLgroup.After configuring the SPEv1, call the vcpu ioctl with attribute
KVM_ARM_VCPU_SPE_V1_INITto initialize the SPEv1.Signed-off-by Sudeep Holla <[email protected]>Signed-off-by Andrew Murray <[email protected]>--- Documentation/virt/kvm/devices/vcpu.txt | 28 ++++ arch/arm64/include/asm/kvm_host.h | 2 +- arch/arm64/include/uapi/asm/kvm.h | 4 + arch/arm64/kvm/Makefile | 1 + arch/arm64/kvm/guest.c | 6 + arch/arm64/kvm/reset.c | 3 + include/kvm/arm_spe.h | 45 +++++++ include/uapi/linux/kvm.h | 1 + virt/kvm/arm/arm.c | 1 + virt/kvm/arm/spe.c | 163 ++++++++++++++++++++++++ 10 files changed, 253 insertions(+), 1 deletion(-) create mode 100644 virt/kvm/arm/spe.cDocumentation/virt/kvm/devices/vcpu.txt +28
-0@@ -74,3 +74,31 @@ Specifies the base address of the stolen time structure for this VCPU. The base address must be 64 byte aligned and exist within a valid guest memory region. See Documentation/virt/kvm/arm/pvtime.txt for more information including the layout of the stolen time structure. + +4. GROUP: KVM_ARM_VCPU_SPE_V1_CTRL +Architectures: ARM64 + +4.1. ATTRIBUTE: KVM_ARM_VCPU_SPE_V1_IRQ +Parameters: in kvm_device_attr.addr the address for SPE buffer overflow interrupt + is a pointer to an int +Returns: -EBUSY: The SPE overflow interrupt is already set + -ENXIO: The overflow interrupt not set when attempting to get it + -ENODEV: SPEv1 not supported + -EINVAL: Invalid SPE overflow interrupt number supplied or + trying to set the IRQ number without using an in-kernel + irqchip. + +A value describing the SPEv1 (Statistical Profiling Extension v1) overflow +interrupt number for this vcpu. This interrupt should be PPI and the interrupt +type and number must be same for each vcpu. + +4.2 ATTRIBUTE: KVM_ARM_VCPU_SPE_V1_INIT +Parameters: no additional parameter in kvm_device_attr.addr +Returns: -ENODEV: SPEv1 not supported or GIC not initialized + -ENXIO: SPEv1 not properly configured or in-kernel irqchip not + configured as required prior to calling this attribute + -EBUSY: SPEv1 already initialized + +Request the initialization of the SPEv1. If using the SPEv1 with an in-kernel +virtual GIC implementation, this must be done after initializing the in-kernel +irqchip.arch/arm64/include/asm/kvm_host.h +1
-1@@ -39,7 +39,7 @@#define KVM_MAX_VCPUS VGIC_V3_MAX_CPUS -#define KVM_VCPU_MAX_FEATURES 7 +#define KVM_VCPU_MAX_FEATURES 8#define KVM_REQ_SLEEP \ KVM_ARCH_REQ_FLAGS(0, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)arch/arm64/include/uapi/asm/kvm.h +4
-0@@ -106,6 +106,7 @@ struct kvm_regs {#define KVM_ARM_VCPU_SVE 4/* enable SVE for this CPU */ #define KVM_ARM_VCPU_PTRAUTH_ADDRESS 5/* VCPU uses address authentication */ #define KVM_ARM_VCPU_PTRAUTH_GENERIC 6/* VCPU uses generic authentication */ +#define KVM_ARM_VCPU_SPE_V1 7/* Support guest SPEv1 */ struct kvm_vcpu_init {__u32 target;@@ -326,6 +327,9 @@ struct kvm_vcpu_events {#define KVM_ARM_VCPU_TIMER_IRQ_PTIMER 1#define KVM_ARM_VCPU_PVTIME_CTRL 2#define KVM_ARM_VCPU_PVTIME_IPA 0 +#define KVM_ARM_VCPU_SPE_V1_CTRL 3 +#define KVM_ARM_VCPU_SPE_V1_IRQ 0 +#define KVM_ARM_VCPU_SPE_V1_INIT 1/* KVM_IRQ_LINE irq field index values */ #define KVM_ARM_IRQ_VCPU2_SHIFT 28arch/arm64/kvm/Makefile +1
-0@@ -37,3 +37,4 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-debug.o kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/irqchip.o kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/arch_timer.o kvm-$(CONFIG_KVM_ARM_PMU) += $(KVM)/arm/pmu.o +kvm-$(CONFIG_KVM_ARM_SPE) += $(KVM)/arm/spe.oarch/arm64/kvm/guest.c +6
-0@@ -874,6 +874,8 @@ int kvm_arm_vcpu_arch_set_attr(struct kvm_vcpu *vcpu,break ;case KVM_ARM_VCPU_PVTIME_CTRL : ret =kvm_arm_pvtime_set_attr (vcpu, attr); +case KVM_ARM_VCPU_SPE_V1_CTRL : + ret =kvm_arm_spe_v1_set_attr (vcpu, attr);break ;default : ret = -ENXIO ;@@ -897,6 +899,8 @@ int kvm_arm_vcpu_arch_get_attr(struct kvm_vcpu *vcpu,break ;case KVM_ARM_VCPU_PVTIME_CTRL : ret =kvm_arm_pvtime_get_attr (vcpu, attr); +case KVM_ARM_VCPU_SPE_V1_CTRL : + ret =kvm_arm_spe_v1_get_attr (vcpu, attr);break ;default : ret = -ENXIO ;@@ -920,6 +924,8 @@ int kvm_arm_vcpu_arch_has_attr(struct kvm_vcpu *vcpu,break ;case KVM_ARM_VCPU_PVTIME_CTRL : ret =kvm_arm_pvtime_has_attr (vcpu, attr); +case KVM_ARM_VCPU_SPE_V1_CTRL : + ret =kvm_arm_spe_v1_has_attr (vcpu, attr);break ;default : ret = -ENXIO ;arch/arm64/kvm/reset.c +3
-0@@ -80,6 +80,9 @@ int kvm_arch_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_ARM_INJECT_SERROR_ESR : r =cpus_have_const_cap (ARM64_HAS_RAS_EXTN );break ; +case KVM_CAP_ARM_SPE_V1 : + r =kvm_arm_support_spe_v1 (); +break ;case KVM_CAP_SET_GUEST_DEBUG :case KVM_CAP_VCPU_ATTRIBUTES : r =1 ;include/kvm/arm_spe.h +1
-0@@ -8,6 +8,7 @@#include <uapi/linux/kvm.h> #include <linux/kvm_host.h> +#include <linux/cpufeature.h> struct kvm_spe {int irq_num;
include/kvm/arm_spe.h struct kvm_spe { · line 24
Quoted 6 lines
include/kvm/arm_spe.h +3
-0@@ -18,8 +19,52 @@ struct kvm_spe {#ifdef CONFIG_KVM_ARM_SPE #define kvm_arm_spe_v1_ready (v) ((v)->arch.spe.ready) +#define kvm_arm_spe_irq_initialized (v) \ + ((v)->arch.spe.irq_num >= VGIC_NR_SGIS && \ + (v)->arch.spe.irq_num <= VGIC_MAX_PRIVATE)
This is buggy, as it accepts 32 as a valid interrupt (which obviously isn't a PPI). Having fixed it, this is a duplicate of irq_is_ppi().
And that's where we can finally confirm that 'irq_num' is a GIC INTID. Please name it as such.
Quoted 83 lines
include/kvm/arm_spe.h +41
-0+ +static inline bool kvm_arm_support_spe_v1 (void ) +{ +u64 dfr0 =read_sanitised_ftr_reg (SYS_ID_AA64DFR0_EL1 ); + +return !!cpuid_feature_extract_unsigned_field (dfr0, +ID_AA64DFR0_PMSVER_SHIFT ); +} + +int kvm_arm_spe_v1_set_attr (struct kvm_vcpu *vcpu, +struct kvm_device_attr *attr); +int kvm_arm_spe_v1_get_attr (struct kvm_vcpu *vcpu, +struct kvm_device_attr *attr); +int kvm_arm_spe_v1_has_attr (struct kvm_vcpu *vcpu, +struct kvm_device_attr *attr); +int kvm_arm_spe_v1_enable (struct kvm_vcpu *vcpu);#else #define kvm_arm_spe_v1_ready (v) (false) +#define kvm_arm_support_spe_v1 () (false) +#define kvm_arm_spe_irq_initialized (v) (false) + +static inline int kvm_arm_spe_v1_set_attr (struct kvm_vcpu *vcpu, +struct kvm_device_attr *attr) +{ +return -ENXIO ; +} + +static inline int kvm_arm_spe_v1_get_attr (struct kvm_vcpu *vcpu, +struct kvm_device_attr *attr) +{ +return -ENXIO ; +} + +static inline int kvm_arm_spe_v1_has_attr (struct kvm_vcpu *vcpu, +struct kvm_device_attr *attr) +{ +return -ENXIO ; +} + +static inline int kvm_arm_spe_v1_enable (struct kvm_vcpu *vcpu) +{ +return 0 ; +}#endif /* CONFIG_KVM_ARM_SPE */ #endif /* __ASM_ARM_KVM_SPE_H */ include/uapi/linux/kvm.h +1
-0@@ -1009,6 +1009,7 @@ struct kvm_ppc_resize_hpt {#define KVM_CAP_PPC_GUEST_DEBUG_SSTEP 176#define KVM_CAP_ARM_NISV_TO_USER 177#define KVM_CAP_ARM_INJECT_EXT_DABT 178 +#define KVM_CAP_ARM_SPE_V1 179#ifdef KVM_CAP_IRQ_ROUTING virt/kvm/arm/arm.c +1
-0@@ -22,6 +22,7 @@#include <trace/events/kvm.h> #include <kvm/arm_pmu.h> #include <kvm/arm_psci.h> +#include <kvm/arm_spe.h> #define CREATE_TRACE_POINTS #include "trace.h" virt/kvm/arm/spe.c new file +7
-0@@ -0,0 +1,163 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Copyright (C) 2019 ARM Ltd. +*/ + +#include <linux/cpu.h> +#include <linux/kvm.h>
virt/kvm/arm/spe.c line 17
Quoted 10 lines
virt/kvm/arm/spe.c new file +10
-0+#include <linux/kvm_host.h> +#include <linux/uaccess.h> +#include <asm/kvm_emulate.h> +#include <kvm/arm_spe.h> +#include <kvm/arm_vgic.h> + +int kvm_arm_spe_v1_enable (struct kvm_vcpu *vcpu) +{ +if (!vcpu->arch.spe.created) +return 0 ;
Shouldn't it be an error to enable something that doesn't exist?
virt/kvm/arm/spe.c line 20
Quoted 3 lines
virt/kvm/arm/spe.c new file +3
-0+ +/* +* A valid interrupt configuration for the SPE is either to have a
either?
virt/kvm/arm/spe.c line 33
Quoted 13 lines
virt/kvm/arm/spe.c new file +13
-0+* properly configured interrupt number and using an in-kernel irqchip. +*/ +if (irqchip_in_kernel (vcpu->kvm)) { +int irq = vcpu->arch.spe.irq_num; + +if (!kvm_arm_spe_irq_initialized (vcpu)) +return -EINVAL ; + +if (!irq_is_ppi (irq)) +return -EINVAL ; + } + + vcpu->arch.spe.ready = true;
And SPE is then ready even when we don't have an in-kernel irqchip?
Quoted 23 lines
virt/kvm/arm/spe.c new file +23
-0+ +return 0 ; +} + +static int kvm_arm_spe_v1_init (struct kvm_vcpu *vcpu) +{ +if (!kvm_arm_support_spe_v1 ()) +return -ENODEV ; + +if (!test_bit (KVM_ARM_VCPU_SPE_V1 , vcpu->arch.features)) +return -ENXIO ; + +if (vcpu->arch.spe.created) +return -EBUSY ; + +if (irqchip_in_kernel (vcpu->kvm)) { +int ret; + +/* +* If using the SPE with an in-kernel virtual GIC +* implementation, we require the GIC to be already +* initialized when initializing the SPE. +*/
virt/kvm/arm/spe.c line 66
Quoted 10 lines
virt/kvm/arm/spe.c new file +10
-0+if (!vgic_initialized (vcpu->kvm)) +return -ENODEV ; + + ret =kvm_vgic_set_owner (vcpu, vcpu->arch.spe.irq_num, + &vcpu->arch.spe); +if (ret) +return ret; + } + + vcpu->arch.spe.created = true;
Same problem.
virt/kvm/arm/spe.c line 73
Quoted 7 lines
virt/kvm/arm/spe.c new file +7
-0+return 0 ; +} + +/* +* For one VM the interrupt type must be same for each vcpu. +* As a PPI, the interrupt number is the same for all vcpus, +* while as an SPI it must be a separate number per vcpu.
Why do you want to support SPIs at all? And it isn't what kvm_arm_spe_irq_initialized claims to be doing.
Quoted 16 lines
virt/kvm/arm/spe.c new file +16
-0+*/ +static bool spe_irq_is_valid (struct kvm *kvm,int irq) +{ +int i; +struct kvm_vcpu *vcpu; + +kvm_for_each_vcpu (i, vcpu, kvm) { +if (!kvm_arm_spe_irq_initialized (vcpu)) +continue ; + +if (vcpu->arch.spe.irq_num != irq) +return false; + } + +return true; +}
virt/kvm/arm/spe.c line 99
Quoted 10 lines
virt/kvm/arm/spe.c new file +10
-0+ +int kvm_arm_spe_v1_set_attr (struct kvm_vcpu *vcpu,struct kvm_device_attr *attr) +{ +switch (attr->attr) { +case KVM_ARM_VCPU_SPE_V1_IRQ : { +int __user *uaddr = (int __user *)(long )attr->addr; +int irq; + +if (!irqchip_in_kernel (vcpu->kvm)) +return -EINVAL ;
Here, you forbid setting the IRQ for a VM that doesn't have an in-kernel irqchip. And yet below, you're happy to initialise it.
virt/kvm/arm/spe.c line 109
Quoted 10 lines
virt/kvm/arm/spe.c new file +10
-0+ +if (!test_bit (KVM_ARM_VCPU_SPE_V1 , vcpu->arch.features)) +return -ENODEV ; + +if (get_user (irq, uaddr)) +return -EFAULT ; + +/* The SPE overflow interrupt can be a PPI only */ +if (!(irq_is_ppi (irq))) +return -EINVAL ;
Ah, so you do know about irq_is_ppi()...
virt/kvm/arm/spe.c line 111
Quoted 2 lines
virt/kvm/arm/spe.c new file +2
-0+ +if (!spe_irq_is_valid (vcpu->kvm, irq))
But why don't you fold the interrupt validity checks in this helper?
Quoted 38 lines
virt/kvm/arm/spe.c new file +38
-0+return -EINVAL ; + +if (kvm_arm_spe_irq_initialized (vcpu)) +return -EBUSY ; + +kvm_debug ("Set kvm ARM SPE irq: %d\n" , irq); + vcpu->arch.spe.irq_num = irq; +return 0 ; + } +case KVM_ARM_VCPU_SPE_V1_INIT : +return kvm_arm_spe_v1_init (vcpu); + } + +return -ENXIO ; +} + +int kvm_arm_spe_v1_get_attr (struct kvm_vcpu *vcpu,struct kvm_device_attr *attr) +{ +switch (attr->attr) { +case KVM_ARM_VCPU_SPE_V1_IRQ : { +int __user *uaddr = (int __user *)(long )attr->addr; +int irq; + +if (!irqchip_in_kernel (vcpu->kvm)) +return -EINVAL ; + +if (!test_bit (KVM_ARM_VCPU_SPE_V1 , vcpu->arch.features)) +return -ENODEV ; + +if (!kvm_arm_spe_irq_initialized (vcpu)) +return -ENXIO ; + + irq = vcpu->arch.spe.irq_num; +return put_user (irq, uaddr); + } + } + +return -ENXIO ;
virt/kvm/arm/spe.c line 159
Quoted 10 lines
virt/kvm/arm/spe.c new file +10
-0+} + +int kvm_arm_spe_v1_has_attr (struct kvm_vcpu *vcpu,struct kvm_device_attr *attr) +{ +switch (attr->attr) { +case KVM_ARM_VCPU_SPE_V1_IRQ : +case KVM_ARM_VCPU_SPE_V1_INIT : +if (kvm_arm_support_spe_v1 () && +test_bit (KVM_ARM_VCPU_SPE_V1 , vcpu->arch.features)) +return 0 ;
It is interesting that all the user interface is designed with the feature being optional in mind, and yet you've removed all the necessary handling code.
M.
Signature · 1 line
Jazz is not dead, it just smells funny.
Re: [PATCH v2 12/18] KVM: arm64: add a new vcpu device control group for SPEv1
Quoted 167 lines · Marc Zyngier
Quoted 165 lines · Andrew Murray
From: Sudeep Holla <[email protected]>
To configure the virtual SPEv1 overflow interrupt number, we use the vcpu
kvm_deviceioctl, encapsulating theKVM_ARM_VCPU_SPE_V1_IRQattribute within theKVM_ARM_VCPU_SPE_V1_CTRLgroup.After configuring the SPEv1, call the vcpu ioctl with attribute
KVM_ARM_VCPU_SPE_V1_INITto initialize the SPEv1.Signed-off-by Sudeep Holla <[email protected]>Signed-off-by Andrew Murray <[email protected]>--- Documentation/virt/kvm/devices/vcpu.txt | 28 ++++ arch/arm64/include/asm/kvm_host.h | 2 +- arch/arm64/include/uapi/asm/kvm.h | 4 + arch/arm64/kvm/Makefile | 1 + arch/arm64/kvm/guest.c | 6 + arch/arm64/kvm/reset.c | 3 + include/kvm/arm_spe.h | 45 +++++++ include/uapi/linux/kvm.h | 1 + virt/kvm/arm/arm.c | 1 + virt/kvm/arm/spe.c | 163 ++++++++++++++++++++++++ 10 files changed, 253 insertions(+), 1 deletion(-) create mode 100644 virt/kvm/arm/spe.cDocumentation/virt/kvm/devices/vcpu.txt +28
-0@@ -74,3 +74,31 @@ Specifies the base address of the stolen time structure for this VCPU. The base address must be 64 byte aligned and exist within a valid guest memory region. See Documentation/virt/kvm/arm/pvtime.txt for more information including the layout of the stolen time structure. + +4. GROUP: KVM_ARM_VCPU_SPE_V1_CTRL +Architectures: ARM64 + +4.1. ATTRIBUTE: KVM_ARM_VCPU_SPE_V1_IRQ +Parameters: in kvm_device_attr.addr the address for SPE buffer overflow interrupt + is a pointer to an int +Returns: -EBUSY: The SPE overflow interrupt is already set + -ENXIO: The overflow interrupt not set when attempting to get it + -ENODEV: SPEv1 not supported + -EINVAL: Invalid SPE overflow interrupt number supplied or + trying to set the IRQ number without using an in-kernel + irqchip. + +A value describing the SPEv1 (Statistical Profiling Extension v1) overflow +interrupt number for this vcpu. This interrupt should be PPI and the interrupt +type and number must be same for each vcpu. + +4.2 ATTRIBUTE: KVM_ARM_VCPU_SPE_V1_INIT +Parameters: no additional parameter in kvm_device_attr.addr +Returns: -ENODEV: SPEv1 not supported or GIC not initialized + -ENXIO: SPEv1 not properly configured or in-kernel irqchip not + configured as required prior to calling this attribute + -EBUSY: SPEv1 already initialized + +Request the initialization of the SPEv1. If using the SPEv1 with an in-kernel +virtual GIC implementation, this must be done after initializing the in-kernel +irqchip.arch/arm64/include/asm/kvm_host.h +1
-1@@ -39,7 +39,7 @@#define KVM_MAX_VCPUS VGIC_V3_MAX_CPUS -#define KVM_VCPU_MAX_FEATURES 7 +#define KVM_VCPU_MAX_FEATURES 8#define KVM_REQ_SLEEP \ KVM_ARCH_REQ_FLAGS(0, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)arch/arm64/include/uapi/asm/kvm.h +4
-0@@ -106,6 +106,7 @@ struct kvm_regs {#define KVM_ARM_VCPU_SVE 4/* enable SVE for this CPU */ #define KVM_ARM_VCPU_PTRAUTH_ADDRESS 5/* VCPU uses address authentication */ #define KVM_ARM_VCPU_PTRAUTH_GENERIC 6/* VCPU uses generic authentication */ +#define KVM_ARM_VCPU_SPE_V1 7/* Support guest SPEv1 */ struct kvm_vcpu_init {__u32 target;@@ -326,6 +327,9 @@ struct kvm_vcpu_events {#define KVM_ARM_VCPU_TIMER_IRQ_PTIMER 1#define KVM_ARM_VCPU_PVTIME_CTRL 2#define KVM_ARM_VCPU_PVTIME_IPA 0 +#define KVM_ARM_VCPU_SPE_V1_CTRL 3 +#define KVM_ARM_VCPU_SPE_V1_IRQ 0 +#define KVM_ARM_VCPU_SPE_V1_INIT 1/* KVM_IRQ_LINE irq field index values */ #define KVM_ARM_IRQ_VCPU2_SHIFT 28arch/arm64/kvm/Makefile +1
-0@@ -37,3 +37,4 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-debug.o kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/irqchip.o kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/arch_timer.o kvm-$(CONFIG_KVM_ARM_PMU) += $(KVM)/arm/pmu.o +kvm-$(CONFIG_KVM_ARM_SPE) += $(KVM)/arm/spe.oarch/arm64/kvm/guest.c +6
-0@@ -874,6 +874,8 @@ int kvm_arm_vcpu_arch_set_attr(struct kvm_vcpu *vcpu,break ;case KVM_ARM_VCPU_PVTIME_CTRL : ret =kvm_arm_pvtime_set_attr (vcpu, attr); +case KVM_ARM_VCPU_SPE_V1_CTRL : + ret =kvm_arm_spe_v1_set_attr (vcpu, attr);break ;default : ret = -ENXIO ;@@ -897,6 +899,8 @@ int kvm_arm_vcpu_arch_get_attr(struct kvm_vcpu *vcpu,break ;case KVM_ARM_VCPU_PVTIME_CTRL : ret =kvm_arm_pvtime_get_attr (vcpu, attr); +case KVM_ARM_VCPU_SPE_V1_CTRL : + ret =kvm_arm_spe_v1_get_attr (vcpu, attr);break ;default : ret = -ENXIO ;@@ -920,6 +924,8 @@ int kvm_arm_vcpu_arch_has_attr(struct kvm_vcpu *vcpu,break ;case KVM_ARM_VCPU_PVTIME_CTRL : ret =kvm_arm_pvtime_has_attr (vcpu, attr); +case KVM_ARM_VCPU_SPE_V1_CTRL : + ret =kvm_arm_spe_v1_has_attr (vcpu, attr);break ;default : ret = -ENXIO ;arch/arm64/kvm/reset.c +3
-0@@ -80,6 +80,9 @@ int kvm_arch_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_ARM_INJECT_SERROR_ESR : r =cpus_have_const_cap (ARM64_HAS_RAS_EXTN );break ; +case KVM_CAP_ARM_SPE_V1 : + r =kvm_arm_support_spe_v1 (); +break ;case KVM_CAP_SET_GUEST_DEBUG :case KVM_CAP_VCPU_ATTRIBUTES : r =1 ;include/kvm/arm_spe.h +1
-0@@ -8,6 +8,7 @@#include <uapi/linux/kvm.h> #include <linux/kvm_host.h> +#include <linux/cpufeature.h> struct kvm_spe {int irq_num;@@ -18,8 +19,52 @@ struct kvm_spe {
Quoted 8 lines
include/kvm/arm_spe.h struct kvm_spe { · line 24
Quoted 5 lines
include/kvm/arm_spe.h +3
-0#ifdef CONFIG_KVM_ARM_SPE #define kvm_arm_spe_v1_ready (v) ((v)->arch.spe.ready) +#define kvm_arm_spe_irq_initialized (v) \ + ((v)->arch.spe.irq_num >= VGIC_NR_SGIS && \ + (v)->arch.spe.irq_num <= VGIC_MAX_PRIVATE)This is buggy, as it accepts 32 as a valid interrupt (which obviously isn't a PPI). Having fixed it, this is a duplicate of
irq_is_ppi().
I'll replace that line with irq_is_ppi.
Quoted 3 lines
And that's where we can finally confirm that '
irq_num' is a GIC INTID. Please name it as such.
OK.
Quoted 88 lines
Quoted 88 lines
include/kvm/arm_spe.h +41
-0+ +static inline bool kvm_arm_support_spe_v1 (void ) +{ +u64 dfr0 =read_sanitised_ftr_reg (SYS_ID_AA64DFR0_EL1 ); + +return !!cpuid_feature_extract_unsigned_field (dfr0, +ID_AA64DFR0_PMSVER_SHIFT ); +} + +int kvm_arm_spe_v1_set_attr (struct kvm_vcpu *vcpu, +struct kvm_device_attr *attr); +int kvm_arm_spe_v1_get_attr (struct kvm_vcpu *vcpu, +struct kvm_device_attr *attr); +int kvm_arm_spe_v1_has_attr (struct kvm_vcpu *vcpu, +struct kvm_device_attr *attr); +int kvm_arm_spe_v1_enable (struct kvm_vcpu *vcpu);#else #define kvm_arm_spe_v1_ready (v) (false) +#define kvm_arm_support_spe_v1 () (false) +#define kvm_arm_spe_irq_initialized (v) (false) + +static inline int kvm_arm_spe_v1_set_attr (struct kvm_vcpu *vcpu, +struct kvm_device_attr *attr) +{ +return -ENXIO ; +} + +static inline int kvm_arm_spe_v1_get_attr (struct kvm_vcpu *vcpu, +struct kvm_device_attr *attr) +{ +return -ENXIO ; +} + +static inline int kvm_arm_spe_v1_has_attr (struct kvm_vcpu *vcpu, +struct kvm_device_attr *attr) +{ +return -ENXIO ; +} + +static inline int kvm_arm_spe_v1_enable (struct kvm_vcpu *vcpu) +{ +return 0 ; +}#endif /* CONFIG_KVM_ARM_SPE */ #endif /* __ASM_ARM_KVM_SPE_H */ include/uapi/linux/kvm.h +1
-0@@ -1009,6 +1009,7 @@ struct kvm_ppc_resize_hpt {#define KVM_CAP_PPC_GUEST_DEBUG_SSTEP 176#define KVM_CAP_ARM_NISV_TO_USER 177#define KVM_CAP_ARM_INJECT_EXT_DABT 178 +#define KVM_CAP_ARM_SPE_V1 179#ifdef KVM_CAP_IRQ_ROUTING virt/kvm/arm/arm.c +1
-0@@ -22,6 +22,7 @@#include <trace/events/kvm.h> #include <kvm/arm_pmu.h> #include <kvm/arm_psci.h> +#include <kvm/arm_spe.h> #define CREATE_TRACE_POINTS #include "trace.h" virt/kvm/arm/spe.c new file +11
-0@@ -0,0 +1,163 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Copyright (C) 2019 ARM Ltd. +*/ + +#include <linux/cpu.h> +#include <linux/kvm.h> +#include <linux/kvm_host.h> +#include <linux/uaccess.h> +#include <asm/kvm_emulate.h> +#include <kvm/arm_spe.h>
Quoted 8 lines
virt/kvm/arm/spe.c line 17
Quoted 6 lines
virt/kvm/arm/spe.c new file +6
-0+#include <kvm/arm_vgic.h> + +int kvm_arm_spe_v1_enable (struct kvm_vcpu *vcpu) +{ +if (!vcpu->arch.spe.created) +return 0 ;Shouldn't it be an error to enable something that doesn't exist?
It looks like this has adopted the same approach as kvm_arm_pmu_v3_enable. In kvm_vcpu_first_run_init we attempt to enable pmu v3 and spe - without first checking they exist.
Quoted 7 lines
virt/kvm/arm/spe.c line 20
Quoted 4 lines
virt/kvm/arm/spe.c new file +3
-0+ +/* +* A valid interrupt configuration for the SPE is either to have a either?
Quoted 15 lines
virt/kvm/arm/spe.c line 33
Quoted 13 lines
virt/kvm/arm/spe.c new file +13
-0+* properly configured interrupt number and using an in-kernel irqchip. +*/ +if (irqchip_in_kernel (vcpu->kvm)) { +int irq = vcpu->arch.spe.irq_num; + +if (!kvm_arm_spe_irq_initialized (vcpu)) +return -EINVAL ; + +if (!irq_is_ppi (irq)) +return -EINVAL ; + } + + vcpu->arch.spe.ready = true;And SPE is then ready even when we don't have an in-kernel irqchip?
I recall in Sudeep's previous patchset that you suggested we don't support SPE without an in-kernel irqchip. I'll update to reflect that feedback.
Quoted 37 lines
Quoted 24 lines
virt/kvm/arm/spe.c new file +23
-0+ +return 0 ; +} + +static int kvm_arm_spe_v1_init (struct kvm_vcpu *vcpu) +{ +if (!kvm_arm_support_spe_v1 ()) +return -ENODEV ; + +if (!test_bit (KVM_ARM_VCPU_SPE_V1 , vcpu->arch.features)) +return -ENXIO ; + +if (vcpu->arch.spe.created) +return -EBUSY ; + +if (irqchip_in_kernel (vcpu->kvm)) { +int ret; + +/* +* If using the SPE with an in-kernel virtual GIC +* implementation, we require the GIC to be already +* initialized when initializing the SPE. +*/ virt/kvm/arm/spe.c line 66
Quoted 10 lines
virt/kvm/arm/spe.c new file +10
-0+if (!vgic_initialized (vcpu->kvm)) +return -ENODEV ; + + ret =kvm_vgic_set_owner (vcpu, vcpu->arch.spe.irq_num, + &vcpu->arch.spe); +if (ret) +return ret; + } + + vcpu->arch.spe.created = true;Same problem.
Quoted 10 lines
virt/kvm/arm/spe.c line 73
Quoted 7 lines
virt/kvm/arm/spe.c new file +7
-0+return 0 ; +} + +/* +* For one VM the interrupt type must be same for each vcpu. +* As a PPI, the interrupt number is the same for all vcpus, +* while as an SPI it must be a separate number per vcpu. Why do you want to support SPIs at all? And it isn't what
kvm_arm_spe_irq_initializedclaims to be doing.
I don't think we do. I think it's expected that the SPE interrupt is a PPI, at least that's what the SPE driver expects. I'll simplify this.
Quoted 44 lines
Quoted 17 lines
virt/kvm/arm/spe.c new file +16
-0+*/ +static bool spe_irq_is_valid (struct kvm *kvm,int irq) +{ +int i; +struct kvm_vcpu *vcpu; + +kvm_for_each_vcpu (i, vcpu, kvm) { +if (!kvm_arm_spe_irq_initialized (vcpu)) +continue ; + +if (vcpu->arch.spe.irq_num != irq) +return false; + } + +return true; +}virt/kvm/arm/spe.c line 99
Quoted 10 lines
virt/kvm/arm/spe.c new file +10
-0+ +int kvm_arm_spe_v1_set_attr (struct kvm_vcpu *vcpu,struct kvm_device_attr *attr) +{ +switch (attr->attr) { +case KVM_ARM_VCPU_SPE_V1_IRQ : { +int __user *uaddr = (int __user *)(long )attr->addr; +int irq; + +if (!irqchip_in_kernel (vcpu->kvm)) +return -EINVAL ;Here, you forbid setting the IRQ for a VM that doesn't have an in-kernel irqchip. And yet below, you're happy to initialise it.
virt/kvm/arm/spe.c line 109
Quoted 10 lines
virt/kvm/arm/spe.c new file +10
-0+ +if (!test_bit (KVM_ARM_VCPU_SPE_V1 , vcpu->arch.features)) +return -ENODEV ; + +if (get_user (irq, uaddr)) +return -EFAULT ; + +/* The SPE overflow interrupt can be a PPI only */ +if (!(irq_is_ppi (irq))) +return -EINVAL ;Ah, so you do know about
irq_is_ppi()...
Quoted 4 lines
virt/kvm/arm/spe.c line 111
Quoted 2 lines
virt/kvm/arm/spe.c new file +2
-0+ +if (!spe_irq_is_valid (vcpu->kvm, irq))But why don't you fold the interrupt validity checks in this helper?
That makes sense.
Quoted 45 lines
Quoted 45 lines
virt/kvm/arm/spe.c new file +44
-0+return -EINVAL ; + +if (kvm_arm_spe_irq_initialized (vcpu)) +return -EBUSY ; + +kvm_debug ("Set kvm ARM SPE irq: %d\n" , irq); + vcpu->arch.spe.irq_num = irq; +return 0 ; + } +case KVM_ARM_VCPU_SPE_V1_INIT : +return kvm_arm_spe_v1_init (vcpu); + } + +return -ENXIO ; +} + +int kvm_arm_spe_v1_get_attr (struct kvm_vcpu *vcpu,struct kvm_device_attr *attr) +{ +switch (attr->attr) { +case KVM_ARM_VCPU_SPE_V1_IRQ : { +int __user *uaddr = (int __user *)(long )attr->addr; +int irq; + +if (!irqchip_in_kernel (vcpu->kvm)) +return -EINVAL ; + +if (!test_bit (KVM_ARM_VCPU_SPE_V1 , vcpu->arch.features)) +return -ENODEV ; + +if (!kvm_arm_spe_irq_initialized (vcpu)) +return -ENXIO ; + + irq = vcpu->arch.spe.irq_num; +return put_user (irq, uaddr); + } + } + +return -ENXIO ; +} + +int kvm_arm_spe_v1_has_attr (struct kvm_vcpu *vcpu,struct kvm_device_attr *attr) +{ +switch (attr->attr) { +case KVM_ARM_VCPU_SPE_V1_IRQ :
Quoted 8 lines
virt/kvm/arm/spe.c line 159
Quoted 4 lines
virt/kvm/arm/spe.c new file +4
-0+case KVM_ARM_VCPU_SPE_V1_INIT : +if (kvm_arm_support_spe_v1 () && +test_bit (KVM_ARM_VCPU_SPE_V1 , vcpu->arch.features)) +return 0 ;It is interesting that all the user interface is designed with the feature being optional in mind, and yet you've removed all the necessary handling code.
Yeah I broke that, I'll fix it.
Thanks,
Andrew Murray
Quoted 5 lines
M.Signature · 1 line
Jazz is not dead, it just smells funny.
PATCH v2 13/18 perf: arm_spe: Add KVM structure for obtaining IRQ info
In reply to Andrew MurrayKVM requires knowledge of the physical SPE IRQ number such that it can associate it with any virtual IRQ for guests that require SPE emulation.
Let's create a structure to hold this information and an accessor that KVM can use to retrieve this information.
We expect that each SPE device will have the same physical PPI number and thus will warn when this is not the case.
--- drivers/perf/arm_spe_pmu.c | 23 +++++++++++++++++++++++ include/kvm/arm_spe.h | 6 ++++++ 2 files changed, 29 insertions(+)
drivers/perf/arm_spe_pmu.c +23 -0
@@ -34,6 +34,9 @@#include <linux/smp.h> #include <linux/vmalloc.h> +#include <linux/kvm_host.h> +#include <kvm/arm_spe.h> +#include <asm/barrier.h> #include <asm/cpufeature.h> #include <asm/mmu.h>
@@ -1127,6 +1130,24 @@ static void arm_spe_pmu_dev_teardown(struct arm_spe_pmu *spe_pmu)free_percpu_irq (spe_pmu->irq, spe_pmu->handle); } +#ifdef CONFIG_KVM_ARM_SPE +static struct arm_spe_kvm_info arm_spe_kvm_info; + +struct arm_spe_kvm_info *arm_spe_get_kvm_info (void ) +{ +return &arm_spe_kvm_info; +} + +static void arm_spe_populate_kvm_info (struct arm_spe_pmu *spe_pmu) +{ +WARN_ON_ONCE (arm_spe_kvm_info.physical_irq !=0 && + arm_spe_kvm_info.physical_irq != spe_pmu->irq); + arm_spe_kvm_info.physical_irq = spe_pmu->irq; +} +#else +static void arm_spe_populate_kvm_info (struct arm_spe_pmu *spe_pmu) {} +#endif +/* Driver and device probing */ static int arm_spe_pmu_irq_probe (struct arm_spe_pmu *spe_pmu) {
@@ -1149,6 +1170,8 @@ static int arm_spe_pmu_irq_probe(struct arm_spe_pmu *spe_pmu) } spe_pmu->irq = irq; +arm_spe_populate_kvm_info (spe_pmu); +return 0 ; }
include/kvm/arm_spe.h +6 -0
@@ -17,6 +17,12 @@ struct kvm_spe {bool irq_level; }; +struct arm_spe_kvm_info { +int physical_irq; +}; + +struct arm_spe_kvm_info *arm_spe_get_kvm_info (void ); +#ifdef CONFIG_KVM_ARM_SPE #define kvm_arm_spe_v1_ready (v) ((v)->arch.spe.ready)#define kvm_arm_spe_irq_initialized (v) \
Signature · 6 lines
2.21.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerRe: [PATCH v2 13/18] perf: arm_spe: Add KVM structure for obtaining IRQ info
Quoted 3 lines · Andrew Murray
KVM requires knowledge of the physical SPE IRQ number such that it can associate it with any virtual IRQ for guests that require SPE emulation.
This is at best extremely odd. The only reason for KVM to obtain this IRQ number is if it has exclusive access to the device. This obviously isn't the case, as this device is shared between host and guest.
Quoted 27 lines
Let's create a structure to hold this information and an accessor that KVM can use to retrieve this information.
We expect that each SPE device will have the same physical PPI number and thus will warn when this is not the case.
Signed-off-by Andrew Murray <[email protected]>--- drivers/perf/arm_spe_pmu.c | 23 +++++++++++++++++++++++ include/kvm/arm_spe.h | 6 ++++++ 2 files changed, 29 insertions(+)drivers/perf/arm_spe_pmu.c +3
-0@@ -34,6 +34,9 @@#include <linux/smp.h> #include <linux/vmalloc.h> +#include <linux/kvm_host.h> +#include <kvm/arm_spe.h> +#include <asm/barrier.h> #include <asm/cpufeature.h> #include <asm/mmu.h>
drivers/perf/arm_spe_pmu.c static void arm_spe_pmu_dev_teardown(struct arm_spe_pmu *spe_pmu) · line 1139
Quoted 10 lines
drivers/perf/arm_spe_pmu.c +7
-0@@ -1127,6 +1130,24 @@ static void arm_spe_pmu_dev_teardown(struct arm_spe_pmu *spe_pmu)free_percpu_irq (spe_pmu->irq, spe_pmu->handle); } +#ifdef CONFIG_KVM_ARM_SPE +static struct arm_spe_kvm_info arm_spe_kvm_info; + +struct arm_spe_kvm_info *arm_spe_get_kvm_info (void ) +{ +return &arm_spe_kvm_info; +}
How does this work when SPE is built as a module?
drivers/perf/arm_spe_pmu.c static void arm_spe_pmu_dev_teardown(struct arm_spe_pmu *spe_pmu) · line 1145
Quoted 6 lines
drivers/perf/arm_spe_pmu.c +6
-0+ +static void arm_spe_populate_kvm_info (struct arm_spe_pmu *spe_pmu) +{ +WARN_ON_ONCE (arm_spe_kvm_info.physical_irq !=0 && + arm_spe_kvm_info.physical_irq != spe_pmu->irq); + arm_spe_kvm_info.physical_irq = spe_pmu->irq;
What does 'physical' means here? It's an IRQ in the Linux sense, so it's already some random number that bears no relation to anything 'physical'.
Quoted 24 lines
drivers/perf/arm_spe_pmu.c +7
-0+} +#else +static void arm_spe_populate_kvm_info (struct arm_spe_pmu *spe_pmu) {} +#endif +/* Driver and device probing */ static int arm_spe_pmu_irq_probe (struct arm_spe_pmu *spe_pmu) {@@ -1149,6 +1170,8 @@ static int arm_spe_pmu_irq_probe(struct arm_spe_pmu *spe_pmu) } spe_pmu->irq = irq; +arm_spe_populate_kvm_info (spe_pmu); +return 0 ; }include/kvm/arm_spe.h +0
-0@@ -17,6 +17,12 @@ struct kvm_spe {bool irq_level; };
include/kvm/arm_spe.h struct kvm_spe { · line 28
Quoted 10 lines
include/kvm/arm_spe.h +6
-0+struct arm_spe_kvm_info { +int physical_irq; +}; + +struct arm_spe_kvm_info *arm_spe_get_kvm_info (void ); +#ifdef CONFIG_KVM_ARM_SPE #define kvm_arm_spe_v1_ready (v) ((v)->arch.spe.ready)#define kvm_arm_spe_irq_initialized (v) \
M.
Signature · 1 line
Jazz is not dead, it just smells funny.
Re: [PATCH v2 13/18] perf: arm_spe: Add KVM structure for obtaining IRQ info
Quoted 10 lines · Marc Zyngier
Quoted 3 lines · Andrew Murray
KVM requires knowledge of the physical SPE IRQ number such that it can associate it with any virtual IRQ for guests that require SPE emulation.
This is at best extremely odd. The only reason for KVM to obtain this IRQ number is if it has exclusive access to the device. This obviously isn't the case, as this device is shared between host and guest.
This was an attempt to set the interrupt as active such that host SPE driver doesn't get spurious interrupts due to guest SPE activity. Though let's save the discussion to patch 14.
Quoted 41 lines
Quoted 28 lines
Let's create a structure to hold this information and an accessor that KVM can use to retrieve this information.
We expect that each SPE device will have the same physical PPI number and thus will warn when this is not the case.
Signed-off-by Andrew Murray <[email protected]>--- drivers/perf/arm_spe_pmu.c | 23 +++++++++++++++++++++++ include/kvm/arm_spe.h | 6 ++++++ 2 files changed, 29 insertions(+)drivers/perf/arm_spe_pmu.c +3
-0@@ -34,6 +34,9 @@#include <linux/smp.h> #include <linux/vmalloc.h> +#include <linux/kvm_host.h> +#include <kvm/arm_spe.h> +#include <asm/barrier.h> #include <asm/cpufeature.h> #include <asm/mmu.h> drivers/perf/arm_spe_pmu.c static void arm_spe_pmu_dev_teardown(struct arm_spe_pmu *spe_pmu) · line 1139
Quoted 10 lines
drivers/perf/arm_spe_pmu.c +7
-0@@ -1127,6 +1130,24 @@ static void arm_spe_pmu_dev_teardown(struct arm_spe_pmu *spe_pmu)free_percpu_irq (spe_pmu->irq, spe_pmu->handle); } +#ifdef CONFIG_KVM_ARM_SPE +static struct arm_spe_kvm_info arm_spe_kvm_info; + +struct arm_spe_kvm_info *arm_spe_get_kvm_info (void ) +{ +return &arm_spe_kvm_info; +}How does this work when SPE is built as a module?
Quoted 10 lines
drivers/perf/arm_spe_pmu.c static void arm_spe_pmu_dev_teardown(struct arm_spe_pmu *spe_pmu) · line 1145
Quoted 6 lines
drivers/perf/arm_spe_pmu.c +6
-0+ +static void arm_spe_populate_kvm_info (struct arm_spe_pmu *spe_pmu) +{ +WARN_ON_ONCE (arm_spe_kvm_info.physical_irq !=0 && + arm_spe_kvm_info.physical_irq != spe_pmu->irq); + arm_spe_kvm_info.physical_irq = spe_pmu->irq;What does 'physical' means here? It's an IRQ in the Linux sense, so it's already some random number that bears no relation to anything 'physical'.
It's some random number relating to the SPE device as opposed to the virtual SPE device.
Thanks,
Andrew Murray
Quoted 40 lines
Quoted 25 lines
drivers/perf/arm_spe_pmu.c +7
-0+} +#else +static void arm_spe_populate_kvm_info (struct arm_spe_pmu *spe_pmu) {} +#endif +/* Driver and device probing */ static int arm_spe_pmu_irq_probe (struct arm_spe_pmu *spe_pmu) {@@ -1149,6 +1170,8 @@ static int arm_spe_pmu_irq_probe(struct arm_spe_pmu *spe_pmu) } spe_pmu->irq = irq; +arm_spe_populate_kvm_info (spe_pmu); +return 0 ; }include/kvm/arm_spe.h +0
-0@@ -17,6 +17,12 @@ struct kvm_spe {bool irq_level; };include/kvm/arm_spe.h struct kvm_spe { · line 28
Quoted 10 lines
include/kvm/arm_spe.h +6
-0+struct arm_spe_kvm_info { +int physical_irq; +}; + +struct arm_spe_kvm_info *arm_spe_get_kvm_info (void ); +#ifdef CONFIG_KVM_ARM_SPE #define kvm_arm_spe_v1_ready (v) ((v)->arch.spe.ready)#define kvm_arm_spe_irq_initialized (v) \M.Signature · 1 line
Jazz is not dead, it just smells funny.
PATCH v2 14/18 KVM: arm64: spe: Provide guest virtual interrupts for SPE
In reply to Andrew MurrayUpon the exit of a guest, let's determine if the SPE device has generated an interrupt - if so we'll inject a virtual interrupt to the guest.
Upon the entry and exit of a guest we'll also update the state of the physical IRQ such that it is active when a guest interrupt is pending and the guest is running.
Finally we map the physical IRQ to the virtual IRQ such that the guest can deactivate the interrupt when it handles the interrupt.
--- include/kvm/arm_spe.h | 6 ++++ virt/kvm/arm/arm.c | 5 ++- virt/kvm/arm/spe.c | 71 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 81 insertions(+), 1 deletion(-)
include/kvm/arm_spe.h +6 -0
@@ -37,6 +37,9 @@ static inline bool kvm_arm_support_spe_v1(void) ID_AA64DFR0_PMSVER_SHIFT); } +void kvm_spe_flush_hwstate (struct kvm_vcpu *vcpu); +inline void kvm_spe_sync_hwstate (struct kvm_vcpu *vcpu); +int kvm_arm_spe_v1_set_attr (struct kvm_vcpu *vcpu,struct kvm_device_attr *attr); int kvm_arm_spe_v1_get_attr(struct kvm_vcpu *vcpu,
@@ -49,6 +52,9 @@ int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu);#define kvm_arm_support_spe_v1 () (false)#define kvm_arm_spe_irq_initialized (v) (false) +static inline void kvm_spe_flush_hwstate (struct kvm_vcpu *vcpu) {} +static inline void kvm_spe_sync_hwstate (struct kvm_vcpu *vcpu) {} +static inline int kvm_arm_spe_v1_set_attr (struct kvm_vcpu *vcpu,struct kvm_device_attr *attr) {
virt/kvm/arm/arm.c +4 -1
@@ -741,6 +741,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)preempt_disable ();kvm_pmu_flush_hwstate (vcpu); +kvm_spe_flush_hwstate (vcpu);local_irq_disable ();
@@ -782,6 +783,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run) kvm_request_pending(vcpu)) { vcpu->mode =OUTSIDE_GUEST_MODE ;isb ();/* Ensure work in x_flush_hwstate is committed */ +kvm_spe_sync_hwstate (vcpu);kvm_pmu_sync_hwstate (vcpu);if (static_branch_unlikely (&userspace_irqchip_in_use))kvm_timer_sync_hwstate (vcpu);
@@ -816,11 +818,12 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)kvm_arm_clear_debug (vcpu);/* -* We must sync the PMU state before the vgic state so +* We must sync the PMU and SPE state before the vgic state so * that the vgic can properly sample the updated state of the * interrupt line. */ kvm_pmu_sync_hwstate (vcpu); +kvm_spe_sync_hwstate (vcpu);/* * Sync the vgic state before syncing the timer state because
virt/kvm/arm/spe.c +71 -0
93 lines not on this page · raw message
Signature · 6 lines
2.21.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerRe: [PATCH v2 14/18] KVM: arm64: spe: Provide guest virtual interrupts for SPE
Quoted 63 lines · Andrew Murray
Upon the exit of a guest, let's determine if the SPE device has generated an interrupt - if so we'll inject a virtual interrupt to the guest.
Upon the entry and exit of a guest we'll also update the state of the physical IRQ such that it is active when a guest interrupt is pending and the guest is running.
Finally we map the physical IRQ to the virtual IRQ such that the guest can deactivate the interrupt when it handles the interrupt.
Signed-off-by Andrew Murray <[email protected]>--- include/kvm/arm_spe.h | 6 ++++ virt/kvm/arm/arm.c | 5 ++- virt/kvm/arm/spe.c | 71 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 81 insertions(+), 1 deletion(-)include/kvm/arm_spe.h +6
-0@@ -37,6 +37,9 @@ static inline bool kvm_arm_support_spe_v1(void) ID_AA64DFR0_PMSVER_SHIFT); } +void kvm_spe_flush_hwstate (struct kvm_vcpu *vcpu); +inline void kvm_spe_sync_hwstate (struct kvm_vcpu *vcpu); +int kvm_arm_spe_v1_set_attr (struct kvm_vcpu *vcpu,struct kvm_device_attr *attr); int kvm_arm_spe_v1_get_attr(struct kvm_vcpu *vcpu,@@ -49,6 +52,9 @@ int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu);#define kvm_arm_support_spe_v1 () (false)#define kvm_arm_spe_irq_initialized (v) (false) +static inline void kvm_spe_flush_hwstate (struct kvm_vcpu *vcpu) {} +static inline void kvm_spe_sync_hwstate (struct kvm_vcpu *vcpu) {} +static inline int kvm_arm_spe_v1_set_attr (struct kvm_vcpu *vcpu,struct kvm_device_attr *attr) {virt/kvm/arm/arm.c +2
-0@@ -741,6 +741,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)preempt_disable ();kvm_pmu_flush_hwstate (vcpu); +kvm_spe_flush_hwstate (vcpu);local_irq_disable ();@@ -782,6 +783,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run) kvm_request_pending(vcpu)) { vcpu->mode =OUTSIDE_GUEST_MODE ;isb ();/* Ensure work in x_flush_hwstate is committed */ +kvm_spe_sync_hwstate (vcpu);kvm_pmu_sync_hwstate (vcpu);if (static_branch_unlikely (&userspace_irqchip_in_use))kvm_timer_sync_hwstate (vcpu);
virt/kvm/arm/arm.c int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run) · line 826
Quoted 10 lines
virt/kvm/arm/arm.c +2
-1@@ -816,11 +818,12 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)kvm_arm_clear_debug (vcpu);/* -* We must sync the PMU state before the vgic state so +* We must sync the PMU and SPE state before the vgic state so * that the vgic can properly sample the updated state of the * interrupt line. */ kvm_pmu_sync_hwstate (vcpu); +kvm_spe_sync_hwstate (vcpu);
The *HUGE* difference is that the PMU is purely a virtual interrupt, while you're trying to deal with a HW interrupt here.
Quoted 25 lines
virt/kvm/arm/arm.c +0
-0/* * Sync the vgic state before syncing the timer state because virt/kvm/arm/spe.c +14
-0@@ -35,6 +35,68 @@ int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu)return 0 ; } +static inline void set_spe_irq_phys_active (struct arm_spe_kvm_info *info, +bool active) +{ +int r; + r =irq_set_irqchip_state (info->physical_irq,IRQCHIP_STATE_ACTIVE , + active); +WARN_ON (r); +} + +void kvm_spe_flush_hwstate (struct kvm_vcpu *vcpu) +{ +struct kvm_spe *spe = &vcpu->arch.spe; +bool phys_active = false; +struct arm_spe_kvm_info *info =arm_spe_get_kvm_info ();
virt/kvm/arm/spe.c int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu) · line 61
Quoted 10 lines
virt/kvm/arm/spe.c +10
-0+ +if (!kvm_arm_spe_v1_ready (vcpu)) +return ; + +if (irqchip_in_kernel (vcpu->kvm)) + phys_active =kvm_vgic_map_is_active (vcpu, spe->irq_num); + + phys_active |= spe->irq_level; + +set_spe_irq_phys_active (info, phys_active);
So you're happy to mess with the HW interrupt state even when you don't have a HW irqchip? If you are going to copy paste the timer code here, you'd need to support it all the way (no, don't).
Quoted 27 lines
virt/kvm/arm/spe.c +27
-0+} + +void kvm_spe_sync_hwstate (struct kvm_vcpu *vcpu) +{ +struct kvm_spe *spe = &vcpu->arch.spe; +u64 pmbsr; +int r; +bool service; +struct kvm_cpu_context *ctxt = &vcpu->arch.ctxt; +struct arm_spe_kvm_info *info =arm_spe_get_kvm_info (); + +if (!kvm_arm_spe_v1_ready (vcpu)) +return ; + +set_spe_irq_phys_active (info, false); + + pmbsr = ctxt->sys_regs[PMBSR_EL1 ]; + service = !!(pmbsr &BIT (SYS_PMBSR_EL1_S_SHIFT )); +if (spe->irq_level == service) +return ; + + spe->irq_level = service; + +if (likely (irqchip_in_kernel (vcpu->kvm))) { + r =kvm_vgic_inject_irq (vcpu->kvm, vcpu->vcpu_id, + spe->irq_num, service, spe); +WARN_ON (r);
virt/kvm/arm/spe.c int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu) · line 98
Quoted 10 lines
virt/kvm/arm/spe.c +10
-0+ } +} + +static inline bool kvm_arch_arm_spe_v1_get_input_level (int vintid) +{ +struct kvm_vcpu *vcpu =kvm_arm_get_running_vcpu (); +struct kvm_spe *spe = &vcpu->arch.spe; + +return spe->irq_level; +}
This isn't what such a callback is for. It is supposed to sample the HW, an nothing else.
Quoted 18 lines
virt/kvm/arm/spe.c +4
-0+static int kvm_arm_spe_v1_init (struct kvm_vcpu *vcpu) {if (!kvm_arm_support_spe_v1 ())@@ -48,6 +110,7 @@ static int kvm_arm_spe_v1_init(struct kvm_vcpu *vcpu)if (irqchip_in_kernel (vcpu->kvm)) {int ret; +struct arm_spe_kvm_info *info;/* * If using the SPE with an in-kernel virtual GIC @@ -57,10 +120,18 @@ static int kvm_arm_spe_v1_init(struct kvm_vcpu *vcpu)if (!vgic_initialized (vcpu->kvm))return -ENODEV ; + info =arm_spe_get_kvm_info (); +if (!info->physical_irq)
virt/kvm/arm/spe.c static int kvm_arm_spe_v1_init(struct kvm_vcpu *vcpu) · line 134
Quoted 10 lines
virt/kvm/arm/spe.c +6
-0+return -ENODEV ; + ret =kvm_vgic_set_owner (vcpu, vcpu->arch.spe.irq_num, &vcpu->arch.spe);if (ret)return ret; + + ret =kvm_vgic_map_phys_irq (vcpu, info->physical_irq, + vcpu->arch.spe.irq_num, + kvm_arch_arm_spe_v1_get_input_level);
You're mapping the interrupt int the guest, and yet you have never forwarded the interrupt the first place. All this flow is only going to wreck the host driver as soon as an interrupt occurs.
I think you should rethink the interrupt handling altogether. It would make more sense if the interrupt was actually completely virtualized. If you can isolate the guest state and compute the interrupt state in SW (and from the above, it seems that you can), then you shouldn't mess with the whole forwarding *at all*, as it isn't designed for devices shared between host and guests.
M.
Signature · 1 line
Jazz is not dead, it just smells funny.
Re: [PATCH v2 14/18] KVM: arm64: spe: Provide guest virtual interrupts for SPE
Quoted 193 lines · Marc Zyngier
Quoted 63 lines · Andrew Murray
Upon the exit of a guest, let's determine if the SPE device has generated an interrupt - if so we'll inject a virtual interrupt to the guest.
Upon the entry and exit of a guest we'll also update the state of the physical IRQ such that it is active when a guest interrupt is pending and the guest is running.
Finally we map the physical IRQ to the virtual IRQ such that the guest can deactivate the interrupt when it handles the interrupt.
Signed-off-by Andrew Murray <[email protected]>--- include/kvm/arm_spe.h | 6 ++++ virt/kvm/arm/arm.c | 5 ++- virt/kvm/arm/spe.c | 71 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 81 insertions(+), 1 deletion(-)include/kvm/arm_spe.h +6
-0@@ -37,6 +37,9 @@ static inline bool kvm_arm_support_spe_v1(void) ID_AA64DFR0_PMSVER_SHIFT); } +void kvm_spe_flush_hwstate (struct kvm_vcpu *vcpu); +inline void kvm_spe_sync_hwstate (struct kvm_vcpu *vcpu); +int kvm_arm_spe_v1_set_attr (struct kvm_vcpu *vcpu,struct kvm_device_attr *attr); int kvm_arm_spe_v1_get_attr(struct kvm_vcpu *vcpu,@@ -49,6 +52,9 @@ int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu);#define kvm_arm_support_spe_v1 () (false)#define kvm_arm_spe_irq_initialized (v) (false) +static inline void kvm_spe_flush_hwstate (struct kvm_vcpu *vcpu) {} +static inline void kvm_spe_sync_hwstate (struct kvm_vcpu *vcpu) {} +static inline int kvm_arm_spe_v1_set_attr (struct kvm_vcpu *vcpu,struct kvm_device_attr *attr) {virt/kvm/arm/arm.c +2
-0@@ -741,6 +741,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)preempt_disable ();kvm_pmu_flush_hwstate (vcpu); +kvm_spe_flush_hwstate (vcpu);local_irq_disable ();@@ -782,6 +783,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run) kvm_request_pending(vcpu)) { vcpu->mode =OUTSIDE_GUEST_MODE ;isb ();/* Ensure work in x_flush_hwstate is committed */ +kvm_spe_sync_hwstate (vcpu);kvm_pmu_sync_hwstate (vcpu);if (static_branch_unlikely (&userspace_irqchip_in_use))kvm_timer_sync_hwstate (vcpu);virt/kvm/arm/arm.c int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run) · line 826
Quoted 10 lines
virt/kvm/arm/arm.c +2
-1@@ -816,11 +818,12 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)kvm_arm_clear_debug (vcpu);/* -* We must sync the PMU state before the vgic state so +* We must sync the PMU and SPE state before the vgic state so * that the vgic can properly sample the updated state of the * interrupt line. */ kvm_pmu_sync_hwstate (vcpu); +kvm_spe_sync_hwstate (vcpu);The *HUGE* difference is that the PMU is purely a virtual interrupt, while you're trying to deal with a HW interrupt here.
Quoted 25 lines
virt/kvm/arm/arm.c +0
-0/* * Sync the vgic state before syncing the timer state because virt/kvm/arm/spe.c +14
-0@@ -35,6 +35,68 @@ int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu)return 0 ; } +static inline void set_spe_irq_phys_active (struct arm_spe_kvm_info *info, +bool active) +{ +int r; + r =irq_set_irqchip_state (info->physical_irq,IRQCHIP_STATE_ACTIVE , + active); +WARN_ON (r); +} + +void kvm_spe_flush_hwstate (struct kvm_vcpu *vcpu) +{ +struct kvm_spe *spe = &vcpu->arch.spe; +bool phys_active = false; +struct arm_spe_kvm_info *info =arm_spe_get_kvm_info ();virt/kvm/arm/spe.c int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu) · line 61
Quoted 10 lines
virt/kvm/arm/spe.c +10
-0+ +if (!kvm_arm_spe_v1_ready (vcpu)) +return ; + +if (irqchip_in_kernel (vcpu->kvm)) + phys_active =kvm_vgic_map_is_active (vcpu, spe->irq_num); + + phys_active |= spe->irq_level; + +set_spe_irq_phys_active (info, phys_active);So you're happy to mess with the HW interrupt state even when you don't have a HW irqchip? If you are going to copy paste the timer code here, you'd need to support it all the way (no, don't).
Quoted 27 lines
virt/kvm/arm/spe.c +27
-0+} + +void kvm_spe_sync_hwstate (struct kvm_vcpu *vcpu) +{ +struct kvm_spe *spe = &vcpu->arch.spe; +u64 pmbsr; +int r; +bool service; +struct kvm_cpu_context *ctxt = &vcpu->arch.ctxt; +struct arm_spe_kvm_info *info =arm_spe_get_kvm_info (); + +if (!kvm_arm_spe_v1_ready (vcpu)) +return ; + +set_spe_irq_phys_active (info, false); + + pmbsr = ctxt->sys_regs[PMBSR_EL1 ]; + service = !!(pmbsr &BIT (SYS_PMBSR_EL1_S_SHIFT )); +if (spe->irq_level == service) +return ; + + spe->irq_level = service; + +if (likely (irqchip_in_kernel (vcpu->kvm))) { + r =kvm_vgic_inject_irq (vcpu->kvm, vcpu->vcpu_id, + spe->irq_num, service, spe); +WARN_ON (r);virt/kvm/arm/spe.c int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu) · line 98
Quoted 10 lines
virt/kvm/arm/spe.c +10
-0+ } +} + +static inline bool kvm_arch_arm_spe_v1_get_input_level (int vintid) +{ +struct kvm_vcpu *vcpu =kvm_arm_get_running_vcpu (); +struct kvm_spe *spe = &vcpu->arch.spe; + +return spe->irq_level; +}This isn't what such a callback is for. It is supposed to sample the HW, an nothing else.
Quoted 18 lines
virt/kvm/arm/spe.c +4
-0+static int kvm_arm_spe_v1_init (struct kvm_vcpu *vcpu) {if (!kvm_arm_support_spe_v1 ())@@ -48,6 +110,7 @@ static int kvm_arm_spe_v1_init(struct kvm_vcpu *vcpu)if (irqchip_in_kernel (vcpu->kvm)) {int ret; +struct arm_spe_kvm_info *info;/* * If using the SPE with an in-kernel virtual GIC @@ -57,10 +120,18 @@ static int kvm_arm_spe_v1_init(struct kvm_vcpu *vcpu)if (!vgic_initialized (vcpu->kvm))return -ENODEV ; + info =arm_spe_get_kvm_info (); +if (!info->physical_irq)virt/kvm/arm/spe.c static int kvm_arm_spe_v1_init(struct kvm_vcpu *vcpu) · line 134
Quoted 10 lines
virt/kvm/arm/spe.c +6
-0+return -ENODEV ; + ret =kvm_vgic_set_owner (vcpu, vcpu->arch.spe.irq_num, &vcpu->arch.spe);if (ret)return ret; + + ret =kvm_vgic_map_phys_irq (vcpu, info->physical_irq, + vcpu->arch.spe.irq_num, + kvm_arch_arm_spe_v1_get_input_level);You're mapping the interrupt int the guest, and yet you have never forwarded the interrupt the first place. All this flow is only going to wreck the host driver as soon as an interrupt occurs.
Quoted 6 lines
I think you should rethink the interrupt handling altogether. It would make more sense if the interrupt was actually completely virtualized. If you can isolate the guest state and compute the interrupt state in SW (and from the above, it seems that you can), then you shouldn't mess with the whole forwarding *at all*, as it isn't designed for devices shared between host and guests.
Yes it's possible to read SYS_PMBSR_EL1_S_SHIFT and determine if SPE wants service. If I understand correctly, you're suggesting on entry/exit to the guest we determine this and inject an interrupt to the guest. As well as removing the kvm_vgic_map_phys_irq mapping to the physical interrupt?
My understanding was that I needed knowledge of the physical SPE interrupt number so that I could prevent the host SPE driver from getting spurious interrupts due to guest use of the SPE.
Thanks,
Andrew Murray
Quoted 5 lines
M.Signature · 1 line
Jazz is not dead, it just smells funny.
Marc Zyngier
maintainerRe: [PATCH v2 14/18] KVM: arm64: spe: Provide guest virtual interrupts for SPE
Quoted 218 lines · Andrew Murray
Quoted 210 lines · Marc Zyngier
Quoted 2 lines · Andrew Murray
Upon the exit of a guest, let's determine if the SPE device has
generated
Quoted 1 line
an interrupt - if so we'll inject a virtual interrupt to the
guest.
Quoted 2 lines
Upon the entry and exit of a guest we'll also update the state of
the
Quoted 1 line
physical IRQ such that it is active when a guest interrupt is
pending
Quoted 3 lines
and the guest is running.
Finally we map the physical IRQ to the virtual IRQ such that the
guest
Quoted 7 lines
can deactivate the interrupt when it handles the interrupt.
Signed-off-by Andrew Murray <[email protected]>--- include/kvm/arm_spe.h | 6 ++++ virt/kvm/arm/arm.c | 5 ++- virt/kvm/arm/spe.c | 71+++++++++++++++++++++++++++++++++++++++++++
include/kvm/arm_spe.h int kvm_arm_spe_v1_enable(struct kvm_vcpu
Quoted 17 lines
3 files changed, 81 insertions(+), 1 deletion(-)include/kvm/arm_spe.h +3
-0@@ -37,6 +37,9 @@ static inline bool kvm_arm_support_spe_v1(void) ID_AA64DFR0_PMSVER_SHIFT); } +void kvm_spe_flush_hwstate (struct kvm_vcpu *vcpu); +inline void kvm_spe_sync_hwstate (struct kvm_vcpu *vcpu); +int kvm_arm_spe_v1_set_attr (struct kvm_vcpu *vcpu,struct kvm_device_attr *attr); int kvm_arm_spe_v1_get_attr(struct kvm_vcpu *vcpu,@@ -49,6 +52,9 @@ int kvm_arm_spe_v1_enable(struct kvm_vcpu*vcpu);
include/kvm/arm_spe.h int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu);
Quoted 4 lines
include/kvm/arm_spe.h +1
-0#define kvm_arm_support_spe_v1 () (false)#define kvm_arm_spe_irq_initialized (v) (false) +static inline void kvm_spe_flush_hwstate (struct kvm_vcpu *vcpu){}
virt/kvm/arm/arm.c int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu
Quoted 10 lines
include/kvm/arm_spe.h +2
-0+static inline void kvm_spe_sync_hwstate (struct kvm_vcpu *vcpu) {} +static inline int kvm_arm_spe_v1_set_attr (struct kvm_vcpu *vcpu,struct kvm_device_attr *attr) {virt/kvm/arm/arm.c +0
-0@@ -741,6 +741,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu*vcpu, struct
kvm_run*run)virt/kvm/arm/arm.c int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu
Quoted 8 lines
virt/kvm/arm/arm.c +1
-0preempt_disable ();kvm_pmu_flush_hwstate (vcpu); +kvm_spe_flush_hwstate (vcpu);local_irq_disable ();@@ -782,6 +783,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu*vcpu, struct
kvm_run*run)virt/kvm/arm/arm.c int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu
Quoted 8 lines
virt/kvm/arm/arm.c +1
-0kvm_request_pending(vcpu)) { vcpu->mode =OUTSIDE_GUEST_MODE ;isb ();/* Ensure work in x_flush_hwstate is committed */ +kvm_spe_sync_hwstate (vcpu);kvm_pmu_sync_hwstate (vcpu);if (static_branch_unlikely (&userspace_irqchip_in_use))kvm_timer_sync_hwstate (vcpu);@@ -816,11 +818,12 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu*vcpu, struct
kvm_run*run)virt/kvm/arm/arm.c int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run) · line 826
Quoted 10 lines
virt/kvm/arm/arm.c +2
-1kvm_arm_clear_debug (vcpu);/* -* We must sync the PMU state before the vgic state so +* We must sync the PMU and SPE state before the vgic state so * that the vgic can properly sample the updated state of the * interrupt line. */ kvm_pmu_sync_hwstate (vcpu); +kvm_spe_sync_hwstate (vcpu);The *HUGE* difference is that the PMU is purely a virtual interrupt, while you're trying to deal with a HW interrupt here.
virt/kvm/arm/spe.c int kvm_arm_spe_v1_enable(struct kvm_vcpu
Quoted 8 lines
virt/kvm/arm/arm.c +0
-0/* * Sync the vgic state before syncing the timer state because virt/kvm/arm/spe.c +0
-0@@ -35,6 +35,68 @@ int kvm_arm_spe_v1_enable(struct kvm_vcpu*vcpu)
virt/kvm/arm/spe.c int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu)
Quoted 4 lines
virt/kvm/arm/spe.c +1
-0return 0 ; } +static inline void set_spe_irq_phys_active(struct
arm_spe_kvm_info*info,virt/kvm/arm/spe.c int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu)
Quoted 4 lines
virt/kvm/arm/spe.c +4
-0+ bool active) +{ +int r; + r = irq_set_irqchip_state(info->physical_irq,
IRQCHIP_STATE_ACTIVE,Quoted 9 lines
virt/kvm/arm/spe.c +9
-0+ active); +WARN_ON (r); +} + +void kvm_spe_flush_hwstate (struct kvm_vcpu *vcpu) +{ +struct kvm_spe *spe = &vcpu->arch.spe; +bool phys_active = false; +struct arm_spe_kvm_info *info =arm_spe_get_kvm_info ();virt/kvm/arm/spe.c int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu) · line 61
Quoted 10 lines
virt/kvm/arm/spe.c +10
-0+ +if (!kvm_arm_spe_v1_ready (vcpu)) +return ; + +if (irqchip_in_kernel (vcpu->kvm)) + phys_active =kvm_vgic_map_is_active (vcpu, spe->irq_num); + + phys_active |= spe->irq_level; + +set_spe_irq_phys_active (info, phys_active);So you're happy to mess with the HW interrupt state even when you don't have a HW irqchip? If you are going to copy paste the timer code here, you'd need to support it all the way (no, don't).
Quoted 21 lines
virt/kvm/arm/spe.c +21
-0+} + +void kvm_spe_sync_hwstate (struct kvm_vcpu *vcpu) +{ +struct kvm_spe *spe = &vcpu->arch.spe; +u64 pmbsr; +int r; +bool service; +struct kvm_cpu_context *ctxt = &vcpu->arch.ctxt; +struct arm_spe_kvm_info *info =arm_spe_get_kvm_info (); + +if (!kvm_arm_spe_v1_ready (vcpu)) +return ; + +set_spe_irq_phys_active (info, false); + + pmbsr = ctxt->sys_regs[PMBSR_EL1 ]; + service = !!(pmbsr &BIT (SYS_PMBSR_EL1_S_SHIFT )); +if (spe->irq_level == service) +return ; +virt/kvm/arm/spe.c int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu)
Quoted 10 lines
virt/kvm/arm/spe.c +10
-0+ spe->irq_level = service; + +if (likely (irqchip_in_kernel (vcpu->kvm))) { + r =kvm_vgic_inject_irq (vcpu->kvm, vcpu->vcpu_id, + spe->irq_num, service, spe); +WARN_ON (r); + } +} + +static inline bool kvm_arch_arm_spe_v1_get_input_level(intvintid)
virt/kvm/arm/spe.c int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu) · line 98
Quoted 6 lines
virt/kvm/arm/spe.c +6
-0+{ +struct kvm_vcpu *vcpu =kvm_arm_get_running_vcpu (); +struct kvm_spe *spe = &vcpu->arch.spe; + +return spe->irq_level; +}This isn't what such a callback is for. It is supposed to sample the HW, an nothing else.
virt/kvm/arm/spe.c static int kvm_arm_spe_v1_init(struct kvm_vcpu
Quoted 5 lines
virt/kvm/arm/spe.c +1
-0+static int kvm_arm_spe_v1_init (struct kvm_vcpu *vcpu) {if (!kvm_arm_support_spe_v1 ())@@ -48,6 +110,7 @@ static int kvm_arm_spe_v1_init(struct kvm_vcpu*vcpu)
virt/kvm/arm/spe.c static int kvm_arm_spe_v1_init(struct
Quoted 8 lines
virt/kvm/arm/spe.c +1
-0if (irqchip_in_kernel (vcpu->kvm)) {int ret; +struct arm_spe_kvm_info *info;/* * If using the SPE with an in-kernel virtual GIC @@ -57,10 +120,18 @@ static int kvm_arm_spe_v1_init(struct
kvm_vcpu*vcpu)Quoted 5 lines
virt/kvm/arm/spe.c +2
-0if (!vgic_initialized (vcpu->kvm))return -ENODEV ; + info =arm_spe_get_kvm_info (); +if (!info->physical_irq)virt/kvm/arm/spe.c static int kvm_arm_spe_v1_init(struct kvm_vcpu *vcpu) · line 134
Quoted 10 lines
virt/kvm/arm/spe.c +6
-0+return -ENODEV ; + ret =kvm_vgic_set_owner (vcpu, vcpu->arch.spe.irq_num, &vcpu->arch.spe);if (ret)return ret; + + ret =kvm_vgic_map_phys_irq (vcpu, info->physical_irq, + vcpu->arch.spe.irq_num, + kvm_arch_arm_spe_v1_get_input_level);You're mapping the interrupt int the guest, and yet you have never forwarded the interrupt the first place. All this flow is only going to wreck the host driver as soon as an interrupt occurs.
Quoted 6 lines
I think you should rethink the interrupt handling altogether. It would make more sense if the interrupt was actually completely virtualized. If you can isolate the guest state and compute the interrupt state in SW (and from the above, it seems that you can), then you shouldn't mess with the whole forwarding *at all*, as it isn't designed for devices shared between host and guests.
Quoted 4 lines
Yes it's possible to read
SYS_PMBSR_EL1_S_SHIFTand determine if SPE wants service. If I understand correctly, you're suggesting on entry/exit to the guest we determine this and inject an interrupt to the guest. As well as removing thekvm_vgic_map_phys_irqmapping to the physical interrupt?
The mapping only makes sense for devices that have their interrupt forwarded to a vcpu, where the expected flow is that the interrupt is taken on the host with a normal interrupt handler and then injected in the guest (you still have to manage the active state though). The basic assumption is that such a device is entirely owned by KVM.
Here, you're abusing the mapping interface: you don't have an interrupt handler (the host SPE driver owns it), the interrupt isn't forwarded, and yet you're messing with the active state. None of that is expected, and you are in uncharted territory as far as KVM is concerned.
What bothers me the most is that this looks a lot like a previous implementation of the timers, and we had all the problems in the world to keep track of the interrupt state *and* have a reasonable level of performance (hitting the redistributor on the fast path is a performance killer).
Quoted 3 lines
My understanding was that I needed knowledge of the physical SPE interrupt number so that I could prevent the host SPE driver from getting spurious interrupts due to guest use of the SPE.
You can't completely rule out the host getting interrupted. Even if you set PMBSR_EL1.S to zero, there is no guarantee that the host will not observe the interrupt anyway (the GIC architecture doesn't tell you how quickly it will be retired, if ever). The host driver already checks for this anyway.
What you need to ensure is that PMBSR_EL1.S being set on guest entry doesn't immediately kick you out of the guest and prevent forward progress. This is why you need to manage the active state.
The real question is: how quickly do you want to react to a SPE interrupt firing while in a guest?
If you want to take it into account as soon as it fires, then you need to eagerly save/restore the active state together with the SPE state on each entry/exit, and performance will suffer. This is what you are currently doing.
If you're OK with evaluating the interrupt status on exit, but without the interrupt itself causing an exit, then you can simply manage it as a purely virtual interrupt, and just deal with the active state in load/put (set the interrupt as active on load, clear it on put).
Given that SPE interrupts always indicate that profiling has stopped, this only affects the size of the black hole, and I'm inclined to do the latter.
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
Re: [PATCH v2 14/18] KVM: arm64: spe: Provide guest virtual interrupts for SPE
Quoted 228 lines · Marc Zyngier
Quoted 220 lines · Andrew Murray
Quoted 211 lines · Marc Zyngier
Quoted 2 lines · Andrew Murray
Upon the exit of a guest, let's determine if the SPE device has
generated
Quoted 1 line
an interrupt - if so we'll inject a virtual interrupt to the
guest.
Quoted 2 lines
Upon the entry and exit of a guest we'll also update the state of
the
Quoted 1 line
physical IRQ such that it is active when a guest interrupt is
pending
Quoted 3 lines
and the guest is running.
Finally we map the physical IRQ to the virtual IRQ such that the
guest
Quoted 7 lines
can deactivate the interrupt when it handles the interrupt.
Signed-off-by Andrew Murray <[email protected]>--- include/kvm/arm_spe.h | 6 ++++ virt/kvm/arm/arm.c | 5 ++- virt/kvm/arm/spe.c | 71+++++++++++++++++++++++++++++++++++++++++++
include/kvm/arm_spe.h int kvm_arm_spe_v1_enable(struct kvm_vcpu
Quoted 17 lines
3 files changed, 81 insertions(+), 1 deletion(-)include/kvm/arm_spe.h +3
-0@@ -37,6 +37,9 @@ static inline bool kvm_arm_support_spe_v1(void) ID_AA64DFR0_PMSVER_SHIFT); } +void kvm_spe_flush_hwstate (struct kvm_vcpu *vcpu); +inline void kvm_spe_sync_hwstate (struct kvm_vcpu *vcpu); +int kvm_arm_spe_v1_set_attr (struct kvm_vcpu *vcpu,struct kvm_device_attr *attr); int kvm_arm_spe_v1_get_attr(struct kvm_vcpu *vcpu,@@ -49,6 +52,9 @@ int kvm_arm_spe_v1_enable(struct kvm_vcpu*vcpu);
include/kvm/arm_spe.h int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu);
Quoted 4 lines
include/kvm/arm_spe.h +1
-0#define kvm_arm_support_spe_v1 () (false)#define kvm_arm_spe_irq_initialized (v) (false) +static inline void kvm_spe_flush_hwstate (struct kvm_vcpu *vcpu){}
virt/kvm/arm/arm.c int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu
Quoted 10 lines
include/kvm/arm_spe.h +2
-0+static inline void kvm_spe_sync_hwstate (struct kvm_vcpu *vcpu) {} +static inline int kvm_arm_spe_v1_set_attr (struct kvm_vcpu *vcpu,struct kvm_device_attr *attr) {virt/kvm/arm/arm.c +0
-0@@ -741,6 +741,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu*vcpu, struct
kvm_run*run)virt/kvm/arm/arm.c int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu
Quoted 8 lines
virt/kvm/arm/arm.c +1
-0preempt_disable ();kvm_pmu_flush_hwstate (vcpu); +kvm_spe_flush_hwstate (vcpu);local_irq_disable ();@@ -782,6 +783,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu*vcpu, struct
kvm_run*run)virt/kvm/arm/arm.c int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu
Quoted 8 lines
virt/kvm/arm/arm.c +1
-0kvm_request_pending(vcpu)) { vcpu->mode =OUTSIDE_GUEST_MODE ;isb ();/* Ensure work in x_flush_hwstate is committed */ +kvm_spe_sync_hwstate (vcpu);kvm_pmu_sync_hwstate (vcpu);if (static_branch_unlikely (&userspace_irqchip_in_use))kvm_timer_sync_hwstate (vcpu);@@ -816,11 +818,12 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu*vcpu, struct
kvm_run*run)virt/kvm/arm/arm.c int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run) · line 826
Quoted 10 lines
virt/kvm/arm/arm.c +2
-1kvm_arm_clear_debug (vcpu);/* -* We must sync the PMU state before the vgic state so +* We must sync the PMU and SPE state before the vgic state so * that the vgic can properly sample the updated state of the * interrupt line. */ kvm_pmu_sync_hwstate (vcpu); +kvm_spe_sync_hwstate (vcpu);The *HUGE* difference is that the PMU is purely a virtual interrupt, while you're trying to deal with a HW interrupt here.
virt/kvm/arm/spe.c int kvm_arm_spe_v1_enable(struct kvm_vcpu
Quoted 8 lines
virt/kvm/arm/arm.c +0
-0/* * Sync the vgic state before syncing the timer state because virt/kvm/arm/spe.c +0
-0@@ -35,6 +35,68 @@ int kvm_arm_spe_v1_enable(struct kvm_vcpu*vcpu)
virt/kvm/arm/spe.c int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu)
Quoted 4 lines
virt/kvm/arm/spe.c +1
-0return 0 ; } +static inline void set_spe_irq_phys_active(struct
arm_spe_kvm_info*info,virt/kvm/arm/spe.c int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu)
Quoted 4 lines
virt/kvm/arm/spe.c +4
-0+ bool active) +{ +int r; + r = irq_set_irqchip_state(info->physical_irq,
IRQCHIP_STATE_ACTIVE,Quoted 9 lines
virt/kvm/arm/spe.c +9
-0+ active); +WARN_ON (r); +} + +void kvm_spe_flush_hwstate (struct kvm_vcpu *vcpu) +{ +struct kvm_spe *spe = &vcpu->arch.spe; +bool phys_active = false; +struct arm_spe_kvm_info *info =arm_spe_get_kvm_info ();virt/kvm/arm/spe.c int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu) · line 61
Quoted 10 lines
virt/kvm/arm/spe.c +10
-0+ +if (!kvm_arm_spe_v1_ready (vcpu)) +return ; + +if (irqchip_in_kernel (vcpu->kvm)) + phys_active =kvm_vgic_map_is_active (vcpu, spe->irq_num); + + phys_active |= spe->irq_level; + +set_spe_irq_phys_active (info, phys_active);So you're happy to mess with the HW interrupt state even when you don't have a HW irqchip? If you are going to copy paste the timer code
here, you'd need to support it all the way (no, don't).Quoted 21 lines
virt/kvm/arm/spe.c +21
-0+} + +void kvm_spe_sync_hwstate (struct kvm_vcpu *vcpu) +{ +struct kvm_spe *spe = &vcpu->arch.spe; +u64 pmbsr; +int r; +bool service; +struct kvm_cpu_context *ctxt = &vcpu->arch.ctxt; +struct arm_spe_kvm_info *info =arm_spe_get_kvm_info (); + +if (!kvm_arm_spe_v1_ready (vcpu)) +return ; + +set_spe_irq_phys_active (info, false); + + pmbsr = ctxt->sys_regs[PMBSR_EL1 ]; + service = !!(pmbsr &BIT (SYS_PMBSR_EL1_S_SHIFT )); +if (spe->irq_level == service) +return ; +virt/kvm/arm/spe.c int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu)
Quoted 10 lines
virt/kvm/arm/spe.c +10
-0+ spe->irq_level = service; + +if (likely (irqchip_in_kernel (vcpu->kvm))) { + r =kvm_vgic_inject_irq (vcpu->kvm, vcpu->vcpu_id, + spe->irq_num, service, spe); +WARN_ON (r); + } +} + +static inline bool kvm_arch_arm_spe_v1_get_input_level(intvintid)
virt/kvm/arm/spe.c int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu) · line 98
Quoted 6 lines
virt/kvm/arm/spe.c +6
-0+{ +struct kvm_vcpu *vcpu =kvm_arm_get_running_vcpu (); +struct kvm_spe *spe = &vcpu->arch.spe; + +return spe->irq_level; +}This isn't what such a callback is for. It is supposed to sample the HW, an nothing else.
virt/kvm/arm/spe.c static int kvm_arm_spe_v1_init(struct kvm_vcpu
Quoted 5 lines
virt/kvm/arm/spe.c +1
-0+static int kvm_arm_spe_v1_init (struct kvm_vcpu *vcpu) {if (!kvm_arm_support_spe_v1 ())@@ -48,6 +110,7 @@ static int kvm_arm_spe_v1_init(struct kvm_vcpu*vcpu)
virt/kvm/arm/spe.c static int kvm_arm_spe_v1_init(struct
Quoted 8 lines
virt/kvm/arm/spe.c +1
-0if (irqchip_in_kernel (vcpu->kvm)) {int ret; +struct arm_spe_kvm_info *info;/* * If using the SPE with an in-kernel virtual GIC @@ -57,10 +120,18 @@ static int kvm_arm_spe_v1_init(struct
kvm_vcpu*vcpu)Quoted 5 lines
virt/kvm/arm/spe.c +2
-0if (!vgic_initialized (vcpu->kvm))return -ENODEV ; + info =arm_spe_get_kvm_info (); +if (!info->physical_irq)virt/kvm/arm/spe.c static int kvm_arm_spe_v1_init(struct kvm_vcpu *vcpu) · line 134
Quoted 10 lines
virt/kvm/arm/spe.c +6
-0+return -ENODEV ; + ret =kvm_vgic_set_owner (vcpu, vcpu->arch.spe.irq_num, &vcpu->arch.spe);if (ret)return ret; + + ret =kvm_vgic_map_phys_irq (vcpu, info->physical_irq, + vcpu->arch.spe.irq_num, + kvm_arch_arm_spe_v1_get_input_level);You're mapping the interrupt int the guest, and yet you have never forwarded the interrupt the first place. All this flow is only going to wreck the host driver as soon as an interrupt occurs.
Quoted 7 lines
I think you should rethink the interrupt handling altogether. It would
make more sense if the interrupt was actually completely virtualized. If you can isolate the guest state and compute the interrupt state in SW (and from the above, it seems that you can), then you shouldn't mess with the whole forwarding *at all*, as it isn't designed for devices shared between host and guests.Quoted 6 lines
Yes it's possible to read
SYS_PMBSR_EL1_S_SHIFTand determine if SPE wants
service. If I understand correctly, you're suggesting on entry/exit to the
guest we determine this and inject an interrupt to the guest. As well as removing thekvm_vgic_map_phys_irqmapping to the physical interrupt?
Quoted 6 lines
The mapping only makes sense for devices that have their interrupt forwarded to a vcpu, where the expected flow is that the interrupt is taken on the host with a normal interrupt handler and then injected in the guest (you still have to manage the active state though). The basic assumption is that such a device is entirely owned by KVM.
Though the mapping does mean that if the guest handles the guest SPE interrupt it doesn't have to wait for a guest exit before having the SPE interrupt evaluated again (i.e. another SPE interrupt won't cause a guest exit) - thus increasing the size of any black hole.
Quoted 36 lines
Here, you're abusing the mapping interface: you don't have an interrupt handler (the host SPE driver owns it), the interrupt isn't forwarded, and yet you're messing with the active state. None of that is expected, and you are in uncharted territory as far as KVM is concerned.
What bothers me the most is that this looks a lot like a previous implementation of the timers, and we had all the problems in the world to keep track of the interrupt state *and* have a reasonable level of performance (hitting the redistributor on the fast path is a performance killer).
Quoted 4 lines
My understanding was that I needed knowledge of the physical SPE interrupt
number so that I could prevent the host SPE driver from getting spurious interrupts due to guest use of the SPE.You can't completely rule out the host getting interrupted. Even if you set
PMBSR_EL1.Sto zero, there is no guarantee that the host will not observe the interrupt anyway (the GIC architecture doesn't tell you how quickly it will be retired, if ever). The host driver already checks for this anyway.What you need to ensure is that
PMBSR_EL1.Sbeing set on guest entry doesn't immediately kick you out of the guest and prevent forward progress. This is why you need to manage the active state.The real question is: how quickly do you want to react to a SPE interrupt firing while in a guest?
If you want to take it into account as soon as it fires, then you need to eagerly save/restore the active state together with the SPE state on each entry/exit, and performance will suffer. This is what you are currently doing.
Quoted 4 lines
If you're OK with evaluating the interrupt status on exit, but without the interrupt itself causing an exit, then you can simply manage it as a purely virtual interrupt, and just deal with the active state in load/put (set the interrupt as active on load, clear it on put).
This does feel like the pragmatic approach - a larger black hole in exchange for performance. I imagine the blackhole would be naturally reduced on machines with high workloads.
I'll refine the series to take this approach.
Quoted 2 lines
Given that SPE interrupts always indicate that profiling has stopped,
and faults :|
Thanks,
Andrew Murray
Quoted 6 lines
this only affects the size of the black hole, and I'm inclined to do the latter.
M.Signature · 1 line
Jazz is not dead. It just smells funny...
Marc Zyngier
maintainerRe: [PATCH v2 14/18] KVM: arm64: spe: Provide guest virtual interrupts for SPE
Quoted 262 lines · Andrew Murray
Quoted 252 lines · Marc Zyngier
Quoted 5 lines · Andrew Murray
Quoted 4 lines · Marc Zyngier
Quoted 2 lines · Andrew Murray
Upon the exit of a guest, let's determine if the SPE device
has
Quoted 5 lines
Quoted 5 lines
generated
Quoted 1 line
an interrupt - if so we'll inject a virtual interrupt to the
guest.
Quoted 2 lines
Upon the entry and exit of a guest we'll also update the state
of
Quoted 6 lines
Quoted 6 lines
the
Quoted 1 line
physical IRQ such that it is active when a guest interrupt is
pending
Quoted 3 lines
and the guest is running.
Finally we map the physical IRQ to the virtual IRQ such that
the
Quoted 8 lines
Quoted 8 lines
guest
Quoted 7 lines
can deactivate the interrupt when it handles the interrupt.
Signed-off-by Andrew Murray <[email protected]>--- include/kvm/arm_spe.h | 6 ++++ virt/kvm/arm/arm.c | 5 ++- virt/kvm/arm/spe.c | 71Quoted 8 lines
Quoted 8 lines
+++++++++++++++++++++++++++++++++++++++++++
include/kvm/arm_spe.h static inline bool
Quoted 7 lines
3 files changed, 81 insertions(+), 1 deletion(-)include/kvm/arm_spe.h +0
-0@@ -37,6 +37,9 @@ static inline bool
kvm_arm_support_spe_v1(void)Quoted 11 lines
Quoted 11 lines
include/kvm/arm_spe.h int kvm_arm_spe_v1_enable(struct kvm_vcpu
Quoted 10 lines
include/kvm/arm_spe.h +3
-0ID_AA64DFR0_PMSVER_SHIFT); } +void kvm_spe_flush_hwstate (struct kvm_vcpu *vcpu); +inline void kvm_spe_sync_hwstate (struct kvm_vcpu *vcpu); +int kvm_arm_spe_v1_set_attr (struct kvm_vcpu *vcpu,struct kvm_device_attr *attr); int kvm_arm_spe_v1_get_attr(struct kvm_vcpu *vcpu,@@ -49,6 +52,9 @@ int kvm_arm_spe_v1_enable(struct kvm_vcpu*vcpu);
Quoted 4 lines
Quoted 4 lines
include/kvm/arm_spe.h int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu);
Quoted 4 lines
include/kvm/arm_spe.h +1
-0#define kvm_arm_support_spe_v1 () (false)#define kvm_arm_spe_irq_initialized (v) (false) +static inline void kvm_spe_flush_hwstate(struct kvm_vcpu*vcpu)
Quoted 2 lines
Quoted 2 lines
{}
Quoted 1 line
include/kvm/arm_spe.h +1
-0+static inline void kvm_spe_sync_hwstate(struct kvm_vcpu*vcpu) {}
Quoted 2 lines
Quoted 2 lines
include/kvm/arm_spe.h int kvm_arm_spe_v1_enable(struct kvm_vcpu · line 58
Quoted 2 lines
include/kvm/arm_spe.h +1
-0+ static inline int kvm_arm_spe_v1_set_attr(struct kvm_vcpu*vcpu,
Quoted 7 lines
Quoted 7 lines
virt/kvm/arm/arm.c int kvm_arch_vcpu_ioctl_run(struct
Quoted 7 lines
include/kvm/arm_spe.h +0
-0struct kvm_device_attr *attr) {virt/kvm/arm/arm.c +0
-0@@ -741,6 +741,7 @@ int kvm_arch_vcpu_ioctl_run(struct
kvm_vcpuQuoted 9 lines
Quoted 9 lines
*vcpu, struct
kvm_run*run)virt/kvm/arm/arm.c int kvm_arch_vcpu_ioctl_run(struct
Quoted 8 lines
virt/kvm/arm/arm.c +1
-0preempt_disable ();kvm_pmu_flush_hwstate (vcpu); +kvm_spe_flush_hwstate (vcpu);local_irq_disable ();@@ -782,6 +783,7 @@ int kvm_arch_vcpu_ioctl_run(struct
kvm_vcpuQuoted 9 lines
Quoted 9 lines
*vcpu, struct
kvm_run*run)virt/kvm/arm/arm.c int kvm_arch_vcpu_ioctl_run(struct
Quoted 8 lines
virt/kvm/arm/arm.c +1
-0kvm_request_pending(vcpu)) { vcpu->mode =OUTSIDE_GUEST_MODE ;isb ();/* Ensure work in x_flush_hwstate is committed */ +kvm_spe_sync_hwstate (vcpu);kvm_pmu_sync_hwstate (vcpu);if (static_branch_unlikely (&userspace_irqchip_in_use))kvm_timer_sync_hwstate (vcpu);@@ -816,11 +818,12 @@ int kvm_arch_vcpu_ioctl_run(struct
kvm_vcpuQuoted 6 lines
Quoted 6 lines
*vcpu, struct
kvm_run*run)virt/kvm/arm/arm.c int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)
Quoted 5 lines
virt/kvm/arm/arm.c +1
-1kvm_arm_clear_debug (vcpu);/* -* We must sync the PMU state before the vgic state so +* We must sync the PMU and SPE state before the vgic state so
Quoted 1 line
Quoted 1 line
virt/kvm/arm/arm.c int kvm_arch_vcpu_ioctl_run(struct · line 822
Quoted 1 line
virt/kvm/arm/arm.c +0
-0* that the vgic can properly sample the updated state of the
Quoted 6 lines
Quoted 6 lines
virt/kvm/arm/arm.c int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run) · line 826
Quoted 4 lines
virt/kvm/arm/arm.c +1
-0* interrupt line. */ kvm_pmu_sync_hwstate (vcpu); +kvm_spe_sync_hwstate (vcpu);The *HUGE* difference is that the PMU is purely a virtual
interrupt,
Quoted 5 lines
Quoted 5 lines
while you're trying to deal with a HW interrupt here.
Quoted 3 lines
virt/kvm/arm/arm.c +0
-0/* * Sync the vgic state before syncing the timer state because
Quoted 29 lines
Quoted 29 lines
virt/kvm/arm/spe.c int kvm_arm_spe_v1_enable(struct kvm_vcpu
Quoted 5 lines
virt/kvm/arm/spe.c +0
-0@@ -35,6 +35,68 @@ int kvm_arm_spe_v1_enable(struct kvm_vcpu*vcpu)
virt/kvm/arm/spe.c int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu)
Quoted 4 lines
virt/kvm/arm/spe.c +1
-0return 0 ; } +static inline void set_spe_irq_phys_active(struct
arm_spe_kvm_info*info,virt/kvm/arm/spe.c int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu)
Quoted 4 lines
virt/kvm/arm/spe.c +4
-0+ bool active) +{ +int r; + r = irq_set_irqchip_state(info->physical_irq,
IRQCHIP_STATE_ACTIVE,Quoted 13 lines
virt/kvm/arm/spe.c +13
-0+ active); +WARN_ON (r); +} + +void kvm_spe_flush_hwstate (struct kvm_vcpu *vcpu) +{ +struct kvm_spe *spe = &vcpu->arch.spe; +bool phys_active = false; +struct arm_spe_kvm_info *info =arm_spe_get_kvm_info (); + +if (!kvm_arm_spe_v1_ready (vcpu)) +return ; +Quoted 8 lines
Quoted 8 lines
virt/kvm/arm/spe.c int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu) · line 61
Quoted 6 lines
virt/kvm/arm/spe.c +6
-0+if (irqchip_in_kernel (vcpu->kvm)) + phys_active =kvm_vgic_map_is_active (vcpu, spe->irq_num); + + phys_active |= spe->irq_level; + +set_spe_irq_phys_active (info, phys_active);So you're happy to mess with the HW interrupt state even when
you
Quoted 1 line
Quoted 1 line
don't have a HW irqchip? If you are going to copy paste the
timer
Quoted 35 lines
Quoted 35 lines
code
here, you'd need to support it all the way (no, don't).Quoted 21 lines
virt/kvm/arm/spe.c +21
-0+} + +void kvm_spe_sync_hwstate (struct kvm_vcpu *vcpu) +{ +struct kvm_spe *spe = &vcpu->arch.spe; +u64 pmbsr; +int r; +bool service; +struct kvm_cpu_context *ctxt = &vcpu->arch.ctxt; +struct arm_spe_kvm_info *info =arm_spe_get_kvm_info (); + +if (!kvm_arm_spe_v1_ready (vcpu)) +return ; + +set_spe_irq_phys_active (info, false); + + pmbsr = ctxt->sys_regs[PMBSR_EL1 ]; + service = !!(pmbsr &BIT (SYS_PMBSR_EL1_S_SHIFT )); +if (spe->irq_level == service) +return ; +virt/kvm/arm/spe.c int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu)
Quoted 10 lines
virt/kvm/arm/spe.c +10
-0+ spe->irq_level = service; + +if (likely (irqchip_in_kernel (vcpu->kvm))) { + r =kvm_vgic_inject_irq (vcpu->kvm, vcpu->vcpu_id, + spe->irq_num, service, spe); +WARN_ON (r); + } +} + +static inline bool kvm_arch_arm_spe_v1_get_input_level(intvintid)
Quoted 8 lines
Quoted 8 lines
virt/kvm/arm/spe.c int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu) · line 98
Quoted 6 lines
virt/kvm/arm/spe.c +6
-0+{ +struct kvm_vcpu *vcpu =kvm_arm_get_running_vcpu (); +struct kvm_spe *spe = &vcpu->arch.spe; + +return spe->irq_level; +}This isn't what such a callback is for. It is supposed to sample
the
Quoted 7 lines
Quoted 7 lines
HW, an nothing else.
virt/kvm/arm/spe.c static int kvm_arm_spe_v1_init(struct
Quoted 5 lines
virt/kvm/arm/spe.c +1
-0+static int kvm_arm_spe_v1_init (struct kvm_vcpu *vcpu) {if (!kvm_arm_support_spe_v1 ())@@ -48,6 +110,7 @@ static int kvm_arm_spe_v1_init(struct
kvm_vcpuQuoted 13 lines
Quoted 13 lines
*vcpu)
virt/kvm/arm/spe.c static int kvm_arm_spe_v1_init(struct
Quoted 8 lines
virt/kvm/arm/spe.c +1
-0if (irqchip_in_kernel (vcpu->kvm)) {int ret; +struct arm_spe_kvm_info *info;/* * If using the SPE with an in-kernel virtual GIC @@ -57,10 +120,18 @@ static int kvm_arm_spe_v1_init(struct
kvm_vcpu*vcpu)Quoted 3 lines
virt/kvm/arm/spe.c +0
-0if (!vgic_initialized (vcpu->kvm))return -ENODEV ;Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
virt/kvm/arm/spe.c +4
-0+ info =arm_spe_get_kvm_info (); +if (!info->physical_irq) +return -ENODEV ; + ret =kvm_vgic_set_owner (vcpu, vcpu->arch.spe.irq_num, &vcpu->arch.spe);Quoted 8 lines
virt/kvm/arm/spe.c static int kvm_arm_spe_v1_init(struct kvm_vcpu *vcpu) · line 134
Quoted 6 lines
virt/kvm/arm/spe.c +4
-0if (ret)return ret; + + ret =kvm_vgic_map_phys_irq (vcpu, info->physical_irq, + vcpu->arch.spe.irq_num, + kvm_arch_arm_spe_v1_get_input_level);You're mapping the interrupt int the guest, and yet you have
never
Quoted 1 line
Quoted 1 line
forwarded the interrupt the first place. All this flow is only
going
Quoted 7 lines
Quoted 7 lines
to wreck the host driver as soon as an interrupt occurs.
I think you should rethink the interrupt handling altogether. It would
make more sense if the interrupt was actually completely virtualized. If you can isolate the guest state and compute the interrupt state in SW (and from the above, it seems that youcan),
Quoted 1 line
Quoted 1 line
then you shouldn't mess with the whole forwarding *at all*, as
it
Quoted 3 lines
Quoted 1 line
isn't designed for devices shared between host and guests.
Yes it's possible to read
SYS_PMBSR_EL1_S_SHIFTand determine ifSPE
Quoted 2 lines
wants
service. If I understand correctly, you're suggesting onentry/exit to
Quoted 2 lines
the
guest we determine this and inject an interrupt to the guest. Aswell as
Quoted 1 line
removing the
kvm_vgic_map_phys_irqmapping to the physicalQuoted 8 lines
interrupt?
The mapping only makes sense for devices that have their interrupt forwarded to a vcpu, where the expected flow is that the interrupt is taken on the host with a normal interrupt handler and then injected in the guest (you still have to manage the active state though). The basic assumption is that such a device is entirely owned by KVM.
Quoted 4 lines
Though the mapping does mean that if the guest handles the guest SPE interrupt it doesn't have to wait for a guest exit before having the SPE interrupt evaluated again (i.e. another SPE interrupt won't cause a guest exit) - thus increasing the size of any black hole.
Sure. It still remains that your use case is outside of the scope of this internal API.
Quoted 36 lines
Quoted 36 lines
Here, you're abusing the mapping interface: you don't have an interrupt handler (the host SPE driver owns it), the interrupt isn't forwarded, and yet you're messing with the active state. None of that is expected, and you are in uncharted territory as far as KVM is concerned.
What bothers me the most is that this looks a lot like a previous implementation of the timers, and we had all the problems in the world to keep track of the interrupt state *and* have a reasonable level of performance (hitting the redistributor on the fast path is a performance killer).
Quoted 3 lines
My understanding was that I needed knowledge of the physical SPE interrupt
number so that I could prevent the host SPE driver from gettingspurious
Quoted 1 line
interrupts due to guest use of the SPE.
You can't completely rule out the host getting interrupted. Even if you set
PMBSR_EL1.Sto zero, there is no guarantee that the host will not observe the interrupt anyway (the GIC architecture doesn't tell you how quickly it will be retired, if ever). The host driver already checks for this anyway.What you need to ensure is that
PMBSR_EL1.Sbeing set on guest entry doesn't immediately kick you out of the guest and prevent forward progress. This is why you need to manage the active state.The real question is: how quickly do you want to react to a SPE interrupt firing while in a guest?
If you want to take it into account as soon as it fires, then you need to eagerly save/restore the active state together with the SPE state on each entry/exit, and performance will suffer. This is what you are currently doing.
Quoted 8 lines
Quoted 4 lines
If you're OK with evaluating the interrupt status on exit, but without the interrupt itself causing an exit, then you can simply manage it as a purely virtual interrupt, and just deal with the active state in load/put (set the interrupt as active on load, clear it on put).
This does feel like the pragmatic approach - a larger black hole in exchange for performance. I imagine the blackhole would be naturally reduced on machines with high workloads.
Why? I don't see the relation between how busy the vcpu is and the size of the blackhole. It is strictly a function of the frequency of exits.
M.
Quoted 18 lines
I'll refine the series to take this approach.
Quoted 2 lines
Given that SPE interrupts always indicate that profiling has stopped,
and faults :|
Thanks,
Andrew Murray
Quoted 6 lines
this only affects the size of the black hole, and I'm inclined to do the latter.
M.--
Jazz is not dead. It just smells funny...
Signature · 1 line
Jazz is not dead. It just smells funny...
Re: [PATCH v2 14/18] KVM: arm64: spe: Provide guest virtual interrupts for SPE
Quoted 319 lines · Marc Zyngier
Quoted 262 lines · Andrew Murray
Quoted 252 lines · Marc Zyngier
Quoted 5 lines · Andrew Murray
Quoted 4 lines · Marc Zyngier
Quoted 2 lines · Andrew Murray
Upon the exit of a guest, let's determine if the SPE device
has
Quoted 5 lines
Quoted 5 lines
generated
Quoted 1 line
an interrupt - if so we'll inject a virtual interrupt to the
guest.
Quoted 2 lines
Upon the entry and exit of a guest we'll also update the state
of
Quoted 6 lines
Quoted 6 lines
the
Quoted 1 line
physical IRQ such that it is active when a guest interrupt is
pending
Quoted 3 lines
and the guest is running.
Finally we map the physical IRQ to the virtual IRQ such that
the
Quoted 8 lines
Quoted 8 lines
guest
Quoted 7 lines
can deactivate the interrupt when it handles the interrupt.
Signed-off-by Andrew Murray <[email protected]>--- include/kvm/arm_spe.h | 6 ++++ virt/kvm/arm/arm.c | 5 ++- virt/kvm/arm/spe.c | 71Quoted 8 lines
Quoted 8 lines
+++++++++++++++++++++++++++++++++++++++++++
include/kvm/arm_spe.h static inline bool
Quoted 7 lines
3 files changed, 81 insertions(+), 1 deletion(-)include/kvm/arm_spe.h +0
-0@@ -37,6 +37,9 @@ static inline bool
kvm_arm_support_spe_v1(void)Quoted 11 lines
Quoted 11 lines
include/kvm/arm_spe.h int kvm_arm_spe_v1_enable(struct kvm_vcpu
Quoted 10 lines
include/kvm/arm_spe.h +3
-0ID_AA64DFR0_PMSVER_SHIFT); } +void kvm_spe_flush_hwstate (struct kvm_vcpu *vcpu); +inline void kvm_spe_sync_hwstate (struct kvm_vcpu *vcpu); +int kvm_arm_spe_v1_set_attr (struct kvm_vcpu *vcpu,struct kvm_device_attr *attr); int kvm_arm_spe_v1_get_attr(struct kvm_vcpu *vcpu,@@ -49,6 +52,9 @@ int kvm_arm_spe_v1_enable(struct kvm_vcpu*vcpu);
Quoted 4 lines
Quoted 4 lines
include/kvm/arm_spe.h int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu);
Quoted 4 lines
include/kvm/arm_spe.h +1
-0#define kvm_arm_support_spe_v1 () (false)#define kvm_arm_spe_irq_initialized (v) (false) +static inline void kvm_spe_flush_hwstate(struct kvm_vcpu*vcpu)
Quoted 2 lines
Quoted 2 lines
{}
Quoted 1 line
include/kvm/arm_spe.h +1
-0+static inline void kvm_spe_sync_hwstate(struct kvm_vcpu*vcpu) {}
Quoted 2 lines
Quoted 2 lines
include/kvm/arm_spe.h int kvm_arm_spe_v1_enable(struct kvm_vcpu · line 58
Quoted 2 lines
include/kvm/arm_spe.h +1
-0+ static inline int kvm_arm_spe_v1_set_attr(struct kvm_vcpu*vcpu,
Quoted 7 lines
Quoted 7 lines
virt/kvm/arm/arm.c int kvm_arch_vcpu_ioctl_run(struct
Quoted 7 lines
include/kvm/arm_spe.h +0
-0struct kvm_device_attr *attr) {virt/kvm/arm/arm.c +0
-0@@ -741,6 +741,7 @@ int kvm_arch_vcpu_ioctl_run(struct
kvm_vcpuQuoted 9 lines
Quoted 9 lines
*vcpu, struct
kvm_run*run)virt/kvm/arm/arm.c int kvm_arch_vcpu_ioctl_run(struct
Quoted 8 lines
virt/kvm/arm/arm.c +1
-0preempt_disable ();kvm_pmu_flush_hwstate (vcpu); +kvm_spe_flush_hwstate (vcpu);local_irq_disable ();@@ -782,6 +783,7 @@ int kvm_arch_vcpu_ioctl_run(struct
kvm_vcpuQuoted 9 lines
Quoted 9 lines
*vcpu, struct
kvm_run*run)virt/kvm/arm/arm.c int kvm_arch_vcpu_ioctl_run(struct
Quoted 8 lines
virt/kvm/arm/arm.c +1
-0kvm_request_pending(vcpu)) { vcpu->mode =OUTSIDE_GUEST_MODE ;isb ();/* Ensure work in x_flush_hwstate is committed */ +kvm_spe_sync_hwstate (vcpu);kvm_pmu_sync_hwstate (vcpu);if (static_branch_unlikely (&userspace_irqchip_in_use))kvm_timer_sync_hwstate (vcpu);@@ -816,11 +818,12 @@ int kvm_arch_vcpu_ioctl_run(struct
kvm_vcpuQuoted 6 lines
Quoted 6 lines
*vcpu, struct
kvm_run*run)virt/kvm/arm/arm.c int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)
Quoted 5 lines
virt/kvm/arm/arm.c +1
-1kvm_arm_clear_debug (vcpu);/* -* We must sync the PMU state before the vgic state so +* We must sync the PMU and SPE state before the vgic state so
Quoted 1 line
Quoted 1 line
virt/kvm/arm/arm.c int kvm_arch_vcpu_ioctl_run(struct · line 822
Quoted 1 line
virt/kvm/arm/arm.c +0
-0* that the vgic can properly sample the updated state of the
Quoted 6 lines
Quoted 6 lines
virt/kvm/arm/arm.c int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run) · line 826
Quoted 4 lines
virt/kvm/arm/arm.c +1
-0* interrupt line. */ kvm_pmu_sync_hwstate (vcpu); +kvm_spe_sync_hwstate (vcpu);The *HUGE* difference is that the PMU is purely a virtual
interrupt,
Quoted 5 lines
Quoted 5 lines
while you're trying to deal with a HW interrupt here.
Quoted 3 lines
virt/kvm/arm/arm.c +0
-0/* * Sync the vgic state before syncing the timer state because
Quoted 29 lines
Quoted 29 lines
virt/kvm/arm/spe.c int kvm_arm_spe_v1_enable(struct kvm_vcpu
Quoted 5 lines
virt/kvm/arm/spe.c +0
-0@@ -35,6 +35,68 @@ int kvm_arm_spe_v1_enable(struct kvm_vcpu*vcpu)
virt/kvm/arm/spe.c int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu)
Quoted 4 lines
virt/kvm/arm/spe.c +1
-0return 0 ; } +static inline void set_spe_irq_phys_active(struct
arm_spe_kvm_info*info,virt/kvm/arm/spe.c int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu)
Quoted 4 lines
virt/kvm/arm/spe.c +4
-0+ bool active) +{ +int r; + r = irq_set_irqchip_state(info->physical_irq,
IRQCHIP_STATE_ACTIVE,Quoted 13 lines
virt/kvm/arm/spe.c +13
-0+ active); +WARN_ON (r); +} + +void kvm_spe_flush_hwstate (struct kvm_vcpu *vcpu) +{ +struct kvm_spe *spe = &vcpu->arch.spe; +bool phys_active = false; +struct arm_spe_kvm_info *info =arm_spe_get_kvm_info (); + +if (!kvm_arm_spe_v1_ready (vcpu)) +return ; +Quoted 8 lines
Quoted 8 lines
virt/kvm/arm/spe.c int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu) · line 61
Quoted 6 lines
virt/kvm/arm/spe.c +6
-0+if (irqchip_in_kernel (vcpu->kvm)) + phys_active =kvm_vgic_map_is_active (vcpu, spe->irq_num); + + phys_active |= spe->irq_level; + +set_spe_irq_phys_active (info, phys_active);So you're happy to mess with the HW interrupt state even when
you
Quoted 1 line
Quoted 1 line
don't have a HW irqchip? If you are going to copy paste the
timer
Quoted 35 lines
Quoted 35 lines
code
here, you'd need to support it all the way (no, don't).Quoted 21 lines
virt/kvm/arm/spe.c +21
-0+} + +void kvm_spe_sync_hwstate (struct kvm_vcpu *vcpu) +{ +struct kvm_spe *spe = &vcpu->arch.spe; +u64 pmbsr; +int r; +bool service; +struct kvm_cpu_context *ctxt = &vcpu->arch.ctxt; +struct arm_spe_kvm_info *info =arm_spe_get_kvm_info (); + +if (!kvm_arm_spe_v1_ready (vcpu)) +return ; + +set_spe_irq_phys_active (info, false); + + pmbsr = ctxt->sys_regs[PMBSR_EL1 ]; + service = !!(pmbsr &BIT (SYS_PMBSR_EL1_S_SHIFT )); +if (spe->irq_level == service) +return ; +virt/kvm/arm/spe.c int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu)
Quoted 10 lines
virt/kvm/arm/spe.c +10
-0+ spe->irq_level = service; + +if (likely (irqchip_in_kernel (vcpu->kvm))) { + r =kvm_vgic_inject_irq (vcpu->kvm, vcpu->vcpu_id, + spe->irq_num, service, spe); +WARN_ON (r); + } +} + +static inline bool kvm_arch_arm_spe_v1_get_input_level(intvintid)
Quoted 8 lines
Quoted 8 lines
virt/kvm/arm/spe.c int kvm_arm_spe_v1_enable(struct kvm_vcpu *vcpu) · line 98
Quoted 6 lines
virt/kvm/arm/spe.c +6
-0+{ +struct kvm_vcpu *vcpu =kvm_arm_get_running_vcpu (); +struct kvm_spe *spe = &vcpu->arch.spe; + +return spe->irq_level; +}This isn't what such a callback is for. It is supposed to sample
the
Quoted 7 lines
Quoted 7 lines
HW, an nothing else.
virt/kvm/arm/spe.c static int kvm_arm_spe_v1_init(struct
Quoted 5 lines
virt/kvm/arm/spe.c +1
-0+static int kvm_arm_spe_v1_init (struct kvm_vcpu *vcpu) {if (!kvm_arm_support_spe_v1 ())@@ -48,6 +110,7 @@ static int kvm_arm_spe_v1_init(struct
kvm_vcpuQuoted 13 lines
Quoted 13 lines
*vcpu)
virt/kvm/arm/spe.c static int kvm_arm_spe_v1_init(struct
Quoted 8 lines
virt/kvm/arm/spe.c +1
-0if (irqchip_in_kernel (vcpu->kvm)) {int ret; +struct arm_spe_kvm_info *info;/* * If using the SPE with an in-kernel virtual GIC @@ -57,10 +120,18 @@ static int kvm_arm_spe_v1_init(struct
kvm_vcpu*vcpu)Quoted 3 lines
virt/kvm/arm/spe.c +0
-0if (!vgic_initialized (vcpu->kvm))return -ENODEV ;Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
virt/kvm/arm/spe.c +4
-0+ info =arm_spe_get_kvm_info (); +if (!info->physical_irq) +return -ENODEV ; + ret =kvm_vgic_set_owner (vcpu, vcpu->arch.spe.irq_num, &vcpu->arch.spe);Quoted 8 lines
virt/kvm/arm/spe.c static int kvm_arm_spe_v1_init(struct kvm_vcpu *vcpu) · line 134
Quoted 6 lines
virt/kvm/arm/spe.c +4
-0if (ret)return ret; + + ret =kvm_vgic_map_phys_irq (vcpu, info->physical_irq, + vcpu->arch.spe.irq_num, + kvm_arch_arm_spe_v1_get_input_level);You're mapping the interrupt int the guest, and yet you have
never
Quoted 1 line
Quoted 1 line
forwarded the interrupt the first place. All this flow is only
going
Quoted 7 lines
Quoted 7 lines
to wreck the host driver as soon as an interrupt occurs.
I think you should rethink the interrupt handling altogether. It would
make more sense if the interrupt was actually completely virtualized. If you can isolate the guest state and compute the interrupt state in SW (and from the above, it seems that youcan),
Quoted 1 line
Quoted 1 line
then you shouldn't mess with the whole forwarding *at all*, as
it
Quoted 3 lines
Quoted 1 line
isn't designed for devices shared between host and guests.
Yes it's possible to read
SYS_PMBSR_EL1_S_SHIFTand determine ifSPE
Quoted 2 lines
wants
service. If I understand correctly, you're suggesting onentry/exit to
Quoted 2 lines
the
guest we determine this and inject an interrupt to the guest. Aswell as
Quoted 1 line
removing the
kvm_vgic_map_phys_irqmapping to the physicalQuoted 8 lines
interrupt?
The mapping only makes sense for devices that have their interrupt forwarded to a vcpu, where the expected flow is that the interrupt is taken on the host with a normal interrupt handler and then injected in the guest (you still have to manage the active state though). The basic assumption is that such a device is entirely owned by KVM.
Quoted 4 lines
Though the mapping does mean that if the guest handles the guest SPE interrupt it doesn't have to wait for a guest exit before having the SPE interrupt evaluated again (i.e. another SPE interrupt won't cause a guest exit) - thus increasing the size of any black hole.
Sure. It still remains that your use case is outside of the scope of this internal API.
Quoted 48 lines
Quoted 47 lines
Here, you're abusing the mapping interface: you don't have an interrupt handler (the host SPE driver owns it), the interrupt isn't forwarded, and yet you're messing with the active state. None of that is expected, and you are in uncharted territory as far as KVM is concerned.
What bothers me the most is that this looks a lot like a previous implementation of the timers, and we had all the problems in the world to keep track of the interrupt state *and* have a reasonable level of performance (hitting the redistributor on the fast path is a performance killer).
Quoted 3 lines
My understanding was that I needed knowledge of the physical SPE interrupt
number so that I could prevent the host SPE driver from gettingspurious
Quoted 1 line
interrupts due to guest use of the SPE.
You can't completely rule out the host getting interrupted. Even if you set
PMBSR_EL1.Sto zero, there is no guarantee that the host will not observe
the interrupt anyway (the GIC architecture doesn't tell you how quickly
it will be retired, if ever). The host driver already checks for this
anyway.What you need to ensure is that
PMBSR_EL1.Sbeing set on guest entry doesn't immediately kick you out of the guest and prevent forward progress. This is why you need to manage the active state.The real question is: how quickly do you want to react to a SPE interrupt firing while in a guest?
If you want to take it into account as soon as it fires, then you need
to eagerly save/restore the active state together with the SPE state on
each entry/exit, and performance will suffer. This is what you are currently doing.If you're OK with evaluating the interrupt status on exit, but without
the interrupt itself causing an exit, then you can simply manage it as a purely virtual interrupt, and just deal with the active state in load/put (set the interrupt as active on load, clear it on put).
Quoted 7 lines
Quoted 4 lines
This does feel like the pragmatic approach - a larger black hole in exchange
for performance. I imagine the blackhole would be naturally reduced on machines with high workloads.Why? I don't see the relation between how busy the vcpu is and the size of the blackhole. It is strictly a function of the frequency of exits.
Indeed, my assumption being that the busier a system is the more interrupts, thus leading to more exits and so an increased frequency of SPE interrupt evaluation and thus smaller black hole.
Thanks,
Andrew Murray
Quoted 25 lines
M.Quoted 19 lines
I'll refine the series to take this approach.
Quoted 3 lines
Given that SPE interrupts always indicate that profiling has stopped,
and faults :|
Thanks,
Andrew Murray
Quoted 6 lines
this only affects the size of the black hole, and I'm inclined to do the latter.
M.--
Jazz is not dead. It just smells funny...Signature · 1 line
Jazz is not dead. It just smells funny...
Marc Zyngier
maintainerRe: [PATCH v2 14/18] KVM: arm64: spe: Provide guest virtual interrupts for SPE
Quoted 2 lines · Andrew Murray
Quoted 1 line · Marc Zyngier
On 2019-12-24 13:08, Andrew Murray wrote:
[...]
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
This does feel like the pragmatic approach - a larger black hole
in
Quoted 2 lines
exchange
for performance. I imagine the blackhole would be naturallyreduced on
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
machines with high workloads.
Why? I don't see the relation between how busy the vcpu is and the size of the blackhole. It is strictly a function of the frequency of exits.
Indeed, my assumption being that the busier a system is the more interrupts, thus leading to more exits and so an increased frequency of SPE interrupt evaluation and thus smaller black hole.
On a GICv4-enabled system, this isn't true anymore. My bet is that people won't use SPE to optimize IO-oriented workloads, but more CPU intensive workloads (that don't necessarily exit at all).
But never mind. Let's start with this approach, as it is simple and easy to verify. If the black hole aspect becomes problematic, we know how to reduce it (at the expense of entry/exit performance).
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
PATCH v2 15/18 perf: arm_spe: Handle guest/host exclusion flags
In reply to Andrew MurrayA side effect of supporting the SPE in guests is that we prevent the host from collecting data whilst inside a guest thus creating a black-out window. This occurs because instead of emulating the SPE, we share it with our guests.
Let's accurately describe our capabilities by using the perf exclude flags to prevent !exclude_guest and exclude_host flags from being used.
--- drivers/perf/arm_spe_pmu.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/perf/arm_spe_pmu.c +3 -0
@@ -679,6 +679,9 @@ static int arm_spe_pmu_event_init(struct perf_event *event)if (attr->exclude_idle)return -EOPNOTSUPP ; +if (!attr->exclude_guest || attr->exclude_host) +return -EOPNOTSUPP ; +/* * Feedback-directed frequency throttling doesn't work when we * have a buffer of samples. We'd need to manually count the
Signature · 6 lines
2.21.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Mark Rutland
maintainerRe: [PATCH v2 15/18] perf: arm_spe: Handle guest/host exclusion flags
Quoted 4 lines · Andrew Murray
A side effect of supporting the SPE in guests is that we prevent the host from collecting data whilst inside a guest thus creating a black-out window. This occurs because instead of emulating the SPE, we share it with our guests.
We used to permit this; do we know if anyone is using it?
Thanks,
Mark.
Quoted 25 lines
Let's accurately describe our capabilities by using the perf exclude flags to prevent !exclude_guest and
exclude_hostflags from being used.Signed-off-by Andrew Murray <[email protected]>--- drivers/perf/arm_spe_pmu.c | 3 +++ 1 file changed, 3 insertions(+)drivers/perf/arm_spe_pmu.c +3
-0@@ -679,6 +679,9 @@ static int arm_spe_pmu_event_init(struct perf_event *event)if (attr->exclude_idle)return -EOPNOTSUPP ; +if (!attr->exclude_guest || attr->exclude_host) +return -EOPNOTSUPP ; +/* * Feedback-directed frequency throttling doesn't work when we * have a buffer of samples. We'd need to manually count the Signature · 1 line
2.21.0
Marc Zyngier
maintainerRe: [PATCH v2 15/18] perf: arm_spe: Handle guest/host exclusion flags
In reply to Andrew MurrayQuoted 19 lines · Andrew Murray
A side effect of supporting the SPE in guests is that we prevent the host from collecting data whilst inside a guest thus creating a black-out window. This occurs because instead of emulating the SPE, we share it with our guests.
Let's accurately describe our capabilities by using the perf exclude flags to prevent !exclude_guest and
exclude_hostflags from being used.Signed-off-by Andrew Murray <[email protected]>--- drivers/perf/arm_spe_pmu.c | 3 +++ 1 file changed, 3 insertions(+)drivers/perf/arm_spe_pmu.c +0
-0
drivers/perf/arm_spe_pmu.c static int arm_spe_pmu_event_init(struct perf_event *event) · line 684
Quoted 6 lines
drivers/perf/arm_spe_pmu.c +3
-0@@ -679,6 +679,9 @@ static int arm_spe_pmu_event_init(struct perf_event *event)if (attr->exclude_idle)return -EOPNOTSUPP ; +if (!attr->exclude_guest || attr->exclude_host) +return -EOPNOTSUPP ; +
I have the opposite approach. If the host decides to profile the guest, why should that be denied? If there is a black hole, it should take place in the guest. Today, the host does expect this to work, and there is no way that we unconditionally allow it to regress.
M.
Signature · 1 line
Jazz is not dead, it just smells funny.
Re: [PATCH v2 15/18] perf: arm_spe: Handle guest/host exclusion flags
Quoted 24 lines · Marc Zyngier
Quoted 22 lines · Andrew Murray
A side effect of supporting the SPE in guests is that we prevent the host from collecting data whilst inside a guest thus creating a black-out window. This occurs because instead of emulating the SPE, we share it with our guests.
Let's accurately describe our capabilities by using the perf exclude flags to prevent !exclude_guest and
exclude_hostflags from being used.Signed-off-by Andrew Murray <[email protected]>--- drivers/perf/arm_spe_pmu.c | 3 +++ 1 file changed, 3 insertions(+)drivers/perf/arm_spe_pmu.c +0
-0@@ -679,6 +679,9 @@ static int arm_spe_pmu_event_init(struct perf_event *event)if (attr->exclude_idle)return -EOPNOTSUPP ;
Quoted 8 lines
drivers/perf/arm_spe_pmu.c static int arm_spe_pmu_event_init(struct perf_event *event) · line 684
Quoted 3 lines
drivers/perf/arm_spe_pmu.c +3
-0+if (!attr->exclude_guest || attr->exclude_host) +return -EOPNOTSUPP ; +I have the opposite approach. If the host decides to profile the guest, why should that be denied? If there is a black hole, it should take place in the guest. Today, the host does expect this to work, and there is no way that we unconditionally allow it to regress.
That seems reasonable.
Upon entering the guest we'd have to detect if the host is using SPE, and if so choose not to restore the guest registers. Instead we'd have to trap them and let the guest read/write emulated values until the host has finished with SPE - at which time we could restore the guest SPE registers to hardware.
Does that approach make sense?
Thanks,
Andrew Murray
Quoted 5 lines
M.Signature · 1 line
Jazz is not dead, it just smells funny.
Marc Zyngier
maintainerRe: [PATCH v2 15/18] perf: arm_spe: Handle guest/host exclusion flags
Quoted 41 lines · Andrew Murray
Quoted 31 lines · Marc Zyngier
Quoted 2 lines · Andrew Murray
A side effect of supporting the SPE in guests is that we prevent
the
Quoted 1 line
host from collecting data whilst inside a guest thus creating a
black-out
Quoted 1 line
window. This occurs because instead of emulating the SPE, we share
it
Quoted 3 lines
with our guests.
Let's accurately describe our capabilities by using the perf
exclude
Quoted 1 line
flags to prevent !exclude_guest and
exclude_hostflags from beingused.
Review
Quoted 7 lines
Signed-off-by Andrew Murray <[email protected]>--- drivers/perf/arm_spe_pmu.c | 3 +++ 1 file changed, 3 insertions(+)(quoted diff) +0
-0
b/drivers/perf/arm_spe_pmu.cdrivers/perf/arm_spe_pmu.c static int arm_spe_pmu_event_init(struct
Quoted 4 lines
index
2d24af4cfcab..3703dbf459de100644drivers/perf/arm_spe_pmu.c +0
-0@@ -679,6 +679,9 @@ static int arm_spe_pmu_event_init(struct
perf_event*event)Quoted 3 lines
drivers/perf/arm_spe_pmu.c +0
-0if (attr->exclude_idle)return -EOPNOTSUPP ;Quoted 8 lines
drivers/perf/arm_spe_pmu.c static int arm_spe_pmu_event_init(struct perf_event *event) · line 684
Quoted 3 lines
drivers/perf/arm_spe_pmu.c +3
-0+if (!attr->exclude_guest || attr->exclude_host) +return -EOPNOTSUPP ; +I have the opposite approach. If the host decides to profile the guest, why should that be denied? If there is a black hole, it should take place in the guest. Today, the host does expect this to work, and there is no way that we unconditionally allow it to regress.
Quoted 8 lines
That seems reasonable.
Upon entering the guest we'd have to detect if the host is using SPE, and if so choose not to restore the guest registers. Instead we'd have to trap them and let the guest read/write emulated values until the host has finished with SPE - at which time we could restore the guest SPE registers to hardware.
Does that approach make sense?
Yes, this would be much better. All of this can be found out at vcpu_load() time, and once you've moved most of the SPE sysreg handling there, it will just follow the normal scheduling flow.
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
PATCH v2 16/18 KVM: arm64: enable SPE support
In reply to Andrew MurrayFrom: Sudeep Holla <[email protected]>
We have all the bits and pieces to enable SPE for guest in place, so lets enable it.
--- virt/kvm/arm/arm.c | 4 ++++ 1 file changed, 4 insertions(+)
virt/kvm/arm/arm.c +4 -0
@@ -611,6 +611,10 @@ static int kvm_vcpu_first_run_init(struct kvm_vcpu *vcpu)return ret; ret =kvm_arm_pmu_v3_enable (vcpu); +if (ret) +return ret; + + ret =kvm_arm_spe_v1_enable (vcpu);return ret; }
Signature · 1 line
2.21.0
PATCH v2 17/18 update_headers: Sync kvm UAPI headers with linux v5.5-rc2
In reply to Andrew MurrayThe local copies of the kvm user API headers are getting stale.
In preparation for some arch-specific updated, this patch reflects a re-run of util/update_headers.sh to pull in upstream updates from linux v5.5-rc2.
--- arm/aarch32/include/asm/kvm.h | 7 +++++-- arm/aarch64/include/asm/kvm.h | 13 +++++++++++-- include/linux/kvm.h | 18 ++++++++++++++++++ powerpc/include/asm/kvm.h | 3 +++ 4 files changed, 37 insertions(+), 4 deletions(-)
arm/aarch32/include/asm/kvm.h +5 -2
@@ -131,8 +131,9 @@ struct kvm_vcpu_events {struct {__u8 serror_pending;__u8 serror_has_esr; +__u8 ext_dabt_pending;/* Align it to 8 bytes */ -__u8 pad[6 ]; +__u8 pad[5 ];__u64 serror_esr; } exception;__u32 reserved[12 ];
@@ -266,8 +267,10 @@ struct kvm_vcpu_events {#define KVM_DEV_ARM_ITS_CTRL_RESET 4/* KVM_IRQ_LINE irq field index values */ +#define KVM_ARM_IRQ_VCPU2_SHIFT 28 +#define KVM_ARM_IRQ_VCPU2_MASK 0xf#define KVM_ARM_IRQ_TYPE_SHIFT 24 -#define KVM_ARM_IRQ_TYPE_MASK 0xff +#define KVM_ARM_IRQ_TYPE_MASK 0xf#define KVM_ARM_IRQ_VCPU_SHIFT 16#define KVM_ARM_IRQ_VCPU_MASK 0xff#define KVM_ARM_IRQ_NUM_SHIFT 0
arm/aarch64/include/asm/kvm.h +11 -2
@@ -106,6 +106,7 @@ struct kvm_regs {#define KVM_ARM_VCPU_SVE 4/* enable SVE for this CPU */ #define KVM_ARM_VCPU_PTRAUTH_ADDRESS 5/* VCPU uses address authentication */ #define KVM_ARM_VCPU_PTRAUTH_GENERIC 6/* VCPU uses generic authentication */ +#define KVM_ARM_VCPU_SPE_V1 7/* Support guest SPEv1 */ struct kvm_vcpu_init {__u32 target;
@@ -164,8 +165,9 @@ struct kvm_vcpu_events {struct {__u8 serror_pending;__u8 serror_has_esr; +__u8 ext_dabt_pending;/* Align it to 8 bytes */ -__u8 pad[6 ]; +__u8 pad[5 ];__u64 serror_esr; } exception;__u32 reserved[12 ];
@@ -323,10 +325,17 @@ struct kvm_vcpu_events {#define KVM_ARM_VCPU_TIMER_CTRL 1#define KVM_ARM_VCPU_TIMER_IRQ_VTIMER 0#define KVM_ARM_VCPU_TIMER_IRQ_PTIMER 1 +#define KVM_ARM_VCPU_PVTIME_CTRL 2 +#define KVM_ARM_VCPU_PVTIME_IPA 0 +#define KVM_ARM_VCPU_SPE_V1_CTRL 3 +#define KVM_ARM_VCPU_SPE_V1_IRQ 0 +#define KVM_ARM_VCPU_SPE_V1_INIT 1/* KVM_IRQ_LINE irq field index values */ +#define KVM_ARM_IRQ_VCPU2_SHIFT 28 +#define KVM_ARM_IRQ_VCPU2_MASK 0xf#define KVM_ARM_IRQ_TYPE_SHIFT 24 -#define KVM_ARM_IRQ_TYPE_MASK 0xff +#define KVM_ARM_IRQ_TYPE_MASK 0xf#define KVM_ARM_IRQ_VCPU_SHIFT 16#define KVM_ARM_IRQ_VCPU_MASK 0xff#define KVM_ARM_IRQ_NUM_SHIFT 0
include/linux/kvm.h +18 -0
@@ -235,6 +235,7 @@ struct kvm_hyperv_exit {#define KVM_EXIT_S390_STSI 25#define KVM_EXIT_IOAPIC_EOI 26#define KVM_EXIT_HYPERV 27 +#define KVM_EXIT_ARM_NISV 28/* For KVM_EXIT_INTERNAL_ERROR */ /* Emulate instruction failed. */
@@ -243,6 +244,8 @@ struct kvm_hyperv_exit {#define KVM_INTERNAL_ERROR_SIMUL_EX 2/* Encounter unexpected vm-exit due to delivery event. */ #define KVM_INTERNAL_ERROR_DELIVERY_EV 3 +/* Encounter unexpected vm-exit reason */ +#define KVM_INTERNAL_ERROR_UNEXPECTED_EXIT_REASON 4/* for KVM_RUN, returned by mmap(vcpu_fd, offset=0) */ struct kvm_run {
@@ -392,6 +395,11 @@ struct kvm_run { } eoi;/* KVM_EXIT_HYPERV */ struct kvm_hyperv_exit hyperv; +/* KVM_EXIT_ARM_NISV */ +struct { +__u64 esr_iss; +__u64 fault_ipa; + } arm_nisv;/* Fix the size of the union. */ char padding[256 ]; };
@@ -996,6 +1004,12 @@ struct kvm_ppc_resize_hpt {#define KVM_CAP_ARM_PTRAUTH_ADDRESS 171#define KVM_CAP_ARM_PTRAUTH_GENERIC 172#define KVM_CAP_PMU_EVENT_FILTER 173 +#define KVM_CAP_ARM_IRQ_LINE_LAYOUT_2 174 +#define KVM_CAP_HYPERV_DIRECT_TLBFLUSH 175 +#define KVM_CAP_PPC_GUEST_DEBUG_SSTEP 176 +#define KVM_CAP_ARM_NISV_TO_USER 177 +#define KVM_CAP_ARM_INJECT_EXT_DABT 178 +#define KVM_CAP_ARM_SPE_V1 179#ifdef KVM_CAP_IRQ_ROUTING
@@ -1142,6 +1156,7 @@ struct kvm_dirty_tlb {#define KVM_REG_S390 0x5000000000000000ULL#define KVM_REG_ARM64 0x6000000000000000ULL#define KVM_REG_MIPS 0x7000000000000000ULL +#define KVM_REG_RISCV 0x8000000000000000ULL#define KVM_REG_SIZE_SHIFT 52#define KVM_REG_SIZE_MASK 0x00f0000000000000ULL
@@ -1222,6 +1237,8 @@ enum kvm_device_type {#define KVM_DEV_TYPE_ARM_VGIC_ITS KVM_DEV_TYPE_ARM_VGIC_ITS KVM_DEV_TYPE_XIVE,#define KVM_DEV_TYPE_XIVE KVM_DEV_TYPE_XIVE + KVM_DEV_TYPE_ARM_PV_TIME, +#define KVM_DEV_TYPE_ARM_PV_TIME KVM_DEV_TYPE_ARM_PV_TIME KVM_DEV_TYPE_MAX, };
@@ -1332,6 +1349,7 @@ struct kvm_s390_ucas_mapping {#define KVM_PPC_GET_CPU_CHAR _IOR(KVMIO, 0xb1, struct kvm_ppc_cpu_char)/* Available with KVM_CAP_PMU_EVENT_FILTER */ #define KVM_SET_PMU_EVENT_FILTER _IOW(KVMIO, 0xb2, struct kvm_pmu_event_filter) +#define KVM_PPC_SVM_OFF _IO(KVMIO, 0xb3)/* ioctl for vm fd */ #define KVM_CREATE_DEVICE _IOWR(KVMIO, 0xe0, struct kvm_create_device)
powerpc/include/asm/kvm.h +3 -0
@@ -667,6 +667,8 @@ struct kvm_ppc_cpu_char {/* PPC64 eXternal Interrupt Controller Specification */ #define KVM_DEV_XICS_GRP_SOURCES 1/* 64-bit source attributes */ +#define KVM_DEV_XICS_GRP_CTRL 2 +#define KVM_DEV_XICS_NR_SERVERS 1/* Layout of 64-bit source attribute values */ #define KVM_XICS_DESTINATION_SHIFT 0
@@ -683,6 +685,7 @@ struct kvm_ppc_cpu_char {#define KVM_DEV_XIVE_GRP_CTRL 1#define KVM_DEV_XIVE_RESET 1#define KVM_DEV_XIVE_EQ_SYNC 2 +#define KVM_DEV_XIVE_NR_SERVERS 3#define KVM_DEV_XIVE_GRP_SOURCE 2/* 64-bit source identifier */ #define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3/* 64-bit source identifier */ #define KVM_DEV_XIVE_GRP_EQ_CONFIG 4/* 64-bit EQ identifier */
Signature · 6 lines
2.21.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v2 18/18 kvm: add a vcpu feature for SPEv1 support
In reply to Andrew MurrayThis is a runtime configurable for KVM tool to enable Statistical Profiling Extensions version 1 support in guest kernel. A command line option --spe is required to use the same.
--- Makefile | 2 +- arm/aarch64/arm-cpu.c | 2 + arm/aarch64/include/kvm/kvm-config-arch.h | 2 + arm/aarch64/include/kvm/kvm-cpu-arch.h | 3 +- arm/aarch64/kvm-cpu.c | 4 ++ arm/include/arm-common/kvm-config-arch.h | 1 + arm/include/arm-common/spe.h | 4 ++ arm/spe.c | 81 +++++++++++++++++++++++ 8 files changed, 97 insertions(+), 2 deletions(-) create mode 100644 arm/include/arm-common/spe.h create mode 100644 arm/spe.c
Makefile +1 -1
@@ -158,7 +158,7 @@ endif # ARM OBJS_ARM_COMMON := arm/fdt.o arm/gic.o arm/gicv2m.o arm/ioport.o \ arm/kvm.o arm/kvm-cpu.o arm/pci.o arm/timer.o \ - arm/pmu.o + arm/pmu.o arm/spe.o HDRS_ARM_COMMON := arm/include ifeq ($(ARCH), arm) DEFINES += -DCONFIG_ARM
arm/aarch64/arm-cpu.c +2 -0
@@ -6,6 +6,7 @@#include "arm-common/gic.h" #include "arm-common/timer.h" #include "arm-common/pmu.h" +#include "arm-common/spe.h" #include <linux/byteorder.h> #include <linux/types.h>
@@ -17,6 +18,7 @@ static void generate_fdt_nodes(void *fdt, struct kvm *kvm)gic__generate_fdt_nodes (fdt, kvm->cfg.arch.irqchip);timer__generate_fdt_nodes (fdt, kvm, timer_interrupts);pmu__generate_fdt_nodes (fdt, kvm); +spe__generate_fdt_nodes (fdt, kvm); }static int arm_cpu__vcpu_init (struct kvm_cpu *vcpu)
arm/aarch64/include/kvm/kvm-config-arch.h +2 -0
@@ -6,6 +6,8 @@ "Run AArch32 guest"), \OPT_BOOLEAN ('\0' ,"pmu" , &(cfg)->has_pmuv3, \"Create PMUv3 device" ), \ +OPT_BOOLEAN ('\0' ,"spe" , &(cfg)->has_spev1, \ +"Create SPEv1 device" ), \OPT_U64 ('\0' ,"kaslr-seed" , &(cfg)->kaslr_seed, \"Specify random seed for Kernel Address Space " \ "Layout Randomization (KASLR)"),
arm/aarch64/include/kvm/kvm-cpu-arch.h +2 -1
@@ -8,7 +8,8 @@#define ARM_VCPU_FEATURE_FLAGS (kvm, cpuid) { \ [0] = ((!!(cpuid) << KVM_ARM_VCPU_POWER_OFF) | \ (!!(kvm)->cfg.arch.aarch32_guest << KVM_ARM_VCPU_EL1_32BIT) | \ - (!!(kvm)->cfg.arch.has_pmuv3 << KVM_ARM_VCPU_PMU_V3)) \ + (!!(kvm)->cfg.arch.has_pmuv3 << KVM_ARM_VCPU_PMU_V3) | \ + (!!(kvm)->cfg.arch.has_spev1 << KVM_ARM_VCPU_SPE_V1)) \ }#define ARM_MPIDR_HWID_BITMASK 0xFF00FFFFFFUL
arm/aarch64/kvm-cpu.c +4 -0
@@ -140,6 +140,10 @@ void kvm_cpu__select_features(struct kvm *kvm, struct kvm_vcpu_init *init)/* Enable SVE if available */ if (kvm__supports_extension (kvm,KVM_CAP_ARM_SVE )) init->features[0 ] |=1UL <<KVM_ARM_VCPU_SVE ; + +/* Enable SPE if available */ +if (kvm__supports_extension (kvm,KVM_CAP_ARM_SPE_V1 )) + init->features[0 ] |=1UL <<KVM_ARM_VCPU_SPE_V1 ; }int kvm_cpu__configure_features (struct kvm_cpu *vcpu)
arm/include/arm-common/kvm-config-arch.h +1 -0
@@ -9,6 +9,7 @@ struct kvm_config_arch {bool virtio_trans_pci;bool aarch32_guest;bool has_pmuv3; +bool has_spev1;u64 kaslr_seed;enum irqchip_type irqchip;u64 fw_addr;
arm/include/arm-common/spe.h new file +4 -0
@@ -0,0 +1,4 @@ + +#define KVM_ARM_SPEV1_PPI 21 + +void spe__generate_fdt_nodes (void *fdt,struct kvm *kvm);
arm/spe.c new file +81 -0
81 lines not on this page · raw message
Signature · 6 lines
2.21.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Mark Rutland
maintainerHi Andrew,
Quoted 9 lines · Andrew Murray
This series implements support for allowing KVM guests to use the Arm Statistical Profiling Extension (SPE).
It has been tested on a model to ensure that both host and guest can simultaneously use SPE with valid data. E.g.
$ perf record -e arm_spe/ts_enable=1,pa_enable=1,pct_enable=1/ \ dd if=/dev/zero of=/dev/null count=1000 $ perf report --dump-raw-trace > spe_buf.txt
What happens if I run perf record on the VMM, or on the CPU(s) that the VMM is running on? i.e.
$ perf record -e arm_spe/ts_enable=1,pa_enable=1,pct_enable=1/ \
lkvm ${OPTIONS_FOR_GUEST_USING_SPE}... or:
$ perf record -a -c 0 -e arm_spe/ts_enable=1,pa_enable=1,pct_enable=1/ \
sleep 1000 &
$ taskset -c 0 lkvm ${OPTIONS_FOR_GUEST_USING_SPE} &Quoted 9 lines
As we save and restore the SPE context, the guest can access the SPE registers directly, thus in this version of the series we remove the trapping and emulation.
In the previous series of this support, when KVM SPE isn't supported (e.g. via
CONFIG_KVM_ARM_SPE) we were able to return a value of 0 to all reads of the SPE registers - as we can no longer do this there isn't a mechanism to prevent the guest from using SPE - thus I'm keen for feedback on the best way of resolving this.
When not providing SPE to the guest, surely we should be trapping the registers and injecting an UNDEF?
What happens today, without these patches?
Quoted 3 lines
It appears necessary to pin the entire guest memory in order to provide guest SPE access - otherwise it is possible for the guest to receive Stage-2 faults.
AFAICT these patches do not implement this. I assume that's what you're trying to point out here, but I just want to make sure that's explicit.
Maybe this is a reason to trap+emulate if there's something more sensible that hyp can do if it sees a Stage-2 fault.
Thanks,
Mark.
Quoted 13 lines · Mark Rutland
Hi Andrew,
Quoted 9 lines · Andrew Murray
This series implements support for allowing KVM guests to use the Arm Statistical Profiling Extension (SPE).
It has been tested on a model to ensure that both host and guest can simultaneously use SPE with valid data. E.g.
$ perf record -e arm_spe/ts_enable=1,pa_enable=1,pct_enable=1/ \ dd if=/dev/zero of=/dev/null count=1000 $ perf report --dump-raw-trace > spe_buf.txt
Quoted 6 lines
What happens if I run perf record on the VMM, or on the CPU(s) that the VMM is running on? i.e.
$ perf record -e arm_spe/ts_enable=1,pa_enable=1,pct_enable=1/ \ lkvm ${OPTIONS_FOR_GUEST_USING_SPE}
By default perf excludes the guest, so this works as expected, just recording activity of the process when it is outside the guest. (perf report appears to give valid output).
Patch 15 currently prevents using perf to record inside the guest.
Quoted 10 lines
... or:
$ perf record -a -c 0 -e arm_spe/ts_enable=1,pa_enable=1,pct_enable=1/ \ sleep 1000 & $ taskset -c 0 lkvm ${OPTIONS_FOR_GUEST_USING_SPE} &Quoted 4 lines
As we save and restore the SPE context, the guest can access the SPE registers directly, thus in this version of the series we remove the trapping and emulation.
Quoted 8 lines
Quoted 5 lines
In the previous series of this support, when KVM SPE isn't supported (e.g. via
CONFIG_KVM_ARM_SPE) we were able to return a value of 0 to all reads of the SPE registers - as we can no longer do this there isn't a mechanism to prevent the guest from using SPE - thus I'm keen for feedback on the best way of resolving this.When not providing SPE to the guest, surely we should be trapping the registers and injecting an UNDEF?
Yes we should, I'll update the series.
Quoted 3 lines
What happens today, without these patches?
Prior to this series MDCR_EL2_TPMS is set and E2PB is unset resulting in all SPE registers being trapped (with NULL handlers).
Quoted 6 lines
Quoted 3 lines
It appears necessary to pin the entire guest memory in order to provide guest SPE access - otherwise it is possible for the guest to receive Stage-2 faults.
AFAICT these patches do not implement this. I assume that's what you're trying to point out here, but I just want to make sure that's explicit.
That's right.
Quoted 3 lines
Maybe this is a reason to trap+emulate if there's something more sensible that hyp can do if it sees a Stage-2 fault.
Yes it's not really clear to me at the moment what to do about this.
Thanks,
Andrew Murray
Quoted 3 lines
Thanks,
Mark.
Marc Zyngier
maintainer[fixing email addresses]
Hi Andrew,
Quoted 2 lines · Andrew Murray
This series implements support for allowing KVM guests to use the Arm Statistical Profiling Extension (SPE).
Thanks for this. In future, please Cc me and Will on email addresses we can actually read.
Quoted 11 lines
It has been tested on a model to ensure that both host and guest can simultaneously use SPE with valid data. E.g.
$ perf record -e arm_spe/ts_enable=1,pa_enable=1,pct_enable=1/ \ dd if=/dev/zero of=/dev/null count=1000 $ perf report --dump-raw-trace > spe_buf.txtAs we save and restore the SPE context, the guest can access the SPE registers directly, thus in this version of the series we remove the trapping and emulation.
Quoted 5 lines
In the previous series of this support, when KVM SPE isn't supported (e.g. via
CONFIG_KVM_ARM_SPE) we were able to return a value of 0 to all reads of the SPE registers - as we can no longer do this there isn't a mechanism to prevent the guest from using SPE - thus I'm keen for feedback on the best way of resolving this.
Surely there is a way to conditionally trap SPE registers, right? You should still be able to do this if SPE is not configured for a given guest (as we do for other feature such as PtrAuth).
Quoted 3 lines
It appears necessary to pin the entire guest memory in order to provide guest SPE access - otherwise it is possible for the guest to receive Stage-2 faults.
Really? How can the guest receive a stage-2 fault? This doesn't fit what I understand of the ARMv8 exception model. Or do you mean a SPE interrupt describing a S2 fault?
And this is not just pinning the memory either. You have to ensure that all S2 page tables are created ahead of SPE being able to DMA to guest memory. This may have some impacts on the THP code...
I'll have a look at the actual series ASAP (but that's not very soon).
Thanks,
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
Marc Zyngier
maintainerQuoted 42 lines · Marc Zyngier
[fixing email addresses]
Hi Andrew,
Quoted 2 lines · Andrew Murray
This series implements support for allowing KVM guests to use the Arm Statistical Profiling Extension (SPE).
Thanks for this. In future, please Cc me and Will on email addresses we can actually read.
Quoted 11 lines
It has been tested on a model to ensure that both host and guest can simultaneously use SPE with valid data. E.g.
$ perf record -e arm_spe/ts_enable=1,pa_enable=1,pct_enable=1/ \ dd if=/dev/zero of=/dev/null count=1000 $ perf report --dump-raw-trace > spe_buf.txtAs we save and restore the SPE context, the guest can access the SPE registers directly, thus in this version of the series we remove the trapping and emulation.
Quoted 6 lines
In the previous series of this support, when KVM SPE isn't supported (e.g. via
CONFIG_KVM_ARM_SPE) we were able to return a value of 0 to all reads of the SPE registers - as we can no longer do this there isn't a mechanism to prevent the guest from using SPE - thus I'm keen for feedback on the best way of resolving this.Surely there is a way to conditionally trap SPE registers, right? You should still be able to do this if SPE is not configured for a given guest (as we do for other feature such as PtrAuth).
Quoted 3 lines
It appears necessary to pin the entire guest memory in order to provide guest SPE access - otherwise it is possible for the guest to receive Stage-2 faults.
Really? How can the guest receive a stage-2 fault? This doesn't fit what I understand of the ARMv8 exception model. Or do you mean a SPE interrupt describing a S2 fault?
Quoted 5 lines
And this is not just pinning the memory either. You have to ensure that all S2 page tables are created ahead of SPE being able to DMA to guest memory. This may have some impacts on the THP code...
I'll have a look at the actual series ASAP (but that's not very soon).
I found some time to go through the series, and there is clearly a lot of work left to do:
- There so nothing here to handle memory pinning whatsoever. If it works, it is only thanks to some side effect.
- The missing trapping is deeply worrying. Given that this is an optional feature, you cannot just let the guest do whatever it wants in an uncontrolled manner.
- The interrupt handling is busted. You mix concepts picked from both the PMU and the timer code, while the SPE device doesn't behave like any of these two (it is neither a fully emulated device, nor a device that is exclusively owned by a guest at any given time).
I expect some level of discussion on the list including at least Will and myself before you respin this.
M.
Signature · 1 line
Jazz is not dead, it just smells funny.
Quoted 36 lines · Marc Zyngier
Quoted 34 lines · Marc Zyngier
[fixing email addresses]
Hi Andrew,
Quoted 2 lines · Andrew Murray
This series implements support for allowing KVM guests to use the Arm Statistical Profiling Extension (SPE).
Thanks for this. In future, please Cc me and Will on email addresses we can actually read.
Quoted 11 lines
It has been tested on a model to ensure that both host and guest can simultaneously use SPE with valid data. E.g.
$ perf record -e arm_spe/ts_enable=1,pa_enable=1,pct_enable=1/ \ dd if=/dev/zero of=/dev/null count=1000 $ perf report --dump-raw-trace > spe_buf.txtAs we save and restore the SPE context, the guest can access the SPE registers directly, thus in this version of the series we remove the trapping and emulation.
Quoted 6 lines
In the previous series of this support, when KVM SPE isn't supported (e.g. via
CONFIG_KVM_ARM_SPE) we were able to return a value of 0 to all reads of the SPE registers - as we can no longer do this there isn't a mechanism to prevent the guest from using SPE - thus I'm keen for feedback on the best way of resolving this.Surely there is a way to conditionally trap SPE registers, right? You should still be able to do this if SPE is not configured for a given guest (as we do for other feature such as PtrAuth).
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
It appears necessary to pin the entire guest memory in order to provide guest SPE access - otherwise it is possible for the guest to receive Stage-2 faults.
Really? How can the guest receive a stage-2 fault? This doesn't fit what I understand of the ARMv8 exception model. Or do you mean a SPE interrupt describing a S2 fault?
Yes the latter.
Quoted 10 lines
Quoted 6 lines
And this is not just pinning the memory either. You have to ensure that all S2 page tables are created ahead of SPE being able to DMA to guest memory. This may have some impacts on the THP code...
I'll have a look at the actual series ASAP (but that's not very soon).
I found some time to go through the series, and there is clearly a lot of work left to do:
Quoted 6 lines
- There so nothing here to handle memory pinning whatsoever. If it works, it is only thanks to some side effect.
- The missing trapping is deeply worrying. Given that this is an optional feature, you cannot just let the guest do whatever it wants in an uncontrolled manner.
Yes I'll add this.
Quoted 8 lines
- The interrupt handling is busted. You mix concepts picked from both the PMU and the timer code, while the SPE device doesn't behave like any of these two (it is neither a fully emulated device, nor a device that is exclusively owned by a guest at any given time).
I expect some level of discussion on the list including at least Will and myself before you respin this.
Thanks for the quick feedback.
Andrew Murray
Quoted 5 lines
M.Signature · 1 line
Jazz is not dead, it just smells funny.