Skip to contentSkip to conversation

coresight: Support exclude_guest with Feat_TRF and nVHE

RFC PATCH In review James Clark 3 participants 12 messages Aug 4, 2023 – Aug 9, 2023

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover coresight: Support exclude_guest with Feat_TRF and nVHE

Hi,

I'm looking for help in testing this and for feedback on whether it's useful to anyone. Testing it requires hardware that has Feat_TRF (v8.4) but no TRBE. This is because TRBE usage is disabled in nVHE guests.

I don't currently have any access to any hardware, and the FVP model can only do self hosted trace using TRBE.

Currently with nVHE you would always get trace from guests, and filtering out isn't possible without this patchset. In comparison, with VHE guests, they never generate guest trace without [1]. I think the existence of trace rather than lack of could suggest that this change is less useful than [1]. Also the restricted set of hardware that it works on supports that too.

Apart from compilation and checking that the exclude guest settings are correctly programmed on guest switch, this is untested by me.

Applies to kvmarm/next (3b4e3afb2032)

[1]: https://lore.kernel.org/linux-arm-kernel/[email protected]/

James Clark (3):
arm64: KVM: Add support for exclude_guest and exclude_host for ETM
arm64: KVM: Support exclude_guest for Coresight trace in nVHE
coresight: Support exclude_guest with Feat_TRF and nVHE

 arch/arm64/include/asm/kvm_host.h             | 10 +++-
 arch/arm64/kvm/Makefile                       |  1 +
 arch/arm64/kvm/arm.c                          |  1 +
 arch/arm64/kvm/debug.c                        |  7 +++
 arch/arm64/kvm/etm.c                          | 48 ++++++++++++++++
 arch/arm64/kvm/hyp/nvhe/debug-sr.c            | 56 +++++++++++++++++--
 .../hwtracing/coresight/coresight-etm-perf.c  |  4 ++
 include/kvm/etm.h                             | 43 ++++++++++++++
 8 files changed, 165 insertions(+), 5 deletions(-)
 create mode 100644 arch/arm64/kvm/etm.c
 create mode 100644 include/kvm/etm.h
Signature · 1 line
2.34.1
#1 lore raw

RFC PATCH 1/3 arm64: KVM: Add support for exclude_guest and exclude_host for ETM

Add an interface for the Coresight driver to use to set the current exclude settings for the current CPU. This will be used to configure TRFCR_EL1.

The settings must be copied to the vCPU before each run in the same way that PMU events are because the per-cpu struct isn't accessible in protected mode.

This is only needed for nVHE, otherwise it works automatically with TRFCR_EL{1,2}. Unfortunately it can't be gated on CONFIG_CORESIGHT because Coresight can be built as a module. It can however be gated on CONFIG_PERF_EVENTS because that is required by Coresight.

---
 arch/arm64/include/asm/kvm_host.h | 10 ++++++-
 arch/arm64/kvm/Makefile           |  1 +
 arch/arm64/kvm/arm.c              |  1 +
 arch/arm64/kvm/etm.c              | 48 +++++++++++++++++++++++++++++++
 include/kvm/etm.h                 | 43 +++++++++++++++++++++++++++
 5 files changed, 102 insertions(+), 1 deletion(-)
 create mode 100644 arch/arm64/kvm/etm.c
 create mode 100644 include/kvm/etm.h
arch/arm64/include/asm/kvm_host.h +9 -1
index d7b1403a3fb2..f33262217c84 100644
@@ -35,6 +35,7 @@
 #include <kvm/arm_vgic.h>
 #include <kvm/arm_arch_timer.h>
 #include <kvm/arm_pmu.h>
+#include <kvm/etm.h>
 
 #define KVM_MAX_VCPUS VGIC_V3_MAX_CPUS
 
@@ -500,7 +501,7 @@ struct kvm_vcpu_arch {
 	u8 cflags;
 
 	/* Input flags to the hypervisor code, potentially cleared after use */
-	u8 iflags;
+	u16 iflags;
 
 	/* State flags for kernel bookkeeping, unused by the hypervisor code */
 	u8 sflags;
@@ -541,6 +542,9 @@ struct kvm_vcpu_arch {
 		u64 pmscr_el1;
 		/* Self-hosted trace */
 		u64 trfcr_el1;
+		/* exclude_guest settings for nVHE */
+		struct kvm_etm_event etm_event;
+
 	} host_debug_state;
 
 	/* VGIC state */
@@ -713,6 +717,8 @@ struct kvm_vcpu_arch {
 #define DEBUG_STATE_SAVE_TRBE	__vcpu_single_flag(iflags, BIT(6))
 /* vcpu running in HYP context */
 #define VCPU_HYP_CONTEXT	__vcpu_single_flag(iflags, BIT(7))
+/* Save TRFCR and apply exclude_guest rules */
+#define DEBUG_STATE_SAVE_TRFCR	__vcpu_single_flag(iflags, BIT(8))
 
 /* SVE enabled for host EL0 */
 #define HOST_SVE_ENABLED	__vcpu_single_flag(sflags, BIT(0))
@@ -1096,6 +1102,8 @@ void kvm_arch_vcpu_put_debug_state_flags(struct kvm_vcpu *vcpu);
 void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr);
 void kvm_clr_pmu_events(u32 clr);
 bool kvm_set_pmuserenr(u64 val);
+void kvm_set_etm_events(struct perf_event_attr *attr);
+void kvm_clr_etm_events(void);
 #else
 static inline void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr) {}
 static inline void kvm_clr_pmu_events(u32 clr) {}
arch/arm64/kvm/Makefile +1 -0
index c0c050e53157..0faff57423c4 100644
@@ -23,6 +23,7 @@ kvm-y += arm.o mmu.o mmio.o psci.o hypercalls.o pvtime.o \
 	 vgic/vgic-its.o vgic/vgic-debug.o
 
 kvm-$(CONFIG_HW_PERF_EVENTS)  += pmu-emul.o pmu.o
+kvm-$(CONFIG_PERF_EVENTS) += etm.o
 
 always-y := hyp_constants.h hyp-constants.s
 
arch/arm64/kvm/arm.c +1 -0
index b1a9d47fb2f3..7bd5975328a3 100644
@@ -952,6 +952,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
 		kvm_vgic_flush_hwstate(vcpu);
 
 		kvm_pmu_update_vcpu_events(vcpu);
+		kvm_etm_update_vcpu_events(vcpu);
 
 		/*
 		 * Ensure we set mode to IN_GUEST_MODE after we disable
arch/arm64/kvm/etm.c new file +48 -0

48 lines not on this page · raw message

include/kvm/etm.h new file +43 -0

43 lines not on this page · raw message

Signature · 1 line
2.34.1
maintainer
#2 lore raw

Re: [RFC PATCH 1/3] arm64: KVM: Add support for exclude_guest and exclude_host for ETM

Quoted 4 lines · James Clark

Add an interface for the Coresight driver to use to set the current exclude settings for the current CPU. This will be used to configure TRFCR_EL1.

Can you start by stating the problem? There is *some* rationale in the cover letter, but not enough to get the full picture. Specially if you haven't looked at the trace subsystem in the past... 7 years or so.

Quoted 4 lines

The settings must be copied to the vCPU before each run in the same way that PMU events are because the per-cpu struct isn't accessible in protected mode.

I'm pretty sure that for protected guests, we'd like to disable tracing altogether (debug mode excepted).

Quoted 2 lines

This is only needed for nVHE, otherwise it works automatically with

How about hVHE, which uses VHE at EL2 only? Doesn't it require the same treatment?

Quoted 3 lines

TRFCR_EL{1,2}. Unfortunately it can't be gated on CONFIG_CORESIGHT because Coresight can be built as a module. It can however be gated on CONFIG_PERF_EVENTS because that is required by Coresight.

Why does it need to be gated *at all*? We need this for the PMU because of the way we call into the perf subsystem, but I don't see anything like that here. In general, conditional compilation sucks, and I'd like to avoid it as much as possible.

Quoted 25 lines
---
 arch/arm64/include/asm/kvm_host.h | 10 ++++++-
 arch/arm64/kvm/Makefile           |  1 +
 arch/arm64/kvm/arm.c              |  1 +
 arch/arm64/kvm/etm.c              | 48 +++++++++++++++++++++++++++++++
 include/kvm/etm.h                 | 43 +++++++++++++++++++++++++++
 5 files changed, 102 insertions(+), 1 deletion(-)
 create mode 100644 arch/arm64/kvm/etm.c
 create mode 100644 include/kvm/etm.h
arch/arm64/include/asm/kvm_host.h +1 -0
index d7b1403a3fb2..f33262217c84 100644
@@ -35,6 +35,7 @@
 #include <kvm/arm_vgic.h>
 #include <kvm/arm_arch_timer.h>
 #include <kvm/arm_pmu.h>
+#include <kvm/etm.h>
 
 #define KVM_MAX_VCPUS VGIC_V3_MAX_CPUS
 

arch/arm64/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 504

Quoted 5 lines
arch/arm64/include/asm/kvm_host.h +1 -1
@@ -500,7 +501,7 @@ struct kvm_vcpu_arch {
 	u8 cflags;
 
 	/* Input flags to the hypervisor code, potentially cleared after use */
-	u8 iflags;
+	u16 iflags;

If you make the iflags bigger, what ripple effect does it have on the alignment of the other data structures? Consider reordering things if it helps filling holes.

arch/arm64/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 547

Quoted 10 lines
arch/arm64/include/asm/kvm_host.h +3 -0
 	/* State flags for kernel bookkeeping, unused by the hypervisor code */
 	u8 sflags;
@@ -541,6 +542,9 @@ struct kvm_vcpu_arch {
 		u64 pmscr_el1;
 		/* Self-hosted trace */
 		u64 trfcr_el1;
+		/* exclude_guest settings for nVHE */
+		struct kvm_etm_event etm_event;
+

Spurious blank line. More importantly, how is that related to the trfcr_el1 field just above?

Quoted 89 lines
arch/arm64/include/asm/kvm_host.h +4 -0
 	} host_debug_state;
 
 	/* VGIC state */
@@ -713,6 +717,8 @@ struct kvm_vcpu_arch {
 #define DEBUG_STATE_SAVE_TRBE	__vcpu_single_flag(iflags, BIT(6))
 /* vcpu running in HYP context */
 #define VCPU_HYP_CONTEXT	__vcpu_single_flag(iflags, BIT(7))
+/* Save TRFCR and apply exclude_guest rules */
+#define DEBUG_STATE_SAVE_TRFCR	__vcpu_single_flag(iflags, BIT(8))
 
 /* SVE enabled for host EL0 */
 #define HOST_SVE_ENABLED	__vcpu_single_flag(sflags, BIT(0))
@@ -1096,6 +1102,8 @@ void kvm_arch_vcpu_put_debug_state_flags(struct kvm_vcpu *vcpu);
 void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr);
 void kvm_clr_pmu_events(u32 clr);
 bool kvm_set_pmuserenr(u64 val);
+void kvm_set_etm_events(struct perf_event_attr *attr);
+void kvm_clr_etm_events(void);
 #else
 static inline void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr) {}
 static inline void kvm_clr_pmu_events(u32 clr) {}
arch/arm64/kvm/Makefile +1 -0
index c0c050e53157..0faff57423c4 100644
@@ -23,6 +23,7 @@ kvm-y += arm.o mmu.o mmio.o psci.o hypercalls.o pvtime.o \
 	 vgic/vgic-its.o vgic/vgic-debug.o
 
 kvm-$(CONFIG_HW_PERF_EVENTS)  += pmu-emul.o pmu.o
+kvm-$(CONFIG_PERF_EVENTS) += etm.o
 
 always-y := hyp_constants.h hyp-constants.s
 
arch/arm64/kvm/arm.c +1 -0
index b1a9d47fb2f3..7bd5975328a3 100644
@@ -952,6 +952,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
 		kvm_vgic_flush_hwstate(vcpu);
 
 		kvm_pmu_update_vcpu_events(vcpu);
+		kvm_etm_update_vcpu_events(vcpu);
 
 		/*
 		 * Ensure we set mode to IN_GUEST_MODE after we disable
arch/arm64/kvm/etm.c new file +38 -0
new file mode 100644
index 000000000000..359c37745de2
@@ -0,0 +1,48 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+#include <linux/kvm_host.h>
+
+#include <kvm/etm.h>
+
+static DEFINE_PER_CPU(struct kvm_etm_event, kvm_etm_events);
+
+struct kvm_etm_event *kvm_get_etm_event(void)
+{
+	return this_cpu_ptr(&kvm_etm_events);
+}
+
+void kvm_etm_set_events(struct perf_event_attr *attr)
+{
+	struct kvm_etm_event *etm_event;
+
+	/*
+	 * Exclude guest option only requires extra work with nVHE.
+	 * Otherwise it works automatically with TRFCR_EL{1,2}
+	 */
+	if (has_vhe())
+		return;
+
+	etm_event = kvm_get_etm_event();
+
+	etm_event->exclude_guest = attr->exclude_guest;
+	etm_event->exclude_host = attr->exclude_host;
+	etm_event->exclude_kernel = attr->exclude_kernel;
+	etm_event->exclude_user = attr->exclude_user;
+}
+EXPORT_SYMBOL_GPL(kvm_etm_set_events);
+
+void kvm_etm_clr_events(void)
+{
+	struct kvm_etm_event *etm_event;
+
+	if (has_vhe())

arch/arm64/kvm/etm.c line 48

Quoted 10 lines
arch/arm64/kvm/etm.c new file +10 -0
+		return;
+
+	etm_event = kvm_get_etm_event();
+
+	etm_event->exclude_guest = false;
+	etm_event->exclude_host = false;
+	etm_event->exclude_kernel = false;
+	etm_event->exclude_user = false;
+}
+EXPORT_SYMBOL_GPL(kvm_etm_clr_events);

Does it really need its own compilation unit if we were to build it at all times?

Quoted 28 lines
include/kvm/etm.h new file +22 -0
new file mode 100644
index 000000000000..95c4809fa2b0
@@ -0,0 +1,43 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+
+#ifndef __KVM_DEBUG_H
+#define __KVM_DEBUG_H
+
+struct perf_event_attr;
+struct kvm_vcpu;
+
+#if IS_ENABLED(CONFIG_KVM) && IS_ENABLED(CONFIG_PERF_EVENTS)
+
+struct kvm_etm_event {
+	bool exclude_host;
+	bool exclude_guest;
+	bool exclude_kernel;
+	bool exclude_user;
+};
+
+struct kvm_etm_event *kvm_get_etm_event(void);
+void kvm_etm_clr_events(void);
+void kvm_etm_set_events(struct perf_event_attr *attr);
+
+/*

include/kvm/etm.h line 32

Quoted 10 lines
include/kvm/etm.h new file +10 -0
+ * Updates the vcpu's view of the etm events for this cpu. Must be
+ * called before every vcpu run after disabling interrupts, to ensure
+ * that an interrupt cannot fire and update the structure.
+ */
+#define kvm_etm_update_vcpu_events(vcpu)						\
+	do {										\
+		if (!has_vhe() && vcpu_get_flag(vcpu, DEBUG_STATE_SAVE_TRFCR))		\
+			vcpu->arch.host_debug_state.etm_event = *kvm_get_etm_event();	\
+	} while (0)
+

Why is it a macro and not a function, which would avoid exposing kvm_get_etm_event?

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#3 lore raw

Re: [RFC PATCH 1/3] arm64: KVM: Add support for exclude_guest and exclude_host for ETM

Quoted 11 lines · Marc Zyngier
Quoted 4 lines · James Clark

Add an interface for the Coresight driver to use to set the current exclude settings for the current CPU. This will be used to configure TRFCR_EL1.

Can you start by stating the problem? There is *some* rationale in the cover letter, but not enough to get the full picture. Specially if you haven't looked at the trace subsystem in the past... 7 years or so.

Yeah I will expand on it a bit.

Quoted 8 lines
Quoted 4 lines

The settings must be copied to the vCPU before each run in the same way that PMU events are because the per-cpu struct isn't accessible in protected mode.

I'm pretty sure that for protected guests, we'd like to disable tracing altogether (debug mode excepted).

Do you mean disable the ability to trace guests from the host? Or to disable generation of any trace altogether when in the guest, even if the guest started the session?

Currently the ETM driver isn't exposed to guests, so it's not possible for the guest to start a session. That leaves only the host being able to trace the guest. If you think that shouldn't happen outside of debug mode then maybe we need another change to prevent that.

Another issue is that this only works with Feat_TRF, without that we'd have to do something else again to prevent protected guests from being traced, as currently there is nothing to stop it.

I'm not sure about debug mode though? Is that a kernel config or some kvm option? I suppose this could be disabled for users by not loading the Coresight driver, but there's currently nothing a guest can do to stop its self from being traced.

Quoted 6 lines
Quoted 2 lines

This is only needed for nVHE, otherwise it works automatically with

How about hVHE, which uses VHE at EL2 only? Doesn't it require the same treatment?

Yes it sounds like it would. Does it use the same code as nVHE in arch/arm64/kvm/hyp/nvhe? I think it might already work in that case, but I need to read a bit about hVHE.

Quoted 9 lines
Quoted 3 lines

TRFCR_EL{1,2}. Unfortunately it can't be gated on CONFIG_CORESIGHT because Coresight can be built as a module. It can however be gated on CONFIG_PERF_EVENTS because that is required by Coresight.

Why does it need to be gated *at all*? We need this for the PMU because of the way we call into the perf subsystem, but I don't see anything like that here. In general, conditional compilation sucks, and I'd like to avoid it as much as possible.

No real reason really, I was probably trying to minimise the impact because it added a few bytes here and there, but it's tiny. If you're ok with it I can quite easily remove the conditional compilation and it should simplify it a bit.

Quoted 27 lines
Quoted 27 lines
---
 arch/arm64/include/asm/kvm_host.h | 10 ++++++-
 arch/arm64/kvm/Makefile           |  1 +
 arch/arm64/kvm/arm.c              |  1 +
 arch/arm64/kvm/etm.c              | 48 +++++++++++++++++++++++++++++++
 include/kvm/etm.h                 | 43 +++++++++++++++++++++++++++
 5 files changed, 102 insertions(+), 1 deletion(-)
 create mode 100644 arch/arm64/kvm/etm.c
 create mode 100644 include/kvm/etm.h
arch/arm64/include/asm/kvm_host.h +1 -0
index d7b1403a3fb2..f33262217c84 100644
@@ -35,6 +35,7 @@
 #include <kvm/arm_vgic.h>
 #include <kvm/arm_arch_timer.h>
 #include <kvm/arm_pmu.h>
+#include <kvm/etm.h>
 
 #define KVM_MAX_VCPUS VGIC_V3_MAX_CPUS
 
@@ -500,7 +501,7 @@ struct kvm_vcpu_arch {
 	u8 cflags;
 
Quoted 8 lines

arch/arm64/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 504

Quoted 3 lines
arch/arm64/include/asm/kvm_host.h +1 -1
 	/* Input flags to the hypervisor code, potentially cleared after use */
-	u8 iflags;
+	u16 iflags;

If you make the iflags bigger, what ripple effect does it have on the alignment of the other data structures? Consider reordering things if it helps filling holes.

I think you might be right. Possibly flipping iflags and sflags will be enough but will have to check the exact alignment up to that point.

Quoted 14 lines

arch/arm64/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 547

Quoted 10 lines
arch/arm64/include/asm/kvm_host.h +3 -0
 	/* State flags for kernel bookkeeping, unused by the hypervisor code */
 	u8 sflags;
@@ -541,6 +542,9 @@ struct kvm_vcpu_arch {
 		u64 pmscr_el1;
 		/* Self-hosted trace */
 		u64 trfcr_el1;
+		/* exclude_guest settings for nVHE */
+		struct kvm_etm_event etm_event;
+

Spurious blank line. More importantly, how is that related to the trfcr_el1 field just above?

It's not really related, it's just the same place that the code that runs at a similar time and does a similar thing has its data stored.

Something that you mentioned in another comment might be relevant here. If I do it only in terms of registers that are saved and restored then I don't need to save the etm event settings and the low level code doesn't need to know anything about it.

Quoted 94 lines
Quoted 94 lines
arch/arm64/include/asm/kvm_host.h +4 -0
 	} host_debug_state;
 
 	/* VGIC state */
@@ -713,6 +717,8 @@ struct kvm_vcpu_arch {
 #define DEBUG_STATE_SAVE_TRBE	__vcpu_single_flag(iflags, BIT(6))
 /* vcpu running in HYP context */
 #define VCPU_HYP_CONTEXT	__vcpu_single_flag(iflags, BIT(7))
+/* Save TRFCR and apply exclude_guest rules */
+#define DEBUG_STATE_SAVE_TRFCR	__vcpu_single_flag(iflags, BIT(8))
 
 /* SVE enabled for host EL0 */
 #define HOST_SVE_ENABLED	__vcpu_single_flag(sflags, BIT(0))
@@ -1096,6 +1102,8 @@ void kvm_arch_vcpu_put_debug_state_flags(struct kvm_vcpu *vcpu);
 void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr);
 void kvm_clr_pmu_events(u32 clr);
 bool kvm_set_pmuserenr(u64 val);
+void kvm_set_etm_events(struct perf_event_attr *attr);
+void kvm_clr_etm_events(void);
 #else
 static inline void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr) {}
 static inline void kvm_clr_pmu_events(u32 clr) {}
arch/arm64/kvm/Makefile +1 -0
index c0c050e53157..0faff57423c4 100644
@@ -23,6 +23,7 @@ kvm-y += arm.o mmu.o mmio.o psci.o hypercalls.o pvtime.o \
 	 vgic/vgic-its.o vgic/vgic-debug.o
 
 kvm-$(CONFIG_HW_PERF_EVENTS)  += pmu-emul.o pmu.o
+kvm-$(CONFIG_PERF_EVENTS) += etm.o
 
 always-y := hyp_constants.h hyp-constants.s
 
arch/arm64/kvm/arm.c +1 -0
index b1a9d47fb2f3..7bd5975328a3 100644
@@ -952,6 +952,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
 		kvm_vgic_flush_hwstate(vcpu);
 
 		kvm_pmu_update_vcpu_events(vcpu);
+		kvm_etm_update_vcpu_events(vcpu);
 
 		/*
 		 * Ensure we set mode to IN_GUEST_MODE after we disable
arch/arm64/kvm/etm.c new file +43 -0
new file mode 100644
index 000000000000..359c37745de2
@@ -0,0 +1,48 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+#include <linux/kvm_host.h>
+
+#include <kvm/etm.h>
+
+static DEFINE_PER_CPU(struct kvm_etm_event, kvm_etm_events);
+
+struct kvm_etm_event *kvm_get_etm_event(void)
+{
+	return this_cpu_ptr(&kvm_etm_events);
+}
+
+void kvm_etm_set_events(struct perf_event_attr *attr)
+{
+	struct kvm_etm_event *etm_event;
+
+	/*
+	 * Exclude guest option only requires extra work with nVHE.
+	 * Otherwise it works automatically with TRFCR_EL{1,2}
+	 */
+	if (has_vhe())
+		return;
+
+	etm_event = kvm_get_etm_event();
+
+	etm_event->exclude_guest = attr->exclude_guest;
+	etm_event->exclude_host = attr->exclude_host;
+	etm_event->exclude_kernel = attr->exclude_kernel;
+	etm_event->exclude_user = attr->exclude_user;
+}
+EXPORT_SYMBOL_GPL(kvm_etm_set_events);
+
+void kvm_etm_clr_events(void)
+{
+	struct kvm_etm_event *etm_event;
+
+	if (has_vhe())
+		return;
+
+	etm_event = kvm_get_etm_event();
+
+	etm_event->exclude_guest = false;
Quoted 9 lines

arch/arm64/kvm/etm.c line 48

Quoted 5 lines
arch/arm64/kvm/etm.c new file +5 -0
+	etm_event->exclude_host = false;
+	etm_event->exclude_kernel = false;
+	etm_event->exclude_user = false;
+}
+EXPORT_SYMBOL_GPL(kvm_etm_clr_events);

Does it really need its own compilation unit if we were to build it at all times?

I can probably throw it in arch/arm64/kvm/debug.c. I think I did that originally but it had the conditional stuff around it so I made the new file instead.

But yeah if I remove the conditional stuff it's probably cleaner to put somewhere else.

Quoted 33 lines
Quoted 33 lines
include/kvm/etm.h new file +27 -0
new file mode 100644
index 000000000000..95c4809fa2b0
@@ -0,0 +1,43 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+
+#ifndef __KVM_DEBUG_H
+#define __KVM_DEBUG_H
+
+struct perf_event_attr;
+struct kvm_vcpu;
+
+#if IS_ENABLED(CONFIG_KVM) && IS_ENABLED(CONFIG_PERF_EVENTS)
+
+struct kvm_etm_event {
+	bool exclude_host;
+	bool exclude_guest;
+	bool exclude_kernel;
+	bool exclude_user;
+};
+
+struct kvm_etm_event *kvm_get_etm_event(void);
+void kvm_etm_clr_events(void);
+void kvm_etm_set_events(struct perf_event_attr *attr);
+
+/*
+ * Updates the vcpu's view of the etm events for this cpu. Must be
+ * called before every vcpu run after disabling interrupts, to ensure
+ * that an interrupt cannot fire and update the structure.
+ */
+#define kvm_etm_update_vcpu_events(vcpu)						\
Quoted 9 lines

include/kvm/etm.h line 32

Quoted 5 lines
include/kvm/etm.h new file +5 -0
+	do {										\
+		if (!has_vhe() && vcpu_get_flag(vcpu, DEBUG_STATE_SAVE_TRFCR))		\
+			vcpu->arch.host_debug_state.etm_event = *kvm_get_etm_event();	\
+	} while (0)
+

Why is it a macro and not a function, which would avoid exposing kvm_get_etm_event?

I think I saw that kvm_pmu_update_vcpu_events() was done that way, and I couldn't inline it without some circular header issue so I made it a macro.

But yes a non inlined function works fine and then kvm_get_etm_event can be hidden so I'll do that.

Quoted 4 lines

Thanks,

	M.

Thanks for the review.

James

#4 lore raw

RFC PATCH 2/3 arm64: KVM: Support exclude_guest for Coresight trace in nVHE

In reply to James Clark

Currently trace will always be generated in nVHE as long as TRBE isn't being used. To allow filtering out guest trace, re-apply the filter rules before switching to the guest.

The TRFCR restore function remains the same.

---
 arch/arm64/kvm/debug.c             |  7 ++++
 arch/arm64/kvm/hyp/nvhe/debug-sr.c | 56 +++++++++++++++++++++++++++---
 2 files changed, 59 insertions(+), 4 deletions(-)
arch/arm64/kvm/debug.c +7 -0
index 8725291cb00a..ebb4db20a859 100644
@@ -335,10 +335,17 @@ void kvm_arch_vcpu_load_debug_state_flags(struct kvm_vcpu *vcpu)
 	if (cpuid_feature_extract_unsigned_field(dfr0, ID_AA64DFR0_EL1_TraceBuffer_SHIFT) &&
 	    !(read_sysreg_s(SYS_TRBIDR_EL1) & TRBIDR_EL1_P))
 		vcpu_set_flag(vcpu, DEBUG_STATE_SAVE_TRBE);
+	/*
+	 * Save TRFCR on nVHE if FEAT_TRF exists. This will be done in cases
+	 * where DEBUG_STATE_SAVE_TRBE doesn't completely disable trace.
+	 */
+	if (cpuid_feature_extract_unsigned_field(dfr0, ID_AA64DFR0_EL1_TraceFilt_SHIFT))
+		vcpu_set_flag(vcpu, DEBUG_STATE_SAVE_TRFCR);
 }
 
 void kvm_arch_vcpu_put_debug_state_flags(struct kvm_vcpu *vcpu)
 {
 	vcpu_clear_flag(vcpu, DEBUG_STATE_SAVE_SPE);
 	vcpu_clear_flag(vcpu, DEBUG_STATE_SAVE_TRBE);
+	vcpu_clear_flag(vcpu, DEBUG_STATE_SAVE_TRFCR);
 }
arch/arm64/kvm/hyp/nvhe/debug-sr.c +52 -4

92 lines not on this page · raw message

Signature · 1 line
2.34.1
maintainer
#5 lore raw

Re: [RFC PATCH 2/3] arm64: KVM: Support exclude_guest for Coresight trace in nVHE

Quoted 46 lines · James Clark

Currently trace will always be generated in nVHE as long as TRBE isn't being used. To allow filtering out guest trace, re-apply the filter rules before switching to the guest.

The TRFCR restore function remains the same.

---
 arch/arm64/kvm/debug.c             |  7 ++++
 arch/arm64/kvm/hyp/nvhe/debug-sr.c | 56 +++++++++++++++++++++++++++---
 2 files changed, 59 insertions(+), 4 deletions(-)
arch/arm64/kvm/debug.c +7 -0
index 8725291cb00a..ebb4db20a859 100644
@@ -335,10 +335,17 @@ void kvm_arch_vcpu_load_debug_state_flags(struct kvm_vcpu *vcpu)
 	if (cpuid_feature_extract_unsigned_field(dfr0, ID_AA64DFR0_EL1_TraceBuffer_SHIFT) &&
 	    !(read_sysreg_s(SYS_TRBIDR_EL1) & TRBIDR_EL1_P))
 		vcpu_set_flag(vcpu, DEBUG_STATE_SAVE_TRBE);
+	/*
+	 * Save TRFCR on nVHE if FEAT_TRF exists. This will be done in cases
+	 * where DEBUG_STATE_SAVE_TRBE doesn't completely disable trace.
+	 */
+	if (cpuid_feature_extract_unsigned_field(dfr0, ID_AA64DFR0_EL1_TraceFilt_SHIFT))
+		vcpu_set_flag(vcpu, DEBUG_STATE_SAVE_TRFCR);
 }
 
 void kvm_arch_vcpu_put_debug_state_flags(struct kvm_vcpu *vcpu)
 {
 	vcpu_clear_flag(vcpu, DEBUG_STATE_SAVE_SPE);
 	vcpu_clear_flag(vcpu, DEBUG_STATE_SAVE_TRBE);
+	vcpu_clear_flag(vcpu, DEBUG_STATE_SAVE_TRFCR);
 }
arch/arm64/kvm/hyp/nvhe/debug-sr.c +2 -1
index 4558c02eb352..0e8c85b29b92 100644
@@ -51,13 +51,17 @@ static void __debug_restore_spe(u64 pmscr_el1)
 	write_sysreg_s(pmscr_el1, SYS_PMSCR_EL1);
 }
 
-static void __debug_save_trace(u64 *trfcr_el1)
+/*
+ * Save TRFCR and disable trace completely if TRBE is being used. Return true

arch/arm64/kvm/hyp/nvhe/debug-sr.c static void __debug_restore_spe(u64 pmscr_el1) · line 64

Quoted 10 lines
arch/arm64/kvm/hyp/nvhe/debug-sr.c +4 -1
+ * if trace was disabled.
+ */
+static bool __debug_save_trace(u64 *trfcr_el1)
 {
 	*trfcr_el1 = 0;
 
 	/* Check if the TRBE is enabled */
 	if (!(read_sysreg_s(SYS_TRBLIMITR_EL1) & TRBLIMITR_EL1_E))
-		return;
+		return false;

While you're refactoring this code, please move the zeroing of *trfcr_el1 under the if statement.

Quoted 13 lines
arch/arm64/kvm/hyp/nvhe/debug-sr.c +2 -0
 	/*
 	 * Prohibit trace generation while we are in guest.
 	 * Since access to TRFCR_EL1 is trapped, the guest can't
@@ -68,6 +72,8 @@ static void __debug_save_trace(u64 *trfcr_el1)
 	isb();
 	/* Drain the trace buffer to memory */
 	tsb_csync();
+
+	return true;
 }
 
 static void __debug_restore_trace(u64 trfcr_el1)

arch/arm64/kvm/hyp/nvhe/debug-sr.c static void __debug_restore_trace(u64 trfcr_el1) · line 88

Quoted 4 lines
arch/arm64/kvm/hyp/nvhe/debug-sr.c +1 -0
@@ -79,14 +85,55 @@ static void __debug_restore_trace(u64 trfcr_el1)
 	write_sysreg_s(trfcr_el1, SYS_TRFCR_EL1);
 }
 
+#if IS_ENABLED(CONFIG_PERF_EVENTS)

As previously stated, just always compile this. There shouldn't be anything here that's so large that it becomes a candidate for exclusion. Hell, even the whole of NV+pKVM are permanent features, even of most people won't use *any* of that.

arch/arm64/kvm/hyp/nvhe/debug-sr.c static void __debug_restore_trace(u64 trfcr_el1) · line 97

Quoted 9 lines
arch/arm64/kvm/hyp/nvhe/debug-sr.c +9 -0
+static inline void __debug_save_trfcr(struct kvm_vcpu *vcpu)
+{
+	u64 trfcr;
+	struct kvm_etm_event etm_event = vcpu->arch.host_debug_state.etm_event;
+
+	/* No change if neither are excluded */
+	if (!etm_event.exclude_guest && !etm_event.exclude_host) {
+		/* Zeroing prevents restoring a stale value */
+		vcpu->arch.host_debug_state.trfcr_el1 = 0;

I find this "zero means do nothing" part very odd. I can see it is already done, but I really dislike this sort of assumption to avoid writing to a register.

I'd really prefer we track another version of TRFCR_EL1, compare host and guest, and decide to avoid writing if they are equal. At least, it would be readable.

And in the end, expressing *everything* in terms of the register would really help, instead of the exclude_* stuff that has no place in the low-level arch code.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#6 lore raw

Re: [RFC PATCH 2/3] arm64: KVM: Support exclude_guest for Coresight trace in nVHE

Quoted 99 lines · Marc Zyngier
Quoted 46 lines · James Clark

Currently trace will always be generated in nVHE as long as TRBE isn't being used. To allow filtering out guest trace, re-apply the filter rules before switching to the guest.

The TRFCR restore function remains the same.

---
 arch/arm64/kvm/debug.c             |  7 ++++
 arch/arm64/kvm/hyp/nvhe/debug-sr.c | 56 +++++++++++++++++++++++++++---
 2 files changed, 59 insertions(+), 4 deletions(-)
arch/arm64/kvm/debug.c +7 -0
index 8725291cb00a..ebb4db20a859 100644
@@ -335,10 +335,17 @@ void kvm_arch_vcpu_load_debug_state_flags(struct kvm_vcpu *vcpu)
 	if (cpuid_feature_extract_unsigned_field(dfr0, ID_AA64DFR0_EL1_TraceBuffer_SHIFT) &&
 	    !(read_sysreg_s(SYS_TRBIDR_EL1) & TRBIDR_EL1_P))
 		vcpu_set_flag(vcpu, DEBUG_STATE_SAVE_TRBE);
+	/*
+	 * Save TRFCR on nVHE if FEAT_TRF exists. This will be done in cases
+	 * where DEBUG_STATE_SAVE_TRBE doesn't completely disable trace.
+	 */
+	if (cpuid_feature_extract_unsigned_field(dfr0, ID_AA64DFR0_EL1_TraceFilt_SHIFT))
+		vcpu_set_flag(vcpu, DEBUG_STATE_SAVE_TRFCR);
 }
 
 void kvm_arch_vcpu_put_debug_state_flags(struct kvm_vcpu *vcpu)
 {
 	vcpu_clear_flag(vcpu, DEBUG_STATE_SAVE_SPE);
 	vcpu_clear_flag(vcpu, DEBUG_STATE_SAVE_TRBE);
+	vcpu_clear_flag(vcpu, DEBUG_STATE_SAVE_TRFCR);
 }
arch/arm64/kvm/hyp/nvhe/debug-sr.c +2 -1
index 4558c02eb352..0e8c85b29b92 100644
@@ -51,13 +51,17 @@ static void __debug_restore_spe(u64 pmscr_el1)
 	write_sysreg_s(pmscr_el1, SYS_PMSCR_EL1);
 }
 
-static void __debug_save_trace(u64 *trfcr_el1)
+/*
+ * Save TRFCR and disable trace completely if TRBE is being used. Return true

arch/arm64/kvm/hyp/nvhe/debug-sr.c static void __debug_restore_spe(u64 pmscr_el1) · line 64

Quoted 10 lines
arch/arm64/kvm/hyp/nvhe/debug-sr.c +4 -1
+ * if trace was disabled.
+ */
+static bool __debug_save_trace(u64 *trfcr_el1)
 {
 	*trfcr_el1 = 0;
 
 	/* Check if the TRBE is enabled */
 	if (!(read_sysreg_s(SYS_TRBLIMITR_EL1) & TRBLIMITR_EL1_E))
-		return;
+		return false;

While you're refactoring this code, please move the zeroing of *trfcr_el1 under the if statement.

Quoted 13 lines
arch/arm64/kvm/hyp/nvhe/debug-sr.c +2 -0
 	/*
 	 * Prohibit trace generation while we are in guest.
 	 * Since access to TRFCR_EL1 is trapped, the guest can't
@@ -68,6 +72,8 @@ static void __debug_save_trace(u64 *trfcr_el1)
 	isb();
 	/* Drain the trace buffer to memory */
 	tsb_csync();
+
+	return true;
 }
 
 static void __debug_restore_trace(u64 trfcr_el1)

arch/arm64/kvm/hyp/nvhe/debug-sr.c static void __debug_restore_trace(u64 trfcr_el1) · line 88

Quoted 4 lines
arch/arm64/kvm/hyp/nvhe/debug-sr.c +1 -0
@@ -79,14 +85,55 @@ static void __debug_restore_trace(u64 trfcr_el1)
 	write_sysreg_s(trfcr_el1, SYS_TRFCR_EL1);
 }
 
+#if IS_ENABLED(CONFIG_PERF_EVENTS)

As previously stated, just always compile this. There shouldn't be anything here that's so large that it becomes a candidate for exclusion. Hell, even the whole of NV+pKVM are permanent features, even of most people won't use *any* of that.

arch/arm64/kvm/hyp/nvhe/debug-sr.c static void __debug_restore_trace(u64 trfcr_el1) · line 97

Quoted 9 lines
arch/arm64/kvm/hyp/nvhe/debug-sr.c +9 -0
+static inline void __debug_save_trfcr(struct kvm_vcpu *vcpu)
+{
+	u64 trfcr;
+	struct kvm_etm_event etm_event = vcpu->arch.host_debug_state.etm_event;
+
+	/* No change if neither are excluded */
+	if (!etm_event.exclude_guest && !etm_event.exclude_host) {
+		/* Zeroing prevents restoring a stale value */
+		vcpu->arch.host_debug_state.trfcr_el1 = 0;

I find this "zero means do nothing" part very odd. I can see it is already done, but I really dislike this sort of assumption to avoid writing to a register.

Quoted 8 lines

I'd really prefer we track another version of TRFCR_EL1, compare host and guest, and decide to avoid writing if they are equal. At least, it would be readable.

And in the end, expressing *everything* in terms of the register would really help, instead of the exclude_* stuff that has no place in the low-level arch code.

Yep, I agree with all of the above, I can make these changes for the next version. I just want to clarify your point about disabling trace for protected guests when not in debug mode that I asked about in the review on patch 1.

Quoted 4 lines

Thanks,

	M.
#7 lore raw

RFC PATCH 3/3 coresight: Support exclude_guest with Feat_TRF and nVHE

In reply to James Clark

With nVHE the filters need to be applied before switching to the guest, so supply the per-cpu filter status to KVM.

---
 drivers/hwtracing/coresight/coresight-etm-perf.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/hwtracing/coresight/coresight-etm-perf.c +4 -0
index 5ca6278baff4..f78f05e656f5 100644
@@ -9,6 +9,7 @@
 #include <linux/coresight-pmu.h>
 #include <linux/cpumask.h>
 #include <linux/device.h>
+#include <linux/kvm_host.h>
 #include <linux/list.h>
 #include <linux/mm.h>
 #include <linux/init.h>
@@ -510,6 +511,7 @@ static void etm_event_start(struct perf_event *event, int flags)
 	}
 
 out:
+	kvm_etm_set_events(&event->attr);
 	/* Tell the perf core the event is alive */
 	event->hw.state = 0;
 	/* Save the event_data for this ETM */
@@ -627,6 +629,8 @@ static void etm_event_stop(struct perf_event *event, int mode)
 
 	/* Disabling the path make its elements available to other sessions */
 	coresight_disable_path(path);
+
+	kvm_etm_clr_events();
 }
 
 static int etm_event_add(struct perf_event *event, int mode)
Signature · 1 line
2.34.1
maintainer
#8 lore raw
In reply to James Clark
Quoted 10 lines · James Clark

Hi,

I'm looking for help in testing this and for feedback on whether it's useful to anyone. Testing it requires hardware that has Feat_TRF (v8.4) but no TRBE. This is because TRBE usage is disabled in nVHE guests.

I don't currently have any access to any hardware, and the FVP model can only do self hosted trace using TRBE.

Quoted 6 lines

Currently with nVHE you would always get trace from guests, and filtering out isn't possible without this patchset. In comparison, with VHE guests, they never generate guest trace without [1]. I think the existence of trace rather than lack of could suggest that this change is less useful than [1]. Also the restricted set of hardware that it works on supports that too.

It'd be nice to have some sort of feature parity, but it seems like a vanishingly small target of users having access to an ETM sink.

Quoted 3 lines

Apart from compilation and checking that the exclude guest settings are correctly programmed on guest switch, this is untested by me.

I'll have a look at the series, but none of my HW fits in this description (my ARMv8.4+ boxes don't have any form of tracing).

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
maintainer
#9 lore raw

Hi Marc

Quoted 23 lines · Marc Zyngier
Quoted 10 lines · James Clark

Hi,

I'm looking for help in testing this and for feedback on whether it's useful to anyone. Testing it requires hardware that has Feat_TRF (v8.4) but no TRBE. This is because TRBE usage is disabled in nVHE guests.

I don't currently have any access to any hardware, and the FVP model can only do self hosted trace using TRBE.

Quoted 6 lines

Currently with nVHE you would always get trace from guests, and filtering out isn't possible without this patchset. In comparison, with VHE guests, they never generate guest trace without [1]. I think the existence of trace rather than lack of could suggest that this change is less useful than [1]. Also the restricted set of hardware that it works on supports that too.

It'd be nice to have some sort of feature parity, but it seems like a vanishingly small target of users having access to an ETM sink.

Quoted 1 line
Quoted 5 lines
Quoted 2 lines

Apart from compilation and checking that the exclude guest settings are correctly programmed on guest switch, this is untested by me.

I'll have a look at the series, but none of my HW fits in this description (my ARMv8.4+ boxes don't have any form of tracing).

While I have your attention, we have another series that manages the trace filtering for Guests on VHE, completely within the Coresight etm4x driver here [0]. I personally think, it is good to have the guest filtering for both nVHE and VHE under the KVM control, like we do in this series. I would like your opinion on this.

[0] https://lkml.kernel.org/r/[email protected]

Suzuki

Quoted 5 lines

Thanks,

	M.
maintainer
#10 lore raw
Quoted 30 lines · Suzuki K Poulose

Hi Marc

Quoted 26 lines · Marc Zyngier
Quoted 10 lines · James Clark

Hi,

I'm looking for help in testing this and for feedback on whether it's useful to anyone. Testing it requires hardware that has Feat_TRF (v8.4) but no TRBE. This is because TRBE usage is disabled in nVHE guests.

I don't currently have any access to any hardware, and the FVP model can only do self hosted trace using TRBE.

Quoted 6 lines

Currently with nVHE you would always get trace from guests, and filtering out isn't possible without this patchset. In comparison, with VHE guests, they never generate guest trace without [1]. I think the existence of trace rather than lack of could suggest that this change is less useful than [1]. Also the restricted set of hardware that it works on supports that too.

It'd be nice to have some sort of feature parity, but it seems like a vanishingly small target of users having access to an ETM sink.

Quoted 3 lines

Apart from compilation and checking that the exclude guest settings are correctly programmed on guest switch, this is untested by me.

Quoted 8 lines
Quoted 2 lines

I'll have a look at the series, but none of my HW fits in this description (my ARMv8.4+ boxes don't have any form of tracing).

While I have your attention, we have another series that manages the trace filtering for Guests on VHE, completely within the Coresight etm4x driver here [0]. I personally think, it is good to have the guest filtering for both nVHE and VHE under the KVM control, like we do in this series. I would like your opinion on this.

I just quickly reviewed the first two patches of the nVHE series, and I think the shape is a bit wrong. I can absolutely see the interest of preventing extra tracing when a guest is running, but the way this is currently plugged makes it hard to reason about it.

Quoted 1 line

[0] https://lkml.kernel.org/r/[email protected]

I'll try to have a look at that one later.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
maintainer
#11 lore raw
In reply to James Clark
Cc Ganpatrao, Steve, Tanmay
Quoted 8 lines · James Clark

Hi,

I'm looking for help in testing this and for feedback on whether it's useful to anyone. Testing it requires hardware that has Feat_TRF (v8.4) but no TRBE. This is because TRBE usage is disabled in nVHE guests.

I don't currently have any access to any hardware, and the FVP model can only do self hosted trace using TRBE.

If you have a v8.4+ (and not v9) HW, please could you give this a spin ?

Suzuki

Quoted 32 lines

Currently with nVHE you would always get trace from guests, and filtering out isn't possible without this patchset. In comparison, with VHE guests, they never generate guest trace without [1]. I think the existence of trace rather than lack of could suggest that this change is less useful than [1]. Also the restricted set of hardware that it works on supports that too.

Apart from compilation and checking that the exclude guest settings are correctly programmed on guest switch, this is untested by me.

Applies to kvmarm/next (3b4e3afb2032)

[1]: https://lore.kernel.org/linux-arm-kernel/[email protected]/

James Clark (3):
arm64: KVM: Add support for exclude_guest and exclude_host for ETM
arm64: KVM: Support exclude_guest for Coresight trace in nVHE
coresight: Support exclude_guest with Feat_TRF and nVHE

  arch/arm64/include/asm/kvm_host.h             | 10 +++-
  arch/arm64/kvm/Makefile                       |  1 +
  arch/arm64/kvm/arm.c                          |  1 +
  arch/arm64/kvm/debug.c                        |  7 +++
  arch/arm64/kvm/etm.c                          | 48 ++++++++++++++++
  arch/arm64/kvm/hyp/nvhe/debug-sr.c            | 56 +++++++++++++++++--
  .../hwtracing/coresight/coresight-etm-perf.c  |  4 ++
  include/kvm/etm.h                             | 43 ++++++++++++++
  8 files changed, 165 insertions(+), 5 deletions(-)

create mode 100644 arch/arm64/kvm/etm.c
create mode 100644 include/kvm/etm.h