Skip to contentSkip to conversation

riscv: Add perf support to collect KVM guest statistics from host side

PATCH Applied [email protected] 3 participants 6 messages Oct 15, 2024 – Dec 11, 2024

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

v5 cover riscv: Add perf support to collect KVM guest statistics from host side

From: Quan Zhou <[email protected]>

Add basic guest support to RISC-V perf, enabling it to distinguish whether PMU interrupts occur in the host or the guest, and then collect some basic guest information from the host side (guest os callchain is not supported for now).

Based on the x86/arm implementation, tested with kvm-riscv. test env:
- host: qemu-9.0.0
- guest: qemu-9.0.0 --enable-kvm (only start one guest and run top)

-----------------------------------------
1) perf kvm top
./perf kvm --host --guest \
--guestkallsyms=/<path-to-kallsyms> \
--guestmodules=/<path-to-modules> top

PerfTop: 41 irqs/sec kernel:97.6% us: 0.0% guest kernel: 0.0% guest us: 0.0% exact: 0.0% [250Hz cycles:P], (all, 4 CPUs)
-------------------------------------------------------------------------------

    64.57%  [kernel]        [k] default_idle_call
     3.12%  [kernel]        [k] _raw_spin_unlock_irqrestore
     3.03%  [guest.kernel]  [g] mem_serial_out
     2.61%  [kernel]        [k] handle_softirqs
     2.32%  [kernel]        [k] do_trap_ecall_u
     1.71%  [kernel]        [k] _raw_spin_unlock_irq
     1.26%  [guest.kernel]  [g] do_raw_spin_lock
     1.25%  [kernel]        [k] finish_task_switch.isra.0
     1.16%  [kernel]        [k] do_idle
     0.77%  libc.so.6       [.] ioctl
     0.76%  [kernel]        [k] queue_work_on
     0.69%  [kernel]        [k] __local_bh_enable_ip
     0.67%  [guest.kernel]  [g] __noinstr_text_start
     0.64%  [guest.kernel]  [g] mem_serial_in
     0.41%  libc.so.6       [.] pthread_sigmask
     0.39%  [kernel]        [k] mem_cgroup_uncharge_skmem
     0.39%  [kernel]        [k] __might_resched
     0.39%  [guest.kernel]  [g] _nohz_idle_balance.isra.0
     0.37%  [kernel]        [k] sched_balance_update_blocked_averages
     0.34%  [kernel]        [k] sched_balance_rq

2) perf kvm record
./perf kvm --host --guest \
--guestkallsyms=/<path-to-kallsyms> \
--guestmodules=/<path-to-modules> record -a sleep 60

[ perf record: Woken up 3 times to write data ]
[ perf record: Captured and wrote 1.292 MB perf.data.kvm (17990 samples) ]

3) perf kvm report
./perf kvm --host --guest \
--guestkallsyms=/<path-to-kallsyms> \
--guestmodules=/<path-to-modules> report -i perf.data.kvm

# Total Lost Samples: 0
#
# Samples: 17K of event 'cycles:P'
# Event count (approx.): 269968947184
#

# Overhead  Command          Shared Object            Symbol                                        
# ........  ...............  .......................  ..............................................
#
    61.86%  swapper          [kernel.kallsyms]        [k] default_idle_call
     2.93%  :6463            [guest.kernel.kallsyms]  [g] do_raw_spin_lock
     2.82%  :6462            [guest.kernel.kallsyms]  [g] mem_serial_out
     2.11%  sshd             [kernel.kallsyms]        [k] _raw_spin_unlock_irqrestore
     1.78%  :6462            [guest.kernel.kallsyms]  [g] do_raw_spin_lock
     1.37%  swapper          [kernel.kallsyms]        [k] handle_softirqs
     1.36%  swapper          [kernel.kallsyms]        [k] do_idle
     1.21%  sshd             [kernel.kallsyms]        [k] do_trap_ecall_u
     1.21%  sshd             [kernel.kallsyms]        [k] _raw_spin_unlock_irq
     1.11%  qemu-system-ris  [kernel.kallsyms]        [k] do_trap_ecall_u
     0.93%  qemu-system-ris  libc.so.6                [.] ioctl
     0.89%  sshd             [kernel.kallsyms]        [k] __local_bh_enable_ip
     0.77%  qemu-system-ris  [kernel.kallsyms]        [k] _raw_spin_unlock_irqrestore
     0.68%  qemu-system-ris  [kernel.kallsyms]        [k] queue_work_on
     0.65%  sshd             [kernel.kallsyms]        [k] handle_softirqs
     0.44%  :6462            [guest.kernel.kallsyms]  [g] mem_serial_in
     0.42%  sshd             libc.so.6                [.] pthread_sigmask
     0.34%  :6462            [guest.kernel.kallsyms]  [g] serial8250_tx_chars
     0.30%  swapper          [kernel.kallsyms]        [k] finish_task_switch.isra.0
     0.29%  swapper          [kernel.kallsyms]        [k] sched_balance_rq
     0.29%  sshd             [kernel.kallsyms]        [k] __might_resched
     0.26%  swapper          [kernel.kallsyms]        [k] tick_nohz_idle_exit
     0.26%  swapper          [kernel.kallsyms]        [k] sched_balance_update_blocked_averages
     0.26%  swapper          [kernel.kallsyms]        [k] _nohz_idle_balance.isra.0
     0.24%  qemu-system-ris  [kernel.kallsyms]        [k] finish_task_switch.isra.0
     0.23%  :6462            [guest.kernel.kallsyms]  [g] __noinstr_text_start

---
Change since v4:
- Add Reviewed-by tags

Change since v3:
- Rebased on v6.12-rc3

Change since v2:
- Rebased on v6.11-rc7
- Keep the misc type consistent with other architectures as unsigned long (Andrew)
- Add the same comment for kvm_arch_pmi_in_guest as in arm64. (Andrew)

Change since v1:
- Rebased on v6.11-rc3
- Fix incorrect misc type (Andrew)

---
v4 link:
https://lore.kernel.org/all/cover.1728957131.git.zhouquan at iscas.ac.cn/ v3 link:
https://lore.kernel.org/all/cover.1726126795.git.zhouquan at iscas.ac.cn/ v2 link:
https://lore.kernel.org/all/cover.1723518282.git.zhouquan at iscas.ac.cn/ v1 link:
https://lore.kernel.org/all/cover.1721271251.git.zhouquan at iscas.ac.cn/

Quan Zhou (2):
riscv: perf: add guest vs host distinction
riscv: KVM: add basic support for host vs guest profiling

 arch/riscv/include/asm/kvm_host.h   | 10 ++++++++
 arch/riscv/include/asm/perf_event.h |  6 +++++
 arch/riscv/kernel/perf_callchain.c  | 38 +++++++++++++++++++++++++++++
 arch/riscv/kvm/Kconfig              |  1 +
 arch/riscv/kvm/main.c               | 12 +++++++--
 arch/riscv/kvm/vcpu.c               |  7 ++++++
 6 files changed, 72 insertions(+), 2 deletions(-)

base-commit: 8e929cb546ee42c9a61d24fae60605e9e3192354

Signature · 1 line
2.34.1
#1 lore raw

PATCH v5 1/2 riscv: perf: add guest vs host distinction

From: Quan Zhou <[email protected]>

Introduce basic guest support in perf, enabling it to distinguish between PMU interrupts in the host or guest, and collect fundamental information.

---
 arch/riscv/include/asm/perf_event.h |  6 +++++
 arch/riscv/kernel/perf_callchain.c  | 38 +++++++++++++++++++++++++++++
 2 files changed, 44 insertions(+)
arch/riscv/include/asm/perf_event.h +6 -0
index 665bbc9b2f84..38926b4a902d 100644
@@ -8,7 +8,11 @@
 #ifndef _ASM_RISCV_PERF_EVENT_H
 #define _ASM_RISCV_PERF_EVENT_H
 
+#ifdef CONFIG_PERF_EVENTS
 #include <linux/perf_event.h>
+extern unsigned long perf_instruction_pointer(struct pt_regs *regs);
+extern unsigned long perf_misc_flags(struct pt_regs *regs);
+#define perf_misc_flags(regs) perf_misc_flags(regs)
 #define perf_arch_bpf_user_pt_regs(regs) (struct user_regs_struct *)regs
 
 #define perf_arch_fetch_caller_regs(regs, __ip) { \
@@ -17,4 +21,6 @@
 	(regs)->sp = current_stack_pointer; \
 	(regs)->status = SR_PP; \
 }
+#endif
+
 #endif /* _ASM_RISCV_PERF_EVENT_H */
arch/riscv/kernel/perf_callchain.c +38 -0
index c7468af77c66..c2c81a80f816 100644
@@ -28,11 +28,49 @@ static bool fill_callchain(void *entry, unsigned long pc)
 void perf_callchain_user(struct perf_callchain_entry_ctx *entry,
 			 struct pt_regs *regs)
 {
+	if (perf_guest_state()) {
+		/* TODO: We don't support guest os callchain now */
+		return;
+	}
+
 	arch_stack_walk_user(fill_callchain, entry, regs);
 }
 
 void perf_callchain_kernel(struct perf_callchain_entry_ctx *entry,
 			   struct pt_regs *regs)
 {
+	if (perf_guest_state()) {
+		/* TODO: We don't support guest os callchain now */
+		return;
+	}
+
 	walk_stackframe(NULL, regs, fill_callchain, entry);
 }
+
+unsigned long perf_instruction_pointer(struct pt_regs *regs)
+{
+	if (perf_guest_state())
+		return perf_guest_get_ip();
+
+	return instruction_pointer(regs);
+}
+
+unsigned long perf_misc_flags(struct pt_regs *regs)
+{
+	unsigned int guest_state = perf_guest_state();
+	unsigned long misc = 0;
+
+	if (guest_state) {
+		if (guest_state & PERF_GUEST_USER)
+			misc |= PERF_RECORD_MISC_GUEST_USER;
+		else
+			misc |= PERF_RECORD_MISC_GUEST_KERNEL;
+	} else {
+		if (user_mode(regs))
+			misc |= PERF_RECORD_MISC_USER;
+		else
+			misc |= PERF_RECORD_MISC_KERNEL;
+	}
+
+	return misc;
+}
Signature · 1 line
2.34.1
maintainer
#2 lore raw

Re: [PATCH v5 1/2] riscv: perf: add guest vs host distinction

Quoted 41 lines · zhouquan at iscas.ac.cn

From: Quan Zhou <[email protected]>

Introduce basic guest support in perf, enabling it to distinguish between PMU interrupts in the host or guest, and collect fundamental information.

---
 arch/riscv/include/asm/perf_event.h |  6 +++++
 arch/riscv/kernel/perf_callchain.c  | 38 +++++++++++++++++++++++++++++
 2 files changed, 44 insertions(+)
arch/riscv/include/asm/perf_event.h +6 -0
index 665bbc9b2f84..38926b4a902d 100644
@@ -8,7 +8,11 @@
 #ifndef _ASM_RISCV_PERF_EVENT_H
 #define _ASM_RISCV_PERF_EVENT_H
 
+#ifdef CONFIG_PERF_EVENTS
 #include <linux/perf_event.h>
+extern unsigned long perf_instruction_pointer(struct pt_regs *regs);
+extern unsigned long perf_misc_flags(struct pt_regs *regs);
+#define perf_misc_flags(regs) perf_misc_flags(regs)
 #define perf_arch_bpf_user_pt_regs(regs) (struct user_regs_struct *)regs
 
 #define perf_arch_fetch_caller_regs(regs, __ip) { \
@@ -17,4 +21,6 @@
 	(regs)->sp = current_stack_pointer; \
 	(regs)->status = SR_PP; \
 }
+#endif
+
 #endif /* _ASM_RISCV_PERF_EVENT_H */
arch/riscv/kernel/perf_callchain.c +0 -0
index c7468af77c66..c2c81a80f816 100644

arch/riscv/kernel/perf_callchain.c static bool fill_callchain(void *entry, unsigned long pc) · line 33

Quoted 6 lines
arch/riscv/kernel/perf_callchain.c +3 -0
@@ -28,11 +28,49 @@ static bool fill_callchain(void *entry, unsigned long pc)
 void perf_callchain_user(struct perf_callchain_entry_ctx *entry,
 			 struct pt_regs *regs)
 {
+	if (perf_guest_state()) {
+		/* TODO: We don't support guest os callchain now */
+		return;

That seems kind of weird, but it looks like almost exactly the same thing Marc did in 75e424620a4f ("arm64: perf: add guest vs host discrimination"). I think it's safe, as we'll basically just silently display no backtrace and we can always just fail to backtrace.

That said: I don't understand why we can't backtrace inside a guest? If we can get the registers and memory it seems like we should be able to. Maybe I'm missing something?

Quoted 43 lines
arch/riscv/kernel/perf_callchain.c +35 -0
+	}
+
 	arch_stack_walk_user(fill_callchain, entry, regs);
 }
 
 void perf_callchain_kernel(struct perf_callchain_entry_ctx *entry,
 			   struct pt_regs *regs)
 {
+	if (perf_guest_state()) {
+		/* TODO: We don't support guest os callchain now */
+		return;
+	}
+
 	walk_stackframe(NULL, regs, fill_callchain, entry);
 }
+
+unsigned long perf_instruction_pointer(struct pt_regs *regs)
+{
+	if (perf_guest_state())
+		return perf_guest_get_ip();
+
+	return instruction_pointer(regs);
+}
+
+unsigned long perf_misc_flags(struct pt_regs *regs)
+{
+	unsigned int guest_state = perf_guest_state();
+	unsigned long misc = 0;
+
+	if (guest_state) {
+		if (guest_state & PERF_GUEST_USER)
+			misc |= PERF_RECORD_MISC_GUEST_USER;
+		else
+			misc |= PERF_RECORD_MISC_GUEST_KERNEL;
+	} else {
+		if (user_mode(regs))
+			misc |= PERF_RECORD_MISC_USER;
+		else
+			misc |= PERF_RECORD_MISC_KERNEL;
+	}
+
+	return misc;
+}
author
#3 lore raw

Re: [PATCH v5 1/2] riscv: perf: add guest vs host distinction

Quoted 57 lines · Palmer Dabbelt
Quoted 43 lines · zhouquan at iscas.ac.cn

From: Quan Zhou <[email protected]>

Introduce basic guest support in perf, enabling it to distinguish between PMU interrupts in the host or guest, and collect fundamental information.

---
?arch/riscv/include/asm/perf_event.h |? 6 +++++ ?arch/riscv/kernel/perf_callchain.c? | 38 +++++++++++++++++++++++++++++ ?2 files changed, 44 insertions(+)

arch/riscv/include/asm/perf_event.h +0 -0
index 665bbc9b2f84..38926b4a902d 100644
@@ -8,7 +8,11 @@

?#ifndef _ASM_RISCV_PERF_EVENT_H
?#define _ASM_RISCV_PERF_EVENT_H

+#ifdef CONFIG_PERF_EVENTS
?#include <linux/perf_event.h>

arch/riscv/include/asm/perf_event.h +3 -0
+extern unsigned long perf_instruction_pointer(struct pt_regs *regs);
+extern unsigned long perf_misc_flags(struct pt_regs *regs);
+#define perf_misc_flags(regs) perf_misc_flags(regs)

?#define perf_arch_bpf_user_pt_regs(regs) (struct user_regs_struct *)regs

?#define perf_arch_fetch_caller_regs(regs, __ip) { \

arch/riscv/include/asm/perf_event.h +2 -0
@@ -17,4 +21,6 @@
 ???? (regs)->sp = current_stack_pointer; \
 ???? (regs)->status = SR_PP; \
 ?}
+#endif
+

?#endif /* _ASM_RISCV_PERF_EVENT_H */

arch/riscv/kernel/perf_callchain.c +0 -0
index c7468af77c66..c2c81a80f816 100644

arch/riscv/kernel/perf_callchain.c static bool fill_callchain(void *entry, unsigned · line 34

Quoted 7 lines
arch/riscv/kernel/perf_callchain.c +3 -0
@@ -28,11 +28,49 @@ static bool fill_callchain(void *entry, unsigned 
 long pc)
 ?void perf_callchain_user(struct perf_callchain_entry_ctx *entry,
 ????????????? struct pt_regs *regs)
 ?{
+??? if (perf_guest_state()) {
+??????? /* TODO: We don't support guest os callchain now */
+??????? return;

That seems kind of weird, but it looks like almost exactly the same thing Marc did in 75e424620a4f ("arm64: perf: add guest vs host discrimination").? I think it's safe, as we'll basically just silently display no backtrace and we can always just fail to backtrace.

Quoted 4 lines

That said: I don't understand why we can't backtrace inside a guest?? If we can get the registers and memory it seems like we should be able to. Maybe I'm missing something?

From the community's discussion history, there are two reasons:

1) For backtracing, we must traverse the structures within the guest's virtual address space. These structures can change at any time while the guest is running. The page table data we obtain might be corrupted, making it impossible to complete the traversal of the page tables or leading to incorrect data [1].

2) For security reasons, a significant number of cloud vendors wish to make accessing customer data almost impossible [2].

Currently, community prefer to implement this functionality in user space and access the guest through the KVM API, interrupting the guest to perform the unwind operation.

As with the x86/ARM architectures, stubs are still retained here for RISC-V. If guest os callchain is introduced in the future, they can indicate that additional work needs to be done.

[1]
https://lore.kernel.org/all/ZSlNsn-f1j2bB8pW at FVFF77S0Q05N.cambridge.arm.com/
[2] https://lore.kernel.org/all/ZXeTvCLURmwzpDkP at google.com/

Regards,
Quan

Quoted 43 lines
Quoted 43 lines
arch/riscv/kernel/perf_callchain.c +7 -0
+??? }
+
 ???? arch_stack_walk_user(fill_callchain, entry, regs);
 ?}
 
 ?void perf_callchain_kernel(struct perf_callchain_entry_ctx *entry,
 ??????????????? struct pt_regs *regs)
 ?{
+??? if (perf_guest_state()) {
+??????? /* TODO: We don't support guest os callchain now */
+??????? return;
+??? }
+
 ???? walk_stackframe(NULL, regs, fill_callchain, entry);

?}

arch/riscv/kernel/perf_callchain.c +28 -0
+
+unsigned long perf_instruction_pointer(struct pt_regs *regs)
+{
+??? if (perf_guest_state())
+??????? return perf_guest_get_ip();
+
+??? return instruction_pointer(regs);
+}
+
+unsigned long perf_misc_flags(struct pt_regs *regs)
+{
+??? unsigned int guest_state = perf_guest_state();
+??? unsigned long misc = 0;
+
+??? if (guest_state) {
+??????? if (guest_state & PERF_GUEST_USER)
+??????????? misc |= PERF_RECORD_MISC_GUEST_USER;
+??????? else
+??????????? misc |= PERF_RECORD_MISC_GUEST_KERNEL;
+??? } else {
+??????? if (user_mode(regs))
+??????????? misc |= PERF_RECORD_MISC_USER;
+??????? else
+??????????? misc |= PERF_RECORD_MISC_KERNEL;
+??? }
+
+??? return misc;
+}
#4 lore raw

PATCH v5 2/2 riscv: KVM: add basic support for host vs guest profiling

In reply to [email protected]

From: Quan Zhou <[email protected]>

For the information collected on the host side, we need to identify which data originates from the guest and record these events separately, this can be achieved by having KVM register perf callbacks.

---
 arch/riscv/include/asm/kvm_host.h | 10 ++++++++++
 arch/riscv/kvm/Kconfig            |  1 +
 arch/riscv/kvm/main.c             | 12 ++++++++++--
 arch/riscv/kvm/vcpu.c             |  7 +++++++
 4 files changed, 28 insertions(+), 2 deletions(-)
arch/riscv/include/asm/kvm_host.h +10 -0
index 2e2254fd2a2a..35eab6e0f4ae 100644
@@ -286,6 +286,16 @@ struct kvm_vcpu_arch {
 	} sta;
 };
 
+/*
+ * Returns true if a Performance Monitoring Interrupt (PMI), a.k.a. perf event,
+ * arrived in guest context.  For riscv, any event that arrives while a vCPU is
+ * loaded is considered to be "in guest".
+ */
+static inline bool kvm_arch_pmi_in_guest(struct kvm_vcpu *vcpu)
+{
+	return IS_ENABLED(CONFIG_GUEST_PERF_EVENTS) && !!vcpu;
+}
+
 static inline void kvm_arch_sync_events(struct kvm *kvm) {}
 
 #define KVM_RISCV_GSTAGE_TLB_MIN_ORDER		12
arch/riscv/kvm/Kconfig +1 -0
index 26d1727f0550..0c3cbb0915ff 100644
@@ -32,6 +32,7 @@ config KVM
 	select KVM_XFER_TO_GUEST_WORK
 	select KVM_GENERIC_MMU_NOTIFIER
 	select SCHED_INFO
+	select GUEST_PERF_EVENTS if PERF_EVENTS
 	help
 	  Support hosting virtualized guest machines.
 
arch/riscv/kvm/main.c +10 -2
index f3427f6de608..5682e338ae6d 100644
@@ -51,6 +51,12 @@ void kvm_arch_disable_virtualization_cpu(void)
 	csr_write(CSR_HIDELEG, 0);
 }
 
+static void kvm_riscv_teardown(void)
+{
+	kvm_riscv_aia_exit();
+	kvm_unregister_perf_callbacks();
+}
+
 static int __init riscv_kvm_init(void)
 {
 	int rc;
@@ -105,9 +111,11 @@ static int __init riscv_kvm_init(void)
 		kvm_info("AIA available with %d guest external interrupts\n",
 			 kvm_riscv_aia_nr_hgei);
 
+	kvm_register_perf_callbacks(NULL);
+
 	rc = kvm_init(sizeof(struct kvm_vcpu), 0, THIS_MODULE);
 	if (rc) {
-		kvm_riscv_aia_exit();
+		kvm_riscv_teardown();
 		return rc;
 	}
 
@@ -117,7 +125,7 @@ module_init(riscv_kvm_init);
 
 static void __exit riscv_kvm_exit(void)
 {
-	kvm_riscv_aia_exit();
+	kvm_riscv_teardown();
 
 	kvm_exit();
 }
arch/riscv/kvm/vcpu.c +7 -0
index 8d7d381737ee..e8ffb3456898 100644
@@ -226,6 +226,13 @@ bool kvm_arch_vcpu_in_kernel(struct kvm_vcpu *vcpu)
 	return (vcpu->arch.guest_context.sstatus & SR_SPP) ? true : false;
 }
 
+#ifdef CONFIG_GUEST_PERF_EVENTS
+unsigned long kvm_arch_vcpu_get_ip(struct kvm_vcpu *vcpu)
+{
+	return vcpu->arch.guest_context.sepc;
+}
+#endif
+
 vm_fault_t kvm_arch_vcpu_fault(struct kvm_vcpu *vcpu, struct vm_fault *vmf)
 {
 	return VM_FAULT_SIGBUS;
Signature · 1 line
2.34.1
#5 lore raw
In reply to [email protected]
This series was applied to riscv/linux.git (fixes) by Anup Patel <[email protected]>:

Hello:

This series was applied to riscv/linux.git (fixes) by Anup Patel <[email protected]>:

Quoted 8 lines · On Tue, 15 Oct 2024 16:42:18 +0800 you wrote:

From: Quan Zhou <[email protected]>

Add basic guest support to RISC-V perf, enabling it to distinguish whether PMU interrupts occur in the host or the guest, and then collect some basic guest information from the host side (guest os callchain is not supported for now).

[...]

Here is the summary with links:
- [v5,1/2] riscv: perf: add guest vs host distinction

    https://git.kernel.org/riscv/c/5bb5ccb3e8d8
  - [v5,2/2] riscv: KVM: add basic support for host vs guest profiling
    https://git.kernel.org/riscv/c/eded6754f398

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html