KVM nVHE Hypervisor stack unwinder
v3 cover KVM nVHE Hypervisor stack unwinder
Hi all,
This v3 of the nVHE hypervisor stack unwinder. The previous versions were posted at:
v2: https://lore.kernel.org/r/[email protected]/
v1: https://lore.kernel.org/r/[email protected]/
The version is rebased on 5.19-rc1 which resolves the previously unmerged dependencies, and adds some reviewed tags from Mark Brown.
The cover letter are copied below for convenience.
Thanks,
Kalesh
-------
This new version of the unwinder splits the unwinding and dumping of the stack between the hypervisor and host:
- The hypervisor unwinds its stack and dumps the address entries into a page shared with the host.
- The host then symnolizes and prints the hyp stacktrace from the shared page.
The new approach doesn't depend on CONFIG_NVHE_EL2_DEBUG, and allows dumping hyp stacktraces in prodcution environments (!CONFIG_NVHE_EL2_DEBUG).
arm64/kernel/stacktrace.c is compiled twice: stacktrace.o for the host kernel and stacktrace.nvhe.o for the hypervisor. This allows reusing most of the host unwinding logic in the nVHE hypervisor.
Kalesh Singh (5):
KVM: arm64: Factor out common stack unwinding logic
KVM: arm64: Compile stacktrace.nvhe.o
KVM: arm64: Add hypervisor overflow stack
KVM: arm64: Allocate shared stacktrace pages
KVM: arm64: Unwind and dump nVHE hypervisor stacktrace
arch/arm64/include/asm/kvm_asm.h | 1 + arch/arm64/include/asm/stacktrace.h | 58 +++++++++-- arch/arm64/kernel/stacktrace.c | 151 +++++++++++++++++++++++----- arch/arm64/kvm/arm.c | 34 +++++++ arch/arm64/kvm/handle_exit.c | 4 + arch/arm64/kvm/hyp/nvhe/Makefile | 3 +- arch/arm64/kvm/hyp/nvhe/host.S | 9 +- arch/arm64/kvm/hyp/nvhe/setup.c | 11 ++ arch/arm64/kvm/hyp/nvhe/switch.c | 4 + 9 files changed, 231 insertions(+), 44 deletions(-)
base-commit: f2906aa863381afb0015a9eb7fefad885d4e5a56
Signature · 6 lines
2.36.1.255.ge46751e96f-goog _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v3 1/5 KVM: arm64: Factor out common stack unwinding logic
Factor out the stack unwinding logic common to both the host kernel and the nVHE hypersivor into __unwind_next(). This allows for reuse in the nVHE hypervisor stack unwinding (later in this series).
---
Changes in v3:
- Add Mark's Reviewed-by tag
arch/arm64/kernel/stacktrace.c | 36 +++++++++++++++++++++++----------- 1 file changed, 25 insertions(+), 11 deletions(-)
arch/arm64/kernel/stacktrace.c +25 -11
@@ -81,23 +81,19 @@ NOKPROBE_SYMBOL(unwind_init);* records (e.g. a cycle), determined based on the location and fp value of A * and the location (but not the fp value) of B. */ -static int notrace unwind_next(struct task_struct *tsk, -struct unwind_state *state) +static int notrace __unwind_next(struct task_struct *tsk, +struct unwind_state *state, +struct stack_info *info) {unsigned long fp = state->fp; -struct stack_info info; - -/* Final frame; nothing to unwind */ -if (fp == (unsigned long )task_pt_regs (tsk)->stackframe) -return -ENOENT ;if (fp &0x7 )return -EINVAL ; -if (!on_accessible_stack (tsk, fp,16 , &info)) +if (!on_accessible_stack (tsk, fp,16 , info))return -EINVAL ; -if (test_bit (info.type, state->stacks_done)) +if (test_bit (info->type, state->stacks_done))return -EINVAL ;/*
@@ -113,7 +109,7 @@ static int notrace unwind_next(struct task_struct *tsk,* stack to another, it's never valid to unwind back to that first * stack. */ -if (info.type == state->prev_type) { +if (info->type == state->prev_type) {if (fp <= state->prev_fp)return -EINVAL ; }else {
@@ -127,7 +123,25 @@ static int notrace unwind_next(struct task_struct *tsk, state->fp =READ_ONCE_NOCHECK (*(unsigned long *)(fp)); state->pc =READ_ONCE_NOCHECK (*(unsigned long *)(fp +8 )); state->prev_fp = fp; - state->prev_type = info.type; + state->prev_type = info->type; + +return 0 ; +} +NOKPROBE_SYMBOL (__unwind_next); + +static int notrace unwind_next(struct task_struct *tsk, +struct unwind_state *state) +{ +struct stack_info info; +int err; + +/* Final frame; nothing to unwind */ +if (state->fp == (unsigned long )task_pt_regs (tsk)->stackframe) +return -ENOENT ; + + err =__unwind_next (tsk, state, &info); +if (err) +return err; state->pc =ptrauth_strip_insn_pac (state->pc);
Signature · 6 lines
2.36.1.255.ge46751e96f-goog _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v3 2/5 KVM: arm64: Compile stacktrace.nvhe.o
In reply to Kalesh SinghRecompile stack unwinding code for use with the nVHE hypervisor. This is a preparatory patch that will allow reusing most of the kernel unwinding logic in the nVHE hypervisor.
---
Changes in v3:
- Add Mark's Reviewed-by tag
Changes in v2:
- Split out refactoring of common unwinding logic into a separate patch, per Mark Brown
arch/arm64/include/asm/stacktrace.h | 18 +++++++++----- arch/arm64/kernel/stacktrace.c | 37 ++++++++++++++++------------- arch/arm64/kvm/hyp/nvhe/Makefile | 3 ++- 3 files changed, 35 insertions(+), 23 deletions(-)
arch/arm64/include/asm/stacktrace.h +12 -6
@@ -16,12 +16,14 @@#include <asm/sdei.h> enum stack_type { -STACK_TYPE_UNKNOWN , +#ifndef __KVM_NVHE_HYPERVISOR__STACK_TYPE_TASK ,STACK_TYPE_IRQ ,STACK_TYPE_OVERFLOW ,STACK_TYPE_SDEI_NORMAL ,STACK_TYPE_SDEI_CRITICAL , +#endif /* !__KVM_NVHE_HYPERVISOR__ */ +STACK_TYPE_UNKNOWN , __NR_STACK_TYPES };
@@ -31,11 +33,6 @@ struct stack_info {enum stack_type type; }; -extern void dump_backtrace (struct pt_regs *regs,struct task_struct *tsk, -const char *loglvl); - -DECLARE_PER_CPU(unsigned long *, irq_stack_ptr); -static inline bool on_stack (unsigned long sp,unsigned long size,unsigned long low,unsigned long high,enum stack_type type,struct stack_info *info)
@@ -54,6 +51,12 @@ static inline bool on_stack(unsigned long sp, unsigned long size,return true; } +#ifndef __KVM_NVHE_HYPERVISOR__ +extern void dump_backtrace (struct pt_regs *regs,struct task_struct *tsk, +const char *loglvl); + +DECLARE_PER_CPU(unsigned long *, irq_stack_ptr); +static inline bool on_irq_stack (unsigned long sp,unsigned long size,struct stack_info *info) {
@@ -88,6 +91,7 @@ static inline bool on_overflow_stack(unsigned long sp, unsigned long size,static inline bool on_overflow_stack (unsigned long sp,unsigned long size,struct stack_info *info) {return false; }#endif +#endif /* !__KVM_NVHE_HYPERVISOR__ */ /*
@@ -101,6 +105,7 @@ static inline bool on_accessible_stack(const struct task_struct *tsk,if (info) info->type =STACK_TYPE_UNKNOWN ; +#ifndef __KVM_NVHE_HYPERVISOR__if (on_task_stack (tsk, sp, size, info))return true;if (tsk != current ||preemptible ())
@@ -111,6 +116,7 @@ static inline bool on_accessible_stack(const struct task_struct *tsk,return true;if (on_sdei_stack (sp, size, info))return true; +#endif /* !__KVM_NVHE_HYPERVISOR__ */ return false; }
arch/arm64/kernel/stacktrace.c +21 -16
@@ -129,6 +129,26 @@ static int notrace __unwind_next(struct task_struct *tsk, }NOKPROBE_SYMBOL (__unwind_next); +static int notrace unwind_next(struct task_struct *tsk, +struct unwind_state *state); + +static void notrace unwind(struct task_struct *tsk, +struct unwind_state *state, +stack_trace_consume_fn consume_entry,void *cookie) +{ +while (1 ) { +int ret; + +if (!consume_entry (cookie, state->pc)) +break ; + ret =unwind_next (tsk, state); +if (ret <0 ) +break ; + } +} +NOKPROBE_SYMBOL (unwind); + +#ifndef __KVM_NVHE_HYPERVISOR__ static int notrace unwind_next(struct task_struct *tsk,struct unwind_state *state) {
@@ -171,22 +191,6 @@ static int notrace unwind_next(struct task_struct *tsk, }NOKPROBE_SYMBOL (unwind_next); -static void notrace unwind(struct task_struct *tsk, -struct unwind_state *state, -stack_trace_consume_fn consume_entry,void *cookie) -{ -while (1 ) { -int ret; - -if (!consume_entry (cookie, state->pc)) -break ; - ret =unwind_next (tsk, state); -if (ret <0 ) -break ; - } -} -NOKPROBE_SYMBOL (unwind); -static bool dump_backtrace_entry (void *arg,unsigned long where) {char *loglvl = arg;
@@ -238,3 +242,4 @@ noinline notrace void arch_stack_walk(stack_trace_consume_fn consume_entry, unwind(task, &state, consume_entry, cookie); } +#endif /* !__KVM_NVHE_HYPERVISOR__ */
arch/arm64/kvm/hyp/nvhe/Makefile +2 -1
@@ -14,7 +14,8 @@ lib-objs := $(addprefix ../../../lib/, $(lib-objs)) obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o \ hyp-main.o hyp-smp.o psci-relay.o early_alloc.o page_alloc.o \ - cache.o setup.o mm.o mem_protect.o sys_regs.o pkvm.o + cache.o setup.o mm.o mem_protect.o sys_regs.o pkvm.o \ + ../../../kernel/stacktrace.o obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \ ../fpsimd.o ../hyp-entry.o ../exception.o ../pgtable.o obj-$(CONFIG_DEBUG_LIST) += list_debug.o
Signature · 6 lines
2.36.1.255.ge46751e96f-goog _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerRe: [PATCH v3 2/5] KVM: arm64: Compile stacktrace.nvhe.o
Quoted 29 lines · Kalesh Singh
Recompile stack unwinding code for use with the nVHE hypervisor. This is a preparatory patch that will allow reusing most of the kernel unwinding logic in the nVHE hypervisor.
Suggested-by Mark Rutland <[email protected]>Signed-off-by Kalesh Singh <[email protected]>Reviewed-by Mark Brown <[email protected]>---
Changes in v3:
- Add Mark's Reviewed-by tagChanges in v2:
- Split out refactoring of common unwinding logic into a separate patch, per Mark Brownarch/arm64/include/asm/stacktrace.h | 18 +++++++++----- arch/arm64/kernel/stacktrace.c | 37 ++++++++++++++++------------- arch/arm64/kvm/hyp/nvhe/Makefile | 3 ++- 3 files changed, 35 insertions(+), 23 deletions(-)arch/arm64/include/asm/stacktrace.h +0
-0@@ -16,12 +16,14 @@#include <asm/sdei.h>
arch/arm64/include/asm/stacktrace.h line 26
Quoted 10 lines
arch/arm64/include/asm/stacktrace.h +3
-1enum stack_type { -STACK_TYPE_UNKNOWN , +#ifndef __KVM_NVHE_HYPERVISOR__STACK_TYPE_TASK ,STACK_TYPE_IRQ ,STACK_TYPE_OVERFLOW ,STACK_TYPE_SDEI_NORMAL ,STACK_TYPE_SDEI_CRITICAL , +#endif /* !__KVM_NVHE_HYPERVISOR__ */ +STACK_TYPE_UNKNOWN ,
What is the reason for this reordering? I have the sinking feeling that this could play badly with the logic that assumes that it is legal to switch from a lesser stack type to a higher one, and could allow switching to a duff stack.
I would at least like to see a justification of why this isn't less safe than the current code.
[...]
arch/arm64/kvm/hyp/nvhe/Makefile lib-objs := $(addprefix ../../../lib/, $(lib-objs)) · line 18
Quoted 10 lines
index
f9fe4dc21b1f..c0ff0d6fc403100644arch/arm64/kvm/hyp/nvhe/Makefile +2
-1@@ -14,7 +14,8 @@ lib-objs := $(addprefix ../../../lib/, $(lib-objs)) obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o \ hyp-main.o hyp-smp.o psci-relay.o early_alloc.o page_alloc.o \ - cache.o setup.o mm.o mem_protect.o sys_regs.o pkvm.o + cache.o setup.o mm.o mem_protect.o sys_regs.o pkvm.o \ + ../../../kernel/stacktrace.o
This, I positively hate. It is only a marginally better than the cross-arch references we used to have with arch/arm/kvm. I'd be much more happy with an include file containing the shared code. It would also allow the removal of some of the #ifdeferry. Note that this is the approach that we ended up adopting for the VHE/nVHE split.
Thanks,
M.
Signature · 5 lines
Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v3 2/5] KVM: arm64: Compile stacktrace.nvhe.o
Quoted 32 lines · Marc Zyngier
Quoted 29 lines · Kalesh Singh
Recompile stack unwinding code for use with the nVHE hypervisor. This is a preparatory patch that will allow reusing most of the kernel unwinding logic in the nVHE hypervisor.
Suggested-by Mark Rutland <[email protected]>Signed-off-by Kalesh Singh <[email protected]>Reviewed-by Mark Brown <[email protected]>---
Changes in v3:
- Add Mark's Reviewed-by tagChanges in v2:
- Split out refactoring of common unwinding logic into a separate patch, per Mark Brownarch/arm64/include/asm/stacktrace.h | 18 +++++++++----- arch/arm64/kernel/stacktrace.c | 37 ++++++++++++++++------------- arch/arm64/kvm/hyp/nvhe/Makefile | 3 ++- 3 files changed, 35 insertions(+), 23 deletions(-)arch/arm64/include/asm/stacktrace.h +0
-0@@ -16,12 +16,14 @@#include <asm/sdei.h>
Quoted 15 lines
arch/arm64/include/asm/stacktrace.h line 26
Quoted 10 lines
arch/arm64/include/asm/stacktrace.h +3
-1enum stack_type { -STACK_TYPE_UNKNOWN , +#ifndef __KVM_NVHE_HYPERVISOR__STACK_TYPE_TASK ,STACK_TYPE_IRQ ,STACK_TYPE_OVERFLOW ,STACK_TYPE_SDEI_NORMAL ,STACK_TYPE_SDEI_CRITICAL , +#endif /* !__KVM_NVHE_HYPERVISOR__ */ +STACK_TYPE_UNKNOWN ,What is the reason for this reordering? I have the sinking feeling that this could play badly with the logic that assumes that it is legal to switch from a lesser stack type to a higher one, and could allow switching to a duff stack.
HI Marc. Thanks for reviewing.
I only reordered the enum to group the common types. But I don't have a strong opinion on it. The unwinding doesn't depend on the ordering in this enum. When we transition form stack 'A'-->'B', we set the stack_done bit for stack A so that we never transition back to 'A', as it's not valid to transition back to a previous stack. But the order of the sequence itself is not something enforced.
Quoted 11 lines
I would at least like to see a justification of why this isn't less safe than the current code.
[...]
Quoted 5 lines
index
f9fe4dc21b1f..c0ff0d6fc403100644arch/arm64/kvm/hyp/nvhe/Makefile +0
-0@@ -14,7 +14,8 @@ lib-objs := $(addprefix ../../../lib/, $(lib-objs))
Quoted 12 lines
arch/arm64/kvm/hyp/nvhe/Makefile lib-objs := $(addprefix ../../../lib/, $(lib-objs)) · line 18
Quoted 5 lines
arch/arm64/kvm/hyp/nvhe/Makefile +2
-1obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o \ hyp-main.o hyp-smp.o psci-relay.o early_alloc.o page_alloc.o \ - cache.o setup.o mm.o mem_protect.o sys_regs.o pkvm.o + cache.o setup.o mm.o mem_protect.o sys_regs.o pkvm.o \ + ../../../kernel/stacktrace.oThis, I positively hate. It is only a marginally better than the cross-arch references we used to have with
arch/arm/kvm. I'd be much more happy with an include file containing the shared code. It would also allow the removal of some of the #ifdeferry. Note that this is the approach that we ended up adopting for the VHE/nVHE split.
Also thought about moving stuff to some header file, but I thought this might be less intrusive. Let me prototype to see how they compare.
Thanks,
Kalesh
Quoted 10 lines
Thanks,
M.--
Without deviation from the norm, progress is not possible.--
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v3 3/5 KVM: arm64: Add hypervisor overflow stack
In reply to Kalesh SinghAllocate and switch to 16-byte aligned secondary stack on overflow. This provides us stack space to better handle overflows; and is used in a subsequent patch to dump the hypervisor stacktrace.
--- arch/arm64/kernel/stacktrace.c | 3 +++ arch/arm64/kvm/hyp/nvhe/host.S | 9 ++------- 2 files changed, 5 insertions(+), 7 deletions(-)
arch/arm64/kernel/stacktrace.c +3 -0
@@ -242,4 +242,7 @@ noinline notrace void arch_stack_walk(stack_trace_consume_fn consume_entry,unwind (task, &state, consume_entry, cookie); } +#else /* __KVM_NVHE_HYPERVISOR__ */ +DEFINE_PER_CPU(unsigned long [PAGE_SIZE/sizeof(long)], overflow_stack) +__aligned(16) ;#endif /* !__KVM_NVHE_HYPERVISOR__ */
arch/arm64/kvm/hyp/nvhe/host.S +2 -7
@@ -177,13 +177,8 @@ SYM_FUNC_END(__host_hvc) b hyp_panic .L__hyp_sp_overflow\@: - /* - * Reset SP to the top of the stack, to allow handling the hyp_panic. - * This corrupts the stack but is ok, since we won't be attempting - * any unwinding here. - */ - ldr_this_cpu x0, kvm_init_params + NVHE_INIT_STACK_HYP_VA, x1 - mov sp, x0 + /* Switch to the overflow stack */ + adr_this_cpu sp, overflow_stack + PAGE_SIZE, x0 b hyp_panic_bad_stack ASM_BUG()
Signature · 6 lines
2.36.1.255.ge46751e96f-goog _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerRe: [PATCH v3 3/5] KVM: arm64: Add hypervisor overflow stack
Quoted 16 lines · Kalesh Singh
Allocate and switch to 16-byte aligned secondary stack on overflow. This provides us stack space to better handle overflows; and is used in a subsequent patch to dump the hypervisor stacktrace.
Signed-off-by Kalesh Singh <[email protected]>--- arch/arm64/kernel/stacktrace.c | 3 +++ arch/arm64/kvm/hyp/nvhe/host.S | 9 ++------- 2 files changed, 5 insertions(+), 7 deletions(-)arch/arm64/kernel/stacktrace.c +0
-0
arch/arm64/kernel/stacktrace.c noinline notrace void arch_stack_walk(stack_trace_consume_fn consume_entry, · line 247
Quoted 6 lines
arch/arm64/kernel/stacktrace.c +3
-0@@ -242,4 +242,7 @@ noinline notrace void arch_stack_walk(stack_trace_consume_fn consume_entry,unwind (task, &state, consume_entry, cookie); } +#else /* __KVM_NVHE_HYPERVISOR__ */ +DEFINE_PER_CPU(unsigned long [PAGE_SIZE/sizeof(long)], overflow_stack) +__aligned(16) ;
Does this need to be a whole page? With 64kB pages, this is potentially a lot of memory for something that will hardly ever be used. The rest of the kernel limits this to 4kB, which seems more reasonable. There is no guard page anyway, so PAGE_SIZE doesn't provide any extra protection.
Quoted 25 lines
arch/arm64/kernel/stacktrace.c +0
-0#endif /* !__KVM_NVHE_HYPERVISOR__ */ arch/arm64/kvm/hyp/nvhe/host.S +2
-7@@ -177,13 +177,8 @@ SYM_FUNC_END(__host_hvc) b hyp_panic .L__hyp_sp_overflow\@: - /* - * Reset SP to the top of the stack, to allow handling the hyp_panic. - * This corrupts the stack but is ok, since we won't be attempting - * any unwinding here. - */ - ldr_this_cpu x0, kvm_init_params + NVHE_INIT_STACK_HYP_VA, x1 - mov sp, x0 + /* Switch to the overflow stack */ + adr_this_cpu sp, overflow_stack + PAGE_SIZE, x0 b hyp_panic_bad_stack ASM_BUG()Signature · 1 line
2.36.1.255.ge46751e96f-goog
Thanks,
M.
Signature · 5 lines
Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v3 3/5] KVM: arm64: Add hypervisor overflow stack
Quoted 20 lines · Marc Zyngier
Quoted 17 lines · Kalesh Singh
Allocate and switch to 16-byte aligned secondary stack on overflow. This provides us stack space to better handle overflows; and is used in a subsequent patch to dump the hypervisor stacktrace.
Signed-off-by Kalesh Singh <[email protected]>--- arch/arm64/kernel/stacktrace.c | 3 +++ arch/arm64/kvm/hyp/nvhe/host.S | 9 ++------- 2 files changed, 5 insertions(+), 7 deletions(-)arch/arm64/kernel/stacktrace.c +0
-0@@ -242,4 +242,7 @@ noinline notrace void arch_stack_walk(stack_trace_consume_fn consume_entry,
Quoted 11 lines
arch/arm64/kernel/stacktrace.c noinline notrace void arch_stack_walk(stack_trace_consume_fn consume_entry, · line 247
Quoted 5 lines
arch/arm64/kernel/stacktrace.c +3
-0unwind (task, &state, consume_entry, cookie); } +#else /* __KVM_NVHE_HYPERVISOR__ */ +DEFINE_PER_CPU(unsigned long [PAGE_SIZE/sizeof(long)], overflow_stack) +__aligned(16) ;Does this need to be a whole page? With 64kB pages, this is potentially a lot of memory for something that will hardly ever be used. The rest of the kernel limits this to 4kB, which seems more reasonable. There is no guard page anyway, so
PAGE_SIZEdoesn't provide any extra protection.
My oversight on the !4kB page sizes. I think this could be as small as:
(STACK_SIZE - 1) / 2 + sizeof(long)
'/ 2' : Min frame size (x29, x30)
'+ sizeof(long)' : To round up
since we only save the one address (PC) for each frame. WDYT?
Thanks,
Kalesh
Quoted 29 lines
Quoted 23 lines
arch/arm64/kernel/stacktrace.c +0
-0#endif /* !__KVM_NVHE_HYPERVISOR__ */ arch/arm64/kvm/hyp/nvhe/host.S +2
-8@@ -177,13 +177,8 @@ SYM_FUNC_END(__host_hvc) b hyp_panic .L__hyp_sp_overflow\@: - /* - * Reset SP to the top of the stack, to allow handling the hyp_panic. - * This corrupts the stack but is ok, since we won't be attempting - * any unwinding here. - */ - ldr_this_cpu x0, kvm_init_params + NVHE_INIT_STACK_HYP_VA, x1 - mov sp, x0 + /* Switch to the overflow stack */ + adr_this_cpu sp, overflow_stack + PAGE_SIZE, x0 b hyp_panic_bad_stack ASM_BUG() --Quoted 3 lines
2.36.1.255.ge46751e96f-goog
Thanks,
Quoted 4 lines
M.--
Without deviation from the norm, progress is not possible.
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v3 4/5 KVM: arm64: Allocate shared stacktrace pages
In reply to Kalesh SinghThe nVHE hypervisor can use this shared area to dump its stacktrace addresses on hyp_panic(). Symbolization and printing the stacktrace can then be handled by the host in EL1 (done in a later patch in this series).
--- arch/arm64/include/asm/kvm_asm.h | 1 + arch/arm64/kvm/arm.c | 34 ++++++++++++++++++++++++++++++++ arch/arm64/kvm/hyp/nvhe/setup.c | 11 +++++++++++ 3 files changed, 46 insertions(+)
arch/arm64/include/asm/kvm_asm.h +1 -0
@@ -174,6 +174,7 @@ struct kvm_nvhe_init_params {unsigned long hcr_el2;unsigned long vttbr;unsigned long vtcr; +unsigned long stacktrace_hyp_va; };/* Translate a kernel address @ptr into its equivalent linear mapping */
arch/arm64/kvm/arm.c +34 -0
@@ -50,6 +50,7 @@ DEFINE_STATIC_KEY_FALSE(kvm_protected_mode_initialized); DECLARE_KVM_HYP_PER_CPU(unsigned long, kvm_hyp_vector); static DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_page); +DEFINE_PER_CPU (unsigned long , kvm_arm_hyp_stacktrace_page);unsigned long kvm_arm_hyp_percpu_base[NR_CPUS ]; DECLARE_KVM_NVHE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params);
@@ -1554,6 +1555,7 @@ static void cpu_prepare_hyp_mode(int cpu) tcr |= (idmap_t0sz &GENMASK (TCR_TxSZ_WIDTH -1 ,0 )) <<TCR_T0SZ_OFFSET ; params->tcr_el2 = tcr; + params->stacktrace_hyp_va =kern_hyp_va (per_cpu (kvm_arm_hyp_stacktrace_page, cpu)); params->pgd_pa =kvm_mmu_get_httbr ();if (is_protected_kvm_enabled ()) params->hcr_el2 =HCR_HOST_NVHE_PROTECTED_FLAGS ;
@@ -1845,6 +1847,7 @@ static void teardown_hyp_mode(void)free_hyp_pgds ();for_each_possible_cpu (cpu) {free_page (per_cpu (kvm_arm_hyp_stack_page, cpu)); +free_page (per_cpu (kvm_arm_hyp_stacktrace_page, cpu));free_pages (kvm_arm_hyp_percpu_base[cpu],nvhe_percpu_order ()); } }
@@ -1936,6 +1939,23 @@ static int init_hyp_mode(void)per_cpu (kvm_arm_hyp_stack_page, cpu) = stack_page; } +/* +* Allocate stacktrace pages for Hypervisor-mode. +* This is used by the hypervisor to share its stacktrace +* with the host on a hyp_panic(). +*/ +for_each_possible_cpu (cpu) { +unsigned long stacktrace_page; + + stacktrace_page =__get_free_page (GFP_KERNEL ); +if (!stacktrace_page) { + err = -ENOMEM ; + goto out_err; + } + +per_cpu (kvm_arm_hyp_stacktrace_page, cpu) = stacktrace_page; + } +/* * Allocate and initialize pages for Hypervisor-mode percpu regions. */
@@ -2043,6 +2063,20 @@ static int init_hyp_mode(void) params->stack_hyp_va = hyp_addr + (2 *PAGE_SIZE ); } +/* +* Map the hyp stacktrace pages. +*/ +for_each_possible_cpu (cpu) { +char *stacktrace_page = (char *)per_cpu (kvm_arm_hyp_stacktrace_page, cpu); + + err =create_hyp_mappings (stacktrace_page, stacktrace_page +PAGE_SIZE , +PAGE_HYP ); +if (err) { +kvm_err ("Cannot map hyp stacktrace page\n" ); + goto out_err; + } + } +for_each_possible_cpu (cpu) {char *percpu_begin = (char *)kvm_arm_hyp_percpu_base[cpu];char *percpu_end = percpu_begin +nvhe_percpu_size ();
arch/arm64/kvm/hyp/nvhe/setup.c +11 -0
@@ -135,6 +135,17 @@ static int recreate_hyp_mappings(phys_addr_t phys, unsigned long size,/* Update stack_hyp_va to end of the stack's private VA range */ params->stack_hyp_va = hyp_addr + (2 *PAGE_SIZE ); + +/* +* Map the stacktrace pages as shared and transfer ownership to +* the hypervisor. +*/ + prot =pkvm_mkstate (PAGE_HYP ,PKVM_PAGE_SHARED_OWNED ); + start = (void *)params->stacktrace_hyp_va; + end = start +PAGE_SIZE ; + ret =pkvm_create_mappings (start, end, prot); +if (ret) +return ret; }/*
Signature · 6 lines
2.36.1.255.ge46751e96f-goog _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerRe: [PATCH v3 4/5] KVM: arm64: Allocate shared stacktrace pages
Quoted 29 lines · Kalesh Singh
The nVHE hypervisor can use this shared area to dump its stacktrace addresses on
hyp_panic(). Symbolization and printing the stacktrace can then be handled by the host in EL1 (done in a later patch in this series).Signed-off-by Kalesh Singh <[email protected]>--- arch/arm64/include/asm/kvm_asm.h | 1 + arch/arm64/kvm/arm.c | 34 ++++++++++++++++++++++++++++++++ arch/arm64/kvm/hyp/nvhe/setup.c | 11 +++++++++++ 3 files changed, 46 insertions(+)arch/arm64/include/asm/kvm_asm.h +1
-0@@ -174,6 +174,7 @@ struct kvm_nvhe_init_params {unsigned long hcr_el2;unsigned long vttbr;unsigned long vtcr; +unsigned long stacktrace_hyp_va; };/* Translate a kernel address @ptr into its equivalent linear mapping */ arch/arm64/kvm/arm.c +0
-0
arch/arm64/kvm/arm.c DEFINE_STATIC_KEY_FALSE(kvm_protected_mode_initialized); · line 53
Quoted 4 lines
arch/arm64/kvm/arm.c +1
-0@@ -50,6 +50,7 @@ DEFINE_STATIC_KEY_FALSE(kvm_protected_mode_initialized); DECLARE_KVM_HYP_PER_CPU(unsigned long, kvm_hyp_vector); static DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_page); +DEFINE_PER_CPU (unsigned long , kvm_arm_hyp_stacktrace_page);
Why isn't this static, since the address is passed via the kvm_nvhe_init_params block?
Quoted 28 lines
arch/arm64/kvm/arm.c +7
-0unsigned long kvm_arm_hyp_percpu_base[NR_CPUS ]; DECLARE_KVM_NVHE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params);@@ -1554,6 +1555,7 @@ static void cpu_prepare_hyp_mode(int cpu) tcr |= (idmap_t0sz &GENMASK (TCR_TxSZ_WIDTH -1 ,0 )) <<TCR_T0SZ_OFFSET ; params->tcr_el2 = tcr; + params->stacktrace_hyp_va =kern_hyp_va (per_cpu (kvm_arm_hyp_stacktrace_page, cpu)); params->pgd_pa =kvm_mmu_get_httbr ();if (is_protected_kvm_enabled ()) params->hcr_el2 =HCR_HOST_NVHE_PROTECTED_FLAGS ;@@ -1845,6 +1847,7 @@ static void teardown_hyp_mode(void)free_hyp_pgds ();for_each_possible_cpu (cpu) {free_page (per_cpu (kvm_arm_hyp_stack_page, cpu)); +free_page (per_cpu (kvm_arm_hyp_stacktrace_page, cpu));free_pages (kvm_arm_hyp_percpu_base[cpu],nvhe_percpu_order ()); } }@@ -1936,6 +1939,23 @@ static int init_hyp_mode(void)per_cpu (kvm_arm_hyp_stack_page, cpu) = stack_page; } +/* +* Allocate stacktrace pages for Hypervisor-mode. +* This is used by the hypervisor to share its stacktrace +* with the host on a hyp_panic(). +*/
arch/arm64/kvm/arm.c static int init_hyp_mode(void) · line 1956
Quoted 10 lines
arch/arm64/kvm/arm.c +10
-0+for_each_possible_cpu (cpu) { +unsigned long stacktrace_page; + + stacktrace_page =__get_free_page (GFP_KERNEL ); +if (!stacktrace_page) { + err = -ENOMEM ; + goto out_err; + } + +per_cpu (kvm_arm_hyp_stacktrace_page, cpu) = stacktrace_page;
I have the same feeling as with the overflow stack. This is potentially a huge amount of memory: on my test box, with 64k pages, this is a whole 10MB that I give away for something that is only a debug facility.
Can this somehow be limited? I don't see it being less than a page as a problem, as the memory is always shared back with EL1 in the case of pKVM (and normal KVM doesn't have that problem anyway).
Alternatively, this should be restricted to pKVM. With normal nVHE, the host should be able to parse the EL2 stack directly with some offsetting. Actually, this is probably the best option.
Thanks,
M.
Signature · 5 lines
Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v3 4/5] KVM: arm64: Allocate shared stacktrace pages
Quoted 34 lines · Marc Zyngier
Quoted 31 lines · Kalesh Singh
The nVHE hypervisor can use this shared area to dump its stacktrace addresses on
hyp_panic(). Symbolization and printing the stacktrace can then be handled by the host in EL1 (done in a later patch in this series).Signed-off-by Kalesh Singh <[email protected]>--- arch/arm64/include/asm/kvm_asm.h | 1 + arch/arm64/kvm/arm.c | 34 ++++++++++++++++++++++++++++++++ arch/arm64/kvm/hyp/nvhe/setup.c | 11 +++++++++++ 3 files changed, 46 insertions(+)arch/arm64/include/asm/kvm_asm.h +1
-0@@ -174,6 +174,7 @@ struct kvm_nvhe_init_params {unsigned long hcr_el2;unsigned long vttbr;unsigned long vtcr; +unsigned long stacktrace_hyp_va; };/* Translate a kernel address @ptr into its equivalent linear mapping */ arch/arm64/kvm/arm.c +0
-0@@ -50,6 +50,7 @@ DEFINE_STATIC_KEY_FALSE(kvm_protected_mode_initialized); DECLARE_KVM_HYP_PER_CPU(unsigned long, kvm_hyp_vector);
Quoted 5 lines
arch/arm64/kvm/arm.c DEFINE_STATIC_KEY_FALSE(kvm_protected_mode_initialized); · line 53
Quoted 2 lines
arch/arm64/kvm/arm.c +1
-0static DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_page); +DEFINE_PER_CPU (unsigned long , kvm_arm_hyp_stacktrace_page);Why isn't this static, since the address is passed via the
kvm_nvhe_init_paramsblock?
In the subsequent patch the host will need to read and symbolize the addresses that the hypervisor dumped, it's done in stacktrace.c
Quoted 45 lines
Quoted 29 lines
arch/arm64/kvm/arm.c +7
-0unsigned long kvm_arm_hyp_percpu_base[NR_CPUS ]; DECLARE_KVM_NVHE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params);@@ -1554,6 +1555,7 @@ static void cpu_prepare_hyp_mode(int cpu) tcr |= (idmap_t0sz &GENMASK (TCR_TxSZ_WIDTH -1 ,0 )) <<TCR_T0SZ_OFFSET ; params->tcr_el2 = tcr; + params->stacktrace_hyp_va =kern_hyp_va (per_cpu (kvm_arm_hyp_stacktrace_page, cpu)); params->pgd_pa =kvm_mmu_get_httbr ();if (is_protected_kvm_enabled ()) params->hcr_el2 =HCR_HOST_NVHE_PROTECTED_FLAGS ;@@ -1845,6 +1847,7 @@ static void teardown_hyp_mode(void)free_hyp_pgds ();for_each_possible_cpu (cpu) {free_page (per_cpu (kvm_arm_hyp_stack_page, cpu)); +free_page (per_cpu (kvm_arm_hyp_stacktrace_page, cpu));free_pages (kvm_arm_hyp_percpu_base[cpu],nvhe_percpu_order ()); } }@@ -1936,6 +1939,23 @@ static int init_hyp_mode(void)per_cpu (kvm_arm_hyp_stack_page, cpu) = stack_page; } +/* +* Allocate stacktrace pages for Hypervisor-mode. +* This is used by the hypervisor to share its stacktrace +* with the host on a hyp_panic(). +*/ arch/arm64/kvm/arm.c static int init_hyp_mode(void) · line 1956
Quoted 10 lines
arch/arm64/kvm/arm.c +10
-0+for_each_possible_cpu (cpu) { +unsigned long stacktrace_page; + + stacktrace_page =__get_free_page (GFP_KERNEL ); +if (!stacktrace_page) { + err = -ENOMEM ; + goto out_err; + } + +per_cpu (kvm_arm_hyp_stacktrace_page, cpu) = stacktrace_page;I have the same feeling as with the overflow stack. This is potentially a huge amount of memory: on my test box, with 64k pages, this is a whole 10MB that I give away for something that is only a debug facility.
Quoted 7 lines
Can this somehow be limited? I don't see it being less than a page as a problem, as the memory is always shared back with EL1 in the case of pKVM (and normal KVM doesn't have that problem anyway).
Alternatively, this should be restricted to pKVM. With normal nVHE, the host should be able to parse the EL2 stack directly with some offsetting. Actually, this is probably the best option.
In a previous approach we had something similar to what you propose (unwinding in el1) [1]. But doesn't work for our production use case with protected mode since it disabled the stage 2 protection. We could do the split and take a different approach for the protected and unprotected mode, but I'm not sure if it is worth the extra complexity? If it is the memory usage you are concerned with, it could be reduced to half of the current stack size (page size) [2], if we want to retain all frames. But we could also have some max depth, say like 16? One downside being that it could make stack overflows more challenging to root cause.
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/r/CAC_TJvdCuGNEJC4M+bV6o48CSJRs_4GEUb3iiP_4ro79q=KesA@mail.gmail.com/
Thanks
Kalesh
Quoted 7 lines
Thanks,
M.--
Without deviation from the norm, progress is not possible.
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v3 5/5 KVM: arm64: Unwind and dump nVHE hypervisor stacktrace
In reply to Kalesh SinghOn hyp_panic(), the hypervisor dumps the addresses for its stacktrace entries to a page shared with the host. The host then symbolizes and prints the hyp stacktrace before panicking itself.
Example stacktrace:
[ 122.051187] kvm [380]: Invalid host exception to nVHE hyp! [ 122.052467] kvm [380]: nVHE HYP call trace: [ 122.052814] kvm [380]: [<ffff800008f5b550>] __kvm_nvhe___pkvm_vcpu_init_traps+0x1f0/0x1f0 [ 122.053865] kvm [380]: [<ffff800008f560f0>] __kvm_nvhe_hyp_panic+0x130/0x1c0 [ 122.054367] kvm [380]: [<ffff800008f56190>] __kvm_nvhe___kvm_vcpu_run+0x10/0x10 [ 122.054878] kvm [380]: [<ffff800008f57a40>] __kvm_nvhe_handle___kvm_vcpu_run+0x30/0x50 [ 122.055412] kvm [380]: [<ffff800008f57d2c>] __kvm_nvhe_handle_trap+0xbc/0x160 [ 122.055911] kvm [380]: [<ffff800008f56864>] __kvm_nvhe___host_exit+0x64/0x64 [ 122.056417] kvm [380]: ---- end of nVHE HYP call trace ----
---
Changes in v2:
- Add Mark's Reviewed-by tag
arch/arm64/include/asm/stacktrace.h | 42 ++++++++++++++-- arch/arm64/kernel/stacktrace.c | 75 +++++++++++++++++++++++++++++ arch/arm64/kvm/handle_exit.c | 4 ++ arch/arm64/kvm/hyp/nvhe/switch.c | 4 ++ 4 files changed, 121 insertions(+), 4 deletions(-)
arch/arm64/include/asm/stacktrace.h +38 -4
@@ -5,6 +5,7 @@#ifndef __ASM_STACKTRACE_H#define __ASM_STACKTRACE_H +#include <linux/kvm_host.h> #include <linux/percpu.h> #include <linux/sched.h> #include <linux/sched/task_stack.h>
@@ -19,10 +20,12 @@ enum stack_type {#ifndef __KVM_NVHE_HYPERVISOR__STACK_TYPE_TASK ,STACK_TYPE_IRQ , -STACK_TYPE_OVERFLOW ,STACK_TYPE_SDEI_NORMAL ,STACK_TYPE_SDEI_CRITICAL , +#else /* __KVM_NVHE_HYPERVISOR__ */ +STACK_TYPE_HYP , #endif /* !__KVM_NVHE_HYPERVISOR__ */ +STACK_TYPE_OVERFLOW ,STACK_TYPE_UNKNOWN , __NR_STACK_TYPES };
@@ -55,6 +58,9 @@ static inline bool on_stack(unsigned long sp, unsigned long size,extern void dump_backtrace (struct pt_regs *regs,struct task_struct *tsk,const char *loglvl); +extern void hyp_dump_backtrace (unsigned long hyp_offset); + +DECLARE_PER_CPU(unsigned long, kvm_arm_hyp_stacktrace_page); DECLARE_PER_CPU(unsigned long *, irq_stack_ptr); static inline bool on_irq_stack(unsigned long sp, unsigned long size,
@@ -91,8 +97,32 @@ static inline bool on_overflow_stack(unsigned long sp, unsigned long size,static inline bool on_overflow_stack (unsigned long sp,unsigned long size,struct stack_info *info) {return false; }#endif -#endif /* !__KVM_NVHE_HYPERVISOR__ */ +#else /* __KVM_NVHE_HYPERVISOR__ */ + +extern void hyp_save_backtrace (void ); + +DECLARE_PER_CPU(unsigned long [PAGE_SIZE/sizeof(long)], overflow_stack); +DECLARE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params); + +static inline bool on_overflow_stack (unsigned long sp,unsigned long size, +struct stack_info *info) +{ +unsigned long low = (unsigned long )this_cpu_ptr (overflow_stack); +unsigned long high = low +PAGE_SIZE ; + +return on_stack (sp, size, low, high,STACK_TYPE_OVERFLOW , info); +} + +static inline bool on_hyp_stack (unsigned long sp,unsigned long size, +struct stack_info *info) +{ +struct kvm_nvhe_init_params *params =this_cpu_ptr (&kvm_init_params); +unsigned long high = params->stack_hyp_va; +unsigned long low = high -PAGE_SIZE ; +return on_stack (sp, size, low, high,STACK_TYPE_HYP , info); +} +#endif /* !__KVM_NVHE_HYPERVISOR__ */ /* * We can only safely access per-cpu stacks from current in a non-preemptible
@@ -105,6 +135,9 @@ static inline bool on_accessible_stack(const struct task_struct *tsk,if (info) info->type =STACK_TYPE_UNKNOWN ; +if (on_overflow_stack (sp, size, info)) +return true; +#ifndef __KVM_NVHE_HYPERVISOR__if (on_task_stack (tsk, sp, size, info))return true;
@@ -112,10 +145,11 @@ static inline bool on_accessible_stack(const struct task_struct *tsk,return false;if (on_irq_stack (sp, size, info))return true; -if (on_overflow_stack (sp, size, info)) -return true;if (on_sdei_stack (sp, size, info))return true; +#else /* __KVM_NVHE_HYPERVISOR__ */ +if (on_hyp_stack (sp, size, info)) +return true;#endif /* !__KVM_NVHE_HYPERVISOR__ */ return false;
arch/arm64/kernel/stacktrace.c +75 -0
88 lines not on this page · raw message
arch/arm64/kvm/handle_exit.c +4 -0
@@ -17,6 +17,7 @@#include <asm/kvm_emulate.h> #include <asm/kvm_mmu.h> #include <asm/debug-monitors.h> +#include <asm/stacktrace.h> #include <asm/traps.h> #include <kvm/arm_hypercalls.h>
@@ -353,6 +354,9 @@ void __noreturn __cold nvhe_hyp_panic_handler(u64 esr, u64 spsr, (void *)panic_addr); } +/* Dump the hypervisor stacktrace */ +hyp_dump_backtrace (hyp_offset); +/* * Hyp has panicked and we're going to handle that by panicking the * kernel. The kernel offset will be revealed in the panic so we're
arch/arm64/kvm/hyp/nvhe/switch.c +4 -0
@@ -25,6 +25,7 @@#include <asm/fpsimd.h> #include <asm/debug-monitors.h> #include <asm/processor.h> +#include <asm/stacktrace.h> #include <nvhe/fixed_config.h> #include <nvhe/mem_protect.h>
@@ -375,6 +376,9 @@ asmlinkage void __noreturn hyp_panic(void)__sysreg_restore_state_nvhe (host_ctxt); } +/* Save the hypervisor stacktrace */ +hyp_save_backtrace (); +__hyp_do_panic (host_ctxt, spsr, elr, par);unreachable (); }
Signature · 6 lines
2.36.1.255.ge46751e96f-goog _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
kernel test robot
build botRe: [PATCH v3 5/5] KVM: arm64: Unwind and dump nVHE hypervisor stacktrace
Thank you for the patch! Yet something to improve:
Hi Kalesh,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on f2906aa863381afb0015a9eb7fefad885d4e5a56]
url: https://github.com/intel-lab-lkp/linux/commits/Kalesh-Singh/KVM-nVHE-Hypervisor-stack-unwinder/20220608-011351 base: f2906aa863381afb0015a9eb7fefad885d4e5a56 config: arm64-randconfig-c024-20220612 (https://download.01.org/0day-ci/archive/20220613/[email protected]/config) compiler: aarch64-linux-gcc (GCC) 11.3.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/ac1ce397ffe5b05df06cdb56a30db4099c7428ec
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Kalesh-Singh/KVM-nVHE-Hypervisor-stack-unwinder/20220608-011351 git checkout ac1ce397ffe5b05df06cdb56a30db4099c7428ec
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-11.3.0 make.cross W=1 O=build_dir ARCH=arm64 SHELL=/bin/bash
If you fix the issue, kindly add following tag where applicable
All errors (new ones prefixed by >>):
aarch64-linux-ld: arch/arm64/kernel/stacktrace.o: in function `hyp_dump_backtrace':
>> arch/arm64/kernel/stacktrace.c:254: undefined reference to `kvm_arm_hyp_stacktrace_page' aarch64-linux-ld: arch/arm64/kernel/stacktrace.o: relocation R_AARCH64_ADR_PREL_PG_HI21 against symbol `kvm_arm_hyp_stacktrace_page' which may bind externally can not be used when making a shared object; recompile with -fPIC arch/arm64/kernel/stacktrace.c:254:(.text+0x634): dangerous relocation: unsupported relocation >> aarch64-linux-ld: arch/arm64/kernel/stacktrace.c:254: undefined reference to `kvm_arm_hyp_stacktrace_page'
pahole: .tmp_vmlinux.btf: No such file or directory .btf.vmlinux.bin.o: file not recognized: file format not recognized
vim +254 arch/arm64/kernel/stacktrace.c
246
247 /**
248 * Symbolizes and dumps the hypervisor backtrace from the shared
249 * stacktrace page.
250 */
251 noinline notrace void hyp_dump_backtrace(unsigned long hyp_offset)
252 {
253 unsigned long *stacktrace_pos =
> 254 (unsigned long *)*this_cpu_ptr(&kvm_arm_hyp_stacktrace_page);
255 unsigned long va_mask = GENMASK_ULL(vabits_actual - 1, 0);
256 unsigned long pc = *stacktrace_pos++;
257
258 kvm_err("nVHE HYP call trace:\n");
259
260 while (pc) {
261 pc &= va_mask; /* Mask tags */
262 pc += hyp_offset; /* Convert to kern addr */
263 kvm_err("[<%016lx>] %pB\n", pc, (void *)pc);
264 pc = *stacktrace_pos++;
265 }
266
267 kvm_err("---- end of nVHE HYP call trace ----\n");
268 }
269 #else /* __KVM_NVHE_HYPERVISOR__ */
270 DEFINE_PER_CPU(unsigned long [PAGE_SIZE/sizeof(long)], overflow_stack)
271 __aligned(16);
272
Signature · 6 lines
0-DAY CI Kernel Test Service https://01.org/lkp _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
kernel test robot
build botRe: [PATCH v3 5/5] KVM: arm64: Unwind and dump nVHE hypervisor stacktrace
In reply to Kalesh SinghThank you for the patch! Yet something to improve:
Hi Kalesh,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on f2906aa863381afb0015a9eb7fefad885d4e5a56]
url: https://github.com/intel-lab-lkp/linux/commits/Kalesh-Singh/KVM-nVHE-Hypervisor-stack-unwinder/20220608-011351 base: f2906aa863381afb0015a9eb7fefad885d4e5a56 config: arm64-buildonly-randconfig-r003-20220613 (https://download.01.org/0day-ci/archive/20220615/[email protected]/config) compiler: clang version 15.0.0 (https://github.com/llvm/llvm-project c97436f8b6e2718286e8496faf53a2c800e281cf)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# install arm64 cross compiling tool for clang build
# apt-get install binutils-aarch64-linux-gnu
# https://github.com/intel-lab-lkp/linux/commit/ac1ce397ffe5b05df06cdb56a30db4099c7428ec
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Kalesh-Singh/KVM-nVHE-Hypervisor-stack-unwinder/20220608-011351 git checkout ac1ce397ffe5b05df06cdb56a30db4099c7428ec
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=arm64 SHELL=/bin/bash
If you fix the issue, kindly add following tag where applicable
All errors (new ones prefixed by >>):
>> ld.lld: error: undefined symbol: kvm_arm_hyp_stacktrace_page >>> referenced by stacktrace.c:254 (arch/arm64/kernel/stacktrace.c:254) >>> kernel/stacktrace.o:(hyp_dump_backtrace) in archive arch/arm64/built-in.a >>> referenced by stacktrace.c:254 (arch/arm64/kernel/stacktrace.c:254) >>> kernel/stacktrace.o:(hyp_dump_backtrace) in archive arch/arm64/built-in.a
Signature · 2 lines
0-DAY CI Kernel Test Service https://01.org/lkp