Skip to contentSkip to conversation

pkvm boot failures

DISCUSSION Aneesh Kumar K.V 3 participants 7 messages Jun 9, 2025 – Jun 10, 2025

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

I am hitting the below failure with v6.15 (I tried other kernel versions with similar results). I disabled CONFIG_PROTECTED_NVHE_STACKTRACE because with CONFIG_NVHE_EL2_DEBUG, the stack was pointing at hyp_assert_lock_held() .

[    0.664457] kvm [1]: nVHE hyp panic at: [<ffff80008050b10c>] __kvm_nvhe_handle_trap+0x34/0x10c!
[    0.664538] kvm [1]: Cannot dump pKVM nVHE stacktrace: !CONFIG_PROTECTED_NVHE_STACKTRACE
[    0.664566] kvm [1]: Hyp Offset: 0xffff000007c00000
[    0.664631] Kernel panic - not syncing: HYP panic:
[    0.664631] PS:614023c9 PC:000080007890b10c ESR:0000000096000007
[    0.664631] FAR:0000800078c252f0 HPFAR:0000000000000000 PAR:0000000000000000
[    0.664631] VCPU:0000000000000000
[    0.664938] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.15.0-rc1 #594 NONE 
[    0.665068] Hardware name: FVP Base RevC (DT)
[    0.665140] Call trace:
[    0.665196]  show_stack+0x18/0x24 (C)
[    0.665346]  dump_stack_lvl+0x3c/0x80
[    0.665468]  dump_stack+0x18/0x24
[    0.665588]  panic+0x124/0x2d8
[    0.665699]  nvhe_hyp_panic_handler+0x108/0x180
[    0.665825]  do_pkvm_init+0xb0/0x124
[    0.665957]  do_pkvm_init+0xb0/0x124
[    0.666089]  kvm_hyp_init_protection+0x5c/0x6c
[    0.666226]  init_hyp_mode+0x760/0x790
[    0.666362]  kvm_arm_init+0xac/0x23c
[    0.666492]  do_one_initcall+0xa0/0x1f0
[    0.666617]  do_initcall_level+0x8c/0xac
[    0.666753]  do_initcalls+0x54/0x94
[    0.666885]  do_basic_setup+0x18/0x24
[    0.667019]  kernel_init_freeable+0xc0/0x10c
[    0.667157]  kernel_init+0x20/0x118
[    0.667271]  ret_from_fork+0x10/0x20
[    0.667400] SMP: stopping secondary CPUs
[    0.667475] Kernel Offset: disabled
[    0.667534] CPU features: 0x0000,00000140,064dc298,cb7a552f
[    0.667619] Memory Limit: none
[    0.667681] ---[ end Kernel panic - not syncing: HYP panic:
[    0.667681] PS:614023c9 PC:000080007890b10c ESR:0000000096000007
[    0.667681] FAR:0000800078c252f0 HPFAR:0000000000000000 PAR:0000000000000000
[    0.667681] VCPU:0000000000000000 ]

I was able to locate a .config that make the pkvm work, But i am not able to identify which config dependency is making the difference. I am attaching below the working and non working kernel configs. I am using FVP to test this.

-aneesh

maintainer
#1 lore raw
Quoted 5 lines · Aneesh Kumar K.V

I am hitting the below failure with v6.15 (I tried other kernel versions with similar results). I disabled CONFIG_PROTECTED_NVHE_STACKTRACE because with CONFIG_NVHE_EL2_DEBUG, the stack was pointing at hyp_assert_lock_held() .

Why do you get different results based on the backtrace support?

Quoted 6 lines
[    0.664457] kvm [1]: nVHE hyp panic at: [<ffff80008050b10c>] __kvm_nvhe_handle_trap+0x34/0x10c!
[    0.664538] kvm [1]: Cannot dump pKVM nVHE stacktrace: !CONFIG_PROTECTED_NVHE_STACKTRACE
[    0.664566] kvm [1]: Hyp Offset: 0xffff000007c00000
[    0.664631] Kernel panic - not syncing: HYP panic:
[    0.664631] PS:614023c9 PC:000080007890b10c ESR:0000000096000007

ESR_EL2 shows that this is a S1 translation fault, level 3, from EL2. This is consistent with hitting the BUG() in handle_trap(().

What is at this PC?

Quoted 1 line
[    0.664631] FAR:0000800078c252f0 HPFAR:0000000000000000 PAR:0000000000000000

FAR_EL2 is valid (as per the above), so what is EL2 trying to access at this address?

Quoted 4 lines

I was able to locate a .config that make the pkvm work, But i am not able to identify which config dependency is making the difference. I am attaching below the working and non working kernel configs. I am using FVP to test this.

If you're using a model, it should be pretty easy to track all of the above.

Thanks,

	M.
Signature · 1 line
Jazz isn't dead. It just smells funny.
#2 lore raw
In reply to Aneesh Kumar K.V
Quoted 42 lines · Aneesh Kumar K.V

I am hitting the below failure with v6.15 (I tried other kernel versions with similar results). I disabled CONFIG_PROTECTED_NVHE_STACKTRACE because with CONFIG_NVHE_EL2_DEBUG, the stack was pointing at hyp_assert_lock_held() .

[    0.664457] kvm [1]: nVHE hyp panic at: [<ffff80008050b10c>] __kvm_nvhe_handle_trap+0x34/0x10c!
[    0.664538] kvm [1]: Cannot dump pKVM nVHE stacktrace: !CONFIG_PROTECTED_NVHE_STACKTRACE
[    0.664566] kvm [1]: Hyp Offset: 0xffff000007c00000
[    0.664631] Kernel panic - not syncing: HYP panic:
[    0.664631] PS:614023c9 PC:000080007890b10c ESR:0000000096000007
[    0.664631] FAR:0000800078c252f0 HPFAR:0000000000000000 PAR:0000000000000000
[    0.664631] VCPU:0000000000000000
[    0.664938] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.15.0-rc1 #594 NONE 
[    0.665068] Hardware name: FVP Base RevC (DT)
[    0.665140] Call trace:
[    0.665196]  show_stack+0x18/0x24 (C)
[    0.665346]  dump_stack_lvl+0x3c/0x80
[    0.665468]  dump_stack+0x18/0x24
[    0.665588]  panic+0x124/0x2d8
[    0.665699]  nvhe_hyp_panic_handler+0x108/0x180
[    0.665825]  do_pkvm_init+0xb0/0x124
[    0.665957]  do_pkvm_init+0xb0/0x124
[    0.666089]  kvm_hyp_init_protection+0x5c/0x6c
[    0.666226]  init_hyp_mode+0x760/0x790
[    0.666362]  kvm_arm_init+0xac/0x23c
[    0.666492]  do_one_initcall+0xa0/0x1f0
[    0.666617]  do_initcall_level+0x8c/0xac
[    0.666753]  do_initcalls+0x54/0x94
[    0.666885]  do_basic_setup+0x18/0x24
[    0.667019]  kernel_init_freeable+0xc0/0x10c
[    0.667157]  kernel_init+0x20/0x118
[    0.667271]  ret_from_fork+0x10/0x20
[    0.667400] SMP: stopping secondary CPUs
[    0.667475] Kernel Offset: disabled
[    0.667534] CPU features: 0x0000,00000140,064dc298,cb7a552f
[    0.667619] Memory Limit: none
[    0.667681] ---[ end Kernel panic - not syncing: HYP panic:
[    0.667681] PS:614023c9 PC:000080007890b10c ESR:0000000096000007
[    0.667681] FAR:0000800078c252f0 HPFAR:0000000000000000 PAR:0000000000000000
[    0.667681] VCPU:0000000000000000 ]
Quoted 5 lines

I was able to locate a .config that make the pkvm work, But i am not able to identify which config dependency is making the difference. I am attaching below the working and non working kernel configs. I am using FVP to test this.

I had a look at this and tracked the issue to "CONFIG_JUMP_LABEL=n" It seems that it panics at
if (static_branch_unlikely(&kvm_protected_mode_initialized))
Where "kvm_protected_mode_initialized" is mapped in the initial PGD for the hypervisor, but not mapped in the hypervisor created one.
As the variable is defined outside the hypervisor namespace, it doesn’t exist in the hyp bss section.
And in case of "CONFIG_JUMP_LABEL=n" it won't be patched in this case, causing next access to read the variable and panicking.

I guess moving this key to hyp would cause problems with kernel access after de-privilege as cases from kvm_share_hyp(), So I can only think of having a different key for the hypervisor as

arch/arm64/kvm/hyp/nvhe/hyp-main.c +2 -1
index 8e8848de4d47..8945b335bcea 100644
@@ -21,6 +21,7 @@
 #include <nvhe/trap_handler.h>
 
 DEFINE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params);
+DEFINE_STATIC_KEY_FALSE(kvm_protected_mode_initialized_hyp);
 
 void __kvm_hyp_host_forward_smc(struct kvm_cpu_context *host_ctxt);
 
@@ -626,7 +627,7 @@ static void handle_host_hcall(struct kvm_cpu_context *host_ctxt)
 	 * basis. This is all fine, however, since __pkvm_prot_finalize
 	 * returns -EPERM after the first call for a given CPU.
 	 */
-	if (static_branch_unlikely(&kvm_protected_mode_initialized))
+	if (static_branch_unlikely(&kvm_protected_mode_initialized_hyp))
 		hcall_min = __KVM_HOST_SMCCC_FUNC___pkvm_prot_finalize;
 
 	id &= ~ARM_SMCCC_CALL_HINTS;
arch/arm64/kvm/pkvm.c +2 -0
index fcd70bfe44fb..af0854e98902 100644
@@ -17,6 +17,7 @@
 #include "hyp_constants.h"
 
 DEFINE_STATIC_KEY_FALSE(kvm_protected_mode_initialized);
+DECLARE_STATIC_KEY_FALSE(kvm_nvhe_sym(kvm_protected_mode_initialized_hyp));
 
 static struct memblock_region *hyp_memory = kvm_nvhe_sym(hyp_memory);
 static unsigned int *hyp_memblock_nr_ptr = &kvm_nvhe_sym(hyp_memblock_nr);
@@ -229,6 +230,7 @@ static int __init pkvm_drop_host_privileges(void)
 	 * once the host stage 2 is installed.
 	 */
 	static_branch_enable(&kvm_protected_mode_initialized);
+	static_branch_enable(&kvm_nvhe_sym(kvm_protected_mode_initialized_hyp));
 	on_each_cpu(_kvm_host_prot_finalize, &ret, 1);
 	return ret;
 }

--

Can you please check if that helps in your case?

Thanks,
Mostafa

Quoted 2 lines

-aneesh

#3 lore raw
Quoted 76 lines · Mostafa Saleh
Quoted 42 lines · Aneesh Kumar K.V

I am hitting the below failure with v6.15 (I tried other kernel versions with similar results). I disabled CONFIG_PROTECTED_NVHE_STACKTRACE because with CONFIG_NVHE_EL2_DEBUG, the stack was pointing at hyp_assert_lock_held() .

[    0.664457] kvm [1]: nVHE hyp panic at: [<ffff80008050b10c>] __kvm_nvhe_handle_trap+0x34/0x10c!
[    0.664538] kvm [1]: Cannot dump pKVM nVHE stacktrace: !CONFIG_PROTECTED_NVHE_STACKTRACE
[    0.664566] kvm [1]: Hyp Offset: 0xffff000007c00000
[    0.664631] Kernel panic - not syncing: HYP panic:
[    0.664631] PS:614023c9 PC:000080007890b10c ESR:0000000096000007
[    0.664631] FAR:0000800078c252f0 HPFAR:0000000000000000 PAR:0000000000000000
[    0.664631] VCPU:0000000000000000
[    0.664938] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.15.0-rc1 #594 NONE 
[    0.665068] Hardware name: FVP Base RevC (DT)
[    0.665140] Call trace:
[    0.665196]  show_stack+0x18/0x24 (C)
[    0.665346]  dump_stack_lvl+0x3c/0x80
[    0.665468]  dump_stack+0x18/0x24
[    0.665588]  panic+0x124/0x2d8
[    0.665699]  nvhe_hyp_panic_handler+0x108/0x180
[    0.665825]  do_pkvm_init+0xb0/0x124
[    0.665957]  do_pkvm_init+0xb0/0x124
[    0.666089]  kvm_hyp_init_protection+0x5c/0x6c
[    0.666226]  init_hyp_mode+0x760/0x790
[    0.666362]  kvm_arm_init+0xac/0x23c
[    0.666492]  do_one_initcall+0xa0/0x1f0
[    0.666617]  do_initcall_level+0x8c/0xac
[    0.666753]  do_initcalls+0x54/0x94
[    0.666885]  do_basic_setup+0x18/0x24
[    0.667019]  kernel_init_freeable+0xc0/0x10c
[    0.667157]  kernel_init+0x20/0x118
[    0.667271]  ret_from_fork+0x10/0x20
[    0.667400] SMP: stopping secondary CPUs
[    0.667475] Kernel Offset: disabled
[    0.667534] CPU features: 0x0000,00000140,064dc298,cb7a552f
[    0.667619] Memory Limit: none
[    0.667681] ---[ end Kernel panic - not syncing: HYP panic:
[    0.667681] PS:614023c9 PC:000080007890b10c ESR:0000000096000007
[    0.667681] FAR:0000800078c252f0 HPFAR:0000000000000000 PAR:0000000000000000
[    0.667681] VCPU:0000000000000000 ]
Quoted 5 lines

I was able to locate a .config that make the pkvm work, But i am not able to identify which config dependency is making the difference. I am attaching below the working and non working kernel configs. I am using FVP to test this.

I had a look at this and tracked the issue to "CONFIG_JUMP_LABEL=n" It seems that it panics at
if (static_branch_unlikely(&kvm_protected_mode_initialized))
Where "kvm_protected_mode_initialized" is mapped in the initial PGD for the hypervisor, but not mapped in the hypervisor created one.
As the variable is defined outside the hypervisor namespace, it doesn’t exist in the hyp bss section.
And in case of "CONFIG_JUMP_LABEL=n" it won't be patched in this case, causing next access to read the variable and panicking.

I guess moving this key to hyp would cause problems with kernel access after de-privilege as cases from kvm_share_hyp(), So I can only think of having a different key for the hypervisor as

arch/arm64/kvm/hyp/nvhe/hyp-main.c +1 -0
index 8e8848de4d47..8945b335bcea 100644
@@ -21,6 +21,7 @@
 #include <nvhe/trap_handler.h>
 
 DEFINE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params);
+DEFINE_STATIC_KEY_FALSE(kvm_protected_mode_initialized_hyp);
 
 void __kvm_hyp_host_forward_smc(struct kvm_cpu_context *host_ctxt);
 

arch/arm64/kvm/hyp/nvhe/hyp-main.c static void handle_host_hcall(struct kvm_cpu_context *host_ctxt) · line 632

Quoted 7 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +1 -1
@@ -626,7 +627,7 @@ static void handle_host_hcall(struct kvm_cpu_context *host_ctxt)
 	 * basis. This is all fine, however, since __pkvm_prot_finalize
 	 * returns -EPERM after the first call for a given CPU.
 	 */
-	if (static_branch_unlikely(&kvm_protected_mode_initialized))
+	if (static_branch_unlikely(&kvm_protected_mode_initialized_hyp))
 		hcall_min = __KVM_HOST_SMCCC_FUNC___pkvm_prot_finalize;
 

There are other references to kvm_protected_mode_initialized within nvhe. Shouldn't all of those be updated as well?

Also, do we need to drop this line?

KVM_NVHE_ALIAS(kvm_protected_mode_initialized);

If I understand correctly, nvhe will now have a separate static key, distinct from the one used by the EL1 host. So we'll need to update functions like is_pkvm_initialized() and host_data_ptr() accordingly, right?

Quoted 25 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +0 -0
 	id &= ~ARM_SMCCC_CALL_HINTS;
arch/arm64/kvm/pkvm.c +2 -1
index fcd70bfe44fb..af0854e98902 100644
@@ -17,6 +17,7 @@
 #include "hyp_constants.h"
 
 DEFINE_STATIC_KEY_FALSE(kvm_protected_mode_initialized);
+DECLARE_STATIC_KEY_FALSE(kvm_nvhe_sym(kvm_protected_mode_initialized_hyp));
 
 static struct memblock_region *hyp_memory = kvm_nvhe_sym(hyp_memory);
 static unsigned int *hyp_memblock_nr_ptr = &kvm_nvhe_sym(hyp_memblock_nr);
@@ -229,6 +230,7 @@ static int __init pkvm_drop_host_privileges(void)
 	 * once the host stage 2 is installed.
 	 */
 	static_branch_enable(&kvm_protected_mode_initialized);
+	static_branch_enable(&kvm_nvhe_sym(kvm_protected_mode_initialized_hyp));
 	on_each_cpu(_kvm_host_prot_finalize, &ret, 1);
 	return ret;
 }
 
--
Quoted 5 lines

Can you please check if that helps in your case?

Thanks,
Mostafa

-aneesh

#4 lore raw
Quoted 77 lines · Aneesh Kumar K.V
Quoted 75 lines · Mostafa Saleh
Quoted 42 lines · Aneesh Kumar K.V

I am hitting the below failure with v6.15 (I tried other kernel versions with similar results). I disabled CONFIG_PROTECTED_NVHE_STACKTRACE because with CONFIG_NVHE_EL2_DEBUG, the stack was pointing at hyp_assert_lock_held() .

[    0.664457] kvm [1]: nVHE hyp panic at: [<ffff80008050b10c>] __kvm_nvhe_handle_trap+0x34/0x10c!
[    0.664538] kvm [1]: Cannot dump pKVM nVHE stacktrace: !CONFIG_PROTECTED_NVHE_STACKTRACE
[    0.664566] kvm [1]: Hyp Offset: 0xffff000007c00000
[    0.664631] Kernel panic - not syncing: HYP panic:
[    0.664631] PS:614023c9 PC:000080007890b10c ESR:0000000096000007
[    0.664631] FAR:0000800078c252f0 HPFAR:0000000000000000 PAR:0000000000000000
[    0.664631] VCPU:0000000000000000
[    0.664938] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.15.0-rc1 #594 NONE 
[    0.665068] Hardware name: FVP Base RevC (DT)
[    0.665140] Call trace:
[    0.665196]  show_stack+0x18/0x24 (C)
[    0.665346]  dump_stack_lvl+0x3c/0x80
[    0.665468]  dump_stack+0x18/0x24
[    0.665588]  panic+0x124/0x2d8
[    0.665699]  nvhe_hyp_panic_handler+0x108/0x180
[    0.665825]  do_pkvm_init+0xb0/0x124
[    0.665957]  do_pkvm_init+0xb0/0x124
[    0.666089]  kvm_hyp_init_protection+0x5c/0x6c
[    0.666226]  init_hyp_mode+0x760/0x790
[    0.666362]  kvm_arm_init+0xac/0x23c
[    0.666492]  do_one_initcall+0xa0/0x1f0
[    0.666617]  do_initcall_level+0x8c/0xac
[    0.666753]  do_initcalls+0x54/0x94
[    0.666885]  do_basic_setup+0x18/0x24
[    0.667019]  kernel_init_freeable+0xc0/0x10c
[    0.667157]  kernel_init+0x20/0x118
[    0.667271]  ret_from_fork+0x10/0x20
[    0.667400] SMP: stopping secondary CPUs
[    0.667475] Kernel Offset: disabled
[    0.667534] CPU features: 0x0000,00000140,064dc298,cb7a552f
[    0.667619] Memory Limit: none
[    0.667681] ---[ end Kernel panic - not syncing: HYP panic:
[    0.667681] PS:614023c9 PC:000080007890b10c ESR:0000000096000007
[    0.667681] FAR:0000800078c252f0 HPFAR:0000000000000000 PAR:0000000000000000
[    0.667681] VCPU:0000000000000000 ]
Quoted 5 lines

I was able to locate a .config that make the pkvm work, But i am not able to identify which config dependency is making the difference. I am attaching below the working and non working kernel configs. I am using FVP to test this.

I had a look at this and tracked the issue to "CONFIG_JUMP_LABEL=n" It seems that it panics at
if (static_branch_unlikely(&kvm_protected_mode_initialized))
Where "kvm_protected_mode_initialized" is mapped in the initial PGD for the hypervisor, but not mapped in the hypervisor created one.
As the variable is defined outside the hypervisor namespace, it doesn’t exist in the hyp bss section.
And in case of "CONFIG_JUMP_LABEL=n" it won't be patched in this case, causing next access to read the variable and panicking.

I guess moving this key to hyp would cause problems with kernel access after de-privilege as cases from kvm_share_hyp(), So I can only think of having a different key for the hypervisor as

arch/arm64/kvm/hyp/nvhe/hyp-main.c +1 -0
index 8e8848de4d47..8945b335bcea 100644
@@ -21,6 +21,7 @@
 #include <nvhe/trap_handler.h>
 
 DEFINE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params);
+DEFINE_STATIC_KEY_FALSE(kvm_protected_mode_initialized_hyp);
 
 void __kvm_hyp_host_forward_smc(struct kvm_cpu_context *host_ctxt);
 
Quoted 11 lines

arch/arm64/kvm/hyp/nvhe/hyp-main.c static void handle_host_hcall(struct kvm_cpu_context *host_ctxt) · line 632

Quoted 8 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +1 -1
@@ -626,7 +627,7 @@ static void handle_host_hcall(struct kvm_cpu_context *host_ctxt)
 	 * basis. This is all fine, however, since __pkvm_prot_finalize
 	 * returns -EPERM after the first call for a given CPU.
 	 */
-	if (static_branch_unlikely(&kvm_protected_mode_initialized))
+	if (static_branch_unlikely(&kvm_protected_mode_initialized_hyp))
 		hcall_min = __KVM_HOST_SMCCC_FUNC___pkvm_prot_finalize;
 

There are other references to kvm_protected_mode_initialized within nvhe. Shouldn't all of those be updated as well?

Yes, I missed the ones in the headers, those should be replaced with “kvm_protected_mode_initialized_hyp” also.

Quoted 5 lines

Also, do we need to drop this line?

KVM_NVHE_ALIAS(kvm_protected_mode_initialized);

Yes, that’s not needed now.

Quoted 4 lines

If I understand correctly, nvhe will now have a separate static key, distinct from the one used by the EL1 host. So we'll need to update functions like is_pkvm_initialized() and host_data_ptr() accordingly, right?

As far as I see, those are used by the kernel only, but if needed we can guard that with “__KVM_NVHE_HYPERVISOR__”

host_data_ptr() won’t access the key from the hypervisor because of the macro check.

But I agree with Marc, that is a lot of duplication.

Thanks,
Mostafa

Quoted 33 lines
Quoted 26 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +0 -0
 	id &= ~ARM_SMCCC_CALL_HINTS;
arch/arm64/kvm/pkvm.c +2 -1
index fcd70bfe44fb..af0854e98902 100644
@@ -17,6 +17,7 @@
 #include "hyp_constants.h"
 
 DEFINE_STATIC_KEY_FALSE(kvm_protected_mode_initialized);
+DECLARE_STATIC_KEY_FALSE(kvm_nvhe_sym(kvm_protected_mode_initialized_hyp));
 
 static struct memblock_region *hyp_memory = kvm_nvhe_sym(hyp_memory);
 static unsigned int *hyp_memblock_nr_ptr = &kvm_nvhe_sym(hyp_memblock_nr);
@@ -229,6 +230,7 @@ static int __init pkvm_drop_host_privileges(void)
 	 * once the host stage 2 is installed.
 	 */
 	static_branch_enable(&kvm_protected_mode_initialized);
+	static_branch_enable(&kvm_nvhe_sym(kvm_protected_mode_initialized_hyp));
 	on_each_cpu(_kvm_host_prot_finalize, &ret, 1);
 	return ret;
 }
 
--
Quoted 5 lines

Can you please check if that helps in your case?

Thanks,
Mostafa

-aneesh

maintainer
#5 lore raw
In reply to Mostafa Saleh

Hi Mostafa,

Thanks for looking into this.

Quoted 50 lines · Mostafa Saleh
Quoted 42 lines · Aneesh Kumar K.V

I am hitting the below failure with v6.15 (I tried other kernel versions with similar results). I disabled CONFIG_PROTECTED_NVHE_STACKTRACE because with CONFIG_NVHE_EL2_DEBUG, the stack was pointing at hyp_assert_lock_held() .

[    0.664457] kvm [1]: nVHE hyp panic at: [<ffff80008050b10c>] __kvm_nvhe_handle_trap+0x34/0x10c!
[    0.664538] kvm [1]: Cannot dump pKVM nVHE stacktrace: !CONFIG_PROTECTED_NVHE_STACKTRACE
[    0.664566] kvm [1]: Hyp Offset: 0xffff000007c00000
[    0.664631] Kernel panic - not syncing: HYP panic:
[    0.664631] PS:614023c9 PC:000080007890b10c ESR:0000000096000007
[    0.664631] FAR:0000800078c252f0 HPFAR:0000000000000000 PAR:0000000000000000
[    0.664631] VCPU:0000000000000000
[    0.664938] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.15.0-rc1 #594 NONE 
[    0.665068] Hardware name: FVP Base RevC (DT)
[    0.665140] Call trace:
[    0.665196]  show_stack+0x18/0x24 (C)
[    0.665346]  dump_stack_lvl+0x3c/0x80
[    0.665468]  dump_stack+0x18/0x24
[    0.665588]  panic+0x124/0x2d8
[    0.665699]  nvhe_hyp_panic_handler+0x108/0x180
[    0.665825]  do_pkvm_init+0xb0/0x124
[    0.665957]  do_pkvm_init+0xb0/0x124
[    0.666089]  kvm_hyp_init_protection+0x5c/0x6c
[    0.666226]  init_hyp_mode+0x760/0x790
[    0.666362]  kvm_arm_init+0xac/0x23c
[    0.666492]  do_one_initcall+0xa0/0x1f0
[    0.666617]  do_initcall_level+0x8c/0xac
[    0.666753]  do_initcalls+0x54/0x94
[    0.666885]  do_basic_setup+0x18/0x24
[    0.667019]  kernel_init_freeable+0xc0/0x10c
[    0.667157]  kernel_init+0x20/0x118
[    0.667271]  ret_from_fork+0x10/0x20
[    0.667400] SMP: stopping secondary CPUs
[    0.667475] Kernel Offset: disabled
[    0.667534] CPU features: 0x0000,00000140,064dc298,cb7a552f
[    0.667619] Memory Limit: none
[    0.667681] ---[ end Kernel panic - not syncing: HYP panic:
[    0.667681] PS:614023c9 PC:000080007890b10c ESR:0000000096000007
[    0.667681] FAR:0000800078c252f0 HPFAR:0000000000000000 PAR:0000000000000000
[    0.667681] VCPU:0000000000000000 ]
Quoted 5 lines

I was able to locate a .config that make the pkvm work, But i am not able to identify which config dependency is making the difference. I am attaching below the working and non working kernel configs. I am using FVP to test this.

Quoted 9 lines

I had a look at this and tracked the issue to "CONFIG_JUMP_LABEL=n" It seems that it panics at
if (static_branch_unlikely(&kvm_protected_mode_initialized))
Where "kvm_protected_mode_initialized" is mapped in the initial PGD for the hypervisor, but not mapped in the hypervisor created one.
As the variable is defined outside the hypervisor namespace, it doesn’t exist in the hyp bss section.
And in case of "CONFIG_JUMP_LABEL=n" it won't be patched in this case, causing next access to read the variable and panicking.

It really begs the question: why do we even support JUMP_LABEL=n? It really feels like a backward configuration, and I'd be very glad to either mark it as "always on", or make KVM depend on it.

Quoted 39 lines

I guess moving this key to hyp would cause problems with kernel access after de-privilege as cases from kvm_share_hyp(), So I can only think of having a different key for the hypervisor as

arch/arm64/kvm/hyp/nvhe/hyp-main.c +2 -1
index 8e8848de4d47..8945b335bcea 100644
@@ -21,6 +21,7 @@
 #include <nvhe/trap_handler.h>
 
 DEFINE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params);
+DEFINE_STATIC_KEY_FALSE(kvm_protected_mode_initialized_hyp);
 
 void __kvm_hyp_host_forward_smc(struct kvm_cpu_context *host_ctxt);
 
@@ -626,7 +627,7 @@ static void handle_host_hcall(struct kvm_cpu_context *host_ctxt)
 	 * basis. This is all fine, however, since __pkvm_prot_finalize
 	 * returns -EPERM after the first call for a given CPU.
 	 */
-	if (static_branch_unlikely(&kvm_protected_mode_initialized))
+	if (static_branch_unlikely(&kvm_protected_mode_initialized_hyp))
 		hcall_min = __KVM_HOST_SMCCC_FUNC___pkvm_prot_finalize;
 
 	id &= ~ARM_SMCCC_CALL_HINTS;
arch/arm64/kvm/pkvm.c +1 -0
index fcd70bfe44fb..af0854e98902 100644
@@ -17,6 +17,7 @@
 #include "hyp_constants.h"
 
 DEFINE_STATIC_KEY_FALSE(kvm_protected_mode_initialized);
+DECLARE_STATIC_KEY_FALSE(kvm_nvhe_sym(kvm_protected_mode_initialized_hyp));
 
 static struct memblock_region *hyp_memory = kvm_nvhe_sym(hyp_memory);
 static unsigned int *hyp_memblock_nr_ptr = &kvm_nvhe_sym(hyp_memblock_nr);

arch/arm64/kvm/pkvm.c static int __init pkvm_drop_host_privileges(void) · line 236

Quoted 7 lines
arch/arm64/kvm/pkvm.c +1 -0
@@ -229,6 +230,7 @@ static int __init pkvm_drop_host_privileges(void)
 	 * once the host stage 2 is installed.
 	 */
 	static_branch_enable(&kvm_protected_mode_initialized);
+	static_branch_enable(&kvm_nvhe_sym(kvm_protected_mode_initialized_hyp));
 	on_each_cpu(_kvm_host_prot_finalize, &ret, 1);
 	return ret;
 }

I don't really enjoy this duplication, and unless we have a good reason not too, I'd rather have something like:

arch/arm64/kvm/Kconfig +1 -0
index 713248f240e0..66d232e7c894 100644
@@ -37,6 +37,7 @@ menuconfig KVM
 	select HAVE_KVM_VCPU_RUN_PID_CHANGE
 	select SCHED_INFO
 	select GUEST_PERF_EVENTS if PERF_EVENTS
+	select JUMP_LABEL
 	help
 	  Support hosting virtualized guest machines.
 

It should be OK now that all the supported compilers have asm goto support.

Thanks,

	M.
Signature · 1 line
Jazz isn't dead. It just smells funny.
#6 lore raw

Hi Marc,

Quoted 132 lines · Marc Zyngier

Hi Mostafa,

Thanks for looking into this.

Quoted 50 lines · Mostafa Saleh
Quoted 42 lines · Aneesh Kumar K.V

I am hitting the below failure with v6.15 (I tried other kernel versions with similar results). I disabled CONFIG_PROTECTED_NVHE_STACKTRACE because with CONFIG_NVHE_EL2_DEBUG, the stack was pointing at hyp_assert_lock_held() .

[    0.664457] kvm [1]: nVHE hyp panic at: [<ffff80008050b10c>] __kvm_nvhe_handle_trap+0x34/0x10c!
[    0.664538] kvm [1]: Cannot dump pKVM nVHE stacktrace: !CONFIG_PROTECTED_NVHE_STACKTRACE
[    0.664566] kvm [1]: Hyp Offset: 0xffff000007c00000
[    0.664631] Kernel panic - not syncing: HYP panic:
[    0.664631] PS:614023c9 PC:000080007890b10c ESR:0000000096000007
[    0.664631] FAR:0000800078c252f0 HPFAR:0000000000000000 PAR:0000000000000000
[    0.664631] VCPU:0000000000000000
[    0.664938] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.15.0-rc1 #594 NONE 
[    0.665068] Hardware name: FVP Base RevC (DT)
[    0.665140] Call trace:
[    0.665196]  show_stack+0x18/0x24 (C)
[    0.665346]  dump_stack_lvl+0x3c/0x80
[    0.665468]  dump_stack+0x18/0x24
[    0.665588]  panic+0x124/0x2d8
[    0.665699]  nvhe_hyp_panic_handler+0x108/0x180
[    0.665825]  do_pkvm_init+0xb0/0x124
[    0.665957]  do_pkvm_init+0xb0/0x124
[    0.666089]  kvm_hyp_init_protection+0x5c/0x6c
[    0.666226]  init_hyp_mode+0x760/0x790
[    0.666362]  kvm_arm_init+0xac/0x23c
[    0.666492]  do_one_initcall+0xa0/0x1f0
[    0.666617]  do_initcall_level+0x8c/0xac
[    0.666753]  do_initcalls+0x54/0x94
[    0.666885]  do_basic_setup+0x18/0x24
[    0.667019]  kernel_init_freeable+0xc0/0x10c
[    0.667157]  kernel_init+0x20/0x118
[    0.667271]  ret_from_fork+0x10/0x20
[    0.667400] SMP: stopping secondary CPUs
[    0.667475] Kernel Offset: disabled
[    0.667534] CPU features: 0x0000,00000140,064dc298,cb7a552f
[    0.667619] Memory Limit: none
[    0.667681] ---[ end Kernel panic - not syncing: HYP panic:
[    0.667681] PS:614023c9 PC:000080007890b10c ESR:0000000096000007
[    0.667681] FAR:0000800078c252f0 HPFAR:0000000000000000 PAR:0000000000000000
[    0.667681] VCPU:0000000000000000 ]
Quoted 5 lines

I was able to locate a .config that make the pkvm work, But i am not able to identify which config dependency is making the difference. I am attaching below the working and non working kernel configs. I am using FVP to test this.

Quoted 9 lines

I had a look at this and tracked the issue to "CONFIG_JUMP_LABEL=n" It seems that it panics at
if (static_branch_unlikely(&kvm_protected_mode_initialized))
Where "kvm_protected_mode_initialized" is mapped in the initial PGD for the hypervisor, but not mapped in the hypervisor created one.
As the variable is defined outside the hypervisor namespace, it doesn’t exist in the hyp bss section.
And in case of "CONFIG_JUMP_LABEL=n" it won't be patched in this case, causing next access to read the variable and panicking.

It really begs the question: why do we even support JUMP_LABEL=n? It really feels like a backward configuration, and I'd be very glad to either mark it as "always on", or make KVM depend on it.

Quoted 39 lines

I guess moving this key to hyp would cause problems with kernel access after de-privilege as cases from kvm_share_hyp(), So I can only think of having a different key for the hypervisor as

arch/arm64/kvm/hyp/nvhe/hyp-main.c +2 -1
index 8e8848de4d47..8945b335bcea 100644
@@ -21,6 +21,7 @@
 #include <nvhe/trap_handler.h>
 
 DEFINE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params);
+DEFINE_STATIC_KEY_FALSE(kvm_protected_mode_initialized_hyp);
 
 void __kvm_hyp_host_forward_smc(struct kvm_cpu_context *host_ctxt);
 
@@ -626,7 +627,7 @@ static void handle_host_hcall(struct kvm_cpu_context *host_ctxt)
 	 * basis. This is all fine, however, since __pkvm_prot_finalize
 	 * returns -EPERM after the first call for a given CPU.
 	 */
-	if (static_branch_unlikely(&kvm_protected_mode_initialized))
+	if (static_branch_unlikely(&kvm_protected_mode_initialized_hyp))
 		hcall_min = __KVM_HOST_SMCCC_FUNC___pkvm_prot_finalize;
 
 	id &= ~ARM_SMCCC_CALL_HINTS;
arch/arm64/kvm/pkvm.c +1 -0
index fcd70bfe44fb..af0854e98902 100644
@@ -17,6 +17,7 @@
 #include "hyp_constants.h"
 
 DEFINE_STATIC_KEY_FALSE(kvm_protected_mode_initialized);
+DECLARE_STATIC_KEY_FALSE(kvm_nvhe_sym(kvm_protected_mode_initialized_hyp));
 
 static struct memblock_region *hyp_memory = kvm_nvhe_sym(hyp_memory);
 static unsigned int *hyp_memblock_nr_ptr = &kvm_nvhe_sym(hyp_memblock_nr);

arch/arm64/kvm/pkvm.c static int __init pkvm_drop_host_privileges(void) · line 236

Quoted 7 lines
arch/arm64/kvm/pkvm.c +1 -0
@@ -229,6 +230,7 @@ static int __init pkvm_drop_host_privileges(void)
 	 * once the host stage 2 is installed.
 	 */
 	static_branch_enable(&kvm_protected_mode_initialized);
+	static_branch_enable(&kvm_nvhe_sym(kvm_protected_mode_initialized_hyp));
 	on_each_cpu(_kvm_host_prot_finalize, &ret, 1);
 	return ret;
 }

I don't really enjoy this duplication, and unless we have a good reason not too, I'd rather have something like:

arch/arm64/kvm/Kconfig +1 -0
index 713248f240e0..66d232e7c894 100644
@@ -37,6 +37,7 @@ menuconfig KVM
 	select HAVE_KVM_VCPU_RUN_PID_CHANGE
 	select SCHED_INFO
 	select GUEST_PERF_EVENTS if PERF_EVENTS
+	select JUMP_LABEL
 	help
 	  Support hosting virtualized guest machines.
 
Quoted 2 lines

It should be OK now that all the supported compilers have asm goto support.

Yes, that looks better, I am not sure why that is not automatically selected. AFAIK, it should be ok atleast for arm64.

There are some hidden problems though, if the condition is false, that would panic after de-privilege, but as the key is toggled at pkvm_drop_host_privileges() before the privilege drop, it should be fine for now. (but not if the code is reworked)

Also, ofcourse static_branch_disable() won’t be supported.

I’d say, we can add a comment explaining that this key can't be read from the hypervisor from memory, in case someone reworked this in the future.

Thanks,
Mostafa

Quoted 7 lines

Thanks,

	M.
Signature · 1 line
Jazz isn't dead. It just smells funny.