KVM: selftest: Enhance handling WRMSR ICR register in x2APIC mode
PATCH v2 1/1 KVM: selftest: Enhance handling WRMSR ICR register in x2APIC mode
Hardware would directly write x2APIC ICR register instead of software emulation in some circumstances, e.g when Intel IPI virtualization is enabled. This behavior requires normal reserved bits checking to ensure them input as zero, otherwise it will cause #GP. So we need mask out those reserved bits from the data written to vICR register.
Remove Delivery Status bit emulation in test case as this flag is invalid and not needed in x2APIC mode. KVM may ignore clearing it during interrupt dispatch which will lead to fake test failure.
Opportunstically correct vector number for test sending IPI to non-existent vCPUs.
--- .../selftests/kvm/x86_64/xapic_state_test.c | 20 ++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-)
tools/testing/selftests/kvm/x86_64/xapic_state_test.c +17 -3
@@ -70,13 +70,27 @@ static void ____test_icr(struct kvm_vm *vm, struct kvm_vcpu *vcpu, uint64_t val)vcpu_ioctl (vm, vcpu->id,KVM_GET_LAPIC , &xapic); icr = (u64)(*((u32 *)&xapic.regs[APIC_ICR ])) | (u64)(*((u32 *)&xapic.regs[APIC_ICR2 ])) <<32 ; -if (!vcpu->is_x2apic) +if (!vcpu->is_x2apic) { val &= (-1u | (0xffull << (32 +24 ))); -ASSERT_EQ (icr, val & ~APIC_ICR_BUSY ); +ASSERT_EQ (icr, val & ~APIC_ICR_BUSY ); + }else { +ASSERT_EQ (icr & ~APIC_ICR_BUSY , val & ~APIC_ICR_BUSY ); + } } +#define X2APIC_RSVED_BITS_MASK (GENMASK_ULL(31,20) | \ + GENMASK_ULL(17,16) | \ + GENMASK_ULL(13,13)) +static void __test_icr (struct kvm_vm *vm,struct kvm_vcpu *vcpu,uint64_t val) { +if (vcpu->is_x2apic) { +/* Hardware writing vICR register requires reserved bits 31:20, +* 17:16 and 13 kept as zero to avoid #GP exception. Data value +* written to vICR should mask out those bits above. +*/ + val &= ~X2APIC_RSVED_BITS_MASK ; + }____test_icr (vm, vcpu, val |APIC_ICR_BUSY );____test_icr (vm, vcpu, val & ~(u64 )APIC_ICR_BUSY ); }
@@ -100,7 +114,7 @@ static void test_icr(struct kvm_vm *vm, struct kvm_vcpu *vcpu) icr =APIC_INT_ASSERT |0xff ;for (i = vcpu->id +1 ; i <0xff ; i++) {for (j =0 ; j <8 ; j++) -__test_icr (vm, vcpu, i << (32 +24 ) |APIC_INT_ASSERT | (j <<8 )); +__test_icr (vm, vcpu, i << (32 +24 ) | icr | (j <<8 )); }/* And again with a shorthand destination for all types of IPIs. */
Signature · 1 line
2.27.0
Quoted 5 lines · Zeng Guang
Hardware would directly write x2APIC ICR register instead of software emulation in some circumstances, e.g when Intel IPI virtualization is enabled. This behavior requires normal reserved bits checking to ensure them input as zero, otherwise it will cause #GP. So we need mask out those reserved bits from the data written to vICR register.
OK. One open is:
Current KVM doesn't emulate this #GP. Is there any historical reason? if no, we will fix KVM and add some tests to verify this #GP is correctly emulated.
Quoted 19 lines
Remove Delivery Status bit emulation in test case as this flag is invalid and not needed in x2APIC mode. KVM may ignore clearing it during interrupt dispatch which will lead to fake test failure.
Opportunstically correct vector number for test sending IPI to non-existent vCPUs.
Signed-off-by Zeng Guang <[email protected]>---
.../selftests/kvm/x86_64/xapic_state_test.c| 20 ++++++++++++++++---1 file changed, 17 insertions(+), 3 deletions(-)tools/testing/selftests/kvm/x86_64/xapic_state_test.c +0
-0@@ -70,13 +70,27 @@ static void ____test_icr(struct kvm_vm *vm, struct kvm_vcpu *vcpu, uint64_t val)vcpu_ioctl (vm, vcpu->id,KVM_GET_LAPIC , &xapic);
tools/testing/selftests/kvm/x86_64/xapic_state_test.c static void ____test_icr(struct kvm_vm *vm, struct kvm_vcpu *vcpu, uint64_t val) · line 78
Quoted 10 lines
tools/testing/selftests/kvm/x86_64/xapic_state_test.c +4
-2icr = (u64)(*((u32 *)&xapic.regs[APIC_ICR ])) | (u64)(*((u32 *)&xapic.regs[APIC_ICR2 ])) <<32 ; -if (!vcpu->is_x2apic) +if (!vcpu->is_x2apic) { val &= (-1u | (0xffull << (32 +24 ))); -ASSERT_EQ (icr, val & ~APIC_ICR_BUSY ); +ASSERT_EQ (icr, val & ~APIC_ICR_BUSY ); + }else { +ASSERT_EQ (icr & ~APIC_ICR_BUSY , val & ~APIC_ICR_BUSY );
Probably add a comment for it would be better. E.g.,
APIC_ICR_BUSY is removed and not used when CPU is in x2APIC mode. It is undefined whether write 1 to this bit will be preserved. So, even KVM keeps this bit cleared in some cases even in x2apic mode, no guarantee that hardware (specifically, CPU ucode when Intel IPI virtualization enabled) will clear the bit. So, skip checking this bit.
Sean Christopherson
reviewer+Venkatesh
Quoted 7 lines · Chao Gao
Quoted 5 lines · Zeng Guang
Hardware would directly write x2APIC ICR register instead of software emulation in some circumstances, e.g when Intel IPI virtualization is enabled. This behavior requires normal reserved bits checking to ensure them input as zero, otherwise it will cause #GP. So we need mask out those reserved bits from the data written to vICR register.
Quoted 5 lines
OK. One open is:
Current KVM doesn't emulate this #GP. Is there any historical reason? if no, we will fix KVM and add some tests to verify this #GP is correctly emulated.
It's a bug. There are patches posted[*], but they need to be refreshed to fix a rebase goof.
Venkatesh, are you planning on sending a v3 soonish?
Quoted 14 lines · Sean Christopherson
+Venkatesh
Quoted 11 lines · Chao Gao
Quoted 5 lines · Zeng Guang
Hardware would directly write x2APIC ICR register instead of software emulation in some circumstances, e.g when Intel IPI virtualization is enabled. This behavior requires normal reserved bits checking to ensure them input as zero, otherwise it will cause #GP. So we need mask out those reserved bits from the data written to vICR register.
OK. One open is:
Current KVM doesn't emulate this #GP. Is there any historical reason? if no, we will fix KVM and add some tests to verify this #GP is correctly emulated.
Quoted 6 lines
It's a bug. There are patches posted[*], but they need to be refreshed to fix a rebase goof.
Venkatesh, are you planning on sending a v3 soonish?
This patch set doesn't emulate hardware behavior precisely . Actually #GP will happen only if any of reserved bit ( bit[31:20],bit[17:16],bit[13]) is 1-setting in x2apic mode. Other bits including bit[12] won't have any impact. For xapic mode, it doesn't have this restriction.
Quoted 11 lines · Chao
tools/testing/selftests/kvm/x86_64/xapic_state_test.c static void ____test_icr(struct kvm_vm *vm, struct kvm_vcpu *vcpu, uint64_t val) · line 77
Quoted 1 line · Zeng Guang
tools/testing/selftests/kvm/x86_64/xapic_state_test.c +1
-0+ASSERT_EQ (icr & ~APIC_ICR_BUSY , val & ~APIC_ICR_BUSY );Probably add a comment for it would be better. E.g.,
APIC_ICR_BUSYis removed and not used when CPU is in x2APIC mode. It is undefined whether write 1 to this bit will be preserved. So, even KVM keeps this bit cleared in some cases even in x2apic mode, no guarantee that hardware (specifically, CPU ucode when Intel IPI virtualization enabled) will clear the bit. So, skip checking this bit.
Hardware won't touch APIC_ICR_BUSY in x2apic mode. It totally depends on KVM to clear it or not if set for test purpose. While in Intel IPI virtualization case, KVM doesn't take care of this bit in vICR writes. So how about the comments as below:
APIC_ICR_BUSY is removed and not used when CPU is in x2APIC mode. KVM doesn't guarantee to clear this bit in some cases e.g. When Intel IPI virtualization enabled, if it's set for test purpose. So, skip checking this bit.
Thanks.
Zeng Guang
Quoted 7 lines · Zeng Guang
Quoted 5 lines · Chao
tools/testing/selftests/kvm/x86_64/xapic_state_test.c static void ____test_icr(struct kvm_vm *vm, struct kvm_vcpu *vcpu, uint64_t val) · line 77
Quoted 1 line · Zeng Guang
tools/testing/selftests/kvm/x86_64/xapic_state_test.c +1
-0+ASSERT_EQ (icr & ~APIC_ICR_BUSY , val & ~APIC_ICR_BUSY );Probably add a comment for it would be better. E.g.,
Quoted 7 lines
Quoted 6 lines
APIC_ICR_BUSYis removed and not used when CPU is in x2APIC mode. It is undefined whether write 1 to this bit will be preserved. So, even KVM keeps this bit cleared in some cases even in x2apic mode, no guarantee that hardware (specifically, CPU ucode when Intel IPI virtualization enabled) will clear the bit. So, skip checking this bit.Hardware won't touch
APIC_ICR_BUSYin x2apic mode.
IMO, SDM doesn't say how the processor deals with this bit in x2apic mode. Even if SPR behaves like this, the behavior isn't architectural. Otherwise, KVM shouldn't touch this bit and we can add a test to verify that the bit won't be changed by CPU (or KVM) in x2apic mode.
Quoted 3 lines
It totally depends on KVM to clear it or not if set for test purpose. While in Intel IPI virtualization case, KVM doesn't take care of this bit in vICR writes.
I don't think KVM behavior is the key problem here. If an IPI is virtualized by ucode, KVM isn't involved in processing the IPI. It means KVM has no chance to clear the APIC_ICR_BUSY bit.
Queued, thanks.
Paolo