KVM: x86: Clear bit12 of ICR after APIC-write VM-exit
PATCH v2 1/1 KVM: x86: Clear bit12 of ICR after APIC-write VM-exit
When IPI virtualization is enabled, a WARN is triggered if bit12 of ICR MSR is set after APIC-write VM-exit. The reason is kvm_apic_send_ipi() thinks the APIC_ICR_BUSY bit should be cleared because KVM has no delay, but kvm_apic_write_nodecode() doesn't clear the APIC_ICR_BUSY bit.
Bit12 of ICR is different from other reserved bits(31:20, 17:16 and 13). When bit12 is set, it will cause APIC-wirte VM-exit but not #GP. For reading bit12 back as '0' which is a safer approach, clearing bit12 in x2APIC mode is needed.
Although bit12 of ICR is no longer APIC_ICR_BUSY in x2APIC, keeping it is far easier to understand what's going on, especially given that it may be repurposed for something new.
---
Changelog:
v2:
- Drop the unnecessary alias for bit12 of ICR.
- Add back kvm_lapic_get_reg64() that was removed by mistake.
- Modify the commit message to make it clearer.
v1: https://lore.kernel.org/all/[email protected]/
--- arch/x86/kvm/lapic.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
arch/x86/kvm/lapic.c +3 -3
@@ -2450,13 +2450,13 @@ void kvm_apic_write_nodecode(struct kvm_vcpu *vcpu, u32 offset)* ICR is a single 64-bit register when x2APIC is enabled. For legacy * xAPIC, ICR writes need to go down the common (slightly slower) path * to get the upper half from ICR2. +* +* TODO: optimize to just emulate side effect w/o one more write */ if (apic_x2apic_mode (apic) && offset ==APIC_ICR ) { val =kvm_lapic_get_reg64 (apic,APIC_ICR ); -kvm_apic_send_ipi (apic, (u32 )val, (u32)(val >>32 )); -trace_kvm_apic_write (APIC_ICR , val); +kvm_x2apic_icr_write (apic, val); }else { -/* TODO: optimize to just emulate side effect w/o one more write */ val =kvm_lapic_get_reg (apic, offset);kvm_lapic_reg_write (apic, offset, (u32 )val); }
base-commit: a48fa7efaf1161c1c898931fe4c7f0070964233a
Signature · 1 line
2.34.1
Quoted 7 lines · Tao Su
When IPI virtualization is enabled, a WARN is triggered if bit12 of ICR MSR is set after APIC-write VM-exit. The reason is
kvm_apic_send_ipi()thinks theAPIC_ICR_BUSYbit should be cleared because KVM has no delay, butkvm_apic_write_nodecode()doesn't clear theAPIC_ICR_BUSYbit.Bit12 of ICR is different from other reserved
bits(31:20, 17:16 and 13). When bit12 is set, it will cause APIC-wirte VM-exit but not #GP. For
s/wirte/write
Quoted 2 lines
reading bit12 back as '0' which is a safer approach, clearing bit12 in x2APIC mode is needed.
how about quoting what Sean said:
(w/ a slight change to the last sentence)
Under the x2APIC section, regarding ICR, the SDM says:
It remains readable only to aid in debugging; however, software should not assume the value returned by reading the ICR is the last written value.
I.e. KVM basically has free reign to do whatever it wants, so long as it doesn't confuse userspace or break KVM's ABI.
Clear bit12 so that it reads back as '0'. This approach is safer than "do nothing" and is consistent with the case where IPI virtualization is disabled or not supported, i.e.,
handle_fastpath_set_x2apic_icr_irqoff() -> kvm_x2apic_icr_write()
Quoted 4 lines
Although bit12 of ICR is no longer
APIC_ICR_BUSYin x2APIC, keeping it is far easier to understand what's going on, especially given that it may be repurposed for something new.
Probably you can remove this paragraph. it is not clear w/o the context that there was an attempt to rename APIC_ICR_BUSY for x2apic while fixing the issue.
Quoted 5 lines
Fixes 5413bcba7ed5 ("KVM: x86: Add support for vICR APIC-write VM-Exits in x2APIC mode")Signed-off-by Tao Su <[email protected]>Tested-by Yi Lai <[email protected]>
Apart from above nits on the changelog, this patch looks good to me.
Quoted 39 lines
---
Changelog:v2:
- Drop the unnecessary alias for bit12 of ICR.
- Add backkvm_lapic_get_reg64()that was removed by mistake.
- Modify the commit message to make it clearer.v1: https://lore.kernel.org/all/[email protected]/
---arch/x86/kvm/lapic.c| 6 +++---1 file changed, 3 insertions(+), 3 deletions(-)arch/x86/kvm/lapic.c +3
-3@@ -2450,13 +2450,13 @@ void kvm_apic_write_nodecode(struct kvm_vcpu *vcpu, u32 offset)* ICR is a single 64-bit register when x2APIC is enabled. For legacy * xAPIC, ICR writes need to go down the common (slightly slower) path * to get the upper half from ICR2. +* +* TODO: optimize to just emulate side effect w/o one more write */ if (apic_x2apic_mode (apic) && offset ==APIC_ICR ) { val =kvm_lapic_get_reg64 (apic,APIC_ICR ); -kvm_apic_send_ipi (apic, (u32 )val, (u32)(val >>32 )); -trace_kvm_apic_write (APIC_ICR , val); +kvm_x2apic_icr_write (apic, val); }else { -/* TODO: optimize to just emulate side effect w/o one more write */ val =kvm_lapic_get_reg (apic, offset);kvm_lapic_reg_write (apic, offset, (u32 )val); }base-commit:
a48fa7efaf1161c1c898931fe4c7f0070964233aSignature · 1 line
2.34.1
Quoted 32 lines · Chao Gao
Quoted 7 lines · Tao Su
When IPI virtualization is enabled, a WARN is triggered if bit12 of ICR MSR is set after APIC-write VM-exit. The reason is
kvm_apic_send_ipi()thinks theAPIC_ICR_BUSYbit should be cleared because KVM has no delay, butkvm_apic_write_nodecode()doesn't clear theAPIC_ICR_BUSYbit.Bit12 of ICR is different from other reserved
bits(31:20, 17:16 and 13). When bit12 is set, it will cause APIC-wirte VM-exit but not #GP. Fors/wirte/write
Quoted 2 lines
reading bit12 back as '0' which is a safer approach, clearing bit12 in x2APIC mode is needed.
how about quoting what Sean said:
(w/ a slight change to the last sentence)Under the x2APIC section, regarding ICR, the SDM says:
It remains readable only to aid in debugging; however, software should not assume the value returned by reading the ICR is the last written value.
I.e. KVM basically has free reign to do whatever it wants, so long as it doesn't confuse userspace or break KVM's ABI.
Clear bit12 so that it reads back as '0'. This approach is safer than "do nothing" and is consistent with the case where IPI virtualization is disabled or not supported, i.e.,
handle_fastpath_set_x2apic_icr_irqoff() -> kvm_x2apic_icr_write()Quoted 1 line
Quoted 7 lines
Quoted 3 lines
Although bit12 of ICR is no longer
APIC_ICR_BUSYin x2APIC, keeping it is far easier to understand what's going on, especially given that it may be repurposed for something new.Probably you can remove this paragraph. it is not clear w/o the context that there was an attempt to rename
APIC_ICR_BUSYfor x2apic while fixing the issue.
Yes, agree with all the above, this is more correct and clear description.
Thanks,
Tao
Quoted 50 lines
Quoted 6 lines
Fixes 5413bcba7ed5 ("KVM: x86: Add support for vICR APIC-write VM-Exits in x2APIC mode")Signed-off-by Tao Su <[email protected]>Tested-by Yi Lai <[email protected]>Apart from above nits on the changelog, this patch looks good to me.
Reviewed-by Chao Gao <[email protected]>Quoted 39 lines
---
Changelog:v2:
- Drop the unnecessary alias for bit12 of ICR.
- Add backkvm_lapic_get_reg64()that was removed by mistake.
- Modify the commit message to make it clearer.v1: https://lore.kernel.org/all/[email protected]/
---arch/x86/kvm/lapic.c| 6 +++---1 file changed, 3 insertions(+), 3 deletions(-)arch/x86/kvm/lapic.c +3
-3@@ -2450,13 +2450,13 @@ void kvm_apic_write_nodecode(struct kvm_vcpu *vcpu, u32 offset)* ICR is a single 64-bit register when x2APIC is enabled. For legacy * xAPIC, ICR writes need to go down the common (slightly slower) path * to get the upper half from ICR2. +* +* TODO: optimize to just emulate side effect w/o one more write */ if (apic_x2apic_mode (apic) && offset ==APIC_ICR ) { val =kvm_lapic_get_reg64 (apic,APIC_ICR ); -kvm_apic_send_ipi (apic, (u32 )val, (u32)(val >>32 )); -trace_kvm_apic_write (APIC_ICR , val); +kvm_x2apic_icr_write (apic, val); }else { -/* TODO: optimize to just emulate side effect w/o one more write */ val =kvm_lapic_get_reg (apic, offset);kvm_lapic_reg_write (apic, offset, (u32 )val); }base-commit:
a48fa7efaf1161c1c898931fe4c7f0070964233aSignature · 1 line
2.34.1