target/i386: Mask xstate_bv based on the cpu enabled features
PATCH 1/1 target/i386: Mask xstate_bv based on the cpu enabled features
The following steps describe a migration bug:
1 - Bring up a VM with -cpu EPYC on a host with EPYC-Milan cpu 2 - Migrate to a host with EPYC-Naples cpu
The guest kernel crashes shortly after the migration.
The crash happens due to a fault caused by XRSTOR:
A set bit in XSTATE_BV is not set in XCR0.
The faulting bit is FEATURE_PKRU (enabled in Milan, but not in Naples)
To avoid this kind of bug:
In kvm_get_xsave, mask-out from xstate_bv any bits that are not set in current vcpu's features.
This keeps cpu->env->xstate_bv with feature bits compatible with any host machine capable of running the vcpu model.
--- target/i386/xsave_helper.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
target/i386/xsave_helper.c +1 -1
@@ -167,7 +167,7 @@ void x86_cpu_xrstor_all_areas(X86CPU *cpu, const void *buf, uint32_t buflen) env->xmm_regs[i].ZMM_Q (1 ) =ldq_p (xmm +8 ); } - env->xstate_bv = header->xstate_bv; + env->xstate_bv = header->xstate_bv & env->features[FEAT_XSAVE_COMP_LO ]; e = &x86_ext_save_areas[XSTATE_YMM_BIT ];if (e->size && e->offset) {
Signature · 1 line
2.34.1
Quoted 9 lines · Leonardo Bras
The following steps describe a migration bug:
1 - Bring up a VM with -cpu EPYC on a host with EPYC-Milan cpu 2 - Migrate to a host with EPYC-Naples cpuThe guest kernel crashes shortly after the migration.
The crash happens due to a fault caused by XRSTOR:
A set bit inXSTATE_BVis not set in XCR0.
The faulting bit isFEATURE_PKRU(enabled in Milan, but not in Naples)
I'm trying to understand how this happens.
If we boot on EPYC-Milan with "-cpu EPYC", the PKRU feature should not be exposed to the VM (it is not available in the EPYC CPU).
Given this, how would bit 0x200 (representing PKRU) end up set in xstate_bv?
Quoted 17 lines
To avoid this kind of bug:
Inkvm_get_xsave, mask-out fromxstate_bvany bits that are not set in current vcpu's features.This keeps
cpu->env->xstate_bvwith feature bits compatible with any host machine capable of running the vcpu model.Signed-off-by Leonardo Bras <[email protected]>--- target/i386/xsave_helper.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)target/i386/xsave_helper.c +0
-0
target/i386/xsave_helper.c void x86_cpu_xrstor_all_areas(X86CPU *cpu, const void *buf, uint32_t buflen) · line 173
Quoted 8 lines
target/i386/xsave_helper.c +1
-1@@ -167,7 +167,7 @@ void x86_cpu_xrstor_all_areas(X86CPU *cpu, const void *buf, uint32_t buflen) env->xmm_regs[i].ZMM_Q (1 ) =ldq_p (xmm +8 ); } - env->xstate_bv = header->xstate_bv; + env->xstate_bv = header->xstate_bv & env->features[FEAT_XSAVE_COMP_LO ]; e = &x86_ext_save_areas[XSTATE_YMM_BIT ];if (e->size && e->offset) {
dme.
Signature · 1 line
You have underestimated my power, as you shortly will discover.
Quoted 12 lines · On Mon, 31 Jan 2022 12:53:31 +0000 David Edmondson…
Quoted 9 lines · Leonardo Bras
The following steps describe a migration bug:
1 - Bring up a VM with -cpu EPYC on a host with EPYC-Milan cpu 2 - Migrate to a host with EPYC-Naples cpuThe guest kernel crashes shortly after the migration.
The crash happens due to a fault caused by XRSTOR:
A set bit inXSTATE_BVis not set in XCR0.
The faulting bit isFEATURE_PKRU(enabled in Milan, but not in Naples)
Quoted 11 lines
I'm trying to understand how this happens.
If we boot on EPYC-Milan with "-cpu EPYC", the PKRU feature should not be exposed to the VM (it is not available in the EPYC CPU).
Given this, how would bit
0x200(representing PKRU) end up set inxstate_bv?Quoted 3 lines
To avoid this kind of bug:
Inkvm_get_xsave, mask-out fromxstate_bvany bits that are not set in current vcpu's features.
In addition to above:
it's not good idea to silently mask something out. If we can't ensure the same feature-set for a CPU model and can't verify it by asking QEMU on source and target host, the next best thing would be to explicitly fail migration (i.e. adding check to.post_load hook or doing some other migration magic, CCing David)
Quoted 24 lines
Quoted 14 lines
This keeps
cpu->env->xstate_bvwith feature bits compatible with any host machine capable of running the vcpu model.Signed-off-by Leonardo Bras <[email protected]>--- target/i386/xsave_helper.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)target/i386/xsave_helper.c +0
-0target/i386/xsave_helper.c void x86_cpu_xrstor_all_areas(X86CPU *cpu, const void *buf, uint32_t buflen) · line 173
Quoted 8 lines
target/i386/xsave_helper.c +1
-1@@ -167,7 +167,7 @@ void x86_cpu_xrstor_all_areas(X86CPU *cpu, const void *buf, uint32_t buflen) env->xmm_regs[i].ZMM_Q (1 ) =ldq_p (xmm +8 ); } - env->xstate_bv = header->xstate_bv; + env->xstate_bv = header->xstate_bv & env->features[FEAT_XSAVE_COMP_LO ]; e = &x86_ext_save_areas[XSTATE_YMM_BIT ];if (e->size && e->offset) {dme.
Hello Igor,
Quoted 30 lines · Igor Mammedov
Quoted 13 lines · On Mon, 31 Jan 2022 12:53:31 +0000 David Edmondson…
Quoted 10 lines · Leonardo Bras
The following steps describe a migration bug:
1 - Bring up a VM with -cpu EPYC on a host with EPYC-Milan cpu 2 - Migrate to a host with EPYC-Naples cpuThe guest kernel crashes shortly after the migration.
The crash happens due to a fault caused by XRSTOR:
A set bit inXSTATE_BVis not set in XCR0.
The faulting bit isFEATURE_PKRU(enabled in Milan, but not in Naples)Quoted 13 lines
I'm trying to understand how this happens.
If we boot on EPYC-Milan with "-cpu EPYC", the PKRU feature should not
be exposed to the VM (it is not available in the EPYC CPU).Given this, how would bit
0x200(representing PKRU) end up set inxstate_bv?Quoted 4 lines
To avoid this kind of bug:
Inkvm_get_xsave, mask-out fromxstate_bvany bits that are not set in
current vcpu's features.
Quoted 8 lines
In addition to above:
it's not good idea to silently mask something out. If we can't ensure the same feature-set for a CPU model and can't verify it by asking QEMU on source and target host, the next best thing would be to explicitly fail migration (i.e. adding check to.post_load hook or doing some other migration magic, CCing David)
Maybe there is something to do with the host kernel (kvm) doing some strange stuff.
IIRC qemu ended up getting some masked version for using on migration, since it was not failing as expected.
I will try to investigate further.
Please let me know if you have any information on that.
Best regards,
Leo
Quoted 31 lines
Quoted 29 lines
Quoted 25 lines
This keeps
cpu->env->xstate_bvwith feature bits compatible with any
host machine capable of running the vcpu model.Signed-off-by Leonardo Bras <[email protected]>---
target/i386/xsave_helper.c | 2 +-1 file changed, 1 insertion(+), 1 deletion(-)target/i386/xsave_helper.c +1
-1@@ -167,7 +167,7 @@ void x86_cpu_xrstor_all_areas(X86CPU *cpu, const void *buf, uint32_t buflen) env->xmm_regs[i].ZMM_Q(1) = ldq_p(xmm + 8); } - env->xstate_bv = header->xstate_bv; + env->xstate_bv = header->xstate_bv & env- >features[FEAT_XSAVE_COMP_LO];Quoted 2 lines
e = &x86_ext_save_areas[XSTATE_YMM_BIT];
if (e->size&&e->offset) {dme.
Hello David, thanks for this feedback!
Quoted 13 lines · David Edmondson
Quoted 10 lines · Leonardo Bras
The following steps describe a migration bug:
1 - Bring up a VM with -cpu EPYC on a host with EPYC-Milan cpu 2 - Migrate to a host with EPYC-Naples cpuThe guest kernel crashes shortly after the migration.
The crash happens due to a fault caused by XRSTOR:
A set bit inXSTATE_BVis not set in XCR0.
The faulting bit isFEATURE_PKRU(enabled in Milan, but not in Naples)
Quoted 8 lines
I'm trying to understand how this happens.
If we boot on EPYC-Milan with "-cpu EPYC", the PKRU feature should not
be exposed to the VM (it is not available in the EPYC CPU).Given this, how would bit
0x200(representing PKRU) end up set inxstate_bv?
During my debug, I noticed this bit gets set before the kernel even starts.
It's possible Seabios and/or IPXE are somehow setting 0x200 using the xrstor command. I am not sure if qemu is able to stop this in KVM mode.
If you have any info on this, please let me know.
Best regards,
Leo
Quoted 33 lines
Quoted 29 lines
To avoid this kind of bug:
Inkvm_get_xsave, mask-out fromxstate_bvany bits that are not set in
current vcpu's features.This keeps
cpu->env->xstate_bvwith feature bits compatible with any
host machine capable of running the vcpu model.Signed-off-by Leonardo Bras <[email protected]>---
target/i386/xsave_helper.c | 2 +-1 file changed, 1 insertion(+), 1 deletion(-)target/i386/xsave_helper.c +1
-1@@ -167,7 +167,7 @@ void x86_cpu_xrstor_all_areas(X86CPU *cpu, const void *buf, uint32_t buflen) env->xmm_regs[i].ZMM_Q(1) = ldq_p(xmm + 8); } - env->xstate_bv = header->xstate_bv; + env->xstate_bv = header->xstate_bv & env- >features[FEAT_XSAVE_COMP_LO];Quoted 2 lines
e = &x86_ext_save_areas[XSTATE_YMM_BIT];
if (e->size&&e->offset) {dme.
Quoted 22 lines · Leonardo Brás
Hello David, thanks for this feedback!
Quoted 19 lines · David Edmondson
Quoted 10 lines · Leonardo Bras
The following steps describe a migration bug:
1 - Bring up a VM with -cpu EPYC on a host with EPYC-Milan cpu 2 - Migrate to a host with EPYC-Naples cpuThe guest kernel crashes shortly after the migration.
The crash happens due to a fault caused by XRSTOR:
A set bit inXSTATE_BVis not set in XCR0.
The faulting bit isFEATURE_PKRU(enabled in Milan, but not in Naples)I'm trying to understand how this happens.
If we boot on EPYC-Milan with "-cpu EPYC", the PKRU feature should not
be exposed to the VM (it is not available in the EPYC CPU).
Quoted 8 lines
Quoted 2 lines
Given this, how would bit
0x200(representing PKRU) end up set inxstate_bv?During my debug, I noticed this bit gets set before the kernel even starts.
It's possible Seabios and/or IPXE are somehow setting
0x200using the xrstor command. I am not sure if qemu is able to stop this in KVM mode.
I don't believe that this should be possible.
If the CPU is set to EPYC in QEMU then .features[FEAT_7_0_ECX] does not include CPUID_7_0_ECX_PKU, which in turn means that when x86_cpu_enable_xsave_components() generates FEAT_XSAVE_COMP_LO it should not set XSTATE_PKRU_BIT.
Given that, KVM's vcpu->arch.guest_supported_xcr0 will not include XSTATE_PKRU_BIT, and __kvm_set_xcr() should not allow that bit to be set when it intercepts the guest xsetbv instruction.
dme.
Signature · 1 line
Please forgive me if I act a little strange, for I know not what I do.
Hello David, thank you for the feedback.
Quoted 36 lines · David Edmondson
Quoted 22 lines · Leonardo Brás
Hello David, thanks for this feedback!
Quoted 19 lines · David Edmondson
Quoted 10 lines · Leonardo Bras
The following steps describe a migration bug:
1 - Bring up a VM with -cpu EPYC on a host with EPYC-Milan cpu 2 - Migrate to a host with EPYC-Naples cpuThe guest kernel crashes shortly after the migration.
The crash happens due to a fault caused by XRSTOR:
A set bit inXSTATE_BVis not set in XCR0.
The faulting bit isFEATURE_PKRU(enabled in Milan, but not in Naples)I'm trying to understand how this happens.
If we boot on EPYC-Milan with "-cpu EPYC", the PKRU feature should not
be exposed to the VM (it is not available in the EPYC CPU).Quoted 8 lines
Quoted 2 lines
Given this, how would bit
0x200(representing PKRU) end up set inxstate_bv?During my debug, I noticed this bit gets set before the kernel even starts.
It's possible Seabios and/or IPXE are somehow setting
0x200using the xrstor command. I am not sure if qemu is able to stop this in KVM mode.I don't believe that this should be possible.
Quoted 8 lines
If the CPU is set to EPYC in QEMU then .features[FEAT_7_0_ECX] does not include
CPUID_7_0_ECX_PKU, which in turn means that whenx86_cpu_enable_xsave_components()generatesFEAT_XSAVE_COMP_LOit should not setXSTATE_PKRU_BIT.Given that, KVM's
vcpu->arch.guest_supported_xcr0will not includeXSTATE_PKRU_BIT, and__kvm_set_xcr()should not allow that bit to be set when it intercepts the guest xsetbv instruction.
Thanks for sharing those details, it helped me on the kernel side of this bug.
FWIW, i did send a patchset fixing this bug to kernel list:
https://patchwork.kernel.org/project/kvm/list/?series=611524&state=%2A&archive=both
Best regards,
Leo
Hello David Edmondson and Igor Memmedov,
Thank you for the feedback!
For some reason I did not get your comments in my email. I could only notice them when I opened Patchwork to get the link.
Sorry for the delay. I will do my best to address them in a few minutes.
Best regards,
Leo
Quoted 39 lines · Leonardo Bras
The following steps describe a migration bug:
1 - Bring up a VM with -cpu EPYC on a host with EPYC-Milan cpu 2 - Migrate to a host with EPYC-Naples cpuThe guest kernel crashes shortly after the migration.
The crash happens due to a fault caused by XRSTOR:
A set bit inXSTATE_BVis not set in XCR0.
The faulting bit isFEATURE_PKRU(enabled in Milan, but not in Naples)To avoid this kind of bug:
Inkvm_get_xsave, mask-out fromxstate_bvany bits that are not set in current vcpu's features.This keeps
cpu->env->xstate_bvwith feature bits compatible with any host machine capable of running the vcpu model.Signed-off-by Leonardo Bras <[email protected]>--- target/i386/xsave_helper.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)target/i386/xsave_helper.c +1
-2@@ -167,7 +167,7 @@ void x86_cpu_xrstor_all_areas(X86CPU *cpu, const void *buf, uint32_t buflen) env->xmm_regs[i].ZMM_Q (1 ) =ldq_p (xmm +8 ); } - env->xstate_bv = header->xstate_bv; + env->xstate_bv = header->xstate_bv & env->features[FEAT_XSAVE_COMP_LO ]; e = &x86_ext_save_areas[XSTATE_YMM_BIT ];if (e->size && e->offset) { --2.34.1