ignore set_cr3 GP fault in non-pae mode
RFC PATCH 1/1 ignore set_cr3 GP fault in non-pae mode
Playing around with running VMware-server within a KVM guest and noticed that whenever we launch a VM within the guest, KVM reports a GP fault in set_cr3. Removing the fault injection (raised for attempting to set reserved bits) for the non-pae case allows memtest to boot and run within VMWare Server, running in a KVM Linux guest.
This same test (Linux, VMware-server, booting/running memtest iso) works fine on bare-metal. Thoughts?
Signature · 5 lines
Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 ryanh-r/[email protected]
This patch removes the fault injected when the guest attempts to set reserved bits in cr3. X86 hardware doesn't generate a fault when setting reserved bits. The result of this patch is that vmware-server, running within a kvm guest, boots and runs memtest from an iso.
drivers/kvm/kvm_main.c +0 -7
@@ -573,13 +573,6 @@ void set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3)inject_gp (vcpu);return ; } - }else { -if (cr3 &CR3_NONPAE_RESERVED_BITS ) { -printk (KERN_DEBUG -"set_cr3: #GP, reserved bits\n" ); -inject_gp (vcpu); -return ; - } } }
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
kvm-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-devel
A message that is not in the archive (sent off-list or not yet received): 20070918190516.GC7519-r/[email protected]
Quoted 9 lines · Ryan Harper
Playing around with running VMware-server within a KVM guest and noticed that whenever we launch a VM within the guest, KVM reports a GP fault in
set_cr3. Removing the fault injection (raised for attempting to set reserved bits) for the non-pae case allows memtest to boot and run within VMWare Server, running in a KVM Linux guest.This same test (Linux, VMware-server, booting/running memtest iso) works fine on bare-metal. Thoughts?
Setting reserved bits is different from setting MBZ bits since the behaviors undefined. If something as common as VMware is depending on being able to set a reserved bit then perhaps the right thing to do from KVM's perspective is to let it.
I'm curious if Zach or Jun have any comments about the right thing to do here.
Regards,
Anthony Liguori
Quoted 8 lines
------------------------------------------------------------------------
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
------------------------------------------------------------------------
Quoted 4 lines
_______________________________________________
kvm-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-devel
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 2 lines · Anthony Liguori
Quoted 1 line · Ryan Harper
Playing around with running VMware-server within a KVM guest and
noticed
Quoted 1 line
Quoted 1 line
that whenever we launch a VM within the guest, KVM reports a GP
fault in
Quoted 5 lines
Quoted 5 lines
set_cr3. Removing the fault injection (raised for attempting to set reserved bits) for the non-pae case allows memtest to boot and run within VMWare Server, running in a KVM Linux guest.This same test (Linux, VMware-server, booting/running memtest iso)
works
Quoted 6 lines
Quoted 2 lines
fine on bare-metal. Thoughts?
Setting reserved bits is different from setting MBZ bits since the behaviors undefined. If something as common as VMware is depending on being able to set a reserved bit then perhaps the right thing to do
from
Quoted 3 lines
KVM's perspective is to let it.
I'm curious if Zach or Jun have any comments about the right thing to
do
Quoted 2 lines
here.
As long as the guest is protected mode (unlike the long mode), the Intel spec does _not_ say that reserved bits checking is enforced for CR3. As far as I looked at the AMD spec, looks like #GP is caused even in protected mode... Does the test work for AMD systems?
Quoted 4 lines
Regards,
Anthony Liguori
Jun
---
Intel Open Source Technology Center
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
A message that is not in the archive (sent off-list or not yet received): 97D612E30E1F88419025B06CB4CF1BE1037F9013-1a9uaKK1+wJcIJlls4ac1rfspsVTdybXVpNB7YpNyf8@public.gmane.org
Quoted 22 lines · Nakajima, Jun
Quoted 2 lines · Anthony Liguori
Quoted 1 line · Ryan Harper
Playing around with running VMware-server within a KVM guest and
noticed
Quoted 1 line
Quoted 1 line
that whenever we launch a VM within the guest, KVM reports a GP
fault in
Quoted 5 lines
Quoted 5 lines
set_cr3. Removing the fault injection (raised for attempting to set reserved bits) for the non-pae case allows memtest to boot and run within VMWare Server, running in a KVM Linux guest.This same test (Linux, VMware-server, booting/running memtest iso)
works
Quoted 6 lines
Quoted 2 lines
fine on bare-metal. Thoughts?
Setting reserved bits is different from setting MBZ bits since the behaviors undefined. If something as common as VMware is depending on being able to set a reserved bit then perhaps the right thing to do
from
Quoted 3 lines
KVM's perspective is to let it.
I'm curious if Zach or Jun have any comments about the right thing to
Quoted 8 lines
do
Quoted 2 lines
here.
As long as the guest is protected mode (unlike the long mode), the Intel spec does _not_ say that reserved bits checking is enforced for CR3. As far as I looked at the AMD spec, looks like #GP is caused even in protected mode... Does the test work for AMD systems?
I ran my test on an AMD host.
Quoted 10 lines
Quoted 5 lines
Regards,
Anthony Liguori
Jun
---
Intel Open Source Technology Center
Signature · 10 lines
Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 ryanh-r/[email protected] ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
A message that is not in the archive (sent off-list or not yet received): 20070918212509.GE7519-r/[email protected]
Quoted 4 lines · Ryan Harper
Quoted 3 lines · Nakajima, Jun
Quoted 2 lines · Anthony Liguori
Quoted 1 line · Ryan Harper
Playing around with running VMware-server within a KVM guest and
noticed
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
that whenever we launch a VM within the guest, KVM reports a GP
fault in
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
set_cr3. Removing the fault injection (raised for attempting to
set
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
reserved bits) for the non-pae case allows memtest to boot and
run
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
within VMWare Server, running in a KVM Linux guest.
This same test (Linux, VMware-server, booting/running memtest
iso) works
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
fine on bare-metal. Thoughts?
Setting reserved bits is different from setting MBZ bits since the behaviors undefined. If something as common as VMware is
depending on
Quoted 1 line
Quoted 1 line
Quoted 1 line
being able to set a reserved bit then perhaps the right thing to
do from
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
KVM's perspective is to let it.
I'm curious if Zach or Jun have any comments about the right thing
to do
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
here.
As long as the guest is protected mode (unlike the long mode), the
Intel
Quoted 1 line
Quoted 1 line
spec does _not_ say that reserved bits checking is enforced for CR3.
As
Quoted 4 lines
Quoted 2 lines
far as I looked at the AMD spec, looks like #GP is caused even in protected mode... Does the test work for AMD systems?
I ran my test on an AMD host.
Well then the patch sounds fine to me.
Jun
---
Intel Open Source Technology Center
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Quoted 27 lines · Ryan Harper
Quoted 26 lines · Nakajima, Jun
Quoted 2 lines · Anthony Liguori
Quoted 1 line · Ryan Harper
Playing around with running VMware-server within a KVM guest and
noticed
Quoted 1 line
Quoted 1 line
that whenever we launch a VM within the guest, KVM reports a GP
fault in
Quoted 5 lines
Quoted 5 lines
set_cr3. Removing the fault injection (raised for attempting to set reserved bits) for the non-pae case allows memtest to boot and run within VMWare Server, running in a KVM Linux guest.This same test (Linux, VMware-server, booting/running memtest iso)
works
Quoted 6 lines
Quoted 2 lines
fine on bare-metal. Thoughts?
Setting reserved bits is different from setting MBZ bits since the behaviors undefined. If something as common as VMware is depending on being able to set a reserved bit then perhaps the right thing to do
from
Quoted 3 lines
KVM's perspective is to let it.
I'm curious if Zach or Jun have any comments about the right thing to
do
Quoted 2 lines
here.
Quoted 6 lines
Quoted 4 lines
As long as the guest is protected mode (unlike the long mode), the Intel spec does _not_ say that reserved bits checking is enforced for CR3. As far as I looked at the AMD spec, looks like #GP is caused even in protected mode... Does the test work for AMD systems?
I ran my test on an AMD host.
We have a test which verifies #GP is not caused by setting the bits on either AMD or Intel chips. "Stray" bits can get turned on in some cases when switching between 64-bit, PAE and non-PAE address modes.
Were you testing on a 64-bit host kernel?
Zach
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
A message that is not in the archive (sent off-list or not yet received): 1190152700.8353.14.camel-cxY/[email protected]
Quoted 33 lines · Zachary Amsden
Quoted 32 lines · Ryan Harper
Quoted 30 lines · Nakajima, Jun
Quoted 2 lines · Anthony Liguori
Quoted 1 line · Ryan Harper
Playing around with running VMware-server within a KVM guest and
noticed
Quoted 1 line
Quoted 1 line
that whenever we launch a VM within the guest, KVM reports a GP
fault in
Quoted 5 lines
Quoted 5 lines
set_cr3. Removing the fault injection (raised for attempting to set reserved bits) for the non-pae case allows memtest to boot and run within VMWare Server, running in a KVM Linux guest.This same test (Linux, VMware-server, booting/running memtest iso)
works
Quoted 6 lines
Quoted 2 lines
fine on bare-metal. Thoughts?
Setting reserved bits is different from setting MBZ bits since the behaviors undefined. If something as common as VMware is depending on being able to set a reserved bit then perhaps the right thing to do
from
Quoted 3 lines
KVM's perspective is to let it.
I'm curious if Zach or Jun have any comments about the right thing to
do
Quoted 2 lines
here.
As long as the guest is protected mode (unlike the long mode), the Intel spec does _not_ say that reserved bits checking is enforced for CR3. As far as I looked at the AMD spec, looks like #GP is caused even in protected mode... Does the test work for AMD systems?
Quoted 7 lines
Quoted 1 line
I ran my test on an AMD host.
We have a test which verifies #GP is not caused by setting the bits on either AMD or Intel chips. "Stray" bits can get turned on in some cases when switching between 64-bit, PAE and non-PAE address modes.
Were you testing on a 64-bit host kernel?
64-bit Host running 32-bit KVM Guest. VMware-server shouldn't be seeing anything 64-bit AFAIK.
Quoted 2 lines
Zach
Signature · 10 lines
Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 ryanh-r/[email protected] ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
A message that is not in the archive (sent off-list or not yet received): 20070918220308.GF7519-r/[email protected]
Quoted 11 lines · Ryan Harper
Quoted 7 lines
We have a test which verifies #GP is not caused by setting the bits on either AMD or Intel chips. "Stray" bits can get turned on in some cases when switching between 64-bit, PAE and non-PAE address modes.
Were you testing on a 64-bit host kernel?
64-bit Host running 32-bit KVM Guest. VMware-server shouldn't be seeing anything 64-bit AFAIK.
If the host is 64-bit, so is the guest, although the guest can choose to ignore the 64-bit capabilities (and 32-bit guest OSes do; but VMware might be doing something tricky).
Signature · 7 lines
error compiling committee.c: too many arguments to function ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
A message that is not in the archive (sent off-list or not yet received): 46F0E46C.1060905-atKUWr5tajBWk0Htik3J/[email protected]
Quoted 9 lines · Avi Kivity
Quoted 8 lines · Ryan Harper
Quoted 7 lines
We have a test which verifies #GP is not caused by setting the bits on either AMD or Intel chips. "Stray" bits can get turned on in some cases when switching between 64-bit, PAE and non-PAE address modes.
Were you testing on a 64-bit host kernel?
Quoted 7 lines
Quoted 3 lines
64-bit Host running 32-bit KVM Guest. VMware-server shouldn't be seeing anything 64-bit AFAIK.
If the host is 64-bit, so is the guest, although the guest can choose to ignore the 64-bit capabilities (and 32-bit guest OSes do; but VMware might be doing something tricky).
Hrm. I'm not sure I understand the distiction you are making. I'll give a 32-bit host a try without the patch and see if I see the same behavior.
Signature · 10 lines
Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 ryanh-r/[email protected] ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
A message that is not in the archive (sent off-list or not yet received): 20070919145536.GH7519-r/[email protected]
Quoted 15 lines · Ryan Harper
Quoted 13 lines · Avi Kivity
Quoted 11 lines · Ryan Harper
Quoted 7 lines
We have a test which verifies #GP is not caused by setting the bits on either AMD or Intel chips. "Stray" bits can get turned on in some cases when switching between 64-bit, PAE and non-PAE address modes.
Were you testing on a 64-bit host kernel?
64-bit Host running 32-bit KVM Guest. VMware-server shouldn't be seeing anything 64-bit AFAIK.
Quoted 9 lines
Quoted 4 lines
If the host is 64-bit, so is the guest, although the guest can choose to ignore the 64-bit capabilities (and 32-bit guest OSes do; but VMware might be doing something tricky).
Hrm. I'm not sure I understand the distiction you are making. I'll give a 32-bit host a try without the patch and see if I see the same behavior.
On a 64-bit host, even if the guest kernel is 32-bit the guest could still detect the ability to run in 64-bit mode. The thinking is that VMware may be detecting 64-bit capabilities and doing something weird. I think that's somewhat orthogonal to this patch. It seems like ignoring setting of reserved bits is the right thing to do in 32-bit mode. Does anyone disagree with that?
Regards,
Anthony Liguori
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 26 lines · Anthony Liguori
Quoted 15 lines · Ryan Harper
Quoted 13 lines · Avi Kivity
Quoted 11 lines · Ryan Harper
Quoted 7 lines
We have a test which verifies #GP is not caused by setting the bits on either AMD or Intel chips. "Stray" bits can get turned on in some cases when switching between 64-bit, PAE and non-PAE address modes.
Were you testing on a 64-bit host kernel?
64-bit Host running 32-bit KVM Guest. VMware-server shouldn't be seeing anything 64-bit AFAIK.
Quoted 9 lines
Quoted 4 lines
If the host is 64-bit, so is the guest, although the guest can choose to ignore the 64-bit capabilities (and 32-bit guest OSes do; but VMware might be doing something tricky).
Hrm. I'm not sure I understand the distiction you are making. I'll give a 32-bit host a try without the patch and see if I see the same behavior.
Quoted 6 lines
On a 64-bit host, even if the guest kernel is 32-bit the guest could still detect the ability to run in 64-bit mode. The thinking is that VMware may be detecting 64-bit capabilities and doing something weird. I think that's somewhat orthogonal to this patch. It seems like ignoring setting of reserved bits is the right thing to do in 32-bit mode. Does anyone disagree with that?
No, VMware does this in non-weird cases. It might not actually load a CR3 with reserved bits set (well, it might do that too in a different version of the code), but the standard path in and out of the VMware monitor has a couple of different ways you can effectively end up with reserved bits set (left as an exercise to the reader).
VMware will exhibit this behavior more frequently with 64-bit host, but a PAE / non-PAE mismatch between guest and host also causes the same issue.
Our software tests show that AMD and Intel processors ignore these reserved bits across the board and we've been relying on this behavior for some time.
BTW, VMware can run 64-bit guests on a 32-bit host, all it needs is capable hardware, so unless you mask the 64-bit support in KVM, you could have a strange situation where the 32-bit KVM starts (or tries to start, expecting success) running 64-bit code. I can imagine a couple of ways that could blow up the host if assumptions are made that 32-bit guests will limit themselves to running 32-bit code. So if you don't support EFER.LM activation, you should strip the cpuid bit from hardware.
Zach
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
A message that is not in the archive (sent off-list or not yet received): 1190217717.8353.29.camel-cxY/[email protected]
Quoted 9 lines · Zachary Amsden
BTW, VMware can run 64-bit guests on a 32-bit host, all it needs is capable hardware, so unless you mask the 64-bit support in KVM, you could have a strange situation where the 32-bit KVM starts (or tries to start, expecting success) running 64-bit code. I can imagine a couple of ways that could blow up the host if assumptions are made that 32-bit guests will limit themselves to running 32-bit code. So if you don't support EFER.LM activation, you should strip the cpuid bit from hardware.
We do that.
Signature · 7 lines
error compiling committee.c: too many arguments to function ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Quoted 14 lines · Ryan Harper
Quoted 13 lines · Avi Kivity
Quoted 11 lines · Ryan Harper
Quoted 7 lines
We have a test which verifies #GP is not caused by setting the bits on either AMD or Intel chips. "Stray" bits can get turned on in some cases when switching between 64-bit, PAE and non-PAE address modes.
Were you testing on a 64-bit host kernel?
64-bit Host running 32-bit KVM Guest. VMware-server shouldn't be seeing anything 64-bit AFAIK.
Quoted 7 lines
Quoted 3 lines
If the host is 64-bit, so is the guest, although the guest can choose to ignore the 64-bit capabilities (and 32-bit guest OSes do; but VMware might be doing something tricky).
Hrm. I'm not sure I understand the distiction you are making. I'll give a 32-bit host a try without the patch and see if I see the same behavior.
With, or without the patch, using 32-bit host, I get triple fault in the guest.
Signature · 10 lines
Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 ryanh-r/[email protected] ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Avi Kivity
AppliedQuoted 6 lines · Ryan Harper
Playing around with running VMware-server within a KVM guest and noticed that whenever we launch a VM within the guest, KVM reports a GP fault in
set_cr3. Removing the fault injection (raised for attempting to set reserved bits) for the non-pae case allows memtest to boot and run within VMWare Server, running in a KVM Linux guest.
Applied, thanks.
Signature · 7 lines
error compiling committee.c: too many arguments to function ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/