How does kvm support x2apic?

DISCUSSION Zhang Haoyu 2 participants 12 messages Jun 11, 2014 – Jun 12, 2014 KVM

Share & tools

Zhang Haoyu

author

2014-06-11 01:34 UTC #0 lore raw

Quoted 9 lines

Hi,
According to this:
https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd
It looks like kvm have been supporting x2apic since kernel 2.6.32, or even earlier.

This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio.

Quoted 6 lines

However, this following patch:
https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5
Also claims that adding support for x2apic. But this later patch was for kernel 3.9.

This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps, you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit, which belongs to APIC virtualization from certain angle.

Quoted 6 lines

So I am very confused:
First, what's the difference between these two patches? Or say, does kvm support x2apic since kernel 2.6.32, or since kernel 3.9?

kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9.

Quoted 5 lines

Second, can guest use x2apic even if the host does not? (Assuming qemu has exposed this feature to guest.) The word "use" means something like, accessing x2apic registers, setting the x2apic enable bit in IA32_APICBASE MSR (i.e. bit 10).

Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO. But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic, please see enable_IR_x2apic().

Quoted 1 line

-Jidong

Jidong Xiao

2014-06-11 03:54 UTC #1 lore raw

Quoted 13 lines · Zhang Haoyu

Quoted 9 lines
Hi,
According to this:
https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd
It looks like kvm have been supporting x2apic since kernel 2.6.32, or even earlier.
This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio.
Quoted 2 lines
However, this following patch:

Quoted 7 lines

Quoted 4 lines
https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5
Also claims that adding support for x2apic. But this later patch was for kernel 3.9.
This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps, you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit, which belongs to APIC virtualization from certain angle.

Thanks Haoyu, I am reading the Intel x2apic manual.

http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf

but I don't see the so called "virtual x2apic mode", it seems that the manual does not mention anything about that. Do you mean that when the Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't this the same as the aforementioned "emulate x2apic for guest"? Or, "emulate x2apic for guest" is the foundation of "support virtual x2apic mode"?

-Jidong

Quoted 19 lines

Quoted 6 lines
So I am very confused:
First, what's the difference between these two patches? Or say, does kvm support x2apic since kernel 2.6.32, or since kernel 3.9?
kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9.
Quoted 5 lines
Second, can guest use x2apic even if the host does not? (Assuming qemu has exposed this feature to guest.) The word "use" means something like, accessing x2apic registers, setting the x2apic enable bit in IA32_APICBASE MSR (i.e. bit 10).
Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO. But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic, please see enable_IR_x2apic().
Quoted 1 line
-Jidong

Jidong Xiao

2014-06-11 04:15 UTC #2 lore raw

Quoted 33 lines · Jidong Xiao

Quoted 13 lines · Zhang Haoyu
Quoted 9 lines
Hi,
According to this:
https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd
It looks like kvm have been supporting x2apic since kernel 2.6.32, or even earlier.
This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio.
Quoted 2 lines
However, this following patch:
Quoted 7 lines
Quoted 4 lines
https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5
Also claims that adding support for x2apic. But this later patch was for kernel 3.9.
This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps, you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit, which belongs to APIC virtualization from certain angle.
Thanks Haoyu, I am reading the Intel x2apic manual.
http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf
but I don't see the so called "virtual x2apic mode", it seems that the manual does not mention anything about that. Do you mean that when the Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't this the same as the aforementioned "emulate x2apic for guest"? Or, "emulate x2apic for guest" is the foundation of "support virtual x2apic mode"?

Quoted 2 lines

-Jidong

Oh, I found the answer, the virtualized x2apic mode is not defined in the x2apic manual, but is defined in the intel SDM manual, chapter 29
- APIC virtualization and virtual interrupts. Ignore my questions please, but thank you anyway.

-Jidong

Quoted 19 lines

Quoted 19 lines
Quoted 6 lines
So I am very confused:
First, what's the difference between these two patches? Or say, does kvm support x2apic since kernel 2.6.32, or since kernel 3.9?
kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9.
Quoted 5 lines
Second, can guest use x2apic even if the host does not? (Assuming qemu has exposed this feature to guest.) The word "use" means something like, accessing x2apic registers, setting the x2apic enable bit in IA32_APICBASE MSR (i.e. bit 10).
Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO. But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic, please see enable_IR_x2apic().
Quoted 1 line
-Jidong

Zhang Haoyu

author

2014-06-11 04:54 UTC #3 lore raw

Quoted 32 lines

Quoted 32 lines
Quoted 13 lines
Quoted 9 lines
Hi,
According to this:
https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd
It looks like kvm have been supporting x2apic since kernel 2.6.32, or even earlier.
This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio.
Quoted 2 lines
However, this following patch:
Quoted 7 lines
Quoted 4 lines
https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5
Also claims that adding support for x2apic. But this later patch was for kernel 3.9.
This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps, you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit, which belongs to APIC virtualization from certain angle.
Thanks Haoyu, I am reading the Intel x2apic manual.
http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf
but I don't see the so called "virtual x2apic mode", it seems that the manual does not mention anything about that. Do you mean that when the Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't this the same as the aforementioned "emulate x2apic for guest"? Or, "emulate x2apic for guest" is the foundation of "support virtual x2apic mode"?

Quoted 7 lines

Quoted 2 lines
-Jidong
Oh, I found the answer, the virtualized x2apic mode is not defined in the x2apic manual, but is defined in the intel SDM manual, chapter 29
- APIC virtualization and virtual interrupts. Ignore my questions please, but thank you anyway.

Yes, virtual x2apic mode is part of APICv.
Guest has no idea about it is running in a virtual machine,
VMM also prevent guest from enabling/disabling virtual x2apic mode or some other virtualization configurations.

Quoted 20 lines

-Jidong
Quoted 18 lines
Quoted 18 lines
Quoted 6 lines
So I am very confused:
First, what's the difference between these two patches? Or say, does kvm support x2apic since kernel 2.6.32, or since kernel 3.9?
kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9.
Quoted 5 lines
Second, can guest use x2apic even if the host does not? (Assuming qemu has exposed this feature to guest.) The word "use" means something like, accessing x2apic registers, setting the x2apic enable bit in IA32_APICBASE MSR (i.e. bit 10).
Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO. But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic, please see enable_IR_x2apic().
Quoted 1 line
-Jidong

Jidong Xiao

2014-06-11 05:08 UTC #4 lore raw

Quoted 34 lines · Zhang Haoyu

Quoted 34 lines
Quoted 34 lines
Quoted 13 lines
Quoted 9 lines
Hi,
According to this:
https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd
It looks like kvm have been supporting x2apic since kernel 2.6.32, or even earlier.
This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio.
Quoted 2 lines
However, this following patch:
Quoted 7 lines
Quoted 4 lines
https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5
Also claims that adding support for x2apic. But this later patch was for kernel 3.9.
This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps, you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit, which belongs to APIC virtualization from certain angle.
Thanks Haoyu, I am reading the Intel x2apic manual.
http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf
but I don't see the so called "virtual x2apic mode", it seems that the manual does not mention anything about that. Do you mean that when the Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't this the same as the aforementioned "emulate x2apic for guest"? Or, "emulate x2apic for guest" is the foundation of "support virtual x2apic mode"?
-Jidong

Quoted 9 lines

Quoted 5 lines
Oh, I found the answer, the virtualized x2apic mode is not defined in the x2apic manual, but is defined in the intel SDM manual, chapter 29
- APIC virtualization and virtual interrupts. Ignore my questions please, but thank you anyway.
Yes, virtual x2apic mode is part of APICv.
Guest has no idea about it is running in a virtual machine,
VMM also prevent guest from enabling/disabling virtual x2apic mode or some other virtualization configurations.

Regarding this "VMM also prevent guest from enabling/disabling virtual x2apic mode", do you mean that guest is not allowed to do something like:

Assuming we are in the guest, and we see the original value of the IA32_APICBASE MSR is 0xfee00900, which means EN bit and BSP bit was enabled. And now, we try to set bit 10 (EXTD bit):

wrmsr 0x1b 0xfee00d00.

Do you mean this wrmsr operation is not allowed? If fact, I tried this, and this simply crashes the guest kernel. (I am using Linux 2.6.34 as the guest kernel, and Linux 3.14 as the host kernel.) And this is the exact reason that I initialized this whole discussion. My feeling is this might be a kvm bug, so I wish to figure out and try to fix it.

Quoted 19 lines

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 6 lines
So I am very confused:
First, what's the difference between these two patches? Or say, does kvm support x2apic since kernel 2.6.32, or since kernel 3.9?
kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9.
Quoted 5 lines
Second, can guest use x2apic even if the host does not? (Assuming qemu has exposed this feature to guest.) The word "use" means something like, accessing x2apic registers, setting the x2apic enable bit in IA32_APICBASE MSR (i.e. bit 10).
Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO. But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic, please see enable_IR_x2apic().
Quoted 1 line
-Jidong

Jidong Xiao

2014-06-11 05:30 UTC #5 lore raw

Quoted 53 lines · Jidong Xiao

Quoted 34 lines · Zhang Haoyu
Quoted 34 lines
Quoted 34 lines
Quoted 13 lines
Quoted 9 lines
Hi,
According to this:
https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd
It looks like kvm have been supporting x2apic since kernel 2.6.32, or even earlier.
This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio.
Quoted 2 lines
However, this following patch:
Quoted 7 lines
Quoted 4 lines
https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5
Also claims that adding support for x2apic. But this later patch was for kernel 3.9.
This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps, you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit, which belongs to APIC virtualization from certain angle.
Thanks Haoyu, I am reading the Intel x2apic manual.
http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf
but I don't see the so called "virtual x2apic mode", it seems that the manual does not mention anything about that. Do you mean that when the Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't this the same as the aforementioned "emulate x2apic for guest"? Or, "emulate x2apic for guest" is the foundation of "support virtual x2apic mode"?
-Jidong
Quoted 9 lines
Quoted 5 lines
Oh, I found the answer, the virtualized x2apic mode is not defined in the x2apic manual, but is defined in the intel SDM manual, chapter 29
- APIC virtualization and virtual interrupts. Ignore my questions please, but thank you anyway.
Yes, virtual x2apic mode is part of APICv.
Guest has no idea about it is running in a virtual machine,
VMM also prevent guest from enabling/disabling virtual x2apic mode or some other virtualization configurations.
Regarding this "VMM also prevent guest from enabling/disabling virtual x2apic mode", do you mean that guest is not allowed to do something like:
Assuming we are in the guest, and we see the original value of the IA32_APICBASE MSR is 0xfee00900, which means EN bit and BSP bit was enabled. And now, we try to set bit 10 (EXTD bit):

Quoted 9 lines

wrmsr 0x1b 0xfee00d00.
Do you mean this wrmsr operation is not allowed? If fact, I tried this, and this simply crashes the guest kernel. (I am using Linux 2.6.34 as the guest kernel, and Linux 3.14 as the host kernel.) And this is the exact reason that I initialized this whole discussion. My feeling is this might be a kvm bug, so I wish to figure out and try to fix it.

Can someone test this also? Just set bit 10 of IA32_APICBASE MSR in your guest, and see if the guest crashes. According to Intel SDM manual, this is the approach to enable x2apic mode. And I think normally it should not crash the guest kernel.

One can write that msr with the following tool:

https://01.org/msr-tools

And compile this tool and just run the aforementioned command:

wrmsr 0x1b 0xfee00d00 (rdmsr 0x1b to confirm the original value is 0xfee00900.)

I tried on several machines, and this command just crash the kernel. And I am not sure why.

Quoted 19 lines

Quoted 19 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 6 lines
So I am very confused:
First, what's the difference between these two patches? Or say, does kvm support x2apic since kernel 2.6.32, or since kernel 3.9?
kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9.
Quoted 5 lines
Second, can guest use x2apic even if the host does not? (Assuming qemu has exposed this feature to guest.) The word "use" means something like, accessing x2apic registers, setting the x2apic enable bit in IA32_APICBASE MSR (i.e. bit 10).
Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO. But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic, please see enable_IR_x2apic().
Quoted 1 line
-Jidong

Zhang Haoyu

author

2014-06-11 06:28 UTC #6 lore raw

In reply to Jidong Xiao

Quoted 52 lines

Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 13 lines
Quoted 9 lines
Hi,
According to this:
https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd
It looks like kvm have been supporting x2apic since kernel 2.6.32, or even earlier.
This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio.
Quoted 2 lines
However, this following patch:
Quoted 7 lines
Quoted 4 lines
https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5
Also claims that adding support for x2apic. But this later patch was for kernel 3.9.
This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps, you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit, which belongs to APIC virtualization from certain angle.
Thanks Haoyu, I am reading the Intel x2apic manual.
http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf
but I don't see the so called "virtual x2apic mode", it seems that the manual does not mention anything about that. Do you mean that when the Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't this the same as the aforementioned "emulate x2apic for guest"? Or, "emulate x2apic for guest" is the foundation of "support virtual x2apic mode"?
-Jidong
Quoted 9 lines
Quoted 5 lines
Oh, I found the answer, the virtualized x2apic mode is not defined in the x2apic manual, but is defined in the intel SDM manual, chapter 29
- APIC virtualization and virtual interrupts. Ignore my questions please, but thank you anyway.
Yes, virtual x2apic mode is part of APICv.
Guest has no idea about it is running in a virtual machine,
VMM also prevent guest from enabling/disabling virtual x2apic mode or some other virtualization configurations.
Regarding this "VMM also prevent guest from enabling/disabling virtual x2apic mode", do you mean that guest is not allowed to do something like:
Assuming we are in the guest, and we see the original value of the IA32_APICBASE MSR is 0xfee00900, which means EN bit and BSP bit was enabled. And now, we try to set bit 10 (EXTD bit):

Quoted 9 lines

wrmsr 0x1b 0xfee00d00.
Do you mean this wrmsr operation is not allowed? If fact, I tried this, and this simply crashes the guest kernel. (I am using Linux 2.6.34 as the guest kernel, and Linux 3.14 as the host kernel.) And this is the exact reason that I initialized this whole discussion. My feeling is this might be a kvm bug, so I wish to figure out and try to fix it.

The MSR address range 800H through BFFH is architecturally reserved and dedicated for accessing APIC registers in x2APIC mode, you are writing the wrong MSR address.
Please see SDN 3A 10.12.1.2 x2APIC Register Address Space, you will get the answer.

Quoted 18 lines

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 6 lines
So I am very confused:
First, what's the difference between these two patches? Or say, does kvm support x2apic since kernel 2.6.32, or since kernel 3.9?
kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9.
Quoted 5 lines
Second, can guest use x2apic even if the host does not? (Assuming qemu has exposed this feature to guest.) The word "use" means something like, accessing x2apic registers, setting the x2apic enable bit in IA32_APICBASE MSR (i.e. bit 10).
Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO. But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic, please see enable_IR_x2apic().
Quoted 1 line
-Jidong

Jidong Xiao

2014-06-11 06:46 UTC #7 lore raw

Quoted 61 lines · Zhang Haoyu

Quoted 52 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 13 lines
Quoted 9 lines
Hi,
According to this:
https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd
It looks like kvm have been supporting x2apic since kernel 2.6.32, or even earlier.
This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio.
Quoted 2 lines
However, this following patch:
Quoted 7 lines
Quoted 4 lines
https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5
Also claims that adding support for x2apic. But this later patch was for kernel 3.9.
This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps, you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit, which belongs to APIC virtualization from certain angle.
Thanks Haoyu, I am reading the Intel x2apic manual.
http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf
but I don't see the so called "virtual x2apic mode", it seems that the manual does not mention anything about that. Do you mean that when the Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't this the same as the aforementioned "emulate x2apic for guest"? Or, "emulate x2apic for guest" is the foundation of "support virtual x2apic mode"?
-Jidong
Quoted 9 lines
Quoted 5 lines
Oh, I found the answer, the virtualized x2apic mode is not defined in the x2apic manual, but is defined in the intel SDM manual, chapter 29
- APIC virtualization and virtual interrupts. Ignore my questions please, but thank you anyway.
Yes, virtual x2apic mode is part of APICv.
Guest has no idea about it is running in a virtual machine,
VMM also prevent guest from enabling/disabling virtual x2apic mode or some other virtualization configurations.
Regarding this "VMM also prevent guest from enabling/disabling virtual x2apic mode", do you mean that guest is not allowed to do something like:
Assuming we are in the guest, and we see the original value of the IA32_APICBASE MSR is 0xfee00900, which means EN bit and BSP bit was enabled. And now, we try to set bit 10 (EXTD bit):
Quoted 9 lines
wrmsr 0x1b 0xfee00d00.
Do you mean this wrmsr operation is not allowed? If fact, I tried this, and this simply crashes the guest kernel. (I am using Linux 2.6.34 as the guest kernel, and Linux 3.14 as the host kernel.) And this is the exact reason that I initialized this whole discussion. My feeling is this might be a kvm bug, so I wish to figure out and try to fix it.

Quoted 4 lines

The MSR address range 800H through BFFH is architecturally reserved and dedicated for accessing APIC registers in x2APIC mode, you are writing the wrong MSR address.
Please see SDN 3A 10.12.1.2 x2APIC Register Address Space, you will get the answer.

Oh, I was writing to 0x1b, which is the address of IA32_APIC_BASE MSR. In particular I was trying to write bit 10, which is the x2APIC enable/disable bit. As I said, I just wish to enable that x2APIC. The original value 0xfee00900 suggests x2APIC is not enabled, i.e., bit 10 is 0. According to SDM manual, this bit has to be 1 if we want to enable x2APIC.

Quoted 19 lines

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 6 lines
So I am very confused:
First, what's the difference between these two patches? Or say, does kvm support x2apic since kernel 2.6.32, or since kernel 3.9?
kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9.
Quoted 5 lines
Second, can guest use x2apic even if the host does not? (Assuming qemu has exposed this feature to guest.) The word "use" means something like, accessing x2apic registers, setting the x2apic enable bit in IA32_APICBASE MSR (i.e. bit 10).
Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO. But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic, please see enable_IR_x2apic().
Quoted 1 line
-Jidong

Zhang Haoyu

author

2014-06-11 08:50 UTC #8 lore raw

Quoted 65 lines

Quoted 61 lines
Quoted 52 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 13 lines
Quoted 9 lines
Hi,
According to this:
https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd
It looks like kvm have been supporting x2apic since kernel 2.6.32, or even earlier.
This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio.
Quoted 2 lines
However, this following patch:
Quoted 7 lines
Quoted 4 lines
https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5
Also claims that adding support for x2apic. But this later patch was for kernel 3.9.
This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps, you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit, which belongs to APIC virtualization from certain angle.
Thanks Haoyu, I am reading the Intel x2apic manual.
http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf
but I don't see the so called "virtual x2apic mode", it seems that the manual does not mention anything about that. Do you mean that when the Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't this the same as the aforementioned "emulate x2apic for guest"? Or, "emulate x2apic for guest" is the foundation of "support virtual x2apic mode"?
-Jidong
Quoted 9 lines
Quoted 5 lines
Oh, I found the answer, the virtualized x2apic mode is not defined in the x2apic manual, but is defined in the intel SDM manual, chapter 29
- APIC virtualization and virtual interrupts. Ignore my questions please, but thank you anyway.
Yes, virtual x2apic mode is part of APICv.
Guest has no idea about it is running in a virtual machine,
VMM also prevent guest from enabling/disabling virtual x2apic mode or some other virtualization configurations.
Regarding this "VMM also prevent guest from enabling/disabling virtual x2apic mode", do you mean that guest is not allowed to do something like:
Assuming we are in the guest, and we see the original value of the IA32_APICBASE MSR is 0xfee00900, which means EN bit and BSP bit was enabled. And now, we try to set bit 10 (EXTD bit):
Quoted 9 lines
wrmsr 0x1b 0xfee00d00.
Do you mean this wrmsr operation is not allowed? If fact, I tried this, and this simply crashes the guest kernel. (I am using Linux 2.6.34 as the guest kernel, and Linux 3.14 as the host kernel.) And this is the exact reason that I initialized this whole discussion. My feeling is this might be a kvm bug, so I wish to figure out and try to fix it.
Quoted 4 lines
The MSR address range 800H through BFFH is architecturally reserved and dedicated for accessing APIC registers in x2APIC mode, you are writing the wrong MSR address.
Please see SDN 3A 10.12.1.2 x2APIC Register Address Space, you will get the answer.

Quoted 7 lines

Oh, I was writing to 0x1b, which is the address of IA32_APIC_BASE MSR. In particular I was trying to write bit 10, which is the x2APIC enable/disable bit. As I said, I just wish to enable that x2APIC. The original value 0xfee00900 suggests x2APIC is not enabled, i.e., bit 10 is 0. According to SDM manual, this bit has to be 1 if we want to enable x2APIC.

Sorry, I misread it.
Please try
u64 val;
rdmsrl(0x0000001b, val);
if ( !(val & (1UL << 10)) )

    wrmsrl(0x0000001b, val | (1UL << 10));

If the crash still happened, maybe other settings is needed, like switching apic to physical x2apic, I'm not sure. Please report the crash message.

Quoted 18 lines

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 6 lines
So I am very confused:
First, what's the difference between these two patches? Or say, does kvm support x2apic since kernel 2.6.32, or since kernel 3.9?
kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9.
Quoted 5 lines
Second, can guest use x2apic even if the host does not? (Assuming qemu has exposed this feature to guest.) The word "use" means something like, accessing x2apic registers, setting the x2apic enable bit in IA32_APICBASE MSR (i.e. bit 10).
Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO. But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic, please see enable_IR_x2apic().
Quoted 1 line
-Jidong

Jidong Xiao

2014-06-11 15:12 UTC #9 lore raw

Quoted 77 lines · Zhang Haoyu

Quoted 65 lines
Quoted 61 lines
Quoted 52 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 13 lines
Quoted 9 lines
Hi,
According to this:
https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd
It looks like kvm have been supporting x2apic since kernel 2.6.32, or even earlier.
This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio.
Quoted 2 lines
However, this following patch:
Quoted 7 lines
Quoted 4 lines
https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5
Also claims that adding support for x2apic. But this later patch was for kernel 3.9.
This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps, you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit, which belongs to APIC virtualization from certain angle.
Thanks Haoyu, I am reading the Intel x2apic manual.
http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf
but I don't see the so called "virtual x2apic mode", it seems that the manual does not mention anything about that. Do you mean that when the Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't this the same as the aforementioned "emulate x2apic for guest"? Or, "emulate x2apic for guest" is the foundation of "support virtual x2apic mode"?
-Jidong
Quoted 9 lines
Quoted 5 lines
Oh, I found the answer, the virtualized x2apic mode is not defined in the x2apic manual, but is defined in the intel SDM manual, chapter 29
- APIC virtualization and virtual interrupts. Ignore my questions please, but thank you anyway.
Yes, virtual x2apic mode is part of APICv.
Guest has no idea about it is running in a virtual machine,
VMM also prevent guest from enabling/disabling virtual x2apic mode or some other virtualization configurations.
Regarding this "VMM also prevent guest from enabling/disabling virtual x2apic mode", do you mean that guest is not allowed to do something like:
Assuming we are in the guest, and we see the original value of the IA32_APICBASE MSR is 0xfee00900, which means EN bit and BSP bit was enabled. And now, we try to set bit 10 (EXTD bit):
Quoted 9 lines
wrmsr 0x1b 0xfee00d00.
Do you mean this wrmsr operation is not allowed? If fact, I tried this, and this simply crashes the guest kernel. (I am using Linux 2.6.34 as the guest kernel, and Linux 3.14 as the host kernel.) And this is the exact reason that I initialized this whole discussion. My feeling is this might be a kvm bug, so I wish to figure out and try to fix it.
Quoted 4 lines
The MSR address range 800H through BFFH is architecturally reserved and dedicated for accessing APIC registers in x2APIC mode, you are writing the wrong MSR address.
Please see SDN 3A 10.12.1.2 x2APIC Register Address Space, you will get the answer.
Quoted 7 lines
Oh, I was writing to 0x1b, which is the address of IA32_APIC_BASE MSR. In particular I was trying to write bit 10, which is the x2APIC enable/disable bit. As I said, I just wish to enable that x2APIC. The original value 0xfee00900 suggests x2APIC is not enabled, i.e., bit 10 is 0. According to SDM manual, this bit has to be 1 if we want to enable x2APIC.
Sorry, I misread it.
Please try
u64 val;
rdmsrl(0x0000001b, val);
if ( !(val & (1UL << 10)) )

Quoted 5 lines

    wrmsrl(0x0000001b, val | (1UL << 10));
If the crash still happened, maybe other settings is needed, like switching apic to physical x2apic, I'm not sure. Please report the crash message.

Yes I wrote a simple kernel module which just includes the above four lines in the module init function, and when I try to install the module with insmod command, the crash still happens, the crash message is:

[ 1025.424260] Process bash (pid: 3091, ti=f58b2000 task=f5a725f0
task.ti=f58b2000)
[ 1025.425093] Stack:
[ 1025.425093] Call Trace:
[ 1025.425093] Leftover inexact backtrace:
[ 1025.425093]
[ 1025.425093] Code: 10 5b 5e 5f 5d e9 98 4a 42 00 8b 0d 3c ba 94 c0

8d 93 f0 00 00 00 89 f0 ff 51 78 eb 07 90 8d 74 26 00 f3 90 ba 80 00 00 00 89 f0 <e8> 12 01 1d 00 85 c0 74 ee eb ad 8d b6 00 00 00 00 8d bf 00 00

And this above message will be displayed repeatedly all the time and the system will become unresponsive.

[ 1090.922260] Process bash (pid: 3091, ti=f58b2000 task=f5a725f0
task.ti=f58b2000)
[ 1090.923089] Stack:
[ 1090.923089] Call Trace:
[ 1090.923089] Leftover inexact backtrace:
[ 1090.923089]
[ 1090.923089] Code: 83 c4 10 5b 5e 5f 5d e9 98 4a 42 00 8b 0d 3c ba

94 c0 8d 93 f0 00 00 00 89 f0 ff 51 78 eb 07 90 8d 74 26 00 f3 90 ba 80 00 00 00 <89> f0 e8 12 01 1d 00 85 c0 74 ee eb ad 8d b6 00 00 00 00 8d bf

Quoted 19 lines

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 6 lines
So I am very confused:
First, what's the difference between these two patches? Or say, does kvm support x2apic since kernel 2.6.32, or since kernel 3.9?
kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9.
Quoted 5 lines
Second, can guest use x2apic even if the host does not? (Assuming qemu has exposed this feature to guest.) The word "use" means something like, accessing x2apic registers, setting the x2apic enable bit in IA32_APICBASE MSR (i.e. bit 10).
Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO. But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic, please see enable_IR_x2apic().
Quoted 1 line
-Jidong

Zhang Haoyu

author

2014-06-12 00:43 UTC #10 lore raw

Quoted 94 lines

Quoted 77 lines
Quoted 65 lines
Quoted 61 lines
Quoted 52 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 13 lines
Quoted 9 lines
Hi,
According to this:
https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd
It looks like kvm have been supporting x2apic since kernel 2.6.32, or even earlier.
This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio.
Quoted 2 lines
However, this following patch:
Quoted 7 lines
Quoted 4 lines
https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5
Also claims that adding support for x2apic. But this later patch was for kernel 3.9.
This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps, you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit, which belongs to APIC virtualization from certain angle.
Thanks Haoyu, I am reading the Intel x2apic manual.
http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf
but I don't see the so called "virtual x2apic mode", it seems that the manual does not mention anything about that. Do you mean that when the Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't this the same as the aforementioned "emulate x2apic for guest"? Or, "emulate x2apic for guest" is the foundation of "support virtual x2apic mode"?
-Jidong
Quoted 9 lines
Quoted 5 lines
Oh, I found the answer, the virtualized x2apic mode is not defined in the x2apic manual, but is defined in the intel SDM manual, chapter 29
- APIC virtualization and virtual interrupts. Ignore my questions please, but thank you anyway.
Yes, virtual x2apic mode is part of APICv.
Guest has no idea about it is running in a virtual machine,
VMM also prevent guest from enabling/disabling virtual x2apic mode or some other virtualization configurations.
Regarding this "VMM also prevent guest from enabling/disabling virtual x2apic mode", do you mean that guest is not allowed to do something like:
Assuming we are in the guest, and we see the original value of the IA32_APICBASE MSR is 0xfee00900, which means EN bit and BSP bit was enabled. And now, we try to set bit 10 (EXTD bit):
Quoted 9 lines
wrmsr 0x1b 0xfee00d00.
Do you mean this wrmsr operation is not allowed? If fact, I tried this, and this simply crashes the guest kernel. (I am using Linux 2.6.34 as the guest kernel, and Linux 3.14 as the host kernel.) And this is the exact reason that I initialized this whole discussion. My feeling is this might be a kvm bug, so I wish to figure out and try to fix it.
Quoted 4 lines
The MSR address range 800H through BFFH is architecturally reserved and dedicated for accessing APIC registers in x2APIC mode, you are writing the wrong MSR address.
Please see SDN 3A 10.12.1.2 x2APIC Register Address Space, you will get the answer.
Quoted 7 lines
Oh, I was writing to 0x1b, which is the address of IA32_APIC_BASE MSR. In particular I was trying to write bit 10, which is the x2APIC enable/disable bit. As I said, I just wish to enable that x2APIC. The original value 0xfee00900 suggests x2APIC is not enabled, i.e., bit 10 is 0. According to SDM manual, this bit has to be 1 if we want to enable x2APIC.
Sorry, I misread it.
Please try
u64 val;
rdmsrl(0x0000001b, val);
if ( !(val & (1UL << 10)) )
Quoted 5 lines
    wrmsrl(0x0000001b, val | (1UL << 10));
If the crash still happened, maybe other settings is needed, like switching apic to physical x2apic, I'm not sure. Please report the crash message.
Yes I wrote a simple kernel module which just includes the above four lines in the module init function, and when I try to install the module with insmod command, the crash still happens, the crash message is:
[ 1025.424260] Process bash (pid: 3091, ti=f58b2000 task=f5a725f0
task.ti=f58b2000)
[ 1025.425093] Stack:
[ 1025.425093] Call Trace:
[ 1025.425093] Leftover inexact backtrace:
[ 1025.425093]
[ 1025.425093] Code: 10 5b 5e 5f 5d e9 98 4a 42 00 8b 0d 3c ba 94 c0

Quoted 4 lines

8d 93 f0 00 00 00 89 f0 ff 51 78 eb 07 90 8d 74 26 00 f3 90 ba 80 00 00 00 89 f0 <e8> 12 01 1d 00 85 c0 74 ee eb ad 8d b6 00 00 00 00 8d bf 00 00

Does the vcpu have x2apic feature?
please make a confirmation by 'cat /proc/cpuinfo | grep -i x2apic' Configure crashkernel to kdump more info, please.

Thanks,
Zhang Haoyu

Quoted 33 lines

And this above message will be displayed repeatedly all the time and the system will become unresponsive.
[ 1090.922260] Process bash (pid: 3091, ti=f58b2000 task=f5a725f0
task.ti=f58b2000)
[ 1090.923089] Stack:
[ 1090.923089] Call Trace:
[ 1090.923089] Leftover inexact backtrace:
[ 1090.923089]
[ 1090.923089] Code: 83 c4 10 5b 5e 5f 5d e9 98 4a 42 00 8b 0d 3c ba
94 c0 8d 93 f0 00 00 00 89 f0 ff 51 78 eb 07 90 8d 74 26 00 f3 90 ba 80 00 00 00 <89> f0 e8 12 01 1d 00 85 c0 74 ee eb ad 8d b6 00 00 00 00 8d bf
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 6 lines
So I am very confused:
First, what's the difference between these two patches? Or say, does kvm support x2apic since kernel 2.6.32, or since kernel 3.9?
kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9.
Quoted 5 lines
Second, can guest use x2apic even if the host does not? (Assuming qemu has exposed this feature to guest.) The word "use" means something like, accessing x2apic registers, setting the x2apic enable bit in IA32_APICBASE MSR (i.e. bit 10).
Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO. But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic, please see enable_IR_x2apic().
Quoted 1 line
-Jidong

Jidong Xiao

2014-06-12 03:24 UTC #11 lore raw

Quoted 98 lines · Zhang Haoyu

Quoted 94 lines
Quoted 77 lines
Quoted 65 lines
Quoted 61 lines
Quoted 52 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 13 lines
Quoted 9 lines
Hi,
According to this:
https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd
It looks like kvm have been supporting x2apic since kernel 2.6.32, or even earlier.
This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio.
Quoted 2 lines
However, this following patch:
Quoted 7 lines
Quoted 4 lines
https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5
Also claims that adding support for x2apic. But this later patch was for kernel 3.9.
This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps, you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit, which belongs to APIC virtualization from certain angle.
Thanks Haoyu, I am reading the Intel x2apic manual.
http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf
but I don't see the so called "virtual x2apic mode", it seems that the manual does not mention anything about that. Do you mean that when the Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't this the same as the aforementioned "emulate x2apic for guest"? Or, "emulate x2apic for guest" is the foundation of "support virtual x2apic mode"?
-Jidong
Quoted 9 lines
Quoted 5 lines
Oh, I found the answer, the virtualized x2apic mode is not defined in the x2apic manual, but is defined in the intel SDM manual, chapter 29
- APIC virtualization and virtual interrupts. Ignore my questions please, but thank you anyway.
Yes, virtual x2apic mode is part of APICv.
Guest has no idea about it is running in a virtual machine,
VMM also prevent guest from enabling/disabling virtual x2apic mode or some other virtualization configurations.
Regarding this "VMM also prevent guest from enabling/disabling virtual x2apic mode", do you mean that guest is not allowed to do something like:
Assuming we are in the guest, and we see the original value of the IA32_APICBASE MSR is 0xfee00900, which means EN bit and BSP bit was enabled. And now, we try to set bit 10 (EXTD bit):
Quoted 9 lines
wrmsr 0x1b 0xfee00d00.
Do you mean this wrmsr operation is not allowed? If fact, I tried this, and this simply crashes the guest kernel. (I am using Linux 2.6.34 as the guest kernel, and Linux 3.14 as the host kernel.) And this is the exact reason that I initialized this whole discussion. My feeling is this might be a kvm bug, so I wish to figure out and try to fix it.
Quoted 4 lines
The MSR address range 800H through BFFH is architecturally reserved and dedicated for accessing APIC registers in x2APIC mode, you are writing the wrong MSR address.
Please see SDN 3A 10.12.1.2 x2APIC Register Address Space, you will get the answer.
Quoted 7 lines
Oh, I was writing to 0x1b, which is the address of IA32_APIC_BASE MSR. In particular I was trying to write bit 10, which is the x2APIC enable/disable bit. As I said, I just wish to enable that x2APIC. The original value 0xfee00900 suggests x2APIC is not enabled, i.e., bit 10 is 0. According to SDM manual, this bit has to be 1 if we want to enable x2APIC.
Sorry, I misread it.
Please try
u64 val;
rdmsrl(0x0000001b, val);
if ( !(val & (1UL << 10)) )
Quoted 5 lines
    wrmsrl(0x0000001b, val | (1UL << 10));
If the crash still happened, maybe other settings is needed, like switching apic to physical x2apic, I'm not sure. Please report the crash message.
Yes I wrote a simple kernel module which just includes the above four lines in the module init function, and when I try to install the module with insmod command, the crash still happens, the crash message is:
[ 1025.424260] Process bash (pid: 3091, ti=f58b2000 task=f5a725f0
task.ti=f58b2000)
[ 1025.425093] Stack:
[ 1025.425093] Call Trace:
[ 1025.425093] Leftover inexact backtrace:
[ 1025.425093]
[ 1025.425093] Code: 10 5b 5e 5f 5d e9 98 4a 42 00 8b 0d 3c ba 94 c0
Quoted 4 lines
8d 93 f0 00 00 00 89 f0 ff 51 78 eb 07 90 8d 74 26 00 f3 90 ba 80 00 00 00 89 f0 <e8> 12 01 1d 00 85 c0 74 ee eb ad 8d b6 00 00 00 00 8d bf 00 00

Quoted 6 lines

Does the vcpu have x2apic feature?
please make a confirmation by 'cat /proc/cpuinfo | grep -i x2apic' Configure crashkernel to kdump more info, please.
Thanks,
Zhang Haoyu

Hi, Haoyu,

I think I have found the root cause. As I said I was using 2.6.34 kernel in the guest, and CONFIG_X86_X2APIC was not enabled by default in my kernel. After I set CONFIG_X86_X2APIC and recompile the kernel, the new kernel just works fine in x2apic mode. Thanks for your time and help.

-Jidong

Quoted 34 lines

Quoted 33 lines
And this above message will be displayed repeatedly all the time and the system will become unresponsive.
[ 1090.922260] Process bash (pid: 3091, ti=f58b2000 task=f5a725f0
task.ti=f58b2000)
[ 1090.923089] Stack:
[ 1090.923089] Call Trace:
[ 1090.923089] Leftover inexact backtrace:
[ 1090.923089]
[ 1090.923089] Code: 83 c4 10 5b 5e 5f 5d e9 98 4a 42 00 8b 0d 3c ba
94 c0 8d 93 f0 00 00 00 89 f0 ff 51 78 eb 07 90 8d 74 26 00 f3 90 ba 80 00 00 00 <89> f0 e8 12 01 1d 00 85 c0 74 ee eb ad 8d b6 00 00 00 00 8d bf
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 6 lines
So I am very confused:
First, what's the difference between these two patches? Or say, does kvm support x2apic since kernel 2.6.32, or since kernel 3.9?
kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9.
Quoted 5 lines
Second, can guest use x2apic even if the host does not? (Assuming qemu has exposed this feature to guest.) The word "use" means something like, accessing x2apic registers, setting the x2apic enable bit in IA32_APICBASE MSR (i.e. bit 10).
Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO. But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic, please see enable_IR_x2apic().
Quoted 1 line
-Jidong