Enable more than 255 VCPU support without irq remapping function in the guest
Hi All:
Recently I am working on extending max vcpu to more than 256 on the both KVM/Xen. For some HPC cases, it needs many vcpus. The job requires to use X2APIC in the guest which supports 32-bit APIC id. Linux kernel requires irq remapping function during enabling X2APIC when max APIC id is more than 255(More detail please see try_to_enable_x2apic()).
The irq remapping function helps to deliver irq to cpu 255~. IOAPIC just supports 8-bit target APIC id field and only can deliver irq to cpu 0~255.
So far both KVM/Xen doesn't enable irq remapping function. If enable the function, it seems a huge job which need to rework IO-APIC, local APIC, MSI parts and add virtual VTD support in the KVM.
Other quick way to enable more than 256 VCPUs is to eliminate the dependency between irq remapping and X2APIC in the guest linux kernel. So far I can boot the guest after removing the dependency. The side effect I thought is that irq only can deliver to 0~255 vcpus but 256 vcpus seem enough to balance irq requests in the guest. In the most cases, there are fewer devices in the guest.
I wonder whether it's feasible. There maybe some other side effects I didn't think of. Very appreciate for your comments.
Tianyu Lan
Best regards.
Quoted 23 lines · Tianyu
Hi All:
Recently I am working on extending max vcpu to more than 256 on the both KVM/Xen. For some HPC cases, it needs many vcpus. The job requires to use X2APIC in the guest which supports 32-bit APIC id. Linux kernel requires irq remapping function during enabling X2APIC when max APIC id is more than 255(More detail please see
try_to_enable_x2apic()).The irq remapping function helps to deliver irq to cpu 255~. IOAPIC just supports 8-bit target APIC id field and only can deliver irq to cpu 0~255.
So far both KVM/Xen doesn't enable irq remapping function. If enable the function, it seems a huge job which need to rework IO-APIC, local APIC, MSI parts and add virtual VTD support in the KVM.
Other quick way to enable more than 256 VCPUs is to eliminate the dependency between irq remapping and X2APIC in the guest linux kernel. So far I can boot the guest after removing the dependency. The side effect I thought is that irq only can deliver to 0~255 vcpus but 256 vcpus seem enough to balance irq requests in the guest. In the most cases, there are fewer devices in the guest.
Quoted 2 lines
I wonder whether it's feasible. There maybe some other side effects I didn't think of. Very appreciate for your comments.
Radim is working on the KVM side already, Peter is currently driving the VT-d interrupt emulation topic in QEMU. It's in reach, I would say. :)
Jan
PS: Please no PV mess, at least without good reasons.
Signature · 2 lines
Siemens AG, Corporate Technology, CT RDA ITP SES-DE Corporate Competence Center Embedded Linux
2016-04-26 18:17+0200, Jan Kiszka:
Quoted 8 lines · Tianyu
Quoted 7 lines
Hi All:
Recently I am working on extending max vcpu to more than 256 on the both KVM/Xen. For some HPC cases, it needs many vcpus. The job requires to use X2APIC in the guest which supports 32-bit APIC id. Linux kernel requires irq remapping function during enabling X2APIC when max APIC id is more than 255(More detail please see
try_to_enable_x2apic()).
Our of curiosity, how many VCPUs are you aiming at?
Quoted 15 lines
Quoted 15 lines
The irq remapping function helps to deliver irq to cpu 255~. IOAPIC just supports 8-bit target APIC id field and only can deliver irq to cpu 0~255.
So far both KVM/Xen doesn't enable irq remapping function. If enable the function, it seems a huge job which need to rework IO-APIC, local APIC, MSI parts and add virtual VTD support in the KVM.
Other quick way to enable more than 256 VCPUs is to eliminate the dependency between irq remapping and X2APIC in the guest linux kernel. So far I can boot the guest after removing the dependency. The side effect I thought is that irq only can deliver to 0~255 vcpus but 256 vcpus seem enough to balance irq requests in the guest. In the most cases, there are fewer devices in the guest.
Quoted 5 lines
Quoted 2 lines
I wonder whether it's feasible. There maybe some other side effects I didn't think of. Very appreciate for your comments.
Radim is working on the KVM side already, Peter is currently driving the VT-d interrupt emulation topic in QEMU. It's in reach, I would say. :)
+ Igor extends QEMU to support more than 255 in internal structures and ACPI. What remains mostly untracked is Seabios/OVMF.
Quoted 1 line
PS: Please no PV mess, at least without good reasons.
Seconded.
(If we designed all related devices as virtware, then it would not be
that bad, but slightly modifying and putting hardware drivers into situations that cannot happen in hardware, not even in the spec, and then juggling the KVM side to make them work, is a road to hell.)
Quoted 27 lines · Radim Krčmář
2016-04-26 18:17+0200, Jan Kiszka:
Quoted 8 lines · Tianyu
Quoted 7 lines
Hi All:
Recently I am working on extending max vcpu to more than 256 on the both KVM/Xen. For some HPC cases, it needs many vcpus. The job requires to use X2APIC in the guest which supports 32-bit APIC id. Linux kernel requires irq remapping function during enabling X2APIC when max APIC id is more than 255(More detail please see
try_to_enable_x2apic()).Our of curiosity, how many VCPUs are you aiming at?
Quoted 15 lines
Quoted 15 lines
The irq remapping function helps to deliver irq to cpu 255~. IOAPIC just supports 8-bit target APIC id field and only can deliver irq to cpu 0~255.
So far both KVM/Xen doesn't enable irq remapping function. If enable the function, it seems a huge job which need to rework IO-APIC, local APIC, MSI parts and add virtual VTD support in the KVM.
Other quick way to enable more than 256 VCPUs is to eliminate the dependency between irq remapping and X2APIC in the guest linux kernel. So far I can boot the guest after removing the dependency. The side effect I thought is that irq only can deliver to 0~255 vcpus but 256 vcpus seem enough to balance irq requests in the guest. In the most cases, there are fewer devices in the guest.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
I wonder whether it's feasible. There maybe some other side effects I didn't think of. Very appreciate for your comments.
Radim is working on the KVM side already, Peter is currently driving the VT-d interrupt emulation topic in QEMU. It's in reach, I would say. :)
+ Igor extends QEMU to support more than 255 in internal structures and ACPI. What remains mostly untracked is Seabios/OVMF.
If we don't want the interrupt from internal device delivers to CPU >255, do we still need the VT-d interrupt remapping emulation? I think firmware is able to send IPI to wakeup APs even without IR and OS is able to do it too. So basically, only KVM and Qemu's support is enough.
Quoted 10 lines
Quoted 2 lines
PS: Please no PV mess, at least without good reasons.
Seconded.
(If we designed all related devices as virtware, then it would not be
that bad, but slightly modifying and putting hardware drivers into situations that cannot happen in hardware, not even in the spec, and then juggling the KVM side to make them work, is a road to hell.)
Signature · 2 lines
best regards yang
Quoted 37 lines · Yang Zhang
Quoted 36 lines · Radim Krčmář
2016-04-26 18:17+0200, Jan Kiszka:
Quoted 9 lines · Tianyu
Quoted 8 lines
Hi All:
Recently I am working on extending max vcpu to more than 256 on the both
KVM/Xen. For some HPC cases, it needs many vcpus. The job requires to use X2APIC in the guest which supports 32-bit APIC id. Linux kernel requires irq remapping function during enabling X2APIC when max APIC id is more than 255(More detail please seetry_to_enable_x2apic()).Our of curiosity, how many VCPUs are you aiming at?
Quoted 22 lines
Quoted 17 lines
The irq remapping function helps to deliver irq to cpu 255~. IOAPIC just
supports 8-bit target APIC id field and only can deliver irq to cpu 0~255.So far both KVM/Xen doesn't enable irq remapping function. If enable the
function, it seems a huge job which need to rework IO-APIC, local APIC, MSI parts and add virtual VTD support in the KVM.Other quick way to enable more than 256 VCPUs is to eliminate the dependency between irq remapping and X2APIC in the guest linux kernel. So far I can boot the guest after removing the dependency. The side effect I thought is that irq only can deliver to 0~255 vcpus but 256 vcpus seem enough to balance irq requests in the guest. In the most cases, there are fewer devices in the guest.
Quoted 2 lines
I wonder whether it's feasible. There maybe some other side effects I didn't think of. Very appreciate for your comments.
Radim is working on the KVM side already, Peter is currently driving the VT-d interrupt emulation topic in QEMU. It's in reach, I would say. :)
Quoted 7 lines
Quoted 2 lines
+ Igor extends QEMU to support more than 255 in internal structures and ACPI. What remains mostly untracked is Seabios/OVMF.
If we don't want the interrupt from internal device delivers to CPU
Quoted 1 line
255, do we still need the VT-d interrupt remapping emulation? I think
firmware is able to send IPI to wakeup APs even without IR and OS is able to do it too. So basically, only KVM and Qemu's support is enough.
What are "internal devices" for you? And which OS do you know that would handle such artificial setups without prio massive patching?
We do need VT-d IR emulation in order to present our guest a well specified and support architecture for running > 255 CPUs.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT RDA ITP SES-DE Corporate Competence Center Embedded Linux
Quoted 4 lines · Jan Kiszka
Quoted 4 lines
Quoted 4 lines
If we don't want the interrupt from internal device delivers to CPU
Quoted 1 line
Quoted 1 line
255, do we still need the VT-d interrupt remapping emulation? I think
firmware is able to send IPI to wakeup APs even without IR and OS is able to do it too. So basically, only KVM and Qemu's support is enough.
Yes, just starting more than 255 APs doesn't need IR.
Quoted 5 lines
What are "internal devices" for you? And which OS do you know that would handle such artificial setups without prio massive patching?
We do need VT-d IR emulation in order to present our guest a well specified and support architecture for running > 255 CPUs.
Changing guest kernel will be big concern. I found commit ce69a784 did optimization to use X2APIC without IR in the guest when APIC id is less than 256 and so I proposed my idea to see everyone's feedback. Whether it's possible to relax the IR requirement when APIC id > 255 in the guest.
commit ce69a784504222c3ab6f1b3c357d09ec5772127a
Author: Gleb Natapov <[email protected]> Date: Mon Jul 20 15:24:17 2009 +0300
x86/apic: Enable x2APIC without interrupt remapping under KVM
KVM would like to provide x2APIC interface to a guest without emulating interrupt remapping device. The reason KVM prefers guest to use x2APIC is that x2APIC interface is better virtualizable and provides better performance than mmio xAPIC interface:
- msr exits are faster than mmio (no page table walk, emulation)
- no need to read back ICR to look at the busy bit
- one 64 bit ICR write instead of two 32 bit writes
- shared code with the Hyper-V paravirt interface
Included patch changes x2APIC enabling logic to enable it even if IR initialization failed, but kernel runs under KVM and no apic id is greater than 255 (if there is one spec requires BIOS to move to
x2apic mode before starting an OS).
It's great to know Peter already worked on the IR.
Signature · 2 lines
Best regards Tianyu Lan
Quoted 11 lines · Lan Tianyu
Quoted 4 lines · Jan Kiszka
Quoted 4 lines
Quoted 4 lines
If we don't want the interrupt from internal device delivers to CPU
Quoted 1 line
Quoted 1 line
255, do we still need the VT-d interrupt remapping emulation? I think
firmware is able to send IPI to wakeup APs even without IR and OS is able to do it too. So basically, only KVM and Qemu's support is enough.
Yes, just starting more than 255 APs doesn't need IR.
Quoted 3 lines
What are "internal devices" for you? And which OS do you know that would handle such artificial setups without prio massive patching?
Quoted 7 lines
Quoted 2 lines
We do need VT-d IR emulation in order to present our guest a well specified and support architecture for running > 255 CPUs.
Changing guest kernel will be big concern. I found commit
ce69a784did optimization to use X2APIC without IR in the guest when APIC id is less than 256 and so I proposed my idea to see everyone's feedback. Whether it's possible to relax the IR requirement when APIC id > 255 in the guest.
You can't do that easily because you can't address those additional CPUs from *any* device then, only via IPIs. That means, Linux would have to be changed to only set up IRQ affinity masks in the 0-254 range. I suppose you would even have to patch tools like irqbalanced to not issue mask changes via /proc that include larger CPU IDs. Practically not feasible, already on Linux. Not to speak of other guest OSes.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT RDA ITP SES-DE Corporate Competence Center Embedded Linux
Quoted 41 lines · Jan Kiszka
Quoted 40 lines · Yang Zhang
Quoted 38 lines · Radim Krčmář
2016-04-26 18:17+0200, Jan Kiszka:
Quoted 9 lines · Tianyu
Quoted 8 lines
Hi All:
Recently I am working on extending max vcpu to more than 256 on the both
KVM/Xen. For some HPC cases, it needs many vcpus. The job requires to use X2APIC in the guest which supports 32-bit APIC id. Linux kernel requires irq remapping function during enabling X2APIC when max APIC id is more than 255(More detail please seetry_to_enable_x2apic()).Our of curiosity, how many VCPUs are you aiming at?
Quoted 17 lines
Quoted 17 lines
The irq remapping function helps to deliver irq to cpu 255~. IOAPIC just
supports 8-bit target APIC id field and only can deliver irq to cpu 0~255.So far both KVM/Xen doesn't enable irq remapping function. If enable the
function, it seems a huge job which need to rework IO-APIC, local APIC, MSI parts and add virtual VTD support in the KVM.Other quick way to enable more than 256 VCPUs is to eliminate the dependency between irq remapping and X2APIC in the guest linux kernel. So far I can boot the guest after removing the dependency. The side effect I thought is that irq only can deliver to 0~255 vcpus but 256 vcpus seem enough to balance irq requests in the guest. In the most cases, there are fewer devices in the guest.
Quoted 5 lines
Quoted 2 lines
I wonder whether it's feasible. There maybe some other side effects I didn't think of. Very appreciate for your comments.
Radim is working on the KVM side already, Peter is currently driving the VT-d interrupt emulation topic in QEMU. It's in reach, I would say. :)
+ Igor extends QEMU to support more than 255 in internal structures and ACPI. What remains mostly untracked is Seabios/OVMF.
Quoted 7 lines
Quoted 4 lines
If we don't want the interrupt from internal device delivers to CPU
Quoted 1 line
255, do we still need the VT-d interrupt remapping emulation? I think
firmware is able to send IPI to wakeup APs even without IR and OS is able to do it too. So basically, only KVM and Qemu's support is enough.
What are "internal devices" for you? And which OS do you know that would handle such artificial setups without prio massive patching?
Sorry, a typo. I mean the external devices of IOAPIC/MSI/MSIX. Doesn't current Linux use x2apic without IR in VM?
Quoted 3 lines
We do need VT-d IR emulation in order to present our guest a well specified and support architecture for running > 255 CPUs.
I mean in Tianyu's case, if he doesn't care about to deliver external interrupt to CPU >255, IR is not required.
Signature · 2 lines
best regards yang
Quoted 49 lines · Yang Zhang
Quoted 48 lines · Jan Kiszka
Quoted 47 lines · Yang Zhang
Quoted 37 lines · Radim Krčmář
2016-04-26 18:17+0200, Jan Kiszka:
Quoted 10 lines · Tianyu
Quoted 9 lines
Hi All:
Recently I am working on extending max vcpu to more than 256 on the both
KVM/Xen. For some HPC cases, it needs many vcpus. The job requires to use X2APIC in the guest which supports 32-bit APIC id. Linux kernel requires irq remapping function during enabling X2APIC when max APIC id
is more than 255(More detail please seetry_to_enable_x2apic()).Our of curiosity, how many VCPUs are you aiming at?
Quoted 23 lines
Quoted 22 lines
The irq remapping function helps to deliver irq to cpu 255~. IOAPIC just
supports 8-bit target APIC id field and only can deliver irq to cpu 0~255.So far both KVM/Xen doesn't enable irq remapping function. If enable the
function, it seems a huge job which need to rework IO-APIC, local APIC,
MSI parts and add virtual VTD support in the KVM.Other quick way to enable more than 256 VCPUs is to eliminate the dependency between irq remapping and X2APIC in the guest linux kernel.
So far I can boot the guest after removing the dependency. The side effect I thought is that irq only can deliver to 0~255 vcpus but 256 vcpus seem enough to balance irq requests in the guest. In the
most cases, there are fewer devices in the guest.I wonder whether it's feasible. There maybe some other side effects I didn't think of. Very appreciate for your comments.
Quoted 6 lines
Quoted 3 lines
Radim is working on the KVM side already, Peter is currently driving the
VT-d interrupt emulation topic in QEMU. It's in reach, I would say. :)+ Igor extends QEMU to support more than 255 in internal structures and ACPI. What remains mostly untracked is Seabios/OVMF.
If we don't want the interrupt from internal device delivers to CPU
Quoted 1 line
255, do we still need the VT-d interrupt remapping emulation? I think
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
firmware is able to send IPI to wakeup APs even without IR and OS is able to do it too. So basically, only KVM and Qemu's support is enough.
What are "internal devices" for you? And which OS do you know that would handle such artificial setups without prio massive patching?
Sorry, a typo. I mean the external devices of IOAPIC/MSI/MSIX. Doesn't current Linux use x2apic without IR in VM?
If and only if there only need to be 254 CPUs to be addressed.
Quoted 7 lines
Quoted 4 lines
We do need VT-d IR emulation in order to present our guest a well specified and support architecture for running > 255 CPUs.
I mean in Tianyu's case, if he doesn't care about to deliver external interrupt to CPU >255, IR is not required.
What matters is the guest OS. See my other reply on this why this doesn't work, even for Linux.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT RDA ITP SES-DE Corporate Competence Center Embedded Linux
Quoted 63 lines · Jan Kiszka
Quoted 49 lines · Yang Zhang
Quoted 48 lines · Jan Kiszka
Quoted 47 lines · Yang Zhang
Quoted 37 lines · Radim Krčmář
2016-04-26 18:17+0200, Jan Kiszka:
Quoted 10 lines · Tianyu
Quoted 9 lines
Hi All:
Recently I am working on extending max vcpu to more than 256 on the both
KVM/Xen. For some HPC cases, it needs many vcpus. The job requires to use X2APIC in the guest which supports 32-bit APIC id. Linux kernel requires irq remapping function during enabling X2APIC when max APIC id
is more than 255(More detail please seetry_to_enable_x2apic()).Our of curiosity, how many VCPUs are you aiming at?
Quoted 23 lines
Quoted 22 lines
The irq remapping function helps to deliver irq to cpu 255~. IOAPIC just
supports 8-bit target APIC id field and only can deliver irq to cpu 0~255.So far both KVM/Xen doesn't enable irq remapping function. If enable the
function, it seems a huge job which need to rework IO-APIC, local APIC,
MSI parts and add virtual VTD support in the KVM.Other quick way to enable more than 256 VCPUs is to eliminate the dependency between irq remapping and X2APIC in the guest linux kernel.
So far I can boot the guest after removing the dependency. The side effect I thought is that irq only can deliver to 0~255 vcpus but 256 vcpus seem enough to balance irq requests in the guest. In the
most cases, there are fewer devices in the guest.I wonder whether it's feasible. There maybe some other side effects I didn't think of. Very appreciate for your comments.
Quoted 6 lines
Quoted 3 lines
Radim is working on the KVM side already, Peter is currently driving the
VT-d interrupt emulation topic in QEMU. It's in reach, I would say. :)+ Igor extends QEMU to support more than 255 in internal structures and ACPI. What remains mostly untracked is Seabios/OVMF.
If we don't want the interrupt from internal device delivers to CPU
Quoted 1 line
255, do we still need the VT-d interrupt remapping emulation? I think
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
firmware is able to send IPI to wakeup APs even without IR and OS is able to do it too. So basically, only KVM and Qemu's support is enough.
What are "internal devices" for you? And which OS do you know that would handle such artificial setups without prio massive patching?
Sorry, a typo. I mean the external devices of IOAPIC/MSI/MSIX. Doesn't current Linux use x2apic without IR in VM?
If and only if there only need to be 254 CPUs to be addressed.
Quoted 2 lines
Quoted 2 lines
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
We do need VT-d IR emulation in order to present our guest a well specified and support architecture for running > 255 CPUs.
I mean in Tianyu's case, if he doesn't care about to deliver external interrupt to CPU >255, IR is not required.
What matters is the guest OS. See my other reply on this why this doesn't work, even for Linux.
Since there only few devices in his case, set the irq affinity manually is enough.
Signature · 2 lines
best regards yang
Quoted 75 lines · Yang Zhang
Quoted 74 lines · Jan Kiszka
Quoted 59 lines · Yang Zhang
Quoted 58 lines · Jan Kiszka
Quoted 56 lines · Yang Zhang
Quoted 41 lines · Radim Krčmář
2016-04-26 18:17+0200, Jan Kiszka:
Quoted 11 lines · Tianyu
Quoted 10 lines
Hi All:
Recently I am working on extending max vcpu to more than 256 on the both
KVM/Xen. For some HPC cases, it needs many vcpus. The job requires to
use X2APIC in the guest which supports 32-bit APIC id. Linux kernel requires irq remapping function during enabling X2APIC when max APIC id
is more than 255(More detail please seetry_to_enable_x2apic()).Our of curiosity, how many VCPUs are you aiming at?
Quoted 26 lines
Quoted 25 lines
The irq remapping function helps to deliver irq to cpu 255~. IOAPIC just
supports 8-bit target APIC id field and only can deliver irq to cpu 0~255.So far both KVM/Xen doesn't enable irq remapping function. If enable
the
function, it seems a huge job which need to rework IO-APIC, local APIC,
MSI parts and add virtual VTD support in the KVM.Other quick way to enable more than 256 VCPUs is to eliminate the dependency between irq remapping and X2APIC in the guest linux kernel.
So far I can boot the guest after removing the dependency. The side effect I thought is that irq only can deliver to 0~255 vcpus
but 256 vcpus seem enough to balance irq requests in the guest. In the
most cases, there are fewer devices in the guest.I wonder whether it's feasible. There maybe some other side effects I
didn't think of. Very appreciate for your comments.Quoted 8 lines
Quoted 4 lines
Radim is working on the KVM side already, Peter is currently driving the
VT-d interrupt emulation topic in QEMU. It's in reach, I would say. :)+ Igor extends QEMU to support more than 255 in internal structures and
ACPI. What remains mostly untracked is Seabios/OVMF.If we don't want the interrupt from internal device delivers to CPU
Quoted 1 line
255, do we still need the VT-d interrupt remapping emulation? I think
firmware is able to send IPI to wakeup APs even without IR and OS is able to do it too. So basically, only KVM and Qemu's support is enough.
Quoted 6 lines
Quoted 3 lines
What are "internal devices" for you? And which OS do you know that would
handle such artificial setups without prio massive patching?Sorry, a typo. I mean the external devices of IOAPIC/MSI/MSIX. Doesn't current Linux use x2apic without IR in VM?
If and only if there only need to be 254 CPUs to be addressed.
Quoted 5 lines
Quoted 4 lines
We do need VT-d IR emulation in order to present our guest a well specified and support architecture for running > 255 CPUs.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
I mean in Tianyu's case, if he doesn't care about to deliver external interrupt to CPU >255, IR is not required.
What matters is the guest OS. See my other reply on this why this doesn't work, even for Linux.
Since there only few devices in his case, set the irq affinity manually is enough.
Ah, wait - are we talking about emulating the Xeon Phi architecture in QEMU, accelerated by KVM?
Then maybe you can point to a more detailed description of its interrupt architecture than that rather vague "Xeon Phi Coprocessor System Software Developers Guide" I was just looking at provides. While the Phi may not have VT-d internally, it still has a need to translate incoming MSI/MSI-X messages (via that PEG port?) to something that can address more than 255 APIC IDs, no?
Possibly, you only need an extended KVM kernel interface for the Phi that allows injecting APIC interrupts to more than 255 CPUs. That interface has to be designed anyway, for normal x86 systems, and is what Radim was talking about.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT RDA ITP SES-DE Corporate Competence Center Embedded Linux
2016-04-28 08:54+0200, Jan Kiszka:
Quoted 11 lines · Yang Zhang
Quoted 10 lines · Jan Kiszka
Quoted 6 lines · Yang Zhang
Quoted 2 lines
I mean in Tianyu's case, if he doesn't care about to deliver external interrupt to CPU >255, IR is not required.
What matters is the guest OS. See my other reply on this why this doesn't work, even for Linux.
Since there only few devices in his case, set the irq affinity manually is enough.
You could configure non-IPIs to work, but we want to create options that are hard to break.
Quoted 2 lines
Ah, wait - are we talking about emulating the Xeon Phi architecture in QEMU, accelerated by KVM?
Knights Landing will also be manufactured as a CPU, hopefully without many peculiarities.
I think we are talking about extending KVM's IR-less x2APIC, when standard x2APIC is the future.
Quoted 16 lines · Radim Krčmář
2016-04-28 08:54+0200, Jan Kiszka:
Quoted 11 lines · Yang Zhang
Quoted 10 lines · Jan Kiszka
Quoted 6 lines · Yang Zhang
Quoted 2 lines
I mean in Tianyu's case, if he doesn't care about to deliver external interrupt to CPU >255, IR is not required.
What matters is the guest OS. See my other reply on this why this doesn't work, even for Linux.
Since there only few devices in his case, set the irq affinity manually is enough.
You could configure non-IPIs to work, but we want to create options that are hard to break.
Quoted 8 lines
Quoted 2 lines
Ah, wait - are we talking about emulating the Xeon Phi architecture in QEMU, accelerated by KVM?
Knights Landing will also be manufactured as a CPU, hopefully without many peculiarities.
I think we are talking about extending KVM's IR-less x2APIC, when standard x2APIC is the future.
Yes, Since IR is only useful for the external device, and 255 CPUs is enough to handle the interrupts from external devices. Besides, i think virtual VT-d will bring extra performance impaction for devices, so if IR-less X2APIC also works well with more than 255 VCPUs, maybe extending KVM with IR-less x2apic is not a bad idea.
Signature · 2 lines
best regards yang
Quoted 5 lines · Yang Zhang
Quoted 2 lines · Radim Krčmář
I think we are talking about extending KVM's IR-less x2APIC, when standard x2APIC is the future.
Yes, Since IR is only useful for the external device, and 255 CPUs is enough to handle the interrupts from external devices. Besides, i think virtual VT-d will bring extra performance impaction for devices, so if IR-less X2APIC also works well with more than 255 VCPUs, maybe extending KVM with IR-less x2apic is not a bad idea.
So will you use x2APIC physical mode in this system? Try not to send a multicast IPI to 400 cores in the VM...
Nadav
Quoted 9 lines · Nadav Amit
Quoted 5 lines · Yang Zhang
Quoted 2 lines · Radim Krčmář
I think we are talking about extending KVM's IR-less x2APIC, when standard x2APIC is the future.
Yes, Since IR is only useful for the external device, and 255 CPUs is enough to handle the interrupts from external devices. Besides, i think virtual VT-d will bring extra performance impaction for devices, so if IR-less X2APIC also works well with more than 255 VCPUs, maybe extending KVM with IR-less x2apic is not a bad idea.
So will you use x2APIC physical mode in this system?
Probably, cluster mode is the better choice.
Quoted 1 line
Try not to send a multicast IPI to 400 cores in the VM...
Yes, a multicast IPI to so many cores is a disaster in VM, like flush_tlb_others().
Signature · 2 lines
best regards yang
Quoted 23 lines · Yang Zhang
Quoted 22 lines · Radim Krčmář
2016-04-28 08:54+0200, Jan Kiszka:
Quoted 11 lines · Yang Zhang
Quoted 10 lines · Jan Kiszka
Quoted 6 lines · Yang Zhang
Quoted 2 lines
I mean in Tianyu's case, if he doesn't care about to deliver external interrupt to CPU >255, IR is not required.
What matters is the guest OS. See my other reply on this why this doesn't work, even for Linux.
Since there only few devices in his case, set the irq affinity manually is enough.
You could configure non-IPIs to work, but we want to create options that are hard to break.
Quoted 2 lines
Ah, wait - are we talking about emulating the Xeon Phi architecture in QEMU, accelerated by KVM?
Knights Landing will also be manufactured as a CPU, hopefully without many peculiarities.
Quoted 8 lines
Quoted 2 lines
I think we are talking about extending KVM's IR-less x2APIC, when standard x2APIC is the future.
Yes, Since IR is only useful for the external device, and 255 CPUs is enough to handle the interrupts from external devices. Besides, i think virtual VT-d will bring extra performance impaction for devices, so if IR-less X2APIC also works well with more than 255 VCPUs, maybe extending KVM with IR-less x2apic is not a bad idea.
IR-less x2APIC for guest architectures that are expected to provide IR remains a bad idea, at least until we have hard numbers what this suspected performance impact actually is. Unless you update IRQ affinities an insane rates, the impact should not be relevant because remapping results are cached (for the irqfd hot-path) or you are already taking the long way (userspace device emulation).
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT RDA ITP SES-DE Corporate Competence Center Embedded Linux
Quoted 33 lines · Jan Kiszka
Quoted 23 lines · Yang Zhang
Quoted 22 lines · Radim Krčmář
2016-04-28 08:54+0200, Jan Kiszka:
Quoted 11 lines · Yang Zhang
Quoted 10 lines · Jan Kiszka
Quoted 6 lines · Yang Zhang
Quoted 2 lines
I mean in Tianyu's case, if he doesn't care about to deliver external interrupt to CPU >255, IR is not required.
What matters is the guest OS. See my other reply on this why this doesn't work, even for Linux.
Since there only few devices in his case, set the irq affinity manually is enough.
You could configure non-IPIs to work, but we want to create options that are hard to break.
Quoted 2 lines
Ah, wait - are we talking about emulating the Xeon Phi architecture in QEMU, accelerated by KVM?
Knights Landing will also be manufactured as a CPU, hopefully without many peculiarities.
Quoted 8 lines
Quoted 2 lines
I think we are talking about extending KVM's IR-less x2APIC, when standard x2APIC is the future.
Yes, Since IR is only useful for the external device, and 255 CPUs is enough to handle the interrupts from external devices. Besides, i think virtual VT-d will bring extra performance impaction for devices, so if IR-less X2APIC also works well with more than 255 VCPUs, maybe extending KVM with IR-less x2apic is not a bad idea.
Quoted 6 lines
IR-less x2APIC for guest architectures that are expected to provide IR remains a bad idea, at least until we have hard numbers what this suspected performance impact actually is. Unless you update IRQ affinities an insane rates, the impact should not be relevant because remapping results are cached (for the irqfd hot-path) or you are already taking the long way (userspace device emulation).
I think it is not only interrupt. There must have the DMAR emulation and the cost for DMA is heavy in VM(DMA operations are very frequently). I cannot remember whether there are strong dependency in hardware between DMAR and IR(I know IR is relying on QI). Even hardware dependency is ok, is it ok for OS running in hardware with IR but without DMAR?
Signature · 2 lines
best regards yang
Quoted 5 lines · Yang Zhang
I think it is not only interrupt. There must have the DMAR emulation and the cost for DMA is heavy in VM(DMA operations are very frequently). I cannot remember whether there are strong dependency in hardware between DMAR and IR(I know IR is relying on QI). Even hardware dependency is ok, is it ok for OS running in hardware with IR but without DMAR?
Do you know a way for the IOMMU to report that DMAR is disabled, while IR is enabled?
Anyhow, the VM can use IOMMU passthrough mode to avoid most IOMMU overhead. Regardless, a recent patch-set should improve DMAR performance considerably [1].
Regards,
Nadav
[1] https://www.mail-archive.com/[email protected]/msg12386.html
Quoted 10 lines · Nadav Amit
Quoted 5 lines · Yang Zhang
I think it is not only interrupt. There must have the DMAR emulation and the cost for DMA is heavy in VM(DMA operations are very frequently). I cannot remember whether there are strong dependency in hardware between DMAR and IR(I know IR is relying on QI). Even hardware dependency is ok, is it ok for OS running in hardware with IR but without DMAR?
Do you know a way for the IOMMU to report that DMAR is disabled, while IR is enabled?
The hardware cannot decide about disabling this, but the guest can, of course. In fact, you can even configure Linux to have DMAR off by default until you pass "intel_iommu=on" on the command line (I think distros still do this - at least they used to). No idea about other OSes, though.
Quoted 4 lines
Anyhow, the VM can use IOMMU passthrough mode to avoid most IOMMU overhead. Regardless, a recent patch-set should improve DMAR performance considerably [1].
The bottleneck with emulated DMAR is rather in QEMU. But DMAR can be almost as cheap as IR once we get it running for VFIO and vhost: both need proper caching because they do not work with QEMU in the loop for each and every DMA transfer. Still no need to deviate from physical hardware.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT RDA ITP SES-DE Corporate Competence Center Embedded Linux
Quoted 9 lines · Jan Kiszka
Quoted 8 lines · Nadav Amit
Quoted 5 lines · Yang Zhang
I think it is not only interrupt. There must have the DMAR emulation and the cost for DMA is heavy in VM(DMA operations are very frequently). I cannot remember whether there are strong dependency in hardware between DMAR and IR(I know IR is relying on QI). Even hardware dependency is ok, is it ok for OS running in hardware with IR but without DMAR?
Quoted 8 lines
Quoted 2 lines
Do you know a way for the IOMMU to report that DMAR is disabled, while IR is enabled?
The hardware cannot decide about disabling this, but the guest can, of course. In fact, you can even configure Linux to have DMAR off by default until you pass "intel_iommu=on" on the command line (I think distros still do this - at least they used to). No idea about other OSes, though.
If we can disable DMAR in guest, it should be enough.
Quoted 11 lines
Quoted 5 lines
Anyhow, the VM can use IOMMU passthrough mode to avoid most IOMMU overhead. Regardless, a recent patch-set should improve DMAR performance considerably [1].
The bottleneck with emulated DMAR is rather in QEMU. But DMAR can be almost as cheap as IR once we get it running for VFIO and vhost: both need proper caching because they do not work with QEMU in the loop for each and every DMA transfer. Still no need to deviate from physical hardware.
Sorry, i don't know detail about how VFIO and vhost work with IR. But it seems hard to do proper caching since DMA allocations are very frequently in Linux unless we move the whole iommu emulation to kernel. Another idea is using two iommus: one for Qemu and one for device in kernel like vfio and vhost. I did the similar thing in Xen in several years ago which uses two iommus solution and it works well in my experiment environment. Besides, this solution is easy for nested device pass-through. The Page 32 of [1] has more detail.
[1] http://docplayer.net/10559370-Nested-virtualization.html
Signature · 2 lines
best regards yang
Quoted 9 lines · Yang Zhang
Quoted 7 lines · Jan Kiszka
On 2016-05-03 04:03, Nadav Amit wrote:
The bottleneck with emulated DMAR is rather in QEMU. But DMAR can be almost as cheap as IR once we get it running for VFIO and vhost: both need proper caching because they do not work with QEMU in the loop for each and every DMA transfer. Still no need to deviate from physical hardware.
Quoted 3 lines
Sorry, i don't know detail about how VFIO and vhost work with IR. But it seems hard to do proper caching since DMA allocations are very frequently in Linux unless we move the whole iommu emulation to kernel. Another idea is using two iommus: one for Qemu and one for device in kernel like vfio and vhost. I did the similar thing in Xen in several years ago which uses two iommus solution and it works well in my experiment environment. Besides, this solution is easy for nested device pass-through. The Page 32 of [1] has more detail.
[1] http://docplayer.net/10559370-Nested-virtualization.html
I did a similar work as well several years ago [2], and achieved similar results. The problem with these results is that you don’t show the CPU utilization. Sure, for 1GBE netperf it might be fine, but I am not sure it would be useful for more demanding tasks.
Isn’t it possible to use the PASID as a sort of virtual function?
Regards,
Nadav
[2] https://www.usenix.org/legacy/event/atc11/tech/final_files/Amit.pdf
Quoted 14 lines · Yang Zhang
Quoted 7 lines · Jan Kiszka
Quoted 5 lines · Nadav Amit
Anyhow, the VM can use IOMMU passthrough mode to avoid most IOMMU overhead.
Regardless, a recent patch-set should improve DMAR performance considerably [1].Quoted 5 lines
The bottleneck with emulated DMAR is rather in QEMU. But DMAR can be almost as cheap as IR once we get it running for VFIO and vhost: both need proper caching because they do not work with QEMU in the loop for each and every DMA transfer. Still no need to deviate from physical hardware.
Quoted 3 lines
Sorry, i don't know detail about how VFIO and vhost work with IR. But it seems hard to do proper caching since DMA allocations are very frequently in Linux unless we move the whole iommu emulation to kernel.
There is technically no reason for Linux to reprogram the DMAR units unless it changes partitioning (or really wants to enforce strict DMA containment for each device). You can surely tune this to no updates at all for the guest Linux under normal operations.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT RDA ITP SES-DE Corporate Competence Center Embedded Linux
Hi Radim:
Quoted 11 lines · Radim Krčmář
2016-04-26 18:17+0200, Jan Kiszka:
Quoted 8 lines · Tianyu
Quoted 7 lines
Hi All:
Recently I am working on extending max vcpu to more than 256 on the both KVM/Xen. For some HPC cases, it needs many vcpus. The job requires to use X2APIC in the guest which supports 32-bit APIC id. Linux kernel requires irq remapping function during enabling X2APIC when max APIC id is more than 255(More detail please see
try_to_enable_x2apic()).Our of curiosity, how many VCPUs are you aiming at?
I think it's 1024.
In the short term, we hope hypervisor at least supports 288 vcpus because Xeon phi chip already supports 288 logical cpus. As hardware development, there will be more logical cpus and we hope one guest can totally uses all cpu resources on the chip to meet HPC requirement.
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
The irq remapping function helps to deliver irq to cpu 255~. IOAPIC just supports 8-bit target APIC id field and only can deliver irq to cpu 0~255.
So far both KVM/Xen doesn't enable irq remapping function. If enable the function, it seems a huge job which need to rework IO-APIC, local APIC, MSI parts and add virtual VTD support in the KVM.
Other quick way to enable more than 256 VCPUs is to eliminate the dependency between irq remapping and X2APIC in the guest linux kernel. So far I can boot the guest after removing the dependency. The side effect I thought is that irq only can deliver to 0~255 vcpus but 256 vcpus seem enough to balance irq requests in the guest. In the most cases, there are fewer devices in the guest.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
I wonder whether it's feasible. There maybe some other side effects I didn't think of. Very appreciate for your comments.
Radim is working on the KVM side already, Peter is currently driving the VT-d interrupt emulation topic in QEMU. It's in reach, I would say. :)
+ Igor extends QEMU to support more than 255 in internal structures and ACPI. What remains mostly untracked is Seabios/OVMF.
Thanks for you information. How about KVM X2APIC part? Do you have patch to extend KVM X2APIC to support 32-bit APIC ID?
Quoted 10 lines
Quoted 2 lines
PS: Please no PV mess, at least without good reasons.
Seconded.
(If we designed all related devices as virtware, then it would not be
that bad, but slightly modifying and putting hardware drivers into situations that cannot happen in hardware, not even in the spec, and then juggling the KVM side to make them work, is a road to hell.)
Signature · 2 lines
Best regards Tianyu Lan
2016-04-27 13:39+0800, Lan Tianyu:
Quoted 11 lines · Radim Krčmář
Quoted 10 lines
2016-04-26 18:17+0200, Jan Kiszka:
Quoted 8 lines · Tianyu
Quoted 7 lines
Hi All:
Recently I am working on extending max vcpu to more than 256 on the both KVM/Xen. For some HPC cases, it needs many vcpus. The job requires to use X2APIC in the guest which supports 32-bit APIC id. Linux kernel requires irq remapping function during enabling X2APIC when max APIC id is more than 255(More detail please see
try_to_enable_x2apic()).
Quoted 8 lines
Quoted 1 line
Our of curiosity, how many VCPUs are you aiming at?
I think it's 1024.
In the short term, we hope hypervisor at least supports 288 vcpus because Xeon phi chip already supports 288 logical cpus. As hardware development, there will be more logical cpus and we hope one guest can totally uses all cpu resources on the chip to meet HPC requirement.
Thanks, I think KVM will start by bumping the hard VCPU limit to 512 or 1024, with recommended maximum being 288. You'll be able to raise the hard limit just by configuing and recompiling.
Quoted 2 lines
How about KVM X2APIC part? Do you have patch
to extend KVM X2APIC to support 32-bit APIC ID?
I do, in limbo, as QEMU cannot create VCPUs with higher APIC IDs, yet.
Quoted 24 lines · Jan Kiszka
Quoted 23 lines · Tianyu
Hi All:
Recently I am working on extending max vcpu to more than 256 on the both KVM/Xen. For some HPC cases, it needs many vcpus. The job requires to use X2APIC in the guest which supports 32-bit APIC id. Linux kernel requires irq remapping function during enabling X2APIC when max APIC id is more than 255(More detail please see
try_to_enable_x2apic()).The irq remapping function helps to deliver irq to cpu 255~. IOAPIC just supports 8-bit target APIC id field and only can deliver irq to cpu 0~255.
So far both KVM/Xen doesn't enable irq remapping function. If enable the function, it seems a huge job which need to rework IO-APIC, local APIC, MSI parts and add virtual VTD support in the KVM.
Other quick way to enable more than 256 VCPUs is to eliminate the dependency between irq remapping and X2APIC in the guest linux kernel. So far I can boot the guest after removing the dependency. The side effect I thought is that irq only can deliver to 0~255 vcpus but 256 vcpus seem enough to balance irq requests in the guest. In the most cases, there are fewer devices in the guest.
Quoted 5 lines
Quoted 2 lines
I wonder whether it's feasible. There maybe some other side effects I didn't think of. Very appreciate for your comments.
Radim is working on the KVM side already, Peter is currently driving the VT-d interrupt emulation topic in QEMU. It's in reach, I would say. :)
Oh. Thanks for your information. Very helpful :)
Quoted 5 lines
Jan
PS: Please no PV mess, at least without good reasons.
Signature · 2 lines
Best regards Tianyu Lan