enable x2APIC without interrupt remapping under KVM
PATCH v3 1/1 enable x2APIC without interrupt remapping under KVM
KVM would like to provide x2APIC interface to a guest without emulating interrupt remapping device. The reason KVM prefers guest to use x2APIC is that x2APIC interface is better virtualizable and provides better performance than mmio xAPIC interface:
- msr exits are faster than mmio (no page table walk, emulation)
- no need to read back ICR to look at the busy bit
- one 64 bit ICR write instead of two 32 bit writes
- shared code with the Hyper-V paravirt interface
Included patch changes x2APIC enabling logic to enable it even if IR initialization failed, but kernel runs under KVM and no apic id is greater than 255 (if there is one spec requires BIOS to move to x2apic mode before starting an OS).
---
v2:
Add help message to Kconfig
Force physical mode if IR is not available.
v3:
disable io-apic in outer function. Enable x2apic mode with masked io-apic
arch/x86/Kconfig +5 -2
@@ -260,12 +260,15 @@ config SMP config X86_X2APIC bool "Support x2apic" - depends on X86_LOCAL_APIC && X86_64 && INTR_REMAP + depends on X86_LOCAL_APIC && X86_64 ---help--- This enables x2apic support on CPUs that have this feature. This allows 32-bit apic IDs (so it can support very large systems), - and accesses the local apic via MSRs not via mmio. + and accesses the local apic via MSRs not via mmio. CONFIG_INTR_REMAP + is required for x2apic to take affect when running on physical + machine. If you intend to run the kernel as KVM guest x2apic can + be used without interrupt remapping. If you don't know what to do here, say N.
arch/x86/kernel/apic/apic.c +49 -37
148 lines not on this page · raw message
--
Gleb.
arch/x86/Kconfig config SMP · line 263
Quoted 10 lines · Gleb Natapov
arch/x86/Kconfig +1
-1@@ -260,12 +260,15 @@ config SMP config X86_X2APIC bool "Support x2apic" - depends on X86_LOCAL_APIC && X86_64 && INTR_REMAP + depends on X86_LOCAL_APIC && X86_64
Thinking more, probably we shouldn't remove this dependency. This might encourage people (knowingly or unknowingly) to enable x2apic without interrupt-remapping. Can we remove this? KVM mode will still work even if we fail to enable interrupt-remapping. So this shouldn't be an issue, correct?
Sorry I should have commented about this before.
arch/x86/Kconfig config SMP · line 267
Quoted 7 lines
arch/x86/Kconfig +2
-2ioapic_entries = alloc_ioapic_entries(); if (!ioapic_entries) { - pr_info("Allocate ioapic_entries failed: %d\n", ret); - goto end; + pr_info("Allocate ioapic_entries failed\n"); + return;
We should go to ir_failed ..
arch/x86/Kconfig config SMP
Quoted 8 lines
arch/x86/Kconfig +2
-1} ret = save_IO_APIC_setup(ioapic_entries); if (ret) { pr_info("Saving IO-APIC state failed: %d\n", ret); - goto end; + free_ioapic_entries(ioapic_entries); + return;
same here.
In few hours I will be on 24 hour flight. So please bear with me for delayed responses.
thanks,
suresh
Quoted 7 lines · Suresh Siddha
Quoted 6 lines · Gleb Natapov
arch/x86/Kconfig +0
-0@@ -260,12 +260,15 @@ config SMP
Quoted 11 lines
arch/x86/Kconfig config SMP · line 263
Quoted 4 lines
arch/x86/Kconfig +1
-1config X86_X2APIC bool "Support x2apic" - depends on X86_LOCAL_APIC && X86_64 && INTR_REMAP + depends on X86_LOCAL_APIC && X86_64Thinking more, probably we shouldn't remove this dependency. This might encourage people (knowingly or unknowingly) to enable x2apic without interrupt-remapping. Can we remove this? KVM mode will still work even if we fail to enable interrupt-remapping. So this shouldn't be an issue, correct?
Yes, KVM will still work. I don't have strong fillings one way or the other, but why mandate an option that is no longer mandatory. What others think?
Quoted 12 lines
Sorry I should have commented about this before.
arch/x86/Kconfig config SMP · line 267
Quoted 7 lines
arch/x86/Kconfig +2
-2ioapic_entries = alloc_ioapic_entries(); if (!ioapic_entries) { - pr_info("Allocate ioapic_entries failed: %d\n", ret); - goto end; + pr_info("Allocate ioapic_entries failed\n"); + return;We should go to
ir_failed..
Why? There is not more ir_failed.
Quoted 12 lines
arch/x86/Kconfig config SMP
Quoted 9 lines
arch/x86/Kconfig +2
-1} ret = save_IO_APIC_setup(ioapic_entries); if (ret) { pr_info("Saving IO-APIC state failed: %d\n", ret); - goto end; + free_ioapic_entries(ioapic_entries); + return;same here.
Why? What ir_failed should do? Call free_ioapic_entries(ioapic_entries) and exit?
--
Gleb.
Quoted 10 lines · Gleb Natapov
Quoted 6 lines · Suresh Siddha
Thinking more, probably we shouldn't remove this dependency. This might encourage people (knowingly or unknowingly) to enable x2apic without interrupt-remapping. Can we remove this? KVM mode will still work even if we fail to enable interrupt-remapping. So this shouldn't be an issue, correct?
Yes, KVM will still work. I don't have strong fillings one way or the other, but why mandate an option that is no longer mandatory. What others think?
Only under the presence of KVM we are breaking this dependency. And typically the same kernel runs natively and under the presence of kvm correct. So thats why we shouldn't break this dependency.
Quoted 10 lines
Quoted 9 lines
arch/x86/kernel/apic/apic.c void enable_x2apic(void) · line 1409
Quoted 6 lines
arch/x86/kernel/apic/apic.c +2
-2ioapic_entries =alloc_ioapic_entries ();if (!ioapic_entries) { -pr_info ("Allocate ioapic_entries failed: %d\n" , ret); - goto end; +pr_info ("Allocate ioapic_entries failed\n" ); +return ;We should go to
ir_failed..Why? There is not more
ir_failed.
oops. Not literally. We should goto the point where we should report the x2apic or ir enabling failed and check for x2apic pre-enabled etc at the end.
Quoted 8 lines · Suresh Siddha
Quoted 7 lines · Gleb Natapov
Quoted 6 lines · Suresh Siddha
Thinking more, probably we shouldn't remove this dependency. This might encourage people (knowingly or unknowingly) to enable x2apic without interrupt-remapping. Can we remove this? KVM mode will still work even if we fail to enable interrupt-remapping. So this shouldn't be an issue, correct?
Quoted 8 lines
Quoted 3 lines
Yes, KVM will still work. I don't have strong fillings one way or the other, but why mandate an option that is no longer mandatory. What others think?
Only under the presence of KVM we are breaking this dependency. And typically the same kernel runs natively and under the presence of kvm correct. So thats why we shouldn't break this dependency.
OK, I'll drop Kconfig part in the next version. Unless someone will object till tomorrow.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
arch/x86/kernel/apic/apic.c void enable_x2apic(void) · line 1409
Quoted 7 lines
arch/x86/kernel/apic/apic.c +2
-2ioapic_entries =alloc_ioapic_entries ();if (!ioapic_entries) { -pr_info ("Allocate ioapic_entries failed: %d\n" , ret); - goto end; +pr_info ("Allocate ioapic_entries failed\n" ); +return ;We should go to
ir_failed..
Quoted 6 lines
Quoted 1 line
Why? There is not more
ir_failed.oops. Not literally. We should goto the point where we should report the x2apic or ir enabling failed and check for x2apic pre-enabled etc at the end.
Ah, OK.
--
Gleb.
Quoted 10 lines · Gleb Natapov
KVM would like to provide x2APIC interface to a guest without emulating interrupt remapping device. The reason KVM prefers guest to use x2APIC is that x2APIC interface is better virtualizable and provides better performance than mmio xAPIC interface:
- msr exits are faster than mmio (no page table walk, emulation)
- no need to read back ICR to look at the busy bit
- one 64 bit ICR write instead of two 32 bit writes
- shared code with the Hyper-V paravirt interface
Quoted 4 lines
Included patch changes x2APIC enabling logic to enable it even if IR initialization failed, but kernel runs under KVM and no apic id is greater than 255 (if there is one spec requires BIOS to move to x2apic mode before starting an OS).
How common is hotplug hardware in kvm? In particular hotplug cpus?
To support that seriously you need interrupt remapping.
Eric
Quoted 12 lines · Eric W. Biederman
Quoted 10 lines · Gleb Natapov
KVM would like to provide x2APIC interface to a guest without emulating interrupt remapping device. The reason KVM prefers guest to use x2APIC is that x2APIC interface is better virtualizable and provides better performance than mmio xAPIC interface:
- msr exits are faster than mmio (no page table walk, emulation)
- no need to read back ICR to look at the busy bit
- one 64 bit ICR write instead of two 32 bit writes
- shared code with the Hyper-V paravirt interface
Quoted 8 lines
Quoted 4 lines
Included patch changes x2APIC enabling logic to enable it even if IR initialization failed, but kernel runs under KVM and no apic id is greater than 255 (if there is one spec requires BIOS to move to x2apic mode before starting an OS).
How common is hotplug hardware in kvm? In particular hotplug cpus?
It works for Linux guests.
Quoted 2 lines
To support that seriously you need interrupt remapping.
Can you explain why?
--
Gleb.
Quoted 19 lines · Gleb Natapov
Quoted 18 lines · Eric W. Biederman
Quoted 14 lines · Gleb Natapov
KVM would like to provide x2APIC interface to a guest without emulating interrupt remapping device. The reason KVM prefers guest to use x2APIC is that x2APIC interface is better virtualizable and provides better performance than mmio xAPIC interface:
- msr exits are faster than mmio (no page table walk, emulation)
- no need to read back ICR to look at the busy bit
- one 64 bit ICR write instead of two 32 bit writes
- shared code with the Hyper-V paravirt interfaceIncluded patch changes x2APIC enabling logic to enable it even if IR initialization failed, but kernel runs under KVM and no apic id is greater than 255 (if there is one spec requires BIOS to move to x2apic mode before starting an OS).
Quoted 8 lines
Quoted 2 lines
How common is hotplug hardware in kvm? In particular hotplug cpus?
It works for Linux guests.
Quoted 2 lines
To support that seriously you need interrupt remapping.
Can you explain why?
Because ioapics don't fully function according to spec, and the interrupt code on the hotplug path is a horrible terrible broken hack for ioapics.
It is better than nothing but it certainly is not something I would expect to work all of the time.
Interrupt remapping is the one case where we have hardware that works according to spec and that works reasonably well.
Eric
Quoted 6 lines · Eric W. Biederman
Quoted 5 lines
Quoted 3 lines
To support that seriously you need interrupt remapping.
Can you explain why?
Quoted 7 lines
Because ioapics don't fully function according to spec, and the interrupt code on the hotplug path is a horrible terrible broken hack for ioapics.
It is better than nothing but it certainly is not something I would expect to work all of the time.
Can you elaborate? For kvm guests, the hardware is reasonably will implemented and if not we will fix it. We need not cripple a feature just because some hardware is broken.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 11 lines · Avi Kivity
Quoted 10 lines · Eric W. Biederman
Quoted 5 lines
Quoted 3 lines
To support that seriously you need interrupt remapping.
Can you explain why?
Because ioapics don't fully function according to spec, and the interrupt code on the hotplug path is a horrible terrible broken hack for ioapics.
Quoted 7 lines
Quoted 3 lines
It is better than nothing but it certainly is not something I would expect to work all of the time.
Can you elaborate? For kvm guests, the hardware is reasonably will implemented and if not we will fix it. We need not cripple a feature just because some hardware is broken.
The short version is I don't know what work arounds we will ultimately decide to deploy to work with real hardware.
I have been seriously contemplating causing a cpu hot-unplug request to fail if we are in ioapic mode and we have irqs routed to the cpu that is being unplugged.
Even with perfectly working hardware it is not possible in the general case to migrate an ioapic irq from one cpu to another outside of an interrupt handler without without risking dropping an interrupt. There is no general way to know you have seen the last interrupt floating around your system. PCI ordering rules don't help because the ioapics can potentially take an out of band channel.
The interrupt remapping hardware is pci and is only present on systems with in-band interrupts, so we can flush in-flight interrupts using pci reads. We can safely and do reprogram the hardware to point at different cpus at arbitrary times.
So if you possibly can please emulate hardware that is known to work reliably for cpu hotplug.
Eric
Quoted 5 lines · Eric W. Biederman
Quoted 4 lines
Can you elaborate? For kvm guests, the hardware is reasonably will implemented and if not we will fix it. We need not cripple a feature just because some hardware is broken.
Quoted 7 lines
The short version is I don't know what work arounds we will ultimately decide to deploy to work with real hardware.
I have been seriously contemplating causing a cpu hot-unplug request to fail if we are in ioapic mode and we have irqs routed to the cpu that is being unplugged.
Well, obviously we need to disassociate any irqs from such a cpu. Could be done from the kernel or only enforced by the kernel.
Quoted 4 lines
Even with perfectly working hardware it is not possible in the general case to migrate an ioapic irq from one cpu to another outside of an interrupt handler without without risking dropping an interrupt.
Can't you generate a spurious interrupt immediately after the migration? An extra interrupt shouldn't hurt.
Quoted 4 lines
There is no general way to know you have seen the last interrupt floating around your system. PCI ordering rules don't help because the ioapics can potentially take an out of band channel.
Can you describe the problem scenario? an ioapic->lapic message delivered to a dead cpu?
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 9 lines · Avi Kivity
Quoted 8 lines · Eric W. Biederman
Quoted 4 lines
Can you elaborate? For kvm guests, the hardware is reasonably will implemented and if not we will fix it. We need not cripple a feature just because some hardware is broken.
The short version is I don't know what work arounds we will ultimately decide to deploy to work with real hardware.
Quoted 7 lines
Quoted 4 lines
I have been seriously contemplating causing a cpu hot-unplug request to fail if we are in ioapic mode and we have irqs routed to the cpu that is being unplugged.
Well, obviously we need to disassociate any irqs from such a cpu. Could be done from the kernel or only enforced by the kernel.
Using the normal irq migration path we can move irqs off of a cpu reliably there just aren't any progress guarantees.
Quoted 7 lines
Quoted 4 lines
Even with perfectly working hardware it is not possible in the general case to migrate an ioapic irq from one cpu to another outside of an interrupt handler without without risking dropping an interrupt.
Can't you generate a spurious interrupt immediately after the migration? An extra interrupt shouldn't hurt.
Nope. The ioapics can't be told to send an interrupt.
Quoted 7 lines
Quoted 4 lines
There is no general way to know you have seen the last interrupt floating around your system. PCI ordering rules don't help because the ioapics can potentially take an out of band channel.
Can you describe the problem scenario? an
ioapic->lapicmessage delivered to a dead cpu?
Dropped irqs.. Driver hangs because it is waiting for an irq. Hardware hangs because it is waiting for the cpu to process the irq.
Potentially we get a level triggered irq that is never acked by the cpu that won't arm until the cpu send an ack, and we can't send an ack from another cpu.
Eric
Quoted 8 lines · Eric W. Biederman
Quoted 8 lines
Quoted 8 lines
The short version is I don't know what work arounds we will ultimately decide to deploy to work with real hardware.
I have been seriously contemplating causing a cpu hot-unplug request to fail if we are in ioapic mode and we have irqs routed to the cpu that is being unplugged.
Quoted 7 lines
Quoted 3 lines
Well, obviously we need to disassociate any irqs from such a cpu. Could be done from the kernel or only enforced by the kernel.
Using the normal irq migration path we can move irqs off of a cpu reliably there just aren't any progress guarantees.
Program the ioapic to the new cpu. Wait a few milliseconds. If it takes more than that to get an interrupt from the ioapic to the local apic, the machine has much bigger problems.
Quoted 11 lines
Quoted 8 lines
Quoted 5 lines
Even with perfectly working hardware it is not possible in the general case to migrate an ioapic irq from one cpu to another outside of an interrupt handler without without risking dropping an interrupt.
Can't you generate a spurious interrupt immediately after the migration? An extra interrupt shouldn't hurt.
Nope. The ioapics can't be told to send an interrupt.
You can program the local apic ICR to generate an interrupt with the same vector.
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines
There is no general way to know you have seen the last interrupt floating around your system. PCI ordering rules don't help because the ioapics can potentially take an out of band channel.
Can you describe the problem scenario? an
ioapic->lapicmessage delivered to a dead cpu?
Quoted 8 lines
Dropped irqs.. Driver hangs because it is waiting for an irq. Hardware hangs because it is waiting for the cpu to process the irq.
Potentially we get a level triggered irq that is never acked by the cpu that won't arm until the cpu send an ack, and we can't send an ack from another cpu.
I think a spurious interrupt generated through the local apic solves that problem. For level-triggered interrupts, mask them before offlining the cpu.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 13 lines · Avi Kivity
Quoted 12 lines · Eric W. Biederman
Quoted 11 lines
Quoted 8 lines
The short version is I don't know what work arounds we will ultimately decide to deploy to work with real hardware.
I have been seriously contemplating causing a cpu hot-unplug request to fail if we are in ioapic mode and we have irqs routed to the cpu that is being unplugged.
Well, obviously we need to disassociate any irqs from such a cpu. Could be done from the kernel or only enforced by the kernel.
Quoted 7 lines
Quoted 3 lines
Using the normal irq migration path we can move irqs off of a cpu reliably there just aren't any progress guarantees.
Program the ioapic to the new cpu. Wait a few milliseconds. If it takes more than that to get an interrupt from the ioapic to the local apic, the machine has much bigger problems.
In general you can not reprogram an ioapic safely unless the interrupt is blocked at the source. Which is why you either need the originating device disabled or you have to do it in interrupt context.
I forget all of the details. I just know in real hardware I experimented with it a lot, and wound up hanging the ioapic state machine of both intel and amd ioapics.
Migrating ioapic irqs in interrupt context sucks. It just happens to be what works reliably.
I do think the wait an eternity in computer time a short while in human time is a valid technique when you can do nothing better. If flushing the interrupt was my only problem that would solve it.
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines
Even with perfectly working hardware it is not possible in the general case to migrate an ioapic irq from one cpu to another outside of an interrupt handler without without risking dropping an interrupt.
Can't you generate a spurious interrupt immediately after the migration? An extra interrupt shouldn't hurt.
Quoted 5 lines
Quoted 2 lines
Nope. The ioapics can't be told to send an interrupt.
You can program the local apic ICR to generate an interrupt with the same vector.
But you can not program the apic ICR to generate a level triggered interrupt with the same vector. So you don't get the broadcast behavior when you ack the apic.
Quoted 12 lines
Quoted 12 lines
Quoted 8 lines
Quoted 5 lines
There is no general way to know you have seen the last interrupt floating around your system. PCI ordering rules don't help because the ioapics can potentially take an out of band channel.
Can you describe the problem scenario? an
ioapic->lapicmessage delivered to a dead cpu?Dropped irqs.. Driver hangs because it is waiting for an irq. Hardware hangs because it is waiting for the cpu to process the irq.
Quoted 8 lines
Quoted 5 lines
Potentially we get a level triggered irq that is never acked by the cpu that won't arm until the cpu send an ack, and we can't send an ack from another cpu.
I think a spurious interrupt generated through the local apic solves that problem. For level-triggered interrupts, mask them before offlining the cpu.
So now we have a masked unacked irq. It doesn't help in the slightest that the cpu migration code puts irq migration last and request that we do it all with interrupts disabled.
You might be right that by application of extreme ingenuity and completely in spec ioapics there is a path that makes this all work. Currently I don't have fully in spec ioapcis, and I don't have anyone interested enough in cpu hotplug to be willing to rip things apart until interrupt migration
is a reasonable deal on x86. Instead all I see are patches that mitigate
the worst of the brokenness.
At the same time with the interrupt remapping hardware because it doesn't need the irq disabled at the source when we reprogram it I can make everything stable much more easily.
Eric
Quoted 6 lines · Eric W. Biederman
Dropped irqs.. Driver hangs because it is waiting for an irq. Hardware hangs because it is waiting for the cpu to process the irq.
Potentially we get a level triggered irq that is never acked by the cpu that won't arm until the cpu send an ack, and we can't send an ack from another cpu.
Eric,
Among number of experiments you have tried in the past to fix this, have you tried the experiment of explicitly clearing the remoteIRR by changing the trigger mode to edge and then back to level.
Is there a problem with this?
We can send a spurious IPI (after the RTE migration) with the new vector to another cpu and handler which services the level interrupt will check if we saw the edge mode for a level interrupt and then the handler can explicitly restore the level trigger and reset the remote IRR by mask +edge and unmask+level.
We might have to work with some rough edges but do you recollect any major issue with this approach..
thanks,
suresh
Quoted 8 lines · Suresh Siddha
Quoted 6 lines · Eric W. Biederman
Dropped irqs.. Driver hangs because it is waiting for an irq. Hardware hangs because it is waiting for the cpu to process the irq.
Potentially we get a level triggered irq that is never acked by the cpu that won't arm until the cpu send an ack, and we can't send an ack from another cpu.
Quoted 7 lines
Eric,
Among number of experiments you have tried in the past to fix this, have you tried the experiment of explicitly clearing the remoteIRR by changing the trigger mode to edge and then back to level.
Is there a problem with this?
The problem I had wasn't remoteIRR getting stuck, but the symptoms were largely the same. I did try changing the trigger mode to edge and back and that did not unstick the ioapic in all cases.
Quoted 8 lines
We can send a spurious IPI (after the RTE migration) with the new vector to another cpu and handler which services the level interrupt will check if we saw the edge mode for a level interrupt and then the handler can explicitly restore the level trigger and reset the remote IRR by mask +edge and unmask+level.
We might have to work with some rough edges but do you recollect any major issue with this approach..
This is coming up enough recently I expect it is time to cook up a patch that does the ioapic migration in process context plus some user space code that stress tests things. Just so people can repeat my experiments and see what I am trying to avoid.
Eric
Quoted 11 lines · Eric W. Biederman
Quoted 5 lines · Suresh Siddha
Among number of experiments you have tried in the past to fix this, have you tried the experiment of explicitly clearing the remoteIRR by changing the trigger mode to edge and then back to level.
Is there a problem with this?
The problem I had wasn't remoteIRR getting stuck, but the symptoms were largely the same. I did try changing the trigger mode to edge and back and that did not unstick the ioapic in all cases.
I did some experiments locally here yesterday and on my old ICH5 based system, I couldn't reset the remoteIRR by changing the trigger mode to edge and then back to level.
However what worked was an explicit eoi to the io-apic using the respective vector.
I guess we need to try both the things based on perhaps io-apic version etc.
But what I am nervous about is, did you try both these things aswell and still saw stuck interrupts?
I will cleanup my code and post it, so that we can try on different systems. If this still doesn't work on certain HW platforms, atleast our experiments of what works and what doesn't work and on what platforms will be documented on the web.
thanks,
suresh
Quoted 17 lines · Suresh Siddha
Quoted 11 lines · Eric W. Biederman
Quoted 5 lines · Suresh Siddha
Among number of experiments you have tried in the past to fix this, have you tried the experiment of explicitly clearing the remoteIRR by changing the trigger mode to edge and then back to level.
Is there a problem with this?
The problem I had wasn't remoteIRR getting stuck, but the symptoms were largely the same. I did try changing the trigger mode to edge and back and that did not unstick the ioapic in all cases.
I did some experiments locally here yesterday and on my old ICH5 based system, I couldn't reset the remoteIRR by changing the trigger mode to edge and then back to level.
Quoted 5 lines
However what worked was an explicit eoi to the io-apic using the respective vector.
I guess we need to try both the things based on perhaps io-apic version etc.
In part the deep problems I ran into was something other than RemoteIRR. If it was something as simple as the ioapic being in a documented state I would have kept looking. At the very least the RemoteIRR bit was not set in some of the cases I encountered.
Quoted 2 lines
But what I am nervous about is, did you try both these things aswell and still saw stuck interrupts?
Yes. The work arounds were in the code so I tried them.
Quoted 4 lines
I will cleanup my code and post it, so that we can try on different systems. If this still doesn't work on certain HW platforms, atleast our experiments of what works and what doesn't work and on what platforms will be documented on the web.
Sounds reasonable. My apologies for the long delayed reply. I want to work on this but I have higher priorities.
Eric
Quoted 26 lines · Eric W. Biederman
Quoted 24 lines · Gleb Natapov
Quoted 12 lines · Eric W. Biederman
Quoted 10 lines · Gleb Natapov
KVM would like to provide x2APIC interface to a guest without emulating interrupt remapping device. The reason KVM prefers guest to use x2APIC is that x2APIC interface is better virtualizable and provides better performance than mmio xAPIC interface:
- msr exits are faster than mmio (no page table walk, emulation)
- no need to read back ICR to look at the busy bit
- one 64 bit ICR write instead of two 32 bit writes
- shared code with the Hyper-V paravirt interfaceQuoted 8 lines
Quoted 4 lines
Included patch changes x2APIC enabling logic to enable it even if IR initialization failed, but kernel runs under KVM and no apic id is greater than 255 (if there is one spec requires BIOS to move to x2apic mode before starting an OS).
How common is hotplug hardware in kvm? In particular hotplug cpus?
It works for Linux guests.
Quoted 8 lines
Quoted 3 lines
Quoted 2 lines
To support that seriously you need interrupt remapping.
Can you explain why?
Because ioapics don't fully function according to spec, and the interrupt code on the hotplug path is a horrible terrible broken hack for ioapics.
Considering that interrupt remapping is fairly new feature are you saying that hotplug (pci and cpu) on x86 is a horrible hack on Linux?
Quoted 3 lines
It is better than nothing but it certainly is not something I would expect to work all of the time.
Because of horrible code or non-complaint ioapic implementation out there? If later this is not a big issue for KVM.
Quoted 3 lines
Interrupt remapping is the one case where we have hardware that works according to spec and that works reasonably well.
I am sure when there was only one ioapic implementation in existence it worked according to a spec (and if not spec was changed :)) Give interrupt remapping some time and than we will see how above statement holds.
--
Gleb.
Quoted 3 lines · Gleb Natapov
Considering that interrupt remapping is fairly new feature are you saying that hotplug (pci and cpu) on x86 is a horrible hack on Linux?
Just the current cpu hotplug path.
Quoted 5 lines
Quoted 3 lines
It is better than nothing but it certainly is not something I would expect to work all of the time.
Because of horrible code or non-complaint ioapic implementation out there? If later this is not a big issue for KVM.
Both. And even in spec ioapics can't be made to work 100% reliably.
Quoted 7 lines
Quoted 3 lines
Interrupt remapping is the one case where we have hardware that works according to spec and that works reasonably well.
I am sure when there was only one ioapic implementation in existence it worked according to a spec (and if not spec was changed :)) Give interrupt remapping some time and than we will see how above statement holds.
What we need for cpu hotplug/unplug is on the tested path now. That is a significant difference.
Eric