mask/unmasking while servicing MSI(-X) unnecessary?
While looking into using MSI-X for our myri10ge driver, we have seen that the msi(x) code (driver/pci/msi.c) masks the MSI(-X) vector while servicing an interrupt. We are not sure why this masking is needed (for instance no such thing is done for "edge IOAPIC" interrupts). There seems to be already several mechanisms each individually protecting against "loosing an interrupt" without masking:
- the "x86" architecture is able to queue 2 interrupt messages. That guarantees an interrupt handler will always start after the last MSI received (even in the case of a big burst of MSI messages).
- Even if there wasn't that interrupt queuing, ack_APIC_irq() could be moved in the ack() method. Then things would work without masking even on a hypothetical platform where a new interrupt is completely ignored (with no IRR-like register) while servicing the same vector (anyway since this "msi" code is already tied to "x86" architecture specificities, that hypothetical platform might not be relevant).
- Almost every driver/device have their own way of acknowledging interrupts anyway, which also by itself makes the masking/unmasking unnecessary.
- The masking by itself is racy unless the driver interrupt handler starts by making sure the masking request has reached the device with some kind of MMIO-read. Such a MMIO-read is normally the kind of costly requests you are happy to get rid of in the MSI model.
So if it is not useful, it might be better to avoid that systematic mask/unmask pair. This masking has a small but measurable performance impact for our device/driver combo.
Would you agree to suppress that masking (sample patch following)? Or otherwise, is there is a possibility of making it optional on a per-device basis.
Thanks for any comment!
Loic Prylli
Myricom, Inc.
[PATCH] Don't mask the interrupt vector while servicing a MSI interrupt.
---
drivers/pci/msi.c | 19 ++++++------------- 1 files changed, 6 insertions(+), 13 deletions(-)
98e9a27091c3ccdc8a8a72cea9ab9086fb258af3
drivers/pci/msi.c +6 -13
@@ -200,19 +200,12 @@ static void shutdown_msi_irq(unsigned inspin_unlock_irqrestore (&msi_lock, flags); } -static void end_msi_irq_wo_maskbit (unsigned int vector) +static void end_msi_irq (unsigned int vector) {move_native_irq (vector);ack_APIC_irq (); } -static void end_msi_irq_w_maskbit (unsigned int vector) -{ -move_native_irq (vector); -unmask_MSI_irq (vector); -ack_APIC_irq (); -} -static void do_nothing (unsigned int vector) { }
@@ -227,8 +220,8 @@ static struct hw_interrupt_type msix_irq .shutdown = shutdown_msi_irq, .enable = unmask_MSI_irq, .disable = mask_MSI_irq, - .ack = mask_MSI_irq, - .end = end_msi_irq_w_maskbit, + .ack = do_nothing, + .end = end_msi_irq, .set_affinity = set_msi_affinity };
@@ -243,8 +236,8 @@ static struct hw_interrupt_type msi_irq_ .shutdown = shutdown_msi_irq, .enable = unmask_MSI_irq, .disable = mask_MSI_irq, - .ack = mask_MSI_irq, - .end = end_msi_irq_w_maskbit, + .ack = do_nothing, + .end = end_msi_irq, .set_affinity = set_msi_affinity };
@@ -260,7 +253,7 @@ static struct hw_interrupt_type msi_irq_ .enable = do_nothing, .disable = do_nothing, .ack = do_nothing, - .end = end_msi_irq_wo_maskbit, + .end = end_msi_irq, .set_affinity = set_msi_affinity };
Signature · 1 line
1.3.2
Quoted 23 lines · On Wed, 29 Nov 2006 17:46:07 -0500 Loic Prylli <[email protected]> wrote:
While looking into using MSI-X for our myri10ge driver, we have seen that the msi(x) code (
driver/pci/msi.c) masks the MSI(-X) vector while servicing an interrupt. We are not sure why this masking is needed (for instance no such thing is done for "edge IOAPIC" interrupts). There seems to be already several mechanisms each individually protecting against "loosing an interrupt" without masking:
- the "x86" architecture is able to queue 2 interrupt messages. That guarantees an interrupt handler will always start after the last MSI received (even in the case of a big burst of MSI messages).
- Even if there wasn't that interrupt queuing,ack_APIC_irq()could be moved in theack()method. Then things would work without masking even on a hypothetical platform where a new interrupt is completely ignored (with no IRR-like register) while servicing the same vector (anyway since this "msi" code is already tied to "x86" architecture specificities, that hypothetical platform might not be relevant).
- Almost every driver/device have their own way of acknowledging interrupts anyway, which also by itself makes the masking/unmasking unnecessary.
- The masking by itself is racy unless the driver interrupt handler starts by making sure the masking request has reached the device with some kind of MMIO-read. Such a MMIO-read is normally the kind of costly requests you are happy to get rid of in the MSI model.
Quoted 8 lines
So if it is not useful, it might be better to avoid that systematic mask/unmask pair. This masking has a small but measurable performance impact for our device/driver combo.
Would you agree to suppress that masking (sample patch following)? Or otherwise, is there is a possibility of making it optional on a per-device basis.
Your patch appears to be against the prehistoric 2.6.18 kernel. MSI got changed a lot - please test 2.6.19 and see if that needs fixing.