Skip to contentSkip to conversation

x86 PCI: Do not use interrupt links for devices using MSI-X

PATCH In review Rafael J. Wysocki 4 participants 11 messages Jan 4, 2009 – Jan 7, 2009

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86 PCI: Do not use interrupt links for devices using MSI-X

From: Rafael J. Wysocki <[email protected]>

pcibios_enable_device() and pcibios_disable_device() don't handle IRQs for devices that have MSI enabled and it should tread the devices with MSI-X enabled in the same way.

---
 arch/x86/pci/common.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/pci/common.c +2 -2
Index: linux-2.6/arch/x86/pci/common.c
===================================================================
@@ -551,14 +551,14 @@ int pcibios_enable_device(struct pci_dev
 	if ((err = pci_enable_resources(dev, mask)) < 0)
 		return err;
 
-	if (!dev->msi_enabled)
+	if (!dev->msi_enabled && !dev->msix_enabled)
 		return pcibios_enable_irq(dev);
 	return 0;
 }
 
 void pcibios_disable_device (struct pci_dev *dev)
 {
-	if (!dev->msi_enabled && pcibios_disable_irq)
+	if (!dev->msi_enabled && !dev->msix_enabled && pcibios_disable_irq)
 		pcibios_disable_irq(dev);
 }
 
#1 lore raw
Quoted 4 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

pcibios_enable_device() and pcibios_disable_device() don't handle IRQs for devices that have MSI enabled and it should tread the

s/tread/treat

Quoted 6 lines

devices with MSI-X enabled in the same way.

---
 arch/x86/pci/common.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

looks good - Jesse, what do you think?

Rafael, i'm curious is this in response to some regression/bug? Did some box or driver get confused by us enabling/disabling the GSI? Some IRQ flood perhaps?

btw., there's a small observation:

Quoted 1 line

+ if (!dev->msi_enabled && !dev->msix_enabled)

maybe a "pci_has_gsi_irq()" wrapper would make these checks cleaner and would make things more robust, should there be any new IRQ delivery method be introduced in the future?

Ingo

#2 lore raw
Quoted 9 lines · Ingo Molnar
Quoted 4 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

pcibios_enable_device() and pcibios_disable_device() don't handle IRQs for devices that have MSI enabled and it should tread the

s/tread/treat

Ah, thanks.

Quoted 7 lines
Quoted 6 lines

devices with MSI-X enabled in the same way.

---
 arch/x86/pci/common.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
Quoted 5 lines

looks good - Jesse, what do you think?

Rafael, i'm curious is this in response to some regression/bug? Did some box or driver get confused by us enabling/disabling the GSI? Some IRQ flood perhaps?

Well, I don't have any MSI-X capable boxes around. :-)

I was just reviewing the code and spotted this.

Quoted 7 lines

btw., there's a small observation:

Quoted 1 line

+ if (!dev->msi_enabled && !dev->msix_enabled)

maybe a "pci_has_gsi_irq()" wrapper would make these checks cleaner and would make things more robust, should there be any new IRQ delivery method be introduced in the future?

Well, perhaps something like the patch below?

Thanks,
Rafael

---
Subject: x86 PCI: Do not use interrupt links for devices using MSI-X (rev. 2)
From: Rafael J. Wysocki <[email protected]>

pcibios_enable_device() and pcibios_disable_device() don't handle IRQs for devices that have MSI enabled and it should treat the devices with MSI-X enabled in the same way.

---
 arch/x86/pci/common.c |    4 ++--
 include/linux/pci.h   |    9 +++++++++
 2 files changed, 11 insertions(+), 2 deletions(-)
arch/x86/pci/common.c +2 -2
Index: linux-2.6/arch/x86/pci/common.c
===================================================================
@@ -551,14 +551,14 @@ int pcibios_enable_device(struct pci_dev
 	if ((err = pci_enable_resources(dev, mask)) < 0)
 		return err;
 
-	if (!dev->msi_enabled)
+	if (!pci_msi_enabled(dev))
 		return pcibios_enable_irq(dev);
 	return 0;
 }
 
 void pcibios_disable_device (struct pci_dev *dev)
 {
-	if (!dev->msi_enabled && pcibios_disable_irq)
+	if (!pci_msi_enabled(dev) && pcibios_disable_irq)
 		pcibios_disable_irq(dev);
 }
 
include/linux/pci.h +9 -0
Index: linux-2.6/include/linux/pci.h
===================================================================
@@ -274,6 +274,15 @@ static inline void pci_add_saved_cap(str
 	hlist_add_head(&new_cap->next, &pci_dev->saved_cap_space);
 }
 
+#ifdef CONFIG_PCI_MSI
+static inline bool pci_msi_enabled(struct pci_dev *pci_dev)
+{
+	return dev->msi_enabled || dev->msix_enabled;
+}
+#else
+static inline bool pci_msi_enabled(struct pci_dev *pci_dev) { return false; }
+#endif
+
 /*
  *  For PCI devices, the region numbers are assigned this way:
  *
#3 lore raw
Quoted 44 lines · Rafael J. Wysocki
Quoted 7 lines · Ingo Molnar
Quoted 4 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

pcibios_enable_device() and pcibios_disable_device() don't handle IRQs for devices that have MSI enabled and it should tread the

s/tread/treat

Ah, thanks.

Quoted 7 lines
Quoted 6 lines

devices with MSI-X enabled in the same way.

---
 arch/x86/pci/common.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
Quoted 5 lines

looks good - Jesse, what do you think?

Rafael, i'm curious is this in response to some regression/bug? Did some box or driver get confused by us enabling/disabling the GSI? Some IRQ flood perhaps?

Well, I don't have any MSI-X capable boxes around. :-)

I was just reviewing the code and spotted this.

Quoted 6 lines

btw., there's a small observation:

Quoted 1 line

+ if (!dev->msi_enabled && !dev->msix_enabled)

maybe a "pci_has_gsi_irq()" wrapper would make these checks cleaner and would make things more robust, should there be any new IRQ delivery method be introduced in the future?

Well, perhaps something like the patch below?

Thanks,
Rafael

---
Subject: x86 PCI: Do not use interrupt links for devices using MSI-X (rev.
2) From: Rafael J. Wysocki <[email protected]>

Quoted 5 lines

pcibios_enable_device() and pcibios_disable_device() don't handle IRQs for devices that have MSI enabled and it should treat the devices with MSI-X enabled in the same way.

Looks good, applied this one.

Thanks,

Signature · 1 line
Jesse Barnes, Intel Open Source Technology Center
#4 lore raw
In reply to Ingo Molnar
Quoted 8 lines · Ingo Molnar
Quoted 4 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

pcibios_enable_device() and pcibios_disable_device() don't handle IRQs for devices that have MSI enabled and it should tread the

s/tread/treat

Quoted 8 lines
Quoted 6 lines

devices with MSI-X enabled in the same way.

---
 arch/x86/pci/common.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

looks good - Jesse, what do you think?

Yeah, seems obviously correct, I'll queue it up.

Quoted 10 lines

Rafael, i'm curious is this in response to some regression/bug? Did some box or driver get confused by us enabling/disabling the GSI? Some IRQ flood perhaps?

btw., there's a small observation:

Quoted 1 line

+ if (!dev->msi_enabled && !dev->msix_enabled)

maybe a "pci_has_gsi_irq()" wrapper would make these checks cleaner and would make things more robust, should there be any new IRQ delivery method be introduced in the future?

pci_has_msi_irq surely? Otherwise we'll catch pretty much everything? Or did you mean !pci_has_gsi_irq() here instead?

Thanks,

Signature · 1 line
Jesse Barnes, Intel Open Source Technology Center
#5 lore raw
Quoted 25 lines · Jesse Barnes
Quoted 8 lines · Ingo Molnar
Quoted 4 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

pcibios_enable_device() and pcibios_disable_device() don't handle IRQs for devices that have MSI enabled and it should tread the

s/tread/treat

Quoted 8 lines
Quoted 6 lines

devices with MSI-X enabled in the same way.

---
 arch/x86/pci/common.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

looks good - Jesse, what do you think?

Yeah, seems obviously correct, I'll queue it up.

Quoted 5 lines

Rafael, i'm curious is this in response to some regression/bug? Did some box or driver get confused by us enabling/disabling the GSI? Some IRQ flood perhaps?

btw., there's a small observation:

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

+ if (!dev->msi_enabled && !dev->msix_enabled)

maybe a "pci_has_gsi_irq()" wrapper would make these checks cleaner and would make things more robust, should there be any new IRQ delivery method be introduced in the future?

pci_has_msi_irq surely? Otherwise we'll catch pretty much everything? Or did you mean !pci_has_gsi_irq() here instead?

Well - here the check is: "if (not MSI or MSIX)" in essence. I thought that it might be confusing to call it _msi() as well, so we could approach it via the inverse space: general system interrupts (GSIs) - which are device irqs that are neither MSI nor MSIX.

But if pci_has_msi_irq() can cleanly include the MSIX portion too, that's fine too. (MSI-X is really MSI with wider eventing capabilities but otherwise non-GSI just as much - and we dont want to enable (or even touch) the legacy IRQ line registers for any of them, even if they happen to be enumerated)

Right?

Ingo

#6 lore raw
Quoted 40 lines · Ingo Molnar
Quoted 25 lines · Jesse Barnes
Quoted 8 lines · Ingo Molnar
Quoted 4 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

pcibios_enable_device() and pcibios_disable_device() don't handle IRQs for devices that have MSI enabled and it should tread the

s/tread/treat

Quoted 8 lines
Quoted 6 lines

devices with MSI-X enabled in the same way.

---
 arch/x86/pci/common.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

looks good - Jesse, what do you think?

Yeah, seems obviously correct, I'll queue it up.

Quoted 5 lines

Rafael, i'm curious is this in response to some regression/bug? Did some box or driver get confused by us enabling/disabling the GSI? Some IRQ flood perhaps?

btw., there's a small observation:

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

+ if (!dev->msi_enabled && !dev->msix_enabled)

maybe a "pci_has_gsi_irq()" wrapper would make these checks cleaner and would make things more robust, should there be any new IRQ delivery method be introduced in the future?

pci_has_msi_irq surely? Otherwise we'll catch pretty much everything? Or did you mean !pci_has_gsi_irq() here instead?

Well - here the check is: "if (not MSI or MSIX)" in essence. I thought that it might be confusing to call it _msi() as well, so we could approach it via the inverse space: general system interrupts (GSIs) - which are device irqs that are neither MSI nor MSIX.

Quoted 7 lines

But if pci_has_msi_irq() can cleanly include the MSIX portion too, that's fine too. (MSI-X is really MSI with wider eventing capabilities but otherwise non-GSI just as much - and we dont want to enable (or even touch) the legacy IRQ line registers for any of them, even if they happen to be enumerated)

Right?

Right, I see where you're coming from. However, I've queued up Rafael's last patch with some fixes for dev vs. pci_dev and a name collision (pci_msi_enabled -> pci_dev_msi_enabled). Bjorn caught the fact that some other arches may want similar treatment too, I think Rafael is checking that out.

Signature · 1 line
Jesse Barnes, Intel Open Source Technology Center
#7 lore raw
Quoted 47 lines · Jesse Barnes
Quoted 46 lines · Ingo Molnar
Quoted 25 lines · Jesse Barnes
Quoted 8 lines · Ingo Molnar
Quoted 4 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

pcibios_enable_device() and pcibios_disable_device() don't handle IRQs for devices that have MSI enabled and it should tread the

s/tread/treat

Quoted 8 lines
Quoted 6 lines

devices with MSI-X enabled in the same way.

---
 arch/x86/pci/common.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

looks good - Jesse, what do you think?

Yeah, seems obviously correct, I'll queue it up.

Quoted 5 lines

Rafael, i'm curious is this in response to some regression/bug? Did some box or driver get confused by us enabling/disabling the GSI? Some IRQ flood perhaps?

btw., there's a small observation:

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

+ if (!dev->msi_enabled && !dev->msix_enabled)

maybe a "pci_has_gsi_irq()" wrapper would make these checks cleaner and would make things more robust, should there be any new IRQ delivery method be introduced in the future?

pci_has_msi_irq surely? Otherwise we'll catch pretty much everything? Or did you mean !pci_has_gsi_irq() here instead?

Well - here the check is: "if (not MSI or MSIX)" in essence. I thought that it might be confusing to call it _msi() as well, so we could approach it via the inverse space: general system interrupts (GSIs) - which are device irqs that are neither MSI nor MSIX.

But if pci_has_msi_irq() can cleanly include the MSIX portion too, that's fine too. (MSI-X is really MSI with wider eventing capabilities but otherwise non-GSI just as much - and we dont want to enable (or even touch) the legacy IRQ line registers for any of them, even if they happen to be enumerated)

Quoted 7 lines
Quoted 1 line

Right?

Right, I see where you're coming from. However, I've queued up Rafael's last patch with some fixes for dev vs. pci_dev and a name collision (pci_msi_enabled -> pci_dev_msi_enabled). Bjorn caught the fact that some other arches may want similar treatment too, I think Rafael is checking that out.

Sure - that sounds good too!

Ingo

#8 lore raw
In reply to Jesse Barnes
Quoted 47 lines · Jesse Barnes
Quoted 46 lines · Ingo Molnar
Quoted 25 lines · Jesse Barnes
Quoted 8 lines · Ingo Molnar
Quoted 4 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

pcibios_enable_device() and pcibios_disable_device() don't handle IRQs for devices that have MSI enabled and it should tread the

s/tread/treat

Quoted 8 lines
Quoted 6 lines

devices with MSI-X enabled in the same way.

---
 arch/x86/pci/common.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

looks good - Jesse, what do you think?

Yeah, seems obviously correct, I'll queue it up.

Quoted 5 lines

Rafael, i'm curious is this in response to some regression/bug? Did some box or driver get confused by us enabling/disabling the GSI? Some IRQ flood perhaps?

btw., there's a small observation:

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

+ if (!dev->msi_enabled && !dev->msix_enabled)

maybe a "pci_has_gsi_irq()" wrapper would make these checks cleaner and would make things more robust, should there be any new IRQ delivery method be introduced in the future?

pci_has_msi_irq surely? Otherwise we'll catch pretty much everything? Or did you mean !pci_has_gsi_irq() here instead?

Well - here the check is: "if (not MSI or MSIX)" in essence. I thought that it might be confusing to call it _msi() as well, so we could approach it via the inverse space: general system interrupts (GSIs) - which are device irqs that are neither MSI nor MSIX.

But if pci_has_msi_irq() can cleanly include the MSIX portion too, that's fine too. (MSI-X is really MSI with wider eventing capabilities but otherwise non-GSI just as much - and we dont want to enable (or even touch) the legacy IRQ line registers for any of them, even if they happen to be enumerated)

Quoted 7 lines
Quoted 1 line

Right?

Right, I see where you're coming from. However, I've queued up Rafael's last patch with some fixes for dev vs. pci_dev and a name collision (pci_msi_enabled -> pci_dev_msi_enabled). Bjorn caught the fact that some other arches may want similar treatment too, I think Rafael is checking that out.

Yes, I am.

Thanks,
Rafael

#9 lore raw
In reply to Rafael J. Wysocki
Quoted 3 lines · Rafael J. Wysocki

pcibios_enable_device() and pcibios_disable_device() don't handle IRQs for devices that have MSI enabled and it should tread the devices with MSI-X enabled in the same way.

There are other places that need similar fixes, too, aren't there? I see cris, frv, ia64, and a driver or two testing dev->msi_enabled.

Quoted 18 lines
arch/x86/pci/common.c +2 -2
@@ -551,14 +551,14 @@ int pcibios_enable_device(struct pci_dev
 	if ((err = pci_enable_resources(dev, mask)) < 0)
 		return err;
 
-	if (!dev->msi_enabled)
+	if (!dev->msi_enabled && !dev->msix_enabled)
 		return pcibios_enable_irq(dev);
 	return 0;
 }
 
 void pcibios_disable_device (struct pci_dev *dev)
 {
-	if (!dev->msi_enabled && pcibios_disable_irq)
+	if (!dev->msi_enabled && !dev->msix_enabled && pcibios_disable_irq)
 		pcibios_disable_irq(dev);
 }
#10 lore raw
Quoted 7 lines · Bjorn Helgaas
Quoted 3 lines · Rafael J. Wysocki

pcibios_enable_device() and pcibios_disable_device() don't handle IRQs for devices that have MSI enabled and it should tread the devices with MSI-X enabled in the same way.

There are other places that need similar fixes, too, aren't there? I see cris, frv, ia64, and a driver or two testing dev->msi_enabled.

Well, I didn't look at the other places, just found this one while reviewing the code.

I'll check them later this week.

Quoted 18 lines
Quoted 18 lines
arch/x86/pci/common.c +2 -2
@@ -551,14 +551,14 @@ int pcibios_enable_device(struct pci_dev
 	if ((err = pci_enable_resources(dev, mask)) < 0)
 		return err;
 
-	if (!dev->msi_enabled)
+	if (!dev->msi_enabled && !dev->msix_enabled)
 		return pcibios_enable_irq(dev);
 	return 0;
 }
 
 void pcibios_disable_device (struct pci_dev *dev)
 {
-	if (!dev->msi_enabled && pcibios_disable_irq)
+	if (!dev->msi_enabled && !dev->msix_enabled && pcibios_disable_irq)
 		pcibios_disable_irq(dev);
 }