Skip to contentSkip to conversation

Only disable/enable LSI interrupts in EEH

PATCH In review Mike Mason 3 participants 7 messages Feb 10, 2009 – Feb 12, 2009

Subsystems (2)
lore
Share & tools
lore

Mike Mason

author
#0 lore raw

PATCH 1/1 Only disable/enable LSI interrupts in EEH

The EEH code disables and enables interrupts during the
device recovery process. This is unnecessary for MSI
and MSI-X interrupts because they are effectively disabled
by the DMA Stopped state when an EEH error occurs. The current code is also incorrect for MSI-X interrupts. It doesn't take into account that MSI-X interrupts are tracked
in a different way than LSI/MSI interrupts. This patch ensures only LSI interrupts are disabled/enabled.

The patch also includes a couple minor formatting fixes.

arch/powerpc/platforms/pseries/eeh_driver.c +0 -0
@@ -67,7 +67,7 @@ static int irq_in_use(unsigned int irq)
{
	int rc = 0;
	unsigned long flags;

- struct irq_desc *desc = irq_desc + irq;
+ struct irq_desc *desc = irq_desc + irq;

	spin_lock_irqsave(&desc->lock, flags);
	if (desc->action)
@@ -76,6 +76,40 @@ static int irq_in_use(unsigned int irq)
	return rc;
}

+/**
+ * eeh_disable_irq - disable interrupt for the recovering device
+ */
+void eeh_disable_irq(struct pci_dev *dev)
+{
+ struct device_node *dn = pci_device_to_OF_node(dev);
+
+ /* Don't disable MSI and MSI-X interrupts. They are
+ * effectively disabled by the DMA Stopped state + * when an EEH error occurs.
+ */
+ if (dev->msi_enabled || dev->msix_enabled)
+ return;
+
+ if (!irq_in_use(dev->irq))
+ return;
+
+ PCI_DN(dn)->eeh_mode |= EEH_MODE_IRQ_DISABLED;
+ disable_irq_nosync(dev->irq);
+}
+
+/**
+ * eeh_enable_irq - enable interrupt for the recovering device
+ */
+void eeh_enable_irq(struct pci_dev *dev)
+{
+ struct device_node *dn = pci_device_to_OF_node(dev);
+
+ if ((PCI_DN(dn)->eeh_mode) & EEH_MODE_IRQ_DISABLED) {
+ PCI_DN(dn)->eeh_mode &= ~EEH_MODE_IRQ_DISABLED;
+ enable_irq(dev->irq);
+ }
+}
+
/* ------------------------------------------------------- */ /**
* eeh_report_error - report pci error to each device driver
@@ -95,11 +129,8 @@ static void eeh_report_error(struct pci_

	if (!driver)
		return;

- if (irq_in_use (dev->irq)) {
- struct device_node *dn = pci_device_to_OF_node(dev);
- PCI_DN(dn)->eeh_mode |= EEH_MODE_IRQ_DISABLED;
- disable_irq_nosync(dev->irq);
- }
+ eeh_disable_irq(dev);
+

	if (!driver->err_handler ||
	    !driver->err_handler->error_detected)
		return;
@@ -144,15 +175,12 @@ static void eeh_report_reset(struct pci_
{
	enum pci_ers_result rc, *res = userdata;
	struct pci_driver *driver = dev->driver;

- struct device_node *dn = pci_device_to_OF_node(dev);

	if (!driver)
		return;

- if ((PCI_DN(dn)->eeh_mode) & EEH_MODE_IRQ_DISABLED) {
- PCI_DN(dn)->eeh_mode &= ~EEH_MODE_IRQ_DISABLED;
- enable_irq(dev->irq);
- }
+ eeh_enable_irq(dev);
+

	if (!driver->err_handler ||
	    !driver->err_handler->slot_reset)
		return;
@@ -171,17 +199,14 @@ static void eeh_report_reset(struct pci_
static void eeh_report_resume(struct pci_dev *dev, void *userdata)
{
	struct pci_driver *driver = dev->driver;

- struct device_node *dn = pci_device_to_OF_node(dev);

	dev->error_state = pci_channel_io_normal;

	if (!driver)
		return;

- if ((PCI_DN(dn)->eeh_mode) & EEH_MODE_IRQ_DISABLED) {
- PCI_DN(dn)->eeh_mode &= ~EEH_MODE_IRQ_DISABLED;
- enable_irq(dev->irq);
- }
+ eeh_enable_irq(dev);
+

	if (!driver->err_handler ||
	    !driver->err_handler->resume)
		return;
@@ -205,15 +230,12 @@ static void eeh_report_failure(struct pc
	if (!driver)
		return;

- if (irq_in_use (dev->irq)) {
- struct device_node *dn = pci_device_to_OF_node(dev);
- PCI_DN(dn)->eeh_mode |= EEH_MODE_IRQ_DISABLED;
- disable_irq_nosync(dev->irq);
- }
- if (!driver->err_handler)
- return;
- if (!driver->err_handler->error_detected)
+ eeh_disable_irq(dev);
+
+ if (!driver->err_handler ||
+ !driver->err_handler->error_detected)

		return;
+
	driver->err_handler->error_detected(dev, pci_channel_io_perm_failure);
}
Acked-by on v1 1/1
#1 lore raw

2009/2/9 Mike Mason <[email protected]>:

Quoted 9 lines

The EEH code disables and enables interrupts during the
device recovery process. This is unnecessary for MSI
and MSI-X interrupts because they are effectively disabled
by the DMA Stopped state when an EEH error occurs. The current code is also incorrect for MSI-X interrupts. It
doesn't take into account that MSI-X interrupts are tracked
in a different way than LSI/MSI interrupts. This patch ensures only LSI interrupts are disabled/enabled.

Quoted 4 lines

The patch also includes a couple minor formatting fixes.

Looks good to me.

On a somewhat-related note: there was an issue (I forget the details) where the kernel needed to shadow some sort of MSI state so that it could be correctly, um, kept-track-of, after an EEH reset (it didn't need to be restored, because firmware did this(?)). After some digging around and discussion, we concluded that some generic PPC MSI code needed to be altered to track this state, and/or the main kernel MSI code needed to be changed to (not?) track this state. Mike Ellerman seemed to best grasp this area ... was this ever fixed?

Or perhaps this is an alternate fix for that bug? It may well have been that calling the MSI disable triggered the problem, I don't remember now.

--linas

#2 lore raw
Quoted 29 lines · Linas Vepstas

2009/2/9 Mike Mason <[email protected]>:

Quoted 9 lines

The EEH code disables and enables interrupts during the
device recovery process. This is unnecessary for MSI
and MSI-X interrupts because they are effectively disabled
by the DMA Stopped state when an EEH error occurs. The current code is also incorrect for MSI-X interrupts. It
doesn't take into account that MSI-X interrupts are tracked
in a different way than LSI/MSI interrupts. This patch ensures only LSI interrupts are disabled/enabled.

Quoted 4 lines

The patch also includes a couple minor formatting fixes.

Looks good to me.

On a somewhat-related note: there was an issue (I forget the details) where the kernel needed to shadow some sort of MSI state so that it could be correctly, um, kept-track-of, after an EEH reset (it didn't need to be restored, because firmware did this(?)). After some digging around and discussion, we concluded that some generic PPC MSI code needed to be altered to track this state, and/or the main kernel MSI code needed to be changed to (not?) track this state. Mike Ellerman seemed to best grasp this area ... was this ever fixed?

Quoted 3 lines

Or perhaps this is an alternate fix for that bug? It may well have been that calling the MSI disable triggered the problem, I don't remember now.

I'm pretty sure you're referring to this patch, which you acked :)

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=1db3e890aed3ac39cded30d6e94618bda086f7ce

I don't know of anything else that fits your description?

cheers

Signature · 8 lines
Michael Ellerman
OzLabs, IBM Australia Development Lab

wwweb: http://michael.ellerman.id.au
phone: +61 2 6212 1183 (tie line 70 21183)

We do not inherit the earth from our ancestors,
we borrow it from our children. - S.M.A.R.T Person
#3 lore raw

2009/2/10 Michael Ellerman <[email protected]>:

Quoted 16 lines · Linas Vepstas
Quoted 11 lines

On a somewhat-related note: there was an issue (I forget the details) where the kernel needed to shadow some sort of MSI state so that it could be correctly, um, kept-track-of, after an EEH reset (it didn't need to be restored, because firmware did this(?)). After some digging around and discussion, we concluded that some generic PPC MSI code needed to be altered to track this state, and/or the main kernel MSI code needed to be changed to (not?) track this state. Mike Ellerman seemed to best grasp this area ... was this ever fixed?

Quoted 3 lines

Or perhaps this is an alternate fix for that bug? It may well have been that calling the MSI disable triggered the problem, I don't remember now.

Quoted 5 lines

I'm pretty sure you're referring to this patch, which you acked :)

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=1db3e890aed3ac39cded30d6e94618bda086f7ce

I don't know of anything else that fits your description?

Yes, that's the one. I wasn't sure if it ever made it in or not, and I just wanted to make sure it wasn't what was biting you.

--linas

author Acked-by Linas Vepstas on v1 1/1
#4 lore raw
In reply to Mike Mason

I'm resubmitting this patch with a couple changes suggested by Michael Ellerman. 1) the new functions should be static, and 2) some people may object to including unrelated formating changes.

=========================================================

The EEH code disables and enables interrupts during the
device recovery process. This is unnecessary for MSI
and MSI-X interrupts because they are effectively disabled
by the DMA Stopped state when an EEH error occurs. The
current code is also incorrect for MSI-X interrupts. It
doesn't take into account that MSI-X interrupts are tracked
in a different way than LSI/MSI interrupts. This patch ensures only LSI interrupts are disabled/enabled.

powerpc/platforms/pseries/eeh_driver.c renamed from powerpc/platforms/pseries/eeh_driver.c-orig +0 -0
@@ -76,6 +76,40 @@ static int irq_in_use(unsigned int irq)
	return rc;
}

+/**
+ * eeh_disable_irq - disable interrupt for the recovering device
+ */
+static void eeh_disable_irq(struct pci_dev *dev)
+{
+ struct device_node *dn = pci_device_to_OF_node(dev);
+
+ /* Don't disable MSI and MSI-X interrupts. They are
+ * effectively disabled by the DMA Stopped state + * when an EEH error occurs.
+ */
+ if (dev->msi_enabled || dev->msix_enabled)
+ return;
+
+ if (!irq_in_use(dev->irq))
+ return;
+
+ PCI_DN(dn)->eeh_mode |= EEH_MODE_IRQ_DISABLED;
+ disable_irq_nosync(dev->irq);
+}
+
+/**
+ * eeh_enable_irq - enable interrupt for the recovering device
+ */
+static void eeh_enable_irq(struct pci_dev *dev)
+{
+ struct device_node *dn = pci_device_to_OF_node(dev);
+
+ if ((PCI_DN(dn)->eeh_mode) & EEH_MODE_IRQ_DISABLED) {
+ PCI_DN(dn)->eeh_mode &= ~EEH_MODE_IRQ_DISABLED;
+ enable_irq(dev->irq);
+ }
+}
+
/* ------------------------------------------------------- */ /**
* eeh_report_error - report pci error to each device driver
@@ -95,11 +129,8 @@ static void eeh_report_error(struct pci_

	if (!driver)
		return;

- if (irq_in_use (dev->irq)) {
- struct device_node *dn = pci_device_to_OF_node(dev);
- PCI_DN(dn)->eeh_mode |= EEH_MODE_IRQ_DISABLED;
- disable_irq_nosync(dev->irq);
- }
+ eeh_disable_irq(dev);
+

	if (!driver->err_handler ||
	    !driver->err_handler->error_detected)
		return;
@@ -144,15 +175,12 @@ static void eeh_report_reset(struct pci_
{
	enum pci_ers_result rc, *res = userdata;
	struct pci_driver *driver = dev->driver;

- struct device_node *dn = pci_device_to_OF_node(dev);

	if (!driver)
		return;

- if ((PCI_DN(dn)->eeh_mode) & EEH_MODE_IRQ_DISABLED) {
- PCI_DN(dn)->eeh_mode &= ~EEH_MODE_IRQ_DISABLED;
- enable_irq(dev->irq);
- }
+ eeh_enable_irq(dev);
+

	if (!driver->err_handler ||
	    !driver->err_handler->slot_reset)
		return;
@@ -171,17 +199,14 @@ static void eeh_report_reset(struct pci_
static void eeh_report_resume(struct pci_dev *dev, void *userdata)
{
	struct pci_driver *driver = dev->driver;

- struct device_node *dn = pci_device_to_OF_node(dev);

	dev->error_state = pci_channel_io_normal;

	if (!driver)
		return;

- if ((PCI_DN(dn)->eeh_mode) & EEH_MODE_IRQ_DISABLED) {
- PCI_DN(dn)->eeh_mode &= ~EEH_MODE_IRQ_DISABLED;
- enable_irq(dev->irq);
- }
+ eeh_enable_irq(dev);
+

	if (!driver->err_handler ||
	    !driver->err_handler->resume)
		return;
@@ -205,15 +230,12 @@ static void eeh_report_failure(struct pc
	if (!driver)
		return;

- if (irq_in_use (dev->irq)) {
- struct device_node *dn = pci_device_to_OF_node(dev);
- PCI_DN(dn)->eeh_mode |= EEH_MODE_IRQ_DISABLED;
- disable_irq_nosync(dev->irq);
- }
- if (!driver->err_handler)
- return;
- if (!driver->err_handler->error_detected)
+ eeh_disable_irq(dev);
+
+ if (!driver->err_handler ||
+ !driver->err_handler->error_detected)

		return;
+
	driver->err_handler->error_detected(dev, pci_channel_io_perm_failure);
}
#5 lore raw
Quoted 16 lines · Mike Mason

I'm resubmitting this patch with a couple changes suggested by Michael Ellerman. 1) the new functions should be static, and 2) some people may object to including unrelated formating changes.

=========================================================

The EEH code disables and enables interrupts during the device recovery process. This is unnecessary for MSI and MSI-X interrupts because they are effectively disabled by the DMA Stopped state when an EEH error occurs. The current code is also incorrect for MSI-X interrupts. It doesn't take into account that MSI-X interrupts are tracked in a different way than LSI/MSI interrupts. This patch ensures only LSI interrupts are disabled/enabled.

Quoted 2 lines

Looks good. Assuming you've tested it :)

Signature · 8 lines
Michael Ellerman
OzLabs, IBM Australia Development Lab

wwweb: http://michael.ellerman.id.au
phone: +61 2 6212 1183 (tie line 70 21183)

We do not inherit the earth from our ancestors,
we borrow it from our children. - S.M.A.R.T Person
author
#6 lore raw
Quoted 16 lines · Michael Ellerman
Quoted 15 lines · Mike Mason

I'm resubmitting this patch with a couple changes suggested by Michael Ellerman. 1) the new functions should be static, and 2) some people may object to including unrelated formating changes.

=========================================================

The EEH code disables and enables interrupts during the
device recovery process. This is unnecessary for MSI
and MSI-X interrupts because they are effectively disabled
by the DMA Stopped state when an EEH error occurs. The
current code is also incorrect for MSI-X interrupts. It
doesn't take into account that MSI-X interrupts are tracked
in a different way than LSI/MSI interrupts. This patch ensures only LSI interrupts are disabled/enabled.

Quoted 5 lines
Quoted 2 lines

Looks good. Assuming you've tested it :)

Yes, it's been tested with network devices that use LSI, MSI and MSI-X interrupts. All recovered fine.

Quoted 4 lines