bnx2: Fix IRQ failures during kdump.
Subsystems (3)
PATCH 1/1 bnx2: Fix IRQ failures during kdump.
When switching from the crashed kernel to the kdump kernel without going through PCI reset, IRQs may not work if a different IRQ mode is used on the kdump kernel. The original IRQ mode used in the crashed kernel may still be enabled and the new IRQ mode may not work. For example, it will fail when going from MSI-X mode to MSI mode.
We fix this by disabling MSI/MSI-X and enabling INTX in bnx2_init_board().
pci_save_state() is also moved to the end of bnx2_init_board() after all config register fixups (including the new IRQ fixups) have been done.
Export pci_msi_off() from drivers/pci/pci.c for this purpose.
Update bnx2 version to 2.0.16.
--- drivers/net/bnx2.c | 17 ++++++++++++++--- drivers/pci/pci.c | 1 + 2 files changed, 15 insertions(+), 3 deletions(-)
drivers/net/bnx2.c +14 -3
@@ -58,8 +58,8 @@#include "bnx2_fw.h" #define DRV_MODULE_NAME "bnx2" -#define DRV_MODULE_VERSION "2.0.15" -#define DRV_MODULE_RELDATE "May 4, 2010" +#define DRV_MODULE_VERSION "2.0.16" +#define DRV_MODULE_RELDATE "May 28, 2010"#define FW_MIPS_FILE_06 "bnx2/bnx2-mips-06-5.0.0.j6.fw"#define FW_RV2P_FILE_06 "bnx2/bnx2-rv2p-06-5.0.0.j3.fw"#define FW_MIPS_FILE_09 "bnx2/bnx2-mips-09-5.0.0.j15.fw"
@@ -7877,7 +7877,6 @@ bnx2_init_board(struct pci_dev *pdev, struct net_device *dev) }pci_set_master (pdev); -pci_save_state (pdev); bp->pm_cap =pci_find_capability (pdev,PCI_CAP_ID_PM );if (bp->pm_cap ==0 ) {
@@ -7953,6 +7952,16 @@ bnx2_init_board(struct pci_dev *pdev, struct net_device *dev) bp->flags |=BNX2_FLAG_MSI_CAP ; } +/* When going from a crashed kernel to a kdump kernel without PCI +* reset, MSI/MSI-X may still be enabled. We need to disable +* MSI/MSI-X and enable INTX because the kdump driver may operate +* the device in a different IRQ mode. +*/ +if (bp->flags & (BNX2_FLAG_MSI_CAP |BNX2_FLAG_MSIX_CAP )) { +pci_msi_off (pdev); +pci_intx (pdev,1 ); + } +/* 5708 cannot support DMA addresses > 40-bit. */ if (CHIP_NUM (bp) ==CHIP_NUM_5708 ) persist_dma_mask = dma_mask =DMA_BIT_MASK (40 );
@@ -8188,6 +8197,8 @@ bnx2_init_board(struct pci_dev *pdev, struct net_device *dev) bp->timer.data = (unsigned long ) bp; bp->timer.function = bnx2_timer; +pci_save_state (pdev); +return 0 ; err_out_unmap:
drivers/pci/pci.c +1 -0
@@ -2294,6 +2294,7 @@ void pci_msi_off(struct pci_dev *dev)pci_write_config_word (dev, pos +PCI_MSIX_FLAGS , control); } } +EXPORT_SYMBOL (pci_msi_off);#ifndef HAVE_ARCH_PCI_SET_DMA_MAX_SEGMENT_SIZE int pci_set_dma_max_seg_size (struct pci_dev *dev,unsigned int size)
Signature · 1 line
1.6.4.GIT
David Miller
maintainerFrom: "Michael Chan" <[email protected]>
Date: Fri, 28 May 2010 20:24:22 -0700
Quoted 8 lines
When switching from the crashed kernel to the kdump kernel without going through PCI reset, IRQs may not work if a different IRQ mode is used on the kdump kernel. The original IRQ mode used in the crashed kernel may still be enabled and the new IRQ mode may not work. For example, it will fail when going from MSI-X mode to MSI mode.
We fix this by disabling MSI/MSI-X and enabling INTX in
bnx2_init_board().
Quoted 8 lines
pci_save_state()is also moved to the end ofbnx2_init_board()after all config register fixups (including the new IRQ fixups) have been done.Export
pci_msi_off()fromdrivers/pci/pci.cfor this purpose.Update bnx2 version to 2.0.16.
Signed-off-by Michael Chan <[email protected]>
I sincerely doubt that your's will be the only device which will ever run into this issue. Therefore handling it manually in each and every device driver, which is the trend you will be setting with this patch, doesn't make much sense.
Any device which uses MSI in any way can run into this scenerio, wherein the device will be left with MSI enabled when we leave the crash kernel and jump into the kdump kernel.
So this needs to be handled generically in the PCI layer or similar.
I'm not applying this patch.
Quoted 46 lines · Michael Chan
When switching from the crashed kernel to the kdump kernel without going through PCI reset, IRQs may not work if a different IRQ mode is used on the kdump kernel. The original IRQ mode used in the crashed kernel may still be enabled and the new IRQ mode may not work. For example, it will fail when going from MSI-X mode to MSI mode.
We fix this by disabling MSI/MSI-X and enabling INTX in
bnx2_init_board().
pci_save_state()is also moved to the end ofbnx2_init_board()after all config register fixups (including the new IRQ fixups) have been done.Export
pci_msi_off()fromdrivers/pci/pci.cfor this purpose.Update bnx2 version to 2.0.16.
Signed-off-by Michael Chan <[email protected]>--- drivers/net/bnx2.c | 17 ++++++++++++++--- drivers/pci/pci.c | 1 + 2 files changed, 15 insertions(+), 3 deletions(-)drivers/net/bnx2.c +2
-3@@ -58,8 +58,8 @@#include "bnx2_fw.h" #define DRV_MODULE_NAME "bnx2" -#define DRV_MODULE_VERSION "2.0.15" -#define DRV_MODULE_RELDATE "May 4, 2010" +#define DRV_MODULE_VERSION "2.0.16" +#define DRV_MODULE_RELDATE "May 28, 2010"#define FW_MIPS_FILE_06 "bnx2/bnx2-mips-06-5.0.0.j6.fw"#define FW_RV2P_FILE_06 "bnx2/bnx2-rv2p-06-5.0.0.j3.fw"#define FW_MIPS_FILE_09 "bnx2/bnx2-mips-09-5.0.0.j15.fw"@@ -7877,7 +7877,6 @@ bnx2_init_board(struct pci_dev *pdev, struct net_device *dev) }pci_set_master (pdev); -pci_save_state (pdev); bp->pm_cap =pci_find_capability (pdev,PCI_CAP_ID_PM );if (bp->pm_cap ==0 ) {@@ -7953,6 +7952,16 @@ bnx2_init_board(struct pci_dev *pdev, struct net_device *dev) bp->flags |=BNX2_FLAG_MSI_CAP ;
drivers/net/bnx2.c bnx2_init_board(struct pci_dev *pdev, struct net_device *dev) · line 7962
Quoted 10 lines
drivers/net/bnx2.c +8
-0} +/* When going from a crashed kernel to a kdump kernel without PCI +* reset, MSI/MSI-X may still be enabled. We need to disable +* MSI/MSI-X and enable INTX because the kdump driver may operate +* the device in a different IRQ mode. +*/ +if (bp->flags & (BNX2_FLAG_MSI_CAP |BNX2_FLAG_MSIX_CAP )) { +pci_msi_off (pdev); +pci_intx (pdev,1 );
Does the driver have to register a different Interrupt handler when switching from MSI(-X) to IRQ interrupts?
(I'm thinking of IRQ interrupts might have DMA vs IRQ ordering dependency)
If that's true, then this can't be handled in the generic PCI layer (as suggested by davem) unless the device driver could register multiple interrupt handlers even if only one is active at a time.
thanks,
grant
Quoted 34 lines
drivers/net/bnx2.c +4
-0+ } +/* 5708 cannot support DMA addresses > 40-bit. */ if (CHIP_NUM (bp) ==CHIP_NUM_5708 ) persist_dma_mask = dma_mask =DMA_BIT_MASK (40 );@@ -8188,6 +8197,8 @@ bnx2_init_board(struct pci_dev *pdev, struct net_device *dev) bp->timer.data = (unsigned long ) bp; bp->timer.function = bnx2_timer; +pci_save_state (pdev); +return 0 ; err_out_unmap:drivers/pci/pci.c +1
-0@@ -2294,6 +2294,7 @@ void pci_msi_off(struct pci_dev *dev)pci_write_config_word (dev, pos +PCI_MSIX_FLAGS , control); } } +EXPORT_SYMBOL (pci_msi_off);#ifndef HAVE_ARCH_PCI_SET_DMA_MAX_SEGMENT_SIZE int pci_set_dma_max_seg_size (struct pci_dev *dev,unsigned int size)Signature · 7 lines
1.6.4.GIT -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
David Miller
maintainerFrom: Grant Grundler <[email protected]>
Date: Sat, 29 May 2010 00:45:12 -0600
Quoted 3 lines
If that's true, then this can't be handled in the generic PCI layer (as suggested by davem) unless the device driver could register multiple interrupt handlers even if only one is active at a time.
The generic PCI layer very well can turn off MSI on all devices when it starts up or a device is plugged in.
That's all he is doing.
Drivers essentially expect that the device comes up in INTX mode when the driver probes the device. All his change is doing is forcing that to be true, and there is no reason the generic PCI code can't do that.
Quoted 11 lines · David Miller
The generic PCI layer very well can turn off MSI on all devices when it starts up or a device is plugged in.
That's all he is doing.
Drivers essentially expect that the device comes up in INTX mode when the driver probes the device. All his change is doing is forcing that to be true, and there is no reason the generic PCI code can't do that.
I think there may be more issues after thinking about it some more. The device is essentially still active at this time. The PCI layer can turn off certain things, but enabling INTX can lead to "irq x: nobody cared" if the driver is not ready for it. The device really needs to be reset by the driver to be totally reliable.
David Miller
maintainerFrom: "Michael Chan" <[email protected]>
Date: Sat, 29 May 2010 09:22:07 -0700
Quoted 6 lines
I think there may be more issues after thinking about it some more. The device is essentially still active at this time. The PCI layer can turn off certain things, but enabling INTX can lead to "irq x: nobody cared" if the driver is not ready for it. The device really needs to be reset by the driver to be totally reliable.
We still have to find some generic way to do this.
My position still stands, and it is entirely rediculious to have every single driver have to attend to all of these esoteric details just to handle interrupts properly. Drivers are hard enough to write as-is.
Quoted 10 lines · David Miller
From: "Michael Chan" <[email protected]>
Date: Sat, 29 May 2010 09:22:07 -0700Quoted 6 lines
I think there may be more issues after thinking about it some more. The device is essentially still active at this time. The PCI layer can turn off certain things, but enabling INTX can lead to "irq x: nobody cared" if the driver is not ready for it. The device really needs to be reset by the driver to be totally reliable.
Quoted 6 lines
We still have to find some generic way to do this.
My position still stands, and it is entirely rediculious to have every single driver have to attend to all of these esoteric details just to handle interrupts properly. Drivers are hard enough to write as-is.
I think we can do this generically. PCI has disable bits for MSI, MSI-X and pin-based interrupts. So we can leave MSIs enabled, but disable interrupt generation.
We should probably set the interrupt type back to pin-based before the kexec kernel starts, right? Or do we expect drivers to handle being initialised with the device still set to MSI mode?
Signature · 4 lines
Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step."
David Miller
maintainerFrom: Matthew Wilcox <[email protected]>
Date: Sat, 29 May 2010 19:24:01 -0600
Quoted 3 lines
We should probably set the interrupt type back to pin-based before the kexec kernel starts, right? Or do we expect drivers to handle being initialised with the device still set to MSI mode?
The expectation is that the device comes up in INTX mode, which is the default after a PCI reset.
Basically all of these issues tend to be about the fact that unlike on a normal boot, after a kexec an intermediate PCI reset has not occured.
Quoted 3 lines · David Miller
Basically all of these issues tend to be about the fact that unlike on a normal boot, after a kexec an intermediate PCI reset has not occured.
This could be fixed, assuming the system has AER capability ...
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 10 lines · David Miller
From: Matthew Wilcox <[email protected]>
Date: Sat, 29 May 2010 19:24:01 -0600Quoted 1 line
We should probably set the interrupt type back to pin-based before
the
Quoted 2 lines
kexec kernel starts, right? Or do we expect drivers to handle being initialised with the device still set to MSI mode?
The expectation is that the device comes up in INTX mode, which is the default after a PCI reset.
We need to be very careful because the device may still be active as I said earlier. Turning INTX on may lead to an IRQ storm that nobody will handle. Some older devices don't have the INTX enable bit, and INTX will automatically be enabled when MSI is disabled.
Quoted 3 lines
Basically all of these issues tend to be about the fact that unlike on a normal boot, after a kexec an intermediate PCI reset has not occured.
Quoted 6 lines · On Fri, 28 May 2010 20:24:22 -0700 "Michael Chan" <[email protected]> wrote:
When switching from the crashed kernel to the kdump kernel without going through PCI reset, IRQs may not work if a different IRQ mode is used on the kdump kernel. The original IRQ mode used in the crashed kernel may still be enabled and the new IRQ mode may not work. For example, it will fail when going from MSI-X mode to MSI mode.
Quoted 8 lines
We fix this by disabling MSI/MSI-X and enabling INTX in
bnx2_init_board().
pci_save_state()is also moved to the end ofbnx2_init_board()after all config register fixups (including the new IRQ fixups) have been done.Export
pci_msi_off()fromdrivers/pci/pci.cfor this purpose.Update bnx2 version to 2.0.16.
This is probably a generic problem for many drivers. So why not get the kdump code to fix it for all.
Quoted 2 lines · Michael Chan
When switching from the crashed kernel to the kdump kernel without going through PCI reset, IRQs may not work if a different IRQ mode is used on
PCIe with AER actually does support per link root port reset (e.g. used for AER)
I've been wondering for some time if kexec should not simply use that to reset all the devices, instead of addings hacks around this to all drivers.
That would fix your problems too, right?
The question is just if AER is widely enough supported for this.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 9 lines · Andi Kleen
Quoted 1 line · Michael Chan
When switching from the crashed kernel to the kdump kernel without
going
Quoted 1 line
through PCI reset, IRQs may not work if a different IRQ mode is used
on
PCIe with AER actually does support per link root port reset (e.g. used for AER)
Do you mean the slot_reset function in the pci_error_handlers? This needs to be called in the context of the crashed kernel, right?
Quoted 6 lines
I've been wondering for some time if kexec should not simply use that to reset all the devices, instead of addings hacks around this to all drivers.
That would fix your problems too, right?
If it is called in the context of the crashed kernel, it won't work. We would reset it and put in back into the same IRQ mode.
Quoted 3 lines
The question is just if AER is widely enough supported for this.
Some newer PCIe devices support Function Level Reset, and that would be ideal. But most existing devices including bnx2 devices don't have this feature.
Quoted 5 lines · Michael Chan
Quoted 3 lines · Andi Kleen
Quoted 1 line · Michael Chan
When switching from the crashed kernel to the kdump kernel without
Quoted 8 lines
Quoted 6 lines
going
Quoted 1 line
through PCI reset, IRQs may not work if a different IRQ mode is used
on
PCIe with AER actually does support per link root port reset (e.g. used for AER)
Do you mean the
slot_resetfunction in thepci_error_handlers? This
Well the fallback code in the PCIE root port driver that does the actual resets.
It could be called directly before kexec.
Quoted 1 line
needs to be called in the context of the crashed kernel, right?
It could be done on kexec, however of course you would rely on PCI root port data structures still being intact on a crash (I guess that's reasonable, they are not very complicated)
Quoted 10 lines
Quoted 7 lines
I've been wondering for some time if kexec should not simply use that to reset all the devices, instead of addings hacks around this to all drivers.
That would fix your problems too, right?
If it is called in the context of the crashed kernel, it won't work. We would reset it and put in back into the same IRQ mode.
Who would put it back? Your driver wouldn't be called anymore.
Quoted 8 lines
Quoted 4 lines
The question is just if AER is widely enough supported for this.
Some newer PCIe devices support Function Level Reset, and that would be ideal. But most existing devices including bnx2 devices don't have this feature.
Root port reset should be fine for this case. Even if some innocent device on the same root port gets reset too that shouldn't matter. Only drawback for the NIC would be that you have to renegotiate links I think.
Also there are systems without AER support.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 12 lines · Andi Kleen
Quoted 5 lines · Michael Chan
Quoted 3 lines · Andi Kleen
Quoted 1 line · Michael Chan
When switching from the crashed kernel to the kdump kernel
without
Quoted 2 lines
Quoted 2 lines
going
Quoted 1 line
through PCI reset, IRQs may not work if a different IRQ mode is
used
Quoted 2 lines
Quoted 2 lines
on
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
PCIe with AER actually does support per link root port reset (e.g. used for AER)
Do you mean the
slot_resetfunction in thepci_error_handlers? ThisWell the fallback code in the PCIE root port driver that does the actual resets.
aer_root_reset() in aerdrv.c?
Quoted 15 lines
It could be called directly before kexec.
Quoted 1 line
needs to be called in the context of the crashed kernel, right?
It could be done on kexec, however of course you would rely on PCI root port data structures still being intact on a crash (I guess that's reasonable, they are not very complicated)
Quoted 6 lines
Quoted 6 lines
I've been wondering for some time if kexec should not simply use that to reset all the devices, instead of addings hacks around this to all drivers.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
That would fix your problems too, right?
If it is called in the context of the crashed kernel, it won't work. We would reset it and put in back into the same IRQ mode.
Who would put it back? Your driver wouldn't be called anymore.
The bnx2 driver like many other drivers has a slot_reset function in the pci_driver struct's err_handler. If the AER code calls this function, we would reset the chip and put it back to the same IRQ mode. Without calling this per driver reset function, I'm not sure if you can reset the device if the device does not support Function Level Reset.
Quoted 22 lines
Quoted 8 lines
Quoted 5 lines
The question is just if AER is widely enough supported for this.
Some newer PCIe devices support Function Level Reset, and that would be ideal. But most existing devices including bnx2 devices don't
have
Quoted 1 line
this feature.
Root port reset should be fine for this case. Even if some innocent device on the same root port gets reset too that shouldn't matter.
Only drawback for the NIC would be that you have to renegotiate links I think.Also there are systems without AER support.
-Andi
--
[email protected] -- Speaking for myself only.