Reset PCIe devices to address DMA problem on kdump with iommu

PATCH Superseded Takao Indoh 4 participants 10 messages Oct 2, 2012 – Oct 10, 2012

Subsystems (4)

x86 PCI Live update kexec handover (KHO)

lore

Share & tools

lore

Takao Indoh

author

2012-10-02 07:49 UTC #0 lore raw

v2 cover Reset PCIe devices to address DMA problem on kdump with iommu

These patches reset PCIe devices at boot time to address DMA problem on kdump with iommu. When "reset_devices" is specified, a hot reset is triggered on each PCIe root port and downstream port to reset its downstream endpoint.

Background:
A kdump problem about DMA has been discussed for a long time. That is, when a kernel is switched to the kdump kernel DMA derived from first kernel affects second kernel. Recently this problem surfaces when iommu is used for PCI passthrough on KVM guest. In the case of the machine I use, when intel_iommu=on is specified, DMAR error is detected in kdump kernel and PCI SERR is also detected. Finally kdump fails because some devices does not work correctly.

The root cause is that ongoing DMA from first kernel causes DMAR fault because page table of DMAR is initialized while kdump kernel is booting up. Therefore to address this problem DMA needs to be stopped before DMAR is initialized at kdump kernel boot time. By these patches, PCIe devices are reset by hot reset and its DMA is stopped when reset_devices is specified. One problem of this solution is that the monitor blacks out when VGA controller is reset. So this patch does not reset the port whose child endpoint is VGA device.

v2:
Reset devices in setup_arch() because reset need to be done before interrupt remapping is initialized.

v1:
https://lkml.org/lkml/2012/8/3/160

Thanks,
Takao Indoh

Takao Indoh

author

2012-10-02 07:50 UTC #1 lore raw

PATCH v2 1/2 x86, pci: Reset PCIe devices at boot time

This patch resets PCIe devices at boot time by hot reset when "reset_devices" is specified.

Signed-off-by Takao Indoh <[email protected]>

---
 arch/x86/include/asm/pci-direct.h |    1 
 arch/x86/kernel/setup.c           |    3 
 arch/x86/pci/early.c              |  302 ++++++++++++++++++++++++++++
 drivers/pci/pci.c                 |   18 -
 include/linux/pci.h               |   18 +
 init/main.c                       |    4 
 6 files changed, 326 insertions(+), 20 deletions(-)

arch/x86/include/asm/pci-direct.h +1 -0

index b1e7a45..de30db2 100644
@@ -18,4 +18,5 @@ extern int early_pci_allowed(void);
 extern unsigned int pci_early_dump_regs;
 extern void early_dump_pci_device(u8 bus, u8 slot, u8 func);
 extern void early_dump_pci_devices(void);
+extern void early_reset_pcie_devices(void);
 #endif /* _ASM_X86_PCI_DIRECT_H */

arch/x86/kernel/setup.c +3 -0

index f4b9b80..24b011c 100644
@@ -988,6 +988,9 @@ void __init setup_arch(char **cmdline_p)
 	generic_apic_probe();
 
 	early_quirks();
+#ifdef CONFIG_PCI
+	early_reset_pcie_devices();
+#endif
 
 	/*
 	 * Read APIC and some other early information from ACPI tables.

arch/x86/pci/early.c +302 -0

310 lines not on this page · raw message

drivers/pci/pci.c +0 ~~-18~~

index ab4bf5a..a7a4125 100644
@@ -852,24 +852,6 @@ pci_power_t pci_choose_state(struct pci_dev *dev, pm_message_t state)
 
 EXPORT_SYMBOL(pci_choose_state);
 
-#define PCI_EXP_SAVE_REGS	7
-
-#define pcie_cap_has_devctl(type, flags)	1
-#define pcie_cap_has_lnkctl(type, flags)		\
-		((flags & PCI_EXP_FLAGS_VERS) > 1 ||	\
-		 (type == PCI_EXP_TYPE_ROOT_PORT ||	\
-		  type == PCI_EXP_TYPE_ENDPOINT ||	\
-		  type == PCI_EXP_TYPE_LEG_END))
-#define pcie_cap_has_sltctl(type, flags)		\
-		((flags & PCI_EXP_FLAGS_VERS) > 1 ||	\
-		 ((type == PCI_EXP_TYPE_ROOT_PORT) ||	\
-		  (type == PCI_EXP_TYPE_DOWNSTREAM &&	\
-		   (flags & PCI_EXP_FLAGS_SLOT))))
-#define pcie_cap_has_rtctl(type, flags)			\
-		((flags & PCI_EXP_FLAGS_VERS) > 1 ||	\
-		 (type == PCI_EXP_TYPE_ROOT_PORT ||	\
-		  type == PCI_EXP_TYPE_RC_EC))
-
 static struct pci_cap_saved_state *pci_find_saved_cap(
 	struct pci_dev *pci_dev, char cap)
 {

include/linux/pci.h +18 -0

index 5faa831..8e10401 100644
@@ -1790,5 +1790,23 @@ static inline struct eeh_dev *pci_dev_to_eeh_dev(struct pci_dev *pdev)
  */
 struct pci_dev *pci_find_upstream_pcie_bridge(struct pci_dev *pdev);
 
+#define PCI_EXP_SAVE_REGS	7
+
+#define pcie_cap_has_devctl(type, flags)	1
+#define pcie_cap_has_lnkctl(type, flags)		\
+		((flags & PCI_EXP_FLAGS_VERS) > 1 ||	\
+		 (type == PCI_EXP_TYPE_ROOT_PORT ||	\
+		  type == PCI_EXP_TYPE_ENDPOINT ||	\
+		  type == PCI_EXP_TYPE_LEG_END))
+#define pcie_cap_has_sltctl(type, flags)		\
+		((flags & PCI_EXP_FLAGS_VERS) > 1 ||	\
+		 ((type == PCI_EXP_TYPE_ROOT_PORT) ||	\
+		  (type == PCI_EXP_TYPE_DOWNSTREAM &&	\
+		   (flags & PCI_EXP_FLAGS_SLOT))))
+#define pcie_cap_has_rtctl(type, flags)			\
+		((flags & PCI_EXP_FLAGS_VERS) > 1 ||	\
+		 (type == PCI_EXP_TYPE_ROOT_PORT ||	\
+		  type == PCI_EXP_TYPE_RC_EC))
+
 #endif /* __KERNEL__ */
 #endif /* LINUX_PCI_H */

init/main.c +2 -2

index b286730..ebaf067 100644
@@ -144,10 +144,10 @@ EXPORT_SYMBOL(reset_devices);
 static int __init set_reset_devices(char *str)
 {
 	reset_devices = 1;
-	return 1;
+	return 0;
 }
 
-__setup("reset_devices", set_reset_devices);
+early_param("reset_devices", set_reset_devices);
 
 static const char * argv_init[MAX_INIT_ARGS+2] = { "init", NULL, };
 const char * envp_init[MAX_INIT_ENVS+2] = { "HOME=/", "TERM=linux", NULL, };

Takao Indoh

author

2012-10-02 07:50 UTC #2 lore raw

PATCH v2 2/2 x86, pci: Enable PCI INTx when MSI is disabled

In reply to Takao Indoh

This patch enables INTx if MSI is disabled in pcibios_enable_device(). In normal case interrupt disable bit in command register is 0b on boot time, but in case of kdump, this bit may be 1b. It causes problems of some drivers. At leaset I confirmed mptsas driver does not work in such a case. This patch fix this problem.

Signed-off-by Takao Indoh <[email protected]>

---
 arch/x86/pci/common.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

arch/x86/pci/common.c +3 -1

index 720e973..2bb7ecc 100644
@@ -615,8 +615,10 @@ int pcibios_enable_device(struct pci_dev *dev, int mask)
 	if ((err = pci_enable_resources(dev, mask)) < 0)
 		return err;
 
-	if (!pci_dev_msi_enabled(dev))
+	if (!pci_dev_msi_enabled(dev)) {
+		pci_intx(dev, true);
 		return pcibios_enable_irq(dev);
+	}
 	return 0;
 }

Andi Kleen

2012-10-02 19:37 UTC #3 lore raw

In reply to Takao Indoh

Quoted 4 lines · Takao Indoh

These patches reset PCIe devices at boot time to address DMA problem on kdump with iommu. When "reset_devices" is specified, a hot reset is triggered on each PCIe root port and downstream port to reset its downstream endpoint.

Great. I've been pondering this for a long time, but you did finally implement it. I hope this will make kdump a lot more reliable at least on the systems that support per port reset.

Now the only question is: why make it a option and not default?

-Andi

Signature · 1 line

[email protected] -- Speaking for myself only

Vivek Goyal

2012-10-02 19:44 UTC #4 lore raw

Quoted 7 lines · Andi Kleen

Quoted 4 lines · Takao Indoh
These patches reset PCIe devices at boot time to address DMA problem on kdump with iommu. When "reset_devices" is specified, a hot reset is triggered on each PCIe root port and downstream port to reset its downstream endpoint.

Quoted 5 lines

Great. I've been pondering this for a long time, but you did finally implement it. I hope this will make kdump a lot more reliable at least on the systems that support per port reset.
Now the only question is: why make it a option and not default?

Do you mean why to hide it behind "reset_devices" kernel parameter?

One thing still is concerning is how to avoid double resets. We probably will not want first bus resets and then driver loads and does another reset after seeing "reset_devices".

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

Takao Indoh

author

2012-10-03 01:02 UTC #5 lore raw

In reply to Andi Kleen

Quoted 10 lines · Andi Kleen

Quoted 4 lines · Takao Indoh
These patches reset PCIe devices at boot time to address DMA problem on kdump with iommu. When "reset_devices" is specified, a hot reset is triggered on each PCIe root port and downstream port to reset its downstream endpoint.
Great. I've been pondering this for a long time, but you did finally implement it. I hope this will make kdump a lot more reliable at least on the systems that support per port reset.

Actually I got the idea of this patch from your comment below;-)
http://permalink.gmane.org/gmane.linux.network/162414

Quoted 2 lines

Now the only question is: why make it a option and not default?

Except kdump I don't know the situation where this reset is useful, so I introduced it as option so that it works only during kdump.

Thanks,
Takao Indoh

Don Dutile

2012-10-03 13:23 UTC #6 lore raw

In reply to Takao Indoh

Quoted 30 lines · Takao Indoh

These patches reset PCIe devices at boot time to address DMA problem on kdump with iommu. When "reset_devices" is specified, a hot reset is triggered on each PCIe root port and downstream port to reset its downstream endpoint.
Background:
A kdump problem about DMA has been discussed for a long time. That is, when a kernel is switched to the kdump kernel DMA derived from first kernel affects second kernel. Recently this problem surfaces when iommu is used for PCI passthrough on KVM guest. In the case of the machine I use, when intel_iommu=on is specified, DMAR error is detected in kdump kernel and PCI SERR is also detected. Finally kdump fails because some devices does not work correctly.
The root cause is that ongoing DMA from first kernel causes DMAR fault because page table of DMAR is initialized while kdump kernel is booting up. Therefore to address this problem DMA needs to be stopped before DMAR is initialized at kdump kernel boot time. By these patches, PCIe devices are reset by hot reset and its DMA is stopped when reset_devices is specified. One problem of this solution is that the monitor blacks out when VGA controller is reset. So this patch does not reset the port whose child endpoint is VGA device.
v2:
Reset devices in setup_arch() because reset need to be done before interrupt remapping is initialized.
v1:
https://lkml.org/lkml/2012/8/3/160

Quoted 7 lines

Thanks,
Takao Indoh
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Maybe you've tried the following, and I missed a thread on it, but instead of a somewhat-large, reset hammer, did any one try just reading all the endpoint-only device CMD register, flip the MasterEnable bit off, and write it back? .... that would stop all DMA (should stop all MSI writes as well since they are just another DMA), and then restart the system?
May also have to do PCI INT Disable as well... and note, that's a PCI 2.3 optional feature.... so devices using INT signalling vs MSI
is just borked on IOMMU/intr-remapping systems... which I would expect are few. Then again, if this is foolishly done, then reset legacy PCI busses as the fallback.

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

Takao Indoh

author

2012-10-09 09:08 UTC #7 lore raw

Quoted 40 lines · Don Dutile

Quoted 30 lines · Takao Indoh
These patches reset PCIe devices at boot time to address DMA problem on kdump with iommu. When "reset_devices" is specified, a hot reset is triggered on each PCIe root port and downstream port to reset its downstream endpoint.
Background:
A kdump problem about DMA has been discussed for a long time. That is, when a kernel is switched to the kdump kernel DMA derived from first kernel affects second kernel. Recently this problem surfaces when iommu is used for PCI passthrough on KVM guest. In the case of the machine I use, when intel_iommu=on is specified, DMAR error is detected in kdump kernel and PCI SERR is also detected. Finally kdump fails because some devices does not work correctly.
The root cause is that ongoing DMA from first kernel causes DMAR fault because page table of DMAR is initialized while kdump kernel is booting up. Therefore to address this problem DMA needs to be stopped before DMAR is initialized at kdump kernel boot time. By these patches, PCIe devices are reset by hot reset and its DMA is stopped when reset_devices is specified. One problem of this solution is that the monitor blacks out when VGA controller is reset. So this patch does not reset the port whose child endpoint is VGA device.
v2:
Reset devices in setup_arch() because reset need to be done before interrupt remapping is initialized.
v1:
https://lkml.org/lkml/2012/8/3/160
Quoted 7 lines
Thanks,
Takao Indoh
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 11 lines

Maybe you've tried the following, and I missed a thread on it, but instead of a somewhat-large, reset hammer, did any one try just reading all the endpoint-only device CMD register, flip the MasterEnable bit off, and write it back? .... that would stop all DMA (should stop all MSI writes as well since they are just another DMA), and then restart the system?
May also have to do PCI INT Disable as well... and note, that's a PCI 2.3 optional feature.... so devices using INT signalling vs MSI
is just borked on IOMMU/intr-remapping systems... which I would expect are few. Then again, if this is foolishly done, then reset legacy PCI busses as the fallback.

Just clearing bus master bit and INTx disable bit in setup_arch() did not solve this problem. I still got DMAR error on devices(for exmaple, igb and megaraid_sas).

Clearing bus master in setup_arch() and resetting devices in fixup_final like v1 patch is better, DMAR error was not detected. But on certain machine kdump kernel hung up when resetting devices. It seems to be a problem specific to the platform.

And, resetting devices in setup_arch() like v2 patch solves all problems I found so far.

Thanks,
Takao Indoh

Don Dutile

2012-10-09 16:05 UTC #8 lore raw

Quoted 62 lines · Takao Indoh

Quoted 40 lines · Don Dutile
Quoted 30 lines · Takao Indoh
These patches reset PCIe devices at boot time to address DMA problem on kdump with iommu. When "reset_devices" is specified, a hot reset is triggered on each PCIe root port and downstream port to reset its downstream endpoint.
Background:
A kdump problem about DMA has been discussed for a long time. That is, when a kernel is switched to the kdump kernel DMA derived from first kernel affects second kernel. Recently this problem surfaces when iommu is used for PCI passthrough on KVM guest. In the case of the machine I use, when intel_iommu=on is specified, DMAR error is detected in kdump kernel and PCI SERR is also detected. Finally kdump fails because some devices does not work correctly.
The root cause is that ongoing DMA from first kernel causes DMAR fault because page table of DMAR is initialized while kdump kernel is booting up. Therefore to address this problem DMA needs to be stopped before DMAR is initialized at kdump kernel boot time. By these patches, PCIe devices are reset by hot reset and its DMA is stopped when reset_devices is specified. One problem of this solution is that the monitor blacks out when VGA controller is reset. So this patch does not reset the port whose child endpoint is VGA device.
v2:
Reset devices in setup_arch() because reset need to be done before interrupt remapping is initialized.
v1:
https://lkml.org/lkml/2012/8/3/160
Quoted 7 lines
Thanks,
Takao Indoh
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 11 lines
Maybe you've tried the following, and I missed a thread on it, but instead of a somewhat-large, reset hammer, did any one try just reading all the endpoint-only device CMD register, flip the MasterEnable bit off, and write it back? .... that would stop all DMA (should stop all MSI writes as well since they are just another DMA), and then restart the system?
May also have to do PCI INT Disable as well... and note, that's a PCI 2.3 optional feature.... so devices using INT signalling vs MSI
is just borked on IOMMU/intr-remapping systems... which I would expect are few. Then again, if this is foolishly done, then reset legacy PCI busses as the fallback.
Just clearing bus master bit and INTx disable bit in setup_arch() did not solve this problem. I still got DMAR error on devices(for exmaple, igb and megaraid_sas).
Clearing bus master in setup_arch() and resetting devices in fixup_final like v1 patch is better, DMAR error was not detected. But on certain machine kdump kernel hung up when resetting devices. It seems to be a problem specific to the platform.

Quoted 6 lines

And, resetting devices in setup_arch() like v2 patch solves all problems I found so far.
Thanks,
Takao Indoh

this summary should be in the patch set, so others know/learn what was attempted, what failed, and how you reached this working conclusion.

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

Takao Indoh

author

2012-10-10 04:22 UTC #9 lore raw

Quoted 63 lines · Don Dutile

Quoted 62 lines · Takao Indoh
Quoted 40 lines · Don Dutile
Quoted 30 lines · Takao Indoh
These patches reset PCIe devices at boot time to address DMA problem on kdump with iommu. When "reset_devices" is specified, a hot reset is triggered on each PCIe root port and downstream port to reset its downstream endpoint.
Background:
A kdump problem about DMA has been discussed for a long time. That is, when a kernel is switched to the kdump kernel DMA derived from first kernel affects second kernel. Recently this problem surfaces when iommu is used for PCI passthrough on KVM guest. In the case of the machine I use, when intel_iommu=on is specified, DMAR error is detected in kdump kernel and PCI SERR is also detected. Finally kdump fails because some devices does not work correctly.
The root cause is that ongoing DMA from first kernel causes DMAR fault because page table of DMAR is initialized while kdump kernel is booting up. Therefore to address this problem DMA needs to be stopped before DMAR is initialized at kdump kernel boot time. By these patches, PCIe devices are reset by hot reset and its DMA is stopped when reset_devices is specified. One problem of this solution is that the monitor blacks out when VGA controller is reset. So this patch does not reset the port whose child endpoint is VGA device.
v2:
Reset devices in setup_arch() because reset need to be done before interrupt remapping is initialized.
v1:
https://lkml.org/lkml/2012/8/3/160
Quoted 7 lines
Thanks,
Takao Indoh
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 11 lines
Maybe you've tried the following, and I missed a thread on it, but instead of a somewhat-large, reset hammer, did any one try just reading all the endpoint-only device CMD register, flip the MasterEnable bit off, and write it back? .... that would stop all DMA (should stop all MSI writes as well since they are just another DMA), and then restart the system?
May also have to do PCI INT Disable as well... and note, that's a PCI 2.3 optional feature.... so devices using INT signalling vs MSI
is just borked on IOMMU/intr-remapping systems... which I would expect are few. Then again, if this is foolishly done, then reset legacy PCI busses as the fallback.
Just clearing bus master bit and INTx disable bit in setup_arch() did not solve this problem. I still got DMAR error on devices(for exmaple, igb and megaraid_sas).
Clearing bus master in setup_arch() and resetting devices in fixup_final like v1 patch is better, DMAR error was not detected. But on certain machine kdump kernel hung up when resetting devices. It seems to be a problem specific to the platform.

Quoted 8 lines

Quoted 6 lines
And, resetting devices in setup_arch() like v2 patch solves all problems I found so far.
Thanks,
Takao Indoh
this summary should be in the patch set, so others know/learn what was attempted, what failed, and how you reached this working conclusion.

Ok, I'll post new patch with this information.

Thanks,
Takao Indoh