Skip to contentSkip to conversation

Fix DPC hotplug race and enhance error hanlding

PATCH Superseded Ethan Zhao 4 participants 16 messages Sep 25, 2020 – Sep 30, 2020 3 messages not in the archive

Subsystems (2)

v1 of 8 Superseded Latest is v8 →

lore
Share & tools
lore

Ethan Zhao

author
#0 lore raw

cover Fix DPC hotplug race and enhance error hanlding

This simple patch set fixed some serious security issues found when DPC error injection and NVMe SSD hotplug brute force test were doing -- race condition between DPC handler and pciehp, AER interrupt handlers, caused system hang and system with DPC feature couldn't recover to normal working state as expected (NVMe instance lost, mount operation hang, race PCIe access caused uncorrectable errors reported alternativly etc).

With this patch set applied, stable 5.9-rc6 could pass the PCIe Gen4 NVMe SSD brute force hotplug test with any time interval between hot-remove and plug-in operation tens of times without any errors occur and system works normal.

With this patch set applied, system with DPC feature could recover from NON-FATAL and FATAL errors injection test and works as expected.

System works smoothly when errors happen while hotplug is doing, no uncorrectable errors found.

Brute DPC error injection script:

for i in {0..100}
do

        setpci -s 64:02.0 0x196.w=000a
        setpci -s 65:00.0 0x04.w=0544
        mount /dev/nvme0n1p1 /root/nvme
        sleep 1

done

Other details see every commits description part.

This patch set could be applied to stable 5.9-rc6 directly.

Help to review and test.

Thanks,
Ethan

Ethan Zhao (5):
PCI: define a function to check and wait till port finish DPC handling
PCI: pciehp: check and wait port status out of DPC before handling
DLLSC and PDC
PCI/ERR: get device before call device driver to avoid null pointer
reference
PCI: only return true when dev io state is really changed PCI/ERR: don't mix io state not changed and no driver together

 drivers/pci/hotplug/pciehp_hpc.c |  4 +++-
 drivers/pci/pci.h                | 31 +++----------------------------
 drivers/pci/pcie/err.c           | 18 ++++++++++++++++--
 include/linux/pci.h              | 31 +++++++++++++++++++++++++++++++
 4 files changed, 53 insertions(+), 31 deletions(-)
Signature · 1 line
2.18.4
author
#1 lore raw

PATCH 4/5 PCI: only return true when dev io state is really changed

When uncorrectable error happens, AER driver and DPC driver interrupt handlers likely call
pcie_do_recovery()->pci_walk_bus()->report_frozen_detected() with
pci_channel_io_frozen the same time.
If pci_dev_set_io_state() return true even if the original state is
pci_channel_io_frozen, that will cause AER or DPC handler re-enter the error detecting and recovery procedure one after another.
The result is the recovery flow mixed between AER and DPC.
So simplify the pci_dev_set_io_state() function to only return true when dev->error_state is changed.

---
 drivers/pci/pci.h | 31 +++----------------------------
 1 file changed, 3 insertions(+), 28 deletions(-)
drivers/pci/pci.h +3 -28
index fa12f7cbc1a0..d420bb977f3b 100644
@@ -362,35 +362,10 @@ static inline bool pci_dev_set_io_state(struct pci_dev *dev,
 	bool changed = false;
 
 	device_lock_assert(&dev->dev);
-	switch (new) {
-	case pci_channel_io_perm_failure:
-		switch (dev->error_state) {
-		case pci_channel_io_frozen:
-		case pci_channel_io_normal:
-		case pci_channel_io_perm_failure:
-			changed = true;
-			break;
-		}
-		break;
-	case pci_channel_io_frozen:
-		switch (dev->error_state) {
-		case pci_channel_io_frozen:
-		case pci_channel_io_normal:
-			changed = true;
-			break;
-		}
-		break;
-	case pci_channel_io_normal:
-		switch (dev->error_state) {
-		case pci_channel_io_frozen:
-		case pci_channel_io_normal:
-			changed = true;
-			break;
-		}
-		break;
-	}
-	if (changed)
+	if (dev->error_state != new) {
 		dev->error_state = new;
+		changed = true;
+	}
 	return changed;
 }
 
Signature · 1 line
2.18.4
Reviewed-by on v1 4/5
#2 lore raw

Re: [PATCH 4/5] PCI: only return true when dev io state is really changed

Hi Ethan,

Quoted 50 lines · Ethan Zhao

When uncorrectable error happens, AER driver and DPC driver interrupt handlers likely call

    pcie_do_recovery()->pci_walk_bus()->report_frozen_detected() with

pci_channel_io_frozen the same time.
If pci_dev_set_io_state() return true even if the original state is
pci_channel_io_frozen, that will cause AER or DPC handler re-enter the error detecting and recovery procedure one after another.
The result is the recovery flow mixed between AER and DPC.
So simplify the pci_dev_set_io_state() function to only return true when dev->error_state is changed.

---
  drivers/pci/pci.h | 31 +++----------------------------
  1 file changed, 3 insertions(+), 28 deletions(-)
drivers/pci/pci.h +0 -24
index fa12f7cbc1a0..d420bb977f3b 100644
@@ -362,35 +362,10 @@ static inline bool pci_dev_set_io_state(struct pci_dev *dev,
  	bool changed = false;
 
  	device_lock_assert(&dev->dev);
-	switch (new) {
-	case pci_channel_io_perm_failure:
-		switch (dev->error_state) {
-		case pci_channel_io_frozen:
-		case pci_channel_io_normal:
-		case pci_channel_io_perm_failure:
-			changed = true;
-			break;
-		}
-		break;
-	case pci_channel_io_frozen:
-		switch (dev->error_state) {
-		case pci_channel_io_frozen:
-		case pci_channel_io_normal:
-			changed = true;
-			break;
-		}
-		break;
-	case pci_channel_io_normal:
-		switch (dev->error_state) {
-		case pci_channel_io_frozen:
-		case pci_channel_io_normal:
-			changed = true;
-			break;

drivers/pci/pci.h static inline bool pci_dev_set_io_state(struct pci_dev *dev, · line 370

Quoted 10 lines
drivers/pci/pci.h +3 -4
-		}
-		break;
-	}
-	if (changed)
+	if (dev->error_state != new) {
  		dev->error_state = new;
+		changed = true;
+	}
  	return changed;
  }

The flow is a lot easier to follow now. Thank you.

#3 lore raw

Re: [PATCH 4/5] PCI: only return true when dev io state is really changed

In reply to Ethan Zhao
Quoted 4 lines · Ethan Zhao

When uncorrectable error happens, AER driver and DPC driver interrupt handlers likely call
pcie_do_recovery()->pci_walk_bus()->report_frozen_detected() with
pci_channel_io_frozen the same time.

Call chains are better to read if they split like

foo() ->

     bar() ->
       baz()
Quoted 6 lines

If pci_dev_set_io_state() return true even if the original state is
pci_channel_io_frozen, that will cause AER or DPC handler re-enter the error detecting and recovery procedure one after another.
The result is the recovery flow mixed between AER and DPC.
So simplify the pci_dev_set_io_state() function to only return true when dev->error_state is changed.

...

drivers/pci/pci.h static inline bool pci_dev_set_io_state(struct pci_dev *dev, · line 369

Quoted 5 lines
drivers/pci/pci.h +3 -0
+	if (dev->error_state != new) {
 		dev->error_state = new;
+		changed = true;
+	}
 	return changed;

Perhaps

	if (dev->error_state == new)
		return changed;

	dev->error_state = new;
	return true;

?

Signature · 2 lines
With Best Regards,
Andy Shevchenko
#4 lore raw

RE: [PATCH 4/5] PCI: only return true when dev io state is really changed

Yes, better !

-----Original Message-----
From: Andy Shevchenko <[email protected]>
Sent: Friday, September 25, 2020 8:38 PM
To: Zhao, Haifeng <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Jia, Pei P <[email protected]>
Subject: Re: [PATCH 4/5] PCI: only return true when dev io state is really changed

Quoted 4 lines · Ethan Zhao

When uncorrectable error happens, AER driver and DPC driver interrupt handlers likely call
pcie_do_recovery()->pci_walk_bus()->report_frozen_detected() with
pci_channel_io_frozen the same time.

Call chains are better to read if they split like

foo() ->

     bar() ->
       baz()
Quoted 6 lines

If pci_dev_set_io_state() return true even if the original state is
pci_channel_io_frozen, that will cause AER or DPC handler re-enter the error detecting and recovery procedure one after another.
The result is the recovery flow mixed between AER and DPC.
So simplify the pci_dev_set_io_state() function to only return true when dev->error_state is changed.

...

drivers/pci/pci.h static inline bool pci_dev_set_io_state(struct pci_dev *dev, · line 369

Quoted 5 lines
drivers/pci/pci.h +3 -0
+	if (dev->error_state != new) {
 		dev->error_state = new;
+		changed = true;
+	}
 	return changed;

Perhaps

	if (dev->error_state == new)
		return changed;

	dev->error_state = new;
	return true;

?

--
With Best Regards,
Andy Shevchenko

author
#5 lore raw

PATCH 5/5 PCI/ERR: don't mix io state not changed and no driver together

In reply to Ethan Zhao

When we see 'can't recover (no error_detected callback)' on console, Maybe the reason is io state is not changed by calling pci_dev_set_io_state(), that is confused. fix it.

---
 drivers/pci/pcie/err.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/pci/pcie/err.c +4 -2
index e35c4480c86b..d85f27c90c26 100644
@@ -55,8 +55,10 @@ static int report_error_detected(struct pci_dev *dev,
 	if (!pci_dev_get(dev))
 		return 0;
 	device_lock(&dev->dev);
-	if (!pci_dev_set_io_state(dev, state) ||
-		!dev->driver ||
+	if (!pci_dev_set_io_state(dev, state)) {
+		pci_dbg(dev, "Device might already being in error handling ...\n");
+		vote = PCI_ERS_RESULT_NONE;
+	} else if (!dev->driver ||
 		!dev->driver->err_handler ||
 		!dev->driver->err_handler->error_detected) {
 		/*
Signature · 1 line
2.18.4

A message that is not in the archive (sent off-list or not yet received): [email protected]

#7 lore raw

Re: [PATCH 1/5] PCI: define a function to check and wait till port finish DPC handling

Quoted 7 lines · Ethan Zhao

Once root port DPC capability is enabled and triggered, at the beginning of DPC is triggered, the DPC status bits are set by hardware and then sends DPC/DLLSC/PDC interrupts to OS DPC and pciehp drivers, it will take the port and software DPC interrupt handler 10ms to 50ms (test data on ICX platform & stable 5.9-rc6) to complete the DPC containment procedure till the DPC status is cleared at the end of the DPC interrupt handler.

Quoted 8 lines

We use this function to check if the root port is in DPC handling status and wait till the hardware and software completed the procedure.

#include <linux/interrupt.h>
#include <linux/io.h>
#include <linux/resource_ext.h>

+#include <linux/delay.h>

Keep it sorted?

Quoted 1 line

#include <uapi/linux/pci.h>

...

Review

Quoted 9 lines
(quoted diff) +8 -0
+#ifdef CONFIG_PCIE_DPC
+static inline bool pci_wait_port_outdpc(struct pci_dev *pdev)
+{
+	u16 cap = pdev->dpc_cap, status;
+	u16 loop = 0;
+
+	if (!cap) {
 
+		pci_WARN_ONCE(pdev, !cap, "No DPC capability initiated\n");

But why? Is this feature mandatory to have? Then the same question about ifdeffery, otherwise it's pretty normal to not have a feature, right?

Review

Quoted 10 lines
(quoted diff) +9 -0
+		return false;
+	}
+	pci_read_config_word(pdev, cap + PCI_EXP_DPC_STATUS, &status);
+	pci_dbg(pdev, "DPC status %x, cap %x\n", status, cap);
 
+	while (status & PCI_EXP_DPC_STATUS_TRIGGER && loop < 100) {
+		msleep(10);
+		loop++;
+		pci_read_config_word(pdev, cap + PCI_EXP_DPC_STATUS, &status);
+	}

Can we have rather something like readx_poll_timeout() for PCI and use them here?

Quoted 17 lines
(quoted diff) +13 -0
+	if (!(status & PCI_EXP_DPC_STATUS_TRIGGER)) {
+		pci_dbg(pdev, "Out of DPC status %x, time cost %d ms\n", status, loop*10);
+		return true;
+	}
+	pci_dbg(pdev, "Timeout to wait port out of DPC status\n");
+	return false;
+}
+#else
+static inline bool pci_wait_port_outdpc(struct pci_dev *pdev)
+{
+	return true;
+}
+#endif
 #endif /* LINUX_PCI_H */
Signature · 1 line
2.18.4
Signature · 2 lines
With Best Regards,
Andy Shevchenko
#8 lore raw

RE: [PATCH 1/5] PCI: define a function to check and wait till port finish DPC handling

Andy,
About the header file, yes, to keep the order.
The function was already defined with #ifdef CONFIG_PCIE_DPC.
As to ' readx_poll_timeout()' if there is generic one, I would like to use it. Seems there is no yet ?

Thanks,
Ethan

-----Original Message-----
From: Andy Shevchenko <[email protected]>
Sent: Friday, September 25, 2020 8:25 PM
To: Zhao, Haifeng <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Jia, Pei P <[email protected]>
Subject: Re: [PATCH 1/5] PCI: define a function to check and wait till port finish DPC handling

Quoted 7 lines · Ethan Zhao

Once root port DPC capability is enabled and triggered, at the
beginning of DPC is triggered, the DPC status bits are set by hardware
and then sends DPC/DLLSC/PDC interrupts to OS DPC and pciehp drivers,
it will take the port and software DPC interrupt handler 10ms to 50ms
(test data on ICX platform & stable 5.9-rc6) to complete the DPC
containment procedure till the DPC status is cleared at the end of the DPC interrupt handler.

Quoted 8 lines

We use this function to check if the root port is in DPC handling status and wait till the hardware and software completed the procedure.

#include <linux/interrupt.h>
#include <linux/io.h>
#include <linux/resource_ext.h>

+#include <linux/delay.h>

Keep it sorted?

Quoted 1 line

#include <uapi/linux/pci.h>

...

Review

Quoted 8 lines
(quoted diff) +7 -0
+#ifdef CONFIG_PCIE_DPC
+static inline bool pci_wait_port_outdpc(struct pci_dev *pdev) {
+	u16 cap = pdev->dpc_cap, status;
+	u16 loop = 0;
+
+	if (!cap) {
 
+		pci_WARN_ONCE(pdev, !cap, "No DPC capability initiated\n");

But why? Is this feature mandatory to have? Then the same question about ifdeffery, otherwise it's pretty normal to not have a feature, right?

Review

Quoted 10 lines
(quoted diff) +9 -0
+		return false;
+	}
+	pci_read_config_word(pdev, cap + PCI_EXP_DPC_STATUS, &status);
+	pci_dbg(pdev, "DPC status %x, cap %x\n", status, cap);
 
+	while (status & PCI_EXP_DPC_STATUS_TRIGGER && loop < 100) {
+		msleep(10);
+		loop++;
+		pci_read_config_word(pdev, cap + PCI_EXP_DPC_STATUS, &status);
+	}

Can we have rather something like readx_poll_timeout() for PCI and use them here?

Quoted 16 lines
(quoted diff) +12 -1
+	if (!(status & PCI_EXP_DPC_STATUS_TRIGGER)) {
+		pci_dbg(pdev, "Out of DPC status %x, time cost %d ms\n", status, loop*10);
+		return true;
+	}
+	pci_dbg(pdev, "Timeout to wait port out of DPC status\n");
+	return false;
+}
+#else
+static inline bool pci_wait_port_outdpc(struct pci_dev *pdev) {
+	return true;
+}
+#endif
 #endif /* LINUX_PCI_H */
--

2.18.4

Signature · 2 lines
With Best Regards,
Andy Shevchenko

A message that is not in the archive (sent off-list or not yet received): [email protected]

#10 lore raw

Re: [PATCH 3/5] PCI/ERR: get device before call device driver to avoid null pointer reference

Quoted 2 lines · Ethan Zhao

During DPC error injection test we found there is race condition between pciehp and DPC driver, null pointer reference caused panic as following

null -> NULL

Quoted 7 lines
 # setpci -s 64:02.0 0x196.w=000a
  // 64:02.0 is rootport has DPC capability
 # setpci -s 65:00.0 0x04.w=0544
  // 65:00.0 is NVMe SSD populated in above port
 # mount /dev/nvme0n1p1 nvme
Quoted 7 lines

(tested on stable 5.8 & ICX platform)

Buffer I/O error on dev nvme0n1p1, logical block 468843328, async page read

 BUG: kernel NULL pointer dereference, address: 0000000000000050
 #PF: supervisor read access in kernel mode
 #PF: error_code(0x0000) - not-present page

Same comment about Oops.

Signature · 2 lines
With Best Regards,
Andy Shevchenko
#11 lore raw

Re: [PATCH 3/5] PCI/ERR: get device before call device driver to avoid null pointer reference

Preferred style, there will be cleared comment in v6.

Thanks,
Ethan

Quoted 29 lines · Andy Shevchenko
Quoted 2 lines · Ethan Zhao

During DPC error injection test we found there is race condition between pciehp and DPC driver, null pointer reference caused panic as following

null -> NULL

Quoted 7 lines
 # setpci -s 64:02.0 0x196.w=000a
  // 64:02.0 is rootport has DPC capability
 # setpci -s 65:00.0 0x04.w=0544
  // 65:00.0 is NVMe SSD populated in above port
 # mount /dev/nvme0n1p1 nvme
Quoted 7 lines

(tested on stable 5.8 & ICX platform)

Buffer I/O error on dev nvme0n1p1, logical block 468843328, async page read

 BUG: kernel NULL pointer dereference, address: 0000000000000050
 #PF: supervisor read access in kernel mode
 #PF: error_code(0x0000) - not-present page

Same comment about Oops.

--
With Best Regards,
Andy Shevchenko

#12 lore raw

Re: [PATCH 3/5] PCI/ERR: get device before call device driver to avoid null pointer reference

Quoted 1 line · Ethan Zhao

Preferred style, there will be cleared comment in v6.

Avoid top postings.

Quoted 18 lines
Quoted 16 lines · Andy Shevchenko
Quoted 2 lines · Ethan Zhao

During DPC error injection test we found there is race condition between pciehp and DPC driver, null pointer reference caused panic as following

null -> NULL

Quoted 9 lines
 # setpci -s 64:02.0 0x196.w=000a
  // 64:02.0 is rootport has DPC capability
 # setpci -s 65:00.0 0x04.w=0544
  // 65:00.0 is NVMe SSD populated in above port
 # mount /dev/nvme0n1p1 nvme

(tested on stable 5.8 & ICX platform)

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

Buffer I/O error on dev nvme0n1p1, logical block 468843328, async page read

 BUG: kernel NULL pointer dereference, address: 0000000000000050
 #PF: supervisor read access in kernel mode
 #PF: error_code(0x0000) - not-present page

Same comment about Oops.

In another thread it was a good advice to move the full Oops (if you think it's very useful to have) after the cutter '---' line, so it will be in email archives but Git history.

Signature · 2 lines
With Best Regards,
Andy Shevchenko
#13 lore raw

Re: [PATCH 3/5] PCI/ERR: get device before call device driver to avoid null pointer reference

Andy,

Quoted 30 lines · Andy Shevchenko
Quoted 1 line · Ethan Zhao

Preferred style, there will be cleared comment in v6.

Avoid top postings.

Quoted 24 lines
Quoted 22 lines · Andy Shevchenko
Quoted 2 lines · Ethan Zhao

During DPC error injection test we found there is race condition between pciehp and DPC driver, null pointer reference caused panic as following

null -> NULL

Quoted 14 lines
 # setpci -s 64:02.0 0x196.w=000a
  // 64:02.0 is rootport has DPC capability
 # setpci -s 65:00.0 0x04.w=0544
  // 65:00.0 is NVMe SSD populated in above port
 # mount /dev/nvme0n1p1 nvme

(tested on stable 5.8 & ICX platform)

Buffer I/O error on dev nvme0n1p1, logical block 468843328, async page read

 BUG: kernel NULL pointer dereference, address: 0000000000000050
 #PF: supervisor read access in kernel mode
 #PF: error_code(0x0000) - not-present page
Quoted 5 lines
Quoted 1 line
Quoted 1 line

Same comment about Oops.

In another thread it was a good advice to move the full Oops (if you think it's very useful to have) after the cutter '---' line, so it will be in email archives but Git history.

So git history wouldn't give any of the Oops context, and he/she has to access LKML,
if offline, then ...lost.

Thanks,
Ethan

Quoted 6 lines

--
With Best Regards,
Andy Shevchenko

#14 lore raw

Re: [PATCH 3/5] PCI/ERR: get device before call device driver to avoid null pointer reference

Quoted 10 lines · Ethan Zhao
Quoted 8 lines · Andy Shevchenko
Quoted 1 line · Ethan Zhao

Preferred style, there will be cleared comment in v6.

Avoid top postings.

Quoted 3 lines
Quoted 1 line · Andy Shevchenko

On Thu, Sep 24, 2020 at 10:34:21PM -0400, Ethan Zhao wrote:

...

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

Buffer I/O error on dev nvme0n1p1, logical block 468843328, async page read

 BUG: kernel NULL pointer dereference, address: 0000000000000050
 #PF: supervisor read access in kernel mode
 #PF: error_code(0x0000) - not-present page

Same comment about Oops.

Quoted 7 lines
Quoted 3 lines

In another thread it was a good advice to move the full Oops (if you think it's very useful to have) after the cutter '---' line, so it will be in email archives but Git history.

So git history wouldn't give any of the Oops context, and he/she has to access LKML,
if offline, then ...lost.

Tell me, do you really think the line:
#PF: error_code(0x0000) - not-present page
makes any sense in the commit message?

I do not think so. And so on, for almost 60% of the Oops. Really, to help one person you will make millions suffering. It's not okay.

Signature · 2 lines
With Best Regards,
Andy Shevchenko
#15 lore raw

Re: [PATCH 3/5] PCI/ERR: get device before call device driver to avoid null pointer reference

Quoted 31 lines · Andy Shevchenko
Quoted 10 lines · Ethan Zhao
Quoted 8 lines · Andy Shevchenko
Quoted 1 line · Ethan Zhao

Preferred style, there will be cleared comment in v6.

Avoid top postings.

Quoted 3 lines
Quoted 1 line · Andy Shevchenko

On Thu, Sep 24, 2020 at 10:34:21PM -0400, Ethan Zhao wrote:

...

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

Buffer I/O error on dev nvme0n1p1, logical block 468843328, async page read

 BUG: kernel NULL pointer dereference, address: 0000000000000050
 #PF: supervisor read access in kernel mode
 #PF: error_code(0x0000) - not-present page

Same comment about Oops.

Quoted 7 lines
Quoted 3 lines

In another thread it was a good advice to move the full Oops (if you think it's very useful to have) after the cutter '---' line, so it will be in email archives but Git history.

So git history wouldn't give any of the Oops context, and he/she has to access LKML,
if offline, then ...lost.

Quoted 7 lines

Tell me, do you really think the line:

   #PF: error_code(0x0000) - not-present page

makes any sense in the commit message?

I do not think so. And so on, for almost 60% of the Oops. Really, to help one person you will make millions suffering. It's not okay.

If you and millions feel so suffered, why not try to simplify the Oops code to not
output nonsense too much to console and log. :)

They might not be so important to old birds as you. but it is easy to cut off the road for newcomers.

Anyway, it is not the focus of this patchset. help to take a look and try the code.

Thanks,
Ethan

Quoted 5 lines

--
With Best Regards,
Andy Shevchenko

A message that is not in the archive (sent off-list or not yet received): [email protected]

#17 lore raw

Re: [PATCH 2/5] PCI: pciehp: check and wait port status out of DPC before handling DLLSC and PDC

Quoted 17 lines · Ethan Zhao

When root port has DPC capability and it is enabled, then triggered by errors, DPC DLLSC and PDC interrupts will be sent to DPC driver, pciehp driver at the same time. That will cause following result:

1. Link and device are recovered by hardware DPC and software DPC driver, device isn't removed, but the pciehp might treat it as devce was hot removed.

2. Race condition happens between pciehp_unconfigure_device() called by pciehp_ist() in pciehp driver and pci_do_recovery() called by dpc_handler in DPC driver. no luck, there is no lock to protect pci_stop_and_remove_bus_device() against pci_walk_bus(), they hold different semaphore and mutex, pci_stop_and_remove_bus_device holds pci_rescan_remove_lock, and pci_walk_bus() holds pci_bus_sem.

This race condition is not purely code analysis, it could be triggered by following command series:

Quoted 6 lines
  # setpci -s 64:02.0 0x196.w=000a   // 64:02.0 is rootport has DPC capability
  # setpci -s 65:00.0 0x04.w=0544    // 65:00.0 is NVMe SSD populated in above port
  # mount /dev/nvme0n1p1 nvme

One shot will cause system panic and null pointer reference happened. (tested on stable 5.8 & ICX platform)

What is ICX (yes, I know, but you are writing this for wider audience)?

Quoted 4 lines

Buffer I/O error on dev nvme0n1p1, logical block 468843328, async page read

   BUG: kernel NULL pointer dereference, address: 0000000000000050
   #PF: supervisor read access in kernel mode
   #PF: error_code(0x0000) - not-present page

Please, try to remove non-relevant information from the crashes.

Quoted 37 lines
   Oops: 0000 [#1] SMP NOPTI
   CPU: 12 PID: 513 Comm: irq/124-pcie-dp Not tainted 5.8.0-0.0.7.el8.x86_64+ #1
   Hardware name: Intel Corporation Wilxxxx.200x0x0xxx 0x/0x/20x0
   RIP: 0010:report_error_detected.cold.4+0x7d/0xe6
   Code: b6 d0 e8 e8 fe 11 00 e8 16 c5 fb ff be 06 00 00 00 48 89 df e8 d3 65 ff
   ff b8 06 00 00 00 e9 75 fc ff ff 48 8b 43 68 45 31 c9 <48> 8b 50 50 48 83 3a 00
   41 0f 94 c1 45 31 c0 48 85 d2 41 0f 94 c0
   RSP: 0018:ff8e06cf8762fda8 EFLAGS: 00010246
   RAX: 0000000000000000 RBX: ff4e3eaacf42a000 RCX: ff4e3eb31f223c01
   RDX: ff4e3eaacf42a140 RSI: ff4e3eb31f223c00 RDI: ff4e3eaacf42a138
   RBP: ff8e06cf8762fdd0 R08: 00000000000000bf R09: 0000000000000000
   R10: 000000eb8ebeab53 R11: ffffffff93453258 R12: 0000000000000002
   R13: ff4e3eaacf42a130 R14: ff8e06cf8762fe2c R15: ff4e3eab44733828
   FS:  0000000000000000(0000) GS:ff4e3eab1fd00000(0000) knlGS:0000000000000000
   CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
   CR2: 0000000000000050 CR3: 0000000f8f80a004 CR4: 0000000000761ee0
   DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
   DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
   PKRU: 55555554
   Call Trace:
   ? report_normal_detected+0x20/0x20
   report_frozen_detected+0x16/0x20
   pci_walk_bus+0x75/0x90
   ? dpc_irq+0x90/0x90
   pcie_do_recovery+0x157/0x201
   ? irq_finalize_oneshot.part.47+0xe0/0xe0
   dpc_handler+0x29/0x40
   irq_thread_fn+0x24/0x60
   irq_thread+0xea/0x170
   ? irq_forced_thread_fn+0x80/0x80
   ? irq_thread_check_affinity+0xf0/0xf0
   kthread+0x124/0x140
   ? kthread_park+0x90/0x90
   ret_from_fork+0x1f/0x30
   Modules linked in: nft_fib_inet.........
   CR2: 0000000000000050
Quoted 6 lines

With this patch, the handling flow of DPC containment and hotplug is partly ordered and serialized, let hardware DPC do the controller reset etc recovery action first, then DPC driver handling the call-back from device drivers, clear the DPC status, at the end, pciehp driver handles the DLLSC and PDC etc.

This patch closes the race conditon partly.

Not sure if Bjorn require to get rid of "this patch" from commit messages... In any case it's written in documentation.

Signature · 2 lines
With Best Regards,
Andy Shevchenko
#18 lore raw

RE: [PATCH 2/5] PCI: pciehp: check and wait port status out of DPC before handling DLLSC and PDC

Andy,

About the ICX code name, I will align it to public documentation in next version.
' non-relevant information' about the crash, I'm not sure what's not relevant.

Thanks,
Ethan

-----Original Message-----
From: Andy Shevchenko <[email protected]>
Sent: Friday, September 25, 2020 8:33 PM
To: Zhao, Haifeng <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Jia, Pei P <[email protected]>
Subject: Re: [PATCH 2/5] PCI: pciehp: check and wait port status out of DPC before handling DLLSC and PDC

Quoted 17 lines · Ethan Zhao

When root port has DPC capability and it is enabled, then triggered by
errors, DPC DLLSC and PDC interrupts will be sent to DPC driver, pciehp driver at the same time. That will cause following result:

1. Link and device are recovered by hardware DPC and software DPC driver, device isn't removed, but the pciehp might treat it as devce was hot removed.

2. Race condition happens between pciehp_unconfigure_device() called by pciehp_ist() in pciehp driver and pci_do_recovery() called by dpc_handler in DPC driver. no luck, there is no lock to protect pci_stop_and_remove_bus_device() against pci_walk_bus(), they hold different semaphore and mutex, pci_stop_and_remove_bus_device holds pci_rescan_remove_lock, and pci_walk_bus() holds pci_bus_sem.

This race condition is not purely code analysis, it could be triggered by following command series:

Quoted 6 lines
  # setpci -s 64:02.0 0x196.w=000a   // 64:02.0 is rootport has DPC capability
  # setpci -s 65:00.0 0x04.w=0544    // 65:00.0 is NVMe SSD populated in above port
  # mount /dev/nvme0n1p1 nvme

One shot will cause system panic and null pointer reference happened. (tested on stable 5.8 & ICX platform)

What is ICX (yes, I know, but you are writing this for wider audience)?

Quoted 4 lines

Buffer I/O error on dev nvme0n1p1, logical block 468843328, async page read

   BUG: kernel NULL pointer dereference, address: 0000000000000050
   #PF: supervisor read access in kernel mode
   #PF: error_code(0x0000) - not-present page

Please, try to remove non-relevant information from the crashes.

Quoted 37 lines
   Oops: 0000 [#1] SMP NOPTI
   CPU: 12 PID: 513 Comm: irq/124-pcie-dp Not tainted 5.8.0-0.0.7.el8.x86_64+ #1
   Hardware name: Intel Corporation Wilxxxx.200x0x0xxx 0x/0x/20x0
   RIP: 0010:report_error_detected.cold.4+0x7d/0xe6
   Code: b6 d0 e8 e8 fe 11 00 e8 16 c5 fb ff be 06 00 00 00 48 89 df e8 d3 65 ff
   ff b8 06 00 00 00 e9 75 fc ff ff 48 8b 43 68 45 31 c9 <48> 8b 50 50 48 83 3a 00
   41 0f 94 c1 45 31 c0 48 85 d2 41 0f 94 c0
   RSP: 0018:ff8e06cf8762fda8 EFLAGS: 00010246
   RAX: 0000000000000000 RBX: ff4e3eaacf42a000 RCX: ff4e3eb31f223c01
   RDX: ff4e3eaacf42a140 RSI: ff4e3eb31f223c00 RDI: ff4e3eaacf42a138
   RBP: ff8e06cf8762fdd0 R08: 00000000000000bf R09: 0000000000000000
   R10: 000000eb8ebeab53 R11: ffffffff93453258 R12: 0000000000000002
   R13: ff4e3eaacf42a130 R14: ff8e06cf8762fe2c R15: ff4e3eab44733828
   FS:  0000000000000000(0000) GS:ff4e3eab1fd00000(0000) knlGS:0000000000000000
   CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
   CR2: 0000000000000050 CR3: 0000000f8f80a004 CR4: 0000000000761ee0
   DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
   DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
   PKRU: 55555554
   Call Trace:
   ? report_normal_detected+0x20/0x20
   report_frozen_detected+0x16/0x20
   pci_walk_bus+0x75/0x90
   ? dpc_irq+0x90/0x90
   pcie_do_recovery+0x157/0x201
   ? irq_finalize_oneshot.part.47+0xe0/0xe0
   dpc_handler+0x29/0x40
   irq_thread_fn+0x24/0x60
   irq_thread+0xea/0x170
   ? irq_forced_thread_fn+0x80/0x80
   ? irq_thread_check_affinity+0xf0/0xf0
   kthread+0x124/0x140
   ? kthread_park+0x90/0x90
   ret_from_fork+0x1f/0x30
   Modules linked in: nft_fib_inet.........
   CR2: 0000000000000050
Quoted 6 lines

With this patch, the handling flow of DPC containment and hotplug is
partly ordered and serialized, let hardware DPC do the controller
reset etc recovery action first, then DPC driver handling the
call-back from device drivers, clear the DPC status, at the end, pciehp driver handles the DLLSC and PDC etc.

This patch closes the race conditon partly.

Not sure if Bjorn require to get rid of "this patch" from commit messages... In any case it's written in documentation.

--
With Best Regards,
Andy Shevchenko