Fix DPC hotplug race and enhance error hanlding
Subsystems (2)
cover Fix DPC hotplug race and enhance error hanlding
This simple patch set fixed some serious security issues found when DPC error injection and NVMe SSD hotplug brute force test were doing -- race condition between DPC handler and pciehp, AER interrupt handlers, caused system hang and system with DPC feature couldn't recover to normal working state as expected (NVMe instance lost, mount operation hang, race PCIe access caused uncorrectable errors reported alternativly etc).
With this patch set applied, stable 5.9-rc6 could pass the PCIe Gen4 NVMe SSD brute force hotplug test with any time interval between hot-remove and plug-in operation tens of times without any errors occur and system works normal.
With this patch set applied, system with DPC feature could recover from NON-FATAL and FATAL errors injection test and works as expected.
System works smoothly when errors happen while hotplug is doing, no uncorrectable errors found.
Brute DPC error injection script:
for i in {0..100}
do
setpci -s 64:02.0 0x196.w=000a
setpci -s 65:00.0 0x04.w=0544
mount /dev/nvme0n1p1 /root/nvme
sleep 1done
Other details see every commits description part.
This patch set could be applied to stable 5.9-rc6 directly.
Help to review and test.
Thanks,
Ethan
Ethan Zhao (5):
PCI: define a function to check and wait till port finish DPC handling
PCI: pciehp: check and wait port status out of DPC before handling
DLLSC and PDC
PCI/ERR: get device before call device driver to avoid null pointer
reference
PCI: only return true when dev io state is really changed PCI/ERR: don't mix io state not changed and no driver together
drivers/pci/hotplug/pciehp_hpc.c | 4 +++- drivers/pci/pci.h | 31 +++---------------------------- drivers/pci/pcie/err.c | 18 ++++++++++++++++-- include/linux/pci.h | 31 +++++++++++++++++++++++++++++++ 4 files changed, 53 insertions(+), 31 deletions(-)
Signature · 1 line
2.18.4
PATCH 4/5 PCI: only return true when dev io state is really changed
When uncorrectable error happens, AER driver and DPC driver interrupt handlers likely call
pcie_do_recovery()->pci_walk_bus()->report_frozen_detected() withpci_channel_io_frozen the same time.
If pci_dev_set_io_state() return true even if the original state ispci_channel_io_frozen, that will cause AER or DPC handler re-enter the error detecting and recovery procedure one after another.
The result is the recovery flow mixed between AER and DPC.
So simplify the pci_dev_set_io_state() function to only return true when dev->error_state is changed.
--- drivers/pci/pci.h | 31 +++---------------------------- 1 file changed, 3 insertions(+), 28 deletions(-)
drivers/pci/pci.h +3 -28
@@ -362,35 +362,10 @@ static inline bool pci_dev_set_io_state(struct pci_dev *dev,bool changed = false;device_lock_assert (&dev->dev); -switch (new) { -case pci_channel_io_perm_failure: -switch (dev->error_state) { -case pci_channel_io_frozen: -case pci_channel_io_normal: -case pci_channel_io_perm_failure: - changed = true; -break ; - } -break ; -case pci_channel_io_frozen: -switch (dev->error_state) { -case pci_channel_io_frozen: -case pci_channel_io_normal: - changed = true; -break ; - } -break ; -case pci_channel_io_normal: -switch (dev->error_state) { -case pci_channel_io_frozen: -case pci_channel_io_normal: - changed = true; -break ; - } -break ; - } -if (changed) +if (dev->error_state != new) { dev->error_state = new; + changed = true; + }return changed; }
Signature · 1 line
2.18.4
Re: [PATCH 4/5] PCI: only return true when dev io state is really changed
Hi Ethan,
Quoted 50 lines · Ethan Zhao
When uncorrectable error happens, AER driver and DPC driver interrupt handlers likely call
pcie_do_recovery()->pci_walk_bus()->report_frozen_detected() with
pci_channel_io_frozenthe same time.
Ifpci_dev_set_io_state()return true even if the original state ispci_channel_io_frozen, that will cause AER or DPC handler re-enter the error detecting and recovery procedure one after another.
The result is the recovery flow mixed between AER and DPC.
So simplify thepci_dev_set_io_state()function to only return true whendev->error_stateis changed.Signed-off-by Ethan Zhao <[email protected]>Tested-by Wen jin <[email protected]>Tested-by Shanshan Zhang <[email protected]>--- drivers/pci/pci.h | 31 +++---------------------------- 1 file changed, 3 insertions(+), 28 deletions(-)drivers/pci/pci.h +0
-24@@ -362,35 +362,10 @@ static inline bool pci_dev_set_io_state(struct pci_dev *dev,bool changed = false;device_lock_assert (&dev->dev); -switch (new) { -case pci_channel_io_perm_failure: -switch (dev->error_state) { -case pci_channel_io_frozen: -case pci_channel_io_normal: -case pci_channel_io_perm_failure: - changed = true; -break ; - } -break ; -case pci_channel_io_frozen: -switch (dev->error_state) { -case pci_channel_io_frozen: -case pci_channel_io_normal: - changed = true; -break ; - } -break ; -case pci_channel_io_normal: -switch (dev->error_state) { -case pci_channel_io_frozen: -case pci_channel_io_normal: - changed = true; -break ;
drivers/pci/pci.h static inline bool pci_dev_set_io_state(struct pci_dev *dev, · line 370
Quoted 10 lines
drivers/pci/pci.h +3
-4- } -break ; - } -if (changed) +if (dev->error_state != new) { dev->error_state = new; + changed = true; + }return changed; }
The flow is a lot easier to follow now. Thank you.
Re: [PATCH 4/5] PCI: only return true when dev io state is really changed
In reply to Ethan ZhaoQuoted 4 lines · Ethan Zhao
When uncorrectable error happens, AER driver and DPC driver interrupt handlers likely call
pcie_do_recovery()->pci_walk_bus()->report_frozen_detected() withpci_channel_io_frozenthe same time.
Call chains are better to read if they split like
foo() ->
bar() ->
baz()Quoted 6 lines
If
pci_dev_set_io_state()return true even if the original state ispci_channel_io_frozen, that will cause AER or DPC handler re-enter the error detecting and recovery procedure one after another.
The result is the recovery flow mixed between AER and DPC.
So simplify thepci_dev_set_io_state()function to only return true whendev->error_stateis changed.
...
drivers/pci/pci.h static inline bool pci_dev_set_io_state(struct pci_dev *dev, · line 369
Quoted 5 lines
drivers/pci/pci.h +3
-0+if (dev->error_state != new) { dev->error_state = new; + changed = true; + }return changed;
Perhaps
if (dev->error_state == new) return changed; dev->error_state = new; return true;
?
Signature · 2 lines
With Best Regards, Andy Shevchenko
RE: [PATCH 4/5] PCI: only return true when dev io state is really changed
Yes, better !
-----Original Message-----
From: Andy Shevchenko <[email protected]>
Sent: Friday, September 25, 2020 8:38 PM
To: Zhao, Haifeng <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Jia, Pei P <[email protected]>
Subject: Re: [PATCH 4/5] PCI: only return true when dev io state is really changed
Quoted 4 lines · Ethan Zhao
When uncorrectable error happens, AER driver and DPC driver interrupt handlers likely call
pcie_do_recovery()->pci_walk_bus()->report_frozen_detected() withpci_channel_io_frozenthe same time.
Call chains are better to read if they split like
foo() ->
bar() ->
baz()Quoted 6 lines
If
pci_dev_set_io_state()return true even if the original state ispci_channel_io_frozen, that will cause AER or DPC handler re-enter the error detecting and recovery procedure one after another.
The result is the recovery flow mixed between AER and DPC.
So simplify thepci_dev_set_io_state()function to only return true whendev->error_stateis changed.
...
drivers/pci/pci.h static inline bool pci_dev_set_io_state(struct pci_dev *dev, · line 369
Quoted 5 lines
drivers/pci/pci.h +3
-0+if (dev->error_state != new) { dev->error_state = new; + changed = true; + }return changed;
Perhaps
if (dev->error_state == new) return changed; dev->error_state = new; return true;
?
--
With Best Regards,
Andy Shevchenko
PATCH 5/5 PCI/ERR: don't mix io state not changed and no driver together
In reply to Ethan ZhaoWhen we see 'can't recover (no error_detected callback)' on console, Maybe the reason is io state is not changed by calling pci_dev_set_io_state(), that is confused. fix it.
--- drivers/pci/pcie/err.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/pci/pcie/err.c +4 -2
@@ -55,8 +55,10 @@ static int report_error_detected(struct pci_dev *dev,if (!pci_dev_get (dev))return 0 ;device_lock (&dev->dev); -if (!pci_dev_set_io_state (dev, state) || - !dev->driver || +if (!pci_dev_set_io_state (dev, state)) { +pci_dbg (dev,"Device might already being in error handling ...\n" ); + vote =PCI_ERS_RESULT_NONE ; + }else if (!dev->driver || !dev->driver->err_handler || !dev->driver->err_handler->error_detected) {/*
Signature · 1 line
2.18.4
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH 1/5] PCI: define a function to check and wait till port finish DPC handling
Quoted 7 lines · Ethan Zhao
Once root port DPC capability is enabled and triggered, at the beginning of DPC is triggered, the DPC status bits are set by hardware and then sends DPC/DLLSC/PDC interrupts to OS DPC and pciehp drivers, it will take the port and software DPC interrupt handler 10ms to 50ms (test data on ICX platform & stable 5.9-rc6) to complete the DPC containment procedure till the DPC status is cleared at the end of the DPC interrupt handler.
Quoted 8 lines
We use this function to check if the root port is in DPC handling status and wait till the hardware and software completed the procedure.
#include <
linux/interrupt.h>
#include <linux/io.h>
#include <linux/resource_ext.h>+#include <
linux/delay.h>
Keep it sorted?
Quoted 1 line
#include <
uapi/linux/pci.h>
...
Review
Quoted 9 lines
(quoted diff) +8
-0+#ifdef CONFIG_PCIE_DPC +static inline bool pci_wait_port_outdpc(struct pci_dev *pdev) +{ + u16 cap = pdev->dpc_cap, status; + u16 loop = 0; + + if (!cap) { + pci_WARN_ONCE(pdev, !cap, "No DPC capability initiated\n");
But why? Is this feature mandatory to have? Then the same question about ifdeffery, otherwise it's pretty normal to not have a feature, right?
Review
Quoted 10 lines
(quoted diff) +9
-0+ return false; + } + pci_read_config_word(pdev, cap + PCI_EXP_DPC_STATUS, &status); + pci_dbg(pdev, "DPC status %x, cap %x\n", status, cap); + while (status & PCI_EXP_DPC_STATUS_TRIGGER && loop < 100) { + msleep(10); + loop++; + pci_read_config_word(pdev, cap + PCI_EXP_DPC_STATUS, &status); + }
Can we have rather something like readx_poll_timeout() for PCI and use them here?
Quoted 17 lines
(quoted diff) +13
-0+ if (!(status & PCI_EXP_DPC_STATUS_TRIGGER)) { + pci_dbg(pdev, "Out of DPC status %x, time cost %d ms\n", status, loop*10); + return true; + } + pci_dbg(pdev, "Timeout to wait port out of DPC status\n"); + return false; +} +#else +static inline bool pci_wait_port_outdpc(struct pci_dev *pdev) +{ + return true; +} +#endif #endif /* LINUX_PCI_H */Signature · 1 line
2.18.4
Signature · 2 lines
With Best Regards, Andy Shevchenko
RE: [PATCH 1/5] PCI: define a function to check and wait till port finish DPC handling
Andy,
About the header file, yes, to keep the order.
The function was already defined with #ifdef CONFIG_PCIE_DPC.
As to ' readx_poll_timeout()' if there is generic one, I would like to use it. Seems there is no yet ?
Thanks,
Ethan
-----Original Message-----
From: Andy Shevchenko <[email protected]>
Sent: Friday, September 25, 2020 8:25 PM
To: Zhao, Haifeng <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Jia, Pei P <[email protected]>
Subject: Re: [PATCH 1/5] PCI: define a function to check and wait till port finish DPC handling
Quoted 7 lines · Ethan Zhao
Once root port DPC capability is enabled and triggered, at the
beginning of DPC is triggered, the DPC status bits are set by hardware
and then sends DPC/DLLSC/PDC interrupts to OS DPC and pciehp drivers,
it will take the port and software DPC interrupt handler 10ms to 50ms
(test data on ICX platform & stable 5.9-rc6) to complete the DPC
containment procedure till the DPC status is cleared at the end of the DPC interrupt handler.
Quoted 8 lines
We use this function to check if the root port is in DPC handling status and wait till the hardware and software completed the procedure.
#include <
linux/interrupt.h>
#include <linux/io.h>
#include <linux/resource_ext.h>+#include <
linux/delay.h>
Keep it sorted?
Quoted 1 line
#include <
uapi/linux/pci.h>
...
Review
Quoted 8 lines
(quoted diff) +7
-0+#ifdef CONFIG_PCIE_DPC +static inline bool pci_wait_port_outdpc(struct pci_dev *pdev) { + u16 cap = pdev->dpc_cap, status; + u16 loop = 0; + + if (!cap) { + pci_WARN_ONCE(pdev, !cap, "No DPC capability initiated\n");
But why? Is this feature mandatory to have? Then the same question about ifdeffery, otherwise it's pretty normal to not have a feature, right?
Review
Quoted 10 lines
(quoted diff) +9
-0+ return false; + } + pci_read_config_word(pdev, cap + PCI_EXP_DPC_STATUS, &status); + pci_dbg(pdev, "DPC status %x, cap %x\n", status, cap); + while (status & PCI_EXP_DPC_STATUS_TRIGGER && loop < 100) { + msleep(10); + loop++; + pci_read_config_word(pdev, cap + PCI_EXP_DPC_STATUS, &status); + }
Can we have rather something like readx_poll_timeout() for PCI and use them here?
Quoted 16 lines
(quoted diff) +12
-1+ if (!(status & PCI_EXP_DPC_STATUS_TRIGGER)) { + pci_dbg(pdev, "Out of DPC status %x, time cost %d ms\n", status, loop*10); + return true; + } + pci_dbg(pdev, "Timeout to wait port out of DPC status\n"); + return false; +} +#else +static inline bool pci_wait_port_outdpc(struct pci_dev *pdev) { + return true; +} +#endif #endif /* LINUX_PCI_H */ --2.18.4
Signature · 2 lines
With Best Regards, Andy Shevchenko
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH 3/5] PCI/ERR: get device before call device driver to avoid null pointer reference
Quoted 2 lines · Ethan Zhao
During DPC error injection test we found there is race condition between pciehp and DPC driver, null pointer reference caused panic as following
null -> NULL
Quoted 7 lines
# setpci -s 64:02.0 0x196.w=000a // 64:02.0 is rootport has DPC capability # setpci -s 65:00.0 0x04.w=0544 // 65:00.0 is NVMe SSD populated in above port # mount /dev/nvme0n1p1 nvme
Quoted 7 lines
(tested on stable 5.8 & ICX platform)
Buffer I/O error on dev nvme0n1p1, logical block 468843328, async page read
BUG: kernel NULL pointer dereference, address: 0000000000000050 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page
Same comment about Oops.
Signature · 2 lines
With Best Regards, Andy Shevchenko
Re: [PATCH 3/5] PCI/ERR: get device before call device driver to avoid null pointer reference
Preferred style, there will be cleared comment in v6.
Thanks,
Ethan
Quoted 29 lines · Andy Shevchenko
Quoted 2 lines · Ethan Zhao
During DPC error injection test we found there is race condition between pciehp and DPC driver, null pointer reference caused panic as following
null -> NULL
Quoted 7 lines
# setpci -s 64:02.0 0x196.w=000a // 64:02.0 is rootport has DPC capability # setpci -s 65:00.0 0x04.w=0544 // 65:00.0 is NVMe SSD populated in above port # mount /dev/nvme0n1p1 nvmeQuoted 7 lines
(tested on stable 5.8 & ICX platform)
Buffer I/O error on dev nvme0n1p1, logical block 468843328, async page read
BUG: kernel NULL pointer dereference, address: 0000000000000050 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present pageSame comment about Oops.
--
With Best Regards,
Andy Shevchenko
Re: [PATCH 3/5] PCI/ERR: get device before call device driver to avoid null pointer reference
Quoted 1 line · Ethan Zhao
Preferred style, there will be cleared comment in v6.
Avoid top postings.
Quoted 18 lines
Quoted 16 lines · Andy Shevchenko
Quoted 2 lines · Ethan Zhao
During DPC error injection test we found there is race condition between pciehp and DPC driver, null pointer reference caused panic as following
null -> NULL
Quoted 9 lines
# setpci -s 64:02.0 0x196.w=000a // 64:02.0 is rootport has DPC capability # setpci -s 65:00.0 0x04.w=0544 // 65:00.0 is NVMe SSD populated in above port # mount /dev/nvme0n1p1 nvme(tested on stable 5.8 & ICX platform)
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Buffer I/O error on dev nvme0n1p1, logical block 468843328, async page read
BUG: kernel NULL pointer dereference, address: 0000000000000050 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present pageSame comment about Oops.
In another thread it was a good advice to move the full Oops (if you think it's very useful to have) after the cutter '---' line, so it will be in email archives but Git history.
Signature · 2 lines
With Best Regards, Andy Shevchenko
Re: [PATCH 3/5] PCI/ERR: get device before call device driver to avoid null pointer reference
Andy,
Quoted 30 lines · Andy Shevchenko
Quoted 1 line · Ethan Zhao
Preferred style, there will be cleared comment in v6.
Avoid top postings.
Quoted 24 lines
Quoted 22 lines · Andy Shevchenko
Quoted 2 lines · Ethan Zhao
During DPC error injection test we found there is race condition between pciehp and DPC driver, null pointer reference caused panic as following
null -> NULL
Quoted 14 lines
# setpci -s 64:02.0 0x196.w=000a // 64:02.0 is rootport has DPC capability # setpci -s 65:00.0 0x04.w=0544 // 65:00.0 is NVMe SSD populated in above port # mount /dev/nvme0n1p1 nvme(tested on stable 5.8 & ICX platform)
Buffer I/O error on dev nvme0n1p1, logical block 468843328, async page read
BUG: kernel NULL pointer dereference, address: 0000000000000050 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Same comment about Oops.
In another thread it was a good advice to move the full Oops (if you think it's very useful to have) after the cutter '---' line, so it will be in email archives but Git history.
So git history wouldn't give any of the Oops context, and he/she has to access LKML,
if offline, then ...lost.
Thanks,
Ethan
Quoted 6 lines
--
With Best Regards,
Andy Shevchenko
Re: [PATCH 3/5] PCI/ERR: get device before call device driver to avoid null pointer reference
Quoted 10 lines · Ethan Zhao
Quoted 8 lines · Andy Shevchenko
Quoted 1 line · Ethan Zhao
Preferred style, there will be cleared comment in v6.
Avoid top postings.
Quoted 3 lines
Quoted 1 line · Andy Shevchenko
On Thu, Sep 24, 2020 at 10:34:21PM -0400, Ethan Zhao wrote:
...
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Buffer I/O error on dev nvme0n1p1, logical block 468843328, async page read
BUG: kernel NULL pointer dereference, address: 0000000000000050 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present pageSame comment about Oops.
Quoted 7 lines
Quoted 3 lines
In another thread it was a good advice to move the full Oops (if you think it's very useful to have) after the cutter '---' line, so it will be in email archives but Git history.
So git history wouldn't give any of the Oops context, and he/she has to access LKML,
if offline, then ...lost.
Tell me, do you really think the line:
#PF: error_code(0x0000) - not-present page
makes any sense in the commit message?
I do not think so. And so on, for almost 60% of the Oops. Really, to help one person you will make millions suffering. It's not okay.
Signature · 2 lines
With Best Regards, Andy Shevchenko
Re: [PATCH 3/5] PCI/ERR: get device before call device driver to avoid null pointer reference
Quoted 31 lines · Andy Shevchenko
Quoted 10 lines · Ethan Zhao
Quoted 8 lines · Andy Shevchenko
Quoted 1 line · Ethan Zhao
Preferred style, there will be cleared comment in v6.
Avoid top postings.
Quoted 3 lines
Quoted 1 line · Andy Shevchenko
On Thu, Sep 24, 2020 at 10:34:21PM -0400, Ethan Zhao wrote:
...
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Buffer I/O error on dev nvme0n1p1, logical block 468843328, async page read
BUG: kernel NULL pointer dereference, address: 0000000000000050 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present pageSame comment about Oops.
Quoted 7 lines
Quoted 3 lines
In another thread it was a good advice to move the full Oops (if you think it's very useful to have) after the cutter '---' line, so it will be in email archives but Git history.
So git history wouldn't give any of the Oops context, and he/she has to access LKML,
if offline, then ...lost.
Quoted 7 lines
Tell me, do you really think the line:
#PF: error_code(0x0000) - not-present pagemakes any sense in the commit message?
I do not think so. And so on, for almost 60% of the Oops. Really, to help one person you will make millions suffering. It's not okay.
If you and millions feel so suffered, why not try to simplify the Oops code to not
output nonsense too much to console and log. :)
They might not be so important to old birds as you. but it is easy to cut off the road for newcomers.
Anyway, it is not the focus of this patchset. help to take a look and try the code.
Thanks,
Ethan
Quoted 5 lines
--
With Best Regards,
Andy Shevchenko
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH 2/5] PCI: pciehp: check and wait port status out of DPC before handling DLLSC and PDC
Quoted 17 lines · Ethan Zhao
When root port has DPC capability and it is enabled, then triggered by errors, DPC DLLSC and PDC interrupts will be sent to DPC driver, pciehp driver at the same time. That will cause following result:
1. Link and device are recovered by hardware DPC and software DPC driver, device isn't removed, but the pciehp might treat it as devce was hot removed.
2. Race condition happens between
pciehp_unconfigure_device()called bypciehp_ist()in pciehp driver andpci_do_recovery()called bydpc_handlerin DPC driver. no luck, there is no lock to protectpci_stop_and_remove_bus_device()againstpci_walk_bus(), they hold different semaphore and mutex,pci_stop_and_remove_bus_deviceholdspci_rescan_remove_lock, andpci_walk_bus()holdspci_bus_sem.This race condition is not purely code analysis, it could be triggered by following command series:
Quoted 6 lines
# setpci -s 64:02.0 0x196.w=000a // 64:02.0 is rootport has DPC capability # setpci -s 65:00.0 0x04.w=0544 // 65:00.0 is NVMe SSD populated in above port # mount /dev/nvme0n1p1 nvmeOne shot will cause system panic and null pointer reference happened. (tested on stable 5.8 & ICX platform)
What is ICX (yes, I know, but you are writing this for wider audience)?
Quoted 4 lines
Buffer I/O error on dev nvme0n1p1, logical block 468843328, async page read
BUG: kernel NULL pointer dereference, address: 0000000000000050 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page
Please, try to remove non-relevant information from the crashes.
Quoted 37 lines
Oops: 0000 [#1] SMP NOPTI CPU: 12 PID: 513 Comm: irq/124-pcie-dp Not tainted 5.8.0-0.0.7.el8.x86_64+ #1 Hardware name: Intel Corporation Wilxxxx.200x0x0xxx 0x/0x/20x0 RIP: 0010:report_error_detected.cold.4+0x7d/0xe6 Code: b6 d0 e8 e8 fe 11 00 e8 16 c5 fb ff be 06 00 00 00 48 89 df e8 d3 65 ff ff b8 06 00 00 00 e9 75 fc ff ff 48 8b 43 68 45 31 c9 <48> 8b 50 50 48 83 3a 00 41 0f 94 c1 45 31 c0 48 85 d2 41 0f 94 c0 RSP: 0018:ff8e06cf8762fda8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ff4e3eaacf42a000 RCX: ff4e3eb31f223c01 RDX: ff4e3eaacf42a140 RSI: ff4e3eb31f223c00 RDI: ff4e3eaacf42a138 RBP: ff8e06cf8762fdd0 R08: 00000000000000bf R09: 0000000000000000 R10: 000000eb8ebeab53 R11: ffffffff93453258 R12: 0000000000000002 R13: ff4e3eaacf42a130 R14: ff8e06cf8762fe2c R15: ff4e3eab44733828 FS: 0000000000000000(0000) GS:ff4e3eab1fd00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000050 CR3: 0000000f8f80a004 CR4: 0000000000761ee0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 PKRU: 55555554 Call Trace: ? report_normal_detected+0x20/0x20 report_frozen_detected+0x16/0x20 pci_walk_bus+0x75/0x90 ? dpc_irq+0x90/0x90 pcie_do_recovery+0x157/0x201 ? irq_finalize_oneshot.part.47+0xe0/0xe0 dpc_handler+0x29/0x40 irq_thread_fn+0x24/0x60 irq_thread+0xea/0x170 ? irq_forced_thread_fn+0x80/0x80 ? irq_thread_check_affinity+0xf0/0xf0 kthread+0x124/0x140 ? kthread_park+0x90/0x90 ret_from_fork+0x1f/0x30 Modules linked in: nft_fib_inet......... CR2: 0000000000000050
Quoted 6 lines
With this patch, the handling flow of DPC containment and hotplug is partly ordered and serialized, let hardware DPC do the controller reset etc recovery action first, then DPC driver handling the call-back from device drivers, clear the DPC status, at the end, pciehp driver handles the DLLSC and PDC etc.
This patch closes the race conditon partly.
Not sure if Bjorn require to get rid of "this patch" from commit messages... In any case it's written in documentation.
Signature · 2 lines
With Best Regards, Andy Shevchenko
RE: [PATCH 2/5] PCI: pciehp: check and wait port status out of DPC before handling DLLSC and PDC
Andy,
About the ICX code name, I will align it to public documentation in next version.
' non-relevant information' about the crash, I'm not sure what's not relevant.
Thanks,
Ethan
-----Original Message-----
From: Andy Shevchenko <[email protected]>
Sent: Friday, September 25, 2020 8:33 PM
To: Zhao, Haifeng <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Jia, Pei P <[email protected]>
Subject: Re: [PATCH 2/5] PCI: pciehp: check and wait port status out of DPC before handling DLLSC and PDC
Quoted 17 lines · Ethan Zhao
When root port has DPC capability and it is enabled, then triggered by
errors, DPC DLLSC and PDC interrupts will be sent to DPC driver, pciehp driver at the same time. That will cause following result:1. Link and device are recovered by hardware DPC and software DPC driver, device isn't removed, but the pciehp might treat it as devce was hot removed.
2. Race condition happens between
pciehp_unconfigure_device()called bypciehp_ist()in pciehp driver andpci_do_recovery()called bydpc_handlerin DPC driver. no luck, there is no lock to protectpci_stop_and_remove_bus_device()againstpci_walk_bus(), they hold different semaphore and mutex,pci_stop_and_remove_bus_deviceholdspci_rescan_remove_lock, andpci_walk_bus()holdspci_bus_sem.This race condition is not purely code analysis, it could be triggered by following command series:
Quoted 6 lines
# setpci -s 64:02.0 0x196.w=000a // 64:02.0 is rootport has DPC capability # setpci -s 65:00.0 0x04.w=0544 // 65:00.0 is NVMe SSD populated in above port # mount /dev/nvme0n1p1 nvmeOne shot will cause system panic and null pointer reference happened. (tested on stable 5.8 & ICX platform)
What is ICX (yes, I know, but you are writing this for wider audience)?
Quoted 4 lines
Buffer I/O error on dev nvme0n1p1, logical block 468843328, async page read
BUG: kernel NULL pointer dereference, address: 0000000000000050 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page
Please, try to remove non-relevant information from the crashes.
Quoted 37 lines
Oops: 0000 [#1] SMP NOPTI CPU: 12 PID: 513 Comm: irq/124-pcie-dp Not tainted 5.8.0-0.0.7.el8.x86_64+ #1 Hardware name: Intel Corporation Wilxxxx.200x0x0xxx 0x/0x/20x0 RIP: 0010:report_error_detected.cold.4+0x7d/0xe6 Code: b6 d0 e8 e8 fe 11 00 e8 16 c5 fb ff be 06 00 00 00 48 89 df e8 d3 65 ff ff b8 06 00 00 00 e9 75 fc ff ff 48 8b 43 68 45 31 c9 <48> 8b 50 50 48 83 3a 00 41 0f 94 c1 45 31 c0 48 85 d2 41 0f 94 c0 RSP: 0018:ff8e06cf8762fda8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ff4e3eaacf42a000 RCX: ff4e3eb31f223c01 RDX: ff4e3eaacf42a140 RSI: ff4e3eb31f223c00 RDI: ff4e3eaacf42a138 RBP: ff8e06cf8762fdd0 R08: 00000000000000bf R09: 0000000000000000 R10: 000000eb8ebeab53 R11: ffffffff93453258 R12: 0000000000000002 R13: ff4e3eaacf42a130 R14: ff8e06cf8762fe2c R15: ff4e3eab44733828 FS: 0000000000000000(0000) GS:ff4e3eab1fd00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000050 CR3: 0000000f8f80a004 CR4: 0000000000761ee0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 PKRU: 55555554 Call Trace: ? report_normal_detected+0x20/0x20 report_frozen_detected+0x16/0x20 pci_walk_bus+0x75/0x90 ? dpc_irq+0x90/0x90 pcie_do_recovery+0x157/0x201 ? irq_finalize_oneshot.part.47+0xe0/0xe0 dpc_handler+0x29/0x40 irq_thread_fn+0x24/0x60 irq_thread+0xea/0x170 ? irq_forced_thread_fn+0x80/0x80 ? irq_thread_check_affinity+0xf0/0xf0 kthread+0x124/0x140 ? kthread_park+0x90/0x90 ret_from_fork+0x1f/0x30 Modules linked in: nft_fib_inet......... CR2: 0000000000000050
Quoted 6 lines
With this patch, the handling flow of DPC containment and hotplug is
partly ordered and serialized, let hardware DPC do the controller
reset etc recovery action first, then DPC driver handling the
call-back from device drivers, clear the DPC status, at the end, pciehp driver handles the DLLSC and PDC etc.This patch closes the race conditon partly.
Not sure if Bjorn require to get rid of "this patch" from commit messages... In any case it's written in documentation.
--
With Best Regards,
Andy Shevchenko