PCI: avoid SBR for NVIDIA T4
RFC PATCH 1/1 PCI: avoid SBR for NVIDIA T4
Secondary bus reset will fail if NVIDIA T4 card is direct attached to a root port.
So avoid to do bus reset, pci_parent_bus_reset() works nomarlly.
Maybe NVIDIA guys can do some detailed explanation abount the SBR behaviour of GPUs.
--- drivers/pci/quirks.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/pci/quirks.c +1 -1
@@ -3618,7 +3618,7 @@ static void quirk_no_bus_reset(struct pci_dev *dev)*/ static void quirk_nvidia_no_bus_reset (struct pci_dev *dev) { -if ((dev->device &0xffc0 ) ==0x2340 ) +if ((dev->device &0xffc0 ) ==0x2340 || dev->device ==0x1eb8 )quirk_no_bus_reset (dev); } DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID,
Signature · 1 line
2.34.3
Quoted 2 lines · Wu Zongyong
Secondary bus reset will fail if NVIDIA T4 card is direct attached to a root port.
Blank line between paragraphs. Rewrap to fill 75 columns if it's a single paragraph.
Is this only a problem when direct attached to a Root Port? Why would that be? If it's *not* related to being directly under a Root Port, don't mention that at all.
Quoted 4 lines
So avoid to do bus reset,
pci_parent_bus_reset()works nomarlly.Maybe NVIDIA guys can do some detailed explanation abount the SBR behaviour of GPUs.
This is a follow-on to 4c207e7121fa ("PCI: Mark some NVIDIA GPUs to avoid bus reset"), so probably should have a Fixes: tag so it goes whereever that commit goes.
Also copy the subject line from 4c207e7121fa, e.g.,
PCI: Mark NVIDIA T4 GPUs to avoid bus reset
Are there any problem reports or bugzilla issues you can include a URL to?
Quoted 21 lines
Signed-off-by Wu Zongyong <[email protected]>--- drivers/pci/quirks.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/pci/quirks.c +1
-1@@ -3618,7 +3618,7 @@ static void quirk_no_bus_reset(struct pci_dev *dev)*/ static void quirk_nvidia_no_bus_reset (struct pci_dev *dev) { -if ((dev->device &0xffc0 ) ==0x2340 ) +if ((dev->device &0xffc0 ) ==0x2340 || dev->device ==0x1eb8 )quirk_no_bus_reset (dev); } DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID,Signature · 1 line
2.34.3
Quoted 6 lines · Bjorn Helgaas
Quoted 2 lines · Wu Zongyong
Secondary bus reset will fail if NVIDIA T4 card is direct attached to a root port.
Blank line between paragraphs. Rewrap to fill 75 columns if it's a single paragraph.
Will be fixed.
Quoted 4 lines
Is this only a problem when direct attached to a Root Port? Why would that be? If it's *not* related to being directly under a Root Port, don't mention that at all.
Yes, this problem occurs only when the T4 card is direct attached to a Root Port.
I have test it with a T4 card attached to a PCIe Switch or a PCI Bridge, and it works well.
Quoted 6 lines
Quoted 5 lines
So avoid to do bus reset,
pci_parent_bus_reset()works nomarlly.Maybe NVIDIA guys can do some detailed explanation abount the SBR behaviour of GPUs.
Quoted 7 lines
This is a follow-on to
4c207e7121fa("PCI: Mark some NVIDIA GPUs to avoid bus reset"), so probably should have a Fixes: tag so it goes whereever that commit goes.Also copy the subject line from
4c207e7121fa, e.g.,PCI: Mark NVIDIA T4 GPUs to avoid bus reset
Will be fixed too.
Quoted 3 lines
Are there any problem reports or bugzilla issues you can include a URL to?
No, I just find the problem in our test environment and I didn't find a similar report.
Quoted 22 lines
Quoted 22 lines
Signed-off-by Wu Zongyong <[email protected]>--- drivers/pci/quirks.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/pci/quirks.c +1
-1@@ -3618,7 +3618,7 @@ static void quirk_no_bus_reset(struct pci_dev *dev)*/ static void quirk_nvidia_no_bus_reset (struct pci_dev *dev) { -if ((dev->device &0xffc0 ) ==0x2340 ) +if ((dev->device &0xffc0 ) ==0x2340 || dev->device ==0x1eb8 )quirk_no_bus_reset (dev); } DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID,Signature · 1 line
2.34.3
Quoted 5 lines · Wu Zongyong
Quoted 4 lines · Bjorn Helgaas
Quoted 2 lines · Wu Zongyong
Secondary bus reset will fail if NVIDIA T4 card is direct attached to a root port.
Quoted 8 lines
Quoted 3 lines
Is this only a problem when direct attached to a Root Port? Why would that be? If it's *not* related to being directly under a Root Port, don't mention that at all.
Yes, this problem occurs only when the T4 card is direct attached to a Root Port.
I have test it with a T4 card attached to a PCIe Switch or a PCI Bridge, and it works well.
From an electrical and protocol point of view, the device should not be able to tell the difference, so Lukas' suggestion that it may be related to reset delays seems very pertinent.
Bjorn
Quoted 10 lines · Bjorn Helgaas
Quoted 9 lines · Wu Zongyong
Quoted 7 lines · Bjorn Helgaas
Quoted 2 lines · Wu Zongyong
Secondary bus reset will fail if NVIDIA T4 card is direct attached to a root port.
Is this only a problem when direct attached to a Root Port? Why would that be? If it's *not* related to being directly under a Root Port, don't mention that at all.
Quoted 8 lines
Quoted 4 lines
Yes, this problem occurs only when the T4 card is direct attached to a Root Port.
I have test it with a T4 card attached to a PCIe Switch or a PCI Bridge, and it works well.From an electrical and protocol point of view, the device should not be able to tell the difference, so Lukas' suggestion that it may be related to reset delays seems very pertinent.
I will test it with the commits mentioned above.
But it may take some time since it is not easy to replace kernel in our environment.
Quoted 2 lines
Bjorn
Quoted 16 lines · Wu Zongyong
Quoted 15 lines · Bjorn Helgaas
Quoted 13 lines · Wu Zongyong
Quoted 7 lines · Bjorn Helgaas
Quoted 2 lines · Wu Zongyong
Secondary bus reset will fail if NVIDIA T4 card is direct attached to a root port.
Is this only a problem when direct attached to a Root Port? Why would that be? If it's *not* related to being directly under a Root Port, don't mention that at all.
Yes, this problem occurs only when the T4 card is direct attached to a Root Port.
I have test it with a T4 card attached to a PCIe Switch or a PCI Bridge, and it works well.
Quoted 6 lines
Quoted 3 lines
From an electrical and protocol point of view, the device should not be able to tell the difference, so Lukas' suggestion that it may be related to reset delays seems very pertinent.
I will test it with the commits mentioned above.
But it may take some time since it is not easy to replace kernel in our environment.
I have tested it with Lukas' suggestion and it didn't work for T4 cards.
My base kernel is v5.10, and I cherry-picked the following patches:
730643d33e2d ("PCI/PM: Resume subordinate bus in bus type callbacks") 8ef0217227b4 ("PCI/PM: Observe reset delay irrespective of bridge_d3") ac91e6980563 ("PCI: Unify delay handling for reset and resume")
Any other necessary patches I should apply?
Quoted 2 lines
Quoted 2 lines
Bjorn
Quoted 3 lines · Wu Zongyong
Secondary bus reset will fail if NVIDIA T4 card is direct attached to a root port.
So avoid to do bus reset,pci_parent_bus_reset()works nomarlly.
If you cherry-pick commits
8ef0217227b4 ("PCI/PM: Observe reset delay irrespective of bridge_d3") ac91e6980563 ("PCI: Unify delay handling for reset and resume")
from v6.3-rc1, does the issue still persist?
Thanks,
Lukas
PATCH 1/1 PCI: Mark NVIDIA T4 GPUs to avoid bus reset
In reply to Wu ZongyongNVIDIA T4 GPUs do not work with SBR. This problem is found when the T4 card is direct attached to a Root Port only. So avoid bus reset by marking T4 GPUs PCI_DEV_FLAGS_NO_BUS_RESET.
--- drivers/pci/quirks.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/pci/quirks.c +1 -1
@@ -3618,7 +3618,7 @@ static void quirk_no_bus_reset(struct pci_dev *dev)*/ static void quirk_nvidia_no_bus_reset (struct pci_dev *dev) { -if ((dev->device &0xffc0 ) ==0x2340 ) +if ((dev->device &0xffc0 ) ==0x2340 || dev->device ==0x1eb8 )quirk_no_bus_reset (dev); } DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID,
Signature · 1 line
2.34.3
Re: [PATCH] PCI: Mark NVIDIA T4 GPUs to avoid bus reset
Quoted 25 lines · Wu Zongyong
NVIDIA T4 GPUs do not work with SBR. This problem is found when the T4 card is direct attached to a Root Port only. So avoid bus reset by marking T4 GPUs
PCI_DEV_FLAGS_NO_BUS_RESET.Fixes 4c207e7121fa ("PCI: Mark some NVIDIA GPUs to avoid bus reset")Signed-off-by Wu Zongyong <[email protected]>--- drivers/pci/quirks.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/pci/quirks.c +1
-1@@ -3618,7 +3618,7 @@ static void quirk_no_bus_reset(struct pci_dev *dev)*/ static void quirk_nvidia_no_bus_reset (struct pci_dev *dev) { -if ((dev->device &0xffc0 ) ==0x2340 ) +if ((dev->device &0xffc0 ) ==0x2340 || dev->device ==0x1eb8 )quirk_no_bus_reset (dev); } DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID,Signature · 1 line
2.34.3
Any further comments about this patch?
Bjorn Helgaas
AppliedRe: [PATCH] PCI: Mark NVIDIA T4 GPUs to avoid bus reset
In reply to Wu ZongyongQuoted 6 lines · Wu Zongyong
NVIDIA T4 GPUs do not work with SBR. This problem is found when the T4 card is direct attached to a Root Port only. So avoid bus reset by marking T4 GPUs
PCI_DEV_FLAGS_NO_BUS_RESET.Fixes 4c207e7121fa ("PCI: Mark some NVIDIA GPUs to avoid bus reset")Signed-off-by Wu Zongyong <[email protected]>
Applied to pci/virtualization for v6.6, thanks!
Quoted 20 lines
--- drivers/pci/quirks.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/pci/quirks.c +1
-1@@ -3618,7 +3618,7 @@ static void quirk_no_bus_reset(struct pci_dev *dev)*/ static void quirk_nvidia_no_bus_reset (struct pci_dev *dev) { -if ((dev->device &0xffc0 ) ==0x2340 ) +if ((dev->device &0xffc0 ) ==0x2340 || dev->device ==0x1eb8 )quirk_no_bus_reset (dev); } DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID,Signature · 1 line
2.34.3
Re: [PATCH] PCI: Mark NVIDIA T4 GPUs to avoid bus reset
Quoted 9 lines · Bjorn Helgaas
Quoted 6 lines · Wu Zongyong
NVIDIA T4 GPUs do not work with SBR. This problem is found when the T4 card is direct attached to a Root Port only. So avoid bus reset by marking T4 GPUs
PCI_DEV_FLAGS_NO_BUS_RESET.Fixes 4c207e7121fa ("PCI: Mark some NVIDIA GPUs to avoid bus reset")Signed-off-by Wu Zongyong <[email protected]>Applied to pci/virtualization for v6.6, thanks!
I talk about the issue with NVIDIA, and they think the issue is probably related the pci link instead of the T4 GPU card.
I will try to describe the issue I met in detail.
The T4 card which is direct attached to a Root Port and I rebind it to vfio-pci driver. Then I try to use to call some vfio-related api and the ioctl VFIO_GROUP_GET_DEVICE_FD failed.
The stack is (base on kernel v5.10):
vfio_group_fops_unl_ioctl
vfio_group_get_device_fd
vfio_pci_open
vfio_pci_enable // return value is -19
pci_try_reset_function
__pci_reset_function_locked
After the __pci_reset_function_locked(), the dmesg shows:
[12207494.508467] pcieport 0000:3f:00.0: pciehp: Slot(5-1): Link Down [12207494.508535] vfio-pci 0000:40:00.0: No device request channel registered, blocked until released by user [12207494.518426] pci 0000:40:00.0: Removing from iommu group 84 [12207495.532365] pcieport 0000:3f:00.0: pciehp: Slot(5-1): Card present [12207495.532367] pcieport 0000:3f:00.0: pciehp: Slot(5-1): Link Up
NVIDIA people thinks this root port is not going through this reset logic and getting the link down/hot plug interrupts[1].
Can you revert the patch I sent and maybe we should dig it deeply.
Quoted 21 lines
Quoted 21 lines
--- drivers/pci/quirks.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/pci/quirks.c +1
-1@@ -3618,7 +3618,7 @@ static void quirk_no_bus_reset(struct pci_dev *dev)*/ static void quirk_nvidia_no_bus_reset (struct pci_dev *dev) { -if ((dev->device &0xffc0 ) ==0x2340 ) +if ((dev->device &0xffc0 ) ==0x2340 || dev->device ==0x1eb8 )quirk_no_bus_reset (dev); } DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID,Signature · 1 line
2.34.3
Re: [PATCH] PCI: Mark NVIDIA T4 GPUs to avoid bus reset
Quoted 35 lines · On Fri, 8 Sep 2023 10:50:48 +0800 Wu Zongyong <[email protected]>…
Quoted 9 lines · Bjorn Helgaas
Quoted 6 lines · Wu Zongyong
NVIDIA T4 GPUs do not work with SBR. This problem is found when the T4 card is direct attached to a Root Port only. So avoid bus reset by marking T4 GPUs
PCI_DEV_FLAGS_NO_BUS_RESET.Fixes 4c207e7121fa ("PCI: Mark some NVIDIA GPUs to avoid bus reset")Signed-off-by Wu Zongyong <[email protected]>Applied to pci/virtualization for v6.6, thanks!
I talk about the issue with NVIDIA, and they think the issue is probably related the pci link instead of the T4 GPU card.
I will try to describe the issue I met in detail.
The T4 card which is direct attached to a Root Port and I rebind it to vfio-pci driver. Then I try to use to call some vfio-related api and the ioctl
VFIO_GROUP_GET_DEVICE_FDfailed.The stack is (base on kernel v5.10):
vfio_group_fops_unl_ioctl
vfio_group_get_device_fd
vfio_pci_open
vfio_pci_enable// return value is -19
pci_try_reset_function
__pci_reset_function_lockedAfter the
__pci_reset_function_locked(), the dmesg shows:[12207494.508467] pcieport 0000:3f:00.0: pciehp: Slot(5-1): Link Down [12207494.508535] vfio-pci 0000:40:00.0: No device request channel registered, blocked until released by user [12207494.518426] pci 0000:40:00.0: Removing from iommu group 84 [12207495.532365] pcieport 0000:3f:00.0: pciehp: Slot(5-1): Card present [12207495.532367] pcieport 0000:3f:00.0: pciehp: Slot(5-1): Link Up
Quoted 4 lines
NVIDIA people thinks this root port is not going through this reset logic and getting the link down/hot plug interrupts[1].
Can you revert the patch I sent and maybe we should dig it deeply.
Yes, please revert, we do testing with T4 and have not seen any issues with bus reset. The T4 provides neither PM nor FLR reset, so masking bus reset compromises this device for assignment scenarios. I can send a revert patch if requested. Thanks,
Alex
Quoted 21 lines
Quoted 20 lines
Quoted 20 lines
--- drivers/pci/quirks.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/pci/quirks.c +1
-1@@ -3618,7 +3618,7 @@ static void quirk_no_bus_reset(struct pci_dev *dev)*/ static void quirk_nvidia_no_bus_reset (struct pci_dev *dev) { -if ((dev->device &0xffc0 ) ==0x2340 ) +if ((dev->device &0xffc0 ) ==0x2340 || dev->device ==0x1eb8 )quirk_no_bus_reset (dev); } DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID,Signature · 1 line
2.34.3
Re: [PATCH] PCI: Mark NVIDIA T4 GPUs to avoid bus reset
Quoted 41 lines · Alex Williamson
Quoted 38 lines · On Fri, 8 Sep 2023 10:50:48 +0800 Wu Zongyong <[email protected]>…
Quoted 9 lines · Bjorn Helgaas
Quoted 6 lines · Wu Zongyong
NVIDIA T4 GPUs do not work with SBR. This problem is found when the T4 card is direct attached to a Root Port only. So avoid bus reset by marking T4 GPUs
PCI_DEV_FLAGS_NO_BUS_RESET.Fixes 4c207e7121fa ("PCI: Mark some NVIDIA GPUs to avoid bus reset")Signed-off-by Wu Zongyong <[email protected]>Applied to pci/virtualization for v6.6, thanks!
I talk about the issue with NVIDIA, and they think the issue is probably related the pci link instead of the T4 GPU card.
I will try to describe the issue I met in detail.
The T4 card which is direct attached to a Root Port and I rebind it to vfio-pci driver. Then I try to use to call some vfio-related api and the ioctl
VFIO_GROUP_GET_DEVICE_FDfailed.The stack is (base on kernel v5.10):
vfio_group_fops_unl_ioctl
vfio_group_get_device_fd
vfio_pci_open
vfio_pci_enable// return value is -19
pci_try_reset_function
__pci_reset_function_lockedAfter the
__pci_reset_function_locked(), the dmesg shows:[12207494.508467] pcieport 0000:3f:00.0: pciehp: Slot(5-1): Link Down [12207494.508535] vfio-pci 0000:40:00.0: No device request channel registered, blocked until released by user [12207494.518426] pci 0000:40:00.0: Removing from iommu group 84 [12207495.532365] pcieport 0000:3f:00.0: pciehp: Slot(5-1): Card present [12207495.532367] pcieport 0000:3f:00.0: pciehp: Slot(5-1): Link UpNVIDIA people thinks this root port is not going through this reset logic and getting the link down/hot plug interrupts[1].
Quoted 6 lines
Quoted 1 line
Can you revert the patch I sent and maybe we should dig it deeply.
Yes, please revert, we do testing with T4 and have not seen any issues with bus reset. The T4 provides neither PM nor FLR reset, so masking bus reset compromises this device for assignment scenarios. I can send a revert patch if requested. Thanks,
Reverted as below. Hopefully this will make v6.6-rc1.
commit 42f5c40846f3 ("Revert "PCI: Mark NVIDIA T4 GPUs to avoid bus reset"")
Author: Bjorn Helgaas <[email protected]> Date: Fri Sep 8 14:55:30 2023 -0500
Revert "PCI: Mark NVIDIA T4 GPUs to avoid bus reset"
This reverts commit d5af729dc2071273f14cbb94abbc60608142fd83.
d5af729dc207 ("PCI: Mark NVIDIA T4 GPUs to avoid bus reset") avoided Secondary Bus Reset on the T4 because the reset seemed to not work when the T4 was directly attached to a Root Port.
But NVIDIA thinks the issue is probably related to some issue with the Root Port, not with the T4. The T4 provides neither PM nor FLR reset, so masking bus reset compromises this device for assignment scenarios.
Revert d5af729dc207 as requested by Wu Zongyong. This will leave SBR broken in the specific configuration Wu tested, as it was in v6.5, so Wu will debug that further.
drivers/pci/quirks.c +1 -1
@@ -3726,7 +3726,7 @@ static void quirk_no_bus_reset(struct pci_dev *dev)*/ static void quirk_nvidia_no_bus_reset (struct pci_dev *dev) { -if ((dev->device &0xffc0 ) ==0x2340 || dev->device ==0x1eb8 ) +if ((dev->device &0xffc0 ) ==0x2340 )quirk_no_bus_reset (dev); } DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID,