Whitelist AMD host bridge device(s) to enable P2P DMA
Subsystems (2)
PATCH 1/1 Whitelist AMD host bridge device(s) to enable P2P DMA
Current implementation will disallow P2P DMA if the participating devices belong to different root complexes. Implementation allows this default behavior to be overridden for whitelisted devices. The patch adds an AMD host bridge to be whitelisted
--- drivers/pci/p2pdma.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/pci/p2pdma.c +2 -0
@@ -305,6 +305,8 @@ static const struct pci_p2pdma_whitelist_entry { {PCI_VENDOR_ID_INTEL, 0x2032, 0}, {PCI_VENDOR_ID_INTEL, 0x2033, 0}, {PCI_VENDOR_ID_INTEL, 0x2020, 0}, +/* AMD Host Bridge Devices */ + {PCI_VENDOR_ID_AMD, 0x1480, 0}, {} };
Signature · 1 line
2.31.1
Quoted 5 lines · Ramesh Errabolu
Current implementation will disallow P2P DMA if the participating devices belong to different root complexes. Implementation allows this default behavior to be overridden for whitelisted devices. The patch adds an AMD host bridge to be whitelisted
Why do we need this? cpu_supports_p2pdma() should return true for all AMD Zen CPUs.
Alex
Quoted 22 lines
Signed-off-by Ramesh Errabolu <[email protected]>--- drivers/pci/p2pdma.c | 2 ++ 1 file changed, 2 insertions(+)drivers/pci/p2pdma.c +2
-1@@ -305,6 +305,8 @@ static const struct pci_p2pdma_whitelist_entry { {PCI_VENDOR_ID_INTEL, 0x2032, 0}, {PCI_VENDOR_ID_INTEL, 0x2033, 0}, {PCI_VENDOR_ID_INTEL, 0x2020, 0}, +/* AMD Host Bridge Devices */ + {PCI_VENDOR_ID_AMD, 0x1480, 0}, {} }; --2.31.1
Am 2021-08-11 um 3:29 p.m. schrieb Alex Deucher:
Quoted 7 lines · Ramesh Errabolu
Quoted 4 lines
Current implementation will disallow P2P DMA if the participating devices belong to different root complexes. Implementation allows this default behavior to be overridden for whitelisted devices. The patch adds an AMD host bridge to be whitelisted
Why do we need this?
cpu_supports_p2pdma()should return true for all AMD Zen CPUs.
This is part of our on-going work to get P2P support upstream. We want to use pci_p2pdma_distance_many to determine whether P2P is possible between a pair of devices. This whitelist is used in this function. This will affect the P2P links reported in the topology and it will be double-checked in the BO mapping function to ensure a peer mapping is legal.
I think this change is a bit free of context at the moment, as we're still working on a few other loose ends for P2P support in our internal branch. I'm hoping we'll have a bigger patch series for upstreamable KFD P2P support ready in a few weeks. I also think we'll probably want to add a few more PCI IDs for other supported AMD root complexes.
Regards,
Felix
Quoted 24 lines
Alex
Quoted 21 lines
Signed-off-by Ramesh Errabolu <[email protected]>--- drivers/pci/p2pdma.c | 2 ++ 1 file changed, 2 insertions(+)drivers/pci/p2pdma.c +2
-1@@ -305,6 +305,8 @@ static const struct pci_p2pdma_whitelist_entry { {PCI_VENDOR_ID_INTEL, 0x2032, 0}, {PCI_VENDOR_ID_INTEL, 0x2033, 0}, {PCI_VENDOR_ID_INTEL, 0x2020, 0}, +/* AMD Host Bridge Devices */ + {PCI_VENDOR_ID_AMD, 0x1480, 0}, {} }; --2.31.1
Alex Deucher
Reviewed-byQuoted 17 lines · Felix Kuehling
Am 2021-08-11 um 3:29 p.m. schrieb Alex Deucher:
Quoted 7 lines · Ramesh Errabolu
Quoted 4 lines
Current implementation will disallow P2P DMA if the participating devices belong to different root complexes. Implementation allows this default behavior to be overridden for whitelisted devices. The patch adds an AMD host bridge to be whitelisted
Why do we need this?
cpu_supports_p2pdma()should return true for all AMD Zen CPUs.This is part of our on-going work to get P2P support upstream. We want to use
pci_p2pdma_distance_manyto determine whether P2P is possible between a pair of devices. This whitelist is used in this function. This will affect the P2P links reported in the topology and it will be double-checked in the BO mapping function to ensure a peer mapping is legal.
Quoted 5 lines
I think this change is a bit free of context at the moment, as we're still working on a few other loose ends for P2P support in our internal branch. I'm hoping we'll have a bigger patch series for upstreamable KFD P2P support ready in a few weeks. I also think we'll probably want to add a few more PCI IDs for other supported AMD root complexes.
We don't need to keep adding AMD root complexes. You must be using an older kernel or something. All root complexes on all Zen platforms support P2P DMA. See:
commit dea286bb71baded7d2fb4f090e3b9fd2c1ccac58
Author: Logan Gunthorpe <[email protected]> Date: Wed Jul 29 17:18:44 2020 -0600
PCI/P2PDMA: Allow P2PDMA on AMD Zen and newer CPUs
Allow P2PDMA if the CPU vendor is AMD and family is 0x17 (Zen) or greater.
Alex
Quoted 29 lines
Regards,
FelixQuoted 24 lines
Alex
Quoted 21 lines
Signed-off-by Ramesh Errabolu <[email protected]>--- drivers/pci/p2pdma.c | 2 ++ 1 file changed, 2 insertions(+)drivers/pci/p2pdma.c +2
-1@@ -305,6 +305,8 @@ static const struct pci_p2pdma_whitelist_entry { {PCI_VENDOR_ID_INTEL, 0x2032, 0}, {PCI_VENDOR_ID_INTEL, 0x2033, 0}, {PCI_VENDOR_ID_INTEL, 0x2020, 0}, +/* AMD Host Bridge Devices */ + {PCI_VENDOR_ID_AMD, 0x1480, 0}, {} }; --2.31.1
Errabolu, Ramesh
Reviewed-by Alex DeucherI will investigate it further. I am using DKMS-5.11 branch. The codebase I am using to build has the right definition i.e. allow P2PDMA for Zen CPU's.
Regards,
Ramesh
-----Original Message-----
From: Alex Deucher <[email protected]>
Sent: Wednesday, August 11, 2021 4:07 PM
To: Kuehling, Felix <[email protected]>
Cc: Errabolu, Ramesh <[email protected]>; amd-gfx list <[email protected]>; Maling list - DRI developers <[email protected]>; Bjorn Helgaas <[email protected]>; Linux PCI <[email protected]>; LKML <[email protected]>
Subject: Re: [PATCH] Whitelist AMD host bridge device(s) to enable P2P DMA
[CAUTION: External Email]
Quoted 17 lines · Felix Kuehling
Am 2021-08-11 um 3:29 p.m. schrieb Alex Deucher:
Quoted 7 lines · Ramesh Errabolu
Quoted 4 lines
Current implementation will disallow P2P DMA if the participating devices belong to different root complexes. Implementation allows this default behavior to be overridden for whitelisted devices. The patch adds an AMD host bridge to be whitelisted
Why do we need this?
cpu_supports_p2pdma()should return true for all AMD Zen CPUs.This is part of our on-going work to get P2P support upstream. We want to use
pci_p2pdma_distance_manyto determine whether P2P is possible between a pair of devices. This whitelist is used in this function. This will affect the P2P links reported in the topology and it will be double-checked in the BO mapping function to ensure a peer mapping is legal.
Quoted 5 lines
I think this change is a bit free of context at the moment, as we're still working on a few other loose ends for P2P support in our internal branch. I'm hoping we'll have a bigger patch series for upstreamable KFD P2P support ready in a few weeks. I also think we'll probably want to add a few more PCI IDs for other supported AMD root complexes.
We don't need to keep adding AMD root complexes. You must be using an older kernel or something. All root complexes on all Zen platforms support P2P DMA. See:
commit dea286bb71baded7d2fb4f090e3b9fd2c1ccac58
Author: Logan Gunthorpe <[email protected]> Date: Wed Jul 29 17:18:44 2020 -0600
PCI/P2PDMA: Allow P2PDMA on AMD Zen and newer CPUs
Allow P2PDMA if the CPU vendor is AMD and family is 0x17 (Zen) or greater.
Alex
Quoted 29 lines
Regards,
FelixQuoted 24 lines
Alex
Quoted 21 lines
Signed-off-by Ramesh Errabolu <[email protected]>--- drivers/pci/p2pdma.c | 2 ++ 1 file changed, 2 insertions(+)drivers/pci/p2pdma.c index renamed from drivers/pci/p2pdma.c +0
-0196382630363..7003bb9faf23 100644
drivers/pci/p2pdma.c +2
-1@@ -305,6 +305,8 @@ static const struct pci_p2pdma_whitelist_entry { {PCI_VENDOR_ID_INTEL, 0x2032, 0}, {PCI_VENDOR_ID_INTEL, 0x2033, 0}, {PCI_VENDOR_ID_INTEL, 0x2020, 0}, +/* AMD Host Bridge Devices */ + {PCI_VENDOR_ID_AMD, 0x1480, 0}, {} }; --2.31.1