Skip to contentSkip to conversation

PCI/P2PDMA: Add AMD Zen 2 root complex to the list of allowed bridges

PATCH In review Logan Gunthorpe 4 participants 11 messages Jul 23, 2020 – Jul 27, 2020

Subsystems (2)
lore
Share & tools
lore

Logan Gunthorpe

author maintainer
#0 lore raw

PATCH 1/1 PCI/P2PDMA: Add AMD Zen 2 root complex to the list of allowed bridges

The AMD Zen 2 root complex (Starship/Matisse) was tested for P2PDMA transactions between root ports and found to work. Therefore add it to the list.

Cc Bjorn Helgaas <[email protected]>
Cc Christian König <[email protected]>
Cc Huang Rui <[email protected]>
Cc Alex Deucher <[email protected]>
---
 drivers/pci/p2pdma.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/pci/p2pdma.c +2 -0
index e8e444eeb1cd..3d67a1ee083e 100644
@@ -284,6 +284,8 @@ static const struct pci_p2pdma_whitelist_entry {
 	{PCI_VENDOR_ID_AMD,	0x1450,	0},
 	{PCI_VENDOR_ID_AMD,	0x15d0,	0},
 	{PCI_VENDOR_ID_AMD,	0x1630,	0},
+	/* AMD ZEN 2 */
+	{PCI_VENDOR_ID_AMD,	0x1480,	0},
 
 	/* Intel Xeon E5/Core i7 */
 	{PCI_VENDOR_ID_INTEL,	0x3c00, REQ_SAME_HOST_BRIDGE},

base-commit: ba47d845d715a010f7b51f6f89bae32845e6acb7

Signature · 1 line
2.20.1
Reviewed-by on v1 1/1
#1 lore raw
Quoted 10 lines · Logan Gunthorpe

The AMD Zen 2 root complex (Starship/Matisse) was tested for P2PDMA transactions between root ports and found to work. Therefore add it to the list.

Cc Bjorn Helgaas <[email protected]>
Cc Christian König <[email protected]>
Cc Huang Rui <[email protected]>
Cc Alex Deucher <[email protected]>

Starting with Zen, all AMD platforms support P2P for reads and writes.

Quoted 22 lines
---
 drivers/pci/p2pdma.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/pci/p2pdma.c +2 -0
index e8e444eeb1cd..3d67a1ee083e 100644
@@ -284,6 +284,8 @@ static const struct pci_p2pdma_whitelist_entry {
        {PCI_VENDOR_ID_AMD,     0x1450, 0},
        {PCI_VENDOR_ID_AMD,     0x15d0, 0},
        {PCI_VENDOR_ID_AMD,     0x1630, 0},
+       /* AMD ZEN 2 */
+       {PCI_VENDOR_ID_AMD,     0x1480, 0},
 
        /* Intel Xeon E5/Core i7 */
        {PCI_VENDOR_ID_INTEL,   0x3c00, REQ_SAME_HOST_BRIDGE},

base-commit: ba47d845d715a010f7b51f6f89bae32845e6acb7 --
2.20.1

#2 lore raw

[+cc Andrew, Armen, hpa]

Quoted 6 lines · Alex Deucher
Quoted 5 lines · Logan Gunthorpe

The AMD Zen 2 root complex (Starship/Matisse) was tested for P2PDMA transactions between root ports and found to work. Therefore add it to the list.

Quoted 7 lines
Quoted 5 lines
Cc Bjorn Helgaas <[email protected]>
Cc Christian König <[email protected]>
Cc Huang Rui <[email protected]>
Cc Alex Deucher <[email protected]>

Starting with Zen, all AMD platforms support P2P for reads and writes.

What's the plan for getting out of the cycle of "update this list for every new chip"? Any new _DSMs planned, for instance?

A continuous trickle of updates like this is not really appealing. So far we have:

7d5b10fcb81e ("PCI/P2PDMA: Add AMD Zen Raven and Renoir Root Ports to whitelist") 7b94b53db34f ("PCI/P2PDMA: Add Intel Sky Lake-E Root Ports B, C, D to the whitelist") bc123a515cb7 ("PCI/P2PDMA: Add Intel SkyLake-E to the whitelist") 494d63b0d5d0 ("PCI/P2PDMA: Whitelist some Intel host bridges") 0f97da831026 ("PCI/P2PDMA: Allow P2P DMA between any devices under AMD ZEN Root Complex")

And that's just from the last year, not including this patch.

Quoted 22 lines
Quoted 22 lines
---
 drivers/pci/p2pdma.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/pci/p2pdma.c +2 -0
index e8e444eeb1cd..3d67a1ee083e 100644
@@ -284,6 +284,8 @@ static const struct pci_p2pdma_whitelist_entry {
        {PCI_VENDOR_ID_AMD,     0x1450, 0},
        {PCI_VENDOR_ID_AMD,     0x15d0, 0},
        {PCI_VENDOR_ID_AMD,     0x1630, 0},
+       /* AMD ZEN 2 */
+       {PCI_VENDOR_ID_AMD,     0x1480, 0},
 
        /* Intel Xeon E5/Core i7 */
        {PCI_VENDOR_ID_INTEL,   0x3c00, REQ_SAME_HOST_BRIDGE},

base-commit: ba47d845d715a010f7b51f6f89bae32845e6acb7 --
2.20.1

author maintainer
#3 lore raw
Quoted 15 lines · Bjorn Helgaas

[+cc Andrew, Armen, hpa]

Quoted 12 lines · Alex Deucher
Quoted 10 lines · Logan Gunthorpe

The AMD Zen 2 root complex (Starship/Matisse) was tested for P2PDMA transactions between root ports and found to work. Therefore add it to the list.

Cc Bjorn Helgaas <[email protected]>
Cc Christian König <[email protected]>
Cc Huang Rui <[email protected]>
Cc Alex Deucher <[email protected]>
Quoted 4 lines
Quoted 1 line

Starting with Zen, all AMD platforms support P2P for reads and writes.

What's the plan for getting out of the cycle of "update this list for every new chip"? Any new _DSMs planned, for instance?

Well there was an effort to add capabilities in the PCI spec to describe this but, as far as I know, they never got anywhere, and hardware still doesn't self describe with this.

Quoted 10 lines

A continuous trickle of updates like this is not really appealing. So far we have:

7d5b10fcb81e ("PCI/P2PDMA: Add AMD Zen Raven and Renoir Root Ports to whitelist") 7b94b53db34f ("PCI/P2PDMA: Add Intel Sky Lake-E Root Ports B, C, D to the whitelist") bc123a515cb7 ("PCI/P2PDMA: Add Intel SkyLake-E to the whitelist") 494d63b0d5d0 ("PCI/P2PDMA: Whitelist some Intel host bridges") 0f97da831026 ("PCI/P2PDMA: Allow P2P DMA between any devices under AMD ZEN Root Complex")

And that's just from the last year, not including this patch.

Yes, it's not ideal. But most of these are adding old devices as people test and care about running on those platforms -- a lot of this is bootstrapping the list. I'd expect this to slow down a bit as by now we have hopefully got a lot of the existing platforms people care about. But we'd still probably expect to be adding a new Intel and AMD devices about once a year as they produce new hardware designs.

Unless, the Intel and AMD folks know of a way to detect this, or even to query if a root complex is newer than a certain generation, I'm not sure what else we can do here.

Logan

#4 lore raw
Quoted 46 lines · Logan Gunthorpe
Quoted 15 lines · Bjorn Helgaas

[+cc Andrew, Armen, hpa]

Quoted 12 lines · Alex Deucher
Quoted 10 lines · Logan Gunthorpe

The AMD Zen 2 root complex (Starship/Matisse) was tested for P2PDMA transactions between root ports and found to work. Therefore add it to the list.

Cc Bjorn Helgaas <[email protected]>
Cc Christian König <[email protected]>
Cc Huang Rui <[email protected]>
Cc Alex Deucher <[email protected]>
Quoted 4 lines
Quoted 1 line

Starting with Zen, all AMD platforms support P2P for reads and writes.

What's the plan for getting out of the cycle of "update this list for every new chip"? Any new _DSMs planned, for instance?

Well there was an effort to add capabilities in the PCI spec to describe this but, as far as I know, they never got anywhere, and hardware still doesn't self describe with this.

Quoted 10 lines

A continuous trickle of updates like this is not really appealing. So far we have:

7d5b10fcb81e ("PCI/P2PDMA: Add AMD Zen Raven and Renoir Root Ports to whitelist") 7b94b53db34f ("PCI/P2PDMA: Add Intel Sky Lake-E Root Ports B, C, D to the whitelist") bc123a515cb7 ("PCI/P2PDMA: Add Intel SkyLake-E to the whitelist") 494d63b0d5d0 ("PCI/P2PDMA: Whitelist some Intel host bridges") 0f97da831026 ("PCI/P2PDMA: Allow P2P DMA between any devices under AMD ZEN Root Complex")

And that's just from the last year, not including this patch.

Yes, it's not ideal. But most of these are adding old devices as people test and care about running on those platforms -- a lot of this is bootstrapping the list. I'd expect this to slow down a bit as by now we have hopefully got a lot of the existing platforms people care about. But we'd still probably expect to be adding a new Intel and AMD devices about once a year as they produce new hardware designs.

Quoted 3 lines

Unless, the Intel and AMD folks know of a way to detect this, or even to query if a root complex is newer than a certain generation, I'm not sure what else we can do here.

I started a thread internally to see if I can find a way. FWIW, pre-ZEN parts also support p2p DMA, but only for writes. If I can get a definitive list, maybe we could switch to a blacklist for the old ones?

Alex

Quoted 2 lines

Logan

author maintainer
#5 lore raw
Quoted 47 lines · Alex Deucher
Quoted 46 lines · Logan Gunthorpe
Quoted 15 lines · Bjorn Helgaas

[+cc Andrew, Armen, hpa]

Quoted 12 lines · Alex Deucher
Quoted 10 lines · Logan Gunthorpe

The AMD Zen 2 root complex (Starship/Matisse) was tested for P2PDMA transactions between root ports and found to work. Therefore add it to the list.

Cc Bjorn Helgaas <[email protected]>
Cc Christian König <[email protected]>
Cc Huang Rui <[email protected]>
Cc Alex Deucher <[email protected]>
Quoted 4 lines
Quoted 1 line

Starting with Zen, all AMD platforms support P2P for reads and writes.

What's the plan for getting out of the cycle of "update this list for every new chip"? Any new _DSMs planned, for instance?

Well there was an effort to add capabilities in the PCI spec to describe this but, as far as I know, they never got anywhere, and hardware still doesn't self describe with this.

Quoted 10 lines

A continuous trickle of updates like this is not really appealing. So far we have:

7d5b10fcb81e ("PCI/P2PDMA: Add AMD Zen Raven and Renoir Root Ports to whitelist") 7b94b53db34f ("PCI/P2PDMA: Add Intel Sky Lake-E Root Ports B, C, D to the whitelist") bc123a515cb7 ("PCI/P2PDMA: Add Intel SkyLake-E to the whitelist") 494d63b0d5d0 ("PCI/P2PDMA: Whitelist some Intel host bridges") 0f97da831026 ("PCI/P2PDMA: Allow P2P DMA between any devices under AMD ZEN Root Complex")

And that's just from the last year, not including this patch.

Yes, it's not ideal. But most of these are adding old devices as people test and care about running on those platforms -- a lot of this is bootstrapping the list. I'd expect this to slow down a bit as by now we have hopefully got a lot of the existing platforms people care about. But we'd still probably expect to be adding a new Intel and AMD devices about once a year as they produce new hardware designs.

Quoted 8 lines
Quoted 3 lines

Unless, the Intel and AMD folks know of a way to detect this, or even to query if a root complex is newer than a certain generation, I'm not sure what else we can do here.

I started a thread internally to see if I can find a way. FWIW, pre-ZEN parts also support p2p DMA, but only for writes. If I can get a definitive list, maybe we could switch to a blacklist for the old ones?

It would have to be an AMD specific list, falling back to the general whitelist.... I suppose we can also mine the pci_ids database for Intel root complexes and create a blacklist there too. But there are a lot more root complexes outside of the x86 world...

Logan

#6 lore raw
In reply to Alex Deucher
Quoted 48 lines · Alex Deucher
Quoted 46 lines · Logan Gunthorpe
Quoted 15 lines · Bjorn Helgaas

[+cc Andrew, Armen, hpa]

Quoted 12 lines · Alex Deucher
Quoted 10 lines · Logan Gunthorpe

The AMD Zen 2 root complex (Starship/Matisse) was tested for P2PDMA transactions between root ports and found to work. Therefore add it to the list.

Cc Bjorn Helgaas <[email protected]>
Cc Christian König <[email protected]>
Cc Huang Rui <[email protected]>
Cc Alex Deucher <[email protected]>
Quoted 4 lines
Quoted 1 line

Starting with Zen, all AMD platforms support P2P for reads and writes.

What's the plan for getting out of the cycle of "update this list for every new chip"? Any new _DSMs planned, for instance?

Well there was an effort to add capabilities in the PCI spec to describe this but, as far as I know, they never got anywhere, and hardware still doesn't self describe with this.

Quoted 10 lines

A continuous trickle of updates like this is not really appealing. So far we have:

7d5b10fcb81e ("PCI/P2PDMA: Add AMD Zen Raven and Renoir Root Ports to whitelist") 7b94b53db34f ("PCI/P2PDMA: Add Intel Sky Lake-E Root Ports B, C, D to the whitelist") bc123a515cb7 ("PCI/P2PDMA: Add Intel SkyLake-E to the whitelist") 494d63b0d5d0 ("PCI/P2PDMA: Whitelist some Intel host bridges") 0f97da831026 ("PCI/P2PDMA: Allow P2P DMA between any devices under AMD ZEN Root Complex")

And that's just from the last year, not including this patch.

Yes, it's not ideal. But most of these are adding old devices as people test and care about running on those platforms -- a lot of this is bootstrapping the list. I'd expect this to slow down a bit as by now we have hopefully got a lot of the existing platforms people care about. But we'd still probably expect to be adding a new Intel and AMD devices about once a year as they produce new hardware designs.

Quoted 8 lines
Quoted 3 lines

Unless, the Intel and AMD folks know of a way to detect this, or even to query if a root complex is newer than a certain generation, I'm not sure what else we can do here.

I started a thread internally to see if I can find a way. FWIW, pre-ZEN parts also support p2p DMA, but only for writes. If I can get a definitive list, maybe we could switch to a blacklist for the old ones?

After talking with a few people internally, for AMD chips, it would probably be easiest to just whitelist based on the CPU family id for zen and newer (e.g., >= 0x17).

Alex

Quoted 5 lines

Alex

Quoted 2 lines

Logan

author maintainer
#7 lore raw
Quoted 53 lines · Alex Deucher
Quoted 52 lines · Alex Deucher
Quoted 49 lines · Logan Gunthorpe
Quoted 15 lines · Bjorn Helgaas

[+cc Andrew, Armen, hpa]

Quoted 12 lines · Alex Deucher
Quoted 10 lines · Logan Gunthorpe

The AMD Zen 2 root complex (Starship/Matisse) was tested for P2PDMA transactions between root ports and found to work. Therefore add it to the list.

Cc Bjorn Helgaas <[email protected]>
Cc Christian König <[email protected]>
Cc Huang Rui <[email protected]>
Cc Alex Deucher <[email protected]>
Quoted 4 lines
Quoted 1 line

Starting with Zen, all AMD platforms support P2P for reads and writes.

What's the plan for getting out of the cycle of "update this list for every new chip"? Any new _DSMs planned, for instance?

Well there was an effort to add capabilities in the PCI spec to describe this but, as far as I know, they never got anywhere, and hardware still doesn't self describe with this.

Quoted 10 lines

A continuous trickle of updates like this is not really appealing. So far we have:

7d5b10fcb81e ("PCI/P2PDMA: Add AMD Zen Raven and Renoir Root Ports to whitelist") 7b94b53db34f ("PCI/P2PDMA: Add Intel Sky Lake-E Root Ports B, C, D to the whitelist") bc123a515cb7 ("PCI/P2PDMA: Add Intel SkyLake-E to the whitelist") 494d63b0d5d0 ("PCI/P2PDMA: Whitelist some Intel host bridges") 0f97da831026 ("PCI/P2PDMA: Allow P2P DMA between any devices under AMD ZEN Root Complex")

And that's just from the last year, not including this patch.

Yes, it's not ideal. But most of these are adding old devices as people test and care about running on those platforms -- a lot of this is bootstrapping the list. I'd expect this to slow down a bit as by now we have hopefully got a lot of the existing platforms people care about. But we'd still probably expect to be adding a new Intel and AMD devices about once a year as they produce new hardware designs.

Unless, the Intel and AMD folks know of a way to detect this, or even to query if a root complex is newer than a certain generation, I'm not sure what else we can do here.

Quoted 8 lines
Quoted 4 lines

I started a thread internally to see if I can find a way. FWIW, pre-ZEN parts also support p2p DMA, but only for writes. If I can get a definitive list, maybe we could switch to a blacklist for the old ones?

After talking with a few people internally, for AMD chips, it would probably be easiest to just whitelist based on the CPU family id for zen and newer (e.g., >= 0x17).

That seems sensible. I was trying to see if we could do something similar for Intel, and just allow anything after Skylake. I found this[1]. It seems they have been on family 6 for a long time, and I'm not comfortable enabling the whole family. Their model numbers also don't seem to increment in a favorable fashion, in that "small core" atoms (which have different host bridges with very much unknown support) have model numbers interspersed with regular "big core" CPUS.

So we might be stuck with the Intel white list for a while, but using the AMD family number will at least cut the number of additions down a fair amount. I can try to put a patch together in place of this one.

Logan

[1]
https://elixir.bootlin.com/linux/latest/source/arch/x86/include/asm/intel-family.h#L73

#8 lore raw
In reply to Logan Gunthorpe
Quoted 14 lines · Logan Gunthorpe
Quoted 13 lines · Bjorn Helgaas
Quoted 12 lines · Alex Deucher
Quoted 10 lines · Logan Gunthorpe

The AMD Zen 2 root complex (Starship/Matisse) was tested for P2PDMA transactions between root ports and found to work. Therefore add it to the list.

Cc Bjorn Helgaas <[email protected]>
Cc Christian König <[email protected]>
Cc Huang Rui <[email protected]>
Cc Alex Deucher <[email protected]>
Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Starting with Zen, all AMD platforms support P2P for reads and writes.

What's the plan for getting out of the cycle of "update this list for every new chip"? Any new _DSMs planned, for instance?

Well there was an effort to add capabilities in the PCI spec to describe this but, as far as I know, they never got anywhere, and hardware still doesn't self describe with this.

Any idea what happened? Is there hope for the future? I'm really not happy about signing up for open-ended device-specific patches like this. It's certainly not in the plug and play spirit that has made PCI successful. I know, preaching to the choir here.

Bjorn

author maintainer
#9 lore raw

[+cc Jonathan]

Quoted 20 lines · Bjorn Helgaas
Quoted 19 lines · Logan Gunthorpe
Quoted 17 lines · Bjorn Helgaas
Quoted 6 lines · Alex Deucher
Quoted 5 lines · Logan Gunthorpe

The AMD Zen 2 root complex (Starship/Matisse) was tested for P2PDMA transactions between root ports and found to work. Therefore add it to the list.

Quoted 7 lines
Quoted 5 lines
Cc Bjorn Helgaas <[email protected]>
Cc Christian König <[email protected]>
Cc Huang Rui <[email protected]>
Cc Alex Deucher <[email protected]>

Starting with Zen, all AMD platforms support P2P for reads and writes.

What's the plan for getting out of the cycle of "update this list for every new chip"? Any new _DSMs planned, for instance?

Quoted 8 lines
Quoted 3 lines

Well there was an effort to add capabilities in the PCI spec to describe this but, as far as I know, they never got anywhere, and hardware still doesn't self describe with this.

Any idea what happened? Is there hope for the future? I'm really not happy about signing up for open-ended device-specific patches like this. It's certainly not in the plug and play spirit that has made PCI successful. I know, preaching to the choir here.

Agreed, though I'm not really hooked into the PCI SIG. The last email I got about this was an RFC from Jonathan Cameron in late 2018. I've CC'd him here, maybe he'll have a bit more insight.

Logan

#10 lore raw
Quoted 27 lines · On Fri, 24 Jul 2020 09:56:39 -0600 Logan Gunthorpe <[email protected]> wrote:

[+cc Jonathan]

Quoted 24 lines · Bjorn Helgaas
Quoted 22 lines · Logan Gunthorpe
Quoted 13 lines · Bjorn Helgaas
Quoted 12 lines · Alex Deucher
Quoted 10 lines · Logan Gunthorpe

The AMD Zen 2 root complex (Starship/Matisse) was tested for P2PDMA transactions between root ports and found to work. Therefore add it to the list.

Cc Bjorn Helgaas <[email protected]>
Cc Christian König <[email protected]>
Cc Huang Rui <[email protected]>
Cc Alex Deucher <[email protected]>
Quoted 4 lines
Quoted 1 line

Starting with Zen, all AMD platforms support P2P for reads and writes.

What's the plan for getting out of the cycle of "update this list for every new chip"? Any new _DSMs planned, for instance?

Well there was an effort to add capabilities in the PCI spec to describe this but, as far as I know, they never got anywhere, and hardware still doesn't self describe with this.

Quoted 8 lines
Quoted 4 lines

Any idea what happened? Is there hope for the future? I'm really not happy about signing up for open-ended device-specific patches like this. It's certainly not in the plug and play spirit that has made PCI successful. I know, preaching to the choir here.

Agreed, though I'm not really hooked into the PCI SIG. The last email I got about this was an RFC from Jonathan Cameron in late 2018. I've CC'd him here, maybe he'll have a bit more insight.

For non technical reasons, you can probably figure out, that particular ECR stalled. Unfortunately I can't directly provide info on any newer discussions. Eric, could you perhaps find out if there is anything we can share?

This is the same question of trying to find a way to avoid white listing root complexes that can do peer 2 peer that would have been covered by your Advanced Peer to Peer Capabilities ECR.

Thanks,

Jonathan

Quoted 2 lines

Logan