Skip to contentSkip to conversation

PCI/P2PDMA: Root complex whitelist should not apply when an IOMMU is present

PATCH Superseded Logan Gunthorpe 4 participants 7 messages May 22, 2019 – Jun 19, 2019

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Logan Gunthorpe

author maintainer
#0 lore raw

PATCH 1/1 PCI/P2PDMA: Root complex whitelist should not apply when an IOMMU is present

Presently, there is no path to DMA map P2PDMA memory, so if a TLP targeting this memory hits the root complex and an IOMMU is present, the IOMMU will reject the transaction, even if the RC would support P2PDMA.

So until the kernel knows to map these DMA addresses in the IOMMU, we should not enable the whitelist when an IOMMU is present.

While we are at it, remove the comment mentioning future work to add a white list.

Fixes 0f97da831026 ("PCI/P2PDMA: Allow P2P DMA between any devices under AMD ZEN Root Complex")
Cc Christian König <[email protected]>
Cc Bjorn Helgaas <[email protected]>
---
 drivers/pci/p2pdma.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

Hey,

I realized recently that I missed this issue between the IOMMU and the whitelist when reviewing Christian's patch.

Unless there are any objections, I think this should be squashed with the commit marked in the Fixes tag (from pci-v5.2-changes).

Thanks,

Logan

drivers/pci/p2pdma.c +5 -2
index 742928d0053e..4d2f6a44cba3 100644
@@ -18,6 +18,7 @@
 #include <linux/percpu-refcount.h>
 #include <linux/random.h>
 #include <linux/seq_buf.h>
+#include <linux/iommu.h>
 
 struct pci_p2pdma {
 	struct percpu_ref devmap_ref;
@@ -284,6 +285,9 @@ static bool root_complex_whitelist(struct pci_dev *dev)
 	struct pci_dev *root = pci_get_slot(host->bus, PCI_DEVFN(0, 0));
 	unsigned short vendor, device;
 
+	if (iommu_present(dev->dev.bus))
+		return false;
+
 	if (!root)
 		return false;
 
@@ -453,8 +457,7 @@ static int upstream_bridge_distance_warn(struct pci_dev *provider,
  *
  * For now, "compatible" means the provider and the clients are all behind
  * the same PCI root port. This cuts out cases that may work but is safest
- * for the user. Future work can expand this to white-list root complexes that
- * can safely forward between each ports.
+ * for the user.
  */
 int pci_p2pdma_distance_many(struct pci_dev *provider, struct device **clients,
 			     int num_clients, bool verbose)

--
2.20.1

#1 lore raw
Quoted 10 lines · Logan Gunthorpe

Presently, there is no path to DMA map P2PDMA memory, so if a TLP targeting this memory hits the root complex and an IOMMU is present, the IOMMU will reject the transaction, even if the RC would support P2PDMA.

So until the kernel knows to map these DMA addresses in the IOMMU, we should not enable the whitelist when an IOMMU is present.

While we are at it, remove the comment mentioning future work to add a white list.

There was a lot of discussion about this. Did everybody come to a consensus about what should be done? Can you post a patch with reviewed-by if appropriate?

Quoted 54 lines
Fixes 0f97da831026 ("PCI/P2PDMA: Allow P2P DMA between any devices under AMD ZEN Root Complex")
Cc Christian König <[email protected]>
Cc Bjorn Helgaas <[email protected]>
---
 drivers/pci/p2pdma.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

Hey,

I realized recently that I missed this issue between the IOMMU and the whitelist when reviewing Christian's patch.

Unless there are any objections, I think this should be squashed with the commit marked in the Fixes tag (from pci-v5.2-changes).

Thanks,

Logan

drivers/pci/p2pdma.c +5 -3
index 742928d0053e..4d2f6a44cba3 100644
@@ -18,6 +18,7 @@
 #include <linux/percpu-refcount.h>
 #include <linux/random.h>
 #include <linux/seq_buf.h>
+#include <linux/iommu.h>
 
 struct pci_p2pdma {
 	struct percpu_ref devmap_ref;
@@ -284,6 +285,9 @@ static bool root_complex_whitelist(struct pci_dev *dev)
 	struct pci_dev *root = pci_get_slot(host->bus, PCI_DEVFN(0, 0));
 	unsigned short vendor, device;
 
+	if (iommu_present(dev->dev.bus))
+		return false;
+
 	if (!root)
 		return false;
 
@@ -453,8 +457,7 @@ static int upstream_bridge_distance_warn(struct pci_dev *provider,
  *
  * For now, "compatible" means the provider and the clients are all behind
  * the same PCI root port. This cuts out cases that may work but is safest
- * for the user. Future work can expand this to white-list root complexes that
- * can safely forward between each ports.
+ * for the user.
  */
 int pci_p2pdma_distance_many(struct pci_dev *provider, struct device **clients,
 			     int num_clients, bool verbose)
--

2.20.1

author maintainer
#2 lore raw
Quoted 9 lines · Bjorn Helgaas
Quoted 8 lines · Logan Gunthorpe

Presently, there is no path to DMA map P2PDMA memory, so if a TLP targeting this memory hits the root complex and an IOMMU is present, the IOMMU will reject the transaction, even if the RC would support P2PDMA.

So until the kernel knows to map these DMA addresses in the IOMMU, we should not enable the whitelist when an IOMMU is present.

Quoted 6 lines
Quoted 2 lines

While we are at it, remove the comment mentioning future work to add a white list.

There was a lot of discussion about this. Did everybody come to a consensus about what should be done? Can you post a patch with reviewed-by if appropriate?

I think we have consensus that it's broken and needs to be fixed for the short term. Preferably before 5.3. I'm not sure we have consensus on the proper fix.

The two easy things I can see to do is to either revert it or add the iommu_is_present() check that I did in the above patch.

@Christian, which do you prefer? I think I'd prefer the iommu_is_present() route as it maintains the information about white-listed devices and is easier to change once we have the correct solution.

I can send a patch tomorrow one way or another.

Thanks,

Logan

author maintainer
#3 lore raw
Quoted 24 lines · Logan Gunthorpe
Quoted 9 lines · Bjorn Helgaas
Quoted 8 lines · Logan Gunthorpe

Presently, there is no path to DMA map P2PDMA memory, so if a TLP targeting this memory hits the root complex and an IOMMU is present, the IOMMU will reject the transaction, even if the RC would support P2PDMA.

So until the kernel knows to map these DMA addresses in the IOMMU, we should not enable the whitelist when an IOMMU is present.

Quoted 6 lines
Quoted 2 lines

While we are at it, remove the comment mentioning future work to add a white list.

There was a lot of discussion about this. Did everybody come to a consensus about what should be done? Can you post a patch with reviewed-by if appropriate?

I think we have consensus that it's broken and needs to be fixed for the short term. Preferably before 5.3. I'm not sure we have consensus on the proper fix.

The two easy things I can see to do is to either revert it or add the iommu_is_present() check that I did in the above patch.

Quoted 6 lines

@Christian, which do you prefer? I think I'd prefer the iommu_is_present() route as it maintains the information about white-listed devices and is easier to change once we have the correct solution.

I can send a patch tomorrow one way or another.

Also, looks like one of my clients has an interest in seeing work like this happen. So I'll be writing some patches in the next couple weeks to do this properly. I'll try to send them to the lists early next cycle.

Logan

#4 lore raw

Am 19.06.19 um 01:50 schrieb Logan Gunthorpe:

Quoted 12 lines · Logan Gunthorpe
Quoted 10 lines · Bjorn Helgaas
Quoted 9 lines · Logan Gunthorpe
Quoted 8 lines

Presently, there is no path to DMA map P2PDMA memory, so if a TLP targeting this memory hits the root complex and an IOMMU is present, the IOMMU will reject the transaction, even if the RC would support P2PDMA.

So until the kernel knows to map these DMA addresses in the IOMMU, we should not enable the whitelist when an IOMMU is present.

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

While we are at it, remove the comment mentioning future work to add a white list.

There was a lot of discussion about this. Did everybody come to a consensus about what should be done? Can you post a patch with reviewed-by if appropriate?

I think we have consensus that it's broken and needs to be fixed for the short term. Preferably before 5.3.

Yeah, completely agree.

Quoted 10 lines
Quoted 10 lines

I'm not sure we have consensus on the
proper fix.

The two easy things I can see to do is to either revert it or add the iommu_is_present() check that I did in the above patch.

@Christian, which do you prefer? I think I'd prefer the iommu_is_present() route as it maintains the information about white-listed devices and is easier to change once we have the correct solution.

Your original iommu_is_prevent() patch sound like the best option so far.

If that hasn't changed feel free to add a Reviewed-by: Christian König <[email protected]> to that one.

Quoted 4 lines
Quoted 1 line

I can send a patch tomorrow one way or another.

Also, looks like one of my clients has an interest in seeing work like this happen. So I'll be writing some patches in the next couple weeks to do this properly. I'll try to send them to the lists early next cycle.

Oh, nice.

I was hoping to get my use case into 5.4 or 5.5, but we are still stuck with some of the DMA-buf related pieces.

Regards,
Christian.

Quoted 2 lines

Logan

#5 lore raw
Quoted 2 lines · Christian

I was hoping to get my use case into 5.4 or 5.5, but we are still stuck with some of the DMA-buf related pieces.

Can you make sure you have Logand and me, and maybe the iommu list Cced when you posted it? I'd like to make sure we all have a common understanding where we are moving with the APIs and use cases.

#6 lore raw

Am 19.06.19 um 11:29 schrieb Christoph Hellwig:

Quoted 6 lines · Christian
Quoted 2 lines

I was hoping to get my use case into 5.4 or 5.5, but we are still stuck with some of the DMA-buf related pieces.

Can you make sure you have Logand and me, and maybe the iommu list Cced when you posted it? I'd like to make sure we all have a common understanding where we are moving with the APIs and use cases.

Yeah, sure. I'm still not settled on APIs anyway, so feedback on this is highly welcome.

Regards,
Christian.