Skip to contentSkip to conversation

PCI/P2PDMA: Root complex whitelist should not apply when an IOMMU is present

PATCH Logan Gunthorpe 3 participants 16 messages May 22, 2019 – May 24, 2019 1 message not in the archive PCI

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 11 lines · Christian

Hi Logan,

Am 22.05.2019 22:12 schrieb Logan Gunthorpe <[email protected]>:

[CAUTION: External Email]

Presently, there is no path to DMA map P2PDMA memory, so if a TLP targeting this memory hits the root complex and an IOMMU is present, the IOMMU will reject the transaction, even if the RC would support P2PDMA.

Quoted 5 lines

So until the kernel knows to map these DMA addresses in the IOMMU, we should not enable the whitelist when an IOMMU is present.

Well NAK, cause that is exactly what we are doing.

Are you DMA-mapping the addresses outside the P2PDMA code? If so there's a huge mismatch with the existing users of P2PDMA (nvme-fabrics). If you're not dma-mapping then I can't see how it could work because the IOMMU should reject any requests to access those addresses.

By adding the whitelist in this way you will break any user that attempts to use P2P in nvme-fabrics on whitelisted RCs with an IOMMU enabled.

Currently, the users of P2PDMA use pci_p2pdma_map_sg() which only returns the PCI bus address. If P2PDMA transactions can now go through an IOMMU, then this is wrong and broken.

We need to ensure that all users of P2PDMA map this memory in the same way. Which means, if the TLPs will go through an IOMMU they get dma-map'd and, if they don't, they use the PCI Bus address (as the current code does).

Without the change proposed in this patch, I have to retract my review and NAK your patch until we can sort out the mapping issues.

Logan

#2 lore raw

Am 22.05.19 um 22:41 schrieb Logan Gunthorpe:

Quoted 18 lines

[CAUTION: External Email]

Quoted 15 lines · Christian

Hi Logan,

Am 22.05.2019 22:12 schrieb Logan Gunthorpe <[email protected]>:

    [CAUTION: External Email]

    Presently, there is no path to DMA map P2PDMA memory, so if a TLP targeting this memory hits the root complex and an IOMMU is present, the IOMMU will reject the transaction, even if the RC would support P2PDMA.

    So until the kernel knows to map these DMA addresses in the IOMMU, we should not enable the whitelist when an IOMMU is present.

Quoted 6 lines
Quoted 1 line

Well NAK, cause that is exactly what we are doing.

Are you DMA-mapping the addresses outside the P2PDMA code? If so there's a huge mismatch with the existing users of P2PDMA (nvme-fabrics). If you're not dma-mapping then I can't see how it could work because the IOMMU should reject any requests to access those addresses.

Well, we are using the DMA API (dma_map_resource) for this. If the P2P code is not using this then I would rather say that the P2P code is actually broken.

Adding Christoph as well, cause he is usually the one discussion stuff like that with me.

Quoted 4 lines

By adding the whitelist in this way you will break any user that attempts to use P2P in nvme-fabrics on whitelisted RCs with an IOMMU enabled.

Quoted 8 lines

Currently, the users of P2PDMA use pci_p2pdma_map_sg() which only returns the PCI bus address. If P2PDMA transactions can now go through an IOMMU, then this is wrong and broken.

We need to ensure that all users of P2PDMA map this memory in the same way. Which means, if the TLPs will go through an IOMMU they get dma-map'd and, if they don't, they use the PCI Bus address (as the current code does).

Well that is exactly what dma_map_resource() already does, so we should probably just make using the DMA API mandatory.

Quoted 2 lines

Without the change proposed in this patch, I have to retract my review and NAK your patch until we can sort out the mapping issues.

Yeah, completely agree we can't do that right now.

Regards,
Christian.

Quoted 3 lines

Logan

#3 lore raw
Quoted 11 lines · Christian
Quoted 4 lines

Are you DMA-mapping the addresses outside the P2PDMA code? If so there's a huge mismatch with the existing users of P2PDMA (nvme-fabrics). If you're not dma-mapping then I can't see how it could work because the IOMMU should reject any requests to access those addresses.

Well, we are using the DMA API (dma_map_resource) for this. If the P2P code is not using this then I would rather say that the P2P code is actually broken.

Adding Christoph as well, cause he is usually the one discussion stuff like that with me.

Heh. Actually dma_map_resource-ish APIs are the right thing to do, but I'm not sure how you managed to be able to use it for PCIe P2P yet, as it fails to account for any difference in the PCIe level "physical" address with the hosts view of "physical" addresses.

Do these offsets now how up on AMD platforms? Do you adjust for them elsewhere?

#4 lore raw

Am 23.05.19 um 11:43 schrieb Christoph Hellwig:

Quoted 13 lines

[CAUTION: External Email]

Quoted 10 lines · Christian
Quoted 4 lines

Are you DMA-mapping the addresses outside the P2PDMA code? If so there's a huge mismatch with the existing users of P2PDMA (nvme-fabrics). If you're not dma-mapping then I can't see how it could work because the IOMMU should reject any requests to access those addresses.

Well, we are using the DMA API (dma_map_resource) for this. If the P2P code is not using this then I would rather say that the P2P code is actually broken.

Adding Christoph as well, cause he is usually the one discussion stuff like that with me.

Quoted 7 lines

Heh. Actually dma_map_resource-ish APIs are the right thing to do, but I'm not sure how you managed to be able to use it for PCIe P2P yet, as it fails to account for any difference in the PCIe level "physical" address with the hosts view of "physical" addresses.

Do these offsets now how up on AMD platforms? Do you adjust for them elsewhere?

I don't adjust the address manually anywhere. I just call dma_map_resource() and use the resulting DMA address to access the other devices PCI BAR.

At least on my test system (AMD CPU + AMD GPUs) this seems to work totally fine. Currently trying to find time and an Intel box to test it there as well.

Regards,
Christian.

#5 lore raw
Quoted 7 lines · Christian

I don't adjust the address manually anywhere. I just call dma_map_resource() and use the resulting DMA address to access the other devices PCI BAR.

At least on my test system (AMD CPU + AMD GPUs) this seems to work totally fine. Currently trying to find time and an Intel box to test it there as well.

The problem shows up if pci_bus_address() returns a different address than pci_resource_start(), should be easy to check if that happens. IIRC it is something mostly seen on embedded SOCs.

#6 lore raw

Am 23.05.19 um 11:50 schrieb Christoph Hellwig:

Quoted 7 lines

[CAUTION: External Email]

Quoted 4 lines · Christian

I don't adjust the address manually anywhere. I just call dma_map_resource() and use the resulting DMA address to access the other devices PCI BAR.

Quoted 6 lines
Quoted 3 lines

At least on my test system (AMD CPU + AMD GPUs) this seems to work totally fine. Currently trying to find time and an Intel box to test it there as well.

The problem shows up if pci_bus_address() returns a different address than pci_resource_start(), should be easy to check if that happens. IIRC it is something mostly seen on embedded SOCs.

Ok, we certainly don't have a system which exercise this user case. Could ask around if we have an ARM SOC with that properties somewhere.

But asking the other way around: Where is the right place to start fixing all this? dma_map_resource()?

Christian.

#7 lore raw
Quoted 5 lines · Christian

Ok, we certainly don't have a system which exercise this user case. Could ask around if we have an ARM SOC with that properties somewhere.

But asking the other way around: Where is the right place to start fixing all this? dma_map_resource()?

That is the the big gorrilla in the room. The offset applies to the device whos BARs/resources we map. The current dma_map_resource API does not even have the right information. So I think we need to enhance the API to pass a second struct device and we could fix it there and then in the next steps add a map_sg version of dma_map_resource and eventually also convert the PCIe P2P map_sg over to that.

#8 lore raw
Quoted 7 lines · Christoph Hellwig
Quoted 5 lines · Christian

Ok, we certainly don't have a system which exercise this user case. Could ask around if we have an ARM SOC with that properties somewhere.

But asking the other way around: Where is the right place to start fixing all this? dma_map_resource()?

Quoted 7 lines

That is the the big gorrilla in the room. The offset applies to the device whos BARs/resources we map. The current dma_map_resource API does not even have the right information. So I think we need to enhance the API to pass a second struct device and we could fix it there and then in the next steps add a map_sg version of dma_map_resource and eventually also convert the PCIe P2P map_sg over to that.

IMO, this logic belongs in pci_p2pdma_map_* helpers that call dma_map_* helpers when appropriate. Changing dma_map_resource() to take two devices won't always make sense. For example, there are existing use cases of dma_map_resource() that work with the Intel IOAT DMA engine and thus it's known that those transactions will always go through the IOMMU if it's enabled; and therefore the existing dma_map_resource() is appropriate.

Logan

#9 lore raw
In reply to Christoph Hellwig
Quoted 5 lines · Christoph Hellwig
Quoted 4 lines · Christian

I don't adjust the address manually anywhere. I just call dma_map_resource() and use the resulting DMA address to access the other devices PCI BAR.

Quoted 8 lines
Quoted 3 lines

At least on my test system (AMD CPU + AMD GPUs) this seems to work totally fine. Currently trying to find time and an Intel box to test it there as well.

The problem shows up if pci_bus_address() returns a different address than pci_resource_start(), should be easy to check if that happens. IIRC it is something mostly seen on embedded SOCs.

I think it's a bit more complicated then that: If you're calling dma_map_resource() to program the IOMMU then I'm pretty sure you'd want to use the pci_resource_start() address as the phys_addr_t. If you're bypassing the root complex (like the current p2pdma code enforces), then you'd simply use a pci_bus_address() directly as the dma_addr and would not program the IOMMU at all seeing it's not involved (which is what is currently done).

Logan

#10 lore raw
Quoted 5 lines · Logan Gunthorpe
Quoted 4 lines

The problem shows up if pci_bus_address() returns a different address than pci_resource_start(), should be easy to check if that happens. IIRC it is something mostly seen on embedded SOCs.

Quoted 7 lines

I think it's a bit more complicated then that: If you're calling dma_map_resource() to program the IOMMU then I'm pretty sure you'd want to use the pci_resource_start() address as the phys_addr_t. If you're bypassing the root complex (like the current p2pdma code enforces), then you'd simply use a pci_bus_address() directly as the dma_addr and would not program the IOMMU at all seeing it's not involved (which is what is currently done).

True. What we need is:

if (both device are behind the same root port (using a switch)) {
use the current direct map + offset code
} else {
call ->map_resource()
}

#11 lore raw

Am 23.05.19 um 17:59 schrieb Christoph Hellwig:

Quoted 14 lines

[CAUTION: External Email]

Quoted 11 lines · Logan Gunthorpe
Quoted 4 lines

The problem shows up if pci_bus_address() returns a different address than pci_resource_start(), should be easy to check if that happens. IIRC it is something mostly seen on embedded SOCs.

I think it's a bit more complicated then that: If you're calling dma_map_resource() to program the IOMMU then I'm pretty sure you'd want to use the pci_resource_start() address as the phys_addr_t. If you're bypassing the root complex (like the current p2pdma code enforces), then you'd simply use a pci_bus_address() directly as the dma_addr and would not program the IOMMU at all seeing it's not involved (which is what is currently done).

Quoted 7 lines

True. What we need is:

if (both device are behind the same root port (using a switch)) {
use the current direct map + offset code

  } else {
         call ->map_resource()

}

Sounds sane to me as well.

But since I don't have a struct pages backing my PCI BAR I won't be able to use pci_p2pdma_map_sg.

How should we work around that?

Regards,
Christian.

#12 lore raw
Quoted 6 lines · Christian

Sounds sane to me as well.

But since I don't have a struct pages backing my PCI BAR I won't be able to use pci_p2pdma_map_sg.

How should we work around that?

I think we've got two escalation levels here:

(1) fix the NVMe/RDMA p2p breakage for 5.2-rc - for that we just need the above fix in the p2p map_sg routine (and add an unmap_sg routine there)
(2) figure out how to handle the switched case for your code. I'm not sure about that yet, and without a struct page we'd have to somehow track the relationship explicitly.

#13 lore raw
Quoted 5 lines · Christoph Hellwig

I think we've got two escalation levels here:

(1) fix the NVMe/RDMA p2p breakage for 5.2-rc - for that we just need the above fix in the p2p map_sg routine (and add an unmap_sg routine there)

Yes, fixing the p2p_map_sg routine or add the check if the iommu is present (as I did in the patch that started this discussion) until we can get the map stuff sorted out.

Logan

#14 lore raw
In reply to Koenig, Christian
Quoted 23 lines · Christian

Am 23.05.19 um 17:59 schrieb Christoph Hellwig:

Quoted 14 lines

[CAUTION: External Email]

Quoted 11 lines · Logan Gunthorpe
Quoted 4 lines

The problem shows up if pci_bus_address() returns a different address than pci_resource_start(), should be easy to check if that happens. IIRC it is something mostly seen on embedded SOCs.

I think it's a bit more complicated then that: If you're calling dma_map_resource() to program the IOMMU then I'm pretty sure you'd want to use the pci_resource_start() address as the phys_addr_t. If you're bypassing the root complex (like the current p2pdma code enforces), then you'd simply use a pci_bus_address() directly as the dma_addr and would not program the IOMMU at all seeing it's not involved (which is what is currently done).

Quoted 7 lines

True. What we need is:

if (both device are behind the same root port (using a switch)) {
use the current direct map + offset code

  } else {
         call ->map_resource()

}

Quoted 6 lines

Sounds sane to me as well.

But since I don't have a struct pages backing my PCI BAR I won't be able to use pci_p2pdma_map_sg.

How should we work around that?

I'd add something like:

int pci_p2pdma_map_resource(struct pci_dev *provider, struct device *mapper, phys_addr_t addr, size_t size, enum dma_data_direction dir, unsigned long attrs);

But keep in mind we have to update pci_p2pdma_map_sg() as well before we can enable the whitelist as there are existing users. The map_sg() command should be able to get the provider device through the pgmap. And that will be easier to do once Dan's patch[1] lands because he creates a private pgmap struct.

Logan

[1]
https://lore.kernel.org/lkml/155727338646.292046.9922678317501435597.stgit@dwillia2-desk3.amr.corp.intel.com/T/#u

#15 lore raw
In reply to Koenig, Christian
Quoted 22 lines · Christian

Am 23.05.19 um 11:43 schrieb Christoph Hellwig:

Quoted 13 lines

[CAUTION: External Email]

Quoted 10 lines · Christian
Quoted 4 lines

Are you DMA-mapping the addresses outside the P2PDMA code? If so there's a huge mismatch with the existing users of P2PDMA (nvme-fabrics). If you're not dma-mapping then I can't see how it could work because the IOMMU should reject any requests to access those addresses.

Well, we are using the DMA API (dma_map_resource) for this. If the P2P code is not using this then I would rather say that the P2P code is actually broken.

Adding Christoph as well, cause he is usually the one discussion stuff like that with me.

Quoted 7 lines

Heh. Actually dma_map_resource-ish APIs are the right thing to do, but I'm not sure how you managed to be able to use it for PCIe P2P yet, as it fails to account for any difference in the PCIe level "physical" address with the hosts view of "physical" addresses.

Do these offsets now how up on AMD platforms? Do you adjust for them elsewhere?

Quoted 7 lines

I don't adjust the address manually anywhere. I just call dma_map_resource() and use the resulting DMA address to access the other devices PCI BAR.

At least on my test system (AMD CPU + AMD GPUs) this seems to work totally fine. Currently trying to find time and an Intel box to test it there as well.

I'm sure this will work fine in all cases (assuming the RC/IOMMU supports P2P). It's just you're breaking the existing p2pdma users which currently work by avoiding the IOMMU and use the pci_bus_address() instead.

Logan

#16 lore raw
In reply to Koenig, Christian
Quoted 8 lines · Christian
Quoted 4 lines

Are you DMA-mapping the addresses outside the P2PDMA code? If so there's a huge mismatch with the existing users of P2PDMA (nvme-fabrics). If you're not dma-mapping then I can't see how it could work because the IOMMU should reject any requests to access those addresses.

Well, we are using the DMA API (dma_map_resource) for this. If the P2P code is not using this then I would rather say that the P2P code is actually broken.

No, the P2P code is doing what it was designed to do: peer-to-peer without going through the root complex. If you DMA map in the presence of an IOMMU, then you force the TLPs to go through the root complex. This is not what we want and will not be supported by some RCs so you effectively drop support for everything but the one entry you've added in the white list. If we're going to start allowing transactions to go through an IOMMU then the existing users of P2PDMA needs to be updated to support this without dropping support for existing use cases.

Quoted 8 lines
Quoted 8 lines

By adding the whitelist in this way you will break any user that attempts to use P2P in nvme-fabrics on whitelisted RCs with an IOMMU enabled.

Currently, the users of P2PDMA use pci_p2pdma_map_sg() which only returns the PCI bus address. If P2PDMA transactions can now go through an IOMMU, then this is wrong and broken.

Quoted 7 lines
Quoted 4 lines

We need to ensure that all users of P2PDMA map this memory in the same way. Which means, if the TLPs will go through an IOMMU they get dma-map'd and, if they don't, they use the PCI Bus address (as the current code does).

Well that is exactly what dma_map_resource() already does, so we should probably just make using the DMA API mandatory.

No, it's absolutely not what dma_map_resource() does. DMA map resource only adds IOVA addresses in the IOMMU for a physical address (if an IOMMU is present). It does not determine whether that's appropriate to do for a given transaction. Such logic would belong in pci_p2pdma_map_* helpers.

Quoted 4 lines
Quoted 2 lines

Without the change proposed in this patch, I have to retract my review and NAK your patch until we can sort out the mapping issues.

Yeah, completely agree we can't do that right now.

Why not? It's not that complicated. We just need a way for pci_p2pdma_map_* to determine if the transfer path involves the IOMMU. If it does, call an existing dma_map_* function, if it does not then do what it currently does and use the PCI bus address. This probably means creating some kind of cache with flags for the distance between devices.

Logan