Skip to contentSkip to conversation

PCI/P2PDMA: Avoid returning a provider for non_mappable_bars

PATCH Superseded Matt Evans 5 participants 11 messages Apr 21 – Apr 26

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Matt Evans

author
#0 lore raw

PATCH 1/1 PCI/P2PDMA: Avoid returning a provider for non_mappable_bars

Extend pcim_p2pdma_provider()'s checks to exclude functions that have pdev->non_mappable_bars set.

Consumers such as VFIO were previously able to map these for access by the CPU or P2P. Update the comment on non_mappable_bars to show it refers to any access, not just userspace CPU access.

Fixes 372d6d1b8ae3c ("PCI/P2PDMA: Refactor to separate core P2P functionality from memory allocation")

---

This arises from Alex Williamson's suggestion to test non_mappable_bars when getting the provider, with discussion here:

https://lore.kernel.org/kvm/[email protected]/

The goal was to prevent a hole where VFIO could export DMABUFs for BARs marked non-mappable, and to fix for all users of the provider rather than just VFIO. Alex observed that non_mappable_bars should be taken to mean BARs weren't usable by the CPU _or_ peers and, considering that, its comment about userspace access wasn't quite right.

 drivers/pci/p2pdma.c | 3 ++-
 include/linux/pci.h  | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)
drivers/pci/p2pdma.c +2 -1
index 7c898542af8d..4a783413f466 100644
@@ -318,7 +318,8 @@ struct p2pdma_provider *pcim_p2pdma_provider(struct pci_dev *pdev, int bar)
 {
 	struct pci_p2pdma *p2p;
 
-	if (!(pci_resource_flags(pdev, bar) & IORESOURCE_MEM))
+	if (!(pci_resource_flags(pdev, bar) & IORESOURCE_MEM) ||
+	    pdev->non_mappable_bars)
 		return NULL;
 
 	p2p = rcu_dereference_protected(pdev->p2pdma, 1);
include/linux/pci.h +1 -1
index 2c4454583c11..1e6802017d6b 100644
@@ -508,7 +508,7 @@ struct pci_dev {
 	unsigned int	no_command_memory:1;	/* No PCI_COMMAND_MEMORY */
 	unsigned int	rom_bar_overlap:1;	/* ROM BAR disable broken */
 	unsigned int	rom_attr_enabled:1;	/* Display of ROM attribute enabled? */
-	unsigned int	non_mappable_bars:1;	/* BARs can't be mapped to user-space  */
+	unsigned int	non_mappable_bars:1;	/* BARs can't be mapped by CPU or peers */
 	pci_dev_flags_t dev_flags;
 	atomic_t	enable_cnt;	/* pci_enable_device has been called */
 
Signature · 1 line
2.47.3
maintainer Reviewed-by on v1 1/1
#1 lore raw
Quoted 9 lines · Matt Evans

Extend pcim_p2pdma_provider()'s checks to exclude functions that have pdev->non_mappable_bars set.

Consumers such as VFIO were previously able to map these for access by the CPU or P2P. Update the comment on non_mappable_bars to show it refers to any access, not just userspace CPU access.

Fixes 372d6d1b8ae3c ("PCI/P2PDMA: Refactor to separate core P2P functionality from memory allocation")

This makes sense to me, thanks.

#2 lore raw
In reply to Matt Evans
Quoted 8 lines · Matt Evans

Extend pcim_p2pdma_provider()'s checks to exclude functions that have pdev->non_mappable_bars set.

Consumers such as VFIO were previously able to map these for access by the CPU or P2P. Update the comment on non_mappable_bars to show it refers to any access, not just userspace CPU access.

Fixes 372d6d1b8ae3c ("PCI/P2PDMA: Refactor to separate core P2P functionality from memory allocation")

I don't object to the patch, but this Fixes line doesn't look correct. non_mappable_bars applies only to s390, which doesn't support p2p. That wasn't prevented before 372d6d1b8ae3c refactoring too.

Thanks

author
#3 lore raw

Hi Leon,

Quoted 9 lines · Leon Romanovsky
Quoted 7 lines · Matt Evans

Extend pcim_p2pdma_provider()'s checks to exclude functions that have pdev->non_mappable_bars set.

Consumers such as VFIO were previously able to map these for access by the CPU or P2P. Update the comment on non_mappable_bars to show it refers to any access, not just userspace CPU access.

Quoted 5 lines
Quoted 1 line
Fixes 372d6d1b8ae3c ("PCI/P2PDMA: Refactor to separate core P2P functionality from memory allocation")

I don't object to the patch, but this Fixes line doesn't look correct. non_mappable_bars applies only to s390, which doesn't support p2p. That wasn't prevented before 372d6d1b8ae3c refactoring too.

Thanks; I'd chosen that commit as it adds the function that the additional test is being added to.

As an example consumer, vfio_pci_core_get_dmabuf_phys() added in 5d74781ebc86 ("vfio/pci: Add dma-buf export support for MMIO regions") could do an export when it shouldn't. (And with my other mmap series, a CPU could then access it, not even P2P.) I suppose this patch could point to 5d74781ebc86 instead (which allows the naughty DMABUF export), but I couldn't prove VFIO DMABUF export was the only thing using pcim_p2pdma_provider() added in 372d6d1b8ae3c (especially as that symbol's exported).

It still seems most watertight to treat this as a fix for anything containing 372d6d1b8ae3c, but if you feel it's not right what would be a better one?

Also, in the other thread (linked above) Alex made the point that whilst _today_ non_mappable_bars is only set by one s390 device, he expects future quirks or reasons to exclude mappings would also use non_mappable_bars so we shouldn't consider it a s390-specific thing.

Cheers,

Matt

#4 lore raw
Quoted 12 lines · Matt Evans

Hi Leon,

Quoted 9 lines · Leon Romanovsky
Quoted 7 lines · Matt Evans

Extend pcim_p2pdma_provider()'s checks to exclude functions that have pdev->non_mappable_bars set.

Consumers such as VFIO were previously able to map these for access by the CPU or P2P. Update the comment on non_mappable_bars to show it refers to any access, not just userspace CPU access.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Fixes 372d6d1b8ae3c ("PCI/P2PDMA: Refactor to separate core P2P functionality from memory allocation")

I don't object to the patch, but this Fixes line doesn't look correct. non_mappable_bars applies only to s390, which doesn't support p2p. That wasn't prevented before 372d6d1b8ae3c refactoring too.

Thanks; I'd chosen that commit as it adds the function that the additional test is being added to.

Can I suggest slightly different solution?
If CONFIG_..._S390 is set, we will unset CONFIG_PCI_P2PDMA and pcim_p2pdma_provider() will be compiled to return NULL, without changing non_mappable_bars description.

Thanks

#5 lore raw
Quoted 22 lines · Leon Romanovsky
Quoted 12 lines · Matt Evans

Hi Leon,

Quoted 9 lines · Leon Romanovsky
Quoted 7 lines · Matt Evans

Extend pcim_p2pdma_provider()'s checks to exclude functions that have pdev->non_mappable_bars set.

Consumers such as VFIO were previously able to map these for access by the CPU or P2P. Update the comment on non_mappable_bars to show it refers to any access, not just userspace CPU access.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Fixes 372d6d1b8ae3c ("PCI/P2PDMA: Refactor to separate core P2P functionality from memory allocation")

I don't object to the patch, but this Fixes line doesn't look correct. non_mappable_bars applies only to s390, which doesn't support p2p. That wasn't prevented before 372d6d1b8ae3c refactoring too.

Thanks; I'd chosen that commit as it adds the function that the additional test is being added to.

Quoted 6 lines

Can I suggest slightly different solution?
If CONFIG_..._S390 is set, we will unset CONFIG_PCI_P2PDMA and pcim_p2pdma_provider() will be compiled to return NULL, without changing non_mappable_bars description.

Thanks

Hi Leon,

I'm not sure this is an ideal solution for the longer term. While ISM definitely can't do P2P and is more akin to an emulated PCI device in its actual implementation, other PCI devices on the platform such as ConnectX family NICs, NVMes and the IBM Spyre Accelerator are much more run off the mill.

So, I'd prefer not to create restrictions unnecessarily. Now there are other unusual properties like the fact that all OSs on s390 run virtualized with PCI functions passed-through by a machine level hypervisor. That and firmware also control root ports and PCIe switches. Still, I don't believe there is anything fundamentally preventing P2P to work in the future. While we could then take this restriction out again, I'd prefer not to introduce it in the first place unless we have to.

Thanks,
Niklas Schnelle

#6 lore raw
Quoted 30 lines · Niklas Schnelle
Quoted 22 lines · Leon Romanovsky
Quoted 12 lines · Matt Evans

Hi Leon,

Quoted 9 lines · Leon Romanovsky
Quoted 7 lines · Matt Evans

Extend pcim_p2pdma_provider()'s checks to exclude functions that have pdev->non_mappable_bars set.

Consumers such as VFIO were previously able to map these for access by the CPU or P2P. Update the comment on non_mappable_bars to show it refers to any access, not just userspace CPU access.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Fixes 372d6d1b8ae3c ("PCI/P2PDMA: Refactor to separate core P2P functionality from memory allocation")

I don't object to the patch, but this Fixes line doesn't look correct. non_mappable_bars applies only to s390, which doesn't support p2p. That wasn't prevented before 372d6d1b8ae3c refactoring too.

Thanks; I'd chosen that commit as it adds the function that the additional test is being added to.

Quoted 6 lines

Can I suggest slightly different solution?
If CONFIG_..._S390 is set, we will unset CONFIG_PCI_P2PDMA and pcim_p2pdma_provider() will be compiled to return NULL, without changing non_mappable_bars description.

Thanks

Quoted 7 lines

Hi Leon,

I'm not sure this is an ideal solution for the longer term. While ISM definitely can't do P2P and is more akin to an emulated PCI device in its actual implementation, other PCI devices on the platform such as ConnectX family NICs, NVMes and the IBM Spyre Accelerator are much more run off the mill.

Thanks, Niklas and Matt,

I hadn’t realized that the non_mappable_bars apply only to a specific portion of the system. You’re right — the change should be made as you suggested.

Thanks

author
#7 lore raw
In reply to Leon Romanovsky

Hi Leon,

Quoted 20 lines · Leon Romanovsky
Quoted 18 lines · Matt Evans

Hi Leon,

Quoted 14 lines · Leon Romanovsky
Quoted 8 lines · Matt Evans

Extend pcim_p2pdma_provider()'s checks to exclude functions that have pdev->non_mappable_bars set.

Consumers such as VFIO were previously able to map these for access by the CPU or P2P. Update the comment on non_mappable_bars to show it refers to any access, not just userspace CPU access.

Fixes 372d6d1b8ae3c ("PCI/P2PDMA: Refactor to separate core P2P functionality from memory allocation")

I don't object to the patch, but this Fixes line doesn't look correct. non_mappable_bars applies only to s390, which doesn't support p2p. That wasn't prevented before 372d6d1b8ae3c refactoring too.

Quoted 7 lines
Quoted 2 lines

Thanks; I'd chosen that commit as it adds the function that the additional test is being added to.

Can I suggest slightly different solution?
If CONFIG_..._S390 is set, we will unset CONFIG_PCI_P2PDMA and pcim_p2pdma_provider() will be compiled to return NULL, without changing non_mappable_bars description.

Where we got to in

https://lore.kernel.org/kvm/[email protected]/

was that non_mappable_bars could be used in future for non-s390 quirks, i.e. s390 is the current (but potentially not only) user. Secondly, it isn't a P2P-specific issue because in

https://lore.kernel.org/kvm/[email protected]/

I'm aiming to add mmap() support to VFIO DMABUFs, which (without a fix like this) would allow CPU access too.

Thanks,

Matt

Reviewed-by on v1 1/1
#8 lore raw
In reply to Matt Evans
Quoted 47 lines · Matt Evans

Extend pcim_p2pdma_provider()'s checks to exclude functions that have pdev->non_mappable_bars set.

Consumers such as VFIO were previously able to map these for access by the CPU or P2P. Update the comment on non_mappable_bars to show it refers to any access, not just userspace CPU access.

Fixes 372d6d1b8ae3c ("PCI/P2PDMA: Refactor to separate core P2P functionality from memory allocation")

---

This arises from Alex Williamson's suggestion to test non_mappable_bars when getting the provider, with discussion here:

https://lore.kernel.org/kvm/[email protected]/

The goal was to prevent a hole where VFIO could export DMABUFs for BARs marked non-mappable, and to fix for all users of the provider rather than just VFIO. Alex observed that non_mappable_bars should be taken to mean BARs weren't usable by the CPU _or_ peers and, considering that, its comment about userspace access wasn't quite right.

 drivers/pci/p2pdma.c | 3 ++-
 include/linux/pci.h  | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)
drivers/pci/p2pdma.c +2 -1
index 7c898542af8d..4a783413f466 100644
@@ -318,7 +318,8 @@ struct p2pdma_provider *pcim_p2pdma_provider(struct pci_dev *pdev, int bar)
 {
 	struct pci_p2pdma *p2p;
 
-	if (!(pci_resource_flags(pdev, bar) & IORESOURCE_MEM))
+	if (!(pci_resource_flags(pdev, bar) & IORESOURCE_MEM) ||
+	    pdev->non_mappable_bars)
 		return NULL;
 
 	p2p = rcu_dereference_protected(pdev->p2pdma, 1);
include/linux/pci.h +0 -0
index 2c4454583c11..1e6802017d6b 100644

include/linux/pci.h struct pci_dev { · line 514

Quoted 8 lines
include/linux/pci.h +1 -1
@@ -508,7 +508,7 @@ struct pci_dev {
 	unsigned int	no_command_memory:1;	/* No PCI_COMMAND_MEMORY */
 	unsigned int	rom_bar_overlap:1;	/* ROM BAR disable broken */
 	unsigned int	rom_attr_enabled:1;	/* Display of ROM attribute enabled? */
-	unsigned int	non_mappable_bars:1;	/* BARs can't be mapped to user-space  */
+	unsigned int	non_mappable_bars:1;	/* BARs can't be mapped by CPU or peers */
 	pci_dev_flags_t dev_flags;
 	atomic_t	enable_cnt;	/* pci_enable_device has been called */
 

Sorry for the late reply. For s390's ISM device this definitely makes sense. The BAR can indeed not be accessed by P2P and extending this seems reasonable to me.

Feel free to add my:

Thanks,
Niklas

#9 lore raw
In reply to Matt Evans
Quoted 47 lines · On Tue, 21 Apr 2026 10:43:51 -0700 Matt Evans <[email protected]> wrote:

Extend pcim_p2pdma_provider()'s checks to exclude functions that have pdev->non_mappable_bars set.

Consumers such as VFIO were previously able to map these for access by the CPU or P2P. Update the comment on non_mappable_bars to show it refers to any access, not just userspace CPU access.

Fixes 372d6d1b8ae3c ("PCI/P2PDMA: Refactor to separate core P2P functionality from memory allocation")

---

This arises from Alex Williamson's suggestion to test non_mappable_bars when getting the provider, with discussion here:

https://lore.kernel.org/kvm/[email protected]/

The goal was to prevent a hole where VFIO could export DMABUFs for BARs marked non-mappable, and to fix for all users of the provider rather than just VFIO. Alex observed that non_mappable_bars should be taken to mean BARs weren't usable by the CPU _or_ peers and, considering that, its comment about userspace access wasn't quite right.

 drivers/pci/p2pdma.c | 3 ++-
 include/linux/pci.h  | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)
drivers/pci/p2pdma.c +2 -1
index 7c898542af8d..4a783413f466 100644
@@ -318,7 +318,8 @@ struct p2pdma_provider *pcim_p2pdma_provider(struct pci_dev *pdev, int bar)
 {
 	struct pci_p2pdma *p2p;
 
-	if (!(pci_resource_flags(pdev, bar) & IORESOURCE_MEM))
+	if (!(pci_resource_flags(pdev, bar) & IORESOURCE_MEM) ||
+	    pdev->non_mappable_bars)
 		return NULL;
 
 	p2p = rcu_dereference_protected(pdev->p2pdma, 1);
include/linux/pci.h +0 -0
index 2c4454583c11..1e6802017d6b 100644

include/linux/pci.h struct pci_dev { · line 514

Quoted 8 lines
include/linux/pci.h +1 -1
@@ -508,7 +508,7 @@ struct pci_dev {
 	unsigned int	no_command_memory:1;	/* No PCI_COMMAND_MEMORY */
 	unsigned int	rom_bar_overlap:1;	/* ROM BAR disable broken */
 	unsigned int	rom_attr_enabled:1;	/* Display of ROM attribute enabled? */
-	unsigned int	non_mappable_bars:1;	/* BARs can't be mapped to user-space  */
+	unsigned int	non_mappable_bars:1;	/* BARs can't be mapped by CPU or peers */
 	pci_dev_flags_t dev_flags;
 	atomic_t	enable_cnt;	/* pci_enable_device has been called */
 

Should pcim_p2pdma_init() separately test pdev->non_mappable_bars before the rcu-deref/kzalloc of the pci_p2pdma object and return -EOPNOTSUPP?

That then invokes the same error paths we'd see if we simply don't have p2pdma in the kernel and handles the pci_p2pdma_add_resource() path automatically as well.

This pcim_p2pdma_provider() test is really then just suppressing the WARN_ON that we'd otherwise see by not finding the p2p object on the device. Thanks,

Alex

author
#10 lore raw

Hi Alex,

Quoted 64 lines · Alex Williamson
Quoted 47 lines · On Tue, 21 Apr 2026 10:43:51 -0700 Matt Evans <[email protected]> wrote:

Extend pcim_p2pdma_provider()'s checks to exclude functions that have pdev->non_mappable_bars set.

Consumers such as VFIO were previously able to map these for access by the CPU or P2P. Update the comment on non_mappable_bars to show it refers to any access, not just userspace CPU access.

Fixes 372d6d1b8ae3c ("PCI/P2PDMA: Refactor to separate core P2P functionality from memory allocation")

---

This arises from Alex Williamson's suggestion to test non_mappable_bars when getting the provider, with discussion here:

https://lore.kernel.org/kvm/[email protected]/

The goal was to prevent a hole where VFIO could export DMABUFs for BARs marked non-mappable, and to fix for all users of the provider rather than just VFIO. Alex observed that non_mappable_bars should be taken to mean BARs weren't usable by the CPU _or_ peers and, considering that, its comment about userspace access wasn't quite right.

  drivers/pci/p2pdma.c | 3 ++-
  include/linux/pci.h  | 2 +-
  2 files changed, 3 insertions(+), 2 deletions(-)
drivers/pci/p2pdma.c +2 -1
index 7c898542af8d..4a783413f466 100644
@@ -318,7 +318,8 @@ struct p2pdma_provider *pcim_p2pdma_provider(struct pci_dev *pdev, int bar)
  {
  	struct pci_p2pdma *p2p;
 
-	if (!(pci_resource_flags(pdev, bar) & IORESOURCE_MEM))
+	if (!(pci_resource_flags(pdev, bar) & IORESOURCE_MEM) ||
+	    pdev->non_mappable_bars)
  		return NULL;
 
  	p2p = rcu_dereference_protected(pdev->p2pdma, 1);
include/linux/pci.h +0 -0
index 2c4454583c11..1e6802017d6b 100644

include/linux/pci.h struct pci_dev { · line 514

Quoted 8 lines
include/linux/pci.h +1 -1
@@ -508,7 +508,7 @@ struct pci_dev {
  	unsigned int	no_command_memory:1;	/* No PCI_COMMAND_MEMORY */
  	unsigned int	rom_bar_overlap:1;	/* ROM BAR disable broken */
  	unsigned int	rom_attr_enabled:1;	/* Display of ROM attribute enabled? */
-	unsigned int	non_mappable_bars:1;	/* BARs can't be mapped to user-space  */
+	unsigned int	non_mappable_bars:1;	/* BARs can't be mapped by CPU or peers */
  	pci_dev_flags_t dev_flags;
  	atomic_t	enable_cnt;	/* pci_enable_device has been called */
 

Should pcim_p2pdma_init() separately test pdev->non_mappable_bars before the rcu-deref/kzalloc of the pci_p2pdma object and return -EOPNOTSUPP?

Quoted 7 lines

That then invokes the same error paths we'd see if we simply don't have p2pdma in the kernel and handles the pci_p2pdma_add_resource() path automatically as well.

This pcim_p2pdma_provider() test is really then just suppressing the WARN_ON that we'd otherwise see by not finding the p2p object on the device. Thanks,

That makes sense, yes, it'll avoid the WARN that'd otherwise occur in pci_p2pdma_add_resource().

I'll do a v2 adding this check to pcim_p2pdma_init() as well. Hopefully Logan & Niklas's R-B will still apply.

Thanks,

Matt