PCI/P2PDMA: Avoid returning a provider for non_mappable_bars
Subsystems (2)
PATCH 1/1 PCI/P2PDMA: Avoid returning a provider for non_mappable_bars
Extend pcim_p2pdma_provider()'s checks to exclude functions that have pdev->non_mappable_bars set.
Consumers such as VFIO were previously able to map these for access by the CPU or P2P. Update the comment on non_mappable_bars to show it refers to any access, not just userspace CPU access.
---
This arises from Alex Williamson's suggestion to test non_mappable_bars when getting the provider, with discussion here:
https://lore.kernel.org/kvm/[email protected]/
The goal was to prevent a hole where VFIO could export DMABUFs for BARs marked non-mappable, and to fix for all users of the provider rather than just VFIO. Alex observed that non_mappable_bars should be taken to mean BARs weren't usable by the CPU _or_ peers and, considering that, its comment about userspace access wasn't quite right.
drivers/pci/p2pdma.c | 3 ++- include/linux/pci.h | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)
drivers/pci/p2pdma.c +2 -1
@@ -318,7 +318,8 @@ struct p2pdma_provider *pcim_p2pdma_provider(struct pci_dev *pdev, int bar) {struct pci_p2pdma *p2p; -if (!(pci_resource_flags (pdev, bar) &IORESOURCE_MEM )) +if (!(pci_resource_flags (pdev, bar) &IORESOURCE_MEM ) || + pdev->non_mappable_bars)return NULL ; p2p =rcu_dereference_protected (pdev->p2pdma,1 );
include/linux/pci.h +1 -1
@@ -508,7 +508,7 @@ struct pci_dev { unsigned int no_command_memory:1; /* No PCI_COMMAND_MEMORY */ unsigned int rom_bar_overlap:1; /* ROM BAR disable broken */ unsigned int rom_attr_enabled:1; /* Display of ROM attribute enabled? */ - unsigned int non_mappable_bars:1; /* BARs can't be mapped to user-space */ + unsigned int non_mappable_bars:1; /* BARs can't be mapped by CPU or peers */pci_dev_flags_t dev_flags;atomic_t enable_cnt;/* pci_enable_device has been called */
Signature · 1 line
2.47.3
Quoted 9 lines · Matt Evans
Extend pcim_p2pdma_provider()'s checks to exclude functions that have
pdev->non_mappable_barsset.Consumers such as VFIO were previously able to map these for access by the CPU or P2P. Update the comment on
non_mappable_barsto show it refers to any access, not just userspace CPU access.Fixes 372d6d1b8ae3c ("PCI/P2PDMA: Refactor to separate core P2P functionality from memory allocation")Signed-off-by Matt Evans <[email protected]>
This makes sense to me, thanks.
Quoted 8 lines · Matt Evans
Extend pcim_p2pdma_provider()'s checks to exclude functions that have
pdev->non_mappable_barsset.Consumers such as VFIO were previously able to map these for access by the CPU or P2P. Update the comment on
non_mappable_barsto show it refers to any access, not just userspace CPU access.Fixes 372d6d1b8ae3c ("PCI/P2PDMA: Refactor to separate core P2P functionality from memory allocation")
I don't object to the patch, but this Fixes line doesn't look correct. non_mappable_bars applies only to s390, which doesn't support p2p. That wasn't prevented before 372d6d1b8ae3c refactoring too.
Thanks
Hi Leon,
Quoted 9 lines · Leon Romanovsky
Quoted 7 lines · Matt Evans
Extend pcim_p2pdma_provider()'s checks to exclude functions that have
pdev->non_mappable_barsset.Consumers such as VFIO were previously able to map these for access by the CPU or P2P. Update the comment on
non_mappable_barsto show it refers to any access, not just userspace CPU access.
Quoted 5 lines
Quoted 1 line
Fixes 372d6d1b8ae3c ("PCI/P2PDMA: Refactor to separate core P2P functionality from memory allocation")I don't object to the patch, but this Fixes line doesn't look correct.
non_mappable_barsapplies only to s390, which doesn't support p2p. That wasn't prevented before372d6d1b8ae3crefactoring too.
Thanks; I'd chosen that commit as it adds the function that the additional test is being added to.
As an example consumer, vfio_pci_core_get_dmabuf_phys() added in 5d74781ebc86 ("vfio/pci: Add dma-buf export support for MMIO regions") could do an export when it shouldn't. (And with my other mmap series, a CPU could then access it, not even P2P.) I suppose this patch could point to 5d74781ebc86 instead (which allows the naughty DMABUF export), but I couldn't prove VFIO DMABUF export was the only thing using pcim_p2pdma_provider() added in 372d6d1b8ae3c (especially as that symbol's exported).
It still seems most watertight to treat this as a fix for anything containing 372d6d1b8ae3c, but if you feel it's not right what would be a better one?
Also, in the other thread (linked above) Alex made the point that whilst _today_ non_mappable_bars is only set by one s390 device, he expects future quirks or reasons to exclude mappings would also use non_mappable_bars so we shouldn't consider it a s390-specific thing.
Cheers,
Matt
Quoted 12 lines · Matt Evans
Hi Leon,
Quoted 9 lines · Leon Romanovsky
Quoted 7 lines · Matt Evans
Extend pcim_p2pdma_provider()'s checks to exclude functions that have
pdev->non_mappable_barsset.Consumers such as VFIO were previously able to map these for access by the CPU or P2P. Update the comment on
non_mappable_barsto show it refers to any access, not just userspace CPU access.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Fixes 372d6d1b8ae3c ("PCI/P2PDMA: Refactor to separate core P2P functionality from memory allocation")I don't object to the patch, but this Fixes line doesn't look correct.
non_mappable_barsapplies only to s390, which doesn't support p2p. That wasn't prevented before372d6d1b8ae3crefactoring too.Thanks; I'd chosen that commit as it adds the function that the additional test is being added to.
Can I suggest slightly different solution?
If CONFIG_..._S390 is set, we will unset CONFIG_PCI_P2PDMA and pcim_p2pdma_provider() will be compiled to return NULL, without changing non_mappable_bars description.
Thanks
Quoted 22 lines · Leon Romanovsky
Quoted 12 lines · Matt Evans
Hi Leon,
Quoted 9 lines · Leon Romanovsky
Quoted 7 lines · Matt Evans
Extend pcim_p2pdma_provider()'s checks to exclude functions that have
pdev->non_mappable_barsset.Consumers such as VFIO were previously able to map these for access by the CPU or P2P. Update the comment on
non_mappable_barsto show it refers to any access, not just userspace CPU access.Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Fixes 372d6d1b8ae3c ("PCI/P2PDMA: Refactor to separate core P2P functionality from memory allocation")I don't object to the patch, but this Fixes line doesn't look correct.
non_mappable_barsapplies only to s390, which doesn't support p2p. That wasn't prevented before372d6d1b8ae3crefactoring too.Thanks; I'd chosen that commit as it adds the function that the additional test is being added to.
Quoted 6 lines
Can I suggest slightly different solution?
If CONFIG_..._S390 is set, we will unsetCONFIG_PCI_P2PDMAandpcim_p2pdma_provider()will be compiled to return NULL, without changingnon_mappable_barsdescription.Thanks
Hi Leon,
I'm not sure this is an ideal solution for the longer term. While ISM definitely can't do P2P and is more akin to an emulated PCI device in its actual implementation, other PCI devices on the platform such as ConnectX family NICs, NVMes and the IBM Spyre Accelerator are much more run off the mill.
So, I'd prefer not to create restrictions unnecessarily. Now there are other unusual properties like the fact that all OSs on s390 run virtualized with PCI functions passed-through by a machine level hypervisor. That and firmware also control root ports and PCIe switches. Still, I don't believe there is anything fundamentally preventing P2P to work in the future. While we could then take this restriction out again, I'd prefer not to introduce it in the first place unless we have to.
Thanks,
Niklas Schnelle
Quoted 30 lines · Niklas Schnelle
Quoted 22 lines · Leon Romanovsky
Quoted 12 lines · Matt Evans
Hi Leon,
Quoted 9 lines · Leon Romanovsky
Quoted 7 lines · Matt Evans
Extend pcim_p2pdma_provider()'s checks to exclude functions that have
pdev->non_mappable_barsset.Consumers such as VFIO were previously able to map these for access by the CPU or P2P. Update the comment on
non_mappable_barsto show it refers to any access, not just userspace CPU access.Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Fixes 372d6d1b8ae3c ("PCI/P2PDMA: Refactor to separate core P2P functionality from memory allocation")I don't object to the patch, but this Fixes line doesn't look correct.
non_mappable_barsapplies only to s390, which doesn't support p2p. That wasn't prevented before372d6d1b8ae3crefactoring too.Thanks; I'd chosen that commit as it adds the function that the additional test is being added to.
Quoted 6 lines
Can I suggest slightly different solution?
If CONFIG_..._S390 is set, we will unsetCONFIG_PCI_P2PDMAandpcim_p2pdma_provider()will be compiled to return NULL, without changingnon_mappable_barsdescription.Thanks
Quoted 7 lines
Hi Leon,
I'm not sure this is an ideal solution for the longer term. While ISM definitely can't do P2P and is more akin to an emulated PCI device in its actual implementation, other PCI devices on the platform such as ConnectX family NICs, NVMes and the IBM Spyre Accelerator are much more run off the mill.
Thanks, Niklas and Matt,
I hadn’t realized that the non_mappable_bars apply only to a specific portion of the system. You’re right — the change should be made as you suggested.
Thanks
Hi Leon,
Quoted 20 lines · Leon Romanovsky
Quoted 18 lines · Matt Evans
Hi Leon,
Quoted 14 lines · Leon Romanovsky
Quoted 8 lines · Matt Evans
Extend pcim_p2pdma_provider()'s checks to exclude functions that have
pdev->non_mappable_barsset.Consumers such as VFIO were previously able to map these for access by the CPU or P2P. Update the comment on
non_mappable_barsto show it refers to any access, not just userspace CPU access.Fixes 372d6d1b8ae3c ("PCI/P2PDMA: Refactor to separate core P2P functionality from memory allocation")I don't object to the patch, but this Fixes line doesn't look correct.
non_mappable_barsapplies only to s390, which doesn't support p2p. That wasn't prevented before372d6d1b8ae3crefactoring too.
Quoted 7 lines
Quoted 2 lines
Thanks; I'd chosen that commit as it adds the function that the additional test is being added to.
Can I suggest slightly different solution?
If CONFIG_..._S390 is set, we will unsetCONFIG_PCI_P2PDMAandpcim_p2pdma_provider()will be compiled to return NULL, without changingnon_mappable_barsdescription.
Where we got to in
https://lore.kernel.org/kvm/[email protected]/
was that non_mappable_bars could be used in future for non-s390 quirks, i.e. s390 is the current (but potentially not only) user. Secondly, it isn't a P2P-specific issue because in
https://lore.kernel.org/kvm/[email protected]/
I'm aiming to add mmap() support to VFIO DMABUFs, which (without a fix like this) would allow CPU access too.
Thanks,
Matt
Quoted 47 lines · Matt Evans
Extend pcim_p2pdma_provider()'s checks to exclude functions that have
pdev->non_mappable_barsset.Consumers such as VFIO were previously able to map these for access by the CPU or P2P. Update the comment on
non_mappable_barsto show it refers to any access, not just userspace CPU access.Fixes 372d6d1b8ae3c ("PCI/P2PDMA: Refactor to separate core P2P functionality from memory allocation")Signed-off-by Matt Evans <[email protected]>---
This arises from Alex Williamson's suggestion to test
non_mappable_barswhen getting the provider, with discussion here:https://lore.kernel.org/kvm/[email protected]/
The goal was to prevent a hole where VFIO could export DMABUFs for BARs marked non-mappable, and to fix for all users of the provider rather than just VFIO. Alex observed that
non_mappable_barsshould be taken to mean BARs weren't usable by the CPU _or_ peers and, considering that, its comment about userspace access wasn't quite right.drivers/pci/p2pdma.c | 3 ++- include/linux/pci.h | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)drivers/pci/p2pdma.c +2
-1@@ -318,7 +318,8 @@ struct p2pdma_provider *pcim_p2pdma_provider(struct pci_dev *pdev, int bar) {struct pci_p2pdma *p2p; -if (!(pci_resource_flags (pdev, bar) &IORESOURCE_MEM )) +if (!(pci_resource_flags (pdev, bar) &IORESOURCE_MEM ) || + pdev->non_mappable_bars)return NULL ; p2p =rcu_dereference_protected (pdev->p2pdma,1 );include/linux/pci.h +0
-0
include/linux/pci.h struct pci_dev { · line 514
Quoted 8 lines
include/linux/pci.h +1
-1@@ -508,7 +508,7 @@ struct pci_dev { unsigned int no_command_memory:1; /* No PCI_COMMAND_MEMORY */ unsigned int rom_bar_overlap:1; /* ROM BAR disable broken */ unsigned int rom_attr_enabled:1; /* Display of ROM attribute enabled? */ - unsigned int non_mappable_bars:1; /* BARs can't be mapped to user-space */ + unsigned int non_mappable_bars:1; /* BARs can't be mapped by CPU or peers */pci_dev_flags_t dev_flags;atomic_t enable_cnt;/* pci_enable_device has been called */
Sorry for the late reply. For s390's ISM device this definitely makes sense. The BAR can indeed not be accessed by P2P and extending this seems reasonable to me.
Feel free to add my:
Thanks,
Niklas
Quoted 47 lines · On Tue, 21 Apr 2026 10:43:51 -0700 Matt Evans <[email protected]> wrote:
Extend pcim_p2pdma_provider()'s checks to exclude functions that have
pdev->non_mappable_barsset.Consumers such as VFIO were previously able to map these for access by the CPU or P2P. Update the comment on
non_mappable_barsto show it refers to any access, not just userspace CPU access.Fixes 372d6d1b8ae3c ("PCI/P2PDMA: Refactor to separate core P2P functionality from memory allocation")Signed-off-by Matt Evans <[email protected]>---
This arises from Alex Williamson's suggestion to test
non_mappable_barswhen getting the provider, with discussion here:https://lore.kernel.org/kvm/[email protected]/
The goal was to prevent a hole where VFIO could export DMABUFs for BARs marked non-mappable, and to fix for all users of the provider rather than just VFIO. Alex observed that
non_mappable_barsshould be taken to mean BARs weren't usable by the CPU _or_ peers and, considering that, its comment about userspace access wasn't quite right.drivers/pci/p2pdma.c | 3 ++- include/linux/pci.h | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)drivers/pci/p2pdma.c +2
-1@@ -318,7 +318,8 @@ struct p2pdma_provider *pcim_p2pdma_provider(struct pci_dev *pdev, int bar) {struct pci_p2pdma *p2p; -if (!(pci_resource_flags (pdev, bar) &IORESOURCE_MEM )) +if (!(pci_resource_flags (pdev, bar) &IORESOURCE_MEM ) || + pdev->non_mappable_bars)return NULL ; p2p =rcu_dereference_protected (pdev->p2pdma,1 );include/linux/pci.h +0
-0
include/linux/pci.h struct pci_dev { · line 514
Quoted 8 lines
include/linux/pci.h +1
-1@@ -508,7 +508,7 @@ struct pci_dev { unsigned int no_command_memory:1; /* No PCI_COMMAND_MEMORY */ unsigned int rom_bar_overlap:1; /* ROM BAR disable broken */ unsigned int rom_attr_enabled:1; /* Display of ROM attribute enabled? */ - unsigned int non_mappable_bars:1; /* BARs can't be mapped to user-space */ + unsigned int non_mappable_bars:1; /* BARs can't be mapped by CPU or peers */pci_dev_flags_t dev_flags;atomic_t enable_cnt;/* pci_enable_device has been called */
Should pcim_p2pdma_init() separately test pdev->non_mappable_bars before the rcu-deref/kzalloc of the pci_p2pdma object and return -EOPNOTSUPP?
That then invokes the same error paths we'd see if we simply don't have p2pdma in the kernel and handles the pci_p2pdma_add_resource() path automatically as well.
This pcim_p2pdma_provider() test is really then just suppressing the WARN_ON that we'd otherwise see by not finding the p2p object on the device. Thanks,
Alex
Hi Alex,
Quoted 64 lines · Alex Williamson
Quoted 47 lines · On Tue, 21 Apr 2026 10:43:51 -0700 Matt Evans <[email protected]> wrote:
Extend pcim_p2pdma_provider()'s checks to exclude functions that have
pdev->non_mappable_barsset.Consumers such as VFIO were previously able to map these for access by the CPU or P2P. Update the comment on
non_mappable_barsto show it refers to any access, not just userspace CPU access.Fixes 372d6d1b8ae3c ("PCI/P2PDMA: Refactor to separate core P2P functionality from memory allocation")Signed-off-by Matt Evans <[email protected]>---
This arises from Alex Williamson's suggestion to test
non_mappable_barswhen getting the provider, with discussion here:https://lore.kernel.org/kvm/[email protected]/
The goal was to prevent a hole where VFIO could export DMABUFs for BARs marked non-mappable, and to fix for all users of the provider rather than just VFIO. Alex observed that
non_mappable_barsshould be taken to mean BARs weren't usable by the CPU _or_ peers and, considering that, its comment about userspace access wasn't quite right.drivers/pci/p2pdma.c | 3 ++- include/linux/pci.h | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)drivers/pci/p2pdma.c +2
-1@@ -318,7 +318,8 @@ struct p2pdma_provider *pcim_p2pdma_provider(struct pci_dev *pdev, int bar) {struct pci_p2pdma *p2p; -if (!(pci_resource_flags (pdev, bar) &IORESOURCE_MEM )) +if (!(pci_resource_flags (pdev, bar) &IORESOURCE_MEM ) || + pdev->non_mappable_bars)return NULL ; p2p =rcu_dereference_protected (pdev->p2pdma,1 );include/linux/pci.h +0
-0include/linux/pci.h struct pci_dev { · line 514
Quoted 8 lines
include/linux/pci.h +1
-1@@ -508,7 +508,7 @@ struct pci_dev { unsigned int no_command_memory:1; /* No PCI_COMMAND_MEMORY */ unsigned int rom_bar_overlap:1; /* ROM BAR disable broken */ unsigned int rom_attr_enabled:1; /* Display of ROM attribute enabled? */ - unsigned int non_mappable_bars:1; /* BARs can't be mapped to user-space */ + unsigned int non_mappable_bars:1; /* BARs can't be mapped by CPU or peers */pci_dev_flags_t dev_flags;atomic_t enable_cnt;/* pci_enable_device has been called */ Should
pcim_p2pdma_init()separately testpdev->non_mappable_barsbefore the rcu-deref/kzalloc of thepci_p2pdmaobject and return-EOPNOTSUPP?
Quoted 7 lines
That then invokes the same error paths we'd see if we simply don't have p2pdma in the kernel and handles the
pci_p2pdma_add_resource()path automatically as well.This
pcim_p2pdma_provider()test is really then just suppressing theWARN_ONthat we'd otherwise see by not finding the p2p object on the device. Thanks,
That makes sense, yes, it'll avoid the WARN that'd otherwise occur in pci_p2pdma_add_resource().
I'll do a v2 adding this check to pcim_p2pdma_init() as well. Hopefully Logan & Niklas's R-B will still apply.
Thanks,
Matt