vfio/pci: Allow MMIO regions to be exported through dma-buf
Subsystems (12)
cover vfio/pci: Allow MMIO regions to be exported through dma-buf
Changelog:
v1:
* Changed commit messages.
* Reused DMA_ATTR_MMIO attribute.
* Returned support for multiple DMA ranges per-dMABUF.
v0: https://lore.kernel.org/all/[email protected]
---------------------------------------------------------------------------
Based on "[PATCH v1 00/16] dma-mapping: migrate to physical address-based API"
https://lore.kernel.org/all/[email protected] series.
---------------------------------------------------------------------------
This series extends the VFIO PCI subsystem to support exporting MMIO regions from PCI device BARs as dma-buf objects, enabling safe sharing of non-struct page memory with controlled lifetime management. This allows RDMA and other subsystems to import dma-buf FDs and build them into memory regions for PCI P2P operations.
The series supports a use case for SPDK where a NVMe device will be owned by SPDK through VFIO but interacting with a RDMA device. The RDMA device may directly access the NVMe CMB or directly manipulate the NVMe device's doorbell using PCI P2P.
However, as a general mechanism, it can support many other scenarios with VFIO. This dmabuf approach can be usable by iommufd as well for generic and safe P2P mappings.
In addition to the SPDK use-case mentioned above, the capability added in this patch series can also be useful when a buffer (located in device memory such as VRAM) needs to be shared between any two dGPU devices or instances (assuming one of them is bound to VFIO PCI) as long as they are P2P DMA compatible.
The implementation provides a revocable attachment mechanism using dma-buf move operations. MMIO regions are normally pinned as BARs don't change physical addresses, but access is revoked when the VFIO device is closed or a PCI reset is issued. This ensures kernel self-defense against potentially hostile userspace.
The series includes significant refactoring of the PCI P2PDMA subsystem to separate core P2P functionality from memory allocation features, making it more modular and suitable for VFIO use cases that don't need struct page support.
-----------------------------------------------------------------------
The series is based originally on
https://lore.kernel.org/all/[email protected]/
but heavily rewritten to be based on DMA physical API.
-----------------------------------------------------------------------
The WIP branch can be found here:
https://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git/log/?h=dmabuf-vfio-v1
Thanks
Leon Romanovsky (8):
PCI/P2PDMA: Remove redundant bus_offset from map state PCI/P2PDMA: Separate the mmap() support from the core logic PCI/P2PDMA: Simplify bus address mapping API
PCI/P2PDMA: Refactor to separate core P2P functionality from memory
allocation
PCI/P2PDMA: Export pci_p2pdma_map_type() function
types: move phys_vec definition to common header vfio/pci: Enable peer-to-peer DMA transactions by default vfio/pci: Add dma-buf export support for MMIO regions
Vivek Kasireddy (2):
vfio: Export vfio device get and put registration helpers vfio/pci: Share the core device pointer while invoking feature
functions
block/blk-mq-dma.c | 7 +- drivers/iommu/dma-iommu.c | 4 +- drivers/pci/p2pdma.c | 154 ++++++++---- drivers/vfio/pci/Kconfig | 20 ++ drivers/vfio/pci/Makefile | 2 + drivers/vfio/pci/vfio_pci_config.c | 22 +- drivers/vfio/pci/vfio_pci_core.c | 59 +++-- drivers/vfio/pci/vfio_pci_dmabuf.c | 390 +++++++++++++++++++++++++++++ drivers/vfio/pci/vfio_pci_priv.h | 23 ++ drivers/vfio/vfio_main.c | 2 + include/linux/dma-buf.h | 1 + include/linux/pci-p2pdma.h | 114 +++++---- include/linux/types.h | 5 + include/linux/vfio.h | 2 + include/linux/vfio_pci_core.h | 4 + include/uapi/linux/vfio.h | 25 ++ kernel/dma/direct.c | 4 +- mm/hmm.c | 2 +- 18 files changed, 715 insertions(+), 125 deletions(-) create mode 100644 drivers/vfio/pci/vfio_pci_dmabuf.c
Signature · 1 line
2.50.1
PATCH 1/10 PCI/P2PDMA: Remove redundant bus_offset from map state
From: Leon Romanovsky <[email protected]>
Remove the bus_off field from pci_p2pdma_map_state since it duplicates information already available in the pgmap structure. The bus_offset is only used in one location (pci_p2pdma_bus_addr_map) and is always identical to pgmap->bus_offset.
--- drivers/pci/p2pdma.c | 1 - include/linux/pci-p2pdma.h | 3 +-- 2 files changed, 1 insertion(+), 3 deletions(-)
drivers/pci/p2pdma.c +0 -1
@@ -1009,7 +1009,6 @@ void __pci_p2pdma_update_state(struct pci_p2pdma_map_state *state, { state->pgmap =page_pgmap (page); state->map =pci_p2pdma_map_type (state->pgmap, dev); - state->bus_off =to_p2p_pgmap (state->pgmap)->bus_offset; }/**
include/linux/pci-p2pdma.h +1 -2
@@ -146,7 +146,6 @@ enum pci_p2pdma_map_type {struct pci_p2pdma_map_state {struct dev_pagemap *pgmap;enum pci_p2pdma_map_type map; -u64 bus_off; };/* helper for pci_p2pdma_state(), do not use directly */
@@ -186,7 +185,7 @@ static inline dma_addr_t pci_p2pdma_bus_addr_map(struct pci_p2pdma_map_state *state, phys_addr_t paddr) { WARN_ON_ONCE(state->map != PCI_P2PDMA_MAP_BUS_ADDR); -return paddr + state->bus_off; +return paddr +to_p2p_pgmap (state->pgmap)->bus_offsetf; }#endif /* _LINUX_PCI_P2P_H */
Signature · 1 line
2.50.1
PATCH 3/10 PCI/P2PDMA: Simplify bus address mapping API
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Update the pci_p2pdma_bus_addr_map() function to take a direct pointer to the p2pdma_provider structure instead of the pci_p2pdma_map_state. This simplifies the API by removing the need for callers to extract the provider from the state structure.
The change updates all callers across the kernel (block layer, IOMMU, DMA direct, and HMM) to pass the provider pointer directly, making the code more explicit and reducing unnecessary indirection. This also removes the runtime warning check since callers now have direct control over which provider they use.
--- block/blk-mq-dma.c | 2 +- drivers/iommu/dma-iommu.c | 4 ++-- include/linux/pci-p2pdma.h | 7 +++---- kernel/dma/direct.c | 4 ++-- mm/hmm.c | 2 +- 5 files changed, 9 insertions(+), 10 deletions(-)
block/blk-mq-dma.c +1 -1
@@ -79,7 +79,7 @@ static inline bool blk_can_dma_map_iova(struct request *req,static bool blk_dma_map_bus (struct blk_dma_iter *iter,struct phys_vec *vec) { - iter->addr =pci_p2pdma_bus_addr_map (&iter->p2pdma, vec->paddr); + iter->addr =pci_p2pdma_bus_addr_map (iter->p2pdma.mem, vec->paddr); iter->len = vec->len;return true; }
drivers/iommu/dma-iommu.c +2 -2
@@ -1439,8 +1439,8 @@ int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, int nents,* as a bus address, __finalise_sg() will copy the dma * address into the output segment. */ - s->dma_address =pci_p2pdma_bus_addr_map (&p2pdma_state, -sg_phys (s)); + s->dma_address =pci_p2pdma_bus_addr_map ( + p2pdma_state.mem,sg_phys (s));sg_dma_len (s) = sg->length;sg_dma_mark_bus_address (s);continue ;
include/linux/pci-p2pdma.h +3 -4
@@ -186,16 +186,15 @@ pci_p2pdma_state(struct pci_p2pdma_map_state *state, struct device *dev,/** * pci_p2pdma_bus_addr_map - Translate a physical address to a bus address * for a PCI_P2PDMA_MAP_BUS_ADDR transfer. -* @state: P2P state structure +* @provider: P2P provider structure * @paddr: physical address to map * * Map a physically contiguous PCI_P2PDMA_MAP_BUS_ADDR transfer. */ static inline dma_addr_t -pci_p2pdma_bus_addr_map (struct pci_p2pdma_map_state *state,phys_addr_t paddr) +pci_p2pdma_bus_addr_map (struct p2pdma_provider *provider,phys_addr_t paddr) { -WARN_ON_ONCE (state->map !=PCI_P2PDMA_MAP_BUS_ADDR ); -return paddr + state->mem->bus_offset; +return paddr + provider->bus_offset; }#endif /* _LINUX_PCI_P2P_H */
kernel/dma/direct.c +2 -2
@@ -484,8 +484,8 @@ int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, }break ;case PCI_P2PDMA_MAP_BUS_ADDR : - sg->dma_address =pci_p2pdma_bus_addr_map (&p2pdma_state, -sg_phys (sg)); + sg->dma_address =pci_p2pdma_bus_addr_map ( + p2pdma_state.mem,sg_phys (sg));sg_dma_mark_bus_address (sg);continue ;default :
mm/hmm.c +1 -1
@@ -751,7 +751,7 @@ dma_addr_t hmm_dma_map_pfn(struct device *dev, struct hmm_dma_map *map,break ;case PCI_P2PDMA_MAP_BUS_ADDR : pfns[idx] |=HMM_PFN_P2PDMA_BUS |HMM_PFN_DMA_MAPPED ; -return pci_p2pdma_bus_addr_map (p2pdma_state, paddr); +return pci_p2pdma_bus_addr_map (p2pdma_state->mem, paddr);default :return DMA_MAPPING_ERROR ; }
Signature · 1 line
2.50.1
PATCH 2/10 PCI/P2PDMA: Separate the mmap() support from the core logic
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Currently the P2PDMA code requires a pgmap and a struct page to function. The was serving three important purposes:
- DMA API compatibility, where scatterlist required a struct page as input
- Life cycle management, the percpu_ref is used to prevent UAF during device hot unplug
- A way to get the P2P provider data through the pci_p2pdma_pagemap
The DMA API now has a new flow, and has gained phys_addr_t support, so it no longer needs struct pages to perform P2P mapping.
Lifecycle management can be delegated to the user, DMABUF for instance has a suitable invalidation protocol that does not require struct page.
Finding the P2P provider data can also be managed by the caller without need to look it up from the phys_addr.
Split the P2PDMA code into two layers. The optional upper layer, effectively, provides a way to mmap() P2P memory into a VMA by providing struct page, pgmap, a genalloc and sysfs.
The lower layer provides the actual P2P infrastructure and is wrapped up in a new struct p2pdma_provider. Rework the mmap layer to use new p2pdma_provider based APIs.
Drivers that do not want to put P2P memory into VMA's can allocate a struct p2pdma_provider after probe() starts and free it before remove() completes. When DMA mapping the driver must convey the struct p2pdma_provider to the DMA mapping code along with a phys_addr of the MMIO BAR slice to map. The driver must ensure that no DMA mapping outlives the lifetime of the struct p2pdma_provider.
The intended target of this new API layer is DMABUF. There is usually only a single p2pdma_provider for a DMABUF exporter. Most drivers can establish the p2pdma_provider during probe, access the single instance during DMABUF attach and use that to drive the DMA mapping.
DMABUF provides an invalidation mechanism that can guarantee all DMA is halted and the DMA mappings are undone prior to destroying the struct p2pdma_provider. This ensures there is no UAF through DMABUFs that are lingering past driver removal.
The new p2pdma_provider layer cannot be used to create P2P memory that can be mapped into VMA's, be used with pin_user_pages(), O_DIRECT, and so on. These use cases must still use the mmap() layer. The p2pdma_provider layer is principally for DMABUF-like use cases where DMABUF natively manages the life cycle and access instead of vmas/pin_user_pages()/struct page.
--- drivers/pci/p2pdma.c | 42 +++++++++++++++++++++----------------- include/linux/pci-p2pdma.h | 18 ++++++++++++---- 2 files changed, 37 insertions(+), 23 deletions(-)
drivers/pci/p2pdma.c +23 -19
@@ -28,9 +28,8 @@ struct pci_p2pdma { };struct pci_p2pdma_pagemap { -struct pci_dev *provider; -u64 bus_offset;struct dev_pagemap pgmap; +struct p2pdma_provider mem; };static struct pci_p2pdma_pagemap *to_p2p_pgmap (struct dev_pagemap *pgmap)
@@ -204,8 +203,8 @@ static void p2pdma_page_free(struct page *page) {struct pci_p2pdma_pagemap *pgmap =to_p2p_pgmap (page_pgmap (page));/* safe to dereference while a reference is held to the percpu ref */ -struct pci_p2pdma *p2pdma = -rcu_dereference_protected (pgmap->provider->p2pdma,1 ); +struct pci_p2pdma *p2pdma =rcu_dereference_protected ( +to_pci_dev (pgmap->mem.owner)->p2pdma,1 );struct percpu_ref *ref; gen_pool_free_owner(p2pdma->pool, (uintptr_t)page_to_virt(page),
@@ -270,14 +269,15 @@ static int pci_p2pdma_setup(struct pci_dev *pdev)static void pci_p2pdma_unmap_mappings (void *data) { -struct pci_dev *pdev = data; +struct pci_p2pdma_pagemap *p2p_pgmap = data;/* * Removing the alloc attribute from sysfs will call * unmap_mapping_range() on the inode, teardown any existing userspace * mappings and prevent new ones from being created. */ -sysfs_remove_file_from_group (&pdev->dev.kobj, &p2pmem_alloc_attr.attr, +sysfs_remove_file_from_group (&p2p_pgmap->mem.owner->kobj, + &p2pmem_alloc_attr.attr, p2pmem_group.name); }
@@ -328,10 +328,9 @@ int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, pgmap->nr_range =1 ; pgmap->type =MEMORY_DEVICE_PCI_P2PDMA ; pgmap->ops = &p2pdma_pgmap_ops; - - p2p_pgmap->provider = pdev; - p2p_pgmap->bus_offset =pci_bus_address (pdev, bar) - -pci_resource_start (pdev, bar); + p2p_pgmap->mem.owner = &pdev->dev; + p2p_pgmap->mem.bus_offset = +pci_bus_address (pdev, bar) -pci_resource_start (pdev, bar); addr =devm_memremap_pages (&pdev->dev, pgmap);if (IS_ERR (addr)) {
@@ -340,7 +339,7 @@ int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, } error =devm_add_action_or_reset (&pdev->dev, pci_p2pdma_unmap_mappings, - pdev); + p2p_pgmap);if (error) goto pages_free;
@@ -973,16 +972,16 @@ void pci_p2pmem_publish(struct pci_dev *pdev, bool publish) }EXPORT_SYMBOL_GPL (pci_p2pmem_publish); -static enum pci_p2pdma_map_type pci_p2pdma_map_type (struct dev_pagemap *pgmap, -struct device *dev) +static enum pci_p2pdma_map_type +pci_p2pdma_map_type (struct p2pdma_provider *provider,struct device *dev) {enum pci_p2pdma_map_type type =PCI_P2PDMA_MAP_NOT_SUPPORTED ; -struct pci_dev *provider =to_p2p_pgmap (pgmap)->provider; +struct pci_dev *pdev =to_pci_dev (provider->owner);struct pci_dev *client;struct pci_p2pdma *p2pdma;int dist; -if (!provider->p2pdma) +if (!pdev->p2pdma)return PCI_P2PDMA_MAP_NOT_SUPPORTED ;if (!dev_is_pci (dev))
@@ -991,7 +990,7 @@ static enum pci_p2pdma_map_type pci_p2pdma_map_type(struct dev_pagemap *pgmap, client =to_pci_dev (dev);rcu_read_lock (); - p2pdma =rcu_dereference (provider->p2pdma); + p2pdma =rcu_dereference (pdev->p2pdma);if (p2pdma) type = xa_to_value(xa_load(&p2pdma->map_types,
@@ -999,7 +998,7 @@ static enum pci_p2pdma_map_type pci_p2pdma_map_type(struct dev_pagemap *pgmap,rcu_read_unlock ();if (type ==PCI_P2PDMA_MAP_UNKNOWN ) -return calc_map_type_and_dist (provider, client, &dist, true); +return calc_map_type_and_dist (pdev, client, &dist, true);return type; }
@@ -1007,8 +1006,13 @@ static enum pci_p2pdma_map_type pci_p2pdma_map_type(struct dev_pagemap *pgmap,void __pci_p2pdma_update_state (struct pci_p2pdma_map_state *state,struct device *dev,struct page *page) { - state->pgmap =page_pgmap (page); - state->map =pci_p2pdma_map_type (state->pgmap, dev); +struct pci_p2pdma_pagemap *p2p_pgmap =to_p2p_pgmap (page_pgmap (page)); + +if (state->mem == &p2p_pgmap->mem) +return ; + + state->mem = &p2p_pgmap->mem; + state->map =pci_p2pdma_map_type (&p2p_pgmap->mem, dev); }/**
include/linux/pci-p2pdma.h +14 -4
@@ -16,6 +16,16 @@struct block_device ;struct scatterlist ; +/** +* struct p2pdma_provider +* +* A p2pdma provider is a range of MMIO address space available to the CPU. +*/ +struct p2pdma_provider { +struct device *owner; +u64 bus_offset; +}; +#ifdef CONFIG_PCI_P2PDMA int pci_p2pdma_add_resource (struct pci_dev *pdev,int bar,size_t size,u64 offset);
@@ -144,10 +154,11 @@ enum pci_p2pdma_map_type { };struct pci_p2pdma_map_state { -struct dev_pagemap *pgmap; +struct p2pdma_provider *mem;enum pci_p2pdma_map_type map; }; +/* helper for pci_p2pdma_state(), do not use directly */ void __pci_p2pdma_update_state (struct pci_p2pdma_map_state *state,struct device *dev,struct page *page);
@@ -166,8 +177,7 @@ pci_p2pdma_state(struct pci_p2pdma_map_state *state, struct device *dev, struct page *page) {if (IS_ENABLED (CONFIG_PCI_P2PDMA ) &&is_pci_p2pdma_page (page)) { -if (state->pgmap !=page_pgmap (page)) -__pci_p2pdma_update_state (state, dev, page); +__pci_p2pdma_update_state (state, dev, page);return state->map; }return PCI_P2PDMA_MAP_NONE ;
@@ -185,7 +195,7 @@ static inline dma_addr_t pci_p2pdma_bus_addr_map(struct pci_p2pdma_map_state *state, phys_addr_t paddr) { WARN_ON_ONCE(state->map != PCI_P2PDMA_MAP_BUS_ADDR); -return paddr +to_p2p_pgmap (state->pgmap)->bus_offsetf; +return paddr + state->mem->bus_offset; }#endif /* _LINUX_PCI_P2P_H */
Signature · 1 line
2.50.1
PATCH 5/10 PCI/P2PDMA: Export pci_p2pdma_map_type() function
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Export the pci_p2pdma_map_type() function to allow external modules and subsystems to determine the appropriate mapping type for P2PDMA transfers between a provider and target device.
The function determines whether peer-to-peer DMA transfers can be done directly through PCI switches (PCI_P2PDMA_MAP_BUS_ADDR) or must go through the host bridge (PCI_P2PDMA_MAP_THRU_HOST_BRIDGE), or if the transfer is not supported at all.
This export enables subsystems like VFIO to properly handle P2PDMA operations by querying the mapping type before attempting transfers, ensuring correct DMA address programming and error handling.
--- drivers/pci/p2pdma.c | 15 ++++++- include/linux/pci-p2pdma.h | 85 +++++++++++++++++++++----------------- 2 files changed, 59 insertions(+), 41 deletions(-)
drivers/pci/p2pdma.c +13 -2
@@ -1022,8 +1022,18 @@ void pci_p2pmem_publish(struct pci_dev *pdev, bool publish) }EXPORT_SYMBOL_GPL (pci_p2pmem_publish); -static enum pci_p2pdma_map_type -pci_p2pdma_map_type (struct p2pdma_provider *provider,struct device *dev) +/** +* pci_p2pdma_map_type - Determine the mapping type for P2PDMA transfers +* @provider: P2PDMA provider structure +* @dev: Target device for the transfer +* +* Determines how peer-to-peer DMA transfers should be mapped between +* the provider and the target device. The mapping type indicates whether +* the transfer can be done directly through PCI switches or must go +* through the host bridge. +*/ +enum pci_p2pdma_map_type pci_p2pdma_map_type (struct p2pdma_provider *provider, +struct device *dev) {enum pci_p2pdma_map_type type =PCI_P2PDMA_MAP_NOT_SUPPORTED ;struct pci_dev *pdev =to_pci_dev (provider->owner);
@@ -1052,6 +1062,7 @@ pci_p2pdma_map_type(struct p2pdma_provider *provider, struct device *dev)return type; } +EXPORT_SYMBOL_GPL (pci_p2pdma_map_type);void __pci_p2pdma_update_state (struct pci_p2pdma_map_state *state,struct device *dev,struct page *page)
include/linux/pci-p2pdma.h +46 -39
109 lines not on this page · raw message
Signature · 1 line
2.50.1
PATCH 4/10 PCI/P2PDMA: Refactor to separate core P2P functionality from memory allocation
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Refactor the PCI P2PDMA subsystem to separate the core peer-to-peer DMA functionality from the optional memory allocation layer. This creates a two-tier architecture:
The core layer provides P2P mapping functionality for physical addresses based on PCI device MMIO BARs and integrates with the DMA API for mapping operations. This layer is required for all P2PDMA users.
The optional upper layer provides memory allocation capabilities including gen_pool allocator, struct page support, and sysfs interface for user space access.
This separation allows subsystems like VFIO to use only the core P2P mapping functionality without the overhead of memory allocation features they don't need. The core functionality is now available through the new pci_p2pdma_enable() function that returns a p2pdma_provider structure.
--- drivers/pci/p2pdma.c | 118 ++++++++++++++++++++++++++----------- include/linux/pci-p2pdma.h | 5 ++ 2 files changed, 89 insertions(+), 34 deletions(-)
drivers/pci/p2pdma.c +84 -34
@@ -25,11 +25,12 @@ struct pci_p2pdma {struct gen_pool *pool;bool p2pmem_published;struct xarray map_types; +struct p2pdma_provider mem; };struct pci_p2pdma_pagemap {struct dev_pagemap pgmap; -struct p2pdma_provider mem; +struct p2pdma_provider *mem; };static struct pci_p2pdma_pagemap *to_p2p_pgmap (struct dev_pagemap *pgmap)
@@ -204,7 +205,7 @@ static void p2pdma_page_free(struct page *page)struct pci_p2pdma_pagemap *pgmap =to_p2p_pgmap (page_pgmap (page));/* safe to dereference while a reference is held to the percpu ref */ struct pci_p2pdma *p2pdma =rcu_dereference_protected ( -to_pci_dev (pgmap->mem.owner)->p2pdma,1 ); +to_pci_dev (pgmap->mem->owner)->p2pdma,1 );struct percpu_ref *ref; gen_pool_free_owner(p2pdma->pool, (uintptr_t)page_to_virt(page),
@@ -227,44 +228,82 @@ static void pci_p2pdma_release(void *data)/* Flush and disable pci_alloc_p2p_mem() */ pdev->p2pdma =NULL ; -synchronize_rcu (); +if (p2pdma->pool) +synchronize_rcu (); +xa_destroy (&p2pdma->map_types); + +if (!p2pdma->pool) +return ;gen_pool_destroy (p2pdma->pool);sysfs_remove_group (&pdev->dev.kobj, &p2pmem_group); -xa_destroy (&p2pdma->map_types); } -static int pci_p2pdma_setup (struct pci_dev *pdev) +/** +* pci_p2pdma_enable - Enable peer-to-peer DMA support for a PCI device +* @pdev: The PCI device to enable P2PDMA for +* +* This function initializes the peer-to-peer DMA infrastructure for a PCI +* device. It allocates and sets up the necessary data structures to support +* P2PDMA operations, including mapping type tracking. +*/ +struct p2pdma_provider *pci_p2pdma_enable (struct pci_dev *pdev) { -int error = -ENOMEM ;struct pci_p2pdma *p2p; +int ret; + + p2p =rcu_dereference_protected (pdev->p2pdma,1 ); +if (p2p) +/* PCI device was "rebound" to the driver */ +return &p2p->mem; p2p =devm_kzalloc (&pdev->dev,sizeof (*p2p),GFP_KERNEL );if (!p2p) -return -ENOMEM ; +return ERR_PTR (-ENOMEM );xa_init (&p2p->map_types); + p2p->mem.owner = &pdev->dev; +/* On all p2p platforms bus_offset is the same for all BARs */ + p2p->mem.bus_offset = +pci_bus_address (pdev,0 ) -pci_resource_start (pdev,0 ); - p2p->pool =gen_pool_create (PAGE_SHIFT ,dev_to_node (&pdev->dev)); -if (!p2p->pool) - goto out; + ret =devm_add_action_or_reset (&pdev->dev, pci_p2pdma_release, pdev); +if (ret) + goto out_p2p; - error =devm_add_action_or_reset (&pdev->dev, pci_p2pdma_release, pdev); -if (error) - goto out_pool_destroy; +rcu_assign_pointer (pdev->p2pdma, p2p); +return &p2p->mem; - error =sysfs_create_group (&pdev->dev.kobj, &p2pmem_group); -if (error) +out_p2p: +devm_kfree (&pdev->dev, p2p); +return ERR_PTR (ret); +} +EXPORT_SYMBOL_GPL (pci_p2pdma_enable); + +static int pci_p2pdma_setup_pool (struct pci_dev *pdev) +{ +struct pci_p2pdma *p2pdma; +int ret; + + p2pdma =rcu_dereference_protected (pdev->p2pdma,1 ); +if (p2pdma->pool) +/* We already setup pools, do nothing, */ +return 0 ; + + p2pdma->pool =gen_pool_create (PAGE_SHIFT ,dev_to_node (&pdev->dev)); +if (!p2pdma->pool) +return -ENOMEM ; + + ret =sysfs_create_group (&pdev->dev.kobj, &p2pmem_group); +if (ret) goto out_pool_destroy; -rcu_assign_pointer (pdev->p2pdma, p2p);return 0 ; out_pool_destroy: -gen_pool_destroy (p2p->pool); -out: -devm_kfree (&pdev->dev, p2p); -return error; +gen_pool_destroy (p2pdma->pool); + p2pdma->pool =NULL ; +return ret; }static void pci_p2pdma_unmap_mappings (void *data)
@@ -276,7 +315,7 @@ static void pci_p2pdma_unmap_mappings(void *data)* unmap_mapping_range() on the inode, teardown any existing userspace * mappings and prevent new ones from being created. */ -sysfs_remove_file_from_group (&p2p_pgmap->mem.owner->kobj, +sysfs_remove_file_from_group (&p2p_pgmap->mem->owner->kobj, &p2pmem_alloc_attr.attr, p2pmem_group.name); }
@@ -295,6 +334,7 @@ int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, u64 offset) {struct pci_p2pdma_pagemap *p2p_pgmap; +struct p2pdma_provider *mem;struct dev_pagemap *pgmap;struct pci_p2pdma *p2pdma;void *addr;
@@ -312,15 +352,25 @@ int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size,if (size + offset >pci_resource_len (pdev, bar))return -EINVAL ; -if (!pdev->p2pdma) { - error =pci_p2pdma_setup (pdev); + p2pdma =rcu_dereference_protected (pdev->p2pdma,1 ); +if (!p2pdma) { + mem =pci_p2pdma_enable (pdev); +if (IS_ERR (mem)) +return PTR_ERR (mem); + + error =pci_p2pdma_setup_pool (pdev);if (error)return error; - } + + p2pdma =rcu_dereference_protected (pdev->p2pdma,1 ); + }else + mem = &p2pdma->mem; p2p_pgmap =devm_kzalloc (&pdev->dev,sizeof (*p2p_pgmap),GFP_KERNEL ); -if (!p2p_pgmap) -return -ENOMEM ; +if (!p2p_pgmap) { + error = -ENOMEM ; + goto free_pool; + } pgmap = &p2p_pgmap->pgmap; pgmap->range.start =pci_resource_start (pdev, bar) + offset;
@@ -328,9 +378,7 @@ int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, pgmap->nr_range =1 ; pgmap->type =MEMORY_DEVICE_PCI_P2PDMA ; pgmap->ops = &p2pdma_pgmap_ops; - p2p_pgmap->mem.owner = &pdev->dev; - p2p_pgmap->mem.bus_offset = -pci_bus_address (pdev, bar) -pci_resource_start (pdev, bar); + p2p_pgmap->mem = mem; addr =devm_memremap_pages (&pdev->dev, pgmap);if (IS_ERR (addr)) {
@@ -343,7 +391,6 @@ int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size,if (error) goto pages_free; - p2pdma =rcu_dereference_protected (pdev->p2pdma,1 ); error = gen_pool_add_owner(p2pdma->pool, (unsigned long)addr, pci_bus_address(pdev, bar) + offset, range_len(&pgmap->range), dev_to_node(&pdev->dev),
@@ -359,7 +406,10 @@ int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, pages_free:devm_memunmap_pages (&pdev->dev, pgmap); pgmap_free: -devm_kfree (&pdev->dev, pgmap); +devm_kfree (&pdev->dev, p2p_pgmap); +free_pool: +sysfs_remove_group (&pdev->dev.kobj, &p2pmem_group); +gen_pool_destroy (p2pdma->pool);return error; }EXPORT_SYMBOL_GPL (pci_p2pdma_add_resource);
@@ -1008,11 +1058,11 @@ void __pci_p2pdma_update_state(struct pci_p2pdma_map_state *state, {struct pci_p2pdma_pagemap *p2p_pgmap =to_p2p_pgmap (page_pgmap (page)); -if (state->mem == &p2p_pgmap->mem) +if (state->mem == p2p_pgmap->mem)return ; - state->mem = &p2p_pgmap->mem; - state->map =pci_p2pdma_map_type (&p2p_pgmap->mem, dev); + state->mem = p2p_pgmap->mem; + state->map =pci_p2pdma_map_type (p2p_pgmap->mem, dev); }/**
include/linux/pci-p2pdma.h +5 -0
@@ -27,6 +27,7 @@ struct p2pdma_provider { };#ifdef CONFIG_PCI_P2PDMA +struct p2pdma_provider *pci_p2pdma_enable (struct pci_dev *pdev);int pci_p2pdma_add_resource (struct pci_dev *pdev,int bar,size_t size,u64 offset); int pci_p2pdma_distance_many(struct pci_dev *provider, struct device **clients,
@@ -45,6 +46,10 @@ int pci_p2pdma_enable_store(const char *page, struct pci_dev **p2p_dev,ssize_t pci_p2pdma_enable_show (char *page,struct pci_dev *p2p_dev,bool use_p2pdma);#else /* CONFIG_PCI_P2PDMA */ +static inline struct p2pdma_provider *pci_p2pdma_enable (struct pci_dev *pdev) +{ +return ERR_PTR (-EOPNOTSUPP ); +}static inline int pci_p2pdma_add_resource (struct pci_dev *pdev,int bar,size_t size,u64 offset) {
Signature · 1 line
2.50.1
PATCH 7/10 vfio: Export vfio device get and put registration helpers
In reply to Leon RomanovskyFrom: Vivek Kasireddy <[email protected]>
These helpers are useful for managing additional references taken on the device from other associated VFIO modules.
--- drivers/vfio/vfio_main.c | 2 ++ include/linux/vfio.h | 2 ++ 2 files changed, 4 insertions(+)
drivers/vfio/vfio_main.c +2 -0
@@ -171,11 +171,13 @@ void vfio_device_put_registration(struct vfio_device *device)if (refcount_dec_and_test (&device->refcount))complete (&device->comp); } +EXPORT_SYMBOL_GPL (vfio_device_put_registration);bool vfio_device_try_get_registration (struct vfio_device *device) {return refcount_inc_not_zero (&device->refcount); } +EXPORT_SYMBOL_GPL (vfio_device_try_get_registration);/* * VFIO driver API
include/linux/vfio.h +2 -0
@@ -293,6 +293,8 @@ static inline void vfio_put_device(struct vfio_device *device)int vfio_register_group_dev (struct vfio_device *device);int vfio_register_emulated_iommu_dev (struct vfio_device *device);void vfio_unregister_group_dev (struct vfio_device *device); +bool vfio_device_try_get_registration (struct vfio_device *device); +void vfio_device_put_registration (struct vfio_device *device);int vfio_assign_device_set (struct vfio_device *device,void *set_id);unsigned int vfio_device_set_open_count (struct vfio_device_set *dev_set);
Signature · 1 line
2.50.1
PATCH 6/10 types: move phys_vec definition to common header
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Move the struct phys_vec definition from block/blk-mq-dma.c to include/linux/types.h to make it available for use across the kernel.
The phys_vec structure represents a physical address range with a length, which is used by the new physical address-based DMA mapping API. This structure is already used by the block layer and will be needed by upcoming VFIO patches for dma-buf operations.
Moving this definition to types.h provides a centralized location for this common data structure and eliminates code duplication across subsystems that need to work with physical address ranges.
--- block/blk-mq-dma.c | 5 ----- include/linux/types.h | 5 +++++ 2 files changed, 5 insertions(+), 5 deletions(-)
block/blk-mq-dma.c +0 -5
@@ -5,11 +5,6 @@#include <linux/blk-mq-dma.h> #include "blk.h" -struct phys_vec { -phys_addr_t paddr; -u32 len; -}; -static bool blk_map_iter_next (struct request *req,struct req_iterator *iter,struct phys_vec *vec) {
include/linux/types.h +5 -0
@@ -170,6 +170,11 @@ typedef u64 phys_addr_t;typedef u32 phys_addr_t ;#endif +struct phys_vec { +phys_addr_t paddr; +u32 len; +}; +typedef phys_addr_t resource_size_t ;/*
Signature · 1 line
2.50.1
PATCH 9/10 vfio/pci: Share the core device pointer while invoking feature functions
In reply to Leon RomanovskyFrom: Vivek Kasireddy <[email protected]>
There is no need to share the main device pointer (struct vfio_device *) with all the feature functions as they only need the core device pointer. Therefore, extract the core device pointer once in the caller (vfio_pci_core_ioctl_feature) and share it instead.
--- drivers/vfio/pci/vfio_pci_core.c | 30 +++++++++++++----------------- 1 file changed, 13 insertions(+), 17 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +13 -17
@@ -300,11 +300,9 @@ static int vfio_pci_runtime_pm_entry(struct vfio_pci_core_device *vdev,return 0 ; } -static int vfio_pci_core_pm_entry (struct vfio_device *device,u32 flags, +static int vfio_pci_core_pm_entry (struct vfio_pci_core_device *vdev,u32 flags,void __user *arg,size_t argsz) { -struct vfio_pci_core_device *vdev = - container_of(device, struct vfio_pci_core_device, vdev);int ret; ret =vfio_check_feature (flags, argsz,VFIO_DEVICE_FEATURE_SET ,0 );
@@ -321,12 +319,10 @@ static int vfio_pci_core_pm_entry(struct vfio_device *device, u32 flags, }static int vfio_pci_core_pm_entry_with_wakeup ( -struct vfio_device *device,u32 flags, +struct vfio_pci_core_device *vdev,u32 flags,struct vfio_device_low_power_entry_with_wakeup __user *arg,size_t argsz) { -struct vfio_pci_core_device *vdev = - container_of(device, struct vfio_pci_core_device, vdev);struct vfio_device_low_power_entry_with_wakeup entry;struct eventfd_ctx *efdctx;int ret;
@@ -377,11 +373,9 @@ static void vfio_pci_runtime_pm_exit(struct vfio_pci_core_device *vdev)up_write (&vdev->memory_lock); } -static int vfio_pci_core_pm_exit (struct vfio_device *device,u32 flags, +static int vfio_pci_core_pm_exit (struct vfio_pci_core_device *vdev,u32 flags,void __user *arg,size_t argsz) { -struct vfio_pci_core_device *vdev = - container_of(device, struct vfio_pci_core_device, vdev);int ret; ret =vfio_check_feature (flags, argsz,VFIO_DEVICE_FEATURE_SET ,0 );
@@ -1474,11 +1468,10 @@ long vfio_pci_core_ioctl(struct vfio_device *core_vdev, unsigned int cmd, }EXPORT_SYMBOL_GPL (vfio_pci_core_ioctl); -static int vfio_pci_core_feature_token (struct vfio_device *device,u32 flags, -uuid_t __user *arg,size_t argsz) +static int vfio_pci_core_feature_token (struct vfio_pci_core_device *vdev, +u32 flags,uuid_t __user *arg, +size_t argsz) { -struct vfio_pci_core_device *vdev = - container_of(device, struct vfio_pci_core_device, vdev);uuid_t uuid;int ret;
@@ -1505,16 +1498,19 @@ static int vfio_pci_core_feature_token(struct vfio_device *device, u32 flags,int vfio_pci_core_ioctl_feature (struct vfio_device *device,u32 flags,void __user *arg,size_t argsz) { +struct vfio_pci_core_device *vdev = + container_of(device, struct vfio_pci_core_device, vdev); +switch (flags &VFIO_DEVICE_FEATURE_MASK ) {case VFIO_DEVICE_FEATURE_LOW_POWER_ENTRY : -return vfio_pci_core_pm_entry (device, flags, arg, argsz); +return vfio_pci_core_pm_entry (vdev, flags, arg, argsz);case VFIO_DEVICE_FEATURE_LOW_POWER_ENTRY_WITH_WAKEUP : -return vfio_pci_core_pm_entry_with_wakeup (device, flags, +return vfio_pci_core_pm_entry_with_wakeup (vdev, flags, arg, argsz);case VFIO_DEVICE_FEATURE_LOW_POWER_EXIT : -return vfio_pci_core_pm_exit (device, flags, arg, argsz); +return vfio_pci_core_pm_exit (vdev, flags, arg, argsz);case VFIO_DEVICE_FEATURE_PCI_VF_TOKEN : -return vfio_pci_core_feature_token (device, flags, arg, argsz); +return vfio_pci_core_feature_token (vdev, flags, arg, argsz);default :return -ENOTTY ; }
Signature · 1 line
2.50.1
PATCH 10/10 vfio/pci: Add dma-buf export support for MMIO regions
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Add support for exporting PCI device MMIO regions through dma-buf, enabling safe sharing of non-struct page memory with controlled lifetime management. This allows RDMA and other subsystems to import dma-buf FDs and build them into memory regions for PCI P2P operations.
The implementation provides a revocable attachment mechanism using dma-buf move operations. MMIO regions are normally pinned as BARs don't change physical addresses, but access is revoked when the VFIO device is closed or a PCI reset is issued. This ensures kernel self-defense against potentially hostile userspace.
--- drivers/vfio/pci/Kconfig | 20 ++ drivers/vfio/pci/Makefile | 2 + drivers/vfio/pci/vfio_pci_config.c | 22 +- drivers/vfio/pci/vfio_pci_core.c | 25 +- drivers/vfio/pci/vfio_pci_dmabuf.c | 390 +++++++++++++++++++++++++++++ drivers/vfio/pci/vfio_pci_priv.h | 23 ++ include/linux/dma-buf.h | 1 + include/linux/vfio_pci_core.h | 3 + include/uapi/linux/vfio.h | 25 ++ 9 files changed, 506 insertions(+), 5 deletions(-) create mode 100644 drivers/vfio/pci/vfio_pci_dmabuf.c
drivers/vfio/pci/Kconfig +20 -0
@@ -55,6 +55,26 @@ config VFIO_PCI_ZDEV_KVM To enable s390x KVM vfio-pci extensions, say Y. +config VFIO_PCI_DMABUF + bool "VFIO PCI extensions for DMA-BUF" + depends on VFIO_PCI_CORE + depends on PCI_P2PDMA && DMA_SHARED_BUFFER + default y + help + Enable support for VFIO PCI extensions that allow exporting + device MMIO regions as DMA-BUFs for peer devices to access via + peer-to-peer (P2P) DMA. + + This feature enables a VFIO-managed PCI device to export a portion + of its MMIO BAR as a DMA-BUF file descriptor, which can be passed + to other userspace drivers or kernel subsystems capable of + initiating DMA to that region. + + Say Y here if you want to enable VFIO DMABUF-based MMIO export + support for peer-to-peer DMA use cases. + + If unsure, say N. + source "drivers/vfio/pci/mlx5/Kconfig" source "drivers/vfio/pci/hisilicon/Kconfig"
drivers/vfio/pci/Makefile +2 -0
@@ -2,7 +2,9 @@ vfio-pci-core-y := vfio_pci_core.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o vfio-pci-core-$(CONFIG_VFIO_PCI_ZDEV_KVM) += vfio_pci_zdev.o + obj-$(CONFIG_VFIO_PCI_CORE) += vfio-pci-core.o +vfio-pci-core-$(CONFIG_VFIO_PCI_DMABUF) += vfio_pci_dmabuf.o vfio-pci-y := vfio_pci.o vfio-pci-$(CONFIG_VFIO_PCI_IGD) += vfio_pci_igd.o
drivers/vfio/pci/vfio_pci_config.c +18 -4
@@ -589,10 +589,12 @@ static int vfio_basic_config_write(struct vfio_pci_core_device *vdev, int pos, virt_mem = !!(le16_to_cpu (*virt_cmd) &PCI_COMMAND_MEMORY ); new_mem = !!(new_cmd &PCI_COMMAND_MEMORY ); -if (!new_mem) +if (!new_mem) {vfio_pci_zap_and_down_write_memory_lock (vdev); -else +vfio_pci_dma_buf_move (vdev, true); + }else {down_write (&vdev->memory_lock); + }/* * If the user is writing mem/io enable (new_mem/io) and we
@@ -627,6 +629,8 @@ static int vfio_basic_config_write(struct vfio_pci_core_device *vdev, int pos, *virt_cmd &=cpu_to_le16 (~mask); *virt_cmd |=cpu_to_le16 (new_cmd & mask); +if (__vfio_pci_memory_enabled (vdev)) +vfio_pci_dma_buf_move (vdev, false);up_write (&vdev->memory_lock); }
@@ -707,12 +711,16 @@ static int __init init_pci_cap_basic_perm(struct perm_bits *perm)static void vfio_lock_and_set_power_state (struct vfio_pci_core_device *vdev,pci_power_t state) { -if (state >= PCI_D3hot) +if (state >= PCI_D3hot) {vfio_pci_zap_and_down_write_memory_lock (vdev); -else +vfio_pci_dma_buf_move (vdev, true); + }else {down_write (&vdev->memory_lock); + }vfio_pci_set_power_state (vdev, state); +if (__vfio_pci_memory_enabled (vdev)) +vfio_pci_dma_buf_move (vdev, false);up_write (&vdev->memory_lock); }
@@ -900,7 +908,10 @@ static int vfio_exp_config_write(struct vfio_pci_core_device *vdev, int pos,if (!ret && (cap &PCI_EXP_DEVCAP_FLR )) {vfio_pci_zap_and_down_write_memory_lock (vdev); +vfio_pci_dma_buf_move (vdev, true);pci_try_reset_function (vdev->pdev); +if (__vfio_pci_memory_enabled (vdev)) +vfio_pci_dma_buf_move (vdev, true);up_write (&vdev->memory_lock); } }
@@ -982,7 +993,10 @@ static int vfio_af_config_write(struct vfio_pci_core_device *vdev, int pos,if (!ret && (cap &PCI_AF_CAP_FLR ) && (cap &PCI_AF_CAP_TP )) {vfio_pci_zap_and_down_write_memory_lock (vdev); +vfio_pci_dma_buf_move (vdev, true);pci_try_reset_function (vdev->pdev); +if (__vfio_pci_memory_enabled (vdev)) +vfio_pci_dma_buf_move (vdev, true);up_write (&vdev->memory_lock); } }
drivers/vfio/pci/vfio_pci_core.c +24 -1
@@ -28,7 +28,9 @@#include <linux/nospec.h> #include <linux/sched/mm.h> #include <linux/iommufd.h> +#ifdef CONFIG_VFIO_PCI_DMABUF #include <linux/pci-p2pdma.h> +#endif #if IS_ENABLED (CONFIG_EEH )#include <asm/eeh.h> #endif
@@ -287,6 +289,8 @@ static int vfio_pci_runtime_pm_entry(struct vfio_pci_core_device *vdev,* semaphore. */ vfio_pci_zap_and_down_write_memory_lock (vdev); +vfio_pci_dma_buf_move (vdev, true); +if (vdev->pm_runtime_engaged) {up_write (&vdev->memory_lock);return -EINVAL ;
@@ -370,6 +374,8 @@ static void vfio_pci_runtime_pm_exit(struct vfio_pci_core_device *vdev)*/ down_write (&vdev->memory_lock);__vfio_pci_runtime_pm_exit (vdev); +if (__vfio_pci_memory_enabled (vdev)) +vfio_pci_dma_buf_move (vdev, false);up_write (&vdev->memory_lock); }
@@ -690,6 +696,8 @@ void vfio_pci_core_close_device(struct vfio_device *core_vdev)#endif vfio_pci_core_disable (vdev); +vfio_pci_dma_buf_cleanup (vdev); +mutex_lock (&vdev->igate);if (vdev->err_trigger) {eventfd_ctx_put (vdev->err_trigger);
@@ -1222,7 +1230,10 @@ static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,*/ vfio_pci_set_power_state (vdev,PCI_D0 ); +vfio_pci_dma_buf_move (vdev, true); ret =pci_try_reset_function (vdev->pdev); +if (__vfio_pci_memory_enabled (vdev)) +vfio_pci_dma_buf_move (vdev, false);up_write (&vdev->memory_lock);return ret;
@@ -1511,6 +1522,8 @@ int vfio_pci_core_ioctl_feature(struct vfio_device *device, u32 flags,return vfio_pci_core_pm_exit (vdev, flags, arg, argsz);case VFIO_DEVICE_FEATURE_PCI_VF_TOKEN :return vfio_pci_core_feature_token (vdev, flags, arg, argsz); +case VFIO_DEVICE_FEATURE_DMA_BUF : +return vfio_pci_core_feature_dma_buf (vdev, flags, arg, argsz);default :return -ENOTTY ; }
@@ -2085,9 +2098,13 @@ int vfio_pci_core_init_dev(struct vfio_device *core_vdev)INIT_LIST_HEAD (&vdev->dummy_resources_list);INIT_LIST_HEAD (&vdev->ioeventfds_list);INIT_LIST_HEAD (&vdev->sriov_pfs_item); +#ifdef CONFIG_VFIO_PCI_DMABUF vdev->provider =pci_p2pdma_enable (vdev->pdev);if (IS_ERR (vdev->provider))return PTR_ERR (vdev->provider); + +INIT_LIST_HEAD (&vdev->dmabufs); +#endif init_rwsem (&vdev->memory_lock);xa_init (&vdev->ctx);
@@ -2470,11 +2487,17 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,* cause the PCI config space reset without restoring the original * state (saved locally in 'vdev->pm_save'). */ -list_for_each_entry (vdev, &dev_set->device_list, vdev.dev_set_list) +list_for_each_entry (vdev, &dev_set->device_list, vdev.dev_set_list) { +vfio_pci_dma_buf_move (vdev, true);vfio_pci_set_power_state (vdev,PCI_D0 ); + } ret =pci_reset_bus (pdev); +list_for_each_entry (vdev, &dev_set->device_list, vdev.dev_set_list) +if (__vfio_pci_memory_enabled (vdev)) +vfio_pci_dma_buf_move (vdev, false); + vdev =list_last_entry (&dev_set->device_list, struct vfio_pci_core_device, vdev.dev_set_list);
drivers/vfio/pci/vfio_pci_dmabuf.c new file +390 -0
390 lines not on this page · raw message
drivers/vfio/pci/vfio_pci_priv.h +23 -0
@@ -107,4 +107,27 @@ static inline bool vfio_pci_is_vga(struct pci_dev *pdev) return (pdev->class >> 8) == PCI_CLASS_DISPLAY_VGA; } +#ifdef CONFIG_VFIO_PCI_DMABUF +int vfio_pci_core_feature_dma_buf (struct vfio_pci_core_device *vdev,u32 flags, +struct vfio_device_feature_dma_buf __user *arg, +size_t argsz); +void vfio_pci_dma_buf_cleanup (struct vfio_pci_core_device *vdev); +void vfio_pci_dma_buf_move (struct vfio_pci_core_device *vdev,bool revoked); +#else +static inline int +vfio_pci_core_feature_dma_buf (struct vfio_pci_core_device *vdev,u32 flags, +struct vfio_device_feature_dma_buf __user *arg, +size_t argsz) +{ +return -ENOTTY ; +} +static inline void vfio_pci_dma_buf_cleanup (struct vfio_pci_core_device *vdev) +{ +} +static inline void vfio_pci_dma_buf_move (struct vfio_pci_core_device *vdev, +bool revoked) +{ +} +#endif +#endif
include/linux/dma-buf.h +1 -0
@@ -483,6 +483,7 @@ struct dma_buf_attach_ops {* @dev: device attached to the buffer. * @node: list of dma_buf_attachment, protected by dma_resv lock of the dmabuf. * @peer2peer: true if the importer can handle peer resources without pages. +* #state: DMA structure to provide support for physical addresses DMA interface * @priv: exporter specific attachment data. * @importer_ops: importer operations for this attachment, if provided * dma_buf_map/unmap_attachment() must be called with the dma_resv lock held.
include/linux/vfio_pci_core.h +3 -0
@@ -94,7 +94,10 @@ struct vfio_pci_core_device {struct vfio_pci_core_device *sriov_pf_core_dev;struct notifier_block nb;struct rw_semaphore memory_lock; +#ifdef CONFIG_VFIO_PCI_DMABUF struct p2pdma_provider *provider; +struct list_head dmabufs; +#endif };/* Will be exported for vfio pci drivers usage */
include/uapi/linux/vfio.h +25 -0
@@ -1468,6 +1468,31 @@ struct vfio_device_feature_bus_master { };#define VFIO_DEVICE_FEATURE_BUS_MASTER 10 +/** +* Upon VFIO_DEVICE_FEATURE_GET create a dma_buf fd for the +* regions selected. +* +* open_flags are the typical flags passed to open(2), eg O_RDWR, O_CLOEXEC, +* etc. offset/length specify a slice of the region to create the dmabuf from. +* nr_ranges is the total number of (P2P DMA) ranges that comprise the dmabuf. +* +* Return: The fd number on success, -1 and errno is set on failure. +*/ +#define VFIO_DEVICE_FEATURE_DMA_BUF 11 + +struct vfio_region_dma_range { +__u64 offset; +__u64 length; +}; + +struct vfio_device_feature_dma_buf { +__u32 region_index; +__u32 open_flags; +__u32 flags; +__u32 nr_ranges; +struct vfio_region_dma_range dma_ranges[]; +}; +/* -------- API for Type1 VFIO IOMMU -------- */ /**
Signature · 1 line
2.50.1
PATCH 8/10 vfio/pci: Enable peer-to-peer DMA transactions by default
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Make sure that all VFIO PCI devices have peer-to-peer capabilities enables, so we would be able to export their MMIO memory through DMABUF,
--- drivers/vfio/pci/vfio_pci_core.c | 4 ++++ include/linux/vfio_pci_core.h | 1 + 2 files changed, 5 insertions(+)
drivers/vfio/pci/vfio_pci_core.c +4 -0
@@ -28,6 +28,7 @@#include <linux/nospec.h> #include <linux/sched/mm.h> #include <linux/iommufd.h> +#include <linux/pci-p2pdma.h> #if IS_ENABLED (CONFIG_EEH )#include <asm/eeh.h> #endif
@@ -2088,6 +2089,9 @@ int vfio_pci_core_init_dev(struct vfio_device *core_vdev)INIT_LIST_HEAD (&vdev->dummy_resources_list);INIT_LIST_HEAD (&vdev->ioeventfds_list);INIT_LIST_HEAD (&vdev->sriov_pfs_item); + vdev->provider =pci_p2pdma_enable (vdev->pdev); +if (IS_ERR (vdev->provider)) +return PTR_ERR (vdev->provider);init_rwsem (&vdev->memory_lock);xa_init (&vdev->ctx);
include/linux/vfio_pci_core.h +1 -0
@@ -94,6 +94,7 @@ struct vfio_pci_core_device {struct vfio_pci_core_device *sriov_pf_core_dev;struct notifier_block nb;struct rw_semaphore memory_lock; +struct p2pdma_provider *provider; };/* Will be exported for vfio pci drivers usage */
Signature · 1 line
2.50.1
FYI: this entire patch series was rejected as spam by large numbers of linux-mm subscribers using @gmail.com email addresses.
-ben (owner-linux-mm)
Quoted 94 lines · Leon Romanovsky
Changelog:
v1:
* Changed commit messages.
* ReusedDMA_ATTR_MMIOattribute.
* Returned support for multiple DMA ranges per-dMABUF.
v0: https://lore.kernel.org/all/[email protected]---------------------------------------------------------------------------
Based on "[PATCH v1 00/16] dma-mapping: migrate to physical address-based API"
https://lore.kernel.org/all/[email protected] series.
---------------------------------------------------------------------------This series extends the VFIO PCI subsystem to support exporting MMIO regions from PCI device BARs as dma-buf objects, enabling safe sharing of non-struct page memory with controlled lifetime management. This allows RDMA and other subsystems to import dma-buf FDs and build them into memory regions for PCI P2P operations.
The series supports a use case for SPDK where a NVMe device will be owned by SPDK through VFIO but interacting with a RDMA device. The RDMA device may directly access the NVMe CMB or directly manipulate the NVMe device's doorbell using PCI P2P.
However, as a general mechanism, it can support many other scenarios with VFIO. This dmabuf approach can be usable by iommufd as well for generic and safe P2P mappings.
In addition to the SPDK use-case mentioned above, the capability added in this patch series can also be useful when a buffer (located in device memory such as VRAM) needs to be shared between any two dGPU devices or instances (assuming one of them is bound to VFIO PCI) as long as they are P2P DMA compatible.
The implementation provides a revocable attachment mechanism using dma-buf move operations. MMIO regions are normally pinned as BARs don't change physical addresses, but access is revoked when the VFIO device is closed or a PCI reset is issued. This ensures kernel self-defense against potentially hostile userspace.
The series includes significant refactoring of the PCI P2PDMA subsystem to separate core P2P functionality from memory allocation features, making it more modular and suitable for VFIO use cases that don't need struct page support.
-----------------------------------------------------------------------
The series is based originally on
https://lore.kernel.org/all/[email protected]/
but heavily rewritten to be based on DMA physical API.
-----------------------------------------------------------------------
The WIP branch can be found here:
https://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git/log/?h=dmabuf-vfio-v1Thanks
Leon Romanovsky (8):
PCI/P2PDMA: Remove redundantbus_offsetfrom map state PCI/P2PDMA: Separate themmap()support from the core logic PCI/P2PDMA: Simplify bus address mapping API
PCI/P2PDMA: Refactor to separate core P2P functionality from memory
allocation
PCI/P2PDMA: Exportpci_p2pdma_map_type()function
types: movephys_vecdefinition to common header vfio/pci: Enable peer-to-peer DMA transactions by default vfio/pci: Add dma-buf export support for MMIO regionsVivek Kasireddy (2):
vfio: Export vfio device get and put registration helpers vfio/pci: Share the core device pointer while invoking feature
functionsblock/blk-mq-dma.c | 7 +- drivers/iommu/dma-iommu.c | 4 +- drivers/pci/p2pdma.c | 154 ++++++++---- drivers/vfio/pci/Kconfig | 20 ++ drivers/vfio/pci/Makefile | 2 + drivers/vfio/pci/vfio_pci_config.c | 22 +- drivers/vfio/pci/vfio_pci_core.c | 59 +++-- drivers/vfio/pci/vfio_pci_dmabuf.c | 390 +++++++++++++++++++++++++++++ drivers/vfio/pci/vfio_pci_priv.h | 23 ++ drivers/vfio/vfio_main.c | 2 + include/linux/dma-buf.h | 1 + include/linux/pci-p2pdma.h | 114 +++++---- include/linux/types.h | 5 + include/linux/vfio.h | 2 + include/linux/vfio_pci_core.h | 4 + include/uapi/linux/vfio.h | 25 ++ kernel/dma/direct.c | 4 +- mm/hmm.c | 2 +- 18 files changed, 715 insertions(+), 125 deletions(-) create mode 100644 drivers/vfio/pci/vfio_pci_dmabuf.cSignature · 1 line
2.50.1
Signature · 1 line
"Thought is the essence of where you are now."
Quoted 2 lines · Benjamin LaHaise
FYI: this entire patch series was rejected as spam by large numbers of linux-mm subscribers using @gmail.com email addresses.
Thanks for the heads-up. Are you aware of any issues from my side? I'm sending patches with git-send-email through mail.kernel.org SMTP.
Thanks
Quoted 2 lines
-ben (owner-linux-mm)
Re: [PATCH v1 10/10] vfio/pci: Add dma-buf export support for MMIO regions
In reply to Leon RomanovskyQuoted 28 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Add support for exporting PCI device MMIO regions through dma-buf, enabling safe sharing of non-struct page memory with controlled lifetime management. This allows RDMA and other subsystems to import dma-buf FDs and build them into memory regions for PCI P2P operations.
The implementation provides a revocable attachment mechanism using dma-buf move operations. MMIO regions are normally pinned as BARs don't change physical addresses, but access is revoked when the VFIO device is closed or a PCI reset is issued. This ensures kernel self-defense against potentially hostile userspace.
Signed-off-by Jason Gunthorpe <[email protected]>Signed-off-by Vivek Kasireddy <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>--- drivers/vfio/pci/Kconfig | 20 ++ drivers/vfio/pci/Makefile | 2 + drivers/vfio/pci/vfio_pci_config.c | 22 +- drivers/vfio/pci/vfio_pci_core.c | 25 +- drivers/vfio/pci/vfio_pci_dmabuf.c | 390 +++++++++++++++++++++++++++++ drivers/vfio/pci/vfio_pci_priv.h | 23 ++ include/linux/dma-buf.h | 1 + include/linux/vfio_pci_core.h | 3 + include/uapi/linux/vfio.h | 25 ++ 9 files changed, 506 insertions(+), 5 deletions(-) create mode 100644 drivers/vfio/pci/vfio_pci_dmabuf.c
<...>
include/linux/dma-buf.h struct dma_buf_attach_ops { · line 486
Quoted 9 lines
include/linux/dma-buf.h +1
-0@@ -483,6 +483,7 @@ struct dma_buf_attach_ops {* @dev: device attached to the buffer. * @node: list of dma_buf_attachment, protected by dma_resv lock of the dmabuf. * @peer2peer: true if the importer can handle peer resources without pages. +* #state: DMA structure to provide support for physical addresses DMA interface
This is rebase error, there is no need in this hunk.
Thanks
Re: [PATCH v1 01/10] PCI/P2PDMA: Remove redundant bus_offset from map state
In reply to Leon RomanovskyQuoted 40 lines · On Mon, 4 Aug 2025 16:00:36 +0300 Leon Romanovsky <[email protected]> wrote:
From: Leon Romanovsky <[email protected]>
Remove the
bus_offfield frompci_p2pdma_map_statesince it duplicates information already available in the pgmap structure. Thebus_offsetis only used in one location (pci_p2pdma_bus_addr_map) and is always identical topgmap->bus_offset.Reviewed-by Christoph Hellwig <[email protected]>Signed-off-by Jason Gunthorpe <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>--- drivers/pci/p2pdma.c | 1 - include/linux/pci-p2pdma.h | 3 +-- 2 files changed, 1 insertion(+), 3 deletions(-)drivers/pci/p2pdma.c +0
-1@@ -1009,7 +1009,6 @@ void __pci_p2pdma_update_state(struct pci_p2pdma_map_state *state, { state->pgmap =page_pgmap (page); state->map =pci_p2pdma_map_type (state->pgmap, dev); - state->bus_off =to_p2p_pgmap (state->pgmap)->bus_offset; }/** include/linux/pci-p2pdma.h +0
-1@@ -146,7 +146,6 @@ enum pci_p2pdma_map_type {struct pci_p2pdma_map_state {struct dev_pagemap *pgmap;enum pci_p2pdma_map_type map; -u64 bus_off; };/* helper for pci_p2pdma_state(), do not use directly */
include/linux/pci-p2pdma.h static inline dma_addr_t · line 191
Quoted 8 lines
include/linux/pci-p2pdma.h +1
-1@@ -186,7 +185,7 @@ static inline dma_addr_t pci_p2pdma_bus_addr_map(struct pci_p2pdma_map_state *state, phys_addr_t paddr) { WARN_ON_ONCE(state->map != PCI_P2PDMA_MAP_BUS_ADDR); -return paddr + state->bus_off; +return paddr +to_p2p_pgmap (state->pgmap)->bus_offsetf; }#endif /* _LINUX_PCI_P2P_H */
Looks like you're relying on this bogus code getting resolved in the next patch...
In file included from kernel/dma/direct.c:16:./include/linux/pci-p2pdma.h: In function ‘pci_p2pdma_bus_addr_map’:./include/linux/pci-p2pdma.h:188:24: error: implicit declaration of function ‘to_p2p_pgmap’ [-Wimplicit-function-declaration]
188 | return paddr + to_p2p_pgmap(state->pgmap)->bus_offsetf;
| ^~~~~~~~~~~~./include/linux/pci-p2pdma.h:188:50: error: invalid type argument of ‘->’ (have ‘int’)
188 | return paddr + to_p2p_pgmap(state->pgmap)->bus_offsetf;
| ^~./include/linux/pci-p2pdma.h:189:1: error: control reaches end of non-void function [-Werror=return-type]
189 | }
| ^
to_p2p_pgmap() is a static function and struct pci_p2pdma_pagemap doesn't have a bus_offsetf member. Thanks,
Alex
Re: [PATCH v1 04/10] PCI/P2PDMA: Refactor to separate core P2P functionality from memory allocation
In reply to Leon RomanovskyQuoted 90 lines · On Mon, 4 Aug 2025 16:00:39 +0300 Leon Romanovsky <[email protected]> wrote:
From: Leon Romanovsky <[email protected]>
Refactor the PCI P2PDMA subsystem to separate the core peer-to-peer DMA functionality from the optional memory allocation layer. This creates a two-tier architecture:
The core layer provides P2P mapping functionality for physical addresses based on PCI device MMIO BARs and integrates with the DMA API for mapping operations. This layer is required for all P2PDMA users.
The optional upper layer provides memory allocation capabilities including
gen_poolallocator, struct page support, and sysfs interface for user space access.This separation allows subsystems like VFIO to use only the core P2P mapping functionality without the overhead of memory allocation features they don't need. The core functionality is now available through the new
pci_p2pdma_enable()function that returns ap2pdma_providerstructure.Signed-off-by Leon Romanovsky <[email protected]>--- drivers/pci/p2pdma.c | 118 ++++++++++++++++++++++++++----------- include/linux/pci-p2pdma.h | 5 ++ 2 files changed, 89 insertions(+), 34 deletions(-)drivers/pci/p2pdma.c +24
-6@@ -25,11 +25,12 @@ struct pci_p2pdma {struct gen_pool *pool;bool p2pmem_published;struct xarray map_types; +struct p2pdma_provider mem; };struct pci_p2pdma_pagemap {struct dev_pagemap pgmap; -struct p2pdma_provider mem; +struct p2pdma_provider *mem; };static struct pci_p2pdma_pagemap *to_p2p_pgmap (struct dev_pagemap *pgmap)@@ -204,7 +205,7 @@ static void p2pdma_page_free(struct page *page)struct pci_p2pdma_pagemap *pgmap =to_p2p_pgmap (page_pgmap (page));/* safe to dereference while a reference is held to the percpu ref */ struct pci_p2pdma *p2pdma =rcu_dereference_protected ( -to_pci_dev (pgmap->mem.owner)->p2pdma,1 ); +to_pci_dev (pgmap->mem->owner)->p2pdma,1 );struct percpu_ref *ref; gen_pool_free_owner(p2pdma->pool, (uintptr_t)page_to_virt(page),@@ -227,44 +228,82 @@ static void pci_p2pdma_release(void *data)/* Flush and disable pci_alloc_p2p_mem() */ pdev->p2pdma =NULL ; -synchronize_rcu (); +if (p2pdma->pool) +synchronize_rcu (); +xa_destroy (&p2pdma->map_types); + +if (!p2pdma->pool) +return ;gen_pool_destroy (p2pdma->pool);sysfs_remove_group (&pdev->dev.kobj, &p2pmem_group); -xa_destroy (&p2pdma->map_types); } -static int pci_p2pdma_setup (struct pci_dev *pdev) +/** +* pci_p2pdma_enable - Enable peer-to-peer DMA support for a PCI device +* @pdev: The PCI device to enable P2PDMA for +* +* This function initializes the peer-to-peer DMA infrastructure for a PCI +* device. It allocates and sets up the necessary data structures to support +* P2PDMA operations, including mapping type tracking. +*/ +struct p2pdma_provider *pci_p2pdma_enable (struct pci_dev *pdev) { -int error = -ENOMEM ;struct pci_p2pdma *p2p; +int ret; + + p2p =rcu_dereference_protected (pdev->p2pdma,1 ); +if (p2p) +/* PCI device was "rebound" to the driver */ +return &p2p->mem;
drivers/pci/p2pdma.c static void pci_p2pdma_release(void *data) · line 268
Quoted 10 lines
drivers/pci/p2pdma.c +5
-1p2p =devm_kzalloc (&pdev->dev,sizeof (*p2p),GFP_KERNEL );if (!p2p) -return -ENOMEM ; +return ERR_PTR (-ENOMEM );xa_init (&p2p->map_types); + p2p->mem.owner = &pdev->dev; +/* On all p2p platforms bus_offset is the same for all BARs */ + p2p->mem.bus_offset = +pci_bus_address (pdev,0 ) -pci_resource_start (pdev,0 );
But not all devices implement BAR0, nor is BAR0 necessarily in the memory space, wouldn't this calculation be wrong if BAR0 were unimplemented or an IO BAR? Even within memory BARs I can imagine different translations for 32 vs 64 bit, prefetch vs non-prefetch, but per the comment I guess we're excluding those. Thanks,
Alex
Re: [PATCH v1 08/10] vfio/pci: Enable peer-to-peer DMA transactions by default
In reply to Leon RomanovskyQuoted 24 lines · On Mon, 4 Aug 2025 16:00:43 +0300 Leon Romanovsky <[email protected]> wrote:
From: Leon Romanovsky <[email protected]>
Make sure that all VFIO PCI devices have peer-to-peer capabilities enables, so we would be able to export their MMIO memory through DMABUF,
Signed-off-by Leon Romanovsky <[email protected]>--- drivers/vfio/pci/vfio_pci_core.c | 4 ++++ include/linux/vfio_pci_core.h | 1 + 2 files changed, 5 insertions(+)drivers/vfio/pci/vfio_pci_core.c +1
-0@@ -28,6 +28,7 @@#include <linux/nospec.h> #include <linux/sched/mm.h> #include <linux/iommufd.h> +#include <linux/pci-p2pdma.h> #if IS_ENABLED (CONFIG_EEH )#include <asm/eeh.h> #endif
drivers/vfio/pci/vfio_pci_core.c int vfio_pci_core_init_dev(struct vfio_device *core_vdev) · line 2094
Quoted 6 lines
drivers/vfio/pci/vfio_pci_core.c +3
-0@@ -2088,6 +2089,9 @@ int vfio_pci_core_init_dev(struct vfio_device *core_vdev)INIT_LIST_HEAD (&vdev->dummy_resources_list);INIT_LIST_HEAD (&vdev->ioeventfds_list);INIT_LIST_HEAD (&vdev->sriov_pfs_item); + vdev->provider =pci_p2pdma_enable (vdev->pdev); +if (IS_ERR (vdev->provider)) +return PTR_ERR (vdev->provider);
I think this just made all vfio-pci drivers functionally dependent on CONFIG_PCI_P2PDMA. Seems at best exporting a dma-buf should be restricted if this fails. Thanks,
Alex
Quoted 15 lines
drivers/vfio/pci/vfio_pci_core.c +0
-0init_rwsem (&vdev->memory_lock);xa_init (&vdev->ctx);include/linux/vfio_pci_core.h +1
-0@@ -94,6 +94,7 @@ struct vfio_pci_core_device {struct vfio_pci_core_device *sriov_pf_core_dev;struct notifier_block nb;struct rw_semaphore memory_lock; +struct p2pdma_provider *provider; };/* Will be exported for vfio pci drivers usage */
Re: [PATCH v1 10/10] vfio/pci: Add dma-buf export support for MMIO regions
In reply to Leon RomanovskyQuoted 122 lines · On Mon, 4 Aug 2025 16:00:45 +0300 Leon Romanovsky <[email protected]> wrote:
From: Leon Romanovsky <[email protected]>
Add support for exporting PCI device MMIO regions through dma-buf, enabling safe sharing of non-struct page memory with controlled lifetime management. This allows RDMA and other subsystems to import dma-buf FDs and build them into memory regions for PCI P2P operations.
The implementation provides a revocable attachment mechanism using dma-buf move operations. MMIO regions are normally pinned as BARs don't change physical addresses, but access is revoked when the VFIO device is closed or a PCI reset is issued. This ensures kernel self-defense against potentially hostile userspace.
Signed-off-by Jason Gunthorpe <[email protected]>Signed-off-by Vivek Kasireddy <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>--- drivers/vfio/pci/Kconfig | 20 ++ drivers/vfio/pci/Makefile | 2 + drivers/vfio/pci/vfio_pci_config.c | 22 +- drivers/vfio/pci/vfio_pci_core.c | 25 +- drivers/vfio/pci/vfio_pci_dmabuf.c | 390 +++++++++++++++++++++++++++++ drivers/vfio/pci/vfio_pci_priv.h | 23 ++ include/linux/dma-buf.h | 1 + include/linux/vfio_pci_core.h | 3 + include/uapi/linux/vfio.h | 25 ++ 9 files changed, 506 insertions(+), 5 deletions(-) create mode 100644 drivers/vfio/pci/vfio_pci_dmabuf.cdrivers/vfio/pci/Kconfig +20
-0@@ -55,6 +55,26 @@ config VFIO_PCI_ZDEV_KVM To enable s390x KVM vfio-pci extensions, say Y. +config VFIO_PCI_DMABUF + bool "VFIO PCI extensions for DMA-BUF" + depends on VFIO_PCI_CORE + depends on PCI_P2PDMA && DMA_SHARED_BUFFER + default y + help + Enable support for VFIO PCI extensions that allow exporting + device MMIO regions as DMA-BUFs for peer devices to access via + peer-to-peer (P2P) DMA. + + This feature enables a VFIO-managed PCI device to export a portion + of its MMIO BAR as a DMA-BUF file descriptor, which can be passed + to other userspace drivers or kernel subsystems capable of + initiating DMA to that region. + + Say Y here if you want to enable VFIO DMABUF-based MMIO export + support for peer-to-peer DMA use cases. + + If unsure, say N. + source "drivers/vfio/pci/mlx5/Kconfig" source "drivers/vfio/pci/hisilicon/Kconfig"drivers/vfio/pci/Makefile +2
-0@@ -2,7 +2,9 @@ vfio-pci-core-y := vfio_pci_core.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o vfio-pci-core-$(CONFIG_VFIO_PCI_ZDEV_KVM) += vfio_pci_zdev.o + obj-$(CONFIG_VFIO_PCI_CORE) += vfio-pci-core.o +vfio-pci-core-$(CONFIG_VFIO_PCI_DMABUF) += vfio_pci_dmabuf.o vfio-pci-y := vfio_pci.o vfio-pci-$(CONFIG_VFIO_PCI_IGD) += vfio_pci_igd.odrivers/vfio/pci/vfio_pci_config.c +12
-4@@ -589,10 +589,12 @@ static int vfio_basic_config_write(struct vfio_pci_core_device *vdev, int pos, virt_mem = !!(le16_to_cpu (*virt_cmd) &PCI_COMMAND_MEMORY ); new_mem = !!(new_cmd &PCI_COMMAND_MEMORY ); -if (!new_mem) +if (!new_mem) {vfio_pci_zap_and_down_write_memory_lock (vdev); -else +vfio_pci_dma_buf_move (vdev, true); + }else {down_write (&vdev->memory_lock); + }/* * If the user is writing mem/io enable (new_mem/io) and we @@ -627,6 +629,8 @@ static int vfio_basic_config_write(struct vfio_pci_core_device *vdev, int pos, *virt_cmd &=cpu_to_le16 (~mask); *virt_cmd |=cpu_to_le16 (new_cmd & mask); +if (__vfio_pci_memory_enabled (vdev)) +vfio_pci_dma_buf_move (vdev, false);up_write (&vdev->memory_lock); }@@ -707,12 +711,16 @@ static int __init init_pci_cap_basic_perm(struct perm_bits *perm)static void vfio_lock_and_set_power_state (struct vfio_pci_core_device *vdev,pci_power_t state) { -if (state >= PCI_D3hot) +if (state >= PCI_D3hot) {vfio_pci_zap_and_down_write_memory_lock (vdev); -else +vfio_pci_dma_buf_move (vdev, true); + }else {down_write (&vdev->memory_lock); + }vfio_pci_set_power_state (vdev, state); +if (__vfio_pci_memory_enabled (vdev)) +vfio_pci_dma_buf_move (vdev, false);up_write (&vdev->memory_lock); }
drivers/vfio/pci/vfio_pci_config.c static int vfio_exp_config_write(struct vfio_pci_core_device *vdev, int pos, · line 914
Quoted 7 lines
drivers/vfio/pci/vfio_pci_config.c +3
-0@@ -900,7 +908,10 @@ static int vfio_exp_config_write(struct vfio_pci_core_device *vdev, int pos,if (!ret && (cap &PCI_EXP_DEVCAP_FLR )) {vfio_pci_zap_and_down_write_memory_lock (vdev); +vfio_pci_dma_buf_move (vdev, true);pci_try_reset_function (vdev->pdev); +if (__vfio_pci_memory_enabled (vdev)) +vfio_pci_dma_buf_move (vdev, true);
@revoked true -> true seems wrong.
drivers/vfio/pci/vfio_pci_config.c static int vfio_af_config_write(struct vfio_pci_core_device *vdev, int pos, · line 999
Quoted 11 lines
drivers/vfio/pci/vfio_pci_config.c +3
-0up_write (&vdev->memory_lock); } }@@ -982,7 +993,10 @@ static int vfio_af_config_write(struct vfio_pci_core_device *vdev, int pos,if (!ret && (cap &PCI_AF_CAP_FLR ) && (cap &PCI_AF_CAP_TP )) {vfio_pci_zap_and_down_write_memory_lock (vdev); +vfio_pci_dma_buf_move (vdev, true);pci_try_reset_function (vdev->pdev); +if (__vfio_pci_memory_enabled (vdev)) +vfio_pci_dma_buf_move (vdev, true);
Same.
Quoted 79 lines
drivers/vfio/pci/vfio_pci_config.c +0
-0up_write (&vdev->memory_lock); } }drivers/vfio/pci/vfio_pci_core.c +17
-0@@ -28,7 +28,9 @@#include <linux/nospec.h> #include <linux/sched/mm.h> #include <linux/iommufd.h> +#ifdef CONFIG_VFIO_PCI_DMABUF #include <linux/pci-p2pdma.h> +#endif #if IS_ENABLED (CONFIG_EEH )#include <asm/eeh.h> #endif @@ -287,6 +289,8 @@ static int vfio_pci_runtime_pm_entry(struct vfio_pci_core_device *vdev,* semaphore. */ vfio_pci_zap_and_down_write_memory_lock (vdev); +vfio_pci_dma_buf_move (vdev, true); +if (vdev->pm_runtime_engaged) {up_write (&vdev->memory_lock);return -EINVAL ;@@ -370,6 +374,8 @@ static void vfio_pci_runtime_pm_exit(struct vfio_pci_core_device *vdev)*/ down_write (&vdev->memory_lock);__vfio_pci_runtime_pm_exit (vdev); +if (__vfio_pci_memory_enabled (vdev)) +vfio_pci_dma_buf_move (vdev, false);up_write (&vdev->memory_lock); }@@ -690,6 +696,8 @@ void vfio_pci_core_close_device(struct vfio_device *core_vdev)#endif vfio_pci_core_disable (vdev); +vfio_pci_dma_buf_cleanup (vdev); +mutex_lock (&vdev->igate);if (vdev->err_trigger) {eventfd_ctx_put (vdev->err_trigger);@@ -1222,7 +1230,10 @@ static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,*/ vfio_pci_set_power_state (vdev,PCI_D0 ); +vfio_pci_dma_buf_move (vdev, true); ret =pci_try_reset_function (vdev->pdev); +if (__vfio_pci_memory_enabled (vdev)) +vfio_pci_dma_buf_move (vdev, false);up_write (&vdev->memory_lock);return ret;@@ -1511,6 +1522,8 @@ int vfio_pci_core_ioctl_feature(struct vfio_device *device, u32 flags,return vfio_pci_core_pm_exit (vdev, flags, arg, argsz);case VFIO_DEVICE_FEATURE_PCI_VF_TOKEN :return vfio_pci_core_feature_token (vdev, flags, arg, argsz); +case VFIO_DEVICE_FEATURE_DMA_BUF : +return vfio_pci_core_feature_dma_buf (vdev, flags, arg, argsz);default :return -ENOTTY ; }@@ -2085,9 +2098,13 @@ int vfio_pci_core_init_dev(struct vfio_device *core_vdev)INIT_LIST_HEAD (&vdev->dummy_resources_list);INIT_LIST_HEAD (&vdev->ioeventfds_list);INIT_LIST_HEAD (&vdev->sriov_pfs_item); +#ifdef CONFIG_VFIO_PCI_DMABUF vdev->provider =pci_p2pdma_enable (vdev->pdev);if (IS_ERR (vdev->provider))return PTR_ERR (vdev->provider); + +INIT_LIST_HEAD (&vdev->dmabufs); +#endif init_rwsem (&vdev->memory_lock);xa_init (&vdev->ctx);
drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, · line 2493
Quoted 8 lines
drivers/vfio/pci/vfio_pci_core.c +3
-1@@ -2470,11 +2487,17 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,* cause the PCI config space reset without restoring the original * state (saved locally in 'vdev->pm_save'). */ -list_for_each_entry (vdev, &dev_set->device_list, vdev.dev_set_list) +list_for_each_entry (vdev, &dev_set->device_list, vdev.dev_set_list) { +vfio_pci_dma_buf_move (vdev, true);vfio_pci_set_power_state (vdev,PCI_D0 ); + }
The revoke should have happened at the time the BARs were zapped. Thanks,
Alex
Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +4
-0ret =pci_reset_bus (pdev); +list_for_each_entry (vdev, &dev_set->device_list, vdev.dev_set_list) +if (__vfio_pci_memory_enabled (vdev)) +vfio_pci_dma_buf_move (vdev, false); + vdev =list_last_entry (&dev_set->device_list, struct vfio_pci_core_device, vdev.dev_set_list);
Jason Gunthorpe
maintainerRe: [PATCH v1 04/10] PCI/P2PDMA: Refactor to separate core P2P functionality from memory allocation
In reply to Alex WilliamsonQuoted 8 lines · Alex Williamson
drivers/pci/p2pdma.c static void pci_p2pdma_release(void *data) · line 268
Quoted 4 lines
drivers/pci/p2pdma.c +4
-0+ p2p->mem.owner = &pdev->dev; +/* On all p2p platforms bus_offset is the same for all BARs */ + p2p->mem.bus_offset = +pci_bus_address (pdev,0 ) -pci_resource_start (pdev,0 );But not all devices implement BAR0, nor is BAR0 necessarily in the memory space, wouldn't this calculation be wrong if BAR0 were unimplemented or an IO BAR?
I think you are correct about this.
Quoted 3 lines
Even within memory BARs I can imagine different translations for 32 vs 64 bit, prefetch vs non-prefetch, but per the comment I guess we're excluding those. Thanks,
Humm, I had thought it was consistent for the device, but I guess not:
resource_list_for_each_entry(window, &bridge->windows) {
if (resource_contains(window->res, res)) {
offset = window->offset;Seems like each window can have its own offset.
So the p2p should be made per-bar, and hold the bar index from the constructor..
Jason
Re: [PATCH v1 01/10] PCI/P2PDMA: Remove redundant bus_offset from map state
In reply to Alex WilliamsonQuoted 50 lines · Alex Williamson
Quoted 47 lines · On Mon, 4 Aug 2025 16:00:36 +0300 Leon Romanovsky <[email protected]> wrote:
From: Leon Romanovsky <[email protected]>
Remove the
bus_offfield frompci_p2pdma_map_statesince it duplicates information already available in the pgmap structure. Thebus_offsetis only used in one location (pci_p2pdma_bus_addr_map) and is always identical topgmap->bus_offset.Reviewed-by Christoph Hellwig <[email protected]>Signed-off-by Jason Gunthorpe <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>--- drivers/pci/p2pdma.c | 1 - include/linux/pci-p2pdma.h | 3 +-- 2 files changed, 1 insertion(+), 3 deletions(-)drivers/pci/p2pdma.c +0
-1@@ -1009,7 +1009,6 @@ void __pci_p2pdma_update_state(struct pci_p2pdma_map_state *state, { state->pgmap =page_pgmap (page); state->map =pci_p2pdma_map_type (state->pgmap, dev); - state->bus_off =to_p2p_pgmap (state->pgmap)->bus_offset; }/** include/linux/pci-p2pdma.h +1
-2@@ -146,7 +146,6 @@ enum pci_p2pdma_map_type {struct pci_p2pdma_map_state {struct dev_pagemap *pgmap;enum pci_p2pdma_map_type map; -u64 bus_off; };/* helper for pci_p2pdma_state(), do not use directly */ @@ -186,7 +185,7 @@ static inline dma_addr_t pci_p2pdma_bus_addr_map(struct pci_p2pdma_map_state *state, phys_addr_t paddr) { WARN_ON_ONCE(state->map != PCI_P2PDMA_MAP_BUS_ADDR); -return paddr + state->bus_off; +return paddr +to_p2p_pgmap (state->pgmap)->bus_offsetf; }
Quoted 1 line
include/linux/pci-p2pdma.h static inline dma_addr_t · line 191
Quoted 1 line
include/linux/pci-p2pdma.h +0
-0#endif /* _LINUX_PCI_P2P_H */
Sorry for long time to reply, I waited to see what is going on with DMA phys_vec basic series and together with my summer vacation, it took a while.
Quoted 14 lines
Looks like you're relying on this bogus code getting resolved in the next patch...
In file included from
kernel/dma/direct.c:16:./include/linux/pci-p2pdma.h: In function ‘pci_p2pdma_bus_addr_map’:./include/linux/pci-p2pdma.h:188:24: error: implicit declaration of function ‘to_p2p_pgmap’ [-Wimplicit-function-declaration]188 | return paddr + to_p2p_pgmap(state->pgmap)->bus_offsetf; | ^~~~~~~~~~~~
./include/linux/pci-p2pdma.h:188:50: error: invalid type argument of ‘->’ (have ‘int’)
188 | return paddr + to_p2p_pgmap(state->pgmap)->bus_offsetf;
| ^~./include/linux/pci-p2pdma.h:189:1: error: control reaches end of non-void function [-Werror=return-type]
189 | }
Quoted 4 lines
| ^
to_p2p_pgmap()is a static function and structpci_p2pdma_pagemapdoesn't have abus_offsetfmember. Thanks,
You are right, probably the best way to fix the error is simply squash this change into the next patch.
Thanks
Quoted 3 lines
Alex
Re: [PATCH v1 08/10] vfio/pci: Enable peer-to-peer DMA transactions by default
In reply to Alex WilliamsonQuoted 29 lines · Alex Williamson
Quoted 26 lines · On Mon, 4 Aug 2025 16:00:43 +0300 Leon Romanovsky <[email protected]> wrote:
From: Leon Romanovsky <[email protected]>
Make sure that all VFIO PCI devices have peer-to-peer capabilities enables, so we would be able to export their MMIO memory through DMABUF,
Signed-off-by Leon Romanovsky <[email protected]>--- drivers/vfio/pci/vfio_pci_core.c | 4 ++++ include/linux/vfio_pci_core.h | 1 + 2 files changed, 5 insertions(+)drivers/vfio/pci/vfio_pci_core.c +1
-0@@ -28,6 +28,7 @@#include <linux/nospec.h> #include <linux/sched/mm.h> #include <linux/iommufd.h> +#include <linux/pci-p2pdma.h> #if IS_ENABLED (CONFIG_EEH )#include <asm/eeh.h> #endif @@ -2088,6 +2089,9 @@ int vfio_pci_core_init_dev(struct vfio_device *core_vdev)INIT_LIST_HEAD (&vdev->dummy_resources_list);INIT_LIST_HEAD (&vdev->ioeventfds_list);
Quoted 8 lines
drivers/vfio/pci/vfio_pci_core.c int vfio_pci_core_init_dev(struct vfio_device *core_vdev) · line 2094
Quoted 4 lines
drivers/vfio/pci/vfio_pci_core.c +3
-0INIT_LIST_HEAD (&vdev->sriov_pfs_item); + vdev->provider =pci_p2pdma_enable (vdev->pdev); +if (IS_ERR (vdev->provider)) +return PTR_ERR (vdev->provider);I think this just made all vfio-pci drivers functionally dependent on
CONFIG_PCI_P2PDMA. Seems at best exporting a dma-buf should be restricted if this fails. Thanks,
It is temporary solution in next patch "vfio/pci: Add dma-buf export support for MMIO regions", the strict ifdef is added.
2107 #ifdef CONFIG_VFIO_PCI_DMABUF 2108 vdev->provider = pci_p2pdma_enable(vdev->pdev); 2109 if (IS_ERR(vdev->provider)) 2110 return PTR_ERR(vdev->provider); 2111 2112 INIT_LIST_HEAD(&vdev->dmabufs); 2113 #endif
I will split "vfio/pci: Add dma-buf export ..." patch to introduce CONFIG_VFIO_PCI_DMABUF before this "vfio/pci: Enable peer-to-peer ..." patch.
Thanks
Quoted 19 lines
Alex
Quoted 15 lines
drivers/vfio/pci/vfio_pci_core.c +0
-0init_rwsem (&vdev->memory_lock);xa_init (&vdev->ctx);include/linux/vfio_pci_core.h +1
-0@@ -94,6 +94,7 @@ struct vfio_pci_core_device {struct vfio_pci_core_device *sriov_pf_core_dev;struct notifier_block nb;struct rw_semaphore memory_lock; +struct p2pdma_provider *provider; };/* Will be exported for vfio pci drivers usage */
Re: [PATCH v1 10/10] vfio/pci: Add dma-buf export support for MMIO regions
In reply to Leon Romanovskydrivers/vfio/pci/vfio_pci_dmabuf.c line 218
Quoted 12 lines · Leon Romanovsky
drivers/vfio/pci/vfio_pci_dmabuf.c new file +12
-0+static void dma_ranges_to_p2p_phys (struct vfio_pci_dma_buf *priv, +struct vfio_device_feature_dma_buf *dma_buf, +struct vfio_region_dma_range *dma_ranges) +{ +struct pci_dev *pdev = priv->vdev->pdev; +phys_addr_t pci_start; +int i; + + pci_start =pci_resource_start (pdev, dma_buf->region_index); +for (i =0 ; i < dma_buf->nr_ranges; i++) { + priv->phys_vec[i].len = dma_ranges[i].length; + priv->phys_vec[i].paddr += pci_start + dma_ranges[i].offset;
Is the intent really to += paddr? I would have expected a plain assignment.
Quoted 4 lines
drivers/vfio/pci/vfio_pci_dmabuf.c new file +4
-0+ priv->size += priv->phys_vec[i].len; + } + priv->nr_ranges = dma_buf->nr_ranges; +}
...
Quoted 9 lines
drivers/vfio/pci/vfio_pci_dmabuf.c new file +9
-0+ priv->phys_vec =kcalloc (get_dma_buf.nr_ranges,sizeof (*priv->phys_vec), +GFP_KERNEL ); +if (!priv->phys_vec) { + ret = -ENOMEM ; + goto err_free_priv; + } + + priv->vdev = vdev; +dma_ranges_to_p2p_phys (priv, &get_dma_buf, dma_ranges);
Re: [PATCH v1 10/10] vfio/pci: Add dma-buf export support for MMIO regions
Quoted 15 lines · Alex Mastro
drivers/vfio/pci/vfio_pci_dmabuf.c line 218
Quoted 12 lines · Leon Romanovsky
drivers/vfio/pci/vfio_pci_dmabuf.c new file +12
-0+static void dma_ranges_to_p2p_phys (struct vfio_pci_dma_buf *priv, +struct vfio_device_feature_dma_buf *dma_buf, +struct vfio_region_dma_range *dma_ranges) +{ +struct pci_dev *pdev = priv->vdev->pdev; +phys_addr_t pci_start; +int i; + + pci_start =pci_resource_start (pdev, dma_buf->region_index); +for (i =0 ; i < dma_buf->nr_ranges; i++) { + priv->phys_vec[i].len = dma_ranges[i].length; + priv->phys_vec[i].paddr += pci_start + dma_ranges[i].offset;Is the intent really to += paddr? I would have expected a plain assignment.
In this specific case, there is no difference, because phys_vec is initialized to 0, but It needs to be "=" and not "+=".
Quoted 18 lines
Quoted 5 lines
drivers/vfio/pci/vfio_pci_dmabuf.c new file +4
-0+ priv->size += priv->phys_vec[i].len; + } + priv->nr_ranges = dma_buf->nr_ranges; +}...
Quoted 9 lines
drivers/vfio/pci/vfio_pci_dmabuf.c new file +9
-0+ priv->phys_vec =kcalloc (get_dma_buf.nr_ranges,sizeof (*priv->phys_vec), +GFP_KERNEL ); +if (!priv->phys_vec) { + ret = -ENOMEM ; + goto err_free_priv; + } + + priv->vdev = vdev; +dma_ranges_to_p2p_phys (priv, &get_dma_buf, dma_ranges);