vfio/pci: Set up VFIO barmap before creating a DMABUF
Subsystems (6)
PATCH 1/2 vfio/pci: Set up VFIO barmap before creating a DMABUF
A DMABUF exports access to BAR resources which need to be requested before the DMABUF is handed out. Usually the resources are requested when setting up the barmap when the VFIO device fd is mmap()ed, but there's no guarantee that's done before a DMABUF is created.
Set up the barmap (and so request resources) in the DMABUF-creation path.
--- drivers/vfio/pci/vfio_pci_dmabuf.c | 9 +++++++++ 1 file changed, 9 insertions(+)
drivers/vfio/pci/vfio_pci_dmabuf.c +9 -0
@@ -272,6 +272,15 @@ int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags, goto err_free_priv; } +/* +* See comment in vfio_pci_core_mmap(); ensure PCI regions +* were requested before returning DMABUFs that reference +* them. Barmap setup does this: +*/ + ret =vfio_pci_core_setup_barmap (vdev, get_dma_buf.region_index); +if (ret) + goto err_free_phys; + priv->vdev = vdev; priv->nr_ranges = get_dma_buf.nr_ranges; priv->size = length;
Signature · 1 line
2.47.3
PATCH 2/2 vfio/pci: Serialise vfio_pci_core_setup_barmap()
vfio_pci_core_setup_barmap() is used in a couple of paths (vfio_pci_bar_rw(), mmap()) to ensure BARs are mapped before access, and these paths could execute concurrently. Concurrent execution of vfio_pci_core_setup_barmap() could lead to some callers getting -EBUSY, which would be treated as fatal.
Introduce a new vfio_pci_core_lock_setup_barmap() function, which takes the vdev->memory_lock for write across BAR initialization. Current in-kernel use moves to this.
The existing (exported!) vfio_pci_core_setup_barmap() keeps its 'unlocked' behaviour.
--- drivers/vfio/pci/nvgrace-gpu/main.c | 2 +- drivers/vfio/pci/vfio_pci_core.c | 2 +- drivers/vfio/pci/vfio_pci_dmabuf.c | 2 +- drivers/vfio/pci/vfio_pci_rdwr.c | 43 +++++++++++++++++++++++++---- drivers/vfio/pci/virtio/legacy_io.c | 2 +- include/linux/vfio_pci_core.h | 1 + 6 files changed, 42 insertions(+), 10 deletions(-)
drivers/vfio/pci/nvgrace-gpu/main.c +1 -1
@@ -189,7 +189,7 @@ static int nvgrace_gpu_open_device(struct vfio_device *core_vdev)* register reads on first fault before establishing any GPU * memory mapping. */ - ret =vfio_pci_core_setup_barmap (vdev,0 ); + ret =vfio_pci_core_lock_setup_barmap (vdev,0 );if (ret) goto error_exit;
drivers/vfio/pci/vfio_pci_core.c +1 -1
@@ -1764,7 +1764,7 @@ int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_struct *vma* Even though we don't make use of the barmap for the mmap, * we need to request the region and the barmap tracks that. */ - ret =vfio_pci_core_setup_barmap (vdev, index); + ret =vfio_pci_core_lock_setup_barmap (vdev, index);if (ret)return ret;
drivers/vfio/pci/vfio_pci_dmabuf.c +1 -1
@@ -277,7 +277,7 @@ int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,* were requested before returning DMABUFs that reference * them. Barmap setup does this: */ - ret =vfio_pci_core_setup_barmap (vdev, get_dma_buf.region_index); + ret =vfio_pci_core_lock_setup_barmap (vdev, get_dma_buf.region_index);if (ret) goto err_free_phys;
drivers/vfio/pci/vfio_pci_rdwr.c +37 -6
72 lines not on this page · raw message
drivers/vfio/pci/virtio/legacy_io.c +1 -1
@@ -305,7 +305,7 @@ static int virtiovf_set_notify_addr(struct virtiovf_pci_core_device *virtvdev)* Setup the BAR where the 'notify' exists to be used by vfio as well * This will let us mmap it only once and use it when needed. */ - ret =vfio_pci_core_setup_barmap (core_device, + ret =vfio_pci_core_lock_setup_barmap (core_device, virtvdev->notify_bar);if (ret)return ret;
include/linux/vfio_pci_core.h +1 -0
@@ -189,6 +189,7 @@ int vfio_pci_core_enable(struct vfio_pci_core_device *vdev);void vfio_pci_core_disable (struct vfio_pci_core_device *vdev);void vfio_pci_core_finish_enable (struct vfio_pci_core_device *vdev);int vfio_pci_core_setup_barmap (struct vfio_pci_core_device *vdev,int bar); +int vfio_pci_core_lock_setup_barmap (struct vfio_pci_core_device *vdev,int bar);pci_ers_result_t vfio_pci_core_aer_err_detected (struct pci_dev *pdev,pci_channel_state_t state); ssize_t vfio_pci_core_do_io_rw(struct vfio_pci_core_device *vdev, bool test_mem,
Signature · 1 line
2.47.3
Alex Williamson
maintainerQuoted 24 lines · On Wed, 15 Apr 2026 11:14:22 -0700 Matt Evans <[email protected]> wrote:
A DMABUF exports access to BAR resources which need to be requested before the DMABUF is handed out. Usually the resources are requested when setting up the barmap when the VFIO device fd is mmap()ed, but there's no guarantee that's done before a DMABUF is created.
Set up the barmap (and so request resources) in the DMABUF-creation path.
Fixes 5d74781ebc86c ("vfio/pci: Add dma-buf export support for MMIO regions")Signed-off-by Matt Evans <[email protected]>--- drivers/vfio/pci/vfio_pci_dmabuf.c | 9 +++++++++ 1 file changed, 9 insertions(+)drivers/vfio/pci/vfio_pci_dmabuf.c +2
-0@@ -272,6 +272,15 @@ int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags, goto err_free_priv; } +/* +* See comment in vfio_pci_core_mmap(); ensure PCI regions
drivers/vfio/pci/vfio_pci_dmabuf.c int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags, · line 286
Quoted 10 lines
drivers/vfio/pci/vfio_pci_dmabuf.c +7
-0+* were requested before returning DMABUFs that reference +* them. Barmap setup does this: +*/ + ret =vfio_pci_core_setup_barmap (vdev, get_dma_buf.region_index); +if (ret) + goto err_free_phys; + priv->vdev = vdev; priv->nr_ranges = get_dma_buf.nr_ranges; priv->size = length;
Wouldn't this get a lot easier if we just setup all the barmaps in vfio_pci_core_enable(), conditional on pci_resource_len() just like we use to filter in REGION_INFO?
I don't recall if there's some reason we've avoid this so far, maybe others can shout it out if they do.
We already tear them all down in vfio_pci_core_disable(). It would be a small patch to add that, which we would mark as Fixes:, then a small follow-up on top of that that removes any then redundant or unnecessary callers (all of them). Thoughts? Thanks,
Alex
BTW, cover letters are appreciated for multi-part series.
Tian, Kevin
reviewerQuoted 48 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 17, 2026 6:44 AMQuoted 9 lines · On Wed, 15 Apr 2026 11:14:22 -0700 Matt Evans <[email protected]> wrote:
A DMABUF exports access to BAR resources which need to be requested before the DMABUF is handed out. Usually the resources are requested when setting up the barmap when the VFIO device fd is mmap()ed, but there's no guarantee that's done before a DMABUF is created.
Set up the barmap (and so request resources) in the DMABUF-creation path.
Fixes 5d74781ebc86c ("vfio/pci: Add dma-buf export support for MMIOregions")
Review
Quoted 6 lines
Signed-off-by Matt Evans <[email protected]>--- drivers/vfio/pci/vfio_pci_dmabuf.c | 9 +++++++++ 1 file changed, 9 insertions(+)(quoted diff) +0
-0
b/drivers/vfio/pci/vfio_pci_dmabuf.cdrivers/vfio/pci/vfio_pci_dmabuf.c int vfio_pci_core_feature_dma_buf(struct
Quoted 4 lines
index
4ccaf3531e02..fefe7cf4256b100644drivers/vfio/pci/vfio_pci_dmabuf.c +0
-0@@ -272,6 +272,15 @@ int vfio_pci_core_feature_dma_buf(struct
vfio_pci_core_device*vdev, u32 flags,Quoted 5 lines
drivers/vfio/pci/vfio_pci_dmabuf.c +2
-0goto err_free_priv; } +/* +* See comment in vfio_pci_core_mmap(); ensure PCI regions drivers/vfio/pci/vfio_pci_dmabuf.c int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags, · line 286
Quoted 10 lines
drivers/vfio/pci/vfio_pci_dmabuf.c +7
-0+* were requested before returning DMABUFs that reference +* them. Barmap setup does this: +*/ + ret =vfio_pci_core_setup_barmap (vdev, get_dma_buf.region_index); +if (ret) + goto err_free_phys; + priv->vdev = vdev; priv->nr_ranges = get_dma_buf.nr_ranges; priv->size = length;Wouldn't this get a lot easier if we just setup all the barmaps in
vfio_pci_core_enable(), conditional onpci_resource_len()just like we use to filter inREGION_INFO?
Quoted 2 lines
I don't recall if there's some reason we've avoid this so far, maybe others can shout it out if they do.
I don't remember too. probably just because it's not a wide requirement then was made in this on-demand approach...
Quoted 6 lines
We already tear them all down in
vfio_pci_core_disable(). It would be a small patch to add that, which we would mark as Fixes:, then a small follow-up on top of that that removes any then redundant or unnecessary callers (all of them). Thoughts? Thanks,
Agree. then the next patch fixing the racing conditions is also not required.
Hi Kevin, Alex,
Quoted 56 lines · Kevin
Quoted 49 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 17, 2026 6:44 AMQuoted 9 lines · On Wed, 15 Apr 2026 11:14:22 -0700 Matt Evans <[email protected]> wrote:
A DMABUF exports access to BAR resources which need to be requested before the DMABUF is handed out. Usually the resources are requested when setting up the barmap when the VFIO device fd is mmap()ed, but there's no guarantee that's done before a DMABUF is created.
Set up the barmap (and so request resources) in the DMABUF-creation path.
Fixes 5d74781ebc86c ("vfio/pci: Add dma-buf export support for MMIOregions")
Review
Quoted 6 lines
Signed-off-by Matt Evans <[email protected]>--- drivers/vfio/pci/vfio_pci_dmabuf.c | 9 +++++++++ 1 file changed, 9 insertions(+)(quoted diff) +0
-0
b/drivers/vfio/pci/vfio_pci_dmabuf.cdrivers/vfio/pci/vfio_pci_dmabuf.c int vfio_pci_core_feature_dma_buf(struct
Quoted 4 lines
index
4ccaf3531e02..fefe7cf4256b100644drivers/vfio/pci/vfio_pci_dmabuf.c +0
-0@@ -272,6 +272,15 @@ int vfio_pci_core_feature_dma_buf(struct
vfio_pci_core_device*vdev, u32 flags,Quoted 5 lines
drivers/vfio/pci/vfio_pci_dmabuf.c +2
-0goto err_free_priv; } +/* +* See comment in vfio_pci_core_mmap(); ensure PCI regions drivers/vfio/pci/vfio_pci_dmabuf.c int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags, · line 286
Quoted 10 lines
drivers/vfio/pci/vfio_pci_dmabuf.c +7
-0+* were requested before returning DMABUFs that reference +* them. Barmap setup does this: +*/ + ret =vfio_pci_core_setup_barmap (vdev, get_dma_buf.region_index); +if (ret) + goto err_free_phys; + priv->vdev = vdev; priv->nr_ranges = get_dma_buf.nr_ranges; priv->size = length;Wouldn't this get a lot easier if we just setup all the barmaps in
vfio_pci_core_enable(), conditional onpci_resource_len()just like we use to filter inREGION_INFO?Quoted 2 lines
I don't recall if there's some reason we've avoid this so far, maybe others can shout it out if they do.
I don't remember too. probably just because it's not a wide requirement then was made in this on-demand approach...
Quoted 1 line
Quoted 7 lines
Quoted 5 lines
We already tear them all down in
vfio_pci_core_disable(). It would be a small patch to add that, which we would mark as Fixes:, then a small follow-up on top of that that removes any then redundant or unnecessary callers (all of them). Thoughts? Thanks,Agree. then the next patch fixing the racing conditions is also not required.
Oh, if we can do it earlier it's all much cleaner, agreed. I'd naively thought there was a reason it was done lazily. One benefit is truly huge TB-scale BARs don't get ioremap()ed until actually used by userspace. I can't think of a good usage example that relies on open performance.
If you say fine, then great: Ignore these, and I'll post a new pair of patches doing that (...with a cover letter this time).
Thank you,
Matt
Hi,
Quoted 65 lines · Matt Evans
Hi Kevin, Alex,
Quoted 62 lines · Kevin
Quoted 45 lines
From: Alex Williamson <[email protected]>
Sent: Friday, April 17, 2026 6:44 AMQuoted 9 lines · On Wed, 15 Apr 2026 11:14:22 -0700 Matt Evans <[email protected]> wrote:
A DMABUF exports access to BAR resources which need to be requested before the DMABUF is handed out. Usually the resources are requested when setting up the barmap when the VFIO device fd is mmap()ed, but there's no guarantee that's done before a DMABUF is created.
Set up the barmap (and so request resources) in the DMABUF-creation path.
Fixes 5d74781ebc86c ("vfio/pci: Add dma-buf export support for MMIOregions")
Review
Quoted 6 lines
Signed-off-by Matt Evans <[email protected]>---
drivers/vfio/pci/vfio_pci_dmabuf.c| 9 +++++++++1 file changed, 9 insertions(+)(quoted diff) +0
-0
b/drivers/vfio/pci/vfio_pci_dmabuf.cdrivers/vfio/pci/vfio_pci_dmabuf.c int vfio_pci_core_feature_dma_buf(struct
Quoted 4 lines
index
4ccaf3531e02..fefe7cf4256b100644drivers/vfio/pci/vfio_pci_dmabuf.c +0
-0@@ -272,6 +272,15 @@ int vfio_pci_core_feature_dma_buf(struct
vfio_pci_core_device*vdev, u32 flags,Quoted 5 lines
drivers/vfio/pci/vfio_pci_dmabuf.c +2
-0goto err_free_priv; } + /* +* See comment in vfio_pci_core_mmap(); ensure PCI regions drivers/vfio/pci/vfio_pci_dmabuf.c int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags, · line 286
Quoted 10 lines
drivers/vfio/pci/vfio_pci_dmabuf.c +7
-0+* were requested before returning DMABUFs that reference +* them. Barmap setup does this: +*/ + ret = vfio_pci_core_setup_barmap(vdev, get_dma_buf.region_index); + if (ret) + goto err_free_phys; + priv->vdev = vdev; priv->nr_ranges = get_dma_buf.nr_ranges; priv->size = length;Quoted 6 lines
Wouldn't this get a lot easier if we just setup all the barmaps in
vfio_pci_core_enable(), conditional onpci_resource_len()just like we use to filter inREGION_INFO?I don't recall if there's some reason we've avoid this so far, maybe others can shout it out if they do.
I don't remember too. probably just because it's not a wide requirement then was made in this on-demand approach...
Quoted 6 lines
We already tear them all down in vfio_pci_core_disable(). It would be a small patch to add that, which we would mark as Fixes:, then a small follow-up on top of that that removes any then redundant or unnecessary callers (all of them). Thoughts? Thanks,
Quoted 5 lines
Quoted 1 line
Agree. then the next patch fixing the racing conditions is also not required.
Oh, if we can do it earlier it's all much cleaner, agreed. I'd naively thought there was a reason it was done lazily. One benefit is truly huge TB-scale BARs don't get ioremap()ed until actually used by userspace. I can't think of a good usage example that relies on open performance.
If you say fine, then great: Ignore these, and I'll post a new pair of patches doing that (...with a cover letter this time).
New series posted:
https://lore.kernel.org/kvm/[email protected]/
(Titles/semantics very different, so didn't send as a v2 of this.)
Matt