Skip to contentSkip to conversation

vfio/pci: Set up VFIO barmap before creating a DMABUF

PATCH In review Matt Evans 3 participants 6 messages Apr 15 – Apr 21

Subsystems (6)
lore
Share & tools
lore

Matt Evans

author
#0 lore raw

PATCH 1/2 vfio/pci: Set up VFIO barmap before creating a DMABUF

A DMABUF exports access to BAR resources which need to be requested before the DMABUF is handed out. Usually the resources are requested when setting up the barmap when the VFIO device fd is mmap()ed, but there's no guarantee that's done before a DMABUF is created.

Set up the barmap (and so request resources) in the DMABUF-creation path.

Fixes 5d74781ebc86c ("vfio/pci: Add dma-buf export support for MMIO regions")
---
 drivers/vfio/pci/vfio_pci_dmabuf.c | 9 +++++++++
 1 file changed, 9 insertions(+)
drivers/vfio/pci/vfio_pci_dmabuf.c +9 -0
index 4ccaf3531e02..fefe7cf4256b 100644
@@ -272,6 +272,15 @@ int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
 		goto err_free_priv;
 	}
 
+	/*
+	 * See comment in vfio_pci_core_mmap(); ensure PCI regions
+	 * were requested before returning DMABUFs that reference
+	 * them.  Barmap setup does this:
+	 */
+	ret = vfio_pci_core_setup_barmap(vdev, get_dma_buf.region_index);
+	if (ret)
+		goto err_free_phys;
+
 	priv->vdev = vdev;
 	priv->nr_ranges = get_dma_buf.nr_ranges;
 	priv->size = length;
Signature · 1 line
2.47.3
author
#1 lore raw

PATCH 2/2 vfio/pci: Serialise vfio_pci_core_setup_barmap()

vfio_pci_core_setup_barmap() is used in a couple of paths (vfio_pci_bar_rw(), mmap()) to ensure BARs are mapped before access, and these paths could execute concurrently. Concurrent execution of vfio_pci_core_setup_barmap() could lead to some callers getting -EBUSY, which would be treated as fatal.

Introduce a new vfio_pci_core_lock_setup_barmap() function, which takes the vdev->memory_lock for write across BAR initialization. Current in-kernel use moves to this.

The existing (exported!) vfio_pci_core_setup_barmap() keeps its 'unlocked' behaviour.

Fixes 7f5764e179c6 ("vfio: use vfio_pci_core_setup_barmap to map bar in mmap")
Fixes 0d77ed3589ac0 ("vfio/pci: Pull BAR mapping setup from read-write path")
---
 drivers/vfio/pci/nvgrace-gpu/main.c |  2 +-
 drivers/vfio/pci/vfio_pci_core.c    |  2 +-
 drivers/vfio/pci/vfio_pci_dmabuf.c  |  2 +-
 drivers/vfio/pci/vfio_pci_rdwr.c    | 43 +++++++++++++++++++++++++----
 drivers/vfio/pci/virtio/legacy_io.c |  2 +-
 include/linux/vfio_pci_core.h       |  1 +
 6 files changed, 42 insertions(+), 10 deletions(-)
drivers/vfio/pci/nvgrace-gpu/main.c +1 -1
index fa056b69f899..c1df437754f9 100644
@@ -189,7 +189,7 @@ static int nvgrace_gpu_open_device(struct vfio_device *core_vdev)
 	 * register reads on first fault before establishing any GPU
 	 * memory mapping.
 	 */
-	ret = vfio_pci_core_setup_barmap(vdev, 0);
+	ret = vfio_pci_core_lock_setup_barmap(vdev, 0);
 	if (ret)
 		goto error_exit;
 
drivers/vfio/pci/vfio_pci_core.c +1 -1
index 3f8d093aacf8..4e9091e5fcc2 100644
@@ -1764,7 +1764,7 @@ int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_struct *vma
 	 * Even though we don't make use of the barmap for the mmap,
 	 * we need to request the region and the barmap tracks that.
 	 */
-	ret = vfio_pci_core_setup_barmap(vdev, index);
+	ret = vfio_pci_core_lock_setup_barmap(vdev, index);
 	if (ret)
 		return ret;
 
drivers/vfio/pci/vfio_pci_dmabuf.c +1 -1
index fefe7cf4256b..281ba7d69567 100644
@@ -277,7 +277,7 @@ int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
 	 * were requested before returning DMABUFs that reference
 	 * them.  Barmap setup does this:
 	 */
-	ret = vfio_pci_core_setup_barmap(vdev, get_dma_buf.region_index);
+	ret = vfio_pci_core_lock_setup_barmap(vdev, get_dma_buf.region_index);
 	if (ret)
 		goto err_free_phys;
 
drivers/vfio/pci/vfio_pci_rdwr.c +37 -6

72 lines not on this page · raw message

drivers/vfio/pci/virtio/legacy_io.c +1 -1
index 1ed349a55629..c77064e3f5c4 100644
@@ -305,7 +305,7 @@ static int virtiovf_set_notify_addr(struct virtiovf_pci_core_device *virtvdev)
 	 * Setup the BAR where the 'notify' exists to be used by vfio as well
 	 * This will let us mmap it only once and use it when needed.
 	 */
-	ret = vfio_pci_core_setup_barmap(core_device,
+	ret = vfio_pci_core_lock_setup_barmap(core_device,
 					 virtvdev->notify_bar);
 	if (ret)
 		return ret;
include/linux/vfio_pci_core.h +1 -0
index 2ebba746c18f..2ea4e773c121 100644
@@ -189,6 +189,7 @@ int vfio_pci_core_enable(struct vfio_pci_core_device *vdev);
 void vfio_pci_core_disable(struct vfio_pci_core_device *vdev);
 void vfio_pci_core_finish_enable(struct vfio_pci_core_device *vdev);
 int vfio_pci_core_setup_barmap(struct vfio_pci_core_device *vdev, int bar);
+int vfio_pci_core_lock_setup_barmap(struct vfio_pci_core_device *vdev, int bar);
 pci_ers_result_t vfio_pci_core_aer_err_detected(struct pci_dev *pdev,
 						pci_channel_state_t state);
 ssize_t vfio_pci_core_do_io_rw(struct vfio_pci_core_device *vdev, bool test_mem,
Signature · 1 line
2.47.3
maintainer
#2 lore raw
In reply to Matt Evans
Quoted 24 lines · On Wed, 15 Apr 2026 11:14:22 -0700 Matt Evans <[email protected]> wrote:

A DMABUF exports access to BAR resources which need to be requested before the DMABUF is handed out. Usually the resources are requested when setting up the barmap when the VFIO device fd is mmap()ed, but there's no guarantee that's done before a DMABUF is created.

Set up the barmap (and so request resources) in the DMABUF-creation path.

Fixes 5d74781ebc86c ("vfio/pci: Add dma-buf export support for MMIO regions")
---
 drivers/vfio/pci/vfio_pci_dmabuf.c | 9 +++++++++
 1 file changed, 9 insertions(+)
drivers/vfio/pci/vfio_pci_dmabuf.c +2 -0
index 4ccaf3531e02..fefe7cf4256b 100644
@@ -272,6 +272,15 @@ int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
 		goto err_free_priv;
 	}
 
+	/*
+	 * See comment in vfio_pci_core_mmap(); ensure PCI regions

drivers/vfio/pci/vfio_pci_dmabuf.c int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags, · line 286

Quoted 10 lines
drivers/vfio/pci/vfio_pci_dmabuf.c +7 -0
+	 * were requested before returning DMABUFs that reference
+	 * them.  Barmap setup does this:
+	 */
+	ret = vfio_pci_core_setup_barmap(vdev, get_dma_buf.region_index);
+	if (ret)
+		goto err_free_phys;
+
 	priv->vdev = vdev;
 	priv->nr_ranges = get_dma_buf.nr_ranges;
 	priv->size = length;

Wouldn't this get a lot easier if we just setup all the barmaps in vfio_pci_core_enable(), conditional on pci_resource_len() just like we use to filter in REGION_INFO?

I don't recall if there's some reason we've avoid this so far, maybe others can shout it out if they do.

We already tear them all down in vfio_pci_core_disable(). It would be a small patch to add that, which we would mark as Fixes:, then a small follow-up on top of that that removes any then redundant or unnecessary callers (all of them). Thoughts? Thanks,

Alex

BTW, cover letters are appreciated for multi-part series.

reviewer
#3 lore raw
Quoted 48 lines

From: Alex Williamson <[email protected]>
Sent: Friday, April 17, 2026 6:44 AM

Quoted 9 lines · On Wed, 15 Apr 2026 11:14:22 -0700 Matt Evans <[email protected]> wrote:

A DMABUF exports access to BAR resources which need to be requested before the DMABUF is handed out. Usually the resources are requested when setting up the barmap when the VFIO device fd is mmap()ed, but there's no guarantee that's done before a DMABUF is created.

Set up the barmap (and so request resources) in the DMABUF-creation path.

Fixes 5d74781ebc86c ("vfio/pci: Add dma-buf export support for MMIO

regions")

Review

Quoted 6 lines
---
 drivers/vfio/pci/vfio_pci_dmabuf.c | 9 +++++++++
 1 file changed, 9 insertions(+)
(quoted diff) +0 -0

b/drivers/vfio/pci/vfio_pci_dmabuf.c

drivers/vfio/pci/vfio_pci_dmabuf.c int vfio_pci_core_feature_dma_buf(struct

Quoted 4 lines

index 4ccaf3531e02..fefe7cf4256b 100644

drivers/vfio/pci/vfio_pci_dmabuf.c +0 -0
@@ -272,6 +272,15 @@ int vfio_pci_core_feature_dma_buf(struct

vfio_pci_core_device *vdev, u32 flags,

Quoted 5 lines
drivers/vfio/pci/vfio_pci_dmabuf.c +2 -0
 		goto err_free_priv;
 	}
 
+	/*
+	 * See comment in vfio_pci_core_mmap(); ensure PCI regions

drivers/vfio/pci/vfio_pci_dmabuf.c int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags, · line 286

Quoted 10 lines
drivers/vfio/pci/vfio_pci_dmabuf.c +7 -0
+	 * were requested before returning DMABUFs that reference
+	 * them.  Barmap setup does this:
+	 */
+	ret = vfio_pci_core_setup_barmap(vdev, get_dma_buf.region_index);
+	if (ret)
+		goto err_free_phys;
+
 	priv->vdev = vdev;
 	priv->nr_ranges = get_dma_buf.nr_ranges;
 	priv->size = length;

Wouldn't this get a lot easier if we just setup all the barmaps in vfio_pci_core_enable(), conditional on pci_resource_len() just like we use to filter in REGION_INFO?

Quoted 2 lines

I don't recall if there's some reason we've avoid this so far, maybe others can shout it out if they do.

I don't remember too. probably just because it's not a wide requirement then was made in this on-demand approach...

Quoted 6 lines

We already tear them all down in vfio_pci_core_disable(). It would be a small patch to add that, which we would mark as Fixes:, then a small follow-up on top of that that removes any then redundant or unnecessary callers (all of them). Thoughts? Thanks,

Agree. then the next patch fixing the racing conditions is also not required.

author
#4 lore raw

Hi Kevin, Alex,

Quoted 56 lines · Kevin
Quoted 49 lines

From: Alex Williamson <[email protected]>
Sent: Friday, April 17, 2026 6:44 AM

Quoted 9 lines · On Wed, 15 Apr 2026 11:14:22 -0700 Matt Evans <[email protected]> wrote:

A DMABUF exports access to BAR resources which need to be requested before the DMABUF is handed out. Usually the resources are requested when setting up the barmap when the VFIO device fd is mmap()ed, but there's no guarantee that's done before a DMABUF is created.

Set up the barmap (and so request resources) in the DMABUF-creation path.

Fixes 5d74781ebc86c ("vfio/pci: Add dma-buf export support for MMIO

regions")

Review

Quoted 6 lines
---
  drivers/vfio/pci/vfio_pci_dmabuf.c | 9 +++++++++
  1 file changed, 9 insertions(+)
(quoted diff) +0 -0

b/drivers/vfio/pci/vfio_pci_dmabuf.c

drivers/vfio/pci/vfio_pci_dmabuf.c int vfio_pci_core_feature_dma_buf(struct

Quoted 4 lines

index 4ccaf3531e02..fefe7cf4256b 100644

drivers/vfio/pci/vfio_pci_dmabuf.c +0 -0
@@ -272,6 +272,15 @@ int vfio_pci_core_feature_dma_buf(struct

vfio_pci_core_device *vdev, u32 flags,

Quoted 5 lines
drivers/vfio/pci/vfio_pci_dmabuf.c +2 -0
  		goto err_free_priv;
  	}
 
+	/*
+	 * See comment in vfio_pci_core_mmap(); ensure PCI regions

drivers/vfio/pci/vfio_pci_dmabuf.c int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags, · line 286

Quoted 10 lines
drivers/vfio/pci/vfio_pci_dmabuf.c +7 -0
+	 * were requested before returning DMABUFs that reference
+	 * them.  Barmap setup does this:
+	 */
+	ret = vfio_pci_core_setup_barmap(vdev, get_dma_buf.region_index);
+	if (ret)
+		goto err_free_phys;
+
  	priv->vdev = vdev;
  	priv->nr_ranges = get_dma_buf.nr_ranges;
  	priv->size = length;

Wouldn't this get a lot easier if we just setup all the barmaps in vfio_pci_core_enable(), conditional on pci_resource_len() just like we use to filter in REGION_INFO?

Quoted 2 lines

I don't recall if there's some reason we've avoid this so far, maybe others can shout it out if they do.

I don't remember too. probably just because it's not a wide requirement then was made in this on-demand approach...

Quoted 1 line
Quoted 7 lines
Quoted 5 lines

We already tear them all down in vfio_pci_core_disable(). It would be a small patch to add that, which we would mark as Fixes:, then a small follow-up on top of that that removes any then redundant or unnecessary callers (all of them). Thoughts? Thanks,

Agree. then the next patch fixing the racing conditions is also not required.

Oh, if we can do it earlier it's all much cleaner, agreed. I'd naively thought there was a reason it was done lazily. One benefit is truly huge TB-scale BARs don't get ioremap()ed until actually used by userspace. I can't think of a good usage example that relies on open performance.

If you say fine, then great: Ignore these, and I'll post a new pair of patches doing that (...with a cover letter this time).

Thank you,

Matt

author
#5 lore raw

Hi,

Quoted 65 lines · Matt Evans

Hi Kevin, Alex,

Quoted 62 lines · Kevin
Quoted 45 lines

From: Alex Williamson <[email protected]>
Sent: Friday, April 17, 2026 6:44 AM

Quoted 9 lines · On Wed, 15 Apr 2026 11:14:22 -0700 Matt Evans <[email protected]> wrote:

A DMABUF exports access to BAR resources which need to be requested before the DMABUF is handed out.  Usually the resources are requested when setting up the barmap when the VFIO device fd is mmap()ed, but there's no guarantee that's done before a DMABUF is created.

Set up the barmap (and so request resources) in the DMABUF-creation path.

Fixes 5d74781ebc86c ("vfio/pci: Add dma-buf export support for MMIO

regions")

Review

Quoted 6 lines

---
  drivers/vfio/pci/vfio_pci_dmabuf.c | 9 +++++++++

  1 file changed, 9 insertions(+)
(quoted diff) +0 -0

b/drivers/vfio/pci/vfio_pci_dmabuf.c

drivers/vfio/pci/vfio_pci_dmabuf.c int vfio_pci_core_feature_dma_buf(struct

Quoted 4 lines

index 4ccaf3531e02..fefe7cf4256b 100644

drivers/vfio/pci/vfio_pci_dmabuf.c +0 -0
@@ -272,6 +272,15 @@ int vfio_pci_core_feature_dma_buf(struct

vfio_pci_core_device *vdev, u32 flags,

Quoted 5 lines
drivers/vfio/pci/vfio_pci_dmabuf.c +2 -0
           goto err_free_priv;
       }
 
+    /*
+     * See comment in vfio_pci_core_mmap(); ensure PCI regions

drivers/vfio/pci/vfio_pci_dmabuf.c int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags, · line 286

Quoted 10 lines
drivers/vfio/pci/vfio_pci_dmabuf.c +7 -0
+     * were requested before returning DMABUFs that reference
+     * them.  Barmap setup does this:
+     */
+    ret = vfio_pci_core_setup_barmap(vdev, get_dma_buf.region_index);
+    if (ret)
+        goto err_free_phys;
+
       priv->vdev = vdev;
       priv->nr_ranges = get_dma_buf.nr_ranges;
       priv->size = length;
Quoted 6 lines

Wouldn't this get a lot easier if we just setup all the barmaps in vfio_pci_core_enable(), conditional on pci_resource_len() just like we use to filter in REGION_INFO?

I don't recall if there's some reason we've avoid this so far, maybe others can shout it out if they do.

I don't remember too. probably just because it's not a wide requirement then was made in this on-demand approach...

Quoted 6 lines

We already tear them all down in vfio_pci_core_disable().  It would be a small patch to add that, which we would mark as Fixes:, then a small follow-up on top of that that removes any then redundant or unnecessary callers (all of them).  Thoughts?  Thanks,

Quoted 5 lines
Quoted 1 line

Agree. then the next patch fixing the racing conditions is also not required.

Oh, if we can do it earlier it's all much cleaner, agreed.  I'd naively thought there was a reason it was done lazily.  One benefit is truly huge TB-scale BARs don't get ioremap()ed until actually used by userspace.  I can't think of a good usage example that relies on open performance.

If you say fine, then great:  Ignore these, and I'll post a new pair of patches doing that (...with a cover letter this time).

New series posted:

https://lore.kernel.org/kvm/[email protected]/

(Titles/semantics very different, so didn't send as a v2 of this.)

Matt