Skip to contentSkip to conversation

vfio: Prevent open_count decrement to negative

PATCH In review Jacob Pan 3 participants 14 messages Jun 3, 2025 – Jun 18, 2025 VFIO driver

lore
Share & tools
lore

Jacob Pan

author
#0 lore raw

PATCH v2 1/2 vfio: Prevent open_count decrement to negative

When vfio_df_close() is called with open_count=0, it triggers a warning in vfio_assert_device_open() but still decrements open_count to -1. This allows a subsequent open to incorrectly pass the open_count == 0 check, leading to unintended behavior, such as setting df->access_granted = true.

For example, running an IOMMUFD compat no-IOMMU device with VFIO tests
(https://github.com/awilliam/tests/blob/master/vfio-noiommu-pci-device-open.c)
results in a warning and a failed VFIO_GROUP_GET_DEVICE_FD ioctl on the first run, but the second run succeeds incorrectly.

Add checks to avoid decrementing open_count below zero.

---
v2: Added Reviewed-by tags

---
 drivers/vfio/vfio_main.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/vfio/vfio_main.c +2 -1
index 1fd261efc582..5046cae05222 100644
@@ -583,7 +583,8 @@ void vfio_df_close(struct vfio_device_file *df)
 
 	lockdep_assert_held(&device->dev_set->lock);
 
-	vfio_assert_device_open(device);
+	if (!vfio_assert_device_open(device))
+		return;
 	if (device->open_count == 1)
 		vfio_df_device_last_close(df);
 	device->open_count--;
Signature · 1 line
2.34.1
author
#1 lore raw

PATCH v2 2/2 vfio: Fix unbalanced vfio_df_close call in no-iommu mode

From: Jason Gunthorpe <[email protected]>

For devices with no-iommu enabled in IOMMUFD VFIO compat mode, the group open path skips vfio_df_open(), leaving open_count at 0. This causes a warning in vfio_assert_device_open(device) when vfio_df_close() is called during group close.

The correct behavior is to skip only the IOMMUFD bind in the device open path for no-iommu devices. Commit 6086efe73498 omitted vfio_df_open(), which was too broad. This patch restores the previous behavior, ensuring the vfio_df_open is called in the group open path.

Fixes 6086efe73498 ("vfio-iommufd: Move noiommu compat validation out of vfio_iommufd_bind()")

---
v2: Use a fix from Jason

---
 drivers/vfio/group.c     | 10 +++++-----
 drivers/vfio/iommufd.c   |  3 ---
 drivers/vfio/vfio_main.c | 26 ++++++++++++++++----------
 3 files changed, 21 insertions(+), 18 deletions(-)
drivers/vfio/group.c +5 -5
index c321d442f0da..8f5fe8a392de 100644
@@ -192,18 +192,18 @@ static int vfio_df_group_open(struct vfio_device_file *df)
 		 * implies they expected translation to exist
 		 */
 		if (!capable(CAP_SYS_RAWIO) ||
-		    vfio_iommufd_device_has_compat_ioas(device, df->iommufd))
+		    vfio_iommufd_device_has_compat_ioas(device, df->iommufd)) {
 			ret = -EPERM;
-		else
-			ret = 0;
-		goto out_put_kvm;
+			goto out_put_kvm;
+		}
 	}
 
 	ret = vfio_df_open(df);
 	if (ret)
 		goto out_put_kvm;
 
-	if (df->iommufd && device->open_count == 1) {
+	if (df->iommufd && device->open_count == 1 &&
+	    !vfio_device_is_noiommu(device)) {
 		ret = vfio_iommufd_compat_attach_ioas(device, df->iommufd);
 		if (ret)
 			goto out_close_device;
drivers/vfio/iommufd.c +0 -3
index c8c3a2d53f86..26c9c3068c77 100644
@@ -54,9 +54,6 @@ void vfio_df_iommufd_unbind(struct vfio_device_file *df)
 
 	lockdep_assert_held(&vdev->dev_set->lock);
 
-	if (vfio_device_is_noiommu(vdev))
-		return;
-
 	if (vdev->ops->unbind_iommufd)
 		vdev->ops->unbind_iommufd(vdev);
 }
drivers/vfio/vfio_main.c +16 -10
index 5046cae05222..ac2dbd4e5d04 100644
@@ -506,17 +506,19 @@ static int vfio_df_device_first_open(struct vfio_device_file *df)
 {
 	struct vfio_device *device = df->device;
 	struct iommufd_ctx *iommufd = df->iommufd;
-	int ret;
+	int ret = 0;
 
 	lockdep_assert_held(&device->dev_set->lock);
 
 	if (!try_module_get(device->dev->driver->owner))
 		return -ENODEV;
 
-	if (iommufd)
-		ret = vfio_df_iommufd_bind(df);
-	else
+	if (iommufd) {
+		if (!vfio_device_is_noiommu(device))
+			ret = vfio_df_iommufd_bind(df);
+	} else {
 		ret = vfio_device_group_use_iommu(device);
+	}
 	if (ret)
 		goto err_module_put;
 
@@ -528,10 +530,12 @@ static int vfio_df_device_first_open(struct vfio_device_file *df)
 	return 0;
 
 err_unuse_iommu:
-	if (iommufd)
-		vfio_df_iommufd_unbind(df);
-	else
+	if (iommufd) {
+		if (!vfio_device_is_noiommu(device))
+			vfio_df_iommufd_unbind(df);
+	} else {
 		vfio_device_group_unuse_iommu(device);
+	}
 err_module_put:
 	module_put(device->dev->driver->owner);
 	return ret;
@@ -546,10 +550,12 @@ static void vfio_df_device_last_close(struct vfio_device_file *df)
 
 	if (device->ops->close_device)
 		device->ops->close_device(device);
-	if (iommufd)
-		vfio_df_iommufd_unbind(df);
-	else
+	if (iommufd) {
+		if (!vfio_device_is_noiommu(device))
+			vfio_df_iommufd_unbind(df);
+	} else {
 		vfio_device_group_unuse_iommu(device);
+	}
 	module_put(device->dev->driver->owner);
 }
 
Signature · 1 line
2.34.1
#2 lore raw

Re: [PATCH v2 2/2] vfio: Fix unbalanced vfio_df_close call in no-iommu mode

Quoted 39 lines · On Tue, 3 Jun 2025 08:23:43 -0700 Jacob Pan <[email protected]>…

From: Jason Gunthorpe <[email protected]>

For devices with no-iommu enabled in IOMMUFD VFIO compat mode, the group open path skips vfio_df_open(), leaving open_count at 0. This causes a warning in vfio_assert_device_open(device) when vfio_df_close() is called during group close.

The correct behavior is to skip only the IOMMUFD bind in the device open path for no-iommu devices. Commit 6086efe73498 omitted vfio_df_open(), which was too broad. This patch restores the previous behavior, ensuring the vfio_df_open is called in the group open path.

Fixes 6086efe73498 ("vfio-iommufd: Move noiommu compat validation out of vfio_iommufd_bind()")

---
v2: Use a fix from Jason

---
 drivers/vfio/group.c     | 10 +++++-----
 drivers/vfio/iommufd.c   |  3 ---
 drivers/vfio/vfio_main.c | 26 ++++++++++++++++----------
 3 files changed, 21 insertions(+), 18 deletions(-)
drivers/vfio/group.c +2 -4
index c321d442f0da..8f5fe8a392de 100644
@@ -192,18 +192,18 @@ static int vfio_df_group_open(struct vfio_device_file *df)
 		 * implies they expected translation to exist
 		 */
 		if (!capable(CAP_SYS_RAWIO) ||
-		    vfio_iommufd_device_has_compat_ioas(device, df->iommufd))
+		    vfio_iommufd_device_has_compat_ioas(device, df->iommufd)) {
 			ret = -EPERM;
-		else
-			ret = 0;
-		goto out_put_kvm;
+			goto out_put_kvm;

drivers/vfio/group.c static int vfio_df_group_open(struct vfio_device_file *df) · line 206

Quoted 10 lines
drivers/vfio/group.c +3 -1
+		}
 	}
 
 	ret = vfio_df_open(df);
 	if (ret)
 		goto out_put_kvm;
 
-	if (df->iommufd && device->open_count == 1) {
+	if (df->iommufd && device->open_count == 1 &&
+	    !vfio_device_is_noiommu(device)) {

Why do we need this?

int vfio_iommufd_compat_attach_ioas(struct vfio_device *vdev,
struct iommufd_ctx *ictx)

{
        u32 ioas_id;
        int ret;

        lockdep_assert_held(&vdev->dev_set->lock);

        /* compat noiommu does not need to do ioas attach */
        if (vfio_device_is_noiommu(vdev))
                return 0;
Quoted 8 lines
drivers/vfio/group.c +0 -0
 		ret = vfio_iommufd_compat_attach_ioas(device, df->iommufd);
 		if (ret)
 			goto out_close_device;
drivers/vfio/iommufd.c +0 -0
index c8c3a2d53f86..26c9c3068c77 100644

drivers/vfio/iommufd.c void vfio_df_iommufd_unbind(struct vfio_device_file *df) · line 59

Quoted 6 lines
drivers/vfio/iommufd.c +0 -3
@@ -54,9 +54,6 @@ void vfio_df_iommufd_unbind(struct vfio_device_file *df)
 
 	lockdep_assert_held(&vdev->dev_set->lock);
 
-	if (vfio_device_is_noiommu(vdev))
-		return;
-

Why not keep this and add similar to vfio_df_iommufd_bind()? It seems cleaner to me. Thanks,

Alex

Quoted 63 lines
drivers/vfio/iommufd.c +0 -0
 	if (vdev->ops->unbind_iommufd)
 		vdev->ops->unbind_iommufd(vdev);
 }
drivers/vfio/vfio_main.c +16 -10
index 5046cae05222..ac2dbd4e5d04 100644
@@ -506,17 +506,19 @@ static int vfio_df_device_first_open(struct vfio_device_file *df)
 {
 	struct vfio_device *device = df->device;
 	struct iommufd_ctx *iommufd = df->iommufd;
-	int ret;
+	int ret = 0;
 
 	lockdep_assert_held(&device->dev_set->lock);
 
 	if (!try_module_get(device->dev->driver->owner))
 		return -ENODEV;
 
-	if (iommufd)
-		ret = vfio_df_iommufd_bind(df);
-	else
+	if (iommufd) {
+		if (!vfio_device_is_noiommu(device))
+			ret = vfio_df_iommufd_bind(df);
+	} else {
 		ret = vfio_device_group_use_iommu(device);
+	}
 	if (ret)
 		goto err_module_put;
 
@@ -528,10 +530,12 @@ static int vfio_df_device_first_open(struct vfio_device_file *df)
 	return 0;
 
 err_unuse_iommu:
-	if (iommufd)
-		vfio_df_iommufd_unbind(df);
-	else
+	if (iommufd) {
+		if (!vfio_device_is_noiommu(device))
+			vfio_df_iommufd_unbind(df);
+	} else {
 		vfio_device_group_unuse_iommu(device);
+	}
 err_module_put:
 	module_put(device->dev->driver->owner);
 	return ret;
@@ -546,10 +550,12 @@ static void vfio_df_device_last_close(struct vfio_device_file *df)
 
 	if (device->ops->close_device)
 		device->ops->close_device(device);
-	if (iommufd)
-		vfio_df_iommufd_unbind(df);
-	else
+	if (iommufd) {
+		if (!vfio_device_is_noiommu(device))
+			vfio_df_iommufd_unbind(df);
+	} else {
 		vfio_device_group_unuse_iommu(device);
+	}
 	module_put(device->dev->driver->owner);
 }
 
#3 lore raw

Re: [PATCH v2 2/2] vfio: Fix unbalanced vfio_df_close call in no-iommu mode

Quoted 49 lines · Alex Williamson
Quoted 46 lines · On Tue, 3 Jun 2025 08:23:43 -0700 Jacob Pan <[email protected]>…

From: Jason Gunthorpe <[email protected]>

For devices with no-iommu enabled in IOMMUFD VFIO compat mode, the group open path skips vfio_df_open(), leaving open_count at 0. This causes a warning in vfio_assert_device_open(device) when vfio_df_close() is called during group close.

The correct behavior is to skip only the IOMMUFD bind in the device open path for no-iommu devices. Commit 6086efe73498 omitted vfio_df_open(), which was too broad. This patch restores the previous behavior, ensuring the vfio_df_open is called in the group open path.

Fixes 6086efe73498 ("vfio-iommufd: Move noiommu compat validation out of vfio_iommufd_bind()")

---
v2: Use a fix from Jason

---
 drivers/vfio/group.c     | 10 +++++-----
 drivers/vfio/iommufd.c   |  3 ---
 drivers/vfio/vfio_main.c | 26 ++++++++++++++++----------
 3 files changed, 21 insertions(+), 18 deletions(-)
drivers/vfio/group.c +3 -4
index c321d442f0da..8f5fe8a392de 100644
@@ -192,18 +192,18 @@ static int vfio_df_group_open(struct vfio_device_file *df)
 		 * implies they expected translation to exist
 		 */
 		if (!capable(CAP_SYS_RAWIO) ||
-		    vfio_iommufd_device_has_compat_ioas(device, df->iommufd))
+		    vfio_iommufd_device_has_compat_ioas(device, df->iommufd)) {
 			ret = -EPERM;
-		else
-			ret = 0;
-		goto out_put_kvm;
+			goto out_put_kvm;
+		}
 	}
 
 	ret = vfio_df_open(df);
 	if (ret)
 		goto out_put_kvm;
 
Quoted 5 lines

drivers/vfio/group.c static int vfio_df_group_open(struct vfio_device_file *df) · line 206

Quoted 3 lines
drivers/vfio/group.c +2 -1
-	if (df->iommufd && device->open_count == 1) {
+	if (df->iommufd && device->open_count == 1 &&
+	    !vfio_device_is_noiommu(device)) {

Why do we need this?

What I was trying to do is put all the logic about noiommu into only vfio_df..open/close functions instead of sprikling it into a bunch of other functions. That seemed to be the right point to make this cut.

Quoted 11 lines

int vfio_iommufd_compat_attach_ioas(struct vfio_device *vdev,
struct iommufd_ctx *ictx)

{
        u32 ioas_id;
        int ret;

        lockdep_assert_held(&vdev->dev_set->lock);

        /* compat noiommu does not need to do ioas attach */
        if (vfio_device_is_noiommu(vdev))
                return 0;

So this should be removed, I missed it

Quoted 10 lines

drivers/vfio/group.c void vfio_df_iommufd_unbind(struct vfio_device_file *df) · line 59

Quoted 7 lines
drivers/vfio/group.c +0 -3
@@ -54,9 +54,6 @@ void vfio_df_iommufd_unbind(struct vfio_device_file *df)
 
 	lockdep_assert_held(&vdev->dev_set->lock);
 
-	if (vfio_device_is_noiommu(vdev))
-		return;
-

Why not keep this and add similar to vfio_df_iommufd_bind()? It seems cleaner to me. Thanks,

Same as above, we don't check for noiommu in bind, so we should not check it in unbind to have a symetrical API design.

With this patch we move toward the vfio_df..open/close functions being symmetrical in their decision making.

Jason

#4 lore raw

Re: [PATCH v2 2/2] vfio: Fix unbalanced vfio_df_close call in no-iommu mode

Quoted 56 lines · On Fri, 13 Jun 2025 21:15:55 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 49 lines · Alex Williamson
Quoted 46 lines · On Tue, 3 Jun 2025 08:23:43 -0700 Jacob Pan <[email protected]>…

From: Jason Gunthorpe <[email protected]>

For devices with no-iommu enabled in IOMMUFD VFIO compat mode, the group open path skips vfio_df_open(), leaving open_count at 0. This causes a warning in vfio_assert_device_open(device) when vfio_df_close() is called during group close.

The correct behavior is to skip only the IOMMUFD bind in the device open path for no-iommu devices. Commit 6086efe73498 omitted vfio_df_open(), which was too broad. This patch restores the previous behavior, ensuring the vfio_df_open is called in the group open path.

Fixes 6086efe73498 ("vfio-iommufd: Move noiommu compat validation out of vfio_iommufd_bind()")

---
v2: Use a fix from Jason

---
 drivers/vfio/group.c     | 10 +++++-----
 drivers/vfio/iommufd.c   |  3 ---
 drivers/vfio/vfio_main.c | 26 ++++++++++++++++----------
 3 files changed, 21 insertions(+), 18 deletions(-)
drivers/vfio/group.c +3 -4
index c321d442f0da..8f5fe8a392de 100644
@@ -192,18 +192,18 @@ static int vfio_df_group_open(struct vfio_device_file *df)
 		 * implies they expected translation to exist
 		 */
 		if (!capable(CAP_SYS_RAWIO) ||
-		    vfio_iommufd_device_has_compat_ioas(device, df->iommufd))
+		    vfio_iommufd_device_has_compat_ioas(device, df->iommufd)) {
 			ret = -EPERM;
-		else
-			ret = 0;
-		goto out_put_kvm;
+			goto out_put_kvm;
+		}
 	}
 
 	ret = vfio_df_open(df);
 	if (ret)
 		goto out_put_kvm;
 
Quoted 5 lines

drivers/vfio/group.c static int vfio_df_group_open(struct vfio_device_file *df) · line 206

Quoted 3 lines
drivers/vfio/group.c +2 -1
-	if (df->iommufd && device->open_count == 1) {
+	if (df->iommufd && device->open_count == 1 &&
+	    !vfio_device_is_noiommu(device)) {  

Why do we need this?

Quoted 3 lines

What I was trying to do is put all the logic about noiommu into only vfio_df..open/close functions instead of sprikling it into a bunch of other functions. That seemed to be the right point to make this cut.

Alternatively we could be consistent about breaking out of the vfio/iommufd.c functions that aren't relevant to noiommu. The container side handles noiommu internally, why should iommufd push handling up to the device file layer? We're really just missing the bind path.

TBH, it seems like special casing iommufd in the device file layer is what led to the issue introduced in 6086efe73498.

Quoted 25 lines
Quoted 11 lines

int vfio_iommufd_compat_attach_ioas(struct vfio_device *vdev,
struct iommufd_ctx *ictx)

{
        u32 ioas_id;
        int ret;

        lockdep_assert_held(&vdev->dev_set->lock);

        /* compat noiommu does not need to do ioas attach */
        if (vfio_device_is_noiommu(vdev))
                return 0;  

So this should be removed, I missed it

Quoted 10 lines

drivers/vfio/group.c void vfio_df_iommufd_unbind(struct vfio_device_file *df) · line 59

Quoted 7 lines
drivers/vfio/group.c +0 -3
@@ -54,9 +54,6 @@ void vfio_df_iommufd_unbind(struct vfio_device_file *df)
 
 	lockdep_assert_held(&vdev->dev_set->lock);
 
-	if (vfio_device_is_noiommu(vdev))
-		return;
-  

Why not keep this and add similar to vfio_df_iommufd_bind()? It seems cleaner to me. Thanks,

Quoted 2 lines

Same as above, we don't check for noiommu in bind, so we should not check it in unbind to have a symetrical API design.

Or check it in bind since we already check it in unbind. Either way, symmetry.

Quoted 2 lines

With this patch we move toward the vfio_df..open/close functions being symmetrical in their decision making.

But is it? We special case all the iommufd paths to filter out noiommu but it's inconsistent with the legacy paths. Thanks,

Alex

#5 lore raw

Re: [PATCH v2 2/2] vfio: Fix unbalanced vfio_df_close call in no-iommu mode

Quoted 63 lines · Alex Williamson
Quoted 56 lines · On Fri, 13 Jun 2025 21:15:55 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 49 lines · Alex Williamson
Quoted 46 lines · On Tue, 3 Jun 2025 08:23:43 -0700 Jacob Pan <[email protected]>…

From: Jason Gunthorpe <[email protected]>

For devices with no-iommu enabled in IOMMUFD VFIO compat mode, the group open path skips vfio_df_open(), leaving open_count at 0. This causes a warning in vfio_assert_device_open(device) when vfio_df_close() is called during group close.

The correct behavior is to skip only the IOMMUFD bind in the device open path for no-iommu devices. Commit 6086efe73498 omitted vfio_df_open(), which was too broad. This patch restores the previous behavior, ensuring the vfio_df_open is called in the group open path.

Fixes 6086efe73498 ("vfio-iommufd: Move noiommu compat validation out of vfio_iommufd_bind()")

---
v2: Use a fix from Jason

---
 drivers/vfio/group.c     | 10 +++++-----
 drivers/vfio/iommufd.c   |  3 ---
 drivers/vfio/vfio_main.c | 26 ++++++++++++++++----------
 3 files changed, 21 insertions(+), 18 deletions(-)
drivers/vfio/group.c +3 -4
index c321d442f0da..8f5fe8a392de 100644
@@ -192,18 +192,18 @@ static int vfio_df_group_open(struct vfio_device_file *df)
 		 * implies they expected translation to exist
 		 */
 		if (!capable(CAP_SYS_RAWIO) ||
-		    vfio_iommufd_device_has_compat_ioas(device, df->iommufd))
+		    vfio_iommufd_device_has_compat_ioas(device, df->iommufd)) {
 			ret = -EPERM;
-		else
-			ret = 0;
-		goto out_put_kvm;
+			goto out_put_kvm;
+		}
 	}
 
 	ret = vfio_df_open(df);
 	if (ret)
 		goto out_put_kvm;
 
Quoted 5 lines

drivers/vfio/group.c static int vfio_df_group_open(struct vfio_device_file *df) · line 206

Quoted 3 lines
drivers/vfio/group.c +2 -1
-	if (df->iommufd && device->open_count == 1) {
+	if (df->iommufd && device->open_count == 1 &&
+	    !vfio_device_is_noiommu(device)) {  

Why do we need this?

Quoted 3 lines

What I was trying to do is put all the logic about noiommu into only vfio_df..open/close functions instead of sprikling it into a bunch of other functions. That seemed to be the right point to make this cut.

Quoted 5 lines

Alternatively we could be consistent about breaking out of the vfio/iommufd.c functions that aren't relevant to noiommu. The container side handles noiommu internally, why should iommufd push handling up to the device file layer? We're really just missing the bind path.

Broadly what I was going for was to just remove the iommufd stuff entirely from the DF layer rather than to half pretend there is an iommufd layer below it. This should ideally go as far as not having an iommufd_ctx at all. So things start to look really weird calling iommufd functions without an iommufd ctx.

Quoted 5 lines
Quoted 2 lines

With this patch we move toward the vfio_df..open/close functions being symmetrical in their decision making.

But is it? We special case all the iommufd paths to filter out noiommu but it's inconsistent with the legacy paths. Thanks,

The container still exists in noiommu mode and internally does things, eg it has a container->noiommu indicationm and the vfio-noiommu ops to manage this.

The iommufd should not exist and should never be used. They are different cases.

If Jacob eventually does what I suggested in another email then we would have a noiommu special mode inside iommufd and it would look more like the container.

Jason

#6 lore raw

Re: [PATCH v2 2/2] vfio: Fix unbalanced vfio_df_close call in no-iommu mode

Quoted 86 lines · On Mon, 16 Jun 2025 12:34:55 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 63 lines · Alex Williamson
Quoted 56 lines · On Fri, 13 Jun 2025 21:15:55 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 49 lines · Alex Williamson
Quoted 46 lines · On Tue, 3 Jun 2025 08:23:43 -0700 Jacob Pan <[email protected]>…

From: Jason Gunthorpe <[email protected]>

For devices with no-iommu enabled in IOMMUFD VFIO compat mode, the group open path skips vfio_df_open(), leaving open_count at 0. This causes a warning in vfio_assert_device_open(device) when vfio_df_close() is called during group close.

The correct behavior is to skip only the IOMMUFD bind in the device open path for no-iommu devices. Commit 6086efe73498 omitted vfio_df_open(), which was too broad. This patch restores the previous behavior, ensuring the vfio_df_open is called in the group open path.

Fixes 6086efe73498 ("vfio-iommufd: Move noiommu compat validation out of vfio_iommufd_bind()")

---
v2: Use a fix from Jason

---
 drivers/vfio/group.c     | 10 +++++-----
 drivers/vfio/iommufd.c   |  3 ---
 drivers/vfio/vfio_main.c | 26 ++++++++++++++++----------
 3 files changed, 21 insertions(+), 18 deletions(-)
drivers/vfio/group.c +3 -4
index c321d442f0da..8f5fe8a392de 100644
@@ -192,18 +192,18 @@ static int vfio_df_group_open(struct vfio_device_file *df)
 		 * implies they expected translation to exist
 		 */
 		if (!capable(CAP_SYS_RAWIO) ||
-		    vfio_iommufd_device_has_compat_ioas(device, df->iommufd))
+		    vfio_iommufd_device_has_compat_ioas(device, df->iommufd)) {
 			ret = -EPERM;
-		else
-			ret = 0;
-		goto out_put_kvm;
+			goto out_put_kvm;
+		}
 	}
 
 	ret = vfio_df_open(df);
 	if (ret)
 		goto out_put_kvm;
 
Quoted 5 lines

drivers/vfio/group.c static int vfio_df_group_open(struct vfio_device_file *df) · line 206

Quoted 3 lines
drivers/vfio/group.c +2 -1
-	if (df->iommufd && device->open_count == 1) {
+	if (df->iommufd && device->open_count == 1 &&
+	    !vfio_device_is_noiommu(device)) {    

Why do we need this?

Quoted 3 lines

What I was trying to do is put all the logic about noiommu into only vfio_df..open/close functions instead of sprikling it into a bunch of other functions. That seemed to be the right point to make this cut.

Quoted 5 lines

Alternatively we could be consistent about breaking out of the vfio/iommufd.c functions that aren't relevant to noiommu. The container side handles noiommu internally, why should iommufd push handling up to the device file layer? We're really just missing the bind path.

Broadly what I was going for was to just remove the iommufd stuff entirely from the DF layer rather than to half pretend there is an iommufd layer below it. This should ideally go as far as not having an iommufd_ctx at all. So things start to look really weird calling iommufd functions without an iommufd ctx.

Quoted 5 lines
Quoted 2 lines

With this patch we move toward the vfio_df..open/close functions being symmetrical in their decision making.

But is it? We special case all the iommufd paths to filter out noiommu but it's inconsistent with the legacy paths. Thanks,

The container still exists in noiommu mode and internally does things, eg it has a container->noiommu indicationm and the vfio-noiommu ops to manage this.

Quoted 6 lines

The iommufd should not exist and should never be used. They are different cases.

If Jacob eventually does what I suggested in another email then we would have a noiommu special mode inside iommufd and it would look more like the container.

A concise fix would be nice for stable backports though, so even if we want to move to testing noiommu in the device file layer or create a special mode in iommufd, the smallest, most consistent initial fix would be to continue the _group_open:

drivers/vfio/group.c +3 -4
@@ -192,18 +192,18 @@ static int vfio_df_group_open(struct vfio_device_file *df)
 		 * implies they expected translation to exist
 		 */
 		if (!capable(CAP_SYS_RAWIO) ||
-		    vfio_iommufd_device_has_compat_ioas(device, df->iommufd))
+		    vfio_iommufd_device_has_compat_ioas(device, df->iommufd)) {
 			ret = -EPERM;
-		else
-			ret = 0;
-		goto out_put_kvm;
+			goto out_put_kvm;
+		}
 	}
 
 

And add a noiommu exit branch to _iommufd_bind, symmetric to unbind. Right? Thanks,

Alex

#7 lore raw

Re: [PATCH v2 2/2] vfio: Fix unbalanced vfio_df_close call in no-iommu mode

Quoted 17 lines · Alex Williamson
drivers/vfio/group.c +3 -4
@@ -192,18 +192,18 @@ static int vfio_df_group_open(struct vfio_device_file *df)
 		 * implies they expected translation to exist
 		 */
 		if (!capable(CAP_SYS_RAWIO) ||
-		    vfio_iommufd_device_has_compat_ioas(device, df->iommufd))
+		    vfio_iommufd_device_has_compat_ioas(device, df->iommufd)) {
 			ret = -EPERM;
-		else
-			ret = 0;
-		goto out_put_kvm;
+			goto out_put_kvm;
+		}
 	}
 
 
Quoted 2 lines

And add a noiommu exit branch to _iommufd_bind, symmetric to unbind. Right? Thanks,

Just comparing to the original

+ if (iommufd) {
+ if (!vfio_device_is_noiommu(device))
+ ret = vfio_df_iommufd_bind(df);

Isn't being captured, so it needs another hunk in
vfio_df_iommufd_bind()

Jason

author
#8 lore raw

Re: [PATCH v2 2/2] vfio: Fix unbalanced vfio_df_close call in no-iommu mode

Hi Jason,

Quoted 29 lines · On Mon, 16 Jun 2025 17:05:58 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 19 lines · Alex Williamson
drivers/vfio/group.c +3 -4
@@ -192,18 +192,18 @@ static int vfio_df_group_open(struct
 vfio_device_file *df)
 		 * implies they expected translation to exist
 		 */
 		if (!capable(CAP_SYS_RAWIO) ||
-		    vfio_iommufd_device_has_compat_ioas(device,
 df->iommufd))
+		    vfio_iommufd_device_has_compat_ioas(device,
 df->iommufd)) { ret = -EPERM;
-		else
-			ret = 0;
-		goto out_put_kvm;
+			goto out_put_kvm;
+		}
 	}
 
 
Quoted 2 lines

And add a noiommu exit branch to _iommufd_bind, symmetric to unbind. Right? Thanks,

Just comparing to the original

(quoted diff) +3 -0
+	if (iommufd) {
+		if (!vfio_device_is_noiommu(device))
+			ret = vfio_df_iommufd_bind(df);
Quoted 3 lines

Isn't being captured, so it needs another hunk in
vfio_df_iommufd_bind()

OK, will send out v3 with this concise fix. I am also working on the noiommu special mode as you suggested, then we can allow bind again.

Thanks,

Jacob

#9 lore raw

Re: [PATCH v2 2/2] vfio: Fix unbalanced vfio_df_close call in no-iommu mode

Quoted 34 lines · Jacob Pan

Hi Jason,

Quoted 29 lines · On Mon, 16 Jun 2025 17:05:58 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 19 lines · Alex Williamson
drivers/vfio/group.c +3 -4
@@ -192,18 +192,18 @@ static int vfio_df_group_open(struct
 vfio_device_file *df)
 		 * implies they expected translation to exist
 		 */
 		if (!capable(CAP_SYS_RAWIO) ||
-		    vfio_iommufd_device_has_compat_ioas(device,
 df->iommufd))
+		    vfio_iommufd_device_has_compat_ioas(device,
 df->iommufd)) { ret = -EPERM;
-		else
-			ret = 0;
-		goto out_put_kvm;
+			goto out_put_kvm;
+		}
 	}
 
 
Quoted 2 lines

And add a noiommu exit branch to _iommufd_bind, symmetric to unbind. Right? Thanks,

Just comparing to the original

(quoted diff) +3 -0
+	if (iommufd) {
+		if (!vfio_device_is_noiommu(device))
+			ret = vfio_df_iommufd_bind(df);
Quoted 5 lines
Quoted 3 lines

Isn't being captured, so it needs another hunk in
vfio_df_iommufd_bind()

OK, will send out v3 with this concise fix. I am also working on the noiommu special mode as you suggested, then we can allow bind again.

Sure sounds good

Jason

#10 lore raw
In reply to Jacob Pan

Hi Jacob,

Quoted 11 lines · On Tue, 3 Jun 2025 08:23:42 -0700 Jacob Pan <[email protected]>…

When vfio_df_close() is called with open_count=0, it triggers a warning in vfio_assert_device_open() but still decrements open_count to -1. This allows a subsequent open to incorrectly pass the open_count == 0 check, leading to unintended behavior, such as setting df->access_granted = true.

For example, running an IOMMUFD compat no-IOMMU device with VFIO tests
(https://github.com/awilliam/tests/blob/master/vfio-noiommu-pci-device-open.c)
results in a warning and a failed VFIO_GROUP_GET_DEVICE_FD ioctl on the first run, but the second run succeeds incorrectly.

Add checks to avoid decrementing open_count below zero.

The example above suggests to me that this is a means by which we could see this, but in reality it seems it is the only means by which we can create this scenario, right?

Why does VFIO_GROUP_GET_DEVICE_FD fail on the first iteration? It seems like things are pretty broken, we won't have access_granted set, but I don't spot why the ioctl fails.

Doesn't this also (begin) to fix 6086efe73498 as well? I think that introduced skipping vfio_df_open() entirely for noiommu devices. It seems like this should have a Fixes: tag and the warning in the assertion was unreachable until 608... so maybe it should be linked here. Thanks,

Alex

Quoted 23 lines

---
v2: Added Reviewed-by tags

---
 drivers/vfio/vfio_main.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/vfio/vfio_main.c +2 -1
index 1fd261efc582..5046cae05222 100644
@@ -583,7 +583,8 @@ void vfio_df_close(struct vfio_device_file *df)
 
 	lockdep_assert_held(&device->dev_set->lock);
 
-	vfio_assert_device_open(device);
+	if (!vfio_assert_device_open(device))
+		return;
 	if (device->open_count == 1)
 		vfio_df_device_last_close(df);
 	device->open_count--;
#11 lore raw
Quoted 15 lines · Alex Williamson

Hi Jacob,

Quoted 10 lines · On Tue, 3 Jun 2025 08:23:42 -0700 Jacob Pan <[email protected]>…

When vfio_df_close() is called with open_count=0, it triggers a warning in vfio_assert_device_open() but still decrements open_count to -1. This allows a subsequent open to incorrectly pass the open_count == 0 check, leading to unintended behavior, such as setting df->access_granted = true.

For example, running an IOMMUFD compat no-IOMMU device with VFIO tests
(https://github.com/awilliam/tests/blob/master/vfio-noiommu-pci-device-open.c)
results in a warning and a failed VFIO_GROUP_GET_DEVICE_FD ioctl on the first run, but the second run succeeds incorrectly.

Quoted 5 lines
Quoted 1 line

Add checks to avoid decrementing open_count below zero.

The example above suggests to me that this is a means by which we could see this, but in reality it seems it is the only means by which we can create this scenario, right?

I understood this as an assertion hit because of the bug fixed in patch 2 and thus the missed assertion error handling flow was noticed.

Obviously the assertion should never happen, but if it does we should try to recover better than we currently do.

Jason

#12 lore raw
Quoted 22 lines · On Fri, 13 Jun 2025 21:09:26 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 15 lines · Alex Williamson

Hi Jacob,

Quoted 10 lines · On Tue, 3 Jun 2025 08:23:42 -0700 Jacob Pan <[email protected]>…

When vfio_df_close() is called with open_count=0, it triggers a warning in vfio_assert_device_open() but still decrements open_count to -1. This allows a subsequent open to incorrectly pass the open_count == 0 check, leading to unintended behavior, such as setting df->access_granted = true.

For example, running an IOMMUFD compat no-IOMMU device with VFIO tests
(https://github.com/awilliam/tests/blob/master/vfio-noiommu-pci-device-open.c)
results in a warning and a failed VFIO_GROUP_GET_DEVICE_FD ioctl on the first run, but the second run succeeds incorrectly.

Quoted 5 lines
Quoted 1 line

Add checks to avoid decrementing open_count below zero.

The example above suggests to me that this is a means by which we could see this, but in reality it seems it is the only means by which we can create this scenario, right?

Quoted 5 lines

I understood this as an assertion hit because of the bug fixed in patch 2 and thus the missed assertion error handling flow was noticed.

Obviously the assertion should never happen, but if it does we should try to recover better than we currently do.

Certainly. My statement is trying to determine the scope of the issue from a stable perspective. Maybe I'm interpreting "[f]or example" too broadly, but I think this is unreachable outside of the specific described scenario, ie. using iommufd in compatibility mode with no-iommu. Further, it only became reachable with 6086efe73498.

In any case, it fixes something and we should attribute that something, whether it's 6086efe73498 or we want to reach back to when the assert was introduced and claim it should have had a return even if it was unreachable.

It seems these patches should also be re-ordered if not rolled into one. Fixing the issue in 2/ makes this once again unreachable, so I don't mind it coming along as a "also handle this error case better." This alone doesn't really do much. Thanks,

Alex

author
#13 lore raw

Hi Alex,

Quoted 45 lines · On Mon, 16 Jun 2025 08:40:01 -0600 Alex Williamson…
Quoted 24 lines · On Fri, 13 Jun 2025 21:09:26 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 17 lines · Alex Williamson

Hi Jacob,

Quoted 12 lines · On Tue, 3 Jun 2025 08:23:42 -0700 Jacob Pan <[email protected]>…

When vfio_df_close() is called with open_count=0, it triggers a warning in vfio_assert_device_open() but still decrements open_count to -1. This allows a subsequent open to incorrectly pass the open_count == 0 check, leading to unintended behavior, such as setting df->access_granted = true.

For example, running an IOMMUFD compat no-IOMMU device with VFIO tests
(https://github.com/awilliam/tests/blob/master/vfio-noiommu-pci-device-open.c)
results in a warning and a failed VFIO_GROUP_GET_DEVICE_FD ioctl on the first run, but the second run succeeds incorrectly.

Quoted 5 lines
Quoted 1 line

Add checks to avoid decrementing open_count below zero.

The example above suggests to me that this is a means by which we could see this, but in reality it seems it is the only means by which we can create this scenario, right?

Quoted 6 lines

I understood this as an assertion hit because of the bug fixed in patch 2 and thus the missed assertion error handling flow was noticed.

Obviously the assertion should never happen, but if it does we should try to recover better than we currently do.

Certainly. My statement is trying to determine the scope of the issue from a stable perspective. Maybe I'm interpreting "[f]or example" too broadly, but I think this is unreachable outside of the specific described scenario, ie. using iommufd in compatibility mode with no-iommu. Further, it only became reachable with 6086efe73498.

In any case, it fixes something and we should attribute that something, whether it's 6086efe73498 or we want to reach back to when the assert was introduced and claim it should have had a return even if it was unreachable.

Quoted 5 lines

It seems these patches should also be re-ordered if not rolled into one. Fixing the issue in 2/ makes this once again unreachable, so I don't mind it coming along as a "also handle this error case better." This alone doesn't really do much. Thanks,

IMHO, this is an independent exception handling fix, perhaps I just add the missing tag below?

Fixes 05f37e1c03b6 ("vfio: Pass struct vfio_device_file * to

vfio_device_open/close()")

Thanks,

Jacob