Skip to contentSkip to conversation

A bug fix to the error handling path in vfio_device_open()

PATCH Superseded Yi Liu 4 participants 10 messages Jun 20, 2022 – Jun 27, 2022

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Yi Liu

author
#0 lore raw

cover A bug fix to the error handling path in vfio_device_open()

The open_count is unnecessarily placed in the group_rwsem in the error handling path.

Yi Liu (1):
vfio: Move "device->open_count--" out of group_rwsem in
vfio_device_open()

 drivers/vfio/vfio.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Signature · 1 line
2.27.0
author
#1 lore raw

PATCH 1/1 vfio: Move "device->open_count--" out of group_rwsem in vfio_device_open()

No need to protect open_count with group_rwsem

Fixes 421cfe6596f6 ("vfio: remove VFIO_GROUP_NOTIFY_SET_KVM")
cc Matthew Rosato <[email protected]>
cc Jason Gunthorpe <[email protected]>
---
 drivers/vfio/vfio.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/vfio/vfio.c +1 -1
index 61e71c1154be..44c3bf8023ac 100644
@@ -1146,10 +1146,10 @@ static struct file *vfio_device_open(struct vfio_device *device)
 	if (device->open_count == 1 && device->ops->close_device)
 		device->ops->close_device(device);
 err_undo_count:
+	up_read(&device->group->group_rwsem);
 	device->open_count--;
 	if (device->open_count == 0 && device->kvm)
 		device->kvm = NULL;
-	up_read(&device->group->group_rwsem);
 	mutex_unlock(&device->dev_set->lock);
 	module_put(device->dev->driver->owner);
 err_unassign_container:
Signature · 1 line
2.27.0
Reviewed-by on v1 1/1
#2 lore raw

Re: [Patch 1/1] vfio: Move "device->open_count--" out of group_rwsem in vfio_device_open()

Quoted 7 lines · Yi Liu

No need to protect open_count with group_rwsem

Fixes 421cfe6596f6 ("vfio: remove VFIO_GROUP_NOTIFY_SET_KVM")
cc Matthew Rosato <[email protected]>
cc Jason Gunthorpe <[email protected]>

Seems pretty harmless as-is, but you are correct group_rwsem can be dropped earlier; we do not protect the count with group_rwsem elsewhere (see vfio_device_fops_release as a comparison, where we already drop group_rwsem before open_count--)

FWIW, this change now also drops group_rswem before setting device->kvm = NULL, but that's also OK (again, just like vfio_device_fops_release) -- While the setting of device->kvm before open_device is technically done while holding the group_rwsem, this is done to protect the group kvm value we are copying from, and we should not be relying on that to protect the contents of device->kvm; instead we assume this value will not change until after the device is closed and while under the dev_set->lock.

Quoted 20 lines
---
  drivers/vfio/vfio.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
drivers/vfio/vfio.c +1 -1
index 61e71c1154be..44c3bf8023ac 100644
@@ -1146,10 +1146,10 @@ static struct file *vfio_device_open(struct vfio_device *device)
  	if (device->open_count == 1 && device->ops->close_device)
  		device->ops->close_device(device);
  err_undo_count:
+	up_read(&device->group->group_rwsem);
  	device->open_count--;
  	if (device->open_count == 0 && device->kvm)
  		device->kvm = NULL;
-	up_read(&device->group->group_rwsem);
  	mutex_unlock(&device->dev_set->lock);
  	module_put(device->dev->driver->owner);
  err_unassign_container:
author
#3 lore raw

Re: [Patch 1/1] vfio: Move "device->open_count--" out of group_rwsem in vfio_device_open()

Quoted 10 lines · Matthew Rosato
Quoted 7 lines · Yi Liu

No need to protect open_count with group_rwsem

Fixes 421cfe6596f6 ("vfio: remove VFIO_GROUP_NOTIFY_SET_KVM")
cc Matthew Rosato <[email protected]>
cc Jason Gunthorpe <[email protected]>

Seems pretty harmless as-is, but you are correct group_rwsem can be dropped earlier; we do not protect the count with group_rwsem elsewhere (see vfio_device_fops_release as a comparison, where we already drop group_rwsem before open_count--)

yes. this is exactly how I found it. Normally, I compare the err handling path with the release function to see if they are aligned. :-)

Quoted 1 line

FWIW, this change now also drops group_rswem before setting device->kvm = NULL, but that's also OK (again, just like vfio_device_fops_release) -- While the setting of device->kvm before open_device is technically done while holding the group_rwsem, this is done to protect the group kvm value we are copying from, and we should not be relying on that to protect the contents of device->kvm; instead we assume this value will not change until after the device is closed and while under the dev_set->lock.

yes. set device->kvm to be NULL has no need to hold group_rwsem. BTW. I
also doubt whether the device->ops->open_device(device) and device->ops->close_device(device) should be protected by group_rwsem or not. seems not, right? group_rwsem protects the fields under vfio_group.
For the open_device/close_device() device->dev_set->lock is enough. Maybe
another nit fix.

Quoted 24 lines
Quoted 21 lines

---
  drivers/vfio/vfio.c | 2 +-

  1 file changed, 1 insertion(+), 1 deletion(-)
drivers/vfio/vfio.c +0 -0
index 61e71c1154be..44c3bf8023ac 100644
@@ -1146,10 +1146,10 @@ static struct file *vfio_device_open(struct 
 vfio_device *device)
       if (device->open_count == 1 && device->ops->close_device)
           device->ops->close_device(device);

  err_undo_count:
+    up_read(&device->group->group_rwsem);
      device->open_count--;
      if (device->open_count == 0 && device->kvm)
          device->kvm = NULL;
-    up_read(&device->group->group_rwsem);
      mutex_unlock(&device->dev_set->lock);
      module_put(device->dev->driver->owner);
  err_unassign_container:

Signature · 2 lines
Regards,
Yi Liu
#4 lore raw

RE: [Patch 1/1] vfio: Move "device->open_count--" out of group_rwsem in vfio_device_open()

Quoted 14 lines

From: Liu, Yi L <[email protected]>
Sent: Tuesday, June 21, 2022 9:32 AM

Quoted 10 lines · Matthew Rosato
Quoted 7 lines · Yi Liu

No need to protect open_count with group_rwsem

Fixes 421cfe6596f6 ("vfio: remove VFIO_GROUP_NOTIFY_SET_KVM")
cc Matthew Rosato <[email protected]>
cc Jason Gunthorpe <[email protected]>

Seems pretty harmless as-is, but you are correct group_rwsem can be

Quoted 8 lines

dropped

Quoted 2 lines

earlier; we do not protect the count with group_rwsem elsewhere (see vfio_device_fops_release as a comparison, where we already drop

group_rwsem

Quoted 1 line

before open_count--)

yes. this is exactly how I found it. Normally, I compare the err handling path with the release function to see if they are aligned. :-)

In this case we don't need a FIX tag. It's kind of optimization.

Quoted 11 lines
Quoted 6 lines

FWIW, this change now also drops group_rswem before setting device- kvm =
NULL, but that's also OK (again, just like vfio_device_fops_release) -- While the setting of device->kvm before open_device is technically done while holding the group_rwsem, this is done to protect the group kvm

value

Quoted 3 lines

we are copying from, and we should not be relying on that to protect the contents of device->kvm; instead we assume this value will not change until after the device is closed and while under the dev_set->lock.

Quoted 7 lines

yes. set device->kvm to be NULL has no need to hold group_rwsem. BTW. I also doubt whether the device->ops->open_device(device) and device->ops->close_device(device) should be protected by group_rwsem or not. seems not, right? group_rwsem protects the fields under vfio_group. For the open_device/close_device() device->dev_set->lock is enough. Maybe another nit fix.

group->rwsem is to protect device->group->kvm from being changed by vfio_file_set_kvm() before it is copied to device->kvm.

author
#5 lore raw

Re: [Patch 1/1] vfio: Move "device->open_count--" out of group_rwsem in vfio_device_open()

Quoted 17 lines · Kevin
Quoted 17 lines

From: Liu, Yi L <[email protected]>
Sent: Tuesday, June 21, 2022 9:32 AM

Quoted 10 lines · Matthew Rosato
Quoted 7 lines · Yi Liu

No need to protect open_count with group_rwsem

Fixes 421cfe6596f6 ("vfio: remove VFIO_GROUP_NOTIFY_SET_KVM")
cc Matthew Rosato <[email protected]>
cc Jason Gunthorpe <[email protected]>

Seems pretty harmless as-is, but you are correct group_rwsem can be

dropped

Quoted 2 lines

earlier; we do not protect the count with group_rwsem elsewhere (see vfio_device_fops_release as a comparison, where we already drop

Quoted 7 lines
Quoted 5 lines

group_rwsem

Quoted 1 line

before open_count--)

yes. this is exactly how I found it. Normally, I compare the err handling path with the release function to see if they are aligned. :-)

In this case we don't need a FIX tag. It's kind of optimization.

ok.

Quoted 19 lines
Quoted 11 lines
Quoted 6 lines

FWIW, this change now also drops group_rswem before setting device- kvm =
NULL, but that's also OK (again, just like vfio_device_fops_release) -- While the setting of device->kvm before open_device is technically done while holding the group_rwsem, this is done to protect the group kvm

value

Quoted 3 lines

we are copying from, and we should not be relying on that to protect the contents of device->kvm; instead we assume this value will not change until after the device is closed and while under the dev_set->lock.

Quoted 7 lines

yes. set device->kvm to be NULL has no need to hold group_rwsem. BTW. I also doubt whether the device->ops->open_device(device) and device->ops->close_device(device) should be protected by group_rwsem or not. seems not, right? group_rwsem protects the fields under vfio_group. For the open_device/close_device() device->dev_set->lock is enough. Maybe another nit fix.

Quoted 2 lines

group->rwsem is to protect device->group->kvm from being changed by vfio_file_set_kvm() before it is copied to device->kvm.

yes. this is why vfio_device_open() holds the read lock of group_rwsem around the device->group->kvm copy. However, for the open_device(), callback, I don't think it is necessary to be protected by the group_rwsem lock.

Signature · 2 lines
Regards,
Yi Liu
#6 lore raw

RE: [Patch 1/1] vfio: Move "device->open_count--" out of group_rwsem in vfio_device_open()

Quoted 25 lines

From: Liu, Yi L <[email protected]>
Sent: Tuesday, June 21, 2022 10:59 AM

Quoted 9 lines
Quoted 9 lines
Quoted 6 lines

FWIW, this change now also drops group_rswem before setting device- kvm =
NULL, but that's also OK (again, just like vfio_device_fops_release) -- While the setting of device->kvm before open_device is technically done while holding the group_rwsem, this is done to protect the group kvm

value

Quoted 2 lines

we are copying from, and we should not be relying on that to protect the contents of device->kvm; instead we assume this value will not change

until

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

after the device is closed and while under the dev_set->lock.

yes. set device->kvm to be NULL has no need to hold group_rwsem. BTW.

I

Quoted 2 lines
Quoted 2 lines

also doubt whether the device->ops->open_device(device) and device->ops->close_device(device) should be protected by group_rwsem

or

Quoted 2 lines
Quoted 2 lines

not. seems not, right? group_rwsem protects the fields under vfio_group. For the open_device/close_device() device->dev_set->lock is enough.

Maybe

Quoted 3 lines
Quoted 2 lines

another nit fix.

Quoted 8 lines
Quoted 2 lines

group->rwsem is to protect device->group->kvm from being changed by vfio_file_set_kvm() before it is copied to device->kvm.

yes. this is why vfio_device_open() holds the read lock of group_rwsem around the device->group->kvm copy. However, for the open_device(), callback, I don't think it is necessary to be protected by the group_rwsem lock.

The kvm object could be freed after device->kvm is set, if group_rwsem is not held when calling open_device(). Then you'll hit another use-after-free bug when mdev driver tries to obtain a reference on kvm.

author
#7 lore raw

Re: [Patch 1/1] vfio: Move "device->open_count--" out of group_rwsem in vfio_device_open()

Quoted 34 lines · Kevin
Quoted 25 lines

From: Liu, Yi L <[email protected]>
Sent: Tuesday, June 21, 2022 10:59 AM

Quoted 9 lines
Quoted 9 lines
Quoted 6 lines

FWIW, this change now also drops group_rswem before setting device- kvm =
NULL, but that's also OK (again, just like vfio_device_fops_release) -- While the setting of device->kvm before open_device is technically done while holding the group_rwsem, this is done to protect the group kvm

value

Quoted 2 lines

we are copying from, and we should not be relying on that to protect the contents of device->kvm; instead we assume this value will not change

until

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

after the device is closed and while under the dev_set->lock.

yes. set device->kvm to be NULL has no need to hold group_rwsem. BTW.

I

Quoted 2 lines
Quoted 2 lines

also doubt whether the device->ops->open_device(device) and device->ops->close_device(device) should be protected by group_rwsem

or

Quoted 2 lines
Quoted 2 lines

not. seems not, right? group_rwsem protects the fields under vfio_group. For the open_device/close_device() device->dev_set->lock is enough.

Maybe

Quoted 3 lines
Quoted 2 lines

another nit fix.

Quoted 8 lines
Quoted 2 lines

group->rwsem is to protect device->group->kvm from being changed by vfio_file_set_kvm() before it is copied to device->kvm.

yes. this is why vfio_device_open() holds the read lock of group_rwsem around the device->group->kvm copy. However, for the open_device(), callback, I don't think it is necessary to be protected by the group_rwsem lock.

Quoted 4 lines

The kvm object could be freed after device->kvm is set, if group_rwsem is not held when calling open_device(). Then you'll hit another use-after-free bug when mdev driver tries to obtain a reference on kvm.

aha. I see. so group_rwsem prevents the kvm object free as such a thread should be blocked in the vfio_file_set_kvm() if the vfio_device_open() has held the group_rwsem. Hence kvm object is safe to be used. thanks!

Signature · 2 lines
Regards,
Yi Liu
#8 lore raw

Re: [Patch 1/1] vfio: Move "device->open_count--" out of group_rwsem in vfio_device_open()

In reply to Yi Liu
Quoted 1 line · Yi Liu

No need to protect open_count with group_rwsem

You should explain why this is a good change as you did in the emails here, and no cover letter required for single patches

Quoted 2 lines
Fixes 421cfe6596f6 ("vfio: remove VFIO_GROUP_NOTIFY_SET_KVM")

Extra space after fixes line.
No Fixes line for things that are not bugs.

Quoted 2 lines
cc Matthew Rosato <[email protected]>
cc Jason Gunthorpe <[email protected]>

Cc:

Quoted 4 lines
---
 drivers/vfio/vfio.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

It seems OK but a bit unnecessary.

Jason

author
#9 lore raw

Re: [Patch 1/1] vfio: Move "device->open_count--" out of group_rwsem in vfio_device_open()

Quoted 15 lines · Jason Gunthorpe
Quoted 1 line · Yi Liu

No need to protect open_count with group_rwsem

You should explain why this is a good change as you did in the emails here, and no cover letter required for single patches

Quoted 2 lines
Fixes 421cfe6596f6 ("vfio: remove VFIO_GROUP_NOTIFY_SET_KVM")

Extra space after fixes line.
No Fixes line for things that are not bugs.

Quoted 2 lines
cc Matthew Rosato <[email protected]>
cc Jason Gunthorpe <[email protected]>
Quoted 9 lines

Cc:

Quoted 4 lines
---
  drivers/vfio/vfio.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

It seems OK but a bit unnecessary.

thanks, Jason. all good suggestions.I'd just want to see the device->dev_set->lock and the group_rwsem hold/release is symmetric in the case of err_undo_count is hit in the vfio_device_open(). Just like the order in vfio_device_fops_release().

Signature · 2 lines
Regards,
Yi Liu