Skip to contentSkip to conversation

vfio: Get/put KVM only for the first/last vfio_df_open/close in cdev path

PATCH In review Yan Zhao 3 participants 8 messages Jun 28, 2024 – Jul 2, 2024

Subsystems (2)
lore
Share & tools
lore

Yan Zhao

author
#0 lore raw

PATCH 1/1 vfio: Get/put KVM only for the first/last vfio_df_open/close in cdev path

In the device cdev path, adjust the handling of the KVM reference count to only increment with the first vfio_df_open() and decrement after the final vfio_df_close(). This change addresses a KVM reference leak that occurs when a device cdev file is opened multiple times and attempts to bind to iommufd repeatedly.

Currently, vfio_df_get_kvm_safe() is invoked prior to each vfio_df_open() in the cdev path during iommufd binding. The corresponding vfio_device_put_kvm() is executed either when iommufd is unbound or if an error occurs during the binding process.

However, issues arise when a device binds to iommufd more than once. The second vfio_df_open() will fail during iommufd binding, and vfio_device_put_kvm() will be triggered, setting device->kvm to NULL. Consequently, when iommufd is unbound from the first successfully bound device, vfio_device_put_kvm() becomes ineffective, leading to a leak in the KVM reference count.

Below is the calltrace that will be produced in this scenario when the KVM module is unloaded afterwards, reporting "BUG kvm_vcpu (Tainted: G S):
Objects remaining in kvm_vcpu on __kmem_cache_shutdown()".

Call Trace:
 <TASK>
 dump_stack_lvl+0x80/0xc0
 slab_err+0xb0/0xf0
 ? __kmem_cache_shutdown+0xc1/0x4e0
 ? rcu_is_watching+0x11/0x50
 ? lock_acquired+0x144/0x3c0
 __kmem_cache_shutdown+0x1b7/0x4e0
 kmem_cache_destroy+0xa6/0x260
 kvm_exit+0x80/0xc0 [kvm]
 vmx_exit+0xe/0x20 [kvm_intel]
 __x64_sys_delete_module+0x143/0x250
 ? ktime_get_coarse_real_ts64+0xd3/0xe0
 ? syscall_trace_enter+0x143/0x210
 do_syscall_64+0x6f/0x140
 entry_SYSCALL_64_after_hwframe+0x76/0x7e
Fixes 5fcc26969a16 ("vfio: Add VFIO_DEVICE_BIND_IOMMUFD")
---
 drivers/vfio/device_cdev.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
drivers/vfio/device_cdev.c +9 -4
index bb1817bd4ff3..3b85d01d1b27 100644
@@ -65,6 +65,7 @@ long vfio_df_ioctl_bind_iommufd(struct vfio_device_file *df,
 {
 	struct vfio_device *device = df->device;
 	struct vfio_device_bind_iommufd bind;
+	bool put_kvm = false;
 	unsigned long minsz;
 	int ret;
 
@@ -101,12 +102,15 @@ long vfio_df_ioctl_bind_iommufd(struct vfio_device_file *df,
 	}
 
 	/*
-	 * Before the device open, get the KVM pointer currently
+	 * Before the device's first open, get the KVM pointer currently
 	 * associated with the device file (if there is) and obtain
-	 * a reference.  This reference is held until device closed.
+	 * a reference.  This reference is held until device's last closed.
 	 * Save the pointer in the device for use by drivers.
 	 */
-	vfio_df_get_kvm_safe(df);
+	if (device->open_count == 0) {
+		vfio_df_get_kvm_safe(df);
+		put_kvm = true;
+	}
 
 	ret = vfio_df_open(df);
 	if (ret)
@@ -129,7 +133,8 @@ long vfio_df_ioctl_bind_iommufd(struct vfio_device_file *df,
 out_close_device:
 	vfio_df_close(df);
 out_put_kvm:
-	vfio_device_put_kvm(device);
+	if (put_kvm)
+		vfio_device_put_kvm(device);
 	iommufd_ctx_put(df->iommufd);
 	df->iommufd = NULL;
 out_unlock:

base-commit: 6ba59ff4227927d3a8530fc2973b80e94b54d58f

Signature · 1 line
2.43.2
Reviewed-by on v1 1/1
#1 lore raw
Quoted 11 lines · Yan Zhao

In the device cdev path, adjust the handling of the KVM reference count to only increment with the first vfio_df_open() and decrement after the final vfio_df_close(). This change addresses a KVM reference leak that occurs when a device cdev file is opened multiple times and attempts to bind to iommufd repeatedly.

Currently, vfio_df_get_kvm_safe() is invoked prior to each vfio_df_open() in the cdev path during iommufd binding. The corresponding vfio_device_put_kvm() is executed either when iommufd is unbound or if an error occurs during the binding process.

Quoted 6 lines

However, issues arise when a device binds to iommufd more than once. The second vfio_df_open() will fail during iommufd binding, and vfio_device_put_kvm() will be triggered, setting device->kvm to NULL. Consequently, when iommufd is unbound from the first successfully bound device, vfio_device_put_kvm() becomes ineffective, leading to a leak in the KVM reference count.

Good catch!!!

Quoted 59 lines

Below is the calltrace that will be produced in this scenario when the KVM module is unloaded afterwards, reporting "BUG kvm_vcpu (Tainted: G S):
Objects remaining in kvm_vcpu on __kmem_cache_shutdown()".

Call Trace:
  <TASK>
  dump_stack_lvl+0x80/0xc0
  slab_err+0xb0/0xf0
  ? __kmem_cache_shutdown+0xc1/0x4e0
  ? rcu_is_watching+0x11/0x50
  ? lock_acquired+0x144/0x3c0
  __kmem_cache_shutdown+0x1b7/0x4e0
  kmem_cache_destroy+0xa6/0x260
  kvm_exit+0x80/0xc0 [kvm]
  vmx_exit+0xe/0x20 [kvm_intel]
  __x64_sys_delete_module+0x143/0x250
  ? ktime_get_coarse_real_ts64+0xd3/0xe0
  ? syscall_trace_enter+0x143/0x210
  do_syscall_64+0x6f/0x140
  entry_SYSCALL_64_after_hwframe+0x76/0x7e
Fixes 5fcc26969a16 ("vfio: Add VFIO_DEVICE_BIND_IOMMUFD")
---
  drivers/vfio/device_cdev.c | 13 +++++++++----
  1 file changed, 9 insertions(+), 4 deletions(-)
drivers/vfio/device_cdev.c +7 -3
index bb1817bd4ff3..3b85d01d1b27 100644
@@ -65,6 +65,7 @@ long vfio_df_ioctl_bind_iommufd(struct vfio_device_file *df,
  {
  	struct vfio_device *device = df->device;
  	struct vfio_device_bind_iommufd bind;
+	bool put_kvm = false;
  	unsigned long minsz;
  	int ret;
 
@@ -101,12 +102,15 @@ long vfio_df_ioctl_bind_iommufd(struct vfio_device_file *df,
  	}
 
  	/*
-	 * Before the device open, get the KVM pointer currently
+	 * Before the device's first open, get the KVM pointer currently
  	 * associated with the device file (if there is) and obtain
-	 * a reference.  This reference is held until device closed.
+	 * a reference.  This reference is held until device's last closed.
  	 * Save the pointer in the device for use by drivers.
  	 */
-	vfio_df_get_kvm_safe(df);
+	if (device->open_count == 0) {
+		vfio_df_get_kvm_safe(df);
+		put_kvm = true;
+	}
 
  	ret = vfio_df_open(df);
  	if (ret)

drivers/vfio/device_cdev.c long vfio_df_ioctl_bind_iommufd(struct vfio_device_file *df, · line 136

Quoted 5 lines
drivers/vfio/device_cdev.c +1 -1
@@ -129,7 +133,8 @@ long vfio_df_ioctl_bind_iommufd(struct vfio_device_file *df,
  out_close_device:
  	vfio_df_close(df);
  out_put_kvm:
-	vfio_device_put_kvm(device);
+	if (put_kvm)

you may use if (device->open_count == 0) as well here to save a bool. Otherwise looks good to me.

Quoted 6 lines
drivers/vfio/device_cdev.c +1 -0
+		vfio_device_put_kvm(device);
  	iommufd_ctx_put(df->iommufd);
  	df->iommufd = NULL;
  out_unlock:

base-commit: 6ba59ff4227927d3a8530fc2973b80e94b54d58f

BTW. The vfio_device_get_kvm_safe() is not supposed to be invoked multiple times by design as the second call would override the device->put_kvm and device->kvm. This does not change the put_kvm nor the kvm though. But not a good thing anyghow. maybe worth a warn like below.

drivers/vfio/vfio_main.c +2 -0
index ee72c0b61795..a4bead0e5820 100644
@@ -408,6 +408,8 @@ void vfio_device_get_kvm_safe(struct vfio_device *device, struct kvm *kvm)
 	if (!kvm)
 		return;
 
+	WARN_ON(device->put_kvm || device->kvm);
+
 	pfn = symbol_get(kvm_put_kvm);
 	if (WARN_ON(!pfn))
 		return;
Signature · 2 lines
Regards,
Yi Liu
author
#2 lore raw
Quoted 77 lines · Yi Liu
Quoted 11 lines · Yan Zhao

In the device cdev path, adjust the handling of the KVM reference count to only increment with the first vfio_df_open() and decrement after the final vfio_df_close(). This change addresses a KVM reference leak that occurs when a device cdev file is opened multiple times and attempts to bind to iommufd repeatedly.

Currently, vfio_df_get_kvm_safe() is invoked prior to each vfio_df_open() in the cdev path during iommufd binding. The corresponding vfio_device_put_kvm() is executed either when iommufd is unbound or if an error occurs during the binding process.

Quoted 6 lines

However, issues arise when a device binds to iommufd more than once. The second vfio_df_open() will fail during iommufd binding, and vfio_device_put_kvm() will be triggered, setting device->kvm to NULL. Consequently, when iommufd is unbound from the first successfully bound device, vfio_device_put_kvm() becomes ineffective, leading to a leak in the KVM reference count.

Good catch!!!

Quoted 56 lines

Below is the calltrace that will be produced in this scenario when the KVM module is unloaded afterwards, reporting "BUG kvm_vcpu (Tainted: G S):
Objects remaining in kvm_vcpu on __kmem_cache_shutdown()".

Call Trace:
  <TASK>
  dump_stack_lvl+0x80/0xc0
  slab_err+0xb0/0xf0
  ? __kmem_cache_shutdown+0xc1/0x4e0
  ? rcu_is_watching+0x11/0x50
  ? lock_acquired+0x144/0x3c0
  __kmem_cache_shutdown+0x1b7/0x4e0
  kmem_cache_destroy+0xa6/0x260
  kvm_exit+0x80/0xc0 [kvm]
  vmx_exit+0xe/0x20 [kvm_intel]
  __x64_sys_delete_module+0x143/0x250
  ? ktime_get_coarse_real_ts64+0xd3/0xe0
  ? syscall_trace_enter+0x143/0x210
  do_syscall_64+0x6f/0x140
  entry_SYSCALL_64_after_hwframe+0x76/0x7e
Fixes 5fcc26969a16 ("vfio: Add VFIO_DEVICE_BIND_IOMMUFD")
---
  drivers/vfio/device_cdev.c | 13 +++++++++----
  1 file changed, 9 insertions(+), 4 deletions(-)
drivers/vfio/device_cdev.c +7 -3
index bb1817bd4ff3..3b85d01d1b27 100644
@@ -65,6 +65,7 @@ long vfio_df_ioctl_bind_iommufd(struct vfio_device_file *df,
  {
  	struct vfio_device *device = df->device;
  	struct vfio_device_bind_iommufd bind;
+	bool put_kvm = false;
  	unsigned long minsz;
  	int ret;
@@ -101,12 +102,15 @@ long vfio_df_ioctl_bind_iommufd(struct vfio_device_file *df,
  	}
  	/*
-	 * Before the device open, get the KVM pointer currently
+	 * Before the device's first open, get the KVM pointer currently
  	 * associated with the device file (if there is) and obtain
-	 * a reference.  This reference is held until device closed.
+	 * a reference.  This reference is held until device's last closed.
  	 * Save the pointer in the device for use by drivers.
  	 */
-	vfio_df_get_kvm_safe(df);
+	if (device->open_count == 0) {
+		vfio_df_get_kvm_safe(df);
+		put_kvm = true;
+	}
  	ret = vfio_df_open(df);
  	if (ret)
@@ -129,7 +133,8 @@ long vfio_df_ioctl_bind_iommufd(struct vfio_device_file *df,
Quoted 8 lines

drivers/vfio/device_cdev.c long vfio_df_ioctl_bind_iommufd(struct vfio_device_file *df, · line 136

Quoted 5 lines
drivers/vfio/device_cdev.c +1 -1
  out_close_device:
  	vfio_df_close(df);
  out_put_kvm:
-	vfio_device_put_kvm(device);
+	if (put_kvm)

you may use if (device->open_count == 0) as well here to save a bool. Otherwise looks good to me.

Upon here, device->open_count is not necessarily 0 even for the first open. The failure can be after a successful increment of device->open_count.

Maybe renaming "bool put_kvm" to "bool is_first_open" to save an assignment in most common case?

Quoted 1 line

Thanks:)

Quoted 11 lines
Quoted 7 lines
drivers/vfio/device_cdev.c +1 -0
+		vfio_device_put_kvm(device);
  	iommufd_ctx_put(df->iommufd);
  	df->iommufd = NULL;
  out_unlock:

base-commit: 6ba59ff4227927d3a8530fc2973b80e94b54d58f

BTW. The vfio_device_get_kvm_safe() is not supposed to be invoked multiple times by design as the second call would override the device->put_kvm and device->kvm. This does not change the put_kvm nor the kvm though. But not a

"kvm" may also be changed if the second bind is from a different VM.

drivers/vfio/vfio_main.c void vfio_device_get_kvm_safe(struct vfio_device · line 412

Quoted 12 lines

good thing anyghow. maybe worth a warn like below.

drivers/vfio/vfio_main.c +1 -0
index ee72c0b61795..a4bead0e5820 100644
@@ -408,6 +408,8 @@ void vfio_device_get_kvm_safe(struct vfio_device
 *device, struct kvm *kvm)
 	if (!kvm)
 		return;
 
+	WARN_ON(device->put_kvm || device->kvm);

Yes, better.

Quoted 7 lines
drivers/vfio/vfio_main.c +0 -0
 	pfn = symbol_get(kvm_put_kvm);
 	if (WARN_ON(!pfn))
 		return;
Signature · 3 lines
-- 
Regards,
Yi Liu
#3 lore raw
Quoted 101 lines · Yan Zhao
Quoted 77 lines · Yi Liu
Quoted 11 lines · Yan Zhao

In the device cdev path, adjust the handling of the KVM reference count to only increment with the first vfio_df_open() and decrement after the final vfio_df_close(). This change addresses a KVM reference leak that occurs when a device cdev file is opened multiple times and attempts to bind to iommufd repeatedly.

Currently, vfio_df_get_kvm_safe() is invoked prior to each vfio_df_open() in the cdev path during iommufd binding. The corresponding vfio_device_put_kvm() is executed either when iommufd is unbound or if an error occurs during the binding process.

Quoted 6 lines

However, issues arise when a device binds to iommufd more than once. The second vfio_df_open() will fail during iommufd binding, and vfio_device_put_kvm() will be triggered, setting device->kvm to NULL. Consequently, when iommufd is unbound from the first successfully bound device, vfio_device_put_kvm() becomes ineffective, leading to a leak in the KVM reference count.

Good catch!!!

Quoted 56 lines

Below is the calltrace that will be produced in this scenario when the KVM module is unloaded afterwards, reporting "BUG kvm_vcpu (Tainted: G S):
Objects remaining in kvm_vcpu on __kmem_cache_shutdown()".

Call Trace:
   <TASK>
   dump_stack_lvl+0x80/0xc0
   slab_err+0xb0/0xf0
   ? __kmem_cache_shutdown+0xc1/0x4e0
   ? rcu_is_watching+0x11/0x50
   ? lock_acquired+0x144/0x3c0
   __kmem_cache_shutdown+0x1b7/0x4e0
   kmem_cache_destroy+0xa6/0x260
   kvm_exit+0x80/0xc0 [kvm]
   vmx_exit+0xe/0x20 [kvm_intel]
   __x64_sys_delete_module+0x143/0x250
   ? ktime_get_coarse_real_ts64+0xd3/0xe0
   ? syscall_trace_enter+0x143/0x210
   do_syscall_64+0x6f/0x140
   entry_SYSCALL_64_after_hwframe+0x76/0x7e
Fixes 5fcc26969a16 ("vfio: Add VFIO_DEVICE_BIND_IOMMUFD")
---
   drivers/vfio/device_cdev.c | 13 +++++++++----
   1 file changed, 9 insertions(+), 4 deletions(-)
drivers/vfio/device_cdev.c +7 -3
index bb1817bd4ff3..3b85d01d1b27 100644
@@ -65,6 +65,7 @@ long vfio_df_ioctl_bind_iommufd(struct vfio_device_file *df,
   {
   	struct vfio_device *device = df->device;
   	struct vfio_device_bind_iommufd bind;
+	bool put_kvm = false;
   	unsigned long minsz;
   	int ret;
@@ -101,12 +102,15 @@ long vfio_df_ioctl_bind_iommufd(struct vfio_device_file *df,
   	}
   	/*
-	 * Before the device open, get the KVM pointer currently
+	 * Before the device's first open, get the KVM pointer currently
   	 * associated with the device file (if there is) and obtain
-	 * a reference.  This reference is held until device closed.
+	 * a reference.  This reference is held until device's last closed.
   	 * Save the pointer in the device for use by drivers.
   	 */
-	vfio_df_get_kvm_safe(df);
+	if (device->open_count == 0) {
+		vfio_df_get_kvm_safe(df);
+		put_kvm = true;
+	}
   	ret = vfio_df_open(df);
   	if (ret)
@@ -129,7 +133,8 @@ long vfio_df_ioctl_bind_iommufd(struct vfio_device_file *df,
Quoted 8 lines

drivers/vfio/device_cdev.c long vfio_df_ioctl_bind_iommufd(struct vfio_device_file *df, · line 136

Quoted 5 lines
drivers/vfio/device_cdev.c +1 -1
   out_close_device:
   	vfio_df_close(df);
   out_put_kvm:
-	vfio_device_put_kvm(device);
+	if (put_kvm)

you may use if (device->open_count == 0) as well here to save a bool. Otherwise looks good to me.

Upon here, device->open_count is not necessarily 0 even for the first open. The failure can be after a successful increment of device->open_count.

Maybe renaming "bool put_kvm" to "bool is_first_open" to save an assignment in most common case?

Quoted 1 line

Thanks:)

Quoted 6 lines
Quoted 6 lines
drivers/vfio/device_cdev.c +1 -0
+		vfio_device_put_kvm(device);
   	iommufd_ctx_put(df->iommufd);
   	df->iommufd = NULL;
   out_unlock:
Quoted 7 lines
Quoted 5 lines
Quoted 1 line

base-commit: 6ba59ff4227927d3a8530fc2973b80e94b54d58f

BTW. The vfio_device_get_kvm_safe() is not supposed to be invoked multiple times by design as the second call would override the device->put_kvm and device->kvm. This does not change the put_kvm nor the kvm though. But not a

"kvm" may also be changed if the second bind is from a different VM.

yep.

Quoted 21 lines

drivers/vfio/vfio_main.c void vfio_device_get_kvm_safe(struct vfio_device · line 412

Quoted 12 lines

good thing anyghow. maybe worth a warn like below.

drivers/vfio/vfio_main.c +1 -0
index ee72c0b61795..a4bead0e5820 100644
@@ -408,6 +408,8 @@ void vfio_device_get_kvm_safe(struct vfio_device
 *device, struct kvm *kvm)
  	if (!kvm)
  		return;
 
+	WARN_ON(device->put_kvm || device->kvm);

Yes, better.

Quoted 7 lines
drivers/vfio/vfio_main.c +0 -0
  	pfn = symbol_get(kvm_put_kvm);
  	if (WARN_ON(!pfn))
  		return;
Signature · 3 lines
-- 
Regards,
Yi Liu
Signature · 2 lines
Regards,
Yi Liu
#4 lore raw
In reply to Yan Zhao
Quoted 14 lines

From: Zhao, Yan Y <[email protected]>
Sent: Friday, June 28, 2024 11:19 PM

In the device cdev path, adjust the handling of the KVM reference count to only increment with the first vfio_df_open() and decrement after the final vfio_df_close(). This change addresses a KVM reference leak that occurs when a device cdev file is opened multiple times and attempts to bind to iommufd repeatedly.

Currently, vfio_df_get_kvm_safe() is invoked prior to each vfio_df_open() in the cdev path during iommufd binding. The corresponding vfio_device_put_kvm() is executed either when iommufd is unbound or if an error occurs during the binding process.

Quoted 6 lines

However, issues arise when a device binds to iommufd more than once. The second vfio_df_open() will fail during iommufd binding, and vfio_device_put_kvm() will be triggered, setting device->kvm to NULL. Consequently, when iommufd is unbound from the first successfully bound device, vfio_device_put_kvm() becomes ineffective, leading to a leak in the KVM reference count.

To be accurate this happens only when two binds are issued via different fds otherwise below check will happen earlier when two binds are in a same fd:

	/* one device cannot be bound twice */
	if (df->access_granted) {
		ret = -EINVAL;
		goto out_unlock;
	}
Quoted 71 lines

Below is the calltrace that will be produced in this scenario when the KVM module is unloaded afterwards, reporting "BUG kvm_vcpu (Tainted: G S):
Objects remaining in kvm_vcpu on __kmem_cache_shutdown()".

Call Trace:
 <TASK>
 dump_stack_lvl+0x80/0xc0
 slab_err+0xb0/0xf0
 ? __kmem_cache_shutdown+0xc1/0x4e0
 ? rcu_is_watching+0x11/0x50
 ? lock_acquired+0x144/0x3c0
 __kmem_cache_shutdown+0x1b7/0x4e0
 kmem_cache_destroy+0xa6/0x260
 kvm_exit+0x80/0xc0 [kvm]
 vmx_exit+0xe/0x20 [kvm_intel]
 __x64_sys_delete_module+0x143/0x250
 ? ktime_get_coarse_real_ts64+0xd3/0xe0
 ? syscall_trace_enter+0x143/0x210
 do_syscall_64+0x6f/0x140
 entry_SYSCALL_64_after_hwframe+0x76/0x7e
Fixes 5fcc26969a16 ("vfio: Add VFIO_DEVICE_BIND_IOMMUFD")
---
 drivers/vfio/device_cdev.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
drivers/vfio/device_cdev.c +7 -3
index bb1817bd4ff3..3b85d01d1b27 100644
@@ -65,6 +65,7 @@ long vfio_df_ioctl_bind_iommufd(struct
 vfio_device_file *df,
 {
 	struct vfio_device *device = df->device;
 	struct vfio_device_bind_iommufd bind;
+	bool put_kvm = false;
 	unsigned long minsz;
 	int ret;
@@ -101,12 +102,15 @@ long vfio_df_ioctl_bind_iommufd(struct
 vfio_device_file *df,
 	}
 
 	/*
-	 * Before the device open, get the KVM pointer currently
+	 * Before the device's first open, get the KVM pointer currently
 	 * associated with the device file (if there is) and obtain
-	 * a reference.  This reference is held until device closed.
+	 * a reference.  This reference is held until device's last closed.
 	 * Save the pointer in the device for use by drivers.
 	 */
-	vfio_df_get_kvm_safe(df);
+	if (device->open_count == 0) {
+		vfio_df_get_kvm_safe(df);
+		put_kvm = true;
+	}
 
 	ret = vfio_df_open(df);
drivers/vfio/device_cdev.c +2 -1
 	if (ret)
@@ -129,7 +133,8 @@ long vfio_df_ioctl_bind_iommufd(struct
 vfio_device_file *df,
 out_close_device:
 	vfio_df_close(df);
 out_put_kvm:
-	vfio_device_put_kvm(device);
+	if (put_kvm)
+		vfio_device_put_kvm(device);
 	iommufd_ctx_put(df->iommufd);
 	df->iommufd = NULL;
Quoted 2 lines

out_unlock:

what about extending vfio_df_open() to unify the get/put_kvm() and open_count trick in one place?

int vfio_df_open(struct vfio_device_file *df, struct kvm *kvm,
spinlock_t *kvm_ref_lock)

{
	struct vfio_device *device = df->device;
	int ret = 0;
	
	lockdep_assert_held(&device->dev_set->lock);

	if (device->open_count == 0) {
		spin_lock(kvm_ref_lock);
		vfio_device_get_kvm_safe(device, kvm);
		spin_unlock(kvm_ref_lock);
	}

	/*
	 * Only the group path allows the device to be opened multiple
	 * times.  The device cdev path doesn't have a secure way for it.
	 */
	if (device->open_count != 0 && !df->group)
		return -EINVAL;

	device->open_count++;
	if (device->open_count == 1) {
		ret = vfio_df_device_first_open(df);
		if (ret)
			device->open_count--;
	}

	if (ret)
		vfio_device_put_kvm(device);
	return ret;
}

void vfio_df_close(struct vfio_device_file *df)
{
struct vfio_device *device = df->device;

	lockdep_assert_held(&device->dev_set->lock);

	vfio_assert_device_open(device);
	if (device->open_count == 1) {
		vfio_df_device_last_close(df);
		vfio_device_put_kvm(device);
	}
	device->open_count--;
}
#5 lore raw
Quoted 25 lines · Kevin
Quoted 14 lines

From: Zhao, Yan Y <[email protected]>
Sent: Friday, June 28, 2024 11:19 PM

In the device cdev path, adjust the handling of the KVM reference count to only increment with the first vfio_df_open() and decrement after the final vfio_df_close(). This change addresses a KVM reference leak that occurs when a device cdev file is opened multiple times and attempts to bind to iommufd repeatedly.

Currently, vfio_df_get_kvm_safe() is invoked prior to each vfio_df_open() in the cdev path during iommufd binding. The corresponding vfio_device_put_kvm() is executed either when iommufd is unbound or if an error occurs during the binding process.

Quoted 6 lines

However, issues arise when a device binds to iommufd more than once. The second vfio_df_open() will fail during iommufd binding, and vfio_device_put_kvm() will be triggered, setting device->kvm to NULL. Consequently, when iommufd is unbound from the first successfully bound device, vfio_device_put_kvm() becomes ineffective, leading to a leak in the KVM reference count.

To be accurate this happens only when two binds are issued via different fds otherwise below check will happen earlier when two binds are in a same fd:

Quoted 5 lines
	/* one device cannot be bound twice */
	if (df->access_granted) {
		ret = -EINVAL;
		goto out_unlock;
	}

yes

Quoted 77 lines
Quoted 71 lines

Below is the calltrace that will be produced in this scenario when the KVM module is unloaded afterwards, reporting "BUG kvm_vcpu (Tainted: G S):
Objects remaining in kvm_vcpu on __kmem_cache_shutdown()".

Call Trace:
  <TASK>
  dump_stack_lvl+0x80/0xc0
  slab_err+0xb0/0xf0
  ? __kmem_cache_shutdown+0xc1/0x4e0
  ? rcu_is_watching+0x11/0x50
  ? lock_acquired+0x144/0x3c0
  __kmem_cache_shutdown+0x1b7/0x4e0
  kmem_cache_destroy+0xa6/0x260
  kvm_exit+0x80/0xc0 [kvm]
  vmx_exit+0xe/0x20 [kvm_intel]
  __x64_sys_delete_module+0x143/0x250
  ? ktime_get_coarse_real_ts64+0xd3/0xe0
  ? syscall_trace_enter+0x143/0x210
  do_syscall_64+0x6f/0x140
  entry_SYSCALL_64_after_hwframe+0x76/0x7e
Fixes 5fcc26969a16 ("vfio: Add VFIO_DEVICE_BIND_IOMMUFD")
---
  drivers/vfio/device_cdev.c | 13 +++++++++----
  1 file changed, 9 insertions(+), 4 deletions(-)
drivers/vfio/device_cdev.c +7 -3
index bb1817bd4ff3..3b85d01d1b27 100644
@@ -65,6 +65,7 @@ long vfio_df_ioctl_bind_iommufd(struct
 vfio_device_file *df,
  {
  	struct vfio_device *device = df->device;
  	struct vfio_device_bind_iommufd bind;
+	bool put_kvm = false;
  	unsigned long minsz;
  	int ret;
@@ -101,12 +102,15 @@ long vfio_df_ioctl_bind_iommufd(struct
 vfio_device_file *df,
  	}
 
  	/*
-	 * Before the device open, get the KVM pointer currently
+	 * Before the device's first open, get the KVM pointer currently
  	 * associated with the device file (if there is) and obtain
-	 * a reference.  This reference is held until device closed.
+	 * a reference.  This reference is held until device's last closed.
  	 * Save the pointer in the device for use by drivers.
  	 */
-	vfio_df_get_kvm_safe(df);
+	if (device->open_count == 0) {
+		vfio_df_get_kvm_safe(df);
+		put_kvm = true;
+	}
 
  	ret = vfio_df_open(df);

if (ret)

drivers/vfio/device_cdev.c +2 -1
@@ -129,7 +133,8 @@ long vfio_df_ioctl_bind_iommufd(struct
 vfio_device_file *df,
  out_close_device:
  	vfio_df_close(df);
  out_put_kvm:
-	vfio_device_put_kvm(device);
+	if (put_kvm)
+		vfio_device_put_kvm(device);
  	iommufd_ctx_put(df->iommufd);
  	df->iommufd = NULL;
Quoted 2 lines

out_unlock:

what about extending vfio_df_open() to unify the get/put_kvm() and open_count trick in one place?

Quoted 3 lines

int vfio_df_open(struct vfio_device_file *df, struct kvm *kvm,
spinlock_t *kvm_ref_lock)
{

this should work. But need a comment to note why need pass in both kvm and kvm_ref_lock given df has both of them. :)

Quoted 10 lines
	struct vfio_device *device = df->device;
	int ret = 0;
	
	lockdep_assert_held(&device->dev_set->lock);

	if (device->open_count == 0) {
		spin_lock(kvm_ref_lock);
		vfio_device_get_kvm_safe(device, kvm);
		spin_unlock(kvm_ref_lock);
	}

perhaps it can be put in the "if (device->open_count == 1)" branch, just before invoking vfio_df_device_first_open().

Quoted 33 lines
	/*
	 * Only the group path allows the device to be opened multiple
	 * times.  The device cdev path doesn't have a secure way for it.
	 */
	if (device->open_count != 0 && !df->group)
		return -EINVAL;

	device->open_count++;
	if (device->open_count == 1) {
		ret = vfio_df_device_first_open(df);
		if (ret)
			device->open_count--;
	}

	if (ret)
		vfio_device_put_kvm(device);
	return ret;
}

void vfio_df_close(struct vfio_device_file *df)
{
struct vfio_device *device = df->device;

	lockdep_assert_held(&device->dev_set->lock);

	vfio_assert_device_open(device);
	if (device->open_count == 1) {
		vfio_df_device_last_close(df);
		vfio_device_put_kvm(device);
	}
	device->open_count--;
}
Signature · 2 lines
Regards,
Yi Liu
author
#6 lore raw
Quoted 111 lines · Yi Liu
Quoted 25 lines · Kevin
Quoted 14 lines

From: Zhao, Yan Y <[email protected]>
Sent: Friday, June 28, 2024 11:19 PM

In the device cdev path, adjust the handling of the KVM reference count to only increment with the first vfio_df_open() and decrement after the final vfio_df_close(). This change addresses a KVM reference leak that occurs when a device cdev file is opened multiple times and attempts to bind to iommufd repeatedly.

Currently, vfio_df_get_kvm_safe() is invoked prior to each vfio_df_open() in the cdev path during iommufd binding. The corresponding vfio_device_put_kvm() is executed either when iommufd is unbound or if an error occurs during the binding process.

Quoted 6 lines

However, issues arise when a device binds to iommufd more than once. The second vfio_df_open() will fail during iommufd binding, and vfio_device_put_kvm() will be triggered, setting device->kvm to NULL. Consequently, when iommufd is unbound from the first successfully bound device, vfio_device_put_kvm() becomes ineffective, leading to a leak in the KVM reference count.

To be accurate this happens only when two binds are issued via different fds otherwise below check will happen earlier when two binds are in a same fd:

Quoted 5 lines
	/* one device cannot be bound twice */
	if (df->access_granted) {
		ret = -EINVAL;
		goto out_unlock;
	}

yes

Quoted 77 lines
Quoted 71 lines

Below is the calltrace that will be produced in this scenario when the KVM module is unloaded afterwards, reporting "BUG kvm_vcpu (Tainted: G S):
Objects remaining in kvm_vcpu on __kmem_cache_shutdown()".

Call Trace:
  <TASK>
  dump_stack_lvl+0x80/0xc0
  slab_err+0xb0/0xf0
  ? __kmem_cache_shutdown+0xc1/0x4e0
  ? rcu_is_watching+0x11/0x50
  ? lock_acquired+0x144/0x3c0
  __kmem_cache_shutdown+0x1b7/0x4e0
  kmem_cache_destroy+0xa6/0x260
  kvm_exit+0x80/0xc0 [kvm]
  vmx_exit+0xe/0x20 [kvm_intel]
  __x64_sys_delete_module+0x143/0x250
  ? ktime_get_coarse_real_ts64+0xd3/0xe0
  ? syscall_trace_enter+0x143/0x210
  do_syscall_64+0x6f/0x140
  entry_SYSCALL_64_after_hwframe+0x76/0x7e
Fixes 5fcc26969a16 ("vfio: Add VFIO_DEVICE_BIND_IOMMUFD")
---
  drivers/vfio/device_cdev.c | 13 +++++++++----
  1 file changed, 9 insertions(+), 4 deletions(-)
drivers/vfio/device_cdev.c +7 -3
index bb1817bd4ff3..3b85d01d1b27 100644
@@ -65,6 +65,7 @@ long vfio_df_ioctl_bind_iommufd(struct
 vfio_device_file *df,
  {
  	struct vfio_device *device = df->device;
  	struct vfio_device_bind_iommufd bind;
+	bool put_kvm = false;
  	unsigned long minsz;
  	int ret;
@@ -101,12 +102,15 @@ long vfio_df_ioctl_bind_iommufd(struct
 vfio_device_file *df,
  	}
 
  	/*
-	 * Before the device open, get the KVM pointer currently
+	 * Before the device's first open, get the KVM pointer currently
  	 * associated with the device file (if there is) and obtain
-	 * a reference.  This reference is held until device closed.
+	 * a reference.  This reference is held until device's last closed.
  	 * Save the pointer in the device for use by drivers.
  	 */
-	vfio_df_get_kvm_safe(df);
+	if (device->open_count == 0) {
+		vfio_df_get_kvm_safe(df);
+		put_kvm = true;
+	}
 
  	ret = vfio_df_open(df);

if (ret)

drivers/vfio/device_cdev.c +2 -1
@@ -129,7 +133,8 @@ long vfio_df_ioctl_bind_iommufd(struct
 vfio_device_file *df,
  out_close_device:
  	vfio_df_close(df);
  out_put_kvm:
-	vfio_device_put_kvm(device);
+	if (put_kvm)
+		vfio_device_put_kvm(device);
  	iommufd_ctx_put(df->iommufd);
  	df->iommufd = NULL;
Quoted 2 lines

out_unlock:

what about extending vfio_df_open() to unify the get/put_kvm() and open_count trick in one place?

Quoted 6 lines
Quoted 3 lines

int vfio_df_open(struct vfio_device_file *df, struct kvm *kvm,
spinlock_t *kvm_ref_lock)
{

this should work. But need a comment to note why need pass in both kvm and kvm_ref_lock given df has both of them. :)

So why to pass them?

What about making vfio_device_group_get_kvm_safe() or vfio_df_get_kvm_safe() not static and calling one of them in vfio_df_open() according to the df->group ?

Quoted 6 lines
Quoted 6 lines
	struct vfio_device *device = df->device;
	int ret = 0;
	
	lockdep_assert_held(&device->dev_set->lock);
Quoted 8 lines
Quoted 5 lines
	if (device->open_count == 0) {
		spin_lock(kvm_ref_lock);
		vfio_device_get_kvm_safe(device, kvm);
		spin_unlock(kvm_ref_lock);
	}

perhaps it can be put in the "if (device->open_count == 1)" branch, just before invoking vfio_df_device_first_open().

What about just moving the get/put_kvm into vfio_df_device_first_open()?

Quoted 37 lines
Quoted 33 lines
	/*
	 * Only the group path allows the device to be opened multiple
	 * times.  The device cdev path doesn't have a secure way for it.
	 */
	if (device->open_count != 0 && !df->group)
		return -EINVAL;

	device->open_count++;
	if (device->open_count == 1) {
		ret = vfio_df_device_first_open(df);
		if (ret)
			device->open_count--;
	}

	if (ret)
		vfio_device_put_kvm(device);
	return ret;
}

void vfio_df_close(struct vfio_device_file *df)
{
struct vfio_device *device = df->device;

	lockdep_assert_held(&device->dev_set->lock);

	vfio_assert_device_open(device);
	if (device->open_count == 1) {
		vfio_df_device_last_close(df);
		vfio_device_put_kvm(device);
	}
	device->open_count--;
}
Signature · 2 lines
Regards,
Yi Liu
#7 lore raw
Quoted 9 lines

From: Zhao, Yan Y <[email protected]>
Sent: Monday, July 1, 2024 8:02 PM

Quoted 5 lines · Yi Liu
Quoted 4 lines · Kevin

what about extending vfio_df_open() to unify the get/put_kvm() and open_count trick in one place?

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

int vfio_df_open(struct vfio_device_file *df, struct kvm *kvm,
spinlock_t *kvm_ref_lock)
{

this should work. But need a comment to note why need pass in both kvm and kvm_ref_lock given df has both of them. :)

So why to pass them?

hmm actually passing them is wrong especially for the group path. We have to get kvm upon the first reference to the pointer otherwise it is prone to use-after-free issue.

Quoted 7 lines

What about making vfio_device_group_get_kvm_safe() or vfio_df_get_kvm_safe()
not static and calling one of them in vfio_df_open() according to the df-

Quoted 1 line

group

?

yeah, this sounds better.