Skip to contentSkip to conversation

Drivers: hv: vmbus: Fix kernel crash upon unbinding a device from uio_hv_generic driver

PATCH Applied Vitaly Kuznetsov 4 participants 6 messages Aug 31, 2021 – Sep 3, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Drivers: hv: vmbus: Fix kernel crash upon unbinding a device from uio_hv_generic driver

The following crash happens when a never-used device is unbound from uio_hv_generic driver:

kernel BUG at mm/slub.c:321!
invalid opcode: 0000 [#1] SMP PTI

 CPU: 0 PID: 4001 Comm: bash Kdump: loaded Tainted: G               X --------- ---  5.14.0-0.rc2.23.el9.x86_64 #1
 Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008  12/07/2018
 RIP: 0010:__slab_free+0x1d5/0x3d0
...
 Call Trace:
  ? pick_next_task_fair+0x18e/0x3b0
  ? __cond_resched+0x16/0x40
  ? vunmap_pmd_range.isra.0+0x154/0x1c0
  ? __vunmap+0x22d/0x290
  ? hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
  kfree+0x331/0x380
  ? hv_uio_remove+0x43/0x60 [uio_hv_generic]
  hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
  vmbus_free_ring+0x21/0x60 [hv_vmbus]
  hv_uio_remove+0x4f/0x60 [uio_hv_generic]
  vmbus_remove+0x23/0x30 [hv_vmbus]
  __device_release_driver+0x17a/0x230
  device_driver_detach+0x3c/0xa0
  unbind_store+0x113/0x130

...

The problem appears to be that we free 'ring_info->pkt_buffer' twice:
first, when the device is unbound from in-kernel driver (netvsc in this case) and second from hv_uio_remove(). Normally, ring buffer is supposed to be re-initialized from hv_uio_open() but this happens when UIO device is being opened and this is not guaranteed to happen.

Generally, it is OK to call hv_ringbuffer_cleanup() twice for the same channel (which is being handed over between in-kernel drivers and UIO) even if we didn't call hv_ringbuffer_init() in between. We, however, need to avoid kfree() call for an already freed pointer.

Fixes adae1e931acd ("Drivers: hv: vmbus: Copy packets sent by Hyper-V out of the ring buffer")
---
 drivers/hv/ring_buffer.c | 1 +
 1 file changed, 1 insertion(+)
drivers/hv/ring_buffer.c +1 -0
index 2aee356840a2..314015d9e912 100644
@@ -245,6 +245,7 @@ void hv_ringbuffer_cleanup(struct hv_ring_buffer_info *ring_info)
 	mutex_unlock(&ring_info->ring_buffer_mutex);
 
 	kfree(ring_info->pkt_buffer);
+	ring_info->pkt_buffer = NULL;
 	ring_info->pkt_buffer_size = 0;
 }
 
Signature · 1 line
2.31.1
maintainer
#1 lore raw
Quoted 48 lines · Vitaly Kuznetsov

The following crash happens when a never-used device is unbound from uio_hv_generic driver:

kernel BUG at mm/slub.c:321!
invalid opcode: 0000 [#1] SMP PTI

 CPU: 0 PID: 4001 Comm: bash Kdump: loaded Tainted: G               X --------- ---  5.14.0-0.rc2.23.el9.x86_64 #1
 Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008  12/07/2018
 RIP: 0010:__slab_free+0x1d5/0x3d0
...
 Call Trace:
  ? pick_next_task_fair+0x18e/0x3b0
  ? __cond_resched+0x16/0x40
  ? vunmap_pmd_range.isra.0+0x154/0x1c0
  ? __vunmap+0x22d/0x290
  ? hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
  kfree+0x331/0x380
  ? hv_uio_remove+0x43/0x60 [uio_hv_generic]
  hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
  vmbus_free_ring+0x21/0x60 [hv_vmbus]
  hv_uio_remove+0x4f/0x60 [uio_hv_generic]
  vmbus_remove+0x23/0x30 [hv_vmbus]
  __device_release_driver+0x17a/0x230
  device_driver_detach+0x3c/0xa0
  unbind_store+0x113/0x130

...

The problem appears to be that we free 'ring_info->pkt_buffer' twice:
first, when the device is unbound from in-kernel driver (netvsc in this case) and second from hv_uio_remove(). Normally, ring buffer is supposed to be re-initialized from hv_uio_open() but this happens when UIO device is being opened and this is not guaranteed to happen.

Generally, it is OK to call hv_ringbuffer_cleanup() twice for the same channel (which is being handed over between in-kernel drivers and UIO) even if we didn't call hv_ringbuffer_init() in between. We, however, need to avoid kfree() call for an already freed pointer.

Fixes adae1e931acd ("Drivers: hv: vmbus: Copy packets sent by Hyper-V out of the ring buffer")
---
 drivers/hv/ring_buffer.c | 1 +
 1 file changed, 1 insertion(+)
drivers/hv/ring_buffer.c +0 -0
index 2aee356840a2..314015d9e912 100644

drivers/hv/ring_buffer.c void hv_ringbuffer_cleanup(struct hv_ring_buffer_info *ring_info) · line 248

Quoted 4 lines
drivers/hv/ring_buffer.c +1 -0
@@ -245,6 +245,7 @@ void hv_ringbuffer_cleanup(struct hv_ring_buffer_info *ring_info)
 	mutex_unlock(&ring_info->ring_buffer_mutex);
 
 	kfree(ring_info->pkt_buffer);
+	ring_info->pkt_buffer = NULL;

This certainly won't hurt.

I would however like to wait till Andrea and Michael go over the reasoning of this patch before doing anything.

Wei.

Quoted 6 lines
drivers/hv/ring_buffer.c +0 -0
 	ring_info->pkt_buffer_size = 0;
 }
 
Signature · 1 line
2.31.1
#2 lore raw
Quoted 51 lines · Wei Liu
Quoted 50 lines · Vitaly Kuznetsov

The following crash happens when a never-used device is unbound from uio_hv_generic driver:

kernel BUG at mm/slub.c:321!
invalid opcode: 0000 [#1] SMP PTI

 CPU: 0 PID: 4001 Comm: bash Kdump: loaded Tainted: G               X --------- ---  5.14.0-0.rc2.23.el9.x86_64 #1
 Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008  12/07/2018
 RIP: 0010:__slab_free+0x1d5/0x3d0
...
 Call Trace:
  ? pick_next_task_fair+0x18e/0x3b0
  ? __cond_resched+0x16/0x40
  ? vunmap_pmd_range.isra.0+0x154/0x1c0
  ? __vunmap+0x22d/0x290
  ? hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
  kfree+0x331/0x380
  ? hv_uio_remove+0x43/0x60 [uio_hv_generic]
  hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
  vmbus_free_ring+0x21/0x60 [hv_vmbus]
  hv_uio_remove+0x4f/0x60 [uio_hv_generic]
  vmbus_remove+0x23/0x30 [hv_vmbus]
  __device_release_driver+0x17a/0x230
  device_driver_detach+0x3c/0xa0
  unbind_store+0x113/0x130

...

The problem appears to be that we free 'ring_info->pkt_buffer' twice:
first, when the device is unbound from in-kernel driver (netvsc in this case) and second from hv_uio_remove(). Normally, ring buffer is supposed to be re-initialized from hv_uio_open() but this happens when UIO device is being opened and this is not guaranteed to happen.

Generally, it is OK to call hv_ringbuffer_cleanup() twice for the same channel (which is being handed over between in-kernel drivers and UIO) even if we didn't call hv_ringbuffer_init() in between. We, however, need to avoid kfree() call for an already freed pointer.

Fixes adae1e931acd ("Drivers: hv: vmbus: Copy packets sent by Hyper-V out of the ring buffer")
---
 drivers/hv/ring_buffer.c | 1 +
 1 file changed, 1 insertion(+)
drivers/hv/ring_buffer.c +0 -0
index 2aee356840a2..314015d9e912 100644
@@ -245,6 +245,7 @@ void hv_ringbuffer_cleanup(struct hv_ring_buffer_info *ring_info)
 	mutex_unlock(&ring_info->ring_buffer_mutex);
 
Quoted 8 lines

drivers/hv/ring_buffer.c void hv_ringbuffer_cleanup(struct hv_ring_buffer_info *ring_info) · line 248

Quoted 2 lines
drivers/hv/ring_buffer.c +1 -0
 	kfree(ring_info->pkt_buffer);
+	ring_info->pkt_buffer = NULL;

This certainly won't hurt.

I would however like to wait till Andrea and Michael go over the reasoning of this patch before doing anything.

Thanks,

the counter-intuitive thing here is that the channel sometimes continues to live after hv_ringbuffer_cleanup(): when we unbind a device from in-kernel driver and give it to uio_hv_generic the channel is handed over from one driver to another.

Quoted 9 lines

Wei.

Quoted 6 lines
drivers/hv/ring_buffer.c +0 -0
 	ring_info->pkt_buffer_size = 0;
 }
 
Signature · 1 line
2.31.1
Signature · 1 line
Vitaly
Reviewed-by on v1 1/1
#3 lore raw
In reply to Vitaly Kuznetsov
Quoted 32 lines · Vitaly Kuznetsov

The following crash happens when a never-used device is unbound from uio_hv_generic driver:

kernel BUG at mm/slub.c:321!
invalid opcode: 0000 [#1] SMP PTI

 CPU: 0 PID: 4001 Comm: bash Kdump: loaded Tainted: G               X --------- ---  5.14.0-0.rc2.23.el9.x86_64 #1
 Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008  12/07/2018
 RIP: 0010:__slab_free+0x1d5/0x3d0
...
 Call Trace:
  ? pick_next_task_fair+0x18e/0x3b0
  ? __cond_resched+0x16/0x40
  ? vunmap_pmd_range.isra.0+0x154/0x1c0
  ? __vunmap+0x22d/0x290
  ? hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
  kfree+0x331/0x380
  ? hv_uio_remove+0x43/0x60 [uio_hv_generic]
  hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
  vmbus_free_ring+0x21/0x60 [hv_vmbus]
  hv_uio_remove+0x4f/0x60 [uio_hv_generic]
  vmbus_remove+0x23/0x30 [hv_vmbus]
  __device_release_driver+0x17a/0x230
  device_driver_detach+0x3c/0xa0
  unbind_store+0x113/0x130

...

The problem appears to be that we free 'ring_info->pkt_buffer' twice:
first, when the device is unbound from in-kernel driver (netvsc in this case) and second from hv_uio_remove(). Normally, ring buffer is supposed to be re-initialized from hv_uio_open() but this happens when UIO device is being opened and this is not guaranteed to happen.

Quoted 7 lines

Generally, it is OK to call hv_ringbuffer_cleanup() twice for the same channel (which is being handed over between in-kernel drivers and UIO) even if we didn't call hv_ringbuffer_init() in between. We, however, need to avoid kfree() call for an already freed pointer.

Fixes adae1e931acd ("Drivers: hv: vmbus: Copy packets sent by Hyper-V out of the ring buffer")

LGTM.

ae6935ed7d424f appears to have anticipated this problem on ->ring_buffer and adopted the solution proposed here by Vitaly.

Andrea

Quoted 19 lines
---
 drivers/hv/ring_buffer.c | 1 +
 1 file changed, 1 insertion(+)
drivers/hv/ring_buffer.c +1 -0
index 2aee356840a2..314015d9e912 100644
@@ -245,6 +245,7 @@ void hv_ringbuffer_cleanup(struct hv_ring_buffer_info *ring_info)
 	mutex_unlock(&ring_info->ring_buffer_mutex);
 
 	kfree(ring_info->pkt_buffer);
+	ring_info->pkt_buffer = NULL;
 	ring_info->pkt_buffer_size = 0;
 }
 
Signature · 1 line
2.31.1
Reviewed-by on v1 1/1
#4 lore raw
In reply to Vitaly Kuznetsov

From: Vitaly Kuznetsov <[email protected]> Sent: Tuesday, August 31, 2021 7:39 AM

Quoted 57 lines

The following crash happens when a never-used device is unbound from uio_hv_generic driver:

kernel BUG at mm/slub.c:321!
invalid opcode: 0000 [#1] SMP PTI

 CPU: 0 PID: 4001 Comm: bash Kdump: loaded Tainted: G               X --------- ---  5.14.0-0.rc2.23.el9.x86_64 #1
 Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008  12/07/2018
 RIP: 0010:__slab_free+0x1d5/0x3d0
...
 Call Trace:
  ? pick_next_task_fair+0x18e/0x3b0
  ? __cond_resched+0x16/0x40
  ? vunmap_pmd_range.isra.0+0x154/0x1c0
  ? __vunmap+0x22d/0x290
  ? hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
  kfree+0x331/0x380
  ? hv_uio_remove+0x43/0x60 [uio_hv_generic]
  hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
  vmbus_free_ring+0x21/0x60 [hv_vmbus]
  hv_uio_remove+0x4f/0x60 [uio_hv_generic]
  vmbus_remove+0x23/0x30 [hv_vmbus]
  __device_release_driver+0x17a/0x230
  device_driver_detach+0x3c/0xa0
  unbind_store+0x113/0x130

...

The problem appears to be that we free 'ring_info->pkt_buffer' twice:
first, when the device is unbound from in-kernel driver (netvsc in this case) and second from hv_uio_remove(). Normally, ring buffer is supposed to be re-initialized from hv_uio_open() but this happens when UIO device is being opened and this is not guaranteed to happen.

Generally, it is OK to call hv_ringbuffer_cleanup() twice for the same channel (which is being handed over between in-kernel drivers and UIO) even if we didn't call hv_ringbuffer_init() in between. We, however, need to avoid kfree() call for an already freed pointer.

Fixes adae1e931acd ("Drivers: hv: vmbus: Copy packets sent by Hyper-V out of the ring buffer")
---
 drivers/hv/ring_buffer.c | 1 +
 1 file changed, 1 insertion(+)
drivers/hv/ring_buffer.c +1 -1
index 2aee356840a2..314015d9e912 100644
@@ -245,6 +245,7 @@ void hv_ringbuffer_cleanup(struct hv_ring_buffer_info *ring_info)
 	mutex_unlock(&ring_info->ring_buffer_mutex);
 
 	kfree(ring_info->pkt_buffer);
+	ring_info->pkt_buffer = NULL;
 	ring_info->pkt_buffer_size = 0;
 }
 
--
Quoted 1 line

2.31.1

maintainer Applied
#5 lore raw
In reply to Vitaly Kuznetsov
Quoted 3 lines · Vitaly Kuznetsov

The following crash happens when a never-used device is unbound from uio_hv_generic driver:

[...]

Quoted 2 lines
Fixes adae1e931acd ("Drivers: hv: vmbus: Copy packets sent by Hyper-V out of the ring buffer")

Applied to hyperv-fixes. Thanks.