Drivers: hv: vmbus: Fix kernel crash upon unbinding a device from uio_hv_generic driver
Subsystems (2)
PATCH 1/1 Drivers: hv: vmbus: Fix kernel crash upon unbinding a device from uio_hv_generic driver
The following crash happens when a never-used device is unbound from uio_hv_generic driver:
kernel BUG at mm/slub.c:321!
invalid opcode: 0000 [#1] SMP PTI
CPU: 0 PID: 4001 Comm: bash Kdump: loaded Tainted: G X --------- --- 5.14.0-0.rc2.23.el9.x86_64 #1 Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008 12/07/2018 RIP: 0010:__slab_free+0x1d5/0x3d0 ... Call Trace: ? pick_next_task_fair+0x18e/0x3b0 ? __cond_resched+0x16/0x40 ? vunmap_pmd_range.isra.0+0x154/0x1c0 ? __vunmap+0x22d/0x290 ? hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus] kfree+0x331/0x380 ? hv_uio_remove+0x43/0x60 [uio_hv_generic] hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus] vmbus_free_ring+0x21/0x60 [hv_vmbus] hv_uio_remove+0x4f/0x60 [uio_hv_generic] vmbus_remove+0x23/0x30 [hv_vmbus] __device_release_driver+0x17a/0x230 device_driver_detach+0x3c/0xa0 unbind_store+0x113/0x130
...
The problem appears to be that we free 'ring_info->pkt_buffer' twice:
first, when the device is unbound from in-kernel driver (netvsc in this case) and second from hv_uio_remove(). Normally, ring buffer is supposed to be re-initialized from hv_uio_open() but this happens when UIO device is being opened and this is not guaranteed to happen.
Generally, it is OK to call hv_ringbuffer_cleanup() twice for the same channel (which is being handed over between in-kernel drivers and UIO) even if we didn't call hv_ringbuffer_init() in between. We, however, need to avoid kfree() call for an already freed pointer.
--- drivers/hv/ring_buffer.c | 1 + 1 file changed, 1 insertion(+)
drivers/hv/ring_buffer.c +1 -0
@@ -245,6 +245,7 @@ void hv_ringbuffer_cleanup(struct hv_ring_buffer_info *ring_info)mutex_unlock (&ring_info->ring_buffer_mutex);kfree (ring_info->pkt_buffer); + ring_info->pkt_buffer =NULL ; ring_info->pkt_buffer_size =0 ; }
Signature · 1 line
2.31.1
Wei Liu
maintainerQuoted 48 lines · Vitaly Kuznetsov
The following crash happens when a never-used device is unbound from
uio_hv_genericdriver:kernel BUG at
mm/slub.c:321!
invalid opcode: 0000 [#1] SMP PTICPU: 0 PID: 4001 Comm: bash Kdump: loaded Tainted: G X --------- --- 5.14.0-0.rc2.23.el9.x86_64 #1 Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008 12/07/2018 RIP: 0010:__slab_free+0x1d5/0x3d0 ... Call Trace: ? pick_next_task_fair+0x18e/0x3b0 ? __cond_resched+0x16/0x40 ? vunmap_pmd_range.isra.0+0x154/0x1c0 ? __vunmap+0x22d/0x290 ? hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus] kfree+0x331/0x380 ? hv_uio_remove+0x43/0x60 [uio_hv_generic] hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus] vmbus_free_ring+0x21/0x60 [hv_vmbus] hv_uio_remove+0x4f/0x60 [uio_hv_generic] vmbus_remove+0x23/0x30 [hv_vmbus] __device_release_driver+0x17a/0x230 device_driver_detach+0x3c/0xa0 unbind_store+0x113/0x130...
The problem appears to be that we free '
ring_info->pkt_buffer' twice:
first, when the device is unbound from in-kernel driver (netvsc in this case) and second fromhv_uio_remove(). Normally, ring buffer is supposed to be re-initialized fromhv_uio_open()but this happens when UIO device is being opened and this is not guaranteed to happen.Generally, it is OK to call
hv_ringbuffer_cleanup()twice for the same channel (which is being handed over between in-kernel drivers and UIO) even if we didn't callhv_ringbuffer_init()in between. We, however, need to avoidkfree()call for an already freed pointer.Fixes adae1e931acd ("Drivers: hv: vmbus: Copy packets sent by Hyper-V out of the ring buffer")Signed-off-by Vitaly Kuznetsov <[email protected]>--- drivers/hv/ring_buffer.c | 1 + 1 file changed, 1 insertion(+)drivers/hv/ring_buffer.c +0
-0
drivers/hv/ring_buffer.c void hv_ringbuffer_cleanup(struct hv_ring_buffer_info *ring_info) · line 248
Quoted 4 lines
drivers/hv/ring_buffer.c +1
-0@@ -245,6 +245,7 @@ void hv_ringbuffer_cleanup(struct hv_ring_buffer_info *ring_info)mutex_unlock (&ring_info->ring_buffer_mutex);kfree (ring_info->pkt_buffer); + ring_info->pkt_buffer =NULL ;
This certainly won't hurt.
I would however like to wait till Andrea and Michael go over the reasoning of this patch before doing anything.
Wei.
Quoted 6 lines
drivers/hv/ring_buffer.c +0
-0ring_info->pkt_buffer_size =0 ; }Signature · 1 line
2.31.1
Quoted 51 lines · Wei Liu
Quoted 50 lines · Vitaly Kuznetsov
The following crash happens when a never-used device is unbound from
uio_hv_genericdriver:kernel BUG at
mm/slub.c:321!
invalid opcode: 0000 [#1] SMP PTICPU: 0 PID: 4001 Comm: bash Kdump: loaded Tainted: G X --------- --- 5.14.0-0.rc2.23.el9.x86_64 #1 Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008 12/07/2018 RIP: 0010:__slab_free+0x1d5/0x3d0 ... Call Trace: ? pick_next_task_fair+0x18e/0x3b0 ? __cond_resched+0x16/0x40 ? vunmap_pmd_range.isra.0+0x154/0x1c0 ? __vunmap+0x22d/0x290 ? hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus] kfree+0x331/0x380 ? hv_uio_remove+0x43/0x60 [uio_hv_generic] hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus] vmbus_free_ring+0x21/0x60 [hv_vmbus] hv_uio_remove+0x4f/0x60 [uio_hv_generic] vmbus_remove+0x23/0x30 [hv_vmbus] __device_release_driver+0x17a/0x230 device_driver_detach+0x3c/0xa0 unbind_store+0x113/0x130...
The problem appears to be that we free '
ring_info->pkt_buffer' twice:
first, when the device is unbound from in-kernel driver (netvsc in this case) and second fromhv_uio_remove(). Normally, ring buffer is supposed to be re-initialized fromhv_uio_open()but this happens when UIO device is being opened and this is not guaranteed to happen.Generally, it is OK to call
hv_ringbuffer_cleanup()twice for the same channel (which is being handed over between in-kernel drivers and UIO) even if we didn't callhv_ringbuffer_init()in between. We, however, need to avoidkfree()call for an already freed pointer.Fixes adae1e931acd ("Drivers: hv: vmbus: Copy packets sent by Hyper-V out of the ring buffer")Signed-off-by Vitaly Kuznetsov <[email protected]>--- drivers/hv/ring_buffer.c | 1 + 1 file changed, 1 insertion(+)drivers/hv/ring_buffer.c +0
-0@@ -245,6 +245,7 @@ void hv_ringbuffer_cleanup(struct hv_ring_buffer_info *ring_info)mutex_unlock (&ring_info->ring_buffer_mutex);
Quoted 8 lines
drivers/hv/ring_buffer.c void hv_ringbuffer_cleanup(struct hv_ring_buffer_info *ring_info) · line 248
Quoted 2 lines
drivers/hv/ring_buffer.c +1
-0kfree (ring_info->pkt_buffer); + ring_info->pkt_buffer =NULL ;This certainly won't hurt.
I would however like to wait till Andrea and Michael go over the reasoning of this patch before doing anything.
Thanks,
the counter-intuitive thing here is that the channel sometimes continues to live after hv_ringbuffer_cleanup(): when we unbind a device from in-kernel driver and give it to uio_hv_generic the channel is handed over from one driver to another.
Quoted 9 lines
Wei.
Quoted 6 lines
drivers/hv/ring_buffer.c +0
-0ring_info->pkt_buffer_size =0 ; }Signature · 1 line
2.31.1
Signature · 1 line
Vitaly
Quoted 32 lines · Vitaly Kuznetsov
The following crash happens when a never-used device is unbound from
uio_hv_genericdriver:kernel BUG at
mm/slub.c:321!
invalid opcode: 0000 [#1] SMP PTICPU: 0 PID: 4001 Comm: bash Kdump: loaded Tainted: G X --------- --- 5.14.0-0.rc2.23.el9.x86_64 #1 Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008 12/07/2018 RIP: 0010:__slab_free+0x1d5/0x3d0 ... Call Trace: ? pick_next_task_fair+0x18e/0x3b0 ? __cond_resched+0x16/0x40 ? vunmap_pmd_range.isra.0+0x154/0x1c0 ? __vunmap+0x22d/0x290 ? hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus] kfree+0x331/0x380 ? hv_uio_remove+0x43/0x60 [uio_hv_generic] hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus] vmbus_free_ring+0x21/0x60 [hv_vmbus] hv_uio_remove+0x4f/0x60 [uio_hv_generic] vmbus_remove+0x23/0x30 [hv_vmbus] __device_release_driver+0x17a/0x230 device_driver_detach+0x3c/0xa0 unbind_store+0x113/0x130...
The problem appears to be that we free '
ring_info->pkt_buffer' twice:
first, when the device is unbound from in-kernel driver (netvsc in this case) and second fromhv_uio_remove(). Normally, ring buffer is supposed to be re-initialized fromhv_uio_open()but this happens when UIO device is being opened and this is not guaranteed to happen.
Quoted 7 lines
Generally, it is OK to call
hv_ringbuffer_cleanup()twice for the same channel (which is being handed over between in-kernel drivers and UIO) even if we didn't callhv_ringbuffer_init()in between. We, however, need to avoidkfree()call for an already freed pointer.Fixes adae1e931acd ("Drivers: hv: vmbus: Copy packets sent by Hyper-V out of the ring buffer")Signed-off-by Vitaly Kuznetsov <[email protected]>
LGTM.
ae6935ed7d424f appears to have anticipated this problem on ->ring_buffer and adopted the solution proposed here by Vitaly.
Andrea
Quoted 19 lines
--- drivers/hv/ring_buffer.c | 1 + 1 file changed, 1 insertion(+)drivers/hv/ring_buffer.c +1
-0@@ -245,6 +245,7 @@ void hv_ringbuffer_cleanup(struct hv_ring_buffer_info *ring_info)mutex_unlock (&ring_info->ring_buffer_mutex);kfree (ring_info->pkt_buffer); + ring_info->pkt_buffer =NULL ; ring_info->pkt_buffer_size =0 ; }Signature · 1 line
2.31.1
From: Vitaly Kuznetsov <[email protected]> Sent: Tuesday, August 31, 2021 7:39 AM
Quoted 57 lines
The following crash happens when a never-used device is unbound from
uio_hv_genericdriver:kernel BUG at
mm/slub.c:321!
invalid opcode: 0000 [#1] SMP PTICPU: 0 PID: 4001 Comm: bash Kdump: loaded Tainted: G X --------- --- 5.14.0-0.rc2.23.el9.x86_64 #1 Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008 12/07/2018 RIP: 0010:__slab_free+0x1d5/0x3d0 ... Call Trace: ? pick_next_task_fair+0x18e/0x3b0 ? __cond_resched+0x16/0x40 ? vunmap_pmd_range.isra.0+0x154/0x1c0 ? __vunmap+0x22d/0x290 ? hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus] kfree+0x331/0x380 ? hv_uio_remove+0x43/0x60 [uio_hv_generic] hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus] vmbus_free_ring+0x21/0x60 [hv_vmbus] hv_uio_remove+0x4f/0x60 [uio_hv_generic] vmbus_remove+0x23/0x30 [hv_vmbus] __device_release_driver+0x17a/0x230 device_driver_detach+0x3c/0xa0 unbind_store+0x113/0x130...
The problem appears to be that we free '
ring_info->pkt_buffer' twice:
first, when the device is unbound from in-kernel driver (netvsc in this case) and second fromhv_uio_remove(). Normally, ring buffer is supposed to be re-initialized fromhv_uio_open()but this happens when UIO device is being opened and this is not guaranteed to happen.Generally, it is OK to call
hv_ringbuffer_cleanup()twice for the same channel (which is being handed over between in-kernel drivers and UIO) even if we didn't callhv_ringbuffer_init()in between. We, however, need to avoidkfree()call for an already freed pointer.Fixes adae1e931acd ("Drivers: hv: vmbus: Copy packets sent by Hyper-V out of the ring buffer")Signed-off-by Vitaly Kuznetsov <[email protected]>--- drivers/hv/ring_buffer.c | 1 + 1 file changed, 1 insertion(+)drivers/hv/ring_buffer.c +1
-1@@ -245,6 +245,7 @@ void hv_ringbuffer_cleanup(struct hv_ring_buffer_info *ring_info)mutex_unlock (&ring_info->ring_buffer_mutex);kfree (ring_info->pkt_buffer); + ring_info->pkt_buffer =NULL ; ring_info->pkt_buffer_size =0 ; } --
Quoted 1 line
2.31.1
Quoted 3 lines · Vitaly Kuznetsov
The following crash happens when a never-used device is unbound from
uio_hv_genericdriver:
[...]
Quoted 2 lines
Fixes adae1e931acd ("Drivers: hv: vmbus: Copy packets sent by Hyper-V out of the ring buffer")Signed-off-by Vitaly Kuznetsov <[email protected]>
Applied to hyperv-fixes. Thanks.