nbd: fix use-after-freed and double lock bugs
Subsystems (2)
v3 cover nbd: fix use-after-freed and double lock bugs
From: Xiubo Li <[email protected]>
Changed in V3:
- remove the likely()
Changed in V2:
- fixed possible double lock issue in recv_work().
Xiubo Li (2):
nbd: fix use-after-freed crash for nbd->recv_workq
nbd: add comments about double lock for config_lock confusion
drivers/block/nbd.c | 36 +++++++++++++++++++++++++++++++----- 1 file changed, 31 insertions(+), 5 deletions(-)
Signature · 1 line
2.18.4
PATCH v3 1/2 nbd: fix use-after-freed crash for nbd->recv_workq
From: Xiubo Li <[email protected]>
The crash call trace:
<6>[ 1012.319386] block nbd1: NBD_DISCONNECT <1>[ 1012.319437] BUG: kernel NULL pointer dereference, address: 0000000000000020 <1>[ 1012.319439] #PF: supervisor write access in kernel mode <1>[ 1012.319441] #PF: error_code(0x0002) - not-present page <6>[ 1012.319442] PGD 0 P4D 0 <4>[ 1012.319448] Oops: 0002 [#1] SMP NOPTI <4>[ 1012.319454] CPU: 9 PID: 25111 Comm: rbd-nbd Tainted: G E 5.9.0+ #6 [...] <4>[ 1012.319505] PKRU: 55555554 <4>[ 1012.319506] Call Trace: <4>[ 1012.319560] flush_workqueue+0x81/0x440 <4>[ 1012.319598] nbd_disconnect_and_put+0x50/0x70 [nbd] <4>[ 1012.319607] nbd_genl_disconnect+0xc7/0x170 [nbd] <4>[ 1012.319627] genl_rcv_msg+0x1dd/0x2f9 <4>[ 1012.319642] ? genl_start+0x140/0x140 <4>[ 1012.319644] netlink_rcv_skb+0x49/0x110 <4>[ 1012.319649] genl_rcv+0x24/0x40 <4>[ 1012.319651] netlink_unicast+0x1a5/0x280 <4>[ 1012.319653] netlink_sendmsg+0x23d/0x470 <4>[ 1012.319667] sock_sendmsg+0x5b/0x60 <4>[ 1012.319676] ____sys_sendmsg+0x1ef/0x260 <4>[ 1012.319679] ? copy_msghdr_from_user+0x5c/0x90 <4>[ 1012.319680] ? ____sys_recvmsg+0xa5/0x180 <4>[ 1012.319682] ___sys_sendmsg+0x7c/0xc0 <4>[ 1012.319683] ? copy_msghdr_from_user+0x5c/0x90 <4>[ 1012.319685] ? ___sys_recvmsg+0x89/0xc0 <4>[ 1012.319692] ? __wake_up_common_lock+0x87/0xc0 <4>[ 1012.319715] ? __check_object_size+0x46/0x173 <4>[ 1012.319727] ? _copy_to_user+0x22/0x30 <4>[ 1012.319729] ? move_addr_to_user+0xc3/0x100 <4>[ 1012.319731] __sys_sendmsg+0x57/0xa0 <4>[ 1012.319744] do_syscall_64+0x33/0x40 <4>[ 1012.319760] entry_SYSCALL_64_after_hwframe+0x44/0xa9 <4>[ 1012.319780] RIP: 0033:0x7f5baa8e3ad5
In case the reference of nbd->config has reached zero and the related resource has been released, if another process tries to send the DISCONENCT cmd by using the netlink, it will potentially crash like this.
--- drivers/block/nbd.c | 25 ++++++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-)
drivers/block/nbd.c +20 -5
@@ -2003,16 +2003,31 @@ static int nbd_genl_connect(struct sk_buff *skb, struct genl_info *info)static void nbd_disconnect_and_put (struct nbd_device *nbd) { +bool flush = true; +mutex_lock (&nbd->config_lock);nbd_disconnect (nbd);nbd_clear_sock (nbd); -mutex_unlock (&nbd->config_lock);/* -* Make sure recv thread has finished, so it does not drop the last -* config ref and try to destroy the workqueue from inside the work -* queue. +* In very rare case that the nbd->recv_workq may already have been +* released by the recv_work(). */ -flush_workqueue (nbd->recv_workq); +if (nbd->recv_workq) +refcount_inc (&nbd->config_refs); +else + flush = false; +mutex_unlock (&nbd->config_lock); + +if (flush) { +/* +* Make sure recv thread has finished, so it does not drop +* the last config ref and try to destroy the workqueue +* from inside the work queue. +*/ +flush_workqueue (nbd->recv_workq); +nbd_config_put (nbd); + } +if (test_and_clear_bit (NBD_RT_HAS_CONFIG_REF , &nbd->config->runtime_flags))nbd_config_put (nbd);
Signature · 1 line
2.18.4
Re: [PATCH v3 1/2] nbd: fix use-after-freed crash for nbd->recv_workq
Quoted 39 lines · On 11/3/20 1:51 AM, [email protected] wrote:
From: Xiubo Li <[email protected]>
The crash call trace:
<6>[ 1012.319386] block nbd1: NBD_DISCONNECT <1>[ 1012.319437] BUG: kernel NULL pointer dereference, address: 0000000000000020 <1>[ 1012.319439] #PF: supervisor write access in kernel mode <1>[ 1012.319441] #PF: error_code(0x0002) - not-present page <6>[ 1012.319442] PGD 0 P4D 0 <4>[ 1012.319448] Oops: 0002 [#1] SMP NOPTI <4>[ 1012.319454] CPU: 9 PID: 25111 Comm: rbd-nbd Tainted: G E 5.9.0+ #6 [...] <4>[ 1012.319505] PKRU: 55555554 <4>[ 1012.319506] Call Trace: <4>[ 1012.319560] flush_workqueue+0x81/0x440 <4>[ 1012.319598] nbd_disconnect_and_put+0x50/0x70 [nbd] <4>[ 1012.319607] nbd_genl_disconnect+0xc7/0x170 [nbd] <4>[ 1012.319627] genl_rcv_msg+0x1dd/0x2f9 <4>[ 1012.319642] ? genl_start+0x140/0x140 <4>[ 1012.319644] netlink_rcv_skb+0x49/0x110 <4>[ 1012.319649] genl_rcv+0x24/0x40 <4>[ 1012.319651] netlink_unicast+0x1a5/0x280 <4>[ 1012.319653] netlink_sendmsg+0x23d/0x470 <4>[ 1012.319667] sock_sendmsg+0x5b/0x60 <4>[ 1012.319676] ____sys_sendmsg+0x1ef/0x260 <4>[ 1012.319679] ? copy_msghdr_from_user+0x5c/0x90 <4>[ 1012.319680] ? ____sys_recvmsg+0xa5/0x180 <4>[ 1012.319682] ___sys_sendmsg+0x7c/0xc0 <4>[ 1012.319683] ? copy_msghdr_from_user+0x5c/0x90 <4>[ 1012.319685] ? ___sys_recvmsg+0x89/0xc0 <4>[ 1012.319692] ? __wake_up_common_lock+0x87/0xc0 <4>[ 1012.319715] ? __check_object_size+0x46/0x173 <4>[ 1012.319727] ? _copy_to_user+0x22/0x30 <4>[ 1012.319729] ? move_addr_to_user+0xc3/0x100 <4>[ 1012.319731] __sys_sendmsg+0x57/0xa0 <4>[ 1012.319744] do_syscall_64+0x33/0x40 <4>[ 1012.319760] entry_SYSCALL_64_after_hwframe+0x44/0xa9 <4>[ 1012.319780] RIP: 0033:0x7f5baa8e3ad5
Quoted 7 lines
In case the reference of
nbd->confighas reached zero and the related resource has been released, if another process tries to send the DISCONENCT cmd by using the netlink, it will potentially crash like this.Reviewed-by Ming Lei <[email protected]>Signed-off-by Xiubo Li <[email protected]>
Thanks,
Josef
PATCH v3 2/2 nbd: add comments about double lock for config_lock confusion
In reply to [email protected]From: Xiubo Li <[email protected]>
When calling the ioctl(), fget() will be called on this fd, and nbd_release() is only called when the fd's refcount drops to zero. With this we can make sure that the nbd_release() won't be called before the ioctl() finished.
So there won't have the double lock issue for the "config_lock", which has already been held by nbd_ioctl().
--- drivers/block/nbd.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
drivers/block/nbd.c +11 -0
@@ -1345,6 +1345,17 @@ static void nbd_clear_sock_ioctl(struct nbd_device *nbd,sock_shutdown (nbd);__invalidate_device (bdev, true);nbd_bdev_reset (bdev); + +/* +* When calling the ioctl(), fget() will be called on this +* fd, and nbd_release() is only called when the fd's refcount +* drops to zero. With this we can make sure that the +* nbd_release() won't be called before the ioctl() finished. +* +* So there won't have the double lock issue if it will +* call the nbd_config_put() here for the "config_lock", which +* has already been held by nbd_ioctl(). +*/ if (test_and_clear_bit (NBD_RT_HAS_CONFIG_REF , &nbd->config->runtime_flags))nbd_config_put (nbd);
Signature · 1 line
2.18.4
Re: [PATCH v3 2/2] nbd: add comments about double lock for config_lock confusion
Quoted 7 lines · On 11/3/20 1:51 AM, [email protected] wrote:
From: Xiubo Li <[email protected]>
When calling the
ioctl(),fget()will be called on this fd, andnbd_release()is only called when the fd's refcount drops to zero. With this we can make sure that thenbd_release()won't be called before theioctl()finished.
Quoted 5 lines
So there won't have the double lock issue for the "
config_lock", which has already been held bynbd_ioctl().Reviewed-by Ming Lei <[email protected]>Signed-off-by Xiubo Li <[email protected]>
Thanks,
Josef