virtio_vsock: Fix race condition in virtio_transport_recv_pkt
Subsystems (7)
PATCH v2 1/1 virtio_vsock: Fix race condition in virtio_transport_recv_pkt
When client tries to connect(SOCK_STREAM) the server in the guest with NONBLOCK mode, there will be a panic on a ThunderX2 (armv8a server):
[ 463.718844][ T5040] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000 [ 463.718848][ T5040] Mem abort info: [ 463.718849][ T5040] ESR = 0x96000044 [ 463.718852][ T5040] EC = 0x25: DABT (current EL), IL = 32 bits [ 463.718853][ T5040] SET = 0, FnV = 0 [ 463.718854][ T5040] EA = 0, S1PTW = 0 [ 463.718855][ T5040] Data abort info: [ 463.718856][ T5040] ISV = 0, ISS = 0x00000044 [ 463.718857][ T5040] CM = 0, WnR = 1 [ 463.718859][ T5040] user pgtable: 4k pages, 48-bit VAs, pgdp=0000008f6f6e9000 [ 463.718861][ T5040] [0000000000000000] pgd=0000000000000000 [ 463.718866][ T5040] Internal error: Oops: 96000044 [#1] SMP [...] [ 463.718977][ T5040] CPU: 213 PID: 5040 Comm: vhost-5032 Tainted: G O 5.7.0-rc7+ #139 [ 463.718980][ T5040] Hardware name: GIGABYTE R281-T91-00/MT91-FS1-00, BIOS F06 09/25/2018 [ 463.718982][ T5040] pstate: 60400009 (nZCv daif +PAN -UAO) [ 463.718995][ T5040] pc : virtio_transport_recv_pkt+0x4c8/0xd40 [vmw_vsock_virtio_transport_common] [ 463.718999][ T5040] lr : virtio_transport_recv_pkt+0x1fc/0xd40 [vmw_vsock_virtio_transport_common] [ 463.719000][ T5040] sp : ffff80002dbe3c40 [...] [ 463.719025][ T5040] Call trace: [ 463.719030][ T5040] virtio_transport_recv_pkt+0x4c8/0xd40 [vmw_vsock_virtio_transport_common] [ 463.719034][ T5040] vhost_vsock_handle_tx_kick+0x360/0x408 [vhost_vsock] [ 463.719041][ T5040] vhost_worker+0x100/0x1a0 [vhost] [ 463.719048][ T5040] kthread+0x128/0x130 [ 463.719052][ T5040] ret_from_fork+0x10/0x18
The race condition as follows:
Task1 Task2
===== =====
__sock_release virtio_transport_recv_pkt
__vsock_release vsock_find_bound_socket (found)
lock_sock_nested
vsock_remove_sock
sock_orphan
sk_set_socket(sk, NULL)
...
release_sock
lock_sock
virtio_transport_recv_connecting
sk->sk_socket->state (panic)The root cause is that vsock_find_bound_socket can't hold the lock_sock, so there is a small race window between vsock_find_bound_socket() and lock_sock(). If there is __vsock_release() in another task, sk->sk_socket will be set to NULL inadvertently.
This fixes it by checking sk->sk_shutdown.
---
v2: use lightweight checking suggested by Stefano Garzarella
net/vmw_vsock/virtio_transport_common.c | 8 ++++++++ 1 file changed, 8 insertions(+)
net/vmw_vsock/virtio_transport_common.c +8 -0
@@ -1132,6 +1132,14 @@ void virtio_transport_recv_pkt(struct virtio_transport *t,lock_sock (sk); +/* Check if sk has been released before lock_sock */ +if (sk->sk_shutdown ==SHUTDOWN_MASK ) { + (void )virtio_transport_reset_no_sock (t, pkt); +release_sock (sk); +sock_put (sk); + goto free_pkt; + } +/* Update CID in case it has changed after a transport reset event */ vsk->local_addr.svm_cid = dst.svm_cid;
Signature · 1 line
2.17.1
Quoted 20 lines · Jia He
When client tries to
connect(SOCK_STREAM)the server in the guest with NONBLOCK mode, there will be a panic on a ThunderX2 (armv8a server):[ 463.718844][ T5040] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000 [ 463.718848][ T5040] Mem abort info: [ 463.718849][ T5040] ESR = 0x96000044 [ 463.718852][ T5040] EC = 0x25: DABT (current EL), IL = 32 bits [ 463.718853][ T5040] SET = 0, FnV = 0 [ 463.718854][ T5040] EA = 0, S1PTW = 0 [ 463.718855][ T5040] Data abort info: [ 463.718856][ T5040] ISV = 0, ISS = 0x00000044 [ 463.718857][ T5040] CM = 0, WnR = 1 [ 463.718859][ T5040] user pgtable: 4k pages, 48-bit VAs, pgdp=0000008f6f6e9000 [ 463.718861][ T5040] [0000000000000000] pgd=0000000000000000 [ 463.718866][ T5040] Internal error: Oops: 96000044 [#1] SMP [...] [ 463.718977][ T5040] CPU: 213 PID: 5040 Comm: vhost-5032 Tainted: G O 5.7.0-rc7+ #139 [ 463.718980][ T5040] Hardware name: GIGABYTE R281-T91-00/MT91-FS1-00, BIOS F06 09/25/2018 [ 463.718982][ T5040] pstate: 60400009 (nZCv daif +PAN -UAO) [ 463.718995][ T5040] pc : virtio_transport_recv_pkt+0x4c8/0xd40 [vmw_vsock_virtio_transport_common] [ 463.718999][ T5040] lr : virtio_transport_recv_pkt+0x1fc/0xd40 [vmw_vsock_virtio_transport_common]
Quoted 8 lines
[ 463.719000][ T5040] sp : ffff80002dbe3c40 [...] [ 463.719025][ T5040] Call trace: [ 463.719030][ T5040] virtio_transport_recv_pkt+0x4c8/0xd40 [vmw_vsock_virtio_transport_common] [ 463.719034][ T5040] vhost_vsock_handle_tx_kick+0x360/0x408 [vhost_vsock] [ 463.719041][ T5040] vhost_worker+0x100/0x1a0 [vhost] [ 463.719048][ T5040] kthread+0x128/0x130 [ 463.719052][ T5040] ret_from_fork+0x10/0x18
^ ^
Maybe we can remove these two columns from the commit message.
Quoted 10 lines
The race condition as follows:
Task1 Task2 ===== ===== __sock_release virtio_transport_recv_pkt
__vsock_releasevsock_find_bound_socket(found)
lock_sock_nested
vsock_remove_sock
sock_orphan
sk_set_socket(sk, NULL)
Here we can add:
sk->sk_shutdown = SHUTDOWN_MASK;
Quoted 44 lines
... release_sock lock_sock virtio_transport_recv_connecting sk->sk_socket->state (panic)The root cause is that
vsock_find_bound_socketcan't hold thelock_sock, so there is a small race window betweenvsock_find_bound_socket()andlock_sock(). If there is__vsock_release()in another task,sk->sk_socketwill be set to NULL inadvertently.This fixes it by checking
sk->sk_shutdown.Signed-off-by Jia He <[email protected]>Cc Stefano Garzarella <[email protected]>---
v2: use lightweight checking suggested by Stefano Garzarellanet/vmw_vsock/virtio_transport_common.c | 8 ++++++++ 1 file changed, 8 insertions(+)net/vmw_vsock/virtio_transport_common.c +8
-0@@ -1132,6 +1132,14 @@ void virtio_transport_recv_pkt(struct virtio_transport *t,lock_sock (sk); +/* Check if sk has been released before lock_sock */ +if (sk->sk_shutdown ==SHUTDOWN_MASK ) { + (void )virtio_transport_reset_no_sock (t, pkt); +release_sock (sk); +sock_put (sk); + goto free_pkt; + } +/* Update CID in case it has changed after a transport reset event */ vsk->local_addr.svm_cid = dst.svm_cid;Signature · 1 line
2.17.1
Anyway, the patch LGTM, let see what David and other say.
Thanks,
Stefano
Hi Stefano
Quoted 63 lines
-----Original Message-----
From: Stefano Garzarella <[email protected]>
Sent: Saturday, May 30, 2020 12:34 AM
To: Justin He <[email protected]>
Cc: Stefan Hajnoczi <[email protected]>; David S. Miller <[email protected]>; Jakub Kicinski <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Kaly Xin <[email protected]>; [email protected]
Subject: Re: [PATCH v2]virtio_vsock: Fix race condition invirtio_transport_recv_pktQuoted 3 lines · Jia He
When client tries to
connect(SOCK_STREAM)the server in the guest with NONBLOCK mode, there will be a panic on a ThunderX2 (armv8a server):[ 463.718844][ T5040] Unable to handle kernel NULL pointer dereferenceat virtual address 0000000000000000
Quoted 9 lines
[ 463.718848][ T5040] Mem abort info: [ 463.718849][ T5040] ESR = 0x96000044 [ 463.718852][ T5040] EC = 0x25: DABT (current EL), IL = 32 bits [ 463.718853][ T5040] SET = 0, FnV = 0 [ 463.718854][ T5040] EA = 0, S1PTW = 0 [ 463.718855][ T5040] Data abort info: [ 463.718856][ T5040] ISV = 0, ISS = 0x00000044 [ 463.718857][ T5040] CM = 0, WnR = 1 [ 463.718859][ T5040] user pgtable: 4k pages, 48-bit VAs,pgdp=0000008f6f6e9000
Quoted 4 lines
[ 463.718861][ T5040] [0000000000000000] pgd=0000000000000000 [ 463.718866][ T5040] Internal error: Oops: 96000044 [#1] SMP [...] [ 463.718977][ T5040] CPU: 213 PID: 5040 Comm: vhost-5032 Tainted: GO 5.7.0-rc7+ #139
Quoted 1 line
[ 463.718980][ T5040] Hardware name: GIGABYTE R281-T91-00/MT91-FS1-00,BIOS F06 09/25/2018
Quoted 2 lines
[ 463.718982][ T5040] pstate: 60400009 (nZCv daif +PAN -UAO) [ 463.718995][ T5040] pc : virtio_transport_recv_pkt+0x4c8/0xd40[
vmw_vsock_virtio_transport_common]Quoted 1 line
[ 463.718999][ T5040] lr : virtio_transport_recv_pkt+0x1fc/0xd40[
vmw_vsock_virtio_transport_common]Quoted 4 lines
[ 463.719000][ T5040] sp : ffff80002dbe3c40 [...] [ 463.719025][ T5040] Call trace: [ 463.719030][ T5040] virtio_transport_recv_pkt+0x4c8/0xd40[
vmw_vsock_virtio_transport_common]Quoted 1 line
[ 463.719034][ T5040] vhost_vsock_handle_tx_kick+0x360/0x408[
vhost_vsock]Quoted 3 lines
[ 463.719041][ T5040] vhost_worker+0x100/0x1a0 [vhost] [ 463.719048][ T5040] kthread+0x128/0x130 [ 463.719052][ T5040] ret_from_fork+0x10/0x18^ ^
Maybe we can remove these two columns from the commit message.Quoted 10 lines
The race condition as follows:
Task1 Task2 ===== ===== __sock_release virtio_transport_recv_pkt
__vsock_releasevsock_find_bound_socket(found)
lock_sock_nested
vsock_remove_sock
sock_orphan
sk_set_socket(sk, NULL)
Quoted 2 lines
Here we can add:
sk->sk_shutdown = SHUTDOWN_MASK;
Indeed. This makes it more clearly
--
Cheers,
Justin (Jia He)
Quoted 50 lines
Quoted 17 lines
... release_sock lock_sock virtio_transport_recv_connecting sk->sk_socket->state (panic)The root cause is that
vsock_find_bound_socketcan't hold thelock_sock, so there is a small race window betweenvsock_find_bound_socket()andlock_sock(). If there is__vsock_release()in another task,sk->sk_socketwill be set to NULL inadvertently.This fixes it by checking
sk->sk_shutdown.Signed-off-by Jia He <[email protected]>Cc Stefano Garzarella <[email protected]>Review
Quoted 7 lines
---
v2: use lightweight checking suggested by Stefano Garzarellanet/vmw_vsock/virtio_transport_common.c | 8 ++++++++ 1 file changed, 8 insertions(+)(quoted diff) +0
-0
b/net/vmw_vsock/virtio_transport_common.cnet/vmw_vsock/virtio_transport_common.c void virtio_transport_recv_pkt(struct
Quoted 4 lines
index
69efc891885f..0edda1edf988100644net/vmw_vsock/virtio_transport_common.c +0
-0@@ -1132,6 +1132,14 @@ void virtio_transport_recv_pkt(struct
virtio_transport*t,Quoted 15 lines
net/vmw_vsock/virtio_transport_common.c +8
-1lock_sock (sk); +/* Check if sk has been released before lock_sock */ +if (sk->sk_shutdown ==SHUTDOWN_MASK ) { +(void )virtio_transport_reset_no_sock (t, pkt); +release_sock (sk); +sock_put (sk); +goto free_pkt; +} +/* Update CID in case it has changed after a transport reset event */ vsk->local_addr.svm_cid = dst.svm_cid; --Quoted 2 lines
2.17.1
Anyway, the patch LGTM, let see what David and other say.
Quoted 4 lines
Reviewed-by Stefano Garzarella <[email protected]>Thanks,
Stefano
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Sasha Levin
maintainerHi
[This is an automated email]
This commit has been processed because it contains a -stable tag. The stable tag indicates that it's relevant for the following trees: all
The bot has tested the following trees: v5.6.15, v5.4.43, v4.19.125, v4.14.182, v4.9.225, v4.4.225.
v5.6.15: Build OK!
v5.4.43: Build failed! Errors:
net/vmw_vsock/virtio_transport_common.c:1109:40: error: ‘t’ undeclared (first use in this function); did you mean ‘tm’? net/vmw_vsock/virtio_transport_common.c:1109:9: error: too many arguments to function ‘virtio_transport_reset_no_sock’
v4.19.125: Build failed! Errors:
net/vmw_vsock/virtio_transport_common.c:1042:40: error: ‘t’ undeclared (first use in this function); did you mean ‘tm’? net/vmw_vsock/virtio_transport_common.c:1042:9: error: too many arguments to function ‘virtio_transport_reset_no_sock’
v4.14.182: Build failed! Errors:
net/vmw_vsock/virtio_transport_common.c:1038:40: error: ‘t’ undeclared (first use in this function); did you mean ‘tm’? net/vmw_vsock/virtio_transport_common.c:1038:9: error: too many arguments to function ‘virtio_transport_reset_no_sock’
v4.9.225: Build failed! Errors:
net/vmw_vsock/virtio_transport_common.c:968:40: error: ‘t’ undeclared (first use in this function); did you mean ‘tm’? net/vmw_vsock/virtio_transport_common.c:968:9: error: too many arguments to function ‘virtio_transport_reset_no_sock’
v4.4.225: Failed to apply! Possible dependencies:
06a8fc78367d ("VSOCK: Introduce virtio_vsock_common.ko")NOTE: The patch will not be queued to stable trees until it is upstream.
How should we proceed with this patch?
Signature · 2 lines
Thanks Sasha
Stefano Garzarella
maintainerQuoted 26 lines · Sasha Levin
Hi
[This is an automated email]
This commit has been processed because it contains a -stable tag. The stable tag indicates that it's relevant for the following trees: all
The bot has tested the following trees: v5.6.15, v5.4.43, v4.19.125, v4.14.182, v4.9.225, v4.4.225.
v5.6.15: Build OK!
v5.4.43: Build failed! Errors:net/vmw_vsock/virtio_transport_common.c:1109:40: error: ???????????????t?????????????????? undeclared (first use in this function); did you mean ???????????????tm??????????????????? net/vmw_vsock/virtio_transport_common.c:1109:9: error: too many arguments to function ???????????????virtio_transport_reset_no_sock??????????????????v4.19.125: Build failed! Errors:
net/vmw_vsock/virtio_transport_common.c:1042:40: error: ???????????????t?????????????????? undeclared (first use in this function); did you mean ???????????????tm??????????????????? net/vmw_vsock/virtio_transport_common.c:1042:9: error: too many arguments to function ???????????????virtio_transport_reset_no_sock??????????????????v4.14.182: Build failed! Errors:
net/vmw_vsock/virtio_transport_common.c:1038:40: error: ???????????????t?????????????????? undeclared (first use in this function); did you mean ???????????????tm??????????????????? net/vmw_vsock/virtio_transport_common.c:1038:9: error: too many arguments to function ???????????????virtio_transport_reset_no_sock??????????????????v4.9.225: Build failed! Errors:
net/vmw_vsock/virtio_transport_common.c:968:40: error: ???????????????t?????????????????? undeclared (first use in this function); did you mean ???????????????tm??????????????????? net/vmw_vsock/virtio_transport_common.c:968:9: error: too many arguments to function ???????????????virtio_transport_reset_no_sock??????????????????
Quoted 7 lines
v4.4.225: Failed to apply! Possible dependencies:
06a8fc78367d ("VSOCK: Introduce virtio_vsock_common.ko")NOTE: The patch will not be queued to stable trees until it is upstream.
How should we proceed with this patch?
I think we can simply remove the 't' from virtio_transport_reset_no_sock(), the following patch should work:
net/vmw_vsock/virtio_transport_common.c +8 -0
@@ -1104,6 +1104,14 @@ void virtio_transport_recv_pkt(struct virtio_vsock_pkt *pkt)lock_sock (sk); +/* Check if sk has been released before lock_sock */ +if (sk->sk_shutdown ==SHUTDOWN_MASK ) { + (void )virtio_transport_reset_no_sock (pkt); +release_sock (sk); +sock_put (sk); + goto free_pkt; + } +/* Update CID in case it has changed after a transport reset event */ vsk->local_addr.svm_cid = dst.svm_cid;
Thanks,
Stefano
Quoted 74 lines
-----Original Message-----
From: Stefano Garzarella <[email protected]>
Sent: Friday, June 5, 2020 10:30 PM
To: Sasha Levin <[email protected]>
Cc: Justin He <[email protected]>; Stefan Hajnoczi <[email protected]>; David S. Miller <[email protected]>; [email protected]
Subject: Re: [PATCH v2]virtio_vsock: Fix race condition invirtio_transport_recv_pktQuoted 8 lines · Sasha Levin
Hi
[This is an automated email]
This commit has been processed because it contains a -stable tag. The stable tag indicates that it's relevant for the following trees: all
The bot has tested the following trees: v5.6.15, v5.4.43, v4.19.125,
v4.14.182, v4.9.225, v4.4.225.
Quoted 4 lines
v5.6.15: Build OK!
v5.4.43: Build failed! Errors:net/vmw_vsock/virtio_transport_common.c:1109:40:error: ???????????????t?????????????????? undeclared (first use in this function); did you mean ???????????????tm???????????????????
Quoted 1 line
net/vmw_vsock/virtio_transport_common.c:1109:9: error: too manyarguments to
function ???????????????virtio_transport_reset_no_sock??????????????????Quoted 3 lines
v4.19.125: Build failed! Errors:
net/vmw_vsock/virtio_transport_common.c:1042:40:error: ???????????????t?????????????????? undeclared (first use in this function); did you mean ???????????????tm???????????????????
Quoted 1 line
net/vmw_vsock/virtio_transport_common.c:1042:9: error: too manyarguments to
function ???????????????virtio_transport_reset_no_sock??????????????????Quoted 3 lines
v4.14.182: Build failed! Errors:
net/vmw_vsock/virtio_transport_common.c:1038:40:error: ???????????????t?????????????????? undeclared (first use in this function); did you mean ???????????????tm???????????????????
Quoted 1 line
net/vmw_vsock/virtio_transport_common.c:1038:9: error: too manyarguments to
function ???????????????virtio_transport_reset_no_sock??????????????????Quoted 3 lines
v4.9.225: Build failed! Errors:
net/vmw_vsock/virtio_transport_common.c:968:40:error: ???????????????t?????????????????? undeclared (first use in this function); did you mean ???????????????tm???????????????????
Quoted 1 line
net/vmw_vsock/virtio_transport_common.c:968:9: error: too manyarguments to
function ???????????????virtio_transport_reset_no_sock??????????????????Quoted 8 lines
v4.4.225: Failed to apply! Possible dependencies:
06a8fc78367d ("VSOCK: Introduce virtio_vsock_common.ko")NOTE: The patch will not be queued to stable trees until it is upstream.
How should we proceed with this patch?
I think we can simply remove the 't' from
virtio_transport_reset_no_sock(), the following patch should work:net/vmw_vsock/virtio_transport_common.c +0
-0@@ -1104,6 +1104,14 @@ void virtio_transport_recv_pkt(struct virtio_vsock_pkt *pkt)
net/vmw_vsock/virtio_transport_common.c void virtio_transport_recv_pkt(struct · line 1115
Quoted 10 lines
net/vmw_vsock/virtio_transport_common.c +8
-0lock_sock (sk); +/* Check if sk has been released before lock_sock */ +if (sk->sk_shutdown ==SHUTDOWN_MASK ) { + (void )virtio_transport_reset_no_sock (pkt); +release_sock (sk); +sock_put (sk); + goto free_pkt; + } +
Thanks Stefano,
Yes, this (t) parameter changes is introduced after 4c7246dc45e27 (v5.4-rc6-1712). So the stable tree (version less than v5.4) should apply above patch.
Sorry for my late response, the auto mail is filtered by our outlook server ☹
--
Cheers,
Justin (Jia He)
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.