Skip to contentSkip to conversation

virtio_vsock: Fix race condition in virtio_transport_recv_pkt

PATCH Superseded Jia He 3 participants 6 messages May 29, 2020 – Jun 8, 2020

Subsystems (7)

v2 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Jia He

author
#0 lore raw

PATCH v2 1/1 virtio_vsock: Fix race condition in virtio_transport_recv_pkt

When client tries to connect(SOCK_STREAM) the server in the guest with NONBLOCK mode, there will be a panic on a ThunderX2 (armv8a server):

[  463.718844][ T5040] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000
[  463.718848][ T5040] Mem abort info:
[  463.718849][ T5040]   ESR = 0x96000044
[  463.718852][ T5040]   EC = 0x25: DABT (current EL), IL = 32 bits
[  463.718853][ T5040]   SET = 0, FnV = 0
[  463.718854][ T5040]   EA = 0, S1PTW = 0
[  463.718855][ T5040] Data abort info:
[  463.718856][ T5040]   ISV = 0, ISS = 0x00000044
[  463.718857][ T5040]   CM = 0, WnR = 1
[  463.718859][ T5040] user pgtable: 4k pages, 48-bit VAs, pgdp=0000008f6f6e9000
[  463.718861][ T5040] [0000000000000000] pgd=0000000000000000
[  463.718866][ T5040] Internal error: Oops: 96000044 [#1] SMP
[...]
[  463.718977][ T5040] CPU: 213 PID: 5040 Comm: vhost-5032 Tainted: G           O      5.7.0-rc7+ #139
[  463.718980][ T5040] Hardware name: GIGABYTE R281-T91-00/MT91-FS1-00, BIOS F06 09/25/2018
[  463.718982][ T5040] pstate: 60400009 (nZCv daif +PAN -UAO)
[  463.718995][ T5040] pc : virtio_transport_recv_pkt+0x4c8/0xd40 [vmw_vsock_virtio_transport_common]
[  463.718999][ T5040] lr : virtio_transport_recv_pkt+0x1fc/0xd40 [vmw_vsock_virtio_transport_common]
[  463.719000][ T5040] sp : ffff80002dbe3c40
[...]
[  463.719025][ T5040] Call trace:
[  463.719030][ T5040]  virtio_transport_recv_pkt+0x4c8/0xd40 [vmw_vsock_virtio_transport_common]
[  463.719034][ T5040]  vhost_vsock_handle_tx_kick+0x360/0x408 [vhost_vsock]
[  463.719041][ T5040]  vhost_worker+0x100/0x1a0 [vhost]
[  463.719048][ T5040]  kthread+0x128/0x130
[  463.719052][ T5040]  ret_from_fork+0x10/0x18

The race condition as follows:

Task1                            Task2
=====                            =====
__sock_release                   virtio_transport_recv_pkt
  __vsock_release                  vsock_find_bound_socket (found)
    lock_sock_nested
    vsock_remove_sock
    sock_orphan
      sk_set_socket(sk, NULL)
    ...
    release_sock
                                lock_sock
                                   virtio_transport_recv_connecting
                                     sk->sk_socket->state (panic)

The root cause is that vsock_find_bound_socket can't hold the lock_sock, so there is a small race window between vsock_find_bound_socket() and lock_sock(). If there is __vsock_release() in another task, sk->sk_socket will be set to NULL inadvertently.

This fixes it by checking sk->sk_shutdown.

---
v2: use lightweight checking suggested by Stefano Garzarella

 net/vmw_vsock/virtio_transport_common.c | 8 ++++++++
 1 file changed, 8 insertions(+)
net/vmw_vsock/virtio_transport_common.c +8 -0
index 69efc891885f..0edda1edf988 100644
@@ -1132,6 +1132,14 @@ void virtio_transport_recv_pkt(struct virtio_transport *t,
 
 	lock_sock(sk);
 
+	/* Check if sk has been released before lock_sock */
+	if (sk->sk_shutdown == SHUTDOWN_MASK) {
+		(void)virtio_transport_reset_no_sock(t, pkt);
+		release_sock(sk);
+		sock_put(sk);
+		goto free_pkt;
+	}
+
 	/* Update CID in case it has changed after a transport reset event */
 	vsk->local_addr.svm_cid = dst.svm_cid;
 
Signature · 1 line
2.17.1
maintainer Reviewed-by on v2 1/1
#1 lore raw
Quoted 20 lines · Jia He

When client tries to connect(SOCK_STREAM) the server in the guest with NONBLOCK mode, there will be a panic on a ThunderX2 (armv8a server):

[  463.718844][ T5040] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000
[  463.718848][ T5040] Mem abort info:
[  463.718849][ T5040]   ESR = 0x96000044
[  463.718852][ T5040]   EC = 0x25: DABT (current EL), IL = 32 bits
[  463.718853][ T5040]   SET = 0, FnV = 0
[  463.718854][ T5040]   EA = 0, S1PTW = 0
[  463.718855][ T5040] Data abort info:
[  463.718856][ T5040]   ISV = 0, ISS = 0x00000044
[  463.718857][ T5040]   CM = 0, WnR = 1
[  463.718859][ T5040] user pgtable: 4k pages, 48-bit VAs, pgdp=0000008f6f6e9000
[  463.718861][ T5040] [0000000000000000] pgd=0000000000000000
[  463.718866][ T5040] Internal error: Oops: 96000044 [#1] SMP
[...]
[  463.718977][ T5040] CPU: 213 PID: 5040 Comm: vhost-5032 Tainted: G           O      5.7.0-rc7+ #139
[  463.718980][ T5040] Hardware name: GIGABYTE R281-T91-00/MT91-FS1-00, BIOS F06 09/25/2018
[  463.718982][ T5040] pstate: 60400009 (nZCv daif +PAN -UAO)
[  463.718995][ T5040] pc : virtio_transport_recv_pkt+0x4c8/0xd40 [vmw_vsock_virtio_transport_common]
[  463.718999][ T5040] lr : virtio_transport_recv_pkt+0x1fc/0xd40 [vmw_vsock_virtio_transport_common]
Quoted 8 lines
[  463.719000][ T5040] sp : ffff80002dbe3c40
[...]
[  463.719025][ T5040] Call trace:
[  463.719030][ T5040]  virtio_transport_recv_pkt+0x4c8/0xd40 [vmw_vsock_virtio_transport_common]
[  463.719034][ T5040]  vhost_vsock_handle_tx_kick+0x360/0x408 [vhost_vsock]
[  463.719041][ T5040]  vhost_worker+0x100/0x1a0 [vhost]
[  463.719048][ T5040]  kthread+0x128/0x130
[  463.719052][ T5040]  ret_from_fork+0x10/0x18

^ ^
Maybe we can remove these two columns from the commit message.

Quoted 10 lines

The race condition as follows:

Task1                            Task2
=====                            =====
__sock_release                   virtio_transport_recv_pkt

__vsock_release vsock_find_bound_socket (found)
lock_sock_nested
vsock_remove_sock
sock_orphan
sk_set_socket(sk, NULL)

Here we can add:

      sk->sk_shutdown = SHUTDOWN_MASK;
Quoted 44 lines
    ...
    release_sock
                                lock_sock
                                   virtio_transport_recv_connecting
                                     sk->sk_socket->state (panic)

The root cause is that vsock_find_bound_socket can't hold the lock_sock, so there is a small race window between vsock_find_bound_socket() and lock_sock(). If there is __vsock_release() in another task, sk->sk_socket will be set to NULL inadvertently.

This fixes it by checking sk->sk_shutdown.

---
v2: use lightweight checking suggested by Stefano Garzarella

 net/vmw_vsock/virtio_transport_common.c | 8 ++++++++
 1 file changed, 8 insertions(+)
net/vmw_vsock/virtio_transport_common.c +8 -0
index 69efc891885f..0edda1edf988 100644
@@ -1132,6 +1132,14 @@ void virtio_transport_recv_pkt(struct virtio_transport *t,
 
 	lock_sock(sk);
 
+	/* Check if sk has been released before lock_sock */
+	if (sk->sk_shutdown == SHUTDOWN_MASK) {
+		(void)virtio_transport_reset_no_sock(t, pkt);
+		release_sock(sk);
+		sock_put(sk);
+		goto free_pkt;
+	}
+
 	/* Update CID in case it has changed after a transport reset event */
 	vsk->local_addr.svm_cid = dst.svm_cid;
 
Signature · 1 line
2.17.1

Anyway, the patch LGTM, let see what David and other say.

Thanks,
Stefano

author
#2 lore raw

Hi Stefano

Quoted 63 lines

-----Original Message-----
From: Stefano Garzarella <[email protected]>
Sent: Saturday, May 30, 2020 12:34 AM
To: Justin He <[email protected]>
Cc: Stefan Hajnoczi <[email protected]>; David S. Miller <[email protected]>; Jakub Kicinski <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Kaly Xin <[email protected]>; [email protected]
Subject: Re: [PATCH v2] virtio_vsock: Fix race condition in virtio_transport_recv_pkt

Quoted 3 lines · Jia He

When client tries to connect(SOCK_STREAM) the server in the guest with NONBLOCK mode, there will be a panic on a ThunderX2 (armv8a server):

[  463.718844][ T5040] Unable to handle kernel NULL pointer dereference

at virtual address 0000000000000000

Quoted 9 lines
[  463.718848][ T5040] Mem abort info:
[  463.718849][ T5040]   ESR = 0x96000044
[  463.718852][ T5040]   EC = 0x25: DABT (current EL), IL = 32 bits
[  463.718853][ T5040]   SET = 0, FnV = 0
[  463.718854][ T5040]   EA = 0, S1PTW = 0
[  463.718855][ T5040] Data abort info:
[  463.718856][ T5040]   ISV = 0, ISS = 0x00000044
[  463.718857][ T5040]   CM = 0, WnR = 1
[  463.718859][ T5040] user pgtable: 4k pages, 48-bit VAs,

pgdp=0000008f6f6e9000

Quoted 4 lines
[  463.718861][ T5040] [0000000000000000] pgd=0000000000000000
[  463.718866][ T5040] Internal error: Oops: 96000044 [#1] SMP
[...]
[  463.718977][ T5040] CPU: 213 PID: 5040 Comm: vhost-5032 Tainted: G

O 5.7.0-rc7+ #139

Quoted 1 line
[  463.718980][ T5040] Hardware name: GIGABYTE R281-T91-00/MT91-FS1-00,

BIOS F06 09/25/2018

Quoted 2 lines
[  463.718982][ T5040] pstate: 60400009 (nZCv daif +PAN -UAO)
[  463.718995][ T5040] pc : virtio_transport_recv_pkt+0x4c8/0xd40

[vmw_vsock_virtio_transport_common]

Quoted 1 line
[  463.718999][ T5040] lr : virtio_transport_recv_pkt+0x1fc/0xd40

[vmw_vsock_virtio_transport_common]

Quoted 4 lines
[  463.719000][ T5040] sp : ffff80002dbe3c40
[...]
[  463.719025][ T5040] Call trace:
[  463.719030][ T5040]  virtio_transport_recv_pkt+0x4c8/0xd40

[vmw_vsock_virtio_transport_common]

Quoted 1 line
[  463.719034][ T5040]  vhost_vsock_handle_tx_kick+0x360/0x408

[vhost_vsock]

Quoted 3 lines
[  463.719041][ T5040]  vhost_worker+0x100/0x1a0 [vhost]
[  463.719048][ T5040]  kthread+0x128/0x130
[  463.719052][ T5040]  ret_from_fork+0x10/0x18

^ ^
Maybe we can remove these two columns from the commit message.

Quoted 10 lines

The race condition as follows:

Task1                            Task2
=====                            =====
__sock_release                   virtio_transport_recv_pkt

__vsock_release vsock_find_bound_socket (found)
lock_sock_nested
vsock_remove_sock
sock_orphan
sk_set_socket(sk, NULL)

Quoted 2 lines

Here we can add:

      sk->sk_shutdown = SHUTDOWN_MASK;

Indeed. This makes it more clearly

--
Cheers,
Justin (Jia He)

Quoted 50 lines
Quoted 17 lines
    ...
    release_sock
                                lock_sock
                                   virtio_transport_recv_connecting
                                     sk->sk_socket->state (panic)

The root cause is that vsock_find_bound_socket can't hold the lock_sock, so there is a small race window between vsock_find_bound_socket() and lock_sock(). If there is __vsock_release() in another task, sk->sk_socket will be set to NULL inadvertently.

This fixes it by checking sk->sk_shutdown.

Review

Quoted 7 lines

---
v2: use lightweight checking suggested by Stefano Garzarella

 net/vmw_vsock/virtio_transport_common.c | 8 ++++++++
 1 file changed, 8 insertions(+)
(quoted diff) +0 -0

b/net/vmw_vsock/virtio_transport_common.c

net/vmw_vsock/virtio_transport_common.c void virtio_transport_recv_pkt(struct

Quoted 4 lines

index 69efc891885f..0edda1edf988 100644

net/vmw_vsock/virtio_transport_common.c +0 -0
@@ -1132,6 +1132,14 @@ void virtio_transport_recv_pkt(struct

virtio_transport *t,

Quoted 15 lines
net/vmw_vsock/virtio_transport_common.c +8 -1
 lock_sock(sk);
 
+/* Check if sk has been released before lock_sock */
+if (sk->sk_shutdown == SHUTDOWN_MASK) {
+(void)virtio_transport_reset_no_sock(t, pkt);
+release_sock(sk);
+sock_put(sk);
+goto free_pkt;
+}
+
 /* Update CID in case it has changed after a transport reset event */
 vsk->local_addr.svm_cid = dst.svm_cid;
 
--
Quoted 2 lines

2.17.1

Anyway, the patch LGTM, let see what David and other say.

Quoted 4 lines

Thanks,
Stefano

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

maintainer
#3 lore raw
In reply to Jia He

Hi

[This is an automated email]

This commit has been processed because it contains a -stable tag. The stable tag indicates that it's relevant for the following trees: all

The bot has tested the following trees: v5.6.15, v5.4.43, v4.19.125, v4.14.182, v4.9.225, v4.4.225.

v5.6.15: Build OK!
v5.4.43: Build failed! Errors:
net/vmw_vsock/virtio_transport_common.c:1109:40: error: ‘t’ undeclared (first use in this function); did you mean ‘tm’? net/vmw_vsock/virtio_transport_common.c:1109:9: error: too many arguments to function ‘virtio_transport_reset_no_sock’

v4.19.125: Build failed! Errors:
net/vmw_vsock/virtio_transport_common.c:1042:40: error: ‘t’ undeclared (first use in this function); did you mean ‘tm’? net/vmw_vsock/virtio_transport_common.c:1042:9: error: too many arguments to function ‘virtio_transport_reset_no_sock’

v4.14.182: Build failed! Errors:
net/vmw_vsock/virtio_transport_common.c:1038:40: error: ‘t’ undeclared (first use in this function); did you mean ‘tm’? net/vmw_vsock/virtio_transport_common.c:1038:9: error: too many arguments to function ‘virtio_transport_reset_no_sock’

v4.9.225: Build failed! Errors:
net/vmw_vsock/virtio_transport_common.c:968:40: error: ‘t’ undeclared (first use in this function); did you mean ‘tm’? net/vmw_vsock/virtio_transport_common.c:968:9: error: too many arguments to function ‘virtio_transport_reset_no_sock’

v4.4.225: Failed to apply! Possible dependencies:

    06a8fc78367d ("VSOCK: Introduce virtio_vsock_common.ko")

NOTE: The patch will not be queued to stable trees until it is upstream.

How should we proceed with this patch?

Signature · 2 lines
Thanks
Sasha
#4 lore raw
Quoted 26 lines · Sasha Levin

Hi

[This is an automated email]

This commit has been processed because it contains a -stable tag. The stable tag indicates that it's relevant for the following trees: all

The bot has tested the following trees: v5.6.15, v5.4.43, v4.19.125, v4.14.182, v4.9.225, v4.4.225.

v5.6.15: Build OK!
v5.4.43: Build failed! Errors:

    net/vmw_vsock/virtio_transport_common.c:1109:40: error: ???????????????t?????????????????? undeclared (first use in this function); did you mean ???????????????tm???????????????????
    net/vmw_vsock/virtio_transport_common.c:1109:9: error: too many arguments to function ???????????????virtio_transport_reset_no_sock??????????????????

v4.19.125: Build failed! Errors:

    net/vmw_vsock/virtio_transport_common.c:1042:40: error: ???????????????t?????????????????? undeclared (first use in this function); did you mean ???????????????tm???????????????????
    net/vmw_vsock/virtio_transport_common.c:1042:9: error: too many arguments to function ???????????????virtio_transport_reset_no_sock??????????????????

v4.14.182: Build failed! Errors:

    net/vmw_vsock/virtio_transport_common.c:1038:40: error: ???????????????t?????????????????? undeclared (first use in this function); did you mean ???????????????tm???????????????????
    net/vmw_vsock/virtio_transport_common.c:1038:9: error: too many arguments to function ???????????????virtio_transport_reset_no_sock??????????????????

v4.9.225: Build failed! Errors:

    net/vmw_vsock/virtio_transport_common.c:968:40: error: ???????????????t?????????????????? undeclared (first use in this function); did you mean ???????????????tm???????????????????
    net/vmw_vsock/virtio_transport_common.c:968:9: error: too many arguments to function ???????????????virtio_transport_reset_no_sock??????????????????
Quoted 7 lines

v4.4.225: Failed to apply! Possible dependencies:

    06a8fc78367d ("VSOCK: Introduce virtio_vsock_common.ko")

NOTE: The patch will not be queued to stable trees until it is upstream.

How should we proceed with this patch?

I think we can simply remove the 't' from virtio_transport_reset_no_sock(), the following patch should work:

net/vmw_vsock/virtio_transport_common.c +8 -0
index fb2060dffb0a..17f4c93f5e75 100644
@@ -1104,6 +1104,14 @@ void virtio_transport_recv_pkt(struct virtio_vsock_pkt *pkt)
 
        lock_sock(sk);
 
+       /* Check if sk has been released before lock_sock */
+       if (sk->sk_shutdown == SHUTDOWN_MASK) {
+               (void)virtio_transport_reset_no_sock(pkt);
+               release_sock(sk);
+               sock_put(sk);
+               goto free_pkt;
+       }
+
        /* Update CID in case it has changed after a transport reset event */
        vsk->local_addr.svm_cid = dst.svm_cid;
 

Thanks,
Stefano

author
#5 lore raw
Quoted 74 lines

-----Original Message-----
From: Stefano Garzarella <[email protected]>
Sent: Friday, June 5, 2020 10:30 PM
To: Sasha Levin <[email protected]>
Cc: Justin He <[email protected]>; Stefan Hajnoczi <[email protected]>; David S. Miller <[email protected]>; [email protected]
Subject: Re: [PATCH v2] virtio_vsock: Fix race condition in virtio_transport_recv_pkt

Quoted 8 lines · Sasha Levin

Hi

[This is an automated email]

This commit has been processed because it contains a -stable tag. The stable tag indicates that it's relevant for the following trees: all

The bot has tested the following trees: v5.6.15, v5.4.43, v4.19.125,

v4.14.182, v4.9.225, v4.4.225.

Quoted 4 lines

v5.6.15: Build OK!
v5.4.43: Build failed! Errors:

    net/vmw_vsock/virtio_transport_common.c:1109:40:

error: ???????????????t?????????????????? undeclared (first use in this function); did you mean ???????????????tm???????????????????

Quoted 1 line

net/vmw_vsock/virtio_transport_common.c:1109:9: error: too many

arguments to
function ???????????????virtio_transport_reset_no_sock??????????????????

Quoted 3 lines

v4.19.125: Build failed! Errors:

    net/vmw_vsock/virtio_transport_common.c:1042:40:

error: ???????????????t?????????????????? undeclared (first use in this function); did you mean ???????????????tm???????????????????

Quoted 1 line

net/vmw_vsock/virtio_transport_common.c:1042:9: error: too many

arguments to
function ???????????????virtio_transport_reset_no_sock??????????????????

Quoted 3 lines

v4.14.182: Build failed! Errors:

    net/vmw_vsock/virtio_transport_common.c:1038:40:

error: ???????????????t?????????????????? undeclared (first use in this function); did you mean ???????????????tm???????????????????

Quoted 1 line

net/vmw_vsock/virtio_transport_common.c:1038:9: error: too many

arguments to
function ???????????????virtio_transport_reset_no_sock??????????????????

Quoted 3 lines

v4.9.225: Build failed! Errors:

    net/vmw_vsock/virtio_transport_common.c:968:40:

error: ???????????????t?????????????????? undeclared (first use in this function); did you mean ???????????????tm???????????????????

Quoted 1 line

net/vmw_vsock/virtio_transport_common.c:968:9: error: too many

arguments to
function ???????????????virtio_transport_reset_no_sock??????????????????

Quoted 8 lines

v4.4.225: Failed to apply! Possible dependencies:

    06a8fc78367d ("VSOCK: Introduce virtio_vsock_common.ko")

NOTE: The patch will not be queued to stable trees until it is upstream.

How should we proceed with this patch?

I think we can simply remove the 't' from virtio_transport_reset_no_sock(), the following patch should work:

net/vmw_vsock/virtio_transport_common.c +0 -0
index fb2060dffb0a..17f4c93f5e75 100644
@@ -1104,6 +1104,14 @@ void virtio_transport_recv_pkt(struct
 virtio_vsock_pkt *pkt)
 

net/vmw_vsock/virtio_transport_common.c void virtio_transport_recv_pkt(struct · line 1115

Quoted 10 lines
net/vmw_vsock/virtio_transport_common.c +8 -0
        lock_sock(sk);
 
+       /* Check if sk has been released before lock_sock */
+       if (sk->sk_shutdown == SHUTDOWN_MASK) {
+               (void)virtio_transport_reset_no_sock(pkt);
+               release_sock(sk);
+               sock_put(sk);
+               goto free_pkt;
+       }
+

Thanks Stefano,
Yes, this (t) parameter changes is introduced after 4c7246dc45e27 (v5.4-rc6-1712). So the stable tree (version less than v5.4) should apply above patch.

Sorry for my late response, the auto mail is filtered by our outlook server ☹

--
Cheers,
Justin (Jia He)

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.