Skip to contentSkip to conversation

mptcp: fix tcp fallback crash

PATCH In review Florian Westphal 5 participants 6 messages Mar 31, 2020 – Apr 1, 2020

Subsystems (2)
lore
Share & tools
lore

Florian Westphal

author
#0 lore raw

PATCH 1/1 mptcp: fix tcp fallback crash

Christoph Paasch reports following crash:

general protection fault, probably for non-canonical address 0xfe85e717fe88a633: 0000 [#1] PREEMPT SMP
CPU: 0 PID: 2874 Comm: syz-executor072 Not tainted 5.6.0-rc5 #62
RIP: 0010:__pv_queued_spin_lock_slowpath+0x1b3/0x2f0 kernel/locking/qspinlock.c:471
[..]
 queued_spin_lock_slowpath arch/x86/include/asm/qspinlock.h:50 [inline]
 do_raw_spin_lock include/linux/spinlock.h:181 [inline]
 spin_lock_bh include/linux/spinlock.h:343 [inline]
 __mptcp_flush_join_list+0x44/0xb0 net/mptcp/protocol.c:278
 mptcp_shutdown+0xb3/0x230 net/mptcp/protocol.c:1882

[..]

Problem is that mptcp_shutdown() socket isn't an mptcp socket, its a plain tcp_sk. Thus, trying to access mptcp_sk specific members accesses garbage.

Root cause is that accept() returns a fallback (tcp) socket, not an mptcp one. There is code in getpeername to detect this and override the sockets stream_ops. But this will only run when accept() caller provided a sockaddr struct. "accept(fd, NULL, 0)" will therefore result in mptcp stream ops, with sock->sk being a tcp_sk. Update the existing fallback handling to detect this as well.

Moreover, mptcp_shutdown did not have fallback handling, and mptcp_poll did it too late so add that there as well.

Reported-by Christoph Paasch <cpaasch(a)apple.com>
Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 net/mptcp/protocol.c | 50 ++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 46 insertions(+), 4 deletions(-)
net/mptcp/protocol.c +46 -4
index a3523bc0739b..72f3176dc924 100644
@@ -57,10 +57,43 @@ static bool __mptcp_needs_tcp_fallback(const struct mptcp_sock *msk)
 	return msk->first && !sk_is_mptcp(msk->first);
 }
 
+static struct socket *mptcp_is_tcpsk(struct sock *sk)
+{
+	struct socket *sock = sk->sk_socket;
+
+	if (sock->sk != sk)
+		return NULL;
+
+	if (unlikely(sk->sk_prot == &tcp_prot)) {
+		/* we are being invoked after mptcp_accept() has
+		 * accepted a non-mp-capable flow: sk is a tcp_sk,
+		 * not an mptcp one.
+		 *
+		 * Hand the socket over to tcp so all further socket ops
+		 * bypass mptcp.
+		 */
+		sock->ops = &inet_stream_ops;
+		return sock;
+#if IS_ENABLED(CONFIG_MPTCP_IPV6)
+	} else if (unlikely(sk->sk_prot == &tcpv6_prot)) {
+		sock->ops = &inet6_stream_ops;
+		return sock;
+#endif
+	}
+
+	return NULL;
+}
+
 static struct socket *__mptcp_tcp_fallback(struct mptcp_sock *msk)
 {
+	struct socket *sock;
+
 	sock_owned_by_me((const struct sock *)msk);
 
+	sock = mptcp_is_tcpsk((struct sock *)msk);
+	if (unlikely(sock))
+		return sock;
+
 	if (likely(!__mptcp_needs_tcp_fallback(msk)))
 		return NULL;
 
@@ -84,6 +117,10 @@ static struct socket *__mptcp_socket_create(struct mptcp_sock *msk, int state)
 	struct socket *ssock;
 	int err;
 
+	ssock = __mptcp_tcp_fallback(msk);
+	if (unlikely(ssock))
+		return ssock;
+
 	ssock = __mptcp_nmpc_socket(msk);
 	if (ssock)
 		goto set_state;
@@ -1820,7 +1857,9 @@ static __poll_t mptcp_poll(struct file *file, struct socket *sock,
 
 	msk = mptcp_sk(sk);
 	lock_sock(sk);
-	ssock = __mptcp_nmpc_socket(msk);
+	ssock = __mptcp_tcp_fallback(msk);
+	if (!ssock)
+		ssock = __mptcp_nmpc_socket(msk);
 	if (ssock) {
 		mask = ssock->ops->poll(file, ssock, wait);
 		release_sock(sk);
@@ -1830,9 +1869,6 @@ static __poll_t mptcp_poll(struct file *file, struct socket *sock,
 	release_sock(sk);
 	sock_poll_wait(file, sock, wait);
 	lock_sock(sk);
-	ssock = __mptcp_tcp_fallback(msk);
-	if (unlikely(ssock))
-		return ssock->ops->poll(file, ssock, NULL);
 
 	if (test_bit(MPTCP_DATA_READY, &msk->flags))
 		mask = EPOLLIN | EPOLLRDNORM;
@@ -1851,11 +1887,17 @@ static int mptcp_shutdown(struct socket *sock, int how)
 {
 	struct mptcp_sock *msk = mptcp_sk(sock->sk);
 	struct mptcp_subflow_context *subflow;
+	struct socket *ssock;
 	int ret = 0;
 
 	pr_debug("sk=%p, how=%d", msk, how);
 
 	lock_sock(sock->sk);
+	ssock = __mptcp_tcp_fallback(msk);
+	if (ssock) {
+		release_sock(sock->sk);
+		return inet_shutdown(ssock, how);
+	}
 
 	if (how == SHUT_WR || how == SHUT_RDWR)
 		inet_sk_state_store(sock->sk, TCP_FIN_WAIT1);
Signature · 1 line
2.24.1
#1 lore raw
Quoted 24 lines · Florian Westphal

Christoph Paasch reports following crash:

general protection fault, probably for non-canonical address 0xfe85e717fe88a633: 0000 [#1] PREEMPT SMP
CPU: 0 PID: 2874 Comm: syz-executor072 Not tainted 5.6.0-rc5 #62
RIP: 0010:__pv_queued_spin_lock_slowpath+0x1b3/0x2f0 kernel/locking/qspinlock.c:471
[..]
 queued_spin_lock_slowpath arch/x86/include/asm/qspinlock.h:50 [inline]
 do_raw_spin_lock include/linux/spinlock.h:181 [inline]
 spin_lock_bh include/linux/spinlock.h:343 [inline]
 __mptcp_flush_join_list+0x44/0xb0 net/mptcp/protocol.c:278
 mptcp_shutdown+0xb3/0x230 net/mptcp/protocol.c:1882

[..]

Problem is that mptcp_shutdown() socket isn't an mptcp socket, its a plain tcp_sk. Thus, trying to access mptcp_sk specific members accesses garbage.

Root cause is that accept() returns a fallback (tcp) socket, not an mptcp one. There is code in getpeername to detect this and override the sockets stream_ops. But this will only run when accept() caller provided a sockaddr struct. "accept(fd, NULL, 0)" will therefore result in mptcp stream ops, with sock->sk being a tcp_sk. Update the existing fallback handling to detect this as well.

Quoted 2 lines

Moreover, mptcp_shutdown did not have fallback handling, and mptcp_poll did it too late so add that there as well.

hi Florian,

maybe there is still a point where the NULL dereference can happen, in mptcp_sendmsg()

730 lock_sock(ssk);
731 while (msg_data_left(msg)) {
732 ret = mptcp_sendmsg_frag(sk, ssk, msg, NULL, &timeo, &mss_now,
733 &size_goal);
734 if (ret < 0)
735 break;
736 if (ret == 0 && unlikely(__mptcp_needs_tcp_fallback(msk))) {

 737                         release_sock(ssk);
 738                         ssock = __mptcp_tcp_fallback(msk);
 739                         goto fallback; <--- can ssock can be NULL here?

740 }
741
742 copied += ret;
743 }

wdyt?

Signature · 1 line
davide
author
#2 lore raw
Quoted 25 lines · Davide Caratti
Quoted 24 lines · Florian Westphal

Christoph Paasch reports following crash:

general protection fault, probably for non-canonical address 0xfe85e717fe88a633: 0000 [#1] PREEMPT SMP
CPU: 0 PID: 2874 Comm: syz-executor072 Not tainted 5.6.0-rc5 #62
RIP: 0010:__pv_queued_spin_lock_slowpath+0x1b3/0x2f0 kernel/locking/qspinlock.c:471
[..]
 queued_spin_lock_slowpath arch/x86/include/asm/qspinlock.h:50 [inline]
 do_raw_spin_lock include/linux/spinlock.h:181 [inline]
 spin_lock_bh include/linux/spinlock.h:343 [inline]
 __mptcp_flush_join_list+0x44/0xb0 net/mptcp/protocol.c:278
 mptcp_shutdown+0xb3/0x230 net/mptcp/protocol.c:1882

[..]

Problem is that mptcp_shutdown() socket isn't an mptcp socket, its a plain tcp_sk. Thus, trying to access mptcp_sk specific members accesses garbage.

Root cause is that accept() returns a fallback (tcp) socket, not an mptcp one. There is code in getpeername to detect this and override the sockets stream_ops. But this will only run when accept() caller provided a sockaddr struct. "accept(fd, NULL, 0)" will therefore result in mptcp stream ops, with sock->sk being a tcp_sk. Update the existing fallback handling to detect this as well.

Quoted 7 lines
Quoted 2 lines

Moreover, mptcp_shutdown did not have fallback handling, and mptcp_poll did it too late so add that there as well.

hi Florian,

maybe there is still a point where the NULL dereference can happen, in mptcp_sendmsg()

Not because of this specific problem.

Quoted 11 lines

730 lock_sock(ssk);
731 while (msg_data_left(msg)) {
732 ret = mptcp_sendmsg_frag(sk, ssk, msg, NULL, &timeo, &mss_now,
733 &size_goal);
734 if (ret < 0)
735 break;
736 if (ret == 0 && unlikely(__mptcp_needs_tcp_fallback(msk))) {

 737                         release_sock(ssk);
 738                         ssock = __mptcp_tcp_fallback(msk);
 739                         goto fallback; <--- can ssock can be NULL here?

740 }

It looks odd, yes. I propose this separate patch:

Subject: mptcp: consistently retry tcp fallback lookup

There is confusion about when we can assume that the fallback socket is non-null. Handle this consistently and do a full re-lookup.

net/mptcp/protocol.c +2 -3
@@ -734,9 +734,9 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 			goto out;
 	}
 
+fallback:
 	ssock = __mptcp_tcp_fallback(msk);
 	if (unlikely(ssock)) {
-fallback:
 		pr_debug("fallback passthrough");
 		ret = sock_sendmsg(ssock, msg);
 		return ret >= 0 ? ret + copied : (copied ? copied : ret);
@@ -779,7 +779,6 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 			break;
 		if (ret == 0 && unlikely(__mptcp_needs_tcp_fallback(msk))) {
 			release_sock(ssk);
-			ssock = __mptcp_tcp_fallback(msk);
 			goto fallback;
 		}
 
@@ -889,9 +888,9 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 		return -EOPNOTSUPP;
 
 	lock_sock(sk);
+fallback:
 	ssock = __mptcp_tcp_fallback(msk);
 	if (unlikely(ssock)) {
-fallback:
 		pr_debug("fallback-read subflow=%p",
 			 mptcp_subflow_ctx(ssock->sk));
Tested-by on v1 1/1
#3 lore raw
In reply to Florian Westphal
Quoted 24 lines · Florian Westphal

Christoph Paasch reports following crash:

general protection fault, probably for non-canonical address 0xfe85e717fe88a633: 0000 [#1] PREEMPT SMP
CPU: 0 PID: 2874 Comm: syz-executor072 Not tainted 5.6.0-rc5 #62
RIP: 0010:__pv_queued_spin_lock_slowpath+0x1b3/0x2f0 kernel/locking/qspinlock.c:471
[..]
 queued_spin_lock_slowpath arch/x86/include/asm/qspinlock.h:50 [inline]
 do_raw_spin_lock include/linux/spinlock.h:181 [inline]
 spin_lock_bh include/linux/spinlock.h:343 [inline]
 __mptcp_flush_join_list+0x44/0xb0 net/mptcp/protocol.c:278
 mptcp_shutdown+0xb3/0x230 net/mptcp/protocol.c:1882

[..]

Problem is that mptcp_shutdown() socket isn't an mptcp socket, its a plain tcp_sk. Thus, trying to access mptcp_sk specific members accesses garbage.

Root cause is that accept() returns a fallback (tcp) socket, not an mptcp one. There is code in getpeername to detect this and override the sockets stream_ops. But this will only run when accept() caller provided a sockaddr struct. "accept(fd, NULL, 0)" will therefore result in mptcp stream ops, with sock->sk being a tcp_sk. Update the existing fallback handling to detect this as well.

Quoted 8 lines

Moreover, mptcp_shutdown did not have fallback handling, and mptcp_poll did it too late so add that there as well.

Reported-by Christoph Paasch <cpaasch(a)apple.com>
Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 net/mptcp/protocol.c | 50 ++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 46 insertions(+), 4 deletions(-)

Works for me!

Tested-by Christoph Paasch <cpaasch(a)apple.com>

I think it also needs the Fixes-tag:

Fixes ec3edaa7ca6c ("mptcp: Add handling of outgoing MP_JOIN requests")

Christoph

Reviewed-by on v1 1/1
#4 lore raw
In reply to Florian Westphal
Quoted 24 lines · Florian Westphal

Christoph Paasch reports following crash:

general protection fault, probably for non-canonical address 0xfe85e717fe88a633: 0000 [#1] PREEMPT SMP
CPU: 0 PID: 2874 Comm: syz-executor072 Not tainted 5.6.0-rc5 #62
RIP: 0010:__pv_queued_spin_lock_slowpath+0x1b3/0x2f0 kernel/locking/qspinlock.c:471
[..]
queued_spin_lock_slowpath arch/x86/include/asm/qspinlock.h:50 [inline]
do_raw_spin_lock include/linux/spinlock.h:181 [inline]
spin_lock_bh include/linux/spinlock.h:343 [inline]
__mptcp_flush_join_list+0x44/0xb0 net/mptcp/protocol.c:278
mptcp_shutdown+0xb3/0x230 net/mptcp/protocol.c:1882

[..]

Problem is that mptcp_shutdown() socket isn't an mptcp socket, its a plain tcp_sk. Thus, trying to access mptcp_sk specific members accesses garbage.

Root cause is that accept() returns a fallback (tcp) socket, not an mptcp one. There is code in getpeername to detect this and override the sockets stream_ops. But this will only run when accept() caller provided a sockaddr struct. "accept(fd, NULL, 0)" will therefore result in mptcp stream ops, with sock->sk being a tcp_sk. Update the existing fallback handling to detect this as well.

Quoted 9 lines

Moreover, mptcp_shutdown did not have fallback handling, and mptcp_poll did it too late so add that there as well.

Reported-by Christoph Paasch <cpaasch(a)apple.com>
Signed-off-by Florian Westphal <fw(a)strlen.de>

---
net/mptcp/protocol.c | 50 ++++++++++++++++++++++++++++++++++++++++----

1 file changed, 46 insertions(+), 4 deletions(-)

Looks good to me, thanks Florian.

Reviewed-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>

--
Mat Martineau
Intel

#5 lore raw

Hi Florian, Mat, Christoph,

Quoted 38 lines · Mat Martineau
Quoted 26 lines · Florian Westphal

Christoph Paasch reports following crash:

general protection fault, probably for non-canonical address 
0xfe85e717fe88a633: 0000 [#1] PREEMPT SMP
CPU: 0 PID: 2874 Comm: syz-executor072 Not tainted 5.6.0-rc5 #62
RIP: 0010:__pv_queued_spin_lock_slowpath+0x1b3/0x2f0 
kernel/locking/qspinlock.c:471
[..]
queued_spin_lock_slowpath arch/x86/include/asm/qspinlock.h:50 [inline]
do_raw_spin_lock include/linux/spinlock.h:181 [inline]
spin_lock_bh include/linux/spinlock.h:343 [inline]
__mptcp_flush_join_list+0x44/0xb0 net/mptcp/protocol.c:278
mptcp_shutdown+0xb3/0x230 net/mptcp/protocol.c:1882

[..]

Problem is that mptcp_shutdown() socket isn't an mptcp socket, its a plain tcp_sk.  Thus, trying to access mptcp_sk specific members accesses garbage.

Root cause is that accept() returns a fallback (tcp) socket, not an mptcp one.  There is code in getpeername to detect this and override the sockets stream_ops.  But this will only run when accept() caller provided a sockaddr struct.  "accept(fd, NULL, 0)" will therefore result in mptcp stream ops, with sock->sk being a tcp_sk.  Update the existing fallback handling to detect this as well.

Quoted 9 lines

Moreover, mptcp_shutdown did not have fallback handling, and mptcp_poll did it too late so add that there as well.

Reported-by Christoph Paasch <cpaasch(a)apple.com>
Signed-off-by Florian Westphal <fw(a)strlen.de>

---
net/mptcp/protocol.c | 50 ++++++++++++++++++++++++++++++++++++++++----

1 file changed, 46 insertions(+), 4 deletions(-)
Quoted 3 lines

Looks good to me, thanks Florian.

Reviewed-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>

Thank you for the patch, the review, the bug report and the test!

- 1367f91c2e4a: mptcp: fix tcp fallback crash
- 57580002e381: tg: remove accidental signed-off

I just added this patch at the end of the series (export branch)

Tests are all green on my side!

Cheers,
Matt

Signature · 5 lines
Matthieu Baerts | R&D Engineer
matthieu.baerts(a)tessares.net
Tessares SA | Hybrid Access Solutions
www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium