Skip to contentSkip to conversation

mptcp: add fastclose option

PATCH In review Florian Westphal 2 participants 7 messages Nov 25, 2020 – Nov 26, 2020

Subsystems (3)
lore
Share & tools
lore

Florian Westphal

author
#0 lore raw

cover mptcp: add fastclose option

This series adds receive side support for fastclose processing. First patch is needed to avoid UAF after enabling option processing for tcp resets. Second patch calls mptcp option parsing also for reset patckets so fastclose-on-tcp-reset is handled.

Last patch adds fastclose handling.

I've submitted a pull request vs. packetdrill to add test cases for
this: https://github.com/multipath-tcp/packetdrill/pull/25

Those are the three test cases i posted earlier to this list plus a 4th test to also cover the 'two subflows closed via fastclose+reset' case.

I would send the fastclose tx side next.

I plan to hold the MPTCP reset patches back for the time being until netlink + mptcpd are aware of its existence.

 include/net/tcp.h        |    2 +-
 net/ipv4/tcp_input.c     |   13 ++++++++-----
 net/ipv4/tcp_minisocks.c |    2 +-
 net/mptcp/options.c      |   15 +++++++++++++++
 net/mptcp/protocol.c     |   33 +++++++++++++++++++++++++++++++++
 net/mptcp/protocol.h     |    4 ++++
 net/mptcp/subflow.c      |    7 ++++++-
 7 files changed, 68 insertions(+), 8 deletions(-)
author
#1 lore raw

PATCH 1/3 mptcp: hold mptcp socket before calling tcp_done

When processing options from tcp reset path its possible that tcp_done(ssk) drops the last reference on the mptcp socket which results in use-after-free.

Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 net/mptcp/subflow.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
net/mptcp/subflow.c +6 -1
index b6cd42649e5d..ca8db7ae51f0 100644
@@ -276,12 +276,17 @@ void mptcp_subflow_reset(struct sock *ssk)
 	struct mptcp_subflow_context *subflow = mptcp_subflow_ctx(ssk);
 	struct sock *sk = subflow->conn;
 
+	/* must hold: tcp_done() could drop last reference on parent */
+	sock_hold(sk);
+
 	tcp_set_state(ssk, TCP_CLOSE);
 	tcp_send_active_reset(ssk, GFP_ATOMIC);
 	tcp_done(ssk);
 	if (!test_and_set_bit(MPTCP_WORK_CLOSE_SUBFLOW, &mptcp_sk(sk)->flags) &&
 	    schedule_work(&mptcp_sk(sk)->work))
-		sock_hold(sk);
+		return; /* worker will put sk for us */
+
+	sock_put(sk);
 }
 
 static void subflow_finish_connect(struct sock *sk, const struct sk_buff *skb)
Signature · 1 line
2.26.2
author
#2 lore raw

PATCH 2/3 tcp: parse mptcp options contained in reset packets

In reply to Florian Westphal

Because TCP-level resets only affect the subflow, there is a MPTCP option to indicate that the MPTCP-level connection should be closed immediately without a mptcp-level fin exchange.

This is the 'MPTCP fast close option'. It can be carried on ack segments or TCP resets. In the latter case, its needed to parse mptcp options also for reset packets so that MPTCP can act accordingly.

Next patch will add receive side fastclose support in MPTCP.

Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 include/net/tcp.h        |  2 +-
 net/ipv4/tcp_input.c     | 13 ++++++++-----
 net/ipv4/tcp_minisocks.c |  2 +-
 3 files changed, 10 insertions(+), 7 deletions(-)
include/net/tcp.h +1 -1
index d5d22c411918..be105a5be4ba 100644
@@ -610,7 +610,7 @@ void tcp_skb_collapse_tstamp(struct sk_buff *skb,
 /* tcp_input.c */
 void tcp_rearm_rto(struct sock *sk);
 void tcp_synack_rtt_meas(struct sock *sk, struct request_sock *req);
-void tcp_reset(struct sock *sk);
+void tcp_reset(struct sock *sk, struct sk_buff *skb);
 void tcp_skb_mark_lost_uncond_verify(struct tcp_sock *tp, struct sk_buff *skb);
 void tcp_fin(struct sock *sk);
 
net/ipv4/tcp_input.c +8 -5
index fb3a7750f623..fb65d3519439 100644
@@ -4217,10 +4217,13 @@ static inline bool tcp_sequence(const struct tcp_sock *tp, u32 seq, u32 end_seq)
 }
 
 /* When we get a reset we do this. */
-void tcp_reset(struct sock *sk)
+void tcp_reset(struct sock *sk, struct sk_buff *skb)
 {
 	trace_tcp_receive_reset(sk);
 
+	if (sk_is_mptcp(sk))
+		mptcp_incoming_options(sk, skb);
+
 	/* We want the right error as BSD sees it (and indeed as we do). */
 	switch (sk->sk_state) {
 	case TCP_SYN_SENT:
@@ -5603,7 +5606,7 @@ static bool tcp_validate_incoming(struct sock *sk, struct sk_buff *skb,
 						  &tp->last_oow_ack_time))
 				tcp_send_dupack(sk, skb);
 		} else if (tcp_reset_check(sk, skb)) {
-			tcp_reset(sk);
+			tcp_reset(sk, skb);
 		}
 		goto discard;
 	}
@@ -5639,7 +5642,7 @@ static bool tcp_validate_incoming(struct sock *sk, struct sk_buff *skb,
 		}
 
 		if (rst_seq_match)
-			tcp_reset(sk);
+			tcp_reset(sk, skb);
 		else {
 			/* Disable TFO if RST is out-of-order
 			 * and no data has been received
@@ -6076,7 +6079,7 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
 		 */
 
 		if (th->rst) {
-			tcp_reset(sk);
+			tcp_reset(sk, skb);
 			goto discard;
 		}
 
@@ -6518,7 +6521,7 @@ int tcp_rcv_state_process(struct sock *sk, struct sk_buff *skb)
 			if (TCP_SKB_CB(skb)->end_seq != TCP_SKB_CB(skb)->seq &&
 			    after(TCP_SKB_CB(skb)->end_seq - th->fin, tp->rcv_nxt)) {
 				NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPABORTONDATA);
-				tcp_reset(sk);
+				tcp_reset(sk, skb);
 				return 1;
 			}
 		}
net/ipv4/tcp_minisocks.c +1 -1
index 495dda2449fe..0055ae0a3bf8 100644
@@ -801,7 +801,7 @@ struct sock *tcp_check_req(struct sock *sk, struct sk_buff *skb,
 		req->rsk_ops->send_reset(sk, skb);
 	} else if (fastopen) { /* received a valid RST pkt */
 		reqsk_fastopen_remove(sk, req, true);
-		tcp_reset(sk);
+		tcp_reset(sk, skb);
 	}
 	if (!fastopen) {
 		inet_csk_reqsk_queue_drop(sk, req);
Signature · 1 line
2.26.2
author
#3 lore raw

PATCH 3/3 mptcp: parse and act on incoming FASTCLOSE option

In reply to Florian Westphal

parse the MPTCP FASTCLOSE subtype.

If provided key matches the local one, schedule the work queue to close (with tcp reset) all subflows.

The MPTCP socket moves to closed state immediately.

Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 net/mptcp/options.c  | 15 +++++++++++++++
 net/mptcp/protocol.c | 33 +++++++++++++++++++++++++++++++++
 net/mptcp/protocol.h |  4 ++++
 3 files changed, 52 insertions(+)
net/mptcp/options.c +15 -0
index d1b4c5d208a9..6088fc80db75 100644
@@ -281,7 +281,15 @@ static void mptcp_parse_option(const struct sk_buff *skb,
 		mp_opt->rm_id = *ptr++;
 		pr_debug("RM_ADDR: id=%d", mp_opt->rm_id);
 		break;
+	case MPTCPOPT_MP_FASTCLOSE:
+		if (opsize != TCPOLEN_MPTCP_FASTCLOSE)
+			break;
 
+		ptr += 2;
+		mp_opt->rcvr_key = get_unaligned_be64(ptr);
+		ptr += 8;
+		mp_opt->fastclose = 1;
+		break;
 	default:
 		break;
 	}
@@ -299,6 +307,7 @@ void mptcp_get_options(const struct sk_buff *skb,
 	mp_opt->mp_join = 0;
 	mp_opt->add_addr = 0;
 	mp_opt->ahmac = 0;
+	mp_opt->fastclose = 0;
 	mp_opt->port = 0;
 	mp_opt->rm_addr = 0;
 	mp_opt->dss = 0;
@@ -937,6 +946,12 @@ void mptcp_incoming_options(struct sock *sk, struct sk_buff *skb)
 	if (!check_fully_established(msk, sk, subflow, skb, &mp_opt))
 		return;
 
+	if (mp_opt.fastclose &&
+	    msk->local_key == mp_opt.rcvr_key) {
+		WRITE_ONCE(msk->rcv_fastclose, true);
+		mptcp_schedule_work((struct sock *)msk);
+	}
+
 	if (mp_opt.add_addr && add_addr_hmac_valid(msk, &mp_opt)) {
 		struct mptcp_addr_info addr;
 
net/mptcp/protocol.c +33 -0
index d7875e20f8e7..f6d9a3e4cb33 100644
@@ -2209,6 +2209,36 @@ static bool mptcp_check_close_timeout(const struct sock *sk)
 	return true;
 }
 
+static void mptcp_check_fastclose(struct mptcp_sock *msk)
+{
+	struct mptcp_subflow_context *subflow, *tmp;
+	struct sock *sk = &msk->sk.icsk_inet.sk;
+
+	if (likely(!READ_ONCE(msk->rcv_fastclose)))
+		return;
+
+	mptcp_token_destroy(msk);
+
+	list_for_each_entry_safe(subflow, tmp, &msk->conn_list, node) {
+		struct sock *tcp_sk = mptcp_subflow_tcp_sock(subflow);
+
+		lock_sock(tcp_sk);
+		if (tcp_sk->sk_state != TCP_CLOSE) {
+			tcp_send_active_reset(tcp_sk, GFP_ATOMIC);
+			tcp_set_state(tcp_sk, TCP_CLOSE);
+		}
+		release_sock(tcp_sk);
+	}
+
+	inet_sk_state_store(sk, TCP_CLOSE);
+	sk->sk_shutdown = SHUTDOWN_MASK;
+	smp_mb__before_atomic(); /* SHUTDOWN must be visible first */
+	set_bit(MPTCP_DATA_READY, &msk->flags);
+	set_bit(MPTCP_WORK_CLOSE_SUBFLOW, &msk->flags);
+
+	mptcp_close_wake_up(sk);
+}
+
 static void mptcp_worker(struct work_struct *work)
 {
 	struct mptcp_sock *msk = container_of(work, struct mptcp_sock, work);
@@ -2225,6 +2255,9 @@ static void mptcp_worker(struct work_struct *work)
 
 	mptcp_check_data_fin_ack(sk);
 	__mptcp_flush_join_list(msk);
+
+	mptcp_check_fastclose(msk);
+
 	if (test_and_clear_bit(MPTCP_WORK_CLOSE_SUBFLOW, &msk->flags))
 		__mptcp_close_subflow(msk);
 
net/mptcp/protocol.h +4 -0
index f5db6e721fcf..6617cb376795 100644
@@ -23,6 +23,7 @@
 #define OPTION_MPTCP_ADD_ADDR	BIT(6)
 #define OPTION_MPTCP_ADD_ADDR6	BIT(7)
 #define OPTION_MPTCP_RM_ADDR	BIT(8)
+#define OPTION_MPTCP_FASTCLOSE	BIT(9)
 
 /* MPTCP option subtypes */
 #define MPTCPOPT_MP_CAPABLE	0
@@ -59,6 +60,7 @@
 #define TCPOLEN_MPTCP_PORT_LEN		4
 #define TCPOLEN_MPTCP_ADD_ADDR_HMAC	8
 #define TCPOLEN_MPTCP_RM_ADDR_BASE	4
+#define TCPOLEN_MPTCP_FASTCLOSE		12
 
 /* MPTCP MP_JOIN flags */
 #define MPTCPOPT_BACKUP		BIT(0)
@@ -111,6 +113,7 @@ struct mptcp_options_received {
 	u16	data_len;
 	u16	mp_capable : 1,
 		mp_join : 1,
+		fastclose : 1,
 		dss : 1,
 		add_addr : 1,
 		rm_addr : 1,
@@ -237,6 +240,7 @@ struct mptcp_sock {
 	bool		fully_established;
 	bool		rcv_data_fin;
 	bool		snd_data_fin_enable;
+	bool		rcv_fastclose;
 	bool		use_64bit_ack; /* Set when we received a 64-bit DSN */
 	spinlock_t	join_list_lock;
 	struct sock	*ack_hint;
Signature · 1 line
2.26.2
#4 lore raw
In reply to Florian Westphal

Hi Florian,

Quoted 7 lines · Florian Westphal

This series adds receive side support for fastclose processing. First patch is needed to avoid UAF after enabling option processing for tcp resets. Second patch calls mptcp option parsing also for reset patckets so fastclose-on-tcp-reset is handled.

Last patch adds fastclose handling.

Quoted 7 lines

I've submitted a pull request vs. packetdrill to add test cases for
this: https://github.com/multipath-tcp/packetdrill/pull/25

Those are the three test cases i posted earlier to this list plus a 4th test to also cover the 'two subflows closed via fastclose+reset' case.

I would send the fastclose tx side next.

Thank you for these patches, they are very good!

Just added at the end of our tree with my Reviewed-by tag (there were other reviewers on the previous patches, I can of course add more tags if the new version is OK for others):

- 6766eaff5c84: mptcp: hold mptcp socket before calling tcp_done
- 37a53605cd6b: tcp: parse mptcp options contained in reset packets
- 0b9d6f3f5a2b: mptcp: parse and act on incoming FASTCLOSE option
- Results: 9e11c1d2b8db..ba6c5ecb4ea5

Tests + export are in progress!

Quoted 2 lines

I plan to hold the MPTCP reset patches back for the time being until netlink + mptcpd are aware of its existence.

I certainly missed something here but are you here talking about MP_TCPRST support? Or still about the send of fast-close?

If it is about fast-close, why netlink + mptcpd needs to be aware first? Could we eventually send fast-close when the userspace app (here using packetdrill format) does this:

     setsockopt(<fd>, SOL_SOCKET, SO_LINGER, {onoff=1, linger=0}, 8);
     close(<fd>);

(if we accept this socket option)
No rush of course, just in case these patches were already written :)

Cheers,
Matt

Signature · 2 lines
Tessares | Belgium | Hybrid Access Solutions
www.tessares.net
author
#5 lore raw
Quoted 6 lines · Matthieu Baerts
Quoted 2 lines · Florian Westphal

I plan to hold the MPTCP reset patches back for the time being until netlink + mptcpd are aware of its existence.

I certainly missed something here but are you here talking about MP_TCPRST support? Or still about the send of fast-close?

Oh, no, I jumped off the fastclose train here. This is only wrt. MP_TCPRST. Another approach would be to add the *send side* for tcprst first, perhaps it helps with tcpdump deciphering.

But for RX there is really no point when the kernel can't pass that information along to a consumer such as mptcpd.

Quoted 9 lines

If it is about fast-close, why netlink + mptcpd needs to be aware first? Could we eventually send fast-close when the userspace app (here using packetdrill format) does this:

    setsockopt(<fd>, SOL_SOCKET, SO_LINGER, {onoff=1, linger=0}, 8);
    close(<fd>);

(if we accept this socket option)
No rush of course, just in case these patches were already written :)

Yes, I think it might make sense to allow userspace to generate a fastclose at will.

The last iteration I sent autogenerated a fastclose for the 'unread data in socket queue when userspace called close' case, which is analogous to what the linux TCP stack does when you close with unread data in the local socket.

But I think extending it to linger makes sense as well.

#6 lore raw

Hi Florian,

Quoted 11 lines · Florian Westphal
Quoted 6 lines · Matthieu Baerts
Quoted 2 lines · Florian Westphal

I plan to hold the MPTCP reset patches back for the time being until netlink + mptcpd are aware of its existence.

I certainly missed something here but are you here talking about MP_TCPRST support? Or still about the send of fast-close?

Oh, no, I jumped off the fastclose train here. This is only wrt. MP_TCPRST. Another approach would be to add the *send side* for tcprst first, perhaps it helps with tcpdump deciphering.

Good idea yes. MP_TCPRST are not mandatory but can be useful to understand what is going on.

Quoted 2 lines

But for RX there is really no point when the kernel can't pass that information along to a consumer such as mptcpd.

Indeed. But if it is easier for the development, we can also just print in debug the received MP_TCPRST. This can be parked in the export branch even if I think it is OK to send them to netdev.

Quoted 10 lines
Quoted 9 lines

If it is about fast-close, why netlink + mptcpd needs to be aware first? Could we eventually send fast-close when the userspace app (here using packetdrill format) does this:

     setsockopt(<fd>, SOL_SOCKET, SO_LINGER, {onoff=1, linger=0}, 8);
     close(<fd>);

(if we accept this socket option)
No rush of course, just in case these patches were already written :)

Quoted 7 lines

Yes, I think it might make sense to allow userspace to generate a fastclose at will.

The last iteration I sent autogenerated a fastclose for the 'unread data in socket queue when userspace called close' case, which is analogous to what the linux TCP stack does when you close with unread data in the local socket.

Yes, that's not the first case I was thinking when I was thinking about sending FAST_CLOSE but it is a good idea to support that!

Cheers,
Matt

Signature · 2 lines
Tessares | Belgium | Hybrid Access Solutions
www.tessares.net