mptcp: add fastclose option
Subsystems (3)
Florian Westphal
cover mptcp: add fastclose option
This series adds receive side support for fastclose processing. First patch is needed to avoid UAF after enabling option processing for tcp resets. Second patch calls mptcp option parsing also for reset patckets so fastclose-on-tcp-reset is handled.
Last patch adds fastclose handling.
I've submitted a pull request vs. packetdrill to add test cases for
this: https://github.com/multipath-tcp/packetdrill/pull/25
Those are the three test cases i posted earlier to this list plus a 4th test to also cover the 'two subflows closed via fastclose+reset' case.
I would send the fastclose tx side next.
I plan to hold the MPTCP reset patches back for the time being until netlink + mptcpd are aware of its existence.
include/net/tcp.h | 2 +- net/ipv4/tcp_input.c | 13 ++++++++----- net/ipv4/tcp_minisocks.c | 2 +- net/mptcp/options.c | 15 +++++++++++++++ net/mptcp/protocol.c | 33 +++++++++++++++++++++++++++++++++ net/mptcp/protocol.h | 4 ++++ net/mptcp/subflow.c | 7 ++++++- 7 files changed, 68 insertions(+), 8 deletions(-)
Florian Westphal
PATCH 1/3 mptcp: hold mptcp socket before calling tcp_done
When processing options from tcp reset path its possible that tcp_done(ssk) drops the last reference on the mptcp socket which results in use-after-free.
--- net/mptcp/subflow.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
net/mptcp/subflow.c +6 -1
@@ -276,12 +276,17 @@ void mptcp_subflow_reset(struct sock *ssk)struct mptcp_subflow_context *subflow =mptcp_subflow_ctx (ssk);struct sock *sk = subflow->conn; +/* must hold: tcp_done() could drop last reference on parent */ +sock_hold (sk); +tcp_set_state (ssk,TCP_CLOSE );tcp_send_active_reset (ssk,GFP_ATOMIC );tcp_done (ssk);if (!test_and_set_bit (MPTCP_WORK_CLOSE_SUBFLOW , &mptcp_sk (sk)->flags) &&schedule_work (&mptcp_sk (sk)->work)) -sock_hold (sk); +return ;/* worker will put sk for us */ + +sock_put (sk); }static void subflow_finish_connect (struct sock *sk,const struct sk_buff *skb)
Signature · 1 line
2.26.2
Florian Westphal
PATCH 2/3 tcp: parse mptcp options contained in reset packets
In reply to Florian WestphalBecause TCP-level resets only affect the subflow, there is a MPTCP option to indicate that the MPTCP-level connection should be closed immediately without a mptcp-level fin exchange.
This is the 'MPTCP fast close option'. It can be carried on ack segments or TCP resets. In the latter case, its needed to parse mptcp options also for reset packets so that MPTCP can act accordingly.
Next patch will add receive side fastclose support in MPTCP.
--- include/net/tcp.h | 2 +- net/ipv4/tcp_input.c | 13 ++++++++----- net/ipv4/tcp_minisocks.c | 2 +- 3 files changed, 10 insertions(+), 7 deletions(-)
include/net/tcp.h +1 -1
@@ -610,7 +610,7 @@ void tcp_skb_collapse_tstamp(struct sk_buff *skb,/* tcp_input.c */ void tcp_rearm_rto (struct sock *sk);void tcp_synack_rtt_meas (struct sock *sk,struct request_sock *req); -void tcp_reset (struct sock *sk); +void tcp_reset (struct sock *sk,struct sk_buff *skb);void tcp_skb_mark_lost_uncond_verify (struct tcp_sock *tp,struct sk_buff *skb);void tcp_fin (struct sock *sk);
net/ipv4/tcp_input.c +8 -5
@@ -4217,10 +4217,13 @@ static inline bool tcp_sequence(const struct tcp_sock *tp, u32 seq, u32 end_seq) }/* When we get a reset we do this. */ -void tcp_reset (struct sock *sk) +void tcp_reset (struct sock *sk,struct sk_buff *skb) {trace_tcp_receive_reset (sk); +if (sk_is_mptcp (sk)) +mptcp_incoming_options (sk, skb); +/* We want the right error as BSD sees it (and indeed as we do). */ switch (sk->sk_state) {case TCP_SYN_SENT :
@@ -5603,7 +5606,7 @@ static bool tcp_validate_incoming(struct sock *sk, struct sk_buff *skb, &tp->last_oow_ack_time))tcp_send_dupack (sk, skb); }else if (tcp_reset_check (sk, skb)) { -tcp_reset (sk); +tcp_reset (sk, skb); } goto discard; }
@@ -5639,7 +5642,7 @@ static bool tcp_validate_incoming(struct sock *sk, struct sk_buff *skb, }if (rst_seq_match) -tcp_reset (sk); +tcp_reset (sk, skb);else {/* Disable TFO if RST is out-of-order * and no data has been received
@@ -6076,7 +6079,7 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,*/ if (th->rst) { -tcp_reset (sk); +tcp_reset (sk, skb); goto discard; }
@@ -6518,7 +6521,7 @@ int tcp_rcv_state_process(struct sock *sk, struct sk_buff *skb)if (TCP_SKB_CB (skb)->end_seq !=TCP_SKB_CB (skb)->seq &&after (TCP_SKB_CB (skb)->end_seq - th->fin, tp->rcv_nxt)) {NET_INC_STATS (sock_net (sk),LINUX_MIB_TCPABORTONDATA ); -tcp_reset (sk); +tcp_reset (sk, skb);return 1 ; } }
net/ipv4/tcp_minisocks.c +1 -1
@@ -801,7 +801,7 @@ struct sock *tcp_check_req(struct sock *sk, struct sk_buff *skb, req->rsk_ops->send_reset (sk, skb); }else if (fastopen) {/* received a valid RST pkt */ reqsk_fastopen_remove (sk, req, true); -tcp_reset (sk); +tcp_reset (sk, skb); }if (!fastopen) {inet_csk_reqsk_queue_drop (sk, req);
Signature · 1 line
2.26.2
Florian Westphal
PATCH 3/3 mptcp: parse and act on incoming FASTCLOSE option
In reply to Florian Westphalparse the MPTCP FASTCLOSE subtype.
If provided key matches the local one, schedule the work queue to close (with tcp reset) all subflows.
The MPTCP socket moves to closed state immediately.
--- net/mptcp/options.c | 15 +++++++++++++++ net/mptcp/protocol.c | 33 +++++++++++++++++++++++++++++++++ net/mptcp/protocol.h | 4 ++++ 3 files changed, 52 insertions(+)
net/mptcp/options.c +15 -0
@@ -281,7 +281,15 @@ static void mptcp_parse_option(const struct sk_buff *skb, mp_opt->rm_id = *ptr++;pr_debug ("RM_ADDR: id=%d" , mp_opt->rm_id);break ; +case MPTCPOPT_MP_FASTCLOSE : +if (opsize !=TCPOLEN_MPTCP_FASTCLOSE ) +break ; + ptr +=2 ; + mp_opt->rcvr_key =get_unaligned_be64 (ptr); + ptr +=8 ; + mp_opt->fastclose =1 ; +break ;default :break ; }
@@ -299,6 +307,7 @@ void mptcp_get_options(const struct sk_buff *skb, mp_opt->mp_join =0 ; mp_opt->add_addr =0 ; mp_opt->ahmac =0 ; + mp_opt->fastclose =0 ; mp_opt->port =0 ; mp_opt->rm_addr =0 ; mp_opt->dss =0 ;
@@ -937,6 +946,12 @@ void mptcp_incoming_options(struct sock *sk, struct sk_buff *skb)if (!check_fully_established (msk, sk, subflow, skb, &mp_opt))return ; +if (mp_opt.fastclose && + msk->local_key == mp_opt.rcvr_key) { +WRITE_ONCE (msk->rcv_fastclose, true); +mptcp_schedule_work ((struct sock *)msk); + } +if (mp_opt.add_addr &&add_addr_hmac_valid (msk, &mp_opt)) {struct mptcp_addr_info addr;
net/mptcp/protocol.c +33 -0
@@ -2209,6 +2209,36 @@ static bool mptcp_check_close_timeout(const struct sock *sk)return true; } +static void mptcp_check_fastclose (struct mptcp_sock *msk) +{ +struct mptcp_subflow_context *subflow, *tmp; +struct sock *sk = &msk->sk.icsk_inet.sk; + +if (likely (!READ_ONCE (msk->rcv_fastclose))) +return ; + +mptcp_token_destroy (msk); + +list_for_each_entry_safe (subflow, tmp, &msk->conn_list, node) { +struct sock *tcp_sk =mptcp_subflow_tcp_sock (subflow); + +lock_sock (tcp_sk); +if (tcp_sk->sk_state !=TCP_CLOSE ) { +tcp_send_active_reset (tcp_sk,GFP_ATOMIC ); +tcp_set_state (tcp_sk,TCP_CLOSE ); + } +release_sock (tcp_sk); + } + +inet_sk_state_store (sk,TCP_CLOSE ); + sk->sk_shutdown =SHUTDOWN_MASK ; +smp_mb__before_atomic ();/* SHUTDOWN must be visible first */ +set_bit (MPTCP_DATA_READY , &msk->flags); +set_bit (MPTCP_WORK_CLOSE_SUBFLOW , &msk->flags); + +mptcp_close_wake_up (sk); +} +static void mptcp_worker (struct work_struct *work) { struct mptcp_sock *msk = container_of(work, struct mptcp_sock, work);
@@ -2225,6 +2255,9 @@ static void mptcp_worker(struct work_struct *work)mptcp_check_data_fin_ack (sk);__mptcp_flush_join_list (msk); + +mptcp_check_fastclose (msk); +if (test_and_clear_bit (MPTCP_WORK_CLOSE_SUBFLOW , &msk->flags))__mptcp_close_subflow (msk);
net/mptcp/protocol.h +4 -0
@@ -23,6 +23,7 @@#define OPTION_MPTCP_ADD_ADDR BIT(6)#define OPTION_MPTCP_ADD_ADDR6 BIT(7)#define OPTION_MPTCP_RM_ADDR BIT(8) +#define OPTION_MPTCP_FASTCLOSE BIT(9)/* MPTCP option subtypes */ #define MPTCPOPT_MP_CAPABLE 0
@@ -59,6 +60,7 @@#define TCPOLEN_MPTCP_PORT_LEN 4#define TCPOLEN_MPTCP_ADD_ADDR_HMAC 8#define TCPOLEN_MPTCP_RM_ADDR_BASE 4 +#define TCPOLEN_MPTCP_FASTCLOSE 12/* MPTCP MP_JOIN flags */ #define MPTCPOPT_BACKUP BIT(0)
@@ -111,6 +113,7 @@ struct mptcp_options_received {u16 data_len; u16 mp_capable : 1, mp_join : 1, + fastclose : 1, dss : 1, add_addr : 1, rm_addr : 1,
@@ -237,6 +240,7 @@ struct mptcp_sock {bool fully_established;bool rcv_data_fin;bool snd_data_fin_enable; +bool rcv_fastclose;bool use_64bit_ack;/* Set when we received a 64-bit DSN */ spinlock_t join_list_lock;struct sock *ack_hint;
Signature · 1 line
2.26.2
Matthieu Baerts
Hi Florian,
Quoted 7 lines · Florian Westphal
This series adds receive side support for fastclose processing. First patch is needed to avoid UAF after enabling option processing for tcp resets. Second patch calls mptcp option parsing also for reset patckets so fastclose-on-tcp-reset is handled.
Last patch adds fastclose handling.
Quoted 7 lines
I've submitted a pull request vs. packetdrill to add test cases for
this: https://github.com/multipath-tcp/packetdrill/pull/25Those are the three test cases i posted earlier to this list plus a 4th test to also cover the 'two subflows closed via fastclose+reset' case.
I would send the fastclose tx side next.
Thank you for these patches, they are very good!
Just added at the end of our tree with my Reviewed-by tag (there were other reviewers on the previous patches, I can of course add more tags if the new version is OK for others):
- 6766eaff5c84: mptcp: hold mptcp socket before calling tcp_done
- 37a53605cd6b: tcp: parse mptcp options contained in reset packets
- 0b9d6f3f5a2b: mptcp: parse and act on incoming FASTCLOSE option
- Results: 9e11c1d2b8db..ba6c5ecb4ea5
Tests + export are in progress!
Quoted 2 lines
I plan to hold the MPTCP reset patches back for the time being until netlink + mptcpd are aware of its existence.
I certainly missed something here but are you here talking about MP_TCPRST support? Or still about the send of fast-close?
If it is about fast-close, why netlink + mptcpd needs to be aware first? Could we eventually send fast-close when the userspace app (here using packetdrill format) does this:
setsockopt(<fd>, SOL_SOCKET, SO_LINGER, {onoff=1, linger=0}, 8);
close(<fd>);(if we accept this socket option)
No rush of course, just in case these patches were already written :)
Cheers,
Matt
Signature · 2 lines
Tessares | Belgium | Hybrid Access Solutions www.tessares.net
Florian Westphal
Quoted 6 lines · Matthieu Baerts
Quoted 2 lines · Florian Westphal
I plan to hold the MPTCP reset patches back for the time being until netlink + mptcpd are aware of its existence.
I certainly missed something here but are you here talking about
MP_TCPRSTsupport? Or still about the send of fast-close?
Oh, no, I jumped off the fastclose train here. This is only wrt. MP_TCPRST. Another approach would be to add the *send side* for tcprst first, perhaps it helps with tcpdump deciphering.
But for RX there is really no point when the kernel can't pass that information along to a consumer such as mptcpd.
Quoted 9 lines
If it is about fast-close, why netlink + mptcpd needs to be aware first? Could we eventually send fast-close when the userspace app (here using packetdrill format) does this:
setsockopt(<fd>, SOL_SOCKET, SO_LINGER, {onoff=1, linger=0}, 8); close(<fd>);(if we accept this socket option)
No rush of course, just in case these patches were already written :)
Yes, I think it might make sense to allow userspace to generate a fastclose at will.
The last iteration I sent autogenerated a fastclose for the 'unread data in socket queue when userspace called close' case, which is analogous to what the linux TCP stack does when you close with unread data in the local socket.
But I think extending it to linger makes sense as well.
Matthieu Baerts
Hi Florian,
Quoted 11 lines · Florian Westphal
Quoted 6 lines · Matthieu Baerts
Quoted 2 lines · Florian Westphal
I plan to hold the MPTCP reset patches back for the time being until netlink + mptcpd are aware of its existence.
I certainly missed something here but are you here talking about
MP_TCPRSTsupport? Or still about the send of fast-close?Oh, no, I jumped off the fastclose train here. This is only wrt.
MP_TCPRST. Another approach would be to add the *send side* for tcprst first, perhaps it helps with tcpdump deciphering.
Good idea yes. MP_TCPRST are not mandatory but can be useful to understand what is going on.
Quoted 2 lines
But for RX there is really no point when the kernel can't pass that information along to a consumer such as mptcpd.
Indeed. But if it is easier for the development, we can also just print in debug the received MP_TCPRST. This can be parked in the export branch even if I think it is OK to send them to netdev.
Quoted 10 lines
Quoted 9 lines
If it is about fast-close, why netlink + mptcpd needs to be aware first? Could we eventually send fast-close when the userspace app (here using packetdrill format) does this:
setsockopt(<fd>, SOL_SOCKET, SO_LINGER, {onoff=1, linger=0}, 8); close(<fd>);(if we accept this socket option)
No rush of course, just in case these patches were already written :)
Quoted 7 lines
Yes, I think it might make sense to allow userspace to generate a fastclose at will.
The last iteration I sent autogenerated a fastclose for the 'unread data in socket queue when userspace called close' case, which is analogous to what the linux TCP stack does when you close with unread data in the local socket.
Yes, that's not the first case I was thinking when I was thinking about sending FAST_CLOSE but it is a good idea to support that!
Cheers,
Matt
Signature · 2 lines
Tessares | Belgium | Hybrid Access Solutions www.tessares.net