tcp: improving RACK cpu performance
Subsystems (3)
cover tcp: improving RACK cpu performance
This patch set improves the CPU consumption of the RACK TCP loss recovery algorithm, in particular for high-speed networks. Currently, for every ACK in recovery RACK can potentially iterate over all sent packets in the write queue. On large BDP networks with non-trivial losses the RACK write queue walk CPU usage becomes unreasonably high.
This patch introduces a new queue in TCP that keeps only skbs sent and not yet (s)acked or marked lost, in time order instead of sequence order. With that, RACK can examine this time-sorted list and only check packets that were sent recently, within the reordering window, per ACK. This is the fastest way without any write queue walks. The number of skbs examined per ACK is reduced by orders of magnitude.
Eric Dumazet (1):
tcp: new list for sent but unacked skbs for RACK recovery
Yuchung Cheng (2):
tcp: more efficient RACK loss detection
tcp: a small refactor of RACK loss detection
include/linux/skbuff.h | 11 ++++++++-- include/linux/tcp.h | 1 + include/net/tcp.h | 24 +++++++++++++++++++++- net/ipv4/tcp.c | 2 ++ net/ipv4/tcp_input.c | 9 +++++++-- net/ipv4/tcp_minisocks.c | 1 + net/ipv4/tcp_output.c | 42 ++++++++++++++++++++++++++++---------- net/ipv4/tcp_recovery.c | 52 ++++++++++++++++++------------------------------ 8 files changed, 93 insertions(+), 49 deletions(-)
Signature · 1 line
2.14.2.920.gcf0c67979c-goog
PATCH 1/3 tcp: new list for sent but unacked skbs for RACK recovery
From: Eric Dumazet <[email protected]>
This patch adds a new queue (list) that tracks the sent but not yet acked or SACKed skbs for a TCP connection. The list is chronologically ordered by skb->skb_mstamp (the head is the oldest sent skb).
This list will be used to optimize TCP Rack recovery, which checks an skb's timestamp to judge if it has been lost and needs to be retransmitted. Since TCP write queue is ordered by sequence instead of sent time, RACK has to scan over the write queue to catch all eligible packets to detect lost retransmission, and iterates through SACKed skbs repeatedly.
Special cares for rare events:
1. TCP repair fakes skb transmission so the send queue needs adjusted
2. SACK reneging would require re-inserting SACKed skbs into the send queue. For now I believe it's not worth the complexity to make RACK work perfectly on SACK reneging, so we do nothing here.
3. Fast Open: currently for non-TFO, send-queue correctly queues the pure SYN packet. For TFO which queues a pure SYN and then a data packet, send-queue only queues the data packet but not the pure SYN due to the structure of TFO code. This is okay because the SYN receiver would never respond with a SACK on a missing SYN (i.e. SYN is never fast-retransmitted by SACK/RACK).
In order to not grow sk_buff, we use an union for the new list and _skb_refdst/destructor fields. This is a bit complicated because we need to make sure _skb_refdst and destructor are properly zeroed before skb is cloned/copied at transmit, and before being freed.
--- include/linux/skbuff.h | 11 +++++++++-- include/linux/tcp.h | 1 + include/net/tcp.h | 24 +++++++++++++++++++++++- net/ipv4/tcp.c | 2 ++ net/ipv4/tcp_input.c | 9 +++++++-- net/ipv4/tcp_minisocks.c | 1 + net/ipv4/tcp_output.c | 42 +++++++++++++++++++++++++++++++----------- 7 files changed, 74 insertions(+), 16 deletions(-)
include/linux/skbuff.h +9 -2
@@ -617,6 +617,7 @@ typedef unsigned char *sk_buff_data_t;* @nf_trace: netfilter packet trace flag * @protocol: Packet protocol from driver * @destructor: Destruct function +* @tcp_tsorted_anchor: list structure for TCP (tp->tsorted_sent_queue) * @_nfct: Associated connection, if any (with nfctinfo bits) * @nf_bridge: Saved data about a bridged frame - see br_netfilter.c * @skb_iif: ifindex of device we arrived on
@@ -686,8 +687,14 @@ struct sk_buff {*/ char cb[48 ]__aligned(8) ; -unsigned long _skb_refdst; -void (*destructor)(struct sk_buff *skb); +union { +struct { +unsigned long _skb_refdst; +void (*destructor)(struct sk_buff *skb); + }; +struct list_head tcp_tsorted_anchor; + }; +#ifdef CONFIG_XFRM struct sec_path *sp;#endif
include/linux/tcp.h +1 -0
@@ -191,6 +191,7 @@ struct tcp_sock {u32 tsoffset;/* timestamp offset */ struct list_head tsq_node;/* anchor in tsq_tasklet.head list */ +struct list_head tsorted_sent_queue;/* time-sorted sent but un-SACKed skbs */ u32 snd_wl1;/* Sequence for window update */ u32 snd_wnd;/* The window we expect to receive */
include/net/tcp.h +23 -1
@@ -1588,14 +1588,34 @@ enum tcp_chrono {void tcp_chrono_start (struct sock *sk,const enum tcp_chrono type);void tcp_chrono_stop (struct sock *sk,const enum tcp_chrono type); +/* This helper is needed, because skb->tcp_tsorted_anchor uses +* the same memory storage than skb->destructor/_skb_refdst +*/ +static inline void tcp_skb_tsorted_anchor_cleanup (struct sk_buff *skb) +{ + skb->destructor =NULL ; + skb->_skb_refdst =0UL ; +} + +#define tcp_skb_tsorted_save (skb) { \ + unsigned long _save = skb->_skb_refdst; \ + skb->_skb_refdst = 0UL; + +#define tcp_skb_tsorted_restore (skb) \ + skb->_skb_refdst = _save; \ +} +/* write queue abstraction */ static inline void tcp_write_queue_purge (struct sock *sk) {struct sk_buff *skb;tcp_chrono_stop (sk,TCP_CHRONO_BUSY ); -while ((skb =__skb_dequeue (&sk->sk_write_queue)) !=NULL ) +while ((skb =__skb_dequeue (&sk->sk_write_queue)) !=NULL ) { +tcp_skb_tsorted_anchor_cleanup (skb);sk_wmem_free_skb (sk, skb); + } +INIT_LIST_HEAD (&tcp_sk (sk)->tsorted_sent_queue);sk_mem_reclaim (sk);tcp_clear_all_retrans_hints (tcp_sk (sk)); }
@@ -1710,6 +1730,8 @@ static inline void tcp_insert_write_queue_before(struct sk_buff *new,static inline void tcp_unlink_write_queue (struct sk_buff *skb,struct sock *sk) { +list_del (&skb->tcp_tsorted_anchor); +tcp_skb_tsorted_anchor_cleanup (skb);__skb_unlink (skb, &sk->sk_write_queue); }
net/ipv4/tcp.c +2 -0
@@ -415,6 +415,7 @@ void tcp_init_sock(struct sock *sk) tp->out_of_order_queue =RB_ROOT ;tcp_init_xmit_timers (sk);INIT_LIST_HEAD (&tp->tsq_node); +INIT_LIST_HEAD (&tp->tsorted_sent_queue); icsk->icsk_rto =TCP_TIMEOUT_INIT ; tp->mdev_us =jiffies_to_usecs (TCP_TIMEOUT_INIT );
@@ -869,6 +870,7 @@ struct sk_buff *sk_stream_alloc_skb(struct sock *sk, int size, gfp_t gfp,* available to the caller, no more, no less. */ skb->reserved_tailroom = skb->end - skb->tail - size; +INIT_LIST_HEAD (&skb->tcp_tsorted_anchor);return skb; }__kfree_skb (skb);
net/ipv4/tcp_input.c +7 -2
@@ -1593,6 +1593,8 @@ static struct sk_buff *tcp_sacktag_walk(struct sk_buff *skb, struct sock *sk,tcp_skb_pcount (skb), skb->skb_mstamp);tcp_rate_skb_delivered (sk, skb, state->rate); +if (TCP_SKB_CB (skb)->sacked &TCPCB_SACKED_ACKED ) +list_del_init (&skb->tcp_tsorted_anchor);if (!before (TCP_SKB_CB (skb)->seq,tcp_highest_sack_seq (tp)))
@@ -3054,8 +3056,11 @@ static void tcp_ack_tstamp(struct sock *sk, struct sk_buff *skb, shinfo =skb_shinfo (skb);if (!before (shinfo->tskey, prior_snd_una) && -before (shinfo->tskey,tcp_sk (sk)->snd_una)) -__skb_tstamp_tx (skb,NULL , sk,SCM_TSTAMP_ACK ); +before (shinfo->tskey,tcp_sk (sk)->snd_una)) { +tcp_skb_tsorted_save (skb) { +__skb_tstamp_tx (skb,NULL , sk,SCM_TSTAMP_ACK ); + }tcp_skb_tsorted_restore (skb); + } }/* Remove acknowledged frames from the retransmission queue. If our packet
net/ipv4/tcp_minisocks.c +1 -0
@@ -446,6 +446,7 @@ struct sock *tcp_create_openreq_child(const struct sock *sk, newtp->snd_nxt = newtp->snd_up = treq->snt_isn +1 ;INIT_LIST_HEAD (&newtp->tsq_node); +INIT_LIST_HEAD (&newtp->tsorted_sent_queue);tcp_init_wl (newtp, treq->rcv_isn);
net/ipv4/tcp_output.c +31 -11
93 lines not on this page · raw message
Signature · 1 line
2.14.2.920.gcf0c67979c-goog
PATCH 2/3 tcp: more efficient RACK loss detection
In reply to Yuchung ChengUse the new time-ordered list to speed up RACK. The detection logic is identical. But since the list is chronologically ordered by skb_mstamp and contains only skbs not yet acked or sacked, RACK can abort the loop upon hitting skbs that were sent more recently. On YouTube servers this patch reduces the iterations on write queue by 40x. The improvement is even bigger with large BDP networks.
--- net/ipv4/tcp_recovery.c | 20 +++++--------------- 1 file changed, 5 insertions(+), 15 deletions(-)
net/ipv4/tcp_recovery.c +5 -15
@@ -45,7 +45,7 @@ static bool tcp_rack_sent_after(u64 t1, u64 t2, u32 seq1, u32 seq2)static void tcp_rack_detect_loss (struct sock *sk,u32 *reo_timeout) {struct tcp_sock *tp =tcp_sk (sk); -struct sk_buff *skb; +struct sk_buff *skb, *n;u32 reo_wnd; *reo_timeout =0 ;
@@ -58,17 +58,10 @@ static void tcp_rack_detect_loss(struct sock *sk, u32 *reo_timeout)if ((tp->rack.reord || !tp->lost_out) &&tcp_min_rtt (tp) != ~0U ) reo_wnd =max (tcp_min_rtt (tp) >>2 , reo_wnd); - tcp_for_write_queue(skb, sk) { +list_for_each_entry_safe (skb, n, &tp->tsorted_sent_queue, + tcp_tsorted_anchor) {struct tcp_skb_cb *scb =TCP_SKB_CB (skb); -if (skb ==tcp_send_head (sk)) -break ; - -/* Skip ones already (s)acked */ -if (!after (scb->end_seq, tp->snd_una) || - scb->sacked &TCPCB_SACKED_ACKED ) -continue ; -if (tcp_rack_sent_after (tp->rack.mstamp, skb->skb_mstamp, tp->rack.end_seq, scb->end_seq)) {/* Step 3 in draft-cheng-tcpm-rack-00.txt:
@@ -81,6 +74,7 @@ static void tcp_rack_detect_loss(struct sock *sk, u32 *reo_timeout)if (remaining <0 ) {tcp_rack_mark_skb_lost (sk, skb); +list_del_init (&skb->tcp_tsorted_anchor);continue ; }
@@ -91,11 +85,7 @@ static void tcp_rack_detect_loss(struct sock *sk, u32 *reo_timeout)/* Record maximum wait time (+1 to avoid 0) */ *reo_timeout =max_t (u32, *reo_timeout,1 + remaining); - - }else if (!(scb->sacked &TCPCB_RETRANS )) { -/* Original data are sent sequentially so stop early -* b/c the rest are all sent after rack_sent -*/ + }else {break ; } }
Signature · 1 line
2.14.2.920.gcf0c67979c-goog
PATCH 3/3 tcp: a small refactor of RACK loss detection
In reply to Yuchung ChengRefactor the RACK loop to improve readability and speed up the checks.
--- net/ipv4/tcp_recovery.c | 40 ++++++++++++++++++---------------------- 1 file changed, 18 insertions(+), 22 deletions(-)
net/ipv4/tcp_recovery.c +18 -22
@@ -61,32 +61,28 @@ static void tcp_rack_detect_loss(struct sock *sk, u32 *reo_timeout)list_for_each_entry_safe (skb, n, &tp->tsorted_sent_queue, tcp_tsorted_anchor) {struct tcp_skb_cb *scb =TCP_SKB_CB (skb); +s32 remaining; -if (tcp_rack_sent_after (tp->rack.mstamp, skb->skb_mstamp, - tp->rack.end_seq, scb->end_seq)) { -/* Step 3 in draft-cheng-tcpm-rack-00.txt: -* A packet is lost if its elapsed time is beyond -* the recent RTT plus the reordering window. -*/ -u32 elapsed =tcp_stamp_us_delta (tp->tcp_mstamp, - skb->skb_mstamp); -s32 remaining = tp->rack.rtt_us + reo_wnd - elapsed; - -if (remaining <0 ) { -tcp_rack_mark_skb_lost (sk, skb); -list_del_init (&skb->tcp_tsorted_anchor); -continue ; - } - -/* Skip ones marked lost but not yet retransmitted */ -if ((scb->sacked &TCPCB_LOST ) && - !(scb->sacked &TCPCB_SACKED_RETRANS )) -continue ; +/* Skip ones marked lost but not yet retransmitted */ +if ((scb->sacked &TCPCB_LOST ) && + !(scb->sacked &TCPCB_SACKED_RETRANS )) +continue ; +if (!tcp_rack_sent_after (tp->rack.mstamp, skb->skb_mstamp, + tp->rack.end_seq, scb->end_seq)) +break ; + +/* A packet is lost if it has not been s/acked beyond +* the recent RTT plus the reordering window. +*/ + remaining = tp->rack.rtt_us + reo_wnd - +tcp_stamp_us_delta (tp->tcp_mstamp, skb->skb_mstamp); +if (remaining <0 ) { +tcp_rack_mark_skb_lost (sk, skb); +list_del_init (&skb->tcp_tsorted_anchor); + }else {/* Record maximum wait time (+1 to avoid 0) */ *reo_timeout =max_t (u32, *reo_timeout,1 + remaining); - }else { -break ; } } }
Signature · 1 line
2.14.2.920.gcf0c67979c-goog
David Miller
maintainerFrom: Yuchung Cheng <[email protected]>
Date: Wed, 4 Oct 2017 12:59:57 -0700
Quoted 6 lines
This patch set improves the CPU consumption of the RACK TCP loss recovery algorithm, in particular for high-speed networks. Currently, for every ACK in recovery RACK can potentially iterate over all sent packets in the write queue. On large BDP networks with non-trivial losses the RACK write queue walk CPU usage becomes unreasonably high.
Quoted 6 lines
This patch introduces a new queue in TCP that keeps only skbs sent and not yet (s)acked or marked lost, in time order instead of sequence order. With that, RACK can examine this time-sorted list and only check packets that were sent recently, within the reordering window, per ACK. This is the fastest way without any write queue walks. The number of skbs examined per ACK is reduced by orders of magnitude.
That's a pretty risky way to implement the second SKB list.... but you avoided making sk_buff larger so what can I say :-)
Series applied, thank.
Quoted 10 lines · David Miller
From: Yuchung Cheng <[email protected]>
Date: Wed, 4 Oct 2017 12:59:57 -0700Quoted 6 lines
This patch set improves the CPU consumption of the RACK TCP loss recovery algorithm, in particular for high-speed networks. Currently, for every ACK in recovery RACK can potentially iterate over all sent packets in the write queue. On large BDP networks with non-trivial losses the RACK write queue walk CPU usage becomes unreasonably high.
Quoted 8 lines
Quoted 6 lines
This patch introduces a new queue in TCP that keeps only skbs sent and not yet (s)acked or marked lost, in time order instead of sequence order. With that, RACK can examine this time-sorted list and only check packets that were sent recently, within the reordering window, per ACK. This is the fastest way without any write queue walks. The number of skbs examined per ACK is reduced by orders of magnitude.
That's a pretty risky way to implement the second SKB list.... but
Agreed. I really appreciate you accepting this change. We tried a few alternatives but this is the most clean and fastest approach. BBR can reach 8-10Gbps on 300ms RTT link so both RACK and SACK need to scale in the era of hundreds MB BDP :-)
Quoted 3 lines
you avoided making
sk_bufflarger so what can I say :-)Series applied, thank.