tcp: Add TCP_INFO counter for packets received out-of-order
Subsystems (3)
PATCH v5 1/2 tcp: Add TCP_INFO counter for packets received out-of-order
For receive-heavy cases on the server-side, we want to track the connection quality for individual client IPs. This counter, similar to the existing system-wide TCPOFOQueue counter in /proc/net/netstat, tracks out-of-order packet reception. By providing this counter in TCP_INFO, it will allow understanding to what degree receive-heavy sockets are experiencing out-of-order delivery and packet drops indicating congestion.
Please note that this is similar to the counter in NetBSD TCP_INFO, and has the same name.
Also note that we avoid increasing the size of the tcp_sock struct by taking advantage of a hole.
---
changes since v4:
- optimize placement of rcv_ooopack to avoid increasing tcp_sock struct size
include/linux/tcp.h | 2 ++ include/uapi/linux/tcp.h | 2 ++ net/ipv4/tcp.c | 2 ++ net/ipv4/tcp_input.c | 1 + 4 files changed, 7 insertions(+)
include/linux/tcp.h +2 -0
@@ -354,6 +354,8 @@ struct tcp_sock {#define BPF_SOCK_OPS_TEST_FLAG (TP ,ARG ) 0#endif +u32 rcv_ooopack;/* Received out-of-order packets, for tcpinfo */ +/* Receiver side RTT estimation */ u32 rcv_rtt_last_tsecr;struct {
include/uapi/linux/tcp.h +2 -0
@@ -270,6 +270,8 @@ struct tcp_info {__u64 tcpi_bytes_retrans;/* RFC4898 tcpEStatsPerfOctetsRetrans */ __u32 tcpi_dsack_dups;/* RFC4898 tcpEStatsStackDSACKDups */ __u32 tcpi_reord_seen;/* reordering events seen */ + +__u32 tcpi_rcv_ooopack;/* Out-of-order packets received */ };/* netlink attributes types for SCM_TIMESTAMPING_OPT_STATS */
net/ipv4/tcp.c +2 -0
@@ -2653,6 +2653,7 @@ int tcp_disconnect(struct sock *sk, int flags) tp->rx_opt.saw_tstamp =0 ; tp->rx_opt.dsack =0 ; tp->rx_opt.num_sacks =0 ; + tp->rcv_ooopack =0 ;/* Clean up fastopen related fields */
@@ -3295,6 +3296,7 @@ void tcp_get_info(struct sock *sk, struct tcp_info *info) info->tcpi_bytes_retrans = tp->bytes_retrans; info->tcpi_dsack_dups = tp->dsack_dups; info->tcpi_reord_seen = tp->reord_seen; + info->tcpi_rcv_ooopack = tp->rcv_ooopack;unlock_sock_fast (sk, slow); }EXPORT_SYMBOL_GPL (tcp_get_info);
net/ipv4/tcp_input.c +1 -0
@@ -4555,6 +4555,7 @@ static void tcp_data_queue_ofo(struct sock *sk, struct sk_buff *skb) tp->pred_flags =0 ;inet_csk_schedule_ack (sk); + tp->rcv_ooopack +=max_t (u16,1 ,skb_shinfo (skb)->gso_segs);NET_INC_STATS (sock_net (sk),LINUX_MIB_TCPOFOQUEUE ); seq =TCP_SKB_CB (skb)->seq; end_seq =TCP_SKB_CB (skb)->end_seq;
Signature · 1 line
2.17.1
PATCH v5 2/2 tcp: Add snd_wnd to TCP_INFO
Neal Cardwell mentioned that snd_wnd would be useful for diagnosing TCP performance problems --
Quoted 6 lines
(1) Usually when we're diagnosing TCP performance problems, we do so from the sender, since the sender makes most of the performance-critical decisions (cwnd, pacing, TSO size, TSQ, etc). From the sender-side the thing that would be most useful is to see
tp->snd_wnd, the receive window that the receiver has advertised to the sender.
This serves the purpose of adding an additional __u32 to avoid the would-be hole caused by the addition of the tcpi_rcvi_ooopack field.
---
changes since v4:
- clarify comment
include/uapi/linux/tcp.h | 4 ++++ net/ipv4/tcp.c | 1 + 2 files changed, 5 insertions(+)
include/uapi/linux/tcp.h +4 -0
@@ -272,6 +272,10 @@ struct tcp_info {__u32 tcpi_reord_seen;/* reordering events seen */ __u32 tcpi_rcv_ooopack;/* Out-of-order packets received */ + +__u32 tcpi_snd_wnd;/* peer's advertised receive window after +* scaling (bytes) +*/ };/* netlink attributes types for SCM_TIMESTAMPING_OPT_STATS */
net/ipv4/tcp.c +1 -0
@@ -3297,6 +3297,7 @@ void tcp_get_info(struct sock *sk, struct tcp_info *info) info->tcpi_dsack_dups = tp->dsack_dups; info->tcpi_reord_seen = tp->reord_seen; info->tcpi_rcv_ooopack = tp->rcv_ooopack; + info->tcpi_snd_wnd = tp->snd_wnd;unlock_sock_fast (sk, slow); }EXPORT_SYMBOL_GPL (tcp_get_info);
Signature · 1 line
2.17.1
Re: [PATCH v5 2/2] tcp: Add snd_wnd to TCP_INFO
Quoted 10 lines · Thomas Higdon
Neal Cardwell mentioned that
snd_wndwould be useful for diagnosing TCP performance problems --Quoted 6 lines
(1) Usually when we're diagnosing TCP performance problems, we do so from the sender, since the sender makes most of the performance-critical decisions (cwnd, pacing, TSO size, TSQ, etc). From the sender-side the thing that would be most useful is to see
tp->snd_wnd, the receive window that the receiver has advertised to the sender.
Quoted 5 lines
This serves the purpose of adding an additional
__u32to avoid the would-be hole caused by the addition of thetcpi_rcvi_ooopackfield.Signed-off-by Thomas Higdon <[email protected]>---
Quoted 36 lines
changes since v4:
- clarify commentinclude/uapi/linux/tcp.h | 4 ++++ net/ipv4/tcp.c | 1 + 2 files changed, 5 insertions(+)include/uapi/linux/tcp.h +4
-0@@ -272,6 +272,10 @@ struct tcp_info {__u32 tcpi_reord_seen;/* reordering events seen */ __u32 tcpi_rcv_ooopack;/* Out-of-order packets received */ + +__u32 tcpi_snd_wnd;/* peer's advertised receive window after +* scaling (bytes) +*/ };/* netlink attributes types for SCM_TIMESTAMPING_OPT_STATS */ net/ipv4/tcp.c +1
-1@@ -3297,6 +3297,7 @@ void tcp_get_info(struct sock *sk, struct tcp_info *info) info->tcpi_dsack_dups = tp->dsack_dups; info->tcpi_reord_seen = tp->reord_seen; info->tcpi_rcv_ooopack = tp->rcv_ooopack; + info->tcpi_snd_wnd = tp->snd_wnd;unlock_sock_fast (sk, slow); }EXPORT_SYMBOL_GPL (tcp_get_info); --2.17.1
Re: [PATCH v5 2/2] tcp: Add snd_wnd to TCP_INFO
In reply to Thomas HigdonQuoted 10 lines · Thomas Higdon
Neal Cardwell mentioned that
snd_wndwould be useful for diagnosing TCP performance problems --Quoted 6 lines
(1) Usually when we're diagnosing TCP performance problems, we do so from the sender, since the sender makes most of the performance-critical decisions (cwnd, pacing, TSO size, TSQ, etc). From the sender-side the thing that would be most useful is to see
tp->snd_wnd, the receive window that the receiver has advertised to the sender.
Quoted 7 lines
This serves the purpose of adding an additional
__u32to avoid the would-be hole caused by the addition of thetcpi_rcvi_ooopackfield.Signed-off-by Thomas Higdon <[email protected]>---
changes since v4:
- clarify comment
Thanks!
neal
Soheil Hassas Yeganeh
Acked-byRe: [PATCH v5 2/2] tcp: Add snd_wnd to TCP_INFO
Quoted 15 lines · Neal Cardwell
Quoted 13 lines · Thomas Higdon
Neal Cardwell mentioned that
snd_wndwould be useful for diagnosing TCP performance problems --Quoted 6 lines
(1) Usually when we're diagnosing TCP performance problems, we do so from the sender, since the sender makes most of the performance-critical decisions (cwnd, pacing, TSO size, TSQ, etc). From the sender-side the thing that would be most useful is to see
tp->snd_wnd, the receive window that the receiver has advertised to the sender.This serves the purpose of adding an additional
__u32to avoid the would-be hole caused by the addition of thetcpi_rcvi_ooopackfield.
Quoted 6 lines
Quoted 4 lines
Signed-off-by Thomas Higdon <[email protected]>---
changes since v4:
- clarify commentAcked-by Neal Cardwell <[email protected]>
Thank you for adding the new field!
Re: [PATCH v5 2/2] tcp: Add snd_wnd to TCP_INFO
In reply to Thomas HigdonFrom: Thomas Higdon <[email protected]>
Date: Fri, 13 Sep 2019 23:23:35 +0000
Quoted 9 lines
Neal Cardwell mentioned that
snd_wndwould be useful for diagnosing TCP performance problems --Quoted 6 lines
(1) Usually when we're diagnosing TCP performance problems, we do so from the sender, since the sender makes most of the performance-critical decisions (cwnd, pacing, TSO size, TSQ, etc). From the sender-side the thing that would be most useful is to see
tp->snd_wnd, the receive window that the receiver has advertised to the sender.
Quoted 4 lines
This serves the purpose of adding an additional
__u32to avoid the would-be hole caused by the addition of thetcpi_rcvi_ooopackfield.Signed-off-by Thomas Higdon <[email protected]>
Applied.
Quoted 12 lines · Thomas Higdon
For receive-heavy cases on the server-side, we want to track the connection quality for individual client IPs. This counter, similar to the existing system-wide TCPOFOQueue counter in
/proc/net/netstat, tracks out-of-order packet reception. By providing this counter inTCP_INFO, it will allow understanding to what degree receive-heavy sockets are experiencing out-of-order delivery and packet drops indicating congestion.Please note that this is similar to the counter in NetBSD
TCP_INFO, and has the same name.
Quoted 8 lines
Also note that we avoid increasing the size of the
tcp_sockstruct by taking advantage of a hole.Signed-off-by Thomas Higdon <[email protected]>---
changes since v4:
- optimize placement ofrcv_ooopackto avoid increasingtcp_sockstruct size
Thanks, Thomas, for adding this!
After this is merged, would you mind sending a patch to add support to the "ss" command line tool to print these 2 new fields?
My favorite recent example of such a patch to ss is Eric's change:
https://git.kernel.org/pub/scm/network/iproute2/iproute2.git/commit/misc/ss.c?id=5eead6270a19f00464052d4084f32182cfe027ff
thanks,
neal
Quoted 25 lines · Neal Cardwell
Quoted 12 lines · Thomas Higdon
For receive-heavy cases on the server-side, we want to track the connection quality for individual client IPs. This counter, similar to the existing system-wide TCPOFOQueue counter in
/proc/net/netstat, tracks out-of-order packet reception. By providing this counter inTCP_INFO, it will allow understanding to what degree receive-heavy sockets are experiencing out-of-order delivery and packet drops indicating congestion.Please note that this is similar to the counter in NetBSD
TCP_INFO, and has the same name.Quoted 8 lines
Also note that we avoid increasing the size of the
tcp_sockstruct by taking advantage of a hole.Signed-off-by Thomas Higdon <[email protected]>---
changes since v4:
- optimize placement ofrcv_ooopackto avoid increasingtcp_sockstruct sizeAcked-by Neal Cardwell <[email protected]>
Quoted 7 lines
Thanks, Thomas, for adding this!
After this is merged, would you mind sending a patch to add support to the "ss" command line tool to print these 2 new fields?
My favorite recent example of such a patch to ss is Eric's change:
https://git.kernel.org/pub/scm/network/iproute2/iproute2.git/commit/misc/ss.c?id=5eead6270a19f00464052d4084f32182cfe027ff
Yes, and thank you for the help in getting this into a good state!
From looking at that "ss" patch, it seems like we would need to wait until iproute2-next's include/uapi/linux/tcp.h has received a merge from kernel net-next before we'd be able to apply a patch for "ss" that uses the new fields.
In the meantime, as you've asked, I will go ahead and send a patch for iproute2-next's "ss" with the assumption that these tcpinfo changes have already been merged.
From: Thomas Higdon <[email protected]>
Date: Fri, 13 Sep 2019 23:23:34 +0000
Quoted 8 lines
For receive-heavy cases on the server-side, we want to track the connection quality for individual client IPs. This counter, similar to the existing system-wide TCPOFOQueue counter in
/proc/net/netstat, tracks out-of-order packet reception. By providing this counter inTCP_INFO, it will allow understanding to what degree receive-heavy sockets are experiencing out-of-order delivery and packet drops indicating congestion.
Quoted 7 lines
Please note that this is similar to the counter in NetBSD
TCP_INFO, and has the same name.Also note that we avoid increasing the size of the
tcp_sockstruct by taking advantage of a hole.Signed-off-by Thomas Higdon <[email protected]>
Applied.