tcp: Add TCP_INFO counter for packets received out-of-order
Subsystems (3)
PATCH v4 2/2 tcp: Add snd_wnd to TCP_INFO
In reply to Thomas HigdonNeal Cardwell mentioned that snd_wnd would be useful for diagnosing TCP performance problems --
Quoted 6 lines
(1) Usually when we're diagnosing TCP performance problems, we do so from the sender, since the sender makes most of the performance-critical decisions (cwnd, pacing, TSO size, TSQ, etc). From the sender-side the thing that would be most useful is to see
tp->snd_wnd, the receive window that the receiver has advertised to the sender.
This serves the purpose of adding an additional __u32 to avoid the would-be hole caused by the addition of the tcpi_rcvi_ooopack field.
---
changes from v3:
- changed from rcv_wnd to snd_wnd
include/uapi/linux/tcp.h | 1 + net/ipv4/tcp.c | 1 + 2 files changed, 2 insertions(+)
include/uapi/linux/tcp.h +1 -0
@@ -272,6 +272,7 @@ struct tcp_info {__u32 tcpi_reord_seen;/* reordering events seen */ __u32 tcpi_rcv_ooopack;/* Out-of-order packets received */ +__u32 tcpi_snd_wnd;/* Remote peer's advertised recv window size */ };/* netlink attributes types for SCM_TIMESTAMPING_OPT_STATS */
net/ipv4/tcp.c +1 -0
@@ -3297,6 +3297,7 @@ void tcp_get_info(struct sock *sk, struct tcp_info *info) info->tcpi_dsack_dups = tp->dsack_dups; info->tcpi_reord_seen = tp->reord_seen; info->tcpi_rcv_ooopack = tp->rcv_ooopack; + info->tcpi_snd_wnd = tp->snd_wnd;unlock_sock_fast (sk, slow); }EXPORT_SYMBOL_GPL (tcp_get_info);
Signature · 1 line
2.17.1
PATCH v4 1/2 tcp: Add TCP_INFO counter for packets received out-of-order
For receive-heavy cases on the server-side, we want to track the connection quality for individual client IPs. This counter, similar to the existing system-wide TCPOFOQueue counter in /proc/net/netstat, tracks out-of-order packet reception. By providing this counter in TCP_INFO, it will allow understanding to what degree receive-heavy sockets are experiencing out-of-order delivery and packet drops indicating congestion.
Please note that this is similar to the counter in NetBSD TCP_INFO, and has the same name.
---
no changes from v3
include/linux/tcp.h | 2 ++ include/uapi/linux/tcp.h | 2 ++ net/ipv4/tcp.c | 2 ++ net/ipv4/tcp_input.c | 1 + 4 files changed, 7 insertions(+)
include/linux/tcp.h +2 -0
@@ -393,6 +393,8 @@ struct tcp_sock {*/ struct request_sock *fastopen_rsk;u32 *saved_syn; + +u32 rcv_ooopack;/* Received out-of-order packets, for tcpinfo */ };enum tsq_enum {
include/uapi/linux/tcp.h +2 -0
@@ -270,6 +270,8 @@ struct tcp_info {__u64 tcpi_bytes_retrans;/* RFC4898 tcpEStatsPerfOctetsRetrans */ __u32 tcpi_dsack_dups;/* RFC4898 tcpEStatsStackDSACKDups */ __u32 tcpi_reord_seen;/* reordering events seen */ + +__u32 tcpi_rcv_ooopack;/* Out-of-order packets received */ };/* netlink attributes types for SCM_TIMESTAMPING_OPT_STATS */
net/ipv4/tcp.c +2 -0
@@ -2653,6 +2653,7 @@ int tcp_disconnect(struct sock *sk, int flags) tp->rx_opt.saw_tstamp =0 ; tp->rx_opt.dsack =0 ; tp->rx_opt.num_sacks =0 ; + tp->rcv_ooopack =0 ;/* Clean up fastopen related fields */
@@ -3295,6 +3296,7 @@ void tcp_get_info(struct sock *sk, struct tcp_info *info) info->tcpi_bytes_retrans = tp->bytes_retrans; info->tcpi_dsack_dups = tp->dsack_dups; info->tcpi_reord_seen = tp->reord_seen; + info->tcpi_rcv_ooopack = tp->rcv_ooopack;unlock_sock_fast (sk, slow); }EXPORT_SYMBOL_GPL (tcp_get_info);
net/ipv4/tcp_input.c +1 -0
@@ -4555,6 +4555,7 @@ static void tcp_data_queue_ofo(struct sock *sk, struct sk_buff *skb) tp->pred_flags =0 ;inet_csk_schedule_ack (sk); + tp->rcv_ooopack +=max_t (u16,1 ,skb_shinfo (skb)->gso_segs);NET_INC_STATS (sock_net (sk),LINUX_MIB_TCPOFOQUEUE ); seq =TCP_SKB_CB (skb)->seq; end_seq =TCP_SKB_CB (skb)->end_seq;
Signature · 1 line
2.17.1
Neal Cardwell
maintainerQuoted 28 lines · Thomas Higdon
For receive-heavy cases on the server-side, we want to track the connection quality for individual client IPs. This counter, similar to the existing system-wide TCPOFOQueue counter in
/proc/net/netstat, tracks out-of-order packet reception. By providing this counter inTCP_INFO, it will allow understanding to what degree receive-heavy sockets are experiencing out-of-order delivery and packet drops indicating congestion.Please note that this is similar to the counter in NetBSD
TCP_INFO, and has the same name.Signed-off-by Thomas Higdon <[email protected]>---
no changes from v3
include/linux/tcp.h | 2 ++ include/uapi/linux/tcp.h | 2 ++ net/ipv4/tcp.c | 2 ++ net/ipv4/tcp_input.c | 1 + 4 files changed, 7 insertions(+)include/linux/tcp.h +0
-0
include/linux/tcp.h struct tcp_sock { · line 397
Quoted 5 lines
include/linux/tcp.h +2
-0@@ -393,6 +393,8 @@ struct tcp_sock {*/ struct request_sock *fastopen_rsk;u32 *saved_syn; + +u32 rcv_ooopack;/* Received out-of-order packets, for tcpinfo */
Thanks for adding this.
A thought: putting the new rcv_ooopack field here makes struct tcp_sock bigger, and increases the odds of taking a cache miss (according to "pahole" this field is the only one in a new cache line).
I'd suggest putting the new rcv_ooopack field immediately before rcv_rtt_last_tsecr. That would use up a 4-byte hole, and would place it in a cache line already used on TCP receivers (for rcv_rtt logic). This would make it less likely this new field causes more cache misses or uses more space.
Details: looking at the output of "pahole" for tcp_sock in various cases:
net-next before this patch:
-------------------------------------
...
u8 bpf_sock_ops_cb_flags; /* 2076 1 */
/* XXX 3 bytes hole, try to pack */
u32 rcv_rtt_last_tsecr; /* 2080 4 */
/* XXX 4 bytes hole, try to pack */
struct {
u32 rtt_us; /* 2088 4 */
u32 seq; /* 2092 4 */
u64 time; /* 2096 8 */
} rcv_rtt_est; /* 2088 16 */
...
/* size: 2176, cachelines: 34, members: 134 */
/* sum members: 2164, holes: 4, sum holes: 12 */
/* paddings: 3, sum paddings: 12 */net-next with this patch:
-------------------------------------
...
u32 * saved_syn; /* 2168 8 */
/* --- cacheline 34 boundary (2176 bytes) --- */
u32 rcv_ooopack; /* 2176 4 */
...
/* size: 2184, cachelines: 35, members: 135 */
/* sum members: 2168, holes: 4, sum holes: 12 */
/* padding: 4 */
/* paddings: 3, sum paddings: 12 */
/* last cacheline: 8 bytes */net-next with this field in the suggested spot:
-------------------------------------
...
/* XXX 3 bytes hole, try to pack */
u32 rcv_ooopack; /* 2080 4 */
u32 rcv_rtt_last_tsecr; /* 2084 4 */
struct {
u32 rtt_us; /* 2088 4 */
u32 seq; /* 2092 4 */
u64 time; /* 2096 8 */
} rcv_rtt_est; /* 2088 16 */
...
/* size: 2176, cachelines: 34, members: 135 */
/* sum members: 2168, holes: 3, sum holes: 8 */
/* paddings: 3, sum paddings: 12 */neal
neal
Neal Cardwell
maintainerRe: [PATCH v4 2/2] tcp: Add snd_wnd to TCP_INFO
In reply to Thomas HigdonQuoted 27 lines · Thomas Higdon
Neal Cardwell mentioned that
snd_wndwould be useful for diagnosing TCP performance problems --Quoted 6 lines
(1) Usually when we're diagnosing TCP performance problems, we do so from the sender, since the sender makes most of the performance-critical decisions (cwnd, pacing, TSO size, TSQ, etc). From the sender-side the thing that would be most useful is to see
tp->snd_wnd, the receive window that the receiver has advertised to the sender.This serves the purpose of adding an additional
__u32to avoid the would-be hole caused by the addition of thetcpi_rcvi_ooopackfield.Signed-off-by Thomas Higdon <[email protected]>---
changes from v3:
- changed fromrcv_wndtosnd_wndinclude/uapi/linux/tcp.h | 1 + net/ipv4/tcp.c | 1 + 2 files changed, 2 insertions(+)include/uapi/linux/tcp.h +0
-0
include/uapi/linux/tcp.h struct tcp_info { · line 276
Quoted 5 lines
include/uapi/linux/tcp.h +1
-0@@ -272,6 +272,7 @@ struct tcp_info {__u32 tcpi_reord_seen;/* reordering events seen */ __u32 tcpi_rcv_ooopack;/* Out-of-order packets received */ +__u32 tcpi_snd_wnd;/* Remote peer's advertised recv window size */ };
Thanks for adding this!
My run of ./scripts/checkpatch.pl is showing a warning on this line:
WARNING: line over 80 characters
#19: FILE: include/uapi/linux/tcp.h:273:
+ __u32 tcpi_snd_wnd; /* Remote peer's advertised recv
window size */
What if the comment is shortened up to fit in 80 columns and the units (bytes) are added, something like:
__u32 tcpi_snd_wnd; /* peer's advertised recv window (bytes) */
neal
Re: [PATCH v4 2/2] tcp: Add snd_wnd to TCP_INFO
Quoted 44 lines · Neal Cardwell
Quoted 27 lines · Thomas Higdon
Neal Cardwell mentioned that
snd_wndwould be useful for diagnosing TCP performance problems --Quoted 6 lines
(1) Usually when we're diagnosing TCP performance problems, we do so from the sender, since the sender makes most of the performance-critical decisions (cwnd, pacing, TSO size, TSQ, etc). From the sender-side the thing that would be most useful is to see
tp->snd_wnd, the receive window that the receiver has advertised to the sender.This serves the purpose of adding an additional
__u32to avoid the would-be hole caused by the addition of thetcpi_rcvi_ooopackfield.Signed-off-by Thomas Higdon <[email protected]>---
changes from v3:
- changed fromrcv_wndtosnd_wndinclude/uapi/linux/tcp.h | 1 + net/ipv4/tcp.c | 1 + 2 files changed, 2 insertions(+)include/uapi/linux/tcp.h +0
-0include/uapi/linux/tcp.h struct tcp_info { · line 276
Quoted 5 lines
include/uapi/linux/tcp.h +1
-0@@ -272,6 +272,7 @@ struct tcp_info {__u32 tcpi_reord_seen;/* reordering events seen */ __u32 tcpi_rcv_ooopack;/* Out-of-order packets received */ +__u32 tcpi_snd_wnd;/* Remote peer's advertised recv window size */ };Thanks for adding this!
My run of
./scripts/checkpatch.plis showing a warning on this line:WARNING: line over 80 characters
#19: FILE:include/uapi/linux/tcp.h:273:
+__u32tcpi_snd_wnd; /* Remote peer's advertised recv
window size */
Quoted 4 lines
What if the comment is shortened up to fit in 80 columns and the units (bytes) are added, something like:
__u32tcpi_snd_wnd; /* peer's advertised recv window (bytes) */
just a thought: will tcpi_peer_rcv_wnd be more self-explanatory?
Quoted 2 lines
neal
Neal Cardwell
maintainerRe: [PATCH v4 2/2] tcp: Add snd_wnd to TCP_INFO
Quoted 5 lines · Yuchung Cheng
Quoted 4 lines
What if the comment is shortened up to fit in 80 columns and the units (bytes) are added, something like:
__u32tcpi_snd_wnd; /* peer's advertised recv window (bytes) */just a thought: will
tcpi_peer_rcv_wndbe more self-explanatory?
Good suggestion. I'm on the fence about that one. By itself, I agree tcpi_peer_rcv_wnd sounds much more clear. But tcpi_snd_wnd has the virtue of matching both the kernel code (tp->snd_wnd) and RFC 793 (SND.WND). So they both have pros and cons. Maybe someone else feels more strongly one way or the other.
neal
Re: [PATCH v4 2/2] tcp: Add snd_wnd to TCP_INFO
Quoted 5 lines · Neal Cardwell
Quoted 3 lines · Yuchung Cheng
Quoted 3 lines
What if the comment is shortened up to fit in 80 columns and the units (bytes) are added, something like:
Quoted 8 lines
Quoted 2 lines
Quoted 1 line
__u32tcpi_snd_wnd; /* peer's advertised recv window (bytes) */just a thought: will
tcpi_peer_rcv_wndbe more self-explanatory?Good suggestion. I'm on the fence about that one. By itself, I agree
tcpi_peer_rcv_wndsounds much more clear. Buttcpi_snd_wndhas the virtue of matching both the kernel code (tp->snd_wnd) and RFC 793 (SND.WND). So they both have pros and cons. Maybe someone else feels more strongly one way or the other.
no strong preference -- snd_wnd is just fine too with proper comment (for consistency).
also it might be good to comment whether it is scaled or not. it may not be obvious if the actual value and scale are small.
Quoted 2 lines
neal