Skip to contentSkip to conversation

tcp: Add TCP_INFO counter for packets received out-of-order

PATCH Applied Thomas Higdon 5 participants 9 messages Sep 13, 2019 – Sep 16, 2019

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH v5 1/2 tcp: Add TCP_INFO counter for packets received out-of-order

For receive-heavy cases on the server-side, we want to track the connection quality for individual client IPs. This counter, similar to the existing system-wide TCPOFOQueue counter in /proc/net/netstat, tracks out-of-order packet reception. By providing this counter in TCP_INFO, it will allow understanding to what degree receive-heavy sockets are experiencing out-of-order delivery and packet drops indicating congestion.

Please note that this is similar to the counter in NetBSD TCP_INFO, and has the same name.

Also note that we avoid increasing the size of the tcp_sock struct by taking advantage of a hole.

---
changes since v4:
- optimize placement of rcv_ooopack to avoid increasing tcp_sock struct size

 include/linux/tcp.h      | 2 ++
 include/uapi/linux/tcp.h | 2 ++
 net/ipv4/tcp.c           | 2 ++
 net/ipv4/tcp_input.c     | 1 +
 4 files changed, 7 insertions(+)
include/linux/tcp.h +2 -0
index f3a85a7fb4b1..99617e528ea2 100644
@@ -354,6 +354,8 @@ struct tcp_sock {
 #define BPF_SOCK_OPS_TEST_FLAG(TP, ARG) 0
 #endif
 
+	u32 rcv_ooopack; /* Received out-of-order packets, for tcpinfo */
+
 /* Receiver side RTT estimation */
 	u32 rcv_rtt_last_tsecr;
 	struct {
include/uapi/linux/tcp.h +2 -0
index b3564f85a762..20237987ccc8 100644
@@ -270,6 +270,8 @@ struct tcp_info {
 	__u64	tcpi_bytes_retrans;  /* RFC4898 tcpEStatsPerfOctetsRetrans */
 	__u32	tcpi_dsack_dups;     /* RFC4898 tcpEStatsStackDSACKDups */
 	__u32	tcpi_reord_seen;     /* reordering events seen */
+
+	__u32	tcpi_rcv_ooopack;    /* Out-of-order packets received */
 };
 
 /* netlink attributes types for SCM_TIMESTAMPING_OPT_STATS */
net/ipv4/tcp.c +2 -0
index 94df48bcecc2..4cf58208270e 100644
@@ -2653,6 +2653,7 @@ int tcp_disconnect(struct sock *sk, int flags)
 	tp->rx_opt.saw_tstamp = 0;
 	tp->rx_opt.dsack = 0;
 	tp->rx_opt.num_sacks = 0;
+	tp->rcv_ooopack = 0;
 
 
 	/* Clean up fastopen related fields */
@@ -3295,6 +3296,7 @@ void tcp_get_info(struct sock *sk, struct tcp_info *info)
 	info->tcpi_bytes_retrans = tp->bytes_retrans;
 	info->tcpi_dsack_dups = tp->dsack_dups;
 	info->tcpi_reord_seen = tp->reord_seen;
+	info->tcpi_rcv_ooopack = tp->rcv_ooopack;
 	unlock_sock_fast(sk, slow);
 }
 EXPORT_SYMBOL_GPL(tcp_get_info);
net/ipv4/tcp_input.c +1 -0
index 706cbb3b2986..2ef333354026 100644
@@ -4555,6 +4555,7 @@ static void tcp_data_queue_ofo(struct sock *sk, struct sk_buff *skb)
 	tp->pred_flags = 0;
 	inet_csk_schedule_ack(sk);
 
+	tp->rcv_ooopack += max_t(u16, 1, skb_shinfo(skb)->gso_segs);
 	NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPOFOQUEUE);
 	seq = TCP_SKB_CB(skb)->seq;
 	end_seq = TCP_SKB_CB(skb)->end_seq;
Signature · 1 line
2.17.1
#1 lore raw

PATCH v5 2/2 tcp: Add snd_wnd to TCP_INFO

Neal Cardwell mentioned that snd_wnd would be useful for diagnosing TCP performance problems --

Quoted 6 lines

(1) Usually when we're diagnosing TCP performance problems, we do so from the sender, since the sender makes most of the performance-critical decisions (cwnd, pacing, TSO size, TSQ, etc). From the sender-side the thing that would be most useful is to see tp->snd_wnd, the receive window that the receiver has advertised to the sender.

This serves the purpose of adding an additional __u32 to avoid the would-be hole caused by the addition of the tcpi_rcvi_ooopack field.

---
changes since v4:
- clarify comment

 include/uapi/linux/tcp.h | 4 ++++
 net/ipv4/tcp.c           | 1 +
 2 files changed, 5 insertions(+)
include/uapi/linux/tcp.h +4 -0
index 20237987ccc8..81e697978e8b 100644
@@ -272,6 +272,10 @@ struct tcp_info {
 	__u32	tcpi_reord_seen;     /* reordering events seen */
 
 	__u32	tcpi_rcv_ooopack;    /* Out-of-order packets received */
+
+	__u32	tcpi_snd_wnd;	     /* peer's advertised receive window after
+				      * scaling (bytes)
+				      */
 };
 
 /* netlink attributes types for SCM_TIMESTAMPING_OPT_STATS */
net/ipv4/tcp.c +1 -0
index 4cf58208270e..79c325a07ba5 100644
@@ -3297,6 +3297,7 @@ void tcp_get_info(struct sock *sk, struct tcp_info *info)
 	info->tcpi_dsack_dups = tp->dsack_dups;
 	info->tcpi_reord_seen = tp->reord_seen;
 	info->tcpi_rcv_ooopack = tp->rcv_ooopack;
+	info->tcpi_snd_wnd = tp->snd_wnd;
 	unlock_sock_fast(sk, slow);
 }
 EXPORT_SYMBOL_GPL(tcp_get_info);
Signature · 1 line
2.17.1
Acked-by on v5 2/2
#2 lore raw

Re: [PATCH v5 2/2] tcp: Add snd_wnd to TCP_INFO

Quoted 10 lines · Thomas Higdon

Neal Cardwell mentioned that snd_wnd would be useful for diagnosing TCP performance problems --

Quoted 6 lines

(1) Usually when we're diagnosing TCP performance problems, we do so from the sender, since the sender makes most of the performance-critical decisions (cwnd, pacing, TSO size, TSQ, etc). From the sender-side the thing that would be most useful is to see tp->snd_wnd, the receive window that the receiver has advertised to the sender.

Quoted 5 lines

This serves the purpose of adding an additional __u32 to avoid the would-be hole caused by the addition of the tcpi_rcvi_ooopack field.

---

Quoted 36 lines

changes since v4:
- clarify comment

 include/uapi/linux/tcp.h | 4 ++++
 net/ipv4/tcp.c           | 1 +
 2 files changed, 5 insertions(+)
include/uapi/linux/tcp.h +4 -0
index 20237987ccc8..81e697978e8b 100644
@@ -272,6 +272,10 @@ struct tcp_info {
        __u32   tcpi_reord_seen;     /* reordering events seen */
 
        __u32   tcpi_rcv_ooopack;    /* Out-of-order packets received */
+
+       __u32   tcpi_snd_wnd;        /* peer's advertised receive window after
+                                     * scaling (bytes)
+                                     */
 };
 
 /* netlink attributes types for SCM_TIMESTAMPING_OPT_STATS */
net/ipv4/tcp.c +1 -1
index 4cf58208270e..79c325a07ba5 100644
@@ -3297,6 +3297,7 @@ void tcp_get_info(struct sock *sk, struct tcp_info *info)
        info->tcpi_dsack_dups = tp->dsack_dups;
        info->tcpi_reord_seen = tp->reord_seen;
        info->tcpi_rcv_ooopack = tp->rcv_ooopack;
+       info->tcpi_snd_wnd = tp->snd_wnd;
        unlock_sock_fast(sk, slow);
 }
 EXPORT_SYMBOL_GPL(tcp_get_info);
--

2.17.1

maintainer Acked-by on v5 2/2
#3 lore raw

Re: [PATCH v5 2/2] tcp: Add snd_wnd to TCP_INFO

In reply to Thomas Higdon
Quoted 10 lines · Thomas Higdon

Neal Cardwell mentioned that snd_wnd would be useful for diagnosing TCP performance problems --

Quoted 6 lines

(1) Usually when we're diagnosing TCP performance problems, we do so from the sender, since the sender makes most of the performance-critical decisions (cwnd, pacing, TSO size, TSQ, etc). From the sender-side the thing that would be most useful is to see tp->snd_wnd, the receive window that the receiver has advertised to the sender.

Quoted 7 lines

This serves the purpose of adding an additional __u32 to avoid the would-be hole caused by the addition of the tcpi_rcvi_ooopack field.

---
changes since v4:
- clarify comment

Thanks!

neal

maintainer Acked-by on v5 1/2
#6 lore raw
In reply to Thomas Higdon
Quoted 12 lines · Thomas Higdon

For receive-heavy cases on the server-side, we want to track the connection quality for individual client IPs. This counter, similar to the existing system-wide TCPOFOQueue counter in /proc/net/netstat, tracks out-of-order packet reception. By providing this counter in TCP_INFO, it will allow understanding to what degree receive-heavy sockets are experiencing out-of-order delivery and packet drops indicating congestion.

Please note that this is similar to the counter in NetBSD TCP_INFO, and has the same name.

Quoted 8 lines

Also note that we avoid increasing the size of the tcp_sock struct by taking advantage of a hole.

---
changes since v4:
- optimize placement of rcv_ooopack to avoid increasing tcp_sock struct size

Thanks, Thomas, for adding this!

After this is merged, would you mind sending a patch to add support to the "ss" command line tool to print these 2 new fields?

My favorite recent example of such a patch to ss is Eric's change:
https://git.kernel.org/pub/scm/network/iproute2/iproute2.git/commit/misc/ss.c?id=5eead6270a19f00464052d4084f32182cfe027ff

thanks,
neal

#4 lore raw

Re: [PATCH v5 2/2] tcp: Add snd_wnd to TCP_INFO

In reply to Neal Cardwell
Quoted 15 lines · Neal Cardwell
Quoted 13 lines · Thomas Higdon

Neal Cardwell mentioned that snd_wnd would be useful for diagnosing TCP performance problems --

Quoted 6 lines

(1) Usually when we're diagnosing TCP performance problems, we do so from the sender, since the sender makes most of the performance-critical decisions (cwnd, pacing, TSO size, TSQ, etc). From the sender-side the thing that would be most useful is to see tp->snd_wnd, the receive window that the receiver has advertised to the sender.

This serves the purpose of adding an additional __u32 to avoid the would-be hole caused by the addition of the tcpi_rcvi_ooopack field.

Quoted 6 lines
Quoted 4 lines

---
changes since v4:
- clarify comment

Thank you for adding the new field!

maintainer Applied
#8 lore raw
In reply to Thomas Higdon

From: Thomas Higdon <[email protected]>
Date: Fri, 13 Sep 2019 23:23:34 +0000

Quoted 8 lines

For receive-heavy cases on the server-side, we want to track the connection quality for individual client IPs. This counter, similar to the existing system-wide TCPOFOQueue counter in /proc/net/netstat, tracks out-of-order packet reception. By providing this counter in TCP_INFO, it will allow understanding to what degree receive-heavy sockets are experiencing out-of-order delivery and packet drops indicating congestion.

Quoted 7 lines

Please note that this is similar to the counter in NetBSD TCP_INFO, and has the same name.

Also note that we avoid increasing the size of the tcp_sock struct by taking advantage of a hole.

Applied.

maintainer Applied
#5 lore raw

Re: [PATCH v5 2/2] tcp: Add snd_wnd to TCP_INFO

In reply to Thomas Higdon

From: Thomas Higdon <[email protected]>
Date: Fri, 13 Sep 2019 23:23:35 +0000

Quoted 9 lines

Neal Cardwell mentioned that snd_wnd would be useful for diagnosing TCP performance problems --

Quoted 6 lines

(1) Usually when we're diagnosing TCP performance problems, we do so from the sender, since the sender makes most of the performance-critical decisions (cwnd, pacing, TSO size, TSQ, etc). From the sender-side the thing that would be most useful is to see tp->snd_wnd, the receive window that the receiver has advertised to the sender.

Quoted 4 lines

This serves the purpose of adding an additional __u32 to avoid the would-be hole caused by the addition of the tcpi_rcvi_ooopack field.

Applied.

#7 lore raw
In reply to Neal Cardwell
Quoted 25 lines · Neal Cardwell
Quoted 12 lines · Thomas Higdon

For receive-heavy cases on the server-side, we want to track the connection quality for individual client IPs. This counter, similar to the existing system-wide TCPOFOQueue counter in /proc/net/netstat, tracks out-of-order packet reception. By providing this counter in TCP_INFO, it will allow understanding to what degree receive-heavy sockets are experiencing out-of-order delivery and packet drops indicating congestion.

Please note that this is similar to the counter in NetBSD TCP_INFO, and has the same name.

Quoted 8 lines

Also note that we avoid increasing the size of the tcp_sock struct by taking advantage of a hole.

---
changes since v4:
- optimize placement of rcv_ooopack to avoid increasing tcp_sock struct size

Quoted 7 lines

Thanks, Thomas, for adding this!

After this is merged, would you mind sending a patch to add support to the "ss" command line tool to print these 2 new fields?

My favorite recent example of such a patch to ss is Eric's change:
https://git.kernel.org/pub/scm/network/iproute2/iproute2.git/commit/misc/ss.c?id=5eead6270a19f00464052d4084f32182cfe027ff

Yes, and thank you for the help in getting this into a good state!

From looking at that "ss" patch, it seems like we would need to wait until iproute2-next's include/uapi/linux/tcp.h has received a merge from kernel net-next before we'd be able to apply a patch for "ss" that uses the new fields.

In the meantime, as you've asked, I will go ahead and send a patch for iproute2-next's "ss" with the assumption that these tcpinfo changes have already been merged.