Skip to contentSkip to conversation

net: tcp: fix rx timestamp behavior for tcp_recvmsg

PATCH Superseded Kelly Littlepage 6 participants 12 messages May 4, 2020 – May 8, 2020

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 net: tcp: fix rx timestamp behavior for tcp_recvmsg

Timestamping cmsgs are not returned when the user buffer supplied to recvmsg is too small to copy at least one skbuff in entirety. Support for TCP rx timestamps came from commit 98aaa913b4ed ("tcp: Extend SOF_TIMESTAMPING_RX_SOFTWARE to TCP recvmsg") which noted that the cmsg should "return the timestamp corresponding to the highest sequence number data returned." The commit further notes that when coalescing skbs code should "maintain the invariant of returning the timestamp of the last byte in the recvmsg buffer."

This is consistent with Section 1.4 of timestamping.txt, a document that discusses expected behavior when timestamping streaming protocols. It's worth noting that Section 1.4 alludes to a "buffer" in a way that might have resulted in the current behavior:

Quoted 1 line

The SO_TIMESTAMPING interface supports timestamping of bytes in a

bytestream. Each request is interpreted as a request for when the entire contents of the buffer has passed a timestamping point....In practice, timestamps can be correlated with segments of a bytestream consistently, if both semantics of the timestamp and the timing of measurement are chosen correctly....For bytestreams, we chose that a timestamp is generated only when all bytes have passed a point.

An interpretation of skbs as delineators for timestamping points makes sense for tx timestamps but poses implementation challenges on the rx side. Under the current API unless tcp_recvmsg happens to return bytes copied from precisely one skb there's no useful mapping from bytes to timestamps. Some sequences of reads will result in timestamps getting lost and others will result in the user receiving a timestamp from the second to last skb that tcp_recvmsg copied from instead of the last. The proposed change addresses both problems while remaining consistent with 1.4 and the wording of commit 98aaa913b4ed ("tcp: Extend SOF_TIMESTAMPING_RX_SOFTWARE to TCP recvmsg").

---
 net/ipv4/tcp.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
net/ipv4/tcp.c +4 -2
index 6d87de434377..e72bd651d21a 100644
@@ -2154,13 +2154,15 @@ int tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int nonblock,
 			tp->urg_data = 0;
 			tcp_fast_path_check(sk);
 		}
-		if (used + offset < skb->len)
-			continue;
 
 		if (TCP_SKB_CB(skb)->has_rxtstamp) {
 			tcp_update_recv_tstamps(skb, &tss);
 			cmsg_flags |= 2;
 		}
+
+		if (used + offset < skb->len)
+			continue;
+
 		if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN)
 			goto found_fin_ok;
 		if (!(flags & MSG_PEEK))
Signature · 11 lines
2.26.2


-- 
This email and any attachments thereto may contain private, confidential, 
and privileged material for the sole use of the intended recipient. If you 
are not the intended recipient or otherwise believe that you have received 
this message in error, please notify the sender immediately and delete the 
original. Any review, copying, or distribution of this email (or any 
attachments thereto) by others is strictly prohibited. If this message was 
misdirected, OCX Group Inc. does not waive any confidentiality or privilege.
#1 lore raw
Quoted 3 lines · Kelly Littlepage

Timestamping cmsgs are not returned when the user buffer supplied to recvmsg is too small to copy at least one skbuff in entirety.

In general a tcp reader should not make any assumptions on packetization of the bytestream, including the number of skbs that might have made up the bytestream.

Quoted 7 lines

Support
for TCP rx timestamps came from commit 98aaa913b4ed ("tcp: Extend SOF_TIMESTAMPING_RX_SOFTWARE to TCP recvmsg") which noted that the cmsg should "return the timestamp corresponding to the highest sequence number data returned." The commit further notes that when coalescing skbs code should "maintain the invariant of returning the timestamp of the last byte in the recvmsg buffer."

This states that if a byte range spans multiple timestamps, only the last one is returned.

Quoted 13 lines

This is consistent with Section 1.4 of timestamping.txt, a document that discusses expected behavior when timestamping streaming protocols. It's worth noting that Section 1.4 alludes to a "buffer" in a way that might have resulted in the current behavior:

Quoted 1 line

The SO_TIMESTAMPING interface supports timestamping of bytes in a

bytestream. Each request is interpreted as a request for when the entire contents of the buffer has passed a timestamping point....In practice, timestamps can be correlated with segments of a bytestream consistently, if both semantics of the timestamp and the timing of measurement are chosen correctly....For bytestreams, we chose that a timestamp is generated only when all bytes have passed a point.

Quoted 6 lines

An interpretation of skbs as delineators for timestamping points makes sense for tx timestamps but poses implementation challenges on the rx side. Under the current API unless tcp_recvmsg happens to return bytes copied from precisely one skb there's no useful mapping from bytes to timestamps. Some sequences of reads will result in timestamps getting lost

That's a known caveat, see above. This patch does not change that.

Quoted 2 lines

and others will result in the user receiving a timestamp from the second to last skb that tcp_recvmsg copied from instead of the last.

On Tx, the idea was to associate a timestamp with the last byte in the send buffer, so that a timestamp for this seqno informs us of the upper bound on latency of all bytes in the send buffer.

On Rx, we currently return the timestamp of the last skb of which the last byte is read, which is associated with a byte in the recv buffer, but it is not necessarily the last one. Nor the first. As such it is not clear what it defines.

Your patch addresses this by instead always returning the timestamp associated with the last byte in the recv buffer. The same timestamp could then be returned again for a subsequent recv call, if the entire recv buffer is filled from the same skb. Which is fine.

That sounds correct to me.

Quoted 36 lines

The
proposed change addresses both problems while remaining consistent with 1.4 and the wording of commit 98aaa913b4ed ("tcp: Extend SOF_TIMESTAMPING_RX_SOFTWARE to TCP recvmsg").

---
 net/ipv4/tcp.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
net/ipv4/tcp.c +4 -3
index 6d87de434377..e72bd651d21a 100644
@@ -2154,13 +2154,15 @@ int tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int nonblock,
                        tp->urg_data = 0;
                        tcp_fast_path_check(sk);
                }
-               if (used + offset < skb->len)
-                       continue;
 
                if (TCP_SKB_CB(skb)->has_rxtstamp) {
                        tcp_update_recv_tstamps(skb, &tss);
                        cmsg_flags |= 2;
                }
+
+               if (used + offset < skb->len)
+                       continue;
+
                if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN)
                        goto found_fin_ok;
                if (!(flags & MSG_PEEK))
--

2.26.2

#2 lore raw
Quoted 55 lines · Willem de Bruijn
Quoted 3 lines · Kelly Littlepage

Timestamping cmsgs are not returned when the user buffer supplied to recvmsg is too small to copy at least one skbuff in entirety.

In general a tcp reader should not make any assumptions on packetization of the bytestream, including the number of skbs that might have made up the bytestream.

Quoted 7 lines

Support
for TCP rx timestamps came from commit 98aaa913b4ed ("tcp: Extend SOF_TIMESTAMPING_RX_SOFTWARE to TCP recvmsg") which noted that the cmsg should "return the timestamp corresponding to the highest sequence number data returned." The commit further notes that when coalescing skbs code should "maintain the invariant of returning the timestamp of the last byte in the recvmsg buffer."

This states that if a byte range spans multiple timestamps, only the last one is returned.

Quoted 13 lines

This is consistent with Section 1.4 of timestamping.txt, a document that discusses expected behavior when timestamping streaming protocols. It's worth noting that Section 1.4 alludes to a "buffer" in a way that might have resulted in the current behavior:

Quoted 1 line

The SO_TIMESTAMPING interface supports timestamping of bytes in a

bytestream. Each request is interpreted as a request for when the entire contents of the buffer has passed a timestamping point....In practice, timestamps can be correlated with segments of a bytestream consistently, if both semantics of the timestamp and the timing of measurement are chosen correctly....For bytestreams, we chose that a timestamp is generated only when all bytes have passed a point.

Quoted 6 lines

An interpretation of skbs as delineators for timestamping points makes sense for tx timestamps but poses implementation challenges on the rx side. Under the current API unless tcp_recvmsg happens to return bytes copied from precisely one skb there's no useful mapping from bytes to timestamps. Some sequences of reads will result in timestamps getting lost

That's a known caveat, see above. This patch does not change that.

Quoted 2 lines

and others will result in the user receiving a timestamp from the second to last skb that tcp_recvmsg copied from instead of the last.

On Tx, the idea was to associate a timestamp with the last byte in the send buffer, so that a timestamp for this seqno informs us of the upper bound on latency of all bytes in the send buffer.

On Rx, we currently return the timestamp of the last skb of which the last byte is read, which is associated with a byte in the recv buffer, but it is not necessarily the last one. Nor the first. As such it is not clear what it defines.

Quoted 6 lines

Your patch addresses this by instead always returning the timestamp associated with the last byte in the recv buffer. The same timestamp could then be returned again for a subsequent recv call, if the entire recv buffer is filled from the same skb. Which is fine.

That sounds correct to me.

Due to my earlier comments the patch is no longer on patchwork. Can you please resubmit it.

But to be clear, the code looks good to me. Please add

Fixes 98aaa913b4ed ("tcp: Extend SOF_TIMESTAMPING_RX_SOFTWARE to TCP recvmsg")

The commit message can perhaps be a bit shorter. They key points are

1. the stated intent of the original commit is to "return the timestamp corresponding to the highest sequence number data returned."
2. the current implementation returns the timestamp for the last byte of the last fully read skb, which is not necessarily the last byte in the recv buffer.
3. that this patch converts behavior to the original definition.

Previous draft versions of the patch recorded the timestamp before label skip_copy, which also matches this behavior.

I took a quick look at the selftests under tools/testing/selftests/net, but they don't test for this specific behavior. Given that test code should make no assumptions on packetization, it is also not that straightforward to test in a robust manner.

Quoted 37 lines
Quoted 37 lines

The
proposed change addresses both problems while remaining consistent with 1.4 and the wording of commit 98aaa913b4ed ("tcp: Extend SOF_TIMESTAMPING_RX_SOFTWARE to TCP recvmsg").

---
 net/ipv4/tcp.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
net/ipv4/tcp.c +4 -3
index 6d87de434377..e72bd651d21a 100644
@@ -2154,13 +2154,15 @@ int tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int nonblock,
                        tp->urg_data = 0;
                        tcp_fast_path_check(sk);
                }
-               if (used + offset < skb->len)
-                       continue;
 
                if (TCP_SKB_CB(skb)->has_rxtstamp) {
                        tcp_update_recv_tstamps(skb, &tss);
                        cmsg_flags |= 2;
                }
+
+               if (used + offset < skb->len)
+                       continue;
+
                if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN)
                        goto found_fin_ok;
                if (!(flags & MSG_PEEK))
--

2.26.2

#3 lore raw

PATCH v2 1/1 net: tcp: fixes commit 98aaa913b4ed ("tcp: Extend SOF_TIMESTAMPING_RX_SOFTWARE to TCP recvmsg")

The stated intent of the original commit is to is to "return the timestamp corresponding to the highest sequence number data returned." The current implementation returns the timestamp for the last byte of the last fully read skb, which is not necessarily the last byte in the recv buffer. This patch converts behavior to the original definition, and to the behavior of the previous draft versions of commit 98aaa913b4ed ("tcp: Extend SOF_TIMESTAMPING_RX_SOFTWARE to TCP recvmsg") which also match this behavior.

---
Thanks and credit to Willem de Bruijn for the revised commit language

 net/ipv4/tcp.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
net/ipv4/tcp.c +4 -2
index 6d87de434377..e72bd651d21a 100644
@@ -2154,13 +2154,15 @@ int tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int nonblock,
 			tp->urg_data = 0;
 			tcp_fast_path_check(sk);
 		}
-		if (used + offset < skb->len)
-			continue;
 
 		if (TCP_SKB_CB(skb)->has_rxtstamp) {
 			tcp_update_recv_tstamps(skb, &tss);
 			cmsg_flags |= 2;
 		}
+
+		if (used + offset < skb->len)
+			continue;
+
 		if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN)
 			goto found_fin_ok;
 		if (!(flags & MSG_PEEK))
Signature · 11 lines
2.26.2


-- 
This email and any attachments thereto may contain private, confidential, 
and privileged material for the sole use of the intended recipient. If you 
are not the intended recipient or otherwise believe that you have received 
this message in error, please notify the sender immediately and delete the 
original. Any review, copying, or distribution of this email (or any 
attachments thereto) by others is strictly prohibited. If this message was 
misdirected, OCX Group Inc. does not waive any confidentiality or privilege.
#4 lore raw

Re: [PATCH v2] net: tcp: fixes commit 98aaa913b4ed ("tcp: Extend SOF_TIMESTAMPING_RX_SOFTWARE to TCP recvmsg")

Quoted 10 lines · Kelly Littlepage

The stated intent of the original commit is to is to "return the timestamp corresponding to the highest sequence number data returned." The current implementation returns the timestamp for the last byte of the last fully read skb, which is not necessarily the last byte in the recv buffer. This patch converts behavior to the original definition, and to the behavior of the previous draft versions of commit 98aaa913b4ed ("tcp: Extend SOF_TIMESTAMPING_RX_SOFTWARE to TCP recvmsg") which also match this behavior.

Quoted 5 lines

---
Thanks and credit to Willem de Bruijn for the revised commit language

Thanks for resubmitting. I did not mean to put the Fixes tag in the subject line.

The Fixes tag goes at the top of the block of signs-offs. If unclear, please look at a couple of examples on the mailing list or in git log.

The existing subject from v1 was fine. It is now too long. Could you resubmit a v3?

Thanks

Quoted 38 lines
 net/ipv4/tcp.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
net/ipv4/tcp.c +4 -3
index 6d87de434377..e72bd651d21a 100644
@@ -2154,13 +2154,15 @@ int tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int nonblock,
                        tp->urg_data = 0;
                        tcp_fast_path_check(sk);
                }
-               if (used + offset < skb->len)
-                       continue;
 
                if (TCP_SKB_CB(skb)->has_rxtstamp) {
                        tcp_update_recv_tstamps(skb, &tss);
                        cmsg_flags |= 2;
                }
+
+               if (used + offset < skb->len)
+                       continue;
+
                if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN)
                        goto found_fin_ok;
                if (!(flags & MSG_PEEK))
--

2.26.2

--
This email and any attachments thereto may contain private, confidential, and privileged material for the sole use of the intended recipient. If you are not the intended recipient or otherwise believe that you have received this message in error, please notify the sender immediately and delete the original. Any review, copying, or distribution of this email (or any attachments thereto) by others is strictly prohibited. If this message was misdirected, OCX Group Inc. does not waive any confidentiality or privilege.

#5 lore raw

PATCH v3 1/1 net: tcp: fix rx timestamp behavior for tcp_recvmsg

The stated intent of the original commit is to is to "return the timestamp corresponding to the highest sequence number data returned." The current implementation returns the timestamp for the last byte of the last fully read skb, which is not necessarily the last byte in the recv buffer. This patch converts behavior to the original definition, and to the behavior of the previous draft versions of commit 98aaa913b4ed ("tcp: Extend SOF_TIMESTAMPING_RX_SOFTWARE to TCP recvmsg") which also match this behavior.

Fixes 98aaa913b4ed ("tcp: Extend SOF_TIMESTAMPING_RX_SOFTWARE to TCP recvmsg")

---
Reverted to the original subject line

 net/ipv4/tcp.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
net/ipv4/tcp.c +4 -2
index 6d87de434377..e72bd651d21a 100644
@@ -2154,13 +2154,15 @@ int tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int nonblock,
 			tp->urg_data = 0;
 			tcp_fast_path_check(sk);
 		}
-		if (used + offset < skb->len)
-			continue;
 
 		if (TCP_SKB_CB(skb)->has_rxtstamp) {
 			tcp_update_recv_tstamps(skb, &tss);
 			cmsg_flags |= 2;
 		}
+
+		if (used + offset < skb->len)
+			continue;
+
 		if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN)
 			goto found_fin_ok;
 		if (!(flags & MSG_PEEK))
Signature · 11 lines
2.26.2


-- 
This email and any attachments thereto may contain private, confidential, 
and privileged material for the sole use of the intended recipient. If you 
are not the intended recipient or otherwise believe that you have received 
this message in error, please notify the sender immediately and delete the 
original. Any review, copying, or distribution of this email (or any 
attachments thereto) by others is strictly prohibited. If this message was 
misdirected, OCX Group Inc. does not waive any confidentiality or privilege.
#6 lore raw
Quoted 9 lines · Kelly Littlepage

The stated intent of the original commit is to is to "return the timestamp corresponding to the highest sequence number data returned." The current implementation returns the timestamp for the last byte of the last fully read skb, which is not necessarily the last byte in the recv buffer. This patch converts behavior to the original definition, and to the behavior of the previous draft versions of commit 98aaa913b4ed ("tcp: Extend SOF_TIMESTAMPING_RX_SOFTWARE to TCP recvmsg") which also match this behavior.

Quoted 6 lines
Fixes 98aaa913b4ed ("tcp: Extend SOF_TIMESTAMPING_RX_SOFTWARE to TCP recvmsg")

---
Reverted to the original subject line

SGTM, thanks.

#7 lore raw
Quoted 21 lines · Eric Dumazet
Quoted 9 lines · Kelly Littlepage

The stated intent of the original commit is to is to "return the timestamp corresponding to the highest sequence number data returned." The current implementation returns the timestamp for the last byte of the last fully read skb, which is not necessarily the last byte in the recv buffer. This patch converts behavior to the original definition, and to the behavior of the previous draft versions of commit 98aaa913b4ed ("tcp: Extend SOF_TIMESTAMPING_RX_SOFTWARE to TCP recvmsg") which also match this behavior.

Quoted 6 lines
Fixes 98aaa913b4ed ("tcp: Extend SOF_TIMESTAMPING_RX_SOFTWARE to TCP recvmsg")

---
Reverted to the original subject line

Quoted 3 lines

SGTM, thanks.

#8 lore raw
Quoted 23 lines · Soheil Hassas Yeganeh
Quoted 21 lines · Eric Dumazet
Quoted 15 lines · Kelly Littlepage

The stated intent of the original commit is to is to "return the timestamp corresponding to the highest sequence number data returned." The current implementation returns the timestamp for the last byte of the last fully read skb, which is not necessarily the last byte in the recv buffer. This patch converts behavior to the original definition, and to the behavior of the previous draft versions of commit 98aaa913b4ed ("tcp: Extend SOF_TIMESTAMPING_RX_SOFTWARE to TCP recvmsg") which also match this behavior.

Fixes 98aaa913b4ed ("tcp: Extend SOF_TIMESTAMPING_RX_SOFTWARE to TCP recvmsg")

---
Reverted to the original subject line

Quoted 5 lines
Quoted 3 lines

SGTM, thanks.

maintainer
#9 lore raw
In reply to Kelly Littlepage
Quoted 2 lines · On Fri, 8 May 2020 14:23:10 +0000 Kelly Littlepage wrote:

Any review, copying, or distribution of this email (or any attachments thereto) by others is strictly prohibited.

I'm afraid you'll have to do something about this footer if you want the patch to be applied.. Is sending from a different email an option?

(please make sure to add the review and ack tags you received from folks to your commit before reposting)

#10 lore raw

PATCH v4 1/1 net: tcp: fix rx timestamp behavior for tcp_recvmsg

From: Kelly Littlepage <[email protected]>

The stated intent of the original commit is to is to "return the timestamp corresponding to the highest sequence number data returned." The current implementation returns the timestamp for the last byte of the last fully read skb, which is not necessarily the last byte in the recv buffer. This patch converts behavior to the original definition, and to the behavior of the previous draft versions of commit 98aaa913b4ed ("tcp: Extend SOF_TIMESTAMPING_RX_SOFTWARE to TCP recvmsg") which also match this behavior.

Fixes 98aaa913b4ed ("tcp: Extend SOF_TIMESTAMPING_RX_SOFTWARE to TCP recvmsg")

---
Sending from an alternative email given the compliance footer that's unavoidably attached to all emails from my primary account. The patch itself is unchanged.

 net/ipv4/tcp.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
net/ipv4/tcp.c +4 -2
index 6d87de434377..e72bd651d21a 100644
@@ -2154,13 +2154,15 @@ int tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int nonblock,
 			tp->urg_data = 0;
 			tcp_fast_path_check(sk);
 		}
-		if (used + offset < skb->len)
-			continue;
 
 		if (TCP_SKB_CB(skb)->has_rxtstamp) {
 			tcp_update_recv_tstamps(skb, &tss);
 			cmsg_flags |= 2;
 		}
+
+		if (used + offset < skb->len)
+			continue;
+
 		if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN)
 			goto found_fin_ok;
 		if (!(flags & MSG_PEEK))
Signature · 1 line
2.26.2
maintainer Applied
#11 lore raw
Quoted 10 lines · On Fri, 8 May 2020 19:58:46 +0000 Kelly Littlepage wrote:

From: Kelly Littlepage <[email protected]>

The stated intent of the original commit is to is to "return the timestamp corresponding to the highest sequence number data returned." The current implementation returns the timestamp for the last byte of the last fully read skb, which is not necessarily the last byte in the recv buffer. This patch converts behavior to the original definition, and to the behavior of the previous draft versions of commit 98aaa913b4ed ("tcp: Extend SOF_TIMESTAMPING_RX_SOFTWARE to TCP recvmsg") which also match this behavior.

Applied, thank you!