net: tcp: fix rx timestamp behavior for tcp_recvmsg
Subsystems (3)
PATCH 1/1 net: tcp: fix rx timestamp behavior for tcp_recvmsg
Timestamping cmsgs are not returned when the user buffer supplied to recvmsg is too small to copy at least one skbuff in entirety. Support for TCP rx timestamps came from commit 98aaa913b4ed ("tcp: Extend SOF_TIMESTAMPING_RX_SOFTWARE to TCP recvmsg") which noted that the cmsg should "return the timestamp corresponding to the highest sequence number data returned." The commit further notes that when coalescing skbs code should "maintain the invariant of returning the timestamp of the last byte in the recvmsg buffer."
This is consistent with Section 1.4 of timestamping.txt, a document that discusses expected behavior when timestamping streaming protocols. It's worth noting that Section 1.4 alludes to a "buffer" in a way that might have resulted in the current behavior:
Quoted 1 line
The
SO_TIMESTAMPINGinterface supports timestamping of bytes in a
bytestream. Each request is interpreted as a request for when the entire contents of the buffer has passed a timestamping point....In practice, timestamps can be correlated with segments of a bytestream consistently, if both semantics of the timestamp and the timing of measurement are chosen correctly....For bytestreams, we chose that a timestamp is generated only when all bytes have passed a point.
An interpretation of skbs as delineators for timestamping points makes sense for tx timestamps but poses implementation challenges on the rx side. Under the current API unless tcp_recvmsg happens to return bytes copied from precisely one skb there's no useful mapping from bytes to timestamps. Some sequences of reads will result in timestamps getting lost and others will result in the user receiving a timestamp from the second to last skb that tcp_recvmsg copied from instead of the last. The proposed change addresses both problems while remaining consistent with 1.4 and the wording of commit 98aaa913b4ed ("tcp: Extend SOF_TIMESTAMPING_RX_SOFTWARE to TCP recvmsg").
--- net/ipv4/tcp.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
net/ipv4/tcp.c +4 -2
@@ -2154,13 +2154,15 @@ int tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int nonblock, tp->urg_data =0 ;tcp_fast_path_check (sk); } -if (used + offset < skb->len) -continue ;if (TCP_SKB_CB (skb)->has_rxtstamp) {tcp_update_recv_tstamps (skb, &tss); cmsg_flags |=2 ; } + +if (used + offset < skb->len) +continue ; +if (TCP_SKB_CB (skb)->tcp_flags &TCPHDR_FIN ) goto found_fin_ok;if (!(flags &MSG_PEEK ))
Signature · 11 lines
2.26.2 -- This email and any attachments thereto may contain private, confidential, and privileged material for the sole use of the intended recipient. If you are not the intended recipient or otherwise believe that you have received this message in error, please notify the sender immediately and delete the original. Any review, copying, or distribution of this email (or any attachments thereto) by others is strictly prohibited. If this message was misdirected, OCX Group Inc. does not waive any confidentiality or privilege.
Quoted 3 lines · Kelly Littlepage
Timestamping cmsgs are not returned when the user buffer supplied to recvmsg is too small to copy at least one skbuff in entirety.
In general a tcp reader should not make any assumptions on packetization of the bytestream, including the number of skbs that might have made up the bytestream.
Quoted 7 lines
Support
for TCP rx timestamps came from commit98aaa913b4ed("tcp: ExtendSOF_TIMESTAMPING_RX_SOFTWAREto TCP recvmsg") which noted that the cmsg should "return the timestamp corresponding to the highest sequence number data returned." The commit further notes that when coalescing skbs code should "maintain the invariant of returning the timestamp of the last byte in the recvmsg buffer."
This states that if a byte range spans multiple timestamps, only the last one is returned.
Quoted 13 lines
This is consistent with Section 1.4 of timestamping.txt, a document that discusses expected behavior when timestamping streaming protocols. It's worth noting that Section 1.4 alludes to a "buffer" in a way that might have resulted in the current behavior:
Quoted 1 line
The
SO_TIMESTAMPINGinterface supports timestamping of bytes in abytestream. Each request is interpreted as a request for when the entire contents of the buffer has passed a timestamping point....In practice, timestamps can be correlated with segments of a bytestream consistently, if both semantics of the timestamp and the timing of measurement are chosen correctly....For bytestreams, we chose that a timestamp is generated only when all bytes have passed a point.
Quoted 6 lines
An interpretation of skbs as delineators for timestamping points makes sense for tx timestamps but poses implementation challenges on the rx side. Under the current API unless
tcp_recvmsghappens to return bytes copied from precisely one skb there's no useful mapping from bytes to timestamps. Some sequences of reads will result in timestamps getting lost
That's a known caveat, see above. This patch does not change that.
Quoted 2 lines
and others will result in the user receiving a timestamp from the second to last skb that
tcp_recvmsgcopied from instead of the last.
On Tx, the idea was to associate a timestamp with the last byte in the send buffer, so that a timestamp for this seqno informs us of the upper bound on latency of all bytes in the send buffer.
On Rx, we currently return the timestamp of the last skb of which the last byte is read, which is associated with a byte in the recv buffer, but it is not necessarily the last one. Nor the first. As such it is not clear what it defines.
Your patch addresses this by instead always returning the timestamp associated with the last byte in the recv buffer. The same timestamp could then be returned again for a subsequent recv call, if the entire recv buffer is filled from the same skb. Which is fine.
That sounds correct to me.
Quoted 36 lines
The
proposed change addresses both problems while remaining consistent with 1.4 and the wording of commit98aaa913b4ed("tcp: ExtendSOF_TIMESTAMPING_RX_SOFTWAREto TCP recvmsg").Co-developed-by Iris Liu <[email protected]>Signed-off-by Iris Liu <[email protected]>Signed-off-by Kelly Littlepage <[email protected]>--- net/ipv4/tcp.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)net/ipv4/tcp.c +4
-3@@ -2154,13 +2154,15 @@ int tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int nonblock, tp->urg_data =0 ;tcp_fast_path_check (sk); } -if (used + offset < skb->len) -continue ;if (TCP_SKB_CB (skb)->has_rxtstamp) {tcp_update_recv_tstamps (skb, &tss); cmsg_flags |=2 ; } + +if (used + offset < skb->len) +continue ; +if (TCP_SKB_CB (skb)->tcp_flags &TCPHDR_FIN ) goto found_fin_ok;if (!(flags &MSG_PEEK )) --2.26.2
Quoted 55 lines · Willem de Bruijn
Quoted 3 lines · Kelly Littlepage
Timestamping cmsgs are not returned when the user buffer supplied to recvmsg is too small to copy at least one skbuff in entirety.
In general a tcp reader should not make any assumptions on packetization of the bytestream, including the number of skbs that might have made up the bytestream.
Quoted 7 lines
Support
for TCP rx timestamps came from commit98aaa913b4ed("tcp: ExtendSOF_TIMESTAMPING_RX_SOFTWAREto TCP recvmsg") which noted that the cmsg should "return the timestamp corresponding to the highest sequence number data returned." The commit further notes that when coalescing skbs code should "maintain the invariant of returning the timestamp of the last byte in the recvmsg buffer."This states that if a byte range spans multiple timestamps, only the last one is returned.
Quoted 13 lines
This is consistent with Section 1.4 of timestamping.txt, a document that discusses expected behavior when timestamping streaming protocols. It's worth noting that Section 1.4 alludes to a "buffer" in a way that might have resulted in the current behavior:
Quoted 1 line
The
SO_TIMESTAMPINGinterface supports timestamping of bytes in abytestream. Each request is interpreted as a request for when the entire contents of the buffer has passed a timestamping point....In practice, timestamps can be correlated with segments of a bytestream consistently, if both semantics of the timestamp and the timing of measurement are chosen correctly....For bytestreams, we chose that a timestamp is generated only when all bytes have passed a point.
Quoted 6 lines
An interpretation of skbs as delineators for timestamping points makes sense for tx timestamps but poses implementation challenges on the rx side. Under the current API unless
tcp_recvmsghappens to return bytes copied from precisely one skb there's no useful mapping from bytes to timestamps. Some sequences of reads will result in timestamps getting lostThat's a known caveat, see above. This patch does not change that.
Quoted 2 lines
and others will result in the user receiving a timestamp from the second to last skb that
tcp_recvmsgcopied from instead of the last.On Tx, the idea was to associate a timestamp with the last byte in the send buffer, so that a timestamp for this seqno informs us of the upper bound on latency of all bytes in the send buffer.
On Rx, we currently return the timestamp of the last skb of which the last byte is read, which is associated with a byte in the recv buffer, but it is not necessarily the last one. Nor the first. As such it is not clear what it defines.
Quoted 6 lines
Your patch addresses this by instead always returning the timestamp associated with the last byte in the recv buffer. The same timestamp could then be returned again for a subsequent recv call, if the entire recv buffer is filled from the same skb. Which is fine.
That sounds correct to me.
Due to my earlier comments the patch is no longer on patchwork. Can you please resubmit it.
But to be clear, the code looks good to me. Please add
The commit message can perhaps be a bit shorter. They key points are
1. the stated intent of the original commit is to "return the timestamp corresponding to the highest sequence number data returned."
2. the current implementation returns the timestamp for the last byte of the last fully read skb, which is not necessarily the last byte in the recv buffer.
3. that this patch converts behavior to the original definition.
Previous draft versions of the patch recorded the timestamp before label skip_copy, which also matches this behavior.
I took a quick look at the selftests under tools/testing/selftests/net, but they don't test for this specific behavior. Given that test code should make no assumptions on packetization, it is also not that straightforward to test in a robust manner.
Quoted 37 lines
Quoted 37 lines
The
proposed change addresses both problems while remaining consistent with 1.4 and the wording of commit98aaa913b4ed("tcp: ExtendSOF_TIMESTAMPING_RX_SOFTWAREto TCP recvmsg").Co-developed-by Iris Liu <[email protected]>Signed-off-by Iris Liu <[email protected]>Signed-off-by Kelly Littlepage <[email protected]>--- net/ipv4/tcp.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)net/ipv4/tcp.c +4
-3@@ -2154,13 +2154,15 @@ int tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int nonblock, tp->urg_data =0 ;tcp_fast_path_check (sk); } -if (used + offset < skb->len) -continue ;if (TCP_SKB_CB (skb)->has_rxtstamp) {tcp_update_recv_tstamps (skb, &tss); cmsg_flags |=2 ; } + +if (used + offset < skb->len) +continue ; +if (TCP_SKB_CB (skb)->tcp_flags &TCPHDR_FIN ) goto found_fin_ok;if (!(flags &MSG_PEEK )) --2.26.2
PATCH v2 1/1 net: tcp: fixes commit 98aaa913b4ed ("tcp: Extend SOF_TIMESTAMPING_RX_SOFTWARE to TCP recvmsg")
The stated intent of the original commit is to is to "return the timestamp corresponding to the highest sequence number data returned." The current implementation returns the timestamp for the last byte of the last fully read skb, which is not necessarily the last byte in the recv buffer. This patch converts behavior to the original definition, and to the behavior of the previous draft versions of commit 98aaa913b4ed ("tcp: Extend SOF_TIMESTAMPING_RX_SOFTWARE to TCP recvmsg") which also match this behavior.
---
Thanks and credit to Willem de Bruijn for the revised commit language
net/ipv4/tcp.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
net/ipv4/tcp.c +4 -2
@@ -2154,13 +2154,15 @@ int tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int nonblock, tp->urg_data =0 ;tcp_fast_path_check (sk); } -if (used + offset < skb->len) -continue ;if (TCP_SKB_CB (skb)->has_rxtstamp) {tcp_update_recv_tstamps (skb, &tss); cmsg_flags |=2 ; } + +if (used + offset < skb->len) +continue ; +if (TCP_SKB_CB (skb)->tcp_flags &TCPHDR_FIN ) goto found_fin_ok;if (!(flags &MSG_PEEK ))
Signature · 11 lines
2.26.2 -- This email and any attachments thereto may contain private, confidential, and privileged material for the sole use of the intended recipient. If you are not the intended recipient or otherwise believe that you have received this message in error, please notify the sender immediately and delete the original. Any review, copying, or distribution of this email (or any attachments thereto) by others is strictly prohibited. If this message was misdirected, OCX Group Inc. does not waive any confidentiality or privilege.
Re: [PATCH v2] net: tcp: fixes commit 98aaa913b4ed ("tcp: Extend SOF_TIMESTAMPING_RX_SOFTWARE to TCP recvmsg")
Quoted 10 lines · Kelly Littlepage
The stated intent of the original commit is to is to "return the timestamp corresponding to the highest sequence number data returned." The current implementation returns the timestamp for the last byte of the last fully read skb, which is not necessarily the last byte in the recv buffer. This patch converts behavior to the original definition, and to the behavior of the previous draft versions of commit
98aaa913b4ed("tcp: ExtendSOF_TIMESTAMPING_RX_SOFTWAREto TCP recvmsg") which also match this behavior.
Quoted 5 lines
Co-developed-by Iris Liu <[email protected]>Signed-off-by Iris Liu <[email protected]>Signed-off-by Kelly Littlepage <[email protected]>---
Thanks and credit to Willem de Bruijn for the revised commit language
Thanks for resubmitting. I did not mean to put the Fixes tag in the subject line.
The Fixes tag goes at the top of the block of signs-offs. If unclear, please look at a couple of examples on the mailing list or in git log.
The existing subject from v1 was fine. It is now too long. Could you resubmit a v3?
Thanks
Quoted 38 lines
net/ipv4/tcp.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)net/ipv4/tcp.c +4
-3@@ -2154,13 +2154,15 @@ int tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int nonblock, tp->urg_data =0 ;tcp_fast_path_check (sk); } -if (used + offset < skb->len) -continue ;if (TCP_SKB_CB (skb)->has_rxtstamp) {tcp_update_recv_tstamps (skb, &tss); cmsg_flags |=2 ; } + +if (used + offset < skb->len) +continue ; +if (TCP_SKB_CB (skb)->tcp_flags &TCPHDR_FIN ) goto found_fin_ok;if (!(flags &MSG_PEEK )) --2.26.2
--
This email and any attachments thereto may contain private, confidential, and privileged material for the sole use of the intended recipient. If you are not the intended recipient or otherwise believe that you have received this message in error, please notify the sender immediately and delete the original. Any review, copying, or distribution of this email (or any attachments thereto) by others is strictly prohibited. If this message was misdirected, OCX Group Inc. does not waive any confidentiality or privilege.
PATCH v3 1/1 net: tcp: fix rx timestamp behavior for tcp_recvmsg
The stated intent of the original commit is to is to "return the timestamp corresponding to the highest sequence number data returned." The current implementation returns the timestamp for the last byte of the last fully read skb, which is not necessarily the last byte in the recv buffer. This patch converts behavior to the original definition, and to the behavior of the previous draft versions of commit 98aaa913b4ed ("tcp: Extend SOF_TIMESTAMPING_RX_SOFTWARE to TCP recvmsg") which also match this behavior.
---
Reverted to the original subject line
net/ipv4/tcp.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
net/ipv4/tcp.c +4 -2
@@ -2154,13 +2154,15 @@ int tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int nonblock, tp->urg_data =0 ;tcp_fast_path_check (sk); } -if (used + offset < skb->len) -continue ;if (TCP_SKB_CB (skb)->has_rxtstamp) {tcp_update_recv_tstamps (skb, &tss); cmsg_flags |=2 ; } + +if (used + offset < skb->len) +continue ; +if (TCP_SKB_CB (skb)->tcp_flags &TCPHDR_FIN ) goto found_fin_ok;if (!(flags &MSG_PEEK ))
Signature · 11 lines
2.26.2 -- This email and any attachments thereto may contain private, confidential, and privileged material for the sole use of the intended recipient. If you are not the intended recipient or otherwise believe that you have received this message in error, please notify the sender immediately and delete the original. Any review, copying, or distribution of this email (or any attachments thereto) by others is strictly prohibited. If this message was misdirected, OCX Group Inc. does not waive any confidentiality or privilege.
Quoted 9 lines · Kelly Littlepage
The stated intent of the original commit is to is to "return the timestamp corresponding to the highest sequence number data returned." The current implementation returns the timestamp for the last byte of the last fully read skb, which is not necessarily the last byte in the recv buffer. This patch converts behavior to the original definition, and to the behavior of the previous draft versions of commit
98aaa913b4ed("tcp: ExtendSOF_TIMESTAMPING_RX_SOFTWAREto TCP recvmsg") which also match this behavior.
Quoted 6 lines
Fixes 98aaa913b4ed ("tcp: Extend SOF_TIMESTAMPING_RX_SOFTWARE to TCP recvmsg")Co-developed-by Iris Liu <[email protected]>Signed-off-by Iris Liu <[email protected]>Signed-off-by Kelly Littlepage <[email protected]>---
Reverted to the original subject line
SGTM, thanks.
Soheil Hassas Yeganeh
Acked-byQuoted 21 lines · Eric Dumazet
Quoted 9 lines · Kelly Littlepage
The stated intent of the original commit is to is to "return the timestamp corresponding to the highest sequence number data returned." The current implementation returns the timestamp for the last byte of the last fully read skb, which is not necessarily the last byte in the recv buffer. This patch converts behavior to the original definition, and to the behavior of the previous draft versions of commit
98aaa913b4ed("tcp: ExtendSOF_TIMESTAMPING_RX_SOFTWAREto TCP recvmsg") which also match this behavior.Quoted 6 lines
Fixes 98aaa913b4ed ("tcp: Extend SOF_TIMESTAMPING_RX_SOFTWARE to TCP recvmsg")Co-developed-by Iris Liu <[email protected]>Signed-off-by Iris Liu <[email protected]>Signed-off-by Kelly Littlepage <[email protected]>---
Reverted to the original subject line
Quoted 3 lines
SGTM, thanks.
Signed-off-by Eric Dumazet <[email protected]>
Willem de Bruijn
Acked-byQuoted 23 lines · Soheil Hassas Yeganeh
Quoted 21 lines · Eric Dumazet
Quoted 15 lines · Kelly Littlepage
The stated intent of the original commit is to is to "return the timestamp corresponding to the highest sequence number data returned." The current implementation returns the timestamp for the last byte of the last fully read skb, which is not necessarily the last byte in the recv buffer. This patch converts behavior to the original definition, and to the behavior of the previous draft versions of commit
98aaa913b4ed("tcp: ExtendSOF_TIMESTAMPING_RX_SOFTWAREto TCP recvmsg") which also match this behavior.Fixes 98aaa913b4ed ("tcp: Extend SOF_TIMESTAMPING_RX_SOFTWARE to TCP recvmsg")Co-developed-by Iris Liu <[email protected]>Signed-off-by Iris Liu <[email protected]>Signed-off-by Kelly Littlepage <[email protected]>---
Reverted to the original subject line
Quoted 5 lines
Quoted 3 lines
SGTM, thanks.
Signed-off-by Eric Dumazet <[email protected]>
Jakub Kicinski
maintainerQuoted 2 lines · On Fri, 8 May 2020 14:23:10 +0000 Kelly Littlepage wrote:
Any review, copying, or distribution of this email (or any attachments thereto) by others is strictly prohibited.
I'm afraid you'll have to do something about this footer if you want the patch to be applied.. Is sending from a different email an option?
(please make sure to add the review and ack tags you received from folks to your commit before reposting)
PATCH v4 1/1 net: tcp: fix rx timestamp behavior for tcp_recvmsg
From: Kelly Littlepage <[email protected]>
The stated intent of the original commit is to is to "return the timestamp corresponding to the highest sequence number data returned." The current implementation returns the timestamp for the last byte of the last fully read skb, which is not necessarily the last byte in the recv buffer. This patch converts behavior to the original definition, and to the behavior of the previous draft versions of commit 98aaa913b4ed ("tcp: Extend SOF_TIMESTAMPING_RX_SOFTWARE to TCP recvmsg") which also match this behavior.
---
Sending from an alternative email given the compliance footer that's unavoidably attached to all emails from my primary account. The patch itself is unchanged.
net/ipv4/tcp.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
net/ipv4/tcp.c +4 -2
@@ -2154,13 +2154,15 @@ int tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int nonblock, tp->urg_data =0 ;tcp_fast_path_check (sk); } -if (used + offset < skb->len) -continue ;if (TCP_SKB_CB (skb)->has_rxtstamp) {tcp_update_recv_tstamps (skb, &tss); cmsg_flags |=2 ; } + +if (used + offset < skb->len) +continue ; +if (TCP_SKB_CB (skb)->tcp_flags &TCPHDR_FIN ) goto found_fin_ok;if (!(flags &MSG_PEEK ))
Signature · 1 line
2.26.2
Quoted 10 lines · On Fri, 8 May 2020 19:58:46 +0000 Kelly Littlepage wrote:
From: Kelly Littlepage <[email protected]>
The stated intent of the original commit is to is to "return the timestamp corresponding to the highest sequence number data returned." The current implementation returns the timestamp for the last byte of the last fully read skb, which is not necessarily the last byte in the recv buffer. This patch converts behavior to the original definition, and to the behavior of the previous draft versions of commit
98aaa913b4ed("tcp: ExtendSOF_TIMESTAMPING_RX_SOFTWAREto TCP recvmsg") which also match this behavior.
Applied, thank you!