tcp: rearm RTO timer does not comply with RFC6298
Subsystems (2)
hi,
I have a doubt about tcp_rearm_rto().
Early TCP always rearm the RTO timer to NOW+RTO when it receives an ACK that acknowledges new data.
Referring to RFC6298 SECTION 5.3: "When an ACK is received that acknowledges new data, restart the retransmission timer so that it will expire after RTO seconds (for the current value of RTO)."
After ER and TLP, we rearm the RTO timer to *tstamp_of_head+RTO* when switching from ER/TLP/RACK to original RTO in tcp_rearm_rto(), in this case the RTO timer is triggered earlier than described in RFC6298, otherwise the same.
Is this planned? Or can we always rearm the RTO timer to tstamp_of_head+RTO?
Thanks.
Quoted 17 lines · Pengcheng Yang
hi,
I have a doubt about
tcp_rearm_rto().Early TCP always rearm the RTO timer to NOW+RTO when it receives an ACK that acknowledges new data.
Referring to RFC6298 SECTION 5.3: "When an ACK is received that acknowledges new data, restart the retransmission timer so that it will expire after RTO seconds (for the current value of RTO)."
After ER and TLP, we rearm the RTO timer to *tstamp_of_head+RTO* when switching from ER/TLP/RACK to original RTO in
tcp_rearm_rto(), in this case the RTO timer is triggered earlier than described in RFC6298, otherwise the same.
Quoted 5 lines
Is this planned? Or can we always rearm the RTO timer to tstamp_of_head+RTO?
Thanks.
This is a good question. As far as I can tell, this difference in behavior would only come into play in a few corner cases, like:
(1) The TLP timer fires and the connection is unable to transmit a TLP probe packet. This could happen due to memory allocation failure or the local qdisc being full.
(2) The RACK reorder timer fires but the connection does not take the normal course of action and mark some packets lost and retransmit at least one of them. I'm not sure how this would happen. Maybe someone can think of a case.
My sense would be that given how relatively rare (1)/(2) are, it is probably not worth changing the current behavior, given that it seems it would require extra state (an extra u32 snd_una_advanced_tstamp? ) to save the time at which snd_una advanced (a cumulative ACK covered some data) in order to rearm the RTO timer for snd_una_advanced_tstamp
+ rto.
neal
Quoted 37 lines · Neal Cardwell
Quoted 17 lines · Pengcheng Yang
hi,
I have a doubt about
tcp_rearm_rto().Early TCP always rearm the RTO timer to NOW+RTO when it receives an ACK that acknowledges new data.
Referring to RFC6298 SECTION 5.3: "When an ACK is received that acknowledges new data, restart the retransmission timer so that it will expire after RTO seconds (for the current value of RTO)."
After ER and TLP, we rearm the RTO timer to *tstamp_of_head+RTO* when switching from ER/TLP/RACK to original RTO in
tcp_rearm_rto(), in this case the RTO timer is triggered earlier than described in RFC6298, otherwise the same.Quoted 5 lines
Is this planned? Or can we always rearm the RTO timer to tstamp_of_head+RTO?
Thanks.
This is a good question. As far as I can tell, this difference in behavior would only come into play in a few corner cases, like:
(1) The TLP timer fires and the connection is unable to transmit a TLP probe packet. This could happen due to memory allocation failure or the local qdisc being full.
(2) The RACK reorder timer fires but the connection does not take the normal course of action and mark some packets lost and retransmit at least one of them. I'm not sure how this would happen. Maybe someone can think of a case.
Quoted 6 lines
My sense would be that given how relatively rare (1)/(2) are, it is probably not worth changing the current behavior, given that it seems it would require extra state (an extra u32
snd_una_advanced_tstamp? ) to save the time at whichsnd_unaadvanced (a cumulative ACK covered some data) in order to rearm the RTO timer forsnd_una_advanced_tstamp
+ rto.
also there's an experimental proposal
https://tools.ietf.org/html/rfc7765
so Linux actually implements that in a limited way that only applies in specific scenarios.
Quoted 2 lines
neal
Quoted 30 lines · Yuchung Cheng
Quoted 28 lines · Neal Cardwell
Quoted 17 lines · Pengcheng Yang
hi,
I have a doubt about
tcp_rearm_rto().Early TCP always rearm the RTO timer to NOW+RTO when it receives an ACK that acknowledges new data.
Referring to RFC6298 SECTION 5.3: "When an ACK is received that acknowledges new data, restart the retransmission timer so that it will expire after RTO seconds (for the current value of RTO)."
After ER and TLP, we rearm the RTO timer to *tstamp_of_head+RTO* when switching from ER/TLP/RACK to original RTO in
tcp_rearm_rto(), in this case the RTO timer is triggered earlier than described in RFC6298, otherwise the same.Quoted 5 lines
Is this planned? Or can we always rearm the RTO timer to tstamp_of_head+RTO?
Thanks.
This is a good question. As far as I can tell, this difference in behavior would only come into play in a few corner cases, like:
Quoted 8 lines
Quoted 8 lines
(1) The TLP timer fires and the connection is unable to transmit a TLP probe packet. This could happen due to memory allocation failure or the local qdisc being full.
(2) The RACK reorder timer fires but the connection does not take the normal course of action and mark some packets lost and retransmit at least one of them. I'm not sure how this would happen. Maybe someone can think of a case.
Yes, and it also happens when an ACK (a cumulative ACK covered out-of-order data) is received that makes ca_state change from DISORDER to OPEN, by calling tcp_set_xmit_timer(). Because TLP is not triggered under DISORDER and tcp_rearm_rto() is called before the ca_state changes.
Quoted 8 lines
Quoted 7 lines
My sense would be that given how relatively rare (1)/(2) are, it is probably not worth changing the current behavior, given that it seems it would require extra state (an extra u32
snd_una_advanced_tstamp? ) to save the time at whichsnd_unaadvanced (a cumulative ACK covered some data) in order to rearm the RTO timer forsnd_una_advanced_tstamp
+ rto.
Quoted 8 lines
also there's an experimental proposal
https://tools.ietf.org/html/rfc7765so Linux actually implements that in a limited way that only applies in specific scenarios.
Quoted 2 lines
neal
Thank you for answering my questions.
Quoted 41 lines · Pengcheng Yang
Quoted 30 lines · Yuchung Cheng
Quoted 28 lines · Neal Cardwell
Quoted 17 lines · Pengcheng Yang
hi,
I have a doubt about
tcp_rearm_rto().Early TCP always rearm the RTO timer to NOW+RTO when it receives an ACK that acknowledges new data.
Referring to RFC6298 SECTION 5.3: "When an ACK is received that acknowledges new data, restart the retransmission timer so that it will expire after RTO seconds (for the current value of RTO)."
After ER and TLP, we rearm the RTO timer to *tstamp_of_head+RTO* when switching from ER/TLP/RACK to original RTO in
tcp_rearm_rto(), in this case the RTO timer is triggered earlier than described in RFC6298, otherwise the same.Quoted 5 lines
Is this planned? Or can we always rearm the RTO timer to tstamp_of_head+RTO?
Thanks.
This is a good question. As far as I can tell, this difference in behavior would only come into play in a few corner cases, like:
Quoted 8 lines
Quoted 8 lines
(1) The TLP timer fires and the connection is unable to transmit a TLP probe packet. This could happen due to memory allocation failure or the local qdisc being full.
(2) The RACK reorder timer fires but the connection does not take the normal course of action and mark some packets lost and retransmit at least one of them. I'm not sure how this would happen. Maybe someone can think of a case.
Quoted 4 lines
Yes, and it also happens when an ACK (a cumulative ACK covered out-of-order data) is received that makes
ca_statechange from DISORDER to OPEN, by callingtcp_set_xmit_timer(). Because TLP is not triggered under DISORDER andtcp_rearm_rto()is called before theca_statechanges.
Hmm, that sounds like a good catch, and potentially a significant bug. Re-reading the code, it seems that you correctly identify that on an ACK when reordering is resolved (ca_state change from DISORDER to OPEN) we will not set a TLP timer for now+TLP_interval, but instead will set an RTO timer for rtx_head_tx_time+RTO (which could be very soon indeed, if RTTVAR is very low). Seems like that could cause spurious RTOs with connections that experience reordering with low RTT variance.
It seems like we should try to fix this. Perhaps by calling tcp_set_xmit_timer() only after we have settled on a final ca_state implied by this ACK (in this case, to allow DISORDER to be resolved to OPEN). Though that would require some careful surgery, since that would move the tcp_set_xmit_timer() call *after* the point at which the RACK reorder timer would be set.
Other thoughts?
neal
Quoted 64 lines · Neal Cardwell
Quoted 41 lines · Pengcheng Yang
Quoted 30 lines · Yuchung Cheng
Quoted 28 lines · Neal Cardwell
Quoted 17 lines · Pengcheng Yang
hi,
I have a doubt about
tcp_rearm_rto().Early TCP always rearm the RTO timer to NOW+RTO when it receives an ACK that acknowledges new data.
Referring to RFC6298 SECTION 5.3: "When an ACK is received that acknowledges new data, restart the retransmission timer so that it will expire after RTO seconds (for the current value of RTO)."
After ER and TLP, we rearm the RTO timer to *tstamp_of_head+RTO* when switching from ER/TLP/RACK to original RTO in
tcp_rearm_rto(), in this case the RTO timer is triggered earlier than described in RFC6298, otherwise the same.Quoted 5 lines
Is this planned? Or can we always rearm the RTO timer to tstamp_of_head+RTO?
Thanks.
This is a good question. As far as I can tell, this difference in behavior would only come into play in a few corner cases, like:
Quoted 8 lines
Quoted 8 lines
(1) The TLP timer fires and the connection is unable to transmit a TLP probe packet. This could happen due to memory allocation failure or the local qdisc being full.
(2) The RACK reorder timer fires but the connection does not take the normal course of action and mark some packets lost and retransmit at least one of them. I'm not sure how this would happen. Maybe someone can think of a case.
Quoted 4 lines
Yes, and it also happens when an ACK (a cumulative ACK covered out-of-order data) is received that makes
ca_statechange from DISORDER to OPEN, by callingtcp_set_xmit_timer(). Because TLP is not triggered under DISORDER andtcp_rearm_rto()is called before theca_statechanges.Hmm, that sounds like a good catch, and potentially a significant bug. Re-reading the code, it seems that you correctly identify that on an ACK when reordering is resolved (
ca_statechange from DISORDER to OPEN) we will not set a TLP timer for now+TLP_interval, but instead will set an RTO timer for rtx_head_tx_time+RTO (which could be very soon indeed, if RTTVAR is very low). Seems like that could cause spurious RTOs with connections that experience reordering with low RTT variance.It seems like we should try to fix this. Perhaps by calling
tcp_set_xmit_timer()only after we have settled on a finalca_stateimplied by this ACK (in this case, to allow DISORDER to be resolved to OPEN). Though that would require some careful surgery, since that would move thetcp_set_xmit_timer()call *after* the point at which the RACK reorder timer would be set.
Quoted 3 lines
Other thoughts?
neal
I think this fix is necessary. Actually, we also fixed this issue on our branch recently when we fixed an issue where the TLP timer might not fire(Point 1 below), by calling tcp_set_xmit_timer() after tcp_fastretrans_alert() and
then removing FLAG_SET_XMIT_TIMER from ack_flag when the RACK
reorder timer is set.
This repair has two additional benefits according to my understanding:
(1) When ca_state changes from DISORDER to OPEN, the TLP timer can be activated, otherwise, the sender can only wait for the RTO timer when it is app-limited.
(2) Reduce the xmit timer reschedule once per ACK when the RACK reorder timer is set.
I'll send this fix later, I'm not sure whether there is a more elegant way. : )