tcp: fix ICMP-RTO war
Subsystems (3)
PATCH v4 1/1 tcp: fix ICMP-RTO war
Make sure, that TCP has a nonzero RTT estimation after three-way handshake. Currently, a listening TCP has a value of 0 for srtt, rttvar and rto right after the three-way handshake is completed with TCP timestamps disabled.
This will lead to corrupt RTO recalculation and retransmission flood when RTO is recalculated on backoff reversion as introduced in "Revert RTO on ICMP destination unreachable" (f1ecd5d9e7366609d640ff4040304ea197fbc618).
This behaviour can be provoked by connecting to a server which "responds first" (like SMTP) and rejecting every packet after the handshake with dest-unreachable, which will lead to softirq load on the server (up to 30% per socket in some tests).
Thanks to Ilpo Jarvinen for providing debug patches and to Denys Fedoryshchenko for reporting and testing.
Changes since v3: Removed bad characters in patchfile.
--- net/ipv4/tcp_input.c | 8 +++----- 1 files changed, 3 insertions(+), 5 deletions(-)
net/ipv4/tcp_input.c +3 -5
@@ -5783,12 +5783,10 @@ int tcp_rcv_state_process(struct sock *sk, struct sk_buff *skb,/* tcp_ack considers this ACK as duplicate * and does not calculate rtt. -* Fix it at least with timestamps. +* Force it here. */ -if (tp->rx_opt.saw_tstamp && - tp->rx_opt.rcv_tsecr && !tp->srtt) -tcp_ack_saw_tstamp (sk,0 ); - +tcp_ack_update_rtt (sk,0 ,0 ); +if (tp->rx_opt.tstamp_ok) tp->advmss -=TCPOLEN_TSTAMP_ALIGNED ;
-- 1.6.4.4
David Miller
maintainerFrom: Damian Lukowski <[email protected]>
Date: Wed, 10 Feb 2010 20:06:42 +0100
Quoted 13 lines
Make sure, that TCP has a nonzero RTT estimation after three-way handshake. Currently, a listening TCP has a value of 0 for srtt, rttvar and rto right after the three-way handshake is completed with TCP timestamps disabled.
This will lead to corrupt RTO recalculation and retransmission flood when RTO is recalculated on backoff reversion as introduced in "Revert RTO on ICMP destination unreachable" (f1ecd5d9e7366609d640ff4040304ea197fbc618).
This behaviour can be provoked by connecting to a server which "responds first" (like SMTP) and rejecting every packet after the handshake with dest-unreachable, which will lead to softirq load on the server (up to 30% per socket in some tests).
Quoted 7 lines
Thanks to Ilpo Jarvinen for providing debug patches and to Denys Fedoryshchenko for reporting and testing.
Changes since v3: Removed bad characters in patchfile.
Reported-by Denys Fedoryshchenko <[email protected]>Signed-off-by Damian Lukowski <[email protected]>
Thanks for doing this work, I'll study these issues and review this patch today.
From: Damian Lukowski <[email protected]>
Date: Wed, 10 Feb 2010 20:06:42 +0100
Quoted 13 lines
Make sure, that TCP has a nonzero RTT estimation after three-way handshake. Currently, a listening TCP has a value of 0 for srtt, rttvar and rto right after the three-way handshake is completed with TCP timestamps disabled.
This will lead to corrupt RTO recalculation and retransmission flood when RTO is recalculated on backoff reversion as introduced in "Revert RTO on ICMP destination unreachable" (f1ecd5d9e7366609d640ff4040304ea197fbc618).
This behaviour can be provoked by connecting to a server which "responds first" (like SMTP) and rejecting every packet after the handshake with dest-unreachable, which will lead to softirq load on the server (up to 30% per socket in some tests).
Quoted 7 lines
Thanks to Ilpo Jarvinen for providing debug patches and to Denys Fedoryshchenko for reporting and testing.
Changes since v3: Removed bad characters in patchfile.
Reported-by Denys Fedoryshchenko <[email protected]>Signed-off-by Damian Lukowski <[email protected]>
Applied, thank you very much for fixing this bug.
First of all I want to let you know that I've no objection to this fix itself (DaveM applied it already), it certainly does it's jobs in preventing invalid < RTO_MIN state. However, I wonder if we could do further improvements in this area...
Quoted 31 lines · Damian Lukowski
Make sure, that TCP has a nonzero RTT estimation after three-way handshake. Currently, a listening TCP has a value of 0 for srtt, rttvar and rto right after the three-way handshake is completed with TCP timestamps disabled.
This will lead to corrupt RTO recalculation and retransmission flood when RTO is recalculated on backoff reversion as introduced in "Revert RTO on ICMP destination unreachable" (f1ecd5d9e7366609d640ff4040304ea197fbc618).
This behaviour can be provoked by connecting to a server which "responds first" (like SMTP) and rejecting every packet after the handshake with dest-unreachable, which will lead to softirq load on the server (up to 30% per socket in some tests).Thanks to Ilpo Jarvinen for providing debug patches and to Denys Fedoryshchenko for reporting and testing.
Changes since v3: Removed bad characters in patchfile.
Reported-by Denys Fedoryshchenko <[email protected]>Signed-off-by Damian Lukowski <[email protected]>--- net/ipv4/tcp_input.c | 8 +++----- 1 files changed, 3 insertions(+), 5 deletions(-)net/ipv4/tcp_input.c +0
-0@@ -5783,12 +5783,10 @@ int tcp_rcv_state_process(struct sock *sk, struct sk_buff *skb,/* tcp_ack considers this ACK as duplicate
net/ipv4/tcp_input.c int tcp_rcv_state_process(struct sock *sk, struct sk_buff *skb, · line 5789
Quoted 10 lines
net/ipv4/tcp_input.c +3
-5* and does not calculate rtt. -* Fix it at least with timestamps. +* Force it here. */ -if (tp->rx_opt.saw_tstamp && - tp->rx_opt.rcv_tsecr && !tp->srtt) -tcp_ack_saw_tstamp (sk,0 ); - +tcp_ack_update_rtt (sk,0 ,0 ); +
...Here a zero seq_rtt is given to RTT estimator (it will be effective only in the case w/o timestamps, TS case recalculates it from the stored timestamps). Maybe we could use some field (timestamp related one comes to my mind) in request sock to get a real RTT estimate for non-timestamp case too. ...It seems possible to me, though tricky because the request_sock is no longer that easily available here so some parameter passing would be needed.
Signature · 1 line
i.
David Miller
maintainerFrom: "Ilpo Järvinen" <[email protected]>
Date: Tue, 16 Feb 2010 14:45:25 +0200 (EET)
Quoted 16 lines · Damian Lukowski
Quoted 4 lines
net/ipv4/tcp_input.c +0
-0@@ -5783,12 +5783,10 @@ int tcp_rcv_state_process(struct sock *sk, struct sk_buff *skb,/* tcp_ack considers this ACK as duplicate net/ipv4/tcp_input.c int tcp_rcv_state_process(struct sock *sk, struct sk_buff *skb, · line 5789
Quoted 10 lines
net/ipv4/tcp_input.c +3
-5* and does not calculate rtt. -* Fix it at least with timestamps. +* Force it here. */ -if (tp->rx_opt.saw_tstamp && - tp->rx_opt.rcv_tsecr && !tp->srtt) -tcp_ack_saw_tstamp (sk,0 ); - +tcp_ack_update_rtt (sk,0 ,0 ); +
Quoted 7 lines
...Here a zero
seq_rttis given to RTT estimator (it will be effective only in the case w/o timestamps, TS case recalculates it from the stored timestamps). Maybe we could use some field (timestamp related one comes to my mind) in request sock to get a real RTT estimate for non-timestamp case too. ...It seems possible to me, though tricky because therequest_sockis no longer that easily available here so some parameter passing would be needed.
Agreed.
But even more simply I think we should make even the current tcp_ack_update_rtt() call here conditional on at least tp->srtt being zero.
Damian do you at least agree with that?
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi,
I'm working on a patch that tries to measure and use the RTT for the passive open side when the TS option is NOT enabled. My code conflicts with your recently added "tcp_ack_update_rtt(sk, 0, 0);" Could you tell me why do you force this call for the no-TS case when obviously "0" is not a measured RTT? If you try to force icsk_rto to be initialized correctly, it is already initialized toTCP_TIMEOUT_INIT by tcp_create_openreq_child(). What am I missing?
Thanks,
Jerry
Quoted 46 lines
From: David Miller <[email protected]>
Date: Sun, Feb 21, 2010 at 7:10 PM
Subject: Re: [PATCH][v4] tcp: fix ICMP-RTO war
To: [email protected]
Cc: [email protected], [email protected]From: "Ilpo Järvinen" <[email protected]>
Date: Tue, 16 Feb 2010 14:45:25 +0200 (EET)Quoted 16 lines · Damian Lukowski
Quoted 4 lines
net/ipv4/tcp_input.c +0
-0@@ -5783,12 +5783,10 @@ int tcp_rcv_state_process(struct sock *sk, struct sk_buff *skb, /* tcp_ack considers this ACK as duplicatenet/ipv4/tcp_input.c int tcp_rcv_state_process(struct sock *sk, struct sk_buff *skb, · line 5789
Quoted 10 lines
net/ipv4/tcp_input.c +3
-5* and does not calculate rtt. -* Fix it at least with timestamps. +* Force it here. */ - if (tp->rx_opt.saw_tstamp && - tp->rx_opt.rcv_tsecr && !tp->srtt) - tcp_ack_saw_tstamp(sk, 0); - + tcp_ack_update_rtt(sk, 0, 0); +Quoted 7 lines
...Here a zero
seq_rttis given to RTT estimator (it will be effective only in the case w/o timestamps, TS case recalculates it from the stored timestamps). Maybe we could use some field (timestamp related one comes to my mind) in request sock to get a real RTT estimate for non-timestamp case too. ...It seems possible to me, though tricky because therequest_sockis no longer that easily available here so some parameter passing would be needed.Agreed.
But even more simply I think we should make even the current
tcp_ack_update_rtt()call here conditional on at leasttp->srttbeing zero.Damian do you at least agree with that?
--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines
I'm working on a patch that tries to measure and use the RTT for the passive open side when the TS option is NOT enabled. My code conflicts with your recently added "
tcp_ack_update_rtt(sk, 0, 0);" Could you tell me why do you force this call for the no-TS case when obviously "0" is not a measured RTT? If you try to forceicsk_rtoto be initialized correctly, it is already initialized toTCP_TIMEOUT_INITbytcp_create_openreq_child(). What am I missing?
Hi,
the backoff reversion code uses __tcp_set_rto() to recalculate icsk_rto, which itself relies on tp->srtt and rttvar.
srtt is explicitly set to 0 in tcp_create_openreq_child(), so I didn't change it. Initializing it with TCP_TIMEOUT_INIT should also fix that specific bug, but I don't know if there are other impacts.
Regards
Damian
Quoted 50 lines
Thanks,
Jerry
Quoted 46 lines
From: David Miller <[email protected]>
Date: Sun, Feb 21, 2010 at 7:10 PM
Subject: Re: [PATCH][v4] tcp: fix ICMP-RTO war
To: [email protected]
Cc: [email protected], [email protected]From: "Ilpo Järvinen" <[email protected]>
Date: Tue, 16 Feb 2010 14:45:25 +0200 (EET)Quoted 16 lines · Damian Lukowski
Quoted 4 lines
net/ipv4/tcp_input.c +0
-0@@ -5783,12 +5783,10 @@ int tcp_rcv_state_process(struct sock *sk, struct sk_buff *skb,/* tcp_ack considers this ACK as duplicate net/ipv4/tcp_input.c int tcp_rcv_state_process(struct sock *sk, struct sk_buff *skb, · line 5789
Quoted 10 lines
net/ipv4/tcp_input.c +3
-5* and does not calculate rtt. -* Fix it at least with timestamps. +* Force it here. */ -if (tp->rx_opt.saw_tstamp && - tp->rx_opt.rcv_tsecr && !tp->srtt) -tcp_ack_saw_tstamp (sk,0 ); - +tcp_ack_update_rtt (sk,0 ,0 ); +Quoted 7 lines
...Here a zero
seq_rttis given to RTT estimator (it will be effective only in the case w/o timestamps, TS case recalculates it from the stored timestamps). Maybe we could use some field (timestamp related one comes to my mind) in request sock to get a real RTT estimate for non-timestamp case too. ...It seems possible to me, though tricky because therequest_sockis no longer that easily available here so some parameter passing would be needed.Agreed.
But even more simply I think we should make even the current
tcp_ack_update_rtt()call here conditional on at leasttp->srttbeing zero.Damian do you at least agree with that?
--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 9 lines · Damian Lukowski
Quoted 8 lines
I'm working on a patch that tries to measure and use the RTT for the passive open side when the TS option is NOT enabled. My code conflicts with your recently added "
tcp_ack_update_rtt(sk, 0, 0);" Could you tell me why do you force this call for the no-TS case when obviously "0" is not a measured RTT? If you try to forceicsk_rtoto be initialized correctly, it is already initialized toTCP_TIMEOUT_INITbytcp_create_openreq_child(). What am I missing?
Quoted 3 lines
Hi,
the backoff reversion code uses__tcp_set_rto()to recalculateicsk_rto, which itself relies ontp->srttand rttvar.
Guess you are talking about
inet_csk(sk)->icsk_rto = __tcp_set_rto(tp) << icsk->icsk_backoff;
inside tcp_v4_err(), right? (I'm looking at 2.6.33 kernel.)
Yes it seems to be a bug when __tcp_set_rto() is called before tcp_rtt_estimator()
gets a chance to initialize all the variables properly.
But I don't like your fix of adding tcp_ack_update_rtt(sk, 0, 0); to tcp_rcv_state_process()
because that means you've got a measured RTT of 0 (really meaning < 1 tick) for the no-TS case, which will cause tcp_rtt_estimator() to compute all the variables as if
there has been a valid RTT measurement of 1.
A better fix IMHO is to make sure all the variables are properly initialized when exitingtcp_init_metrics(), e.g, if srtt remains 0, make sure tp->mdev = tp->mdev_max = tp->rttvar = TCP_TIMEOUT_INIT; (mdev already been initialized to TCP_TIMEOUT_INIT. I think you got hit by rttvar == 0)
Quoted 3 lines
srtt is explicitly set to 0 in
tcp_create_openreq_child(), so I didn't change it. Initializing it withTCP_TIMEOUT_INITshould also fix that specific bug, but I don't know if there are other impacts.
So what do I care? Because I'm mucking with the code in this area and your fix causes some conflict with my logic.
What do you think?
Best,
Jerry
Quoted 55 lines
Regards
DamianQuoted 50 lines
Thanks,
Jerry
Quoted 46 lines
From: David Miller <[email protected]>
Date: Sun, Feb 21, 2010 at 7:10 PM
Subject: Re: [PATCH][v4] tcp: fix ICMP-RTO war
To: [email protected]
Cc: [email protected], [email protected]From: "Ilpo Järvinen" <[email protected]>
Date: Tue, 16 Feb 2010 14:45:25 +0200 (EET)Quoted 16 lines · Damian Lukowski
Quoted 4 lines
net/ipv4/tcp_input.c +0
-0@@ -5783,12 +5783,10 @@ int tcp_rcv_state_process(struct sock *sk, struct sk_buff *skb, /* tcp_ack considers this ACK as duplicatenet/ipv4/tcp_input.c int tcp_rcv_state_process(struct sock *sk, struct sk_buff *skb, · line 5789
Quoted 10 lines
net/ipv4/tcp_input.c +3
-5* and does not calculate rtt. -* Fix it at least with timestamps. +* Force it here. */ - if (tp->rx_opt.saw_tstamp && - tp->rx_opt.rcv_tsecr && !tp->srtt) - tcp_ack_saw_tstamp(sk, 0); - + tcp_ack_update_rtt(sk, 0, 0); +Quoted 7 lines
...Here a zero
seq_rttis given to RTT estimator (it will be effective only in the case w/o timestamps, TS case recalculates it from the stored timestamps). Maybe we could use some field (timestamp related one comes to my mind) in request sock to get a real RTT estimate for non-timestamp case too. ...It seems possible to me, though tricky because therequest_sockis no longer that easily available here so some parameter passing would be needed.Agreed.
But even more simply I think we should make even the current
tcp_ack_update_rtt()call here conditional on at leasttp->srttbeing zero.Damian do you at least agree with that?
--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 40 lines · Jerry Chu
Quoted 9 lines · Damian Lukowski
Quoted 8 lines
I'm working on a patch that tries to measure and use the RTT for the passive open side when the TS option is NOT enabled. My code conflicts with your recently added "
tcp_ack_update_rtt(sk, 0, 0);" Could you tell me why do you force this call for the no-TS case when obviously "0" is not a measured RTT? If you try to forceicsk_rtoto be initialized correctly, it is already initialized toTCP_TIMEOUT_INITbytcp_create_openreq_child(). What am I missing?Quoted 3 lines
Hi,
the backoff reversion code uses__tcp_set_rto()to recalculateicsk_rto, which itself relies ontp->srttand rttvar.Guess you are talking about
inet_csk(sk)->icsk_rto =
__tcp_set_rto(tp)<<icsk->icsk_backoff;inside
tcp_v4_err(), right? (I'm looking at 2.6.33 kernel.)Yes it seems to be a bug when
__tcp_set_rto()is called beforetcp_rtt_estimator()
gets a chance to initialize all the variables properly.But I don't like your fix of adding
tcp_ack_update_rtt(sk, 0, 0); totcp_rcv_state_process()
because that means you've got a measured RTT of 0 (really meaning < 1 tick) for the no-TS case, which will causetcp_rtt_estimator()to compute all the variables as if
there has been a valid RTT measurement of 1.A better fix IMHO is to make sure all the variables are properly initialized when exiting
tcp_init_metrics(), e.g, if srtt remains 0, make suretp->mdev=tp->mdev_max=tp->rttvar=TCP_TIMEOUT_INIT; (mdev already been initialized toTCP_TIMEOUT_INIT. I think you got hit by rttvar == 0)
Quoted 8 lines
Quoted 3 lines
srtt is explicitly set to 0 in
tcp_create_openreq_child(), so I didn't change it. Initializing it withTCP_TIMEOUT_INITshould also fix that specific bug, but I don't know if there are other impacts.So what do I care? Because I'm mucking with the code in this area and your fix causes some conflict with my logic.
What do you think?
I didn't/don't like the fix either, but it's better than ICMP-RTO war that it fixed.
I think that most sensible solution to this issue would not be to initialize with the TCP_TIMEOUT_INIT bogosity either but do the real measurement for rtt which should be possible already. I suggested this already when the fix was made but nobody has yet found the time and energy to code the rtt measurement for the early rtt when not using TS. Since you're going to make that lack of feature to go away, please change this illogical call too then while you go :-).
Signature · 1 line
i.