Skip to contentSkip to conversation

fix TSO FACK loss marking in tcp_mark_head_lost

PATCH Applied Yuchung Cheng 3 participants 5 messages Sep 24, 2010 – Sep 27, 2010

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 fix TSO FACK loss marking in tcp_mark_head_lost

When TCP uses FACK algorithm to mark lost packets in tcp_mark_head_lost(), if the number of packets in the (TSO) skb is greater than the number of packets that should be marked lost, TCP incorrectly exits the loop and marks no packets lost in the skb. This underestimates tp->lost_out and affects the recovery/retransmission. This patch fargments the skb and marks the correct amount of packets lost.

---
 net/ipv4/tcp_input.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)
net/ipv4/tcp_input.c +2 -1
index 1bc87a0..e4f472e 100644
@@ -2532,7 +2532,8 @@ static void tcp_mark_head_lost(struct sock *sk, int packets)
 			cnt += tcp_skb_pcount(skb);
 
 		if (cnt > packets) {
-			if (tcp_is_sack(tp) || (oldcnt >= packets))
+			if ((tcp_is_sack(tp) && !tcp_is_fack(tp)) ||
+			    (oldcnt >= packets))
 				break;
 
 			mss = skb_shinfo(skb)->gso_size;
Signature · 1 line
1.7.1
Acked-by on v1 1/1
#1 lore raw
Quoted 19 lines · Yuchung Cheng

When TCP uses FACK algorithm to mark lost packets in tcp_mark_head_lost(), if the number of packets in the (TSO) skb is greater than the number of packets that should be marked lost, TCP incorrectly exits the loop and marks no packets lost in the skb. This underestimates tp->lost_out and affects the recovery/retransmission. This patch fargments the skb and marks the correct amount of packets lost.

---
 net/ipv4/tcp_input.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)
net/ipv4/tcp_input.c +0 -0
index 1bc87a0..e4f472e 100644

net/ipv4/tcp_input.c static void tcp_mark_head_lost(struct sock *sk, int packets) · line 2539

Quoted 9 lines
net/ipv4/tcp_input.c +2 -1
@@ -2532,7 +2532,8 @@ static void tcp_mark_head_lost(struct sock *sk, int packets)
 			cnt += tcp_skb_pcount(skb);
 
 		if (cnt > packets) {
-			if (tcp_is_sack(tp) || (oldcnt >= packets))
+			if ((tcp_is_sack(tp) && !tcp_is_fack(tp)) ||
+			    (oldcnt >= packets))
 				break;
 
 			mss = skb_shinfo(skb)->gso_size;
Signature · 1 line
 i.
maintainer
#2 lore raw

From: "Ilpo Järvinen" <[email protected]>
Date: Mon, 27 Sep 2010 15:22:09 +0300 (EEST)

Quoted 28 lines · Yuchung Cheng
Quoted 27 lines

When TCP uses FACK algorithm to mark lost packets in tcp_mark_head_lost(), if the number of packets in the (TSO) skb is greater than the number of packets that should be marked lost, TCP incorrectly exits the loop and marks no packets lost in the skb. This underestimates tp->lost_out and affects the recovery/retransmission. This patch fargments the skb and marks the correct amount of packets lost.

---
 net/ipv4/tcp_input.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)
net/ipv4/tcp_input.c +2 -1
index 1bc87a0..e4f472e 100644
@@ -2532,7 +2532,8 @@ static void tcp_mark_head_lost(struct sock *sk, int packets)
 			cnt += tcp_skb_pcount(skb);
 
 		if (cnt > packets) {
-			if (tcp_is_sack(tp) || (oldcnt >= packets))
+			if ((tcp_is_sack(tp) && !tcp_is_fack(tp)) ||
+			    (oldcnt >= packets))
 				break;
 
Quoted 4 lines

net/ipv4/tcp_input.c static void tcp_mark_head_lost(struct sock *sk, int packets) · line 2539

Quoted 2 lines
net/ipv4/tcp_input.c +0 -0
 			mss = skb_shinfo(skb)->gso_size;

BTW, the history is that this code was added to fix a bug because we didn't handle GSO packets at all at one point.

But, conservatively, we didn't do splits here for SACK, and it was stated in the commit that we would look into it "at some point in the future" :-)

--------------------
commit c137f3dda04b0aee1bc6889cdc69185f53df8a82

Author: Ilpo Järvinen <[email protected]>
Date:   Mon Apr 7 22:32:38 2008 -0700

[TCP]: Fix NewReno's fast rexmit/recovery problems with GSOed skb

Fixes a long-standing bug which makes NewReno recovery crippled. With GSO the whole head skb was marked as LOST which is in violation of NewReno procedure that only wants to mark one packet and ended up breaking our TCP code by causing counter overflow because our code was built on top of assumption about valid NewReno procedure. This manifested as triggering a WARN_ON for the overflow in a number of places.

It seems relatively safe alternative to just do nothing if tcp_fragment fails due to oom because another duplicate ACK is likely to be received soon and the fragmentation will be retried.

Special thanks goes to Soeren Sonnenburg <[email protected]> who was lucky enough to be able to reproduce this so that the warning for the overflow was hit. It's not as easy task as it seems even if this bug happens quite often because the amount of outstanding data is pretty significant for the mismarkings to lead to an overflow.

Because it's very late in 2.6.25-rc cycle (if this even makes in time), I didn't want to touch anything with SACK enabled here. Fragmenting might be useful for it as well but it's more or less a policy decision rather than mandatory fix. Thus there's no need to rush and we can postpone considering tcp_fragment with SACK for 2.6.26.

In 2.6.24 and earlier, this very same bug existed but the effect is slightly different because of a small changes in the if conditions that fit to the patch's context. With them nothing got lost marker and thus no retransmissions happened.

--------------------

To be honest, we should probably just remove the whole tcp_is_sack() test, rather than special case FACK.

The cost isn't what it was when this code was added. Back then we didn't have Ilpo's restransmit queue coalescing code, so it would make retransmit queue packet freeing more expensive. But since the coalescing code is there now, splitting all the time to record accurate loss information should be fine.

Ilpo what do you say?

#3 lore raw
Quoted 88 lines · David Miller

From: "Ilpo Järvinen" <[email protected]>
Date: Mon, 27 Sep 2010 15:22:09 +0300 (EEST)

Quoted 28 lines · Yuchung Cheng
Quoted 27 lines

When TCP uses FACK algorithm to mark lost packets in tcp_mark_head_lost(), if the number of packets in the (TSO) skb is greater than the number of packets that should be marked lost, TCP incorrectly exits the loop and marks no packets lost in the skb. This underestimates tp->lost_out and affects the recovery/retransmission. This patch fargments the skb and marks the correct amount of packets lost.

---
 net/ipv4/tcp_input.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)
net/ipv4/tcp_input.c +2 -1
index 1bc87a0..e4f472e 100644
@@ -2532,7 +2532,8 @@ static void tcp_mark_head_lost(struct sock *sk, int packets)
 			cnt += tcp_skb_pcount(skb);
 
 		if (cnt > packets) {
-			if (tcp_is_sack(tp) || (oldcnt >= packets))
+			if ((tcp_is_sack(tp) && !tcp_is_fack(tp)) ||
+			    (oldcnt >= packets))
 				break;
 
Quoted 4 lines

net/ipv4/tcp_input.c static void tcp_mark_head_lost(struct sock *sk, int packets) · line 2539

Quoted 2 lines
net/ipv4/tcp_input.c +0 -0
 			mss = skb_shinfo(skb)->gso_size;

BTW, the history is that this code was added to fix a bug because we didn't handle GSO packets at all at one point.

But, conservatively, we didn't do splits here for SACK, and it was stated in the commit that we would look into it "at some point in the future" :-)

--------------------
commit c137f3dda04b0aee1bc6889cdc69185f53df8a82

Author: Ilpo Järvinen <[email protected]>
Date:   Mon Apr 7 22:32:38 2008 -0700

[TCP]: Fix NewReno's fast rexmit/recovery problems with GSOed skb

Fixes a long-standing bug which makes NewReno recovery crippled. With GSO the whole head skb was marked as LOST which is in violation of NewReno procedure that only wants to mark one packet and ended up breaking our TCP code by causing counter overflow because our code was built on top of assumption about valid NewReno procedure. This manifested as triggering a WARN_ON for the overflow in a number of places.

It seems relatively safe alternative to just do nothing if tcp_fragment fails due to oom because another duplicate ACK is likely to be received soon and the fragmentation will be retried.

Special thanks goes to Soeren Sonnenburg <[email protected]> who was lucky enough to be able to reproduce this so that the warning for the overflow was hit. It's not as easy task as it seems even if this bug happens quite often because the amount of outstanding data is pretty significant for the mismarkings to lead to an overflow.

Because it's very late in 2.6.25-rc cycle (if this even makes in time), I didn't want to touch anything with SACK enabled here. Fragmenting might be useful for it as well but it's more or less a policy decision rather than mandatory fix. Thus there's no need to rush and we can postpone considering tcp_fragment with SACK for 2.6.26.

In 2.6.24 and earlier, this very same bug existed but the effect is slightly different because of a small changes in the if conditions that fit to the patch's context. With them nothing got lost marker and thus no retransmissions happened.

--------------------

To be honest, we should probably just remove the whole tcp_is_sack() test, rather than special case FACK.

Quoted 7 lines

The cost isn't what it was when this code was added. Back then we didn't have Ilpo's restransmit queue coalescing code, so it would make retransmit queue packet freeing more expensive. But since the coalescing code is there now, splitting all the time to record accurate loss information should be fine.

Ilpo what do you say?

Removing it would add a bug for non-FACK SACK as lost_cnt_hint is calculated a bit differently with it (only SACKed skbs count so lost_cnt_hint is not changing over the whole skb with pcount > 1 while it does change per each seg for FACK, basically non-FACK is all or nothing as only passing by an already SACKed segment increases lost_cnt_hint). ...I also fell to this trap once while reviewing this change and wondered in my mind why not to d it for all variants :-). So Yuchung's patch is fine as is afaict.

I disagree now with myself btw, it's not just "a policy decision" as we do not retransmit some segments at all before RTO in some scenarios.

Signature · 1 line
 i.
maintainer Applied
#4 lore raw

From: "Ilpo Järvinen" <[email protected]>
Date: Mon, 27 Sep 2010 22:20:57 +0300 (EEST)

Quoted 2 lines

I disagree now with myself btw, it's not just "a policy decision" as we do not retransmit some segments at all before RTO in some scenarios.

Ok, thanks for explaining all of this.

I've applied Yuchung's patch to net-2.6, thanks everyone!