Skip to contentSkip to conversation

Fix piggybacked ACKs

PATCH In review Doug Graham 4 participants 29 messages Jul 29, 2009 – Aug 4, 2009

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Fix piggybacked ACKs

This patch corrects the conditions under which a SACK will be piggybacked on a DATA packet. The previous condition was incorrect due to a misinterpretation of RFC 4960 and/or RFC 2960. Specifically, the following paragraph from section 6.2 had not been implemented correctly:

Before an endpoint transmits a DATA chunk, if any received DATA chunks have not been acknowledged (e.g., due to delayed ack), the sender should create a SACK and bundle it with the outbound DATA chunk, as long as the size of the final SCTP packet does not exceed the current MTU. See Section 6.2.

When about to send a DATA chunk, the code now checks to see if the SACK timer is running. If it is, we know we have a SACK to send to the peer, so we append the SACK (assuming available space in the packet) and turn off the timer. For a simple request-response scenario, this will result in the SACK being bundled with the response, meaning the the SACK is received quickly by the client, and also meaning that no separate SACK packet needs to be sent by the server to acknowledge the request. Prior to this patch, a separate SACK packet would have been sent by the server SCTP only after its delayed-ACK timer had expired (usually 200ms). This is wasteful of bandwidth, and can also have a major negative impact on performance due the interaction of delayed ACKs with the Nagle algorithm.

---

net/sctp/output.c +5 -4
@@ -237,18 +237,19 @@ static sctp_xmit_t sctp_packet_bundle_sa
 	if (sctp_chunk_is_data(chunk) && !pkt->has_sack &&
 	    !pkt->has_cookie_echo) {
 		struct sctp_association *asoc;
+		struct timer_list *timer;
 		asoc = pkt->transport->asoc;
+		timer = &asoc->timers[SCTP_EVENT_TIMEOUT_SACK];
 
-		if (asoc->a_rwnd > asoc->rwnd) {
+		/* If the SACK timer is running, we have a pending SACK */
+		if (timer_pending(timer)) {
 			struct sctp_chunk *sack;
 			asoc->a_rwnd = asoc->rwnd;
 			sack = sctp_make_sack(asoc);
 			if (sack) {
-				struct timer_list *timer;
 				retval = sctp_packet_append_chunk(pkt, sack);
 				asoc->peer.sack_needed = 0;
-				timer = &asoc->timers[SCTP_EVENT_TIMEOUT_SACK];
-				if (timer_pending(timer) && del_timer(timer))
+				if (del_timer(timer))
 					sctp_association_put(asoc);
 			}
 		}
#1 lore raw
Quoted 11 lines · Doug Graham

This patch corrects the conditions under which a SACK will be piggybacked on a DATA packet. The previous condition was incorrect due to a misinterpretation of RFC 4960 and/or RFC 2960. Specifically, the following paragraph from section 6.2 had not been implemented correctly:

Before an endpoint transmits a DATA chunk, if any received DATA chunks have not been acknowledged (e.g., due to delayed ack), the sender should create a SACK and bundle it with the outbound DATA chunk, as long as the size of the final SCTP packet does not exceed the current MTU. See Section 6.2.

The above text said that SACK is create when the size of the final SCTP packet
does not exceed the current MTU. With the patch, what will happend? If the packet is too large for bundle with SACK, the packet sequence will like this:

Endpoint A Endpoint B

DATA ------------->

                <-------------     SACK
                <-------------     DATA (size52)
SACK            ------------->
DATA (size52)------------->
                <-------------     SACK
                <-------------     DATA (size52)

The behavior is the same as no delayed ack support. So I think you also need to check the packet size before append the SACK.

Quoted 51 lines

When about to send a DATA chunk, the code now checks to see if the SACK timer is running. If it is, we know we have a SACK to send to the peer, so we append the SACK (assuming available space in the packet) and turn off the timer. For a simple request-response scenario, this will result in the SACK being bundled with the response, meaning the the SACK is received quickly by the client, and also meaning that no separate SACK packet needs to be sent by the server to acknowledge the request. Prior to this patch, a separate SACK packet would have been sent by the server SCTP only after its delayed-ACK timer had expired (usually 200ms). This is wasteful of bandwidth, and can also have a major negative impact on performance due the interaction of delayed ACKs with the Nagle algorithm.

---

net/sctp/output.c +5 -5
@@ -237,18 +237,19 @@ static sctp_xmit_t sctp_packet_bundle_sa
 	if (sctp_chunk_is_data(chunk) && !pkt->has_sack &&
 	    !pkt->has_cookie_echo) {
 		struct sctp_association *asoc;
+		struct timer_list *timer;
 		asoc = pkt->transport->asoc;
+		timer = &asoc->timers[SCTP_EVENT_TIMEOUT_SACK];
 
-		if (asoc->a_rwnd > asoc->rwnd) {
+		/* If the SACK timer is running, we have a pending SACK */
+		if (timer_pending(timer)) {
 			struct sctp_chunk *sack;
 			asoc->a_rwnd = asoc->rwnd;
 			sack = sctp_make_sack(asoc);
 			if (sack) {
-				struct timer_list *timer;
 				retval = sctp_packet_append_chunk(pkt, sack);
 				asoc->peer.sack_needed = 0;
-				timer = &asoc->timers[SCTP_EVENT_TIMEOUT_SACK];
-				if (timer_pending(timer) && del_timer(timer))
+				if (del_timer(timer))
 					sctp_association_put(asoc);
 			}
 		}
--

To unsubscribe from this list: send the line "unsubscribe linux-sctp" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#2 lore raw
In reply to Doug Graham
Quoted 29 lines · Doug Graham
Quoted 5 lines

This patch corrects the conditions under which a SACK will be piggybacked on a DATA packet. The previous condition was incorrect due to a misinterpretation of RFC 4960 and/or RFC 2960. Specifically, the following paragraph from section 6.2 had not been implemented correctly:

Quoted 7 lines

Before an endpoint transmits a DATA chunk, if any received DATA chunks have not been acknowledged (e.g., due to delayed ack), the sender should create a SACK and bundle it with the outbound DATA chunk, as long as the size of the final SCTP packet does not exceed the current MTU. See Section 6.2.

The above text said that SACK is create when the size of the final SCTP packet
does not exceed the current MTU. With the patch, what will happend? If the packet is too large for bundle with SACK, the packet sequence will like this:

Endpoint A Endpoint B

DATA ------------->

                <-------------     SACK
                <-------------     DATA (size52)
SACK            ------------->
DATA (size52)------------->
                <-------------     SACK
                <-------------     DATA (size52)
Quoted 2 lines

The behavior is the same as no delayed ack support. So I think you also need to check the packet size before append the SACK.

The patch should be like this:

[PATCH] sctp: Do not create SACK chunk if the final packet size exceed current MTU

The sender should create a SACK only if the size of the final SCTP packet does not exceed the current MTU. Base on RFC 4960:

6.1. Transmission of DATA Chunks

Before an endpoint transmits a DATA chunk, if any received DATA chunks have not been acknowledged (e.g., due to delayed ack), the sender should create a SACK and bundle it with the outbound DATA chunk, as long as the size of the final SCTP packet does not exceed the current MTU.

---
 net/sctp/output.c |   13 ++++++++-----
 1 files changed, 8 insertions(+), 5 deletions(-)
net/sctp/output.c +8 -5
index 94c110d..1aa3c6e 100644
@@ -274,16 +274,19 @@ sctp_xmit_t sctp_packet_append_chunk(struct sctp_packet *packet,
 	if (retval != SCTP_XMIT_OK)
 		goto finish;
 
-	/* Try to bundle SACK chunk */
-	retval = sctp_packet_bundle_sack(packet, chunk);
-	if (retval != SCTP_XMIT_OK)
-		goto finish;
-
 	psize = packet->size;
 	pmtu  = ((packet->transport->asoc) ?
 		 (packet->transport->asoc->pathmtu) :
 		 (packet->transport->pathmtu));
 
+	/* Try to bundle SACK chunk if not exceed the current MTU */
+	if (psize + chunk_len + sizeof(struct sctp_sack_chunk) <= pmtu) {
+		retval = sctp_packet_bundle_sack(packet, chunk);
+		if (retval != SCTP_XMIT_OK)
+			goto finish;
+		psize = packet->size;
+	}
+
 	too_big = (psize + chunk_len > pmtu);
 
 	/* Decide if we need to fragment or resubmit later. */
Signature · 1 line
1.6.2.2
#3 lore raw
In reply to Doug Graham
Quoted 10 lines · Wei Yongjun

The sender should create a SACK only if the size of the final SCTP packet does not exceed the current MTU. Base on RFC 4960:

6.1. Transmission of DATA Chunks

Before an endpoint transmits a DATA chunk, if any received DATA chunks have not been acknowledged (e.g., due to delayed ack), the sender should create a SACK and bundle it with the outbound DATA chunk, as long as the size of the final SCTP packet does not exceed the current MTU.

[patch deleted]

I think you're right that there's a real problem here, and that a patch similar to yours is needed, but this is not a new problem introduced with my patch. I only changed the conditions under which a SACK chunk was bundled with a DATA chunk, but the same bundling would have been happening before under different conditions.

I'd really need to study the code more than I have to be able to say whether or not your fix is correct (and who cares what I think anyway :-)), but I've just spent all of about 15 minutes looking at parts of the code that I'd never looked at before, and I see something that off the top of my head looks a bit scary. That is that sctp_packet_bundle_sack() calls sctp_packet_append_chunk(), which calls sctp_packet_bundle_sack(). Aside from the possibility of infinite recursion (presumably this must be prevented somehow, because it doesn't seem to happen), the logic of this seems strangely circular to me. If bundle_sack() is going to call append_chunk(), I'd have guessed that append_chunk() would be a lower level routine that just appends the chunk you give it, not one that itself tries to bundle other chunks in.

Anyway, you've got me curious now. I'll have a go at better understanding the code when I get some time.

--Doug.

#4 lore raw
In reply to Doug Graham
Quoted 20 lines · Doug Graham
Quoted 10 lines · Wei Yongjun

The sender should create a SACK only if the size of the final SCTP packet does not exceed the current MTU. Base on RFC 4960:

6.1. Transmission of DATA Chunks

Before an endpoint transmits a DATA chunk, if any received DATA chunks have not been acknowledged (e.g., due to delayed ack), the sender should create a SACK and bundle it with the outbound DATA chunk, as long as the size of the final SCTP packet does not exceed the current MTU.

[patch deleted]

I think you're right that there's a real problem here, and that a patch similar to yours is needed, but this is not a new problem introduced with my patch. I only changed the conditions under which a SACK chunk was bundled with a DATA chunk, but the same bundling would have been happening before under different conditions.

Quoted 12 lines

I'd really need to study the code more than I have to be able to say whether or not your fix is correct (and who cares what I think anyway :-)), but I've just spent all of about 15 minutes looking at parts of the code that I'd never looked at before, and I see something that off the top of my head looks a bit scary. That is that sctp_packet_bundle_sack() calls sctp_packet_append_chunk(), which calls sctp_packet_bundle_sack(). Aside from the possibility of infinite recursion (presumably this must be prevented somehow, because it doesn't seem to happen), the logic of this seems strangely circular to me. If bundle_sack() is going to call append_chunk(), I'd have guessed that append_chunk() would be a lower level routine that just appends the chunk you give it, not one that itself tries to bundle other chunks in.

sctp_append_chunk() tries to be very smart. It's a generic routine that you call that also then tries to figure out if any special bundle can also be done.

The circular bundling doesn't happen because the chunk one passes to
sctp_packet_bundle_sack() changes. The first time in, the chunk is a data
chunk that makes us go into the whole added the SACK in. At this time, append_chunk() is called with a SACK chunk and a seemingly recursive call to bundle_sack() bails since the chunk is not data.

I think the Wei's patch for mtu checking should probably move into bundle_sack().

-vlad

Quoted 6 lines

Anyway, you've got me curious now. I'll have a go at better understanding the code when I get some time.

--Doug.

#5 lore raw
In reply to Doug Graham
Quoted 12 lines · Doug Graham
Quoted 10 lines · Wei Yongjun

The sender should create a SACK only if the size of the final SCTP packet does not exceed the current MTU. Base on RFC 4960:

6.1. Transmission of DATA Chunks

Before an endpoint transmits a DATA chunk, if any received DATA chunks have not been acknowledged (e.g., due to delayed ack), the sender should create a SACK and bundle it with the outbound DATA chunk, as long as the size of the final SCTP packet does not exceed the current MTU.

Quoted 7 lines

[patch deleted]

I think you're right that there's a real problem here, and that a patch similar to yours is needed, but this is not a new problem introduced with my patch. I only changed the conditions under which a SACK chunk was bundled with a DATA chunk, but the same bundling would have been happening before under different conditions.

Doug

If you still have BSD setup, can you try increasing you message size to say 1442 and see what happens.

I'd expect bundles SACKs at 1440 bytes, but then probably a separate SACK and DATA.

-vlad

Quoted 23 lines

I'd really need to study the code more than I have to be able to say whether or not your fix is correct (and who cares what I think anyway :-)), but I've just spent all of about 15 minutes looking at parts of the code that I'd never looked at before, and I see something that off the top of my head looks a bit scary. That is that sctp_packet_bundle_sack() calls sctp_packet_append_chunk(), which calls sctp_packet_bundle_sack(). Aside from the possibility of infinite recursion (presumably this must be prevented somehow, because it doesn't seem to happen), the logic of this seems strangely circular to me. If bundle_sack() is going to call append_chunk(), I'd have guessed that append_chunk() would be a lower level routine that just appends the chunk you give it, not one that itself tries to bundle other chunks in.

Anyway, you've got me curious now. I'll have a go at better understanding the code when I get some time.

--Doug.
--
To unsubscribe from this list: send the line "unsubscribe linux-sctp" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#6 lore raw
In reply to Doug Graham
Quoted 21 lines · Vlad Yasevich
Quoted 12 lines · Doug Graham
Quoted 10 lines · Wei Yongjun

The sender should create a SACK only if the size of the final SCTP packet does not exceed the current MTU. Base on RFC 4960:

6.1. Transmission of DATA Chunks

Before an endpoint transmits a DATA chunk, if any received DATA chunks have not been acknowledged (e.g., due to delayed ack), the sender should create a SACK and bundle it with the outbound DATA chunk, as long as the size of the final SCTP packet does not exceed the current MTU.

Quoted 7 lines

[patch deleted]

I think you're right that there's a real problem here, and that a patch similar to yours is needed, but this is not a new problem introduced with my patch. I only changed the conditions under which a SACK chunk was bundled with a DATA chunk, but the same bundling would have been happening before under different conditions.

Quoted 6 lines

Doug

If you still have BSD setup, can you try increasing you message size to say 1442 and see what happens.

I'd expect bundles SACKs at 1440 bytes, but then probably a separate SACK and DATA.

The largest amount of data I can send and still have the BSD server bundle a SACK with the response is 1436 bytes. The total ethernet frame size at that point is 1514 bytes, so this seems correct. I've attached wireshark captures with data sizes of 1436 bytes and 1438 bytes. It's interesting to note that if BSD decides not to bundle a SACK, it instead sends a separate SACK packet immediately; it does not wait for the SACK timer to timeout. It first sends the SACK, then the DATA immediately follows. I don't think Wei's patch would do this; I think that if his patch determined that bundling a SACK would cause the packet to exceed the MTU, then the behaviour will revert to what it was before my patch is applied: ie the SACK will not be sent for 200ms.

So I guess the logic when about to send a DATA chunk should go something like:

if (sack_timer_running) {
/* We know we owe the peer a SACK */
if (SACK + DATA fits in the MTU)
bundle SACK with DATA and send that
else {
send SACK in a separate packet
send DATA in a separate packet
}
turn_off_sack_timer
}

I don't think the RFC was explicit on what to do if the SACK+DATA exceeds the MTU, but this makes sense to me.

--Doug.

Ôò¡ÿÿC+rJ–!ZZ ¦øÅˆ.EL@@„&

ÐB06‚¥÷k,4ŠÜ
ÿÿ{aQ €ÀÀC+rJ"#¢¢Åˆ. ¦øE”¶@„c

06ÐB4Š+ÃÕtïÿέŽ8
äÉdíÀPLRS€À€
Á€À�‚€$›ÂC�þ½W¶ÁWß9#ût‹xŸË‚uÅ;qX?–l€€€ÁKAME-BSD 1.1ñ�· `ê4Šïÿέ

ÐB06Uİ{Ä,4ŠÜ
ÿÿ{aQ €ÀÀtïÿέŽ8
äÉdíÀPLRS€À€
Á€À�‚€$›ÂC�þ½W¶ÁWß9#ût‹xŸË‚uÅ;qX?–l€€€Á·š½>�§45H
³wzãéš‹\C+rJa#>> ¦øÅˆ.E0@@„%/

ÐB06ïÿέ§þ
KAME-BSD 1.1ñ�· `ê4Šïÿέ

ÐB06Uİ{Ä,4ŠÜ
ÿÿ{aQ €ÀÀtïÿέŽ8
äÉdíÀPLRS€À€
Á€À�‚€$›ÂC�þ½W¶ÁWß9#ût‹xŸË‚uÅ;qX?–l€€€Á·š½>�§45H
³wzãéš‹\C+rJß$<<ň. ¦øE$·@@„$†

06ÐB4Š
n§% C+rJ%êê ¦øÅˆ.EÜ@@„ ƒ

ÐB06ïÿέ_›æ­ô{aQ0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901È{aR2345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345C+rJt'>>ň. ¦øE0¸@@„$y
06ÐB4ŠÍýWÛ{aR†œC+rJ)ÚÚň. ¦øE̹@@„Ú
06ÐB4Šçí(Û¬äÉdí01234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345C+rJ@)>> ¦øÅˆ.E0@@„&/ 
ÐB06ïÿέ�´ ÙäÉdíÖdE+rJ +ÚÚ ¦øÅˆ.EÌ@@„ “

ÐB06ïÿέ'ÛU"¬{aS01234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345E+rJ'/êêň. ¦øEÜ»@@„È 06ÐB4ŠŽtáî{aSŽ8¬äÉdî01234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345E+rJ78 >> ¦øÅˆ.E0@@„&/ 
ÐB06ïÿέ´¸£ÀäÉdîÖdG+rJë0ÚÚ ¦øÅˆ.EÌ@@„ “

ÐB06ïÿέ´«Û¾¬{aT01234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345G+rJ5êêň. ¦øEܽ@@„Æ
06ÐB4ŠÿD‹ {aTŽ8¬äÉdï01234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345G+rJþ@ >> ¦øÅˆ.E0@@„&/ 
ÐB06ïÿέün�4äÉdïÖdI+rJk7ÚÚ ¦øÅˆ.EÌ@@„ “

ÐB06ïÿέ&R²g¬{aU01234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345I+rJ³;êêň. ¦øEÜ¿@@„Ä 06ÐB4ŠFyº¥{aUŽ8¬äÉdð01234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345I+rJ×G >> ¦øÅˆ.E0@@„&/ 
ÐB06ïÿέ®Á…>äÉdðÖdK+rJ>66 ¦øÅˆ.E(@@„&7

ÐB06ïÿέûð°IäÉdðK+rJÏ><<ň. ¦øE$Á@@„$|

06ÐB4Š3ç…GK+rJó>22 ¦øÅˆ.E$@@„&;

ÐB06ïÿέg�¨�

Ôò¡ÿÿ¡+rJ¹jZZ ¦øÅˆ.EL@@„&

§±06„kD,°n³:Ü
ÿÿÁÛ‚U €ÀÀ¡+rJêk¢¢Åˆ. ¦øE”Â@„c

06§±°n³:9Ïé›tc5Ž8
õÎåÖÀPLRS€À€
Á€À�‚€$UlUD�J€Óó¯ï€]›‘ûw櫽\�–aDWD78€€€ÁKAME-BSD 1.1†|ÄOq
`ê°n³:c5

§±06{Ä,°n³:Ü
ÿÿÁÛ‚U €ÀÀtc5Ž8
õÎåÖÀPLRS€À€
Á€À�‚€$UlUD�J€Óó¯ï€]›‘ûw櫽\�–aDWD78€€€Á®ý�ºr ;$mÌk2µ£tãªÏ>Ô¡+rJ%l>> ¦øÅˆ.E0@@„%/

§±06c5#½“
KAME-BSD 1.1†|ÄOq
`ê°n³:c5

§±06{Ä,°n³:Ü
ÿÿÁÛ‚U €ÀÀtc5Ž8
õÎåÖÀPLRS€À€
Á€À�‚€$UlUD�J€Óó¯ï€]›‘ûw櫽\�–aDWD78€€€Á®ý�ºr ;$mÌk2µ£tãªÏ>Ô¡+rJ m<<ň. ¦øE$Ã@@„$z

06§±°n³:6®’è ¡+rJÊm"" ¦øÅˆ.E@@„!K

§±06c5.'V¤ôÁÛ‚U0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901¡+rJ(p>>ň. ¦øE0Ä@@„$m
06§±°n³:*†#sÁÛ‚U‰T¡+rJFpúú ¦øÅˆ.Eì@@„%s

§±06c5.º¢qÊÁÛ‚V234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567¡+rJ˜q>>ň. ¦øE0Å@@„$l

06§±°n³:våÁÛ‚VŽ8¡+rJsÞÞň. ¦øEÐÆ@@„É
06§±°n³:Þ»®®õÎåÖ0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567¡+rJ=s>> ¦øÅˆ.E0@@„&/ 
§±06c5uÀá�õÎåÖÖb£+rJJtÞÞ ¦øÅˆ.EÐ@@„ �

§±06c5�ÂWä®ÁÛ‚W0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567£+rJw>>ň. ¦øE0È@@„$i
06§±°n³:>Ê ÁÛ‚WŽ8£+rJ‰xÞÞň. ¦øEÐÉ@@„Æ
06§±°n³:—çO®õÎå×0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567£+rJm‚ >> ¦øÅˆ.E0@@„&/ 
§±06c5=ßdõÎå×Öb¥+rJÙyÞÞ ¦øÅˆ.EÐ@@„ �

§±06c5FUþH®ÁÛ‚X0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567¥+rJh|>>ň. ¦øE0Ë@@„$f
06§±°n³:³ö3nÁÛ‚XŽ8¥+rJá}ÞÞň. ¦øEÐÌ@@„Ã
06§±°n³:A‹Nã®õÎåØ0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567¥+rJ2‹ >> ¦øÅˆ.E0@@„&/ 
§±06c5°*ÌõÎåØÖb§+rJü~ÞÞ ¦øÅˆ.EÐ@@„ �

§±06c5Ô—¢©®ÁÛ‚Y0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567§+rJË�>>ň. ¦øE0Î@@„$c
06§±°n³:û šÁÛ‚YŽ8§+rJ9ƒÞÞň. ¦øEÐÏ@@„À
06§±°n³:ÓI®õÎåÙ0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567§+rJ#Ž >> ¦øÅˆ.E0@@„&/ 
§±06c5øüòïõÎåÙÖb©+rJY…66 ¦øÅˆ.E(@@„&7

§±06c5½ÂwõÎåÙ©+rJc†<<ň. ¦øE$Ñ@@„$l

06§±°n³:'°Š©+rJ‹†22 ¦øÅˆ.E$@@„&;

§±06c5 aˆ®

#7 lore raw
In reply to Doug Graham
Quoted 33 lines · Doug Graham
Quoted 23 lines · Vlad Yasevich
Quoted 13 lines · Doug Graham
Quoted 11 lines · Wei Yongjun

The sender should create a SACK only if the size of the final SCTP packet does not exceed the current MTU. Base on RFC 4960:

6.1. Transmission of DATA Chunks

Before an endpoint transmits a DATA chunk, if any received DATA chunks have not been acknowledged (e.g., due to delayed ack), the sender should create a SACK and bundle it with the outbound DATA chunk, as long as the size of the final SCTP packet does not exceed the current MTU.

Quoted 8 lines

[patch deleted]

I think you're right that there's a real problem here, and that a patch similar to yours is needed, but this is not a new problem introduced with my patch. I only changed the conditions under which a SACK chunk was bundled with a DATA chunk, but the same bundling would have been happening before under different conditions.

Quoted 7 lines

Doug

If you still have BSD setup, can you try increasing you message size to say 1442 and see what happens.

I'd expect bundles SACKs at 1440 bytes, but then probably a separate SACK and DATA.

Quoted 12 lines

The largest amount of data I can send and still have the BSD server bundle a SACK with the response is 1436 bytes. The total ethernet frame size at that point is 1514 bytes, so this seems correct. I've attached wireshark captures with data sizes of 1436 bytes and 1438 bytes. It's interesting to note that if BSD decides not to bundle a SACK, it instead sends a separate SACK packet immediately; it does not wait for the SACK timer to timeout. It first sends the SACK, then the DATA immediately follows. I don't think Wei's patch would do this; I think that if his patch determined that bundling a SACK would cause the packet to exceed the MTU, then the behaviour will revert to what it was before my patch is applied: ie the SACK will not be sent for 200ms.

Before my patch, SACK sent on linux is the same as BSD. But... BSD's implemention is really correct?

RFC said:

the sender should create a SACK and bundle it with the outbound DATA chunk, as long as the size of the final SCTP packet does not exceed the current MTU.

So, we just need create a SACK only if the final packet size does not exceed the MTU. Always send SACK may cause lower performance.

Quoted 19 lines

So I guess the logic when about to send a DATA chunk should go something like:

if (sack_timer_running) {
/* We know we owe the peer a SACK */
if (SACK + DATA fits in the MTU)
bundle SACK with DATA and send that
else {
send SACK in a separate packet
send DATA in a separate packet
}
turn_off_sack_timer
}

I don't think the RFC was explicit on what to do if the SACK+DATA exceeds the MTU, but this makes sense to me.

--Doug.

#8 lore raw
In reply to Doug Graham
Quoted 6 lines · Doug Graham
Quoted 4 lines · Vlad Yasevich

If you still have BSD setup, can you try increasing you message size to say 1442 and see what happens.

I'd expect bundles SACKs at 1440 bytes, but then probably a separate SACK and DATA.

Quoted 11 lines

The largest amount of data I can send and still have the BSD server bundle a SACK with the response is 1436 bytes. The total ethernet frame size at that point is 1514 bytes, so this seems correct. I've attached wireshark captures with data sizes of 1436 bytes and 1438 bytes. It's interesting to note that if BSD decides not to bundle a SACK, it instead sends a separate SACK packet immediately; it does not wait for the SACK timer to timeout. It first sends the SACK, then the DATA immediately follows. I don't think Wei's patch would do this; I think that if his patch determined that bundling a SACK would cause the packet to exceed the MTU, then the behaviour will revert to what it was before my patch is applied: ie the SACK will not be sent for 200ms.

I think it's about time that I sat down and carefully read the RFC all the way through before trying to do much more analysis of what's happening on the wire, but I did just notice something surprising while try slightly larger packets. For one, I could've sworn that I saw a ethernet frame of 1516 bytes at one point, but I didn't save the capture and don't know whether it was Linux or BSD that sent the oversized frame, or just my imagination. But here's one that I did capture when sending and receiving 1454 bytes of data. 1452 bytes is the most data that will fit in a single 1514 byte ethernet frame, so 1454 bytes must be fragmented. The capture is attached, but here's one iteration:

 13 2.002632    10.0.0.15   10.0.0.11   DATA (1452 bytes data) 
 14 2.203092    10.0.0.11   10.0.0.15   SACK 
 15 2.203153    10.0.0.15   10.0.0.11   DATA (2 bytes data)
 16 2.203427    10.0.0.11   10.0.0.15   SACK 
 17 2.203808    10.0.0.11   10.0.0.15   DATA (1452 bytes data)
 18 2.403524    10.0.0.15   10.0.0.11   SACK 
 19 2.403686    10.0.0.11   10.0.0.15   DATA (2 bytes data)
 20 2.603285    10.0.0.15   10.0.0.11   SACK 

What bothers me about this is that Nagle seems to be introducing a delay here. The first DATA packets in both directions are MTU-sized packets, yet both the Linux client and the BSD server wait 200ms until they get the SACK to the first fragment before sending the second fragment. The server can't send its reply until it gets both fragments, and the client can't reassemble the reply until it gets both fragments, so from the application's point of view, the reply doesn't arrive until 400ms after the request is sent. This could probably be fixed by disabling Nagle with SCTP_NODELAY, but that shouldn't be required. Nagle is only supposed to prevent multiple outstanding *small* packets.

If you tell me I'm full of crap, I promise I'll shut up until I read the whole RFC :-)

--Doug.

#9 lore raw
In reply to Doug Graham
Quoted 14 lines · Wei Yongjun
Quoted 12 lines · Doug Graham

The largest amount of data I can send and still have the BSD server bundle a SACK with the response is 1436 bytes. The total ethernet frame size at that point is 1514 bytes, so this seems correct. I've attached wireshark captures with data sizes of 1436 bytes and 1438 bytes. It's interesting to note that if BSD decides not to bundle a SACK, it instead sends a separate SACK packet immediately; it does not wait for the SACK timer to timeout. It first sends the SACK, then the DATA immediately follows. I don't think Wei's patch would do this; I think that if his patch determined that bundling a SACK would cause the packet to exceed the MTU, then the behaviour will revert to what it was before my patch is applied: ie the SACK will not be sent for 200ms.

Quoted 1 line

Before my patch, SACK sent on linux is the same as BSD.

I had it in my head that without your patch, the combined DATA+SACK packet would have been fragmented at the IP level, but that's very likely my unfamiliarity with the code kicking in.

Quoted 11 lines

But... BSD's
implemention is really correct?

RFC said:

the sender should create a SACK and bundle it with the outbound DATA chunk, as long as the size of the final SCTP packet does not exceed the current MTU.

So, we just need create a SACK only if the final packet size does not exceed the MTU. Always send SACK may cause lower performance.

I agree that this section of the RFC implies that if the SACK won't fit, it simply shouldn't be sent at this point. Which would make BSD's behaviour incorrect. But to my mind, it makes sense to send it, although I'm not sure I could make a strong case for that.

But consider that in the case of a client and server sending equal-sized messages to each other (to keep it simple), there will be a message size at which the behaviour changes noticably. Small messages will be SACK'd immediately. Messages slightly smaller than the MTU will not be SACK'd until the delayed ACK timer expires. Messages slightly larger than the MTU will again be SACKED immediately because the second fragment in the response will have space for a SACK (assuming that the Nagle problem I mentioned in my last email really is a problem that needs to be fixed).

Perhaps Michael could explain which is the correct behaviour.

--Doug.

#10 lore raw
In reply to Doug Graham
Quoted 42 lines · Doug Graham
Quoted 7 lines · Doug Graham
Quoted 5 lines · Vlad Yasevich

If you still have BSD setup, can you try increasing you message size to say 1442 and see what happens.

I'd expect bundles SACKs at 1440 bytes, but then probably a separate SACK and DATA.

Quoted 12 lines

The largest amount of data I can send and still have the BSD server bundle a SACK with the response is 1436 bytes. The total ethernet frame size at that point is 1514 bytes, so this seems correct. I've attached wireshark captures with data sizes of 1436 bytes and 1438 bytes. It's interesting to note that if BSD decides not to bundle a SACK, it instead sends a separate SACK packet immediately; it does not wait for the SACK timer to timeout. It first sends the SACK, then the DATA immediately follows. I don't think Wei's patch would do this; I think that if his patch determined that bundling a SACK would cause the packet to exceed the MTU, then the behaviour will revert to what it was before my patch is applied: ie the SACK will not be sent for 200ms.

I think it's about time that I sat down and carefully read the RFC all the way through before trying to do much more analysis of what's happening on the wire, but I did just notice something surprising while try slightly larger packets. For one, I could've sworn that I saw a ethernet frame of 1516 bytes at one point, but I didn't save the capture and don't know whether it was Linux or BSD that sent the oversized frame, or just my imagination. But here's one that I did capture when sending and receiving 1454 bytes of data. 1452 bytes is the most data that will fit in a single 1514 byte ethernet frame, so 1454 bytes must be fragmented. The capture is attached, but here's one iteration:

 13 2.002632    10.0.0.15   10.0.0.11   DATA (1452 bytes data) 
 14 2.203092    10.0.0.11   10.0.0.15   SACK 
 15 2.203153    10.0.0.15   10.0.0.11   DATA (2 bytes data)
 16 2.203427    10.0.0.11   10.0.0.15   SACK 
 17 2.203808    10.0.0.11   10.0.0.15   DATA (1452 bytes data)
 18 2.403524    10.0.0.15   10.0.0.11   SACK 
 19 2.403686    10.0.0.11   10.0.0.15   DATA (2 bytes data)
 20 2.603285    10.0.0.15   10.0.0.11   SACK 
Quoted 11 lines

What bothers me about this is that Nagle seems to be introducing a delay here. The first DATA packets in both directions are MTU-sized packets, yet both the Linux client and the BSD server wait 200ms until they get the SACK to the first fragment before sending the second fragment. The server can't send its reply until it gets both fragments, and the client can't reassemble the reply until it gets both fragments, so from the application's point of view, the reply doesn't arrive until 400ms after the request is sent. This could probably be fixed by disabling Nagle with SCTP_NODELAY, but that shouldn't be required. Nagle is only supposed to prevent multiple outstanding *small* packets.

I think you hit the point which Nagle's algorithm should be not used.

Can you try the following patch?

[PATCH] sctp: do not used Nagle algorithm while fragmented data is transmitted

If fragmented data is sent, the Nagle's algorithm should not be used. In special case, if only one large packet is sent, the delay send of fragmented data will cause the receiver wait for more fragmented data to reassembe them and not send SACK, but the sender still wait for SACK before send the last fragment.

---
 net/sctp/output.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)
net/sctp/output.c +3 -1
index b94c211..0d8d765 100644
@@ -718,7 +718,9 @@ static sctp_xmit_t sctp_packet_append_data(struct sctp_packet *packet,
 	 * unacknowledged.
 	 */
 	if (!sp->nodelay && sctp_packet_empty(packet) &&
-	    q->outstanding_bytes && sctp_state(asoc, ESTABLISHED)) {
+	    q->outstanding_bytes && sctp_state(asoc, ESTABLISHED) &&
+	    (chunk->chunk_hdr->flags & SCTP_DATA_FRAG_MASK) =
+							SCTP_DATA_NOT_FRAG) {
 		unsigned len = datasize + q->out_qlen;
 
 		/* Check whether this chunk and all the rest of pending
Signature · 1 line
1.6.2.2
Quoted 5 lines

If you tell me I'm full of crap, I promise I'll shut up until I read the whole RFC :-)

--Doug.

#11 lore raw
In reply to Doug Graham
Quoted 51 lines · Doug Graham
Quoted 7 lines · Doug Graham
Quoted 5 lines · Vlad Yasevich

If you still have BSD setup, can you try increasing you message size to say 1442 and see what happens.

I'd expect bundles SACKs at 1440 bytes, but then probably a separate SACK and DATA.

Quoted 16 lines

The largest amount of data I can send and still have the BSD server bundle
a SACK with the response is 1436 bytes. The total ethernet frame size
at that point is 1514 bytes, so this seems correct. I've attached wireshark captures with data sizes of 1436 bytes and 1438 bytes. It's interesting to note that if BSD decides not to bundle a SACK, it instead sends a separate SACK packet immediately; it does not wait for the SACK timer to timeout. It first sends the SACK, then the DATA
immediately follows. I don't think Wei's patch would do this; I think that if his patch determined that bundling a SACK would cause the packet
to exceed the MTU, then the behaviour will revert to what it was before
my patch is applied: ie the SACK will not be sent for 200ms.

I think it's about time that I sat down and carefully read the RFC all the
way through before trying to do much more analysis of what's happening on
the wire, but I did just notice something surprising while try slightly
larger packets. For one, I could've sworn that I saw a ethernet frame of 1516 bytes at one point, but I didn't save the capture and don't know whether it was Linux or BSD that sent the oversized frame, or just
my imagination. But here's one that I did capture when sending and receiving 1454 bytes of data. 1452 bytes is the most data that will fit
in a single 1514 byte ethernet frame, so 1454 bytes must be fragmented.
The capture is attached, but here's one iteration:

13 2.002632    10.0.0.15   10.0.0.11   DATA (1452 bytes data)
14 2.203092    10.0.0.11   10.0.0.15   SACK
15 2.203153    10.0.0.15   10.0.0.11   DATA (2 bytes data)
16 2.203427    10.0.0.11   10.0.0.15   SACK
17 2.203808    10.0.0.11   10.0.0.15   DATA (1452 bytes data)
18 2.403524    10.0.0.15   10.0.0.11   SACK
19 2.403686    10.0.0.11   10.0.0.15   DATA (2 bytes data)
20 2.603285    10.0.0.15   10.0.0.11   SACK
Quoted 2 lines

What bothers me about this is that Nagle seems to be introducing a delay

This is the common bad interaction between Nagle and delayed SACKs.

Quoted 4 lines

here. The first DATA packets in both directions are MTU-sized packets,
yet both the Linux client and the BSD server wait 200ms until they get the SACK to the first fragment before sending the second fragment.

Quoted 8 lines

The server can't send its reply until it gets both fragments, and the client can't reassemble the reply until it gets both fragments, so from
the application's point of view, the reply doesn't arrive until 400ms after the request is sent. This could probably be fixed by disabling Nagle with SCTP_NODELAY, but that shouldn't be required. Nagle is only
supposed to prevent multiple outstanding *small* packets.

Yes, but Nagle operates at the level of chunks...
This problem is one of the reasons why we have draft-tuexen-tsvwg-sctp-sack-immediately-02
The kernel can set the I-Bit on the first chunk... Currently the only way around this is to disable Nagle at all...

Quoted 6 lines

If you tell me I'm full of crap, I promise I'll shut up until I read the whole RFC :-)

--Doug.
<bsd72_server_1454.cap>

#12 lore raw
In reply to Doug Graham
Quoted 58 lines · Doug Graham
Quoted 21 lines · Wei Yongjun
Quoted 10 lines · Doug Graham

The largest amount of data I can send and still have the BSD server bundle
a SACK with the response is 1436 bytes. The total ethernet frame size
at that point is 1514 bytes, so this seems correct. I've attached wireshark captures with data sizes of 1436 bytes and 1438 bytes. It's interesting to note that if BSD decides not to bundle a SACK, it instead sends a separate SACK packet immediately; it does not wait
for the SACK timer to timeout. It first sends the SACK, then the

Quoted 9 lines

DATA
immediately follows. I don't think Wei's patch would do this; I think
that if his patch determined that bundling a SACK would cause the packet
to exceed the MTU, then the behaviour will revert to what it was before
my patch is applied: ie the SACK will not be sent for 200ms.

Quoted 1 line

Before my patch, SACK sent on linux is the same as BSD.

I had it in my head that without your patch, the combined DATA+SACK packet
would have been fragmented at the IP level, but that's very likely my unfamiliarity with the code kicking in.

Quoted 11 lines

But... BSD's
implemention is really correct?

RFC said:

the sender should create a SACK and bundle it with the outbound DATA chunk, as long as the size of the final SCTP packet does not exceed the current MTU.

So, we just need create a SACK only if the final packet size does not exceed the MTU. Always send SACK may cause lower performance.

I agree that this section of the RFC implies that if the SACK won't fit, it simply shouldn't be sent at this point. Which would make BSD's behaviour incorrect. But to my mind, it makes sense to send it, although I'm not sure I could make a strong case for that.

But consider that in the case of a client and server sending equal- sized
messages to each other (to keep it simple), there will be a message size at which the behaviour changes noticably. Small messages will be SACK'd immediately. Messages slightly smaller than the MTU will not be SACK'd until the delayed ACK timer expires. Messages slightly larger than the MTU will again be SACKED immediately because the second
fragment in the response will have space for a SACK (assuming that the Nagle problem I mentioned in my last email really is a problem that needs to be fixed).

Quoted 1 line

Perhaps Michael could explain which is the correct behaviour.

As said in my earlier mail:
I have seen this multiple times: for one packet size the app runs find, make the packet size larger (1 byte enough) and your throughput drops to 5 packets/requests per second (assuming a 200ms delayed ack timer).

I agree that this is something the kernel should take care of and i think draft-tuexen-tsvwg-sctp-sack-immediately-02 is the way to go...

Quoted 3 lines

--Doug.

#13 lore raw
In reply to Doug Graham
Quoted 89 lines · Wei Yongjun
Quoted 42 lines · Doug Graham
Quoted 7 lines · Doug Graham
Quoted 5 lines · Vlad Yasevich

If you still have BSD setup, can you try increasing you message size to say 1442 and see what happens.

I'd expect bundles SACKs at 1440 bytes, but then probably a separate SACK and DATA.

Quoted 12 lines

The largest amount of data I can send and still have the BSD server bundle a SACK with the response is 1436 bytes. The total ethernet frame size at that point is 1514 bytes, so this seems correct. I've attached wireshark captures with data sizes of 1436 bytes and 1438 bytes. It's interesting to note that if BSD decides not to bundle a SACK, it instead sends a separate SACK packet immediately; it does not wait for the SACK timer to timeout. It first sends the SACK, then the DATA immediately follows. I don't think Wei's patch would do this; I think that if his patch determined that bundling a SACK would cause the packet to exceed the MTU, then the behaviour will revert to what it was before my patch is applied: ie the SACK will not be sent for 200ms.

I think it's about time that I sat down and carefully read the RFC all the way through before trying to do much more analysis of what's happening on the wire, but I did just notice something surprising while try slightly larger packets. For one, I could've sworn that I saw a ethernet frame of 1516 bytes at one point, but I didn't save the capture and don't know whether it was Linux or BSD that sent the oversized frame, or just my imagination. But here's one that I did capture when sending and receiving 1454 bytes of data. 1452 bytes is the most data that will fit in a single 1514 byte ethernet frame, so 1454 bytes must be fragmented. The capture is attached, but here's one iteration:

 13 2.002632    10.0.0.15   10.0.0.11   DATA (1452 bytes data) 
 14 2.203092    10.0.0.11   10.0.0.15   SACK 
 15 2.203153    10.0.0.15   10.0.0.11   DATA (2 bytes data)
 16 2.203427    10.0.0.11   10.0.0.15   SACK 
 17 2.203808    10.0.0.11   10.0.0.15   DATA (1452 bytes data)
 18 2.403524    10.0.0.15   10.0.0.11   SACK 
 19 2.403686    10.0.0.11   10.0.0.15   DATA (2 bytes data)
 20 2.603285    10.0.0.15   10.0.0.11   SACK 
Quoted 11 lines

What bothers me about this is that Nagle seems to be introducing a delay here. The first DATA packets in both directions are MTU-sized packets, yet both the Linux client and the BSD server wait 200ms until they get the SACK to the first fragment before sending the second fragment. The server can't send its reply until it gets both fragments, and the client can't reassemble the reply until it gets both fragments, so from the application's point of view, the reply doesn't arrive until 400ms after the request is sent. This could probably be fixed by disabling Nagle with SCTP_NODELAY, but that shouldn't be required. Nagle is only supposed to prevent multiple outstanding *small* packets.

I think you hit the point which Nagle's algorithm should be not used.

Can you try the following patch?

[PATCH] sctp: do not used Nagle algorithm while fragmented data is transmitted

If fragmented data is sent, the Nagle's algorithm should not be used. In special case, if only one large packet is sent, the delay send of fragmented data will cause the receiver wait for more fragmented data to reassembe them and not send SACK, but the sender still wait for SACK before send the last fragment.

---
 net/sctp/output.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)
net/sctp/output.c +3 -1
index b94c211..0d8d765 100644
@@ -718,7 +718,9 @@ static sctp_xmit_t sctp_packet_append_data(struct sctp_packet *packet,
 	 * unacknowledged.
 	 */
 	if (!sp->nodelay && sctp_packet_empty(packet) &&
-	    q->outstanding_bytes && sctp_state(asoc, ESTABLISHED)) {
+	    q->outstanding_bytes && sctp_state(asoc, ESTABLISHED) &&
+	    (chunk->chunk_hdr->flags & SCTP_DATA_FRAG_MASK) =
+							SCTP_DATA_NOT_FRAG) {
 		unsigned len = datasize + q->out_qlen;
 
 		/* Check whether this chunk and all the rest of pending
Signature · 1 line
1.6.2.2

This patch seems to do the job. I applied it in a UML instance and ran my server in that. The client is still unpatched. I see this:

 13 2.002638    10.0.0.15    10.0.0.249    DATA (1452 bytes data)
 14 2.204041    10.0.0.249   10.0.0.15     SACK 
 15 2.204090    10.0.0.15    10.0.0.249    DATA (2 bytes data)
 16 2.204428    10.0.0.249   10.0.0.15     SACK 
 17 2.204822    10.0.0.249   10.0.0.15     DATA (1452 bytes data)
 18 2.204856    10.0.0.249   10.0.0.15     DATA (2 bytes data)
 19 2.204890    10.0.0.15    10.0.0.249    SACK 

So 10.0.0.249 (the patched UML server) did send back-to-back data packets without waiting for the SACK,

I have not applied your MTU patch yet, so the server also sent a separate SACK immediately. This is less than ideal, since it could have piggybacked the SACK on the second DATA fragment (frame 18), which has lots of room. I think your MTU patch might accomplish that.

--Doug

#14 lore raw
In reply to Doug Graham
Quoted 26 lines · Michael T?xen
Quoted 11 lines · Doug Graham
13 2.002632    10.0.0.15   10.0.0.11   DATA (1452 bytes data)
14 2.203092    10.0.0.11   10.0.0.15   SACK
15 2.203153    10.0.0.15   10.0.0.11   DATA (2 bytes data)
16 2.203427    10.0.0.11   10.0.0.15   SACK
17 2.203808    10.0.0.11   10.0.0.15   DATA (1452 bytes data)
18 2.403524    10.0.0.15   10.0.0.11   SACK
19 2.403686    10.0.0.11   10.0.0.15   DATA (2 bytes data)
20 2.603285    10.0.0.15   10.0.0.11   SACK

What bothers me about this is that Nagle seems to be introducing a delay here.

This is the common bad interaction between Nagle and delayed SACKs.

Quoted 10 lines

The first DATA packets in both directions are MTU-sized packets yet both the Linux client and the BSD server wait 200ms until they get the SACK to the first fragment before sending the second fragment. The server can't send its reply until it gets both fragments, and the client can't reassemble the reply until it gets both fragments, so from
the application's point of view, the reply doesn't arrive until 400ms after the request is sent. This could probably be fixed by disabling Nagle with SCTP_NODELAY, but that shouldn't be required. Nagle is only supposed to prevent multiple outstanding *small* packets.

Quoted 5 lines

Yes, but Nagle operates at the level of chunks...
This problem is one of the reasons why we have draft-tuexen-tsvwg-sctp-sack-immediately-02
The kernel can set the I-Bit on the first chunk... Currently the only way around this is to disable Nagle at all...

I don't understand how Nagle operating at the level of chunks makes a difference here. The first DATA chunk contained 1452 bytes of data, which results in an MTU-sized frame. How is this any different than TCP sending 1460 bytes data in one segment followed immediately by sending another 2 bytes in a second segment? ie: user tries to send 1462 bytes, which TCP has to segment. It first sends an MTU-sized packet followed immediately by another packet containing whatever is left over the data, all without Nagle getting in the way.

--Doug

#15 lore raw
In reply to Doug Graham
Quoted 11 lines · Doug Graham

This patch seems to do the job. I applied it in a UML instance and ran my server in that. The client is still unpatched. I see this:

 13 2.002638    10.0.0.15    10.0.0.249    DATA (1452 bytes data)
 14 2.204041    10.0.0.249   10.0.0.15     SACK 
 15 2.204090    10.0.0.15    10.0.0.249    DATA (2 bytes data)
 16 2.204428    10.0.0.249   10.0.0.15     SACK 
 17 2.204822    10.0.0.249   10.0.0.15     DATA (1452 bytes data)
 18 2.204856    10.0.0.249   10.0.0.15     DATA (2 bytes data)
 19 2.204890    10.0.0.15    10.0.0.249    SACK 
Quoted 7 lines

So 10.0.0.249 (the patched UML server) did send back-to-back data packets without waiting for the SACK,

I have not applied your MTU patch yet, so the server also sent a separate SACK immediately. This is less than ideal, since it could have piggybacked the SACK on the second DATA fragment (frame 18), which has lots of room. I think your MTU patch might accomplish that.

Here's what it looks like after I apply your V2 MTU patch:

 12 2.002750    10.0.0.15      10.0.0.249    DATA 
 13 2.204164    10.0.0.249     10.0.0.15     SACK 
 14 2.204204    10.0.0.15      10.0.0.249    DATA 
 15 2.204926    10.0.0.249     10.0.0.15     DATA 
 16 2.204950    10.0.0.249     10.0.0.15     SACK DATA 
 17 2.204974    10.0.0.15      10.0.0.249    SACK 

Starting to look pretty good!

--Doug

#16 lore raw
In reply to Doug Graham
Quoted 55 lines · Michael Tüxen
Quoted 42 lines · Doug Graham
Quoted 7 lines · Doug Graham
Quoted 5 lines · Vlad Yasevich

If you still have BSD setup, can you try increasing you message size to say 1442 and see what happens.

I'd expect bundles SACKs at 1440 bytes, but then probably a separate SACK and DATA.

Quoted 12 lines

The largest amount of data I can send and still have the BSD server bundle
a SACK with the response is 1436 bytes. The total ethernet frame size at that point is 1514 bytes, so this seems correct. I've attached wireshark captures with data sizes of 1436 bytes and 1438 bytes. It's interesting to note that if BSD decides not to bundle a SACK, it instead sends a separate SACK packet immediately; it does not wait for the SACK timer to timeout. It first sends the SACK, then the DATA immediately follows. I don't think Wei's patch would do this; I think that if his patch determined that bundling a SACK would cause the packet to exceed the MTU, then the behaviour will revert to what it was before my patch is applied: ie the SACK will not be sent for 200ms.

I think it's about time that I sat down and carefully read the RFC all the
way through before trying to do much more analysis of what's happening on the wire, but I did just notice something surprising while try slightly larger packets. For one, I could've sworn that I saw a ethernet frame of 1516 bytes at one point, but I didn't save the capture and don't know whether it was Linux or BSD that sent the oversized frame, or just my imagination. But here's one that I did capture when sending and receiving 1454 bytes of data. 1452 bytes is the most data that will fit in a single 1514 byte ethernet frame, so 1454 bytes must be fragmented. The capture is attached, but here's one iteration:

13 2.002632    10.0.0.15   10.0.0.11   DATA (1452 bytes data)
14 2.203092    10.0.0.11   10.0.0.15   SACK
15 2.203153    10.0.0.15   10.0.0.11   DATA (2 bytes data)
16 2.203427    10.0.0.11   10.0.0.15   SACK
17 2.203808    10.0.0.11   10.0.0.15   DATA (1452 bytes data)
18 2.403524    10.0.0.15   10.0.0.11   SACK
19 2.403686    10.0.0.11   10.0.0.15   DATA (2 bytes data)
20 2.603285    10.0.0.15   10.0.0.11   SACK
Quoted 1 line

What bothers me about this is that Nagle seems to be introducing a delay

This is the common bad interaction between Nagle and delayed SACKs.

Quoted 9 lines

here. The first DATA packets in both directions are MTU-sized packets, yet both the Linux client and the BSD server wait 200ms until they get the SACK to the first fragment before sending the second fragment. The server can't send its reply until it gets both fragments, and the client can't reassemble the reply until it gets both fragments, so from the application's point of view, the reply doesn't arrive until 400ms after the request is sent. This could probably be fixed by disabling Nagle with SCTP_NODELAY, but that shouldn't be required. Nagle is only supposed to prevent multiple outstanding *small* packets.

Quoted 2 lines

Yes, but Nagle operates at the level of chunks...
This problem is one of the reasons why we have

Michael

That doesn't make sense. Nagle was meant to prevent sending a bunch small packets. That doesn't apply if the user sends a large enough message that it ends up being fragmenting into a full sized data chunk and a small-sized data chunk. It doesn't sound like Nagle should apply to the second fragment.

-vlad

Quoted 15 lines

draft-tuexen-tsvwg-sctp-sack-immediately-02
The kernel can set the I-Bit on the first chunk... Currently the only way around this is to disable Nagle at all...

Quoted 6 lines

If you tell me I'm full of crap, I promise I'll shut up until I read the whole RFC :-)

--Doug.
<bsd72_server_1454.cap>

Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-sctp" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#17 lore raw
In reply to Doug Graham
Quoted 26 lines · Wei Yongjun
Quoted 9 lines · Doug Graham
 13 2.002632    10.0.0.15   10.0.0.11   DATA (1452 bytes data) 
 14 2.203092    10.0.0.11   10.0.0.15   SACK 
 15 2.203153    10.0.0.15   10.0.0.11   DATA (2 bytes data)
 16 2.203427    10.0.0.11   10.0.0.15   SACK 
 17 2.203808    10.0.0.11   10.0.0.15   DATA (1452 bytes data)
 18 2.403524    10.0.0.15   10.0.0.11   SACK 
 19 2.403686    10.0.0.11   10.0.0.15   DATA (2 bytes data)
 20 2.603285    10.0.0.15   10.0.0.11   SACK 
Quoted 11 lines

What bothers me about this is that Nagle seems to be introducing a delay here. The first DATA packets in both directions are MTU-sized packets, yet both the Linux client and the BSD server wait 200ms until they get the SACK to the first fragment before sending the second fragment. The server can't send its reply until it gets both fragments, and the client can't reassemble the reply until it gets both fragments, so from the application's point of view, the reply doesn't arrive until 400ms after the request is sent. This could probably be fixed by disabling Nagle with SCTP_NODELAY, but that shouldn't be required. Nagle is only supposed to prevent multiple outstanding *small* packets.

I think you hit the point which Nagle's algorithm should be not used.

Can you try the following patch?

Quoted 7 lines

[PATCH] sctp: do not used Nagle algorithm while fragmented data is transmitted

If fragmented data is sent, the Nagle's algorithm should not be used. In special case, if only one large packet is sent, the delay send of fragmented data will cause the receiver wait for more fragmented data to reassembe them and not send SACK, but the sender still wait for SACK before send the last fragment.

[patch deleted]

This patch seems to work quite well, but I think disabling Nagle completely for large messages is not quite the right thing to do. There's a draft-minshall-nagle-01.txt floating around that describes a modified Nagle algorithm for TCP. It appears to have been implemented in Linux TCP even though the draft has expired. The modified algorithm is how I thought Nagle had always worked to begin with. From the draft:

"If a TCP has less than a full-sized packet to transmit, and if any previously transmitted less than full-sized packet has not yet been acknowledged, do not transmit a packet."

so in the case of sending a fragmented SCTP message, all but the last fragment will be full-sized and will be sent without delay. The last fragment will usually not be full-sized, but it too will be sent without delay because there are no outstanding non-full-sized packets.

The difference between this and your method is that yours would allow many small fragments of big messages to be outstanding, whereas this one would only allow the first big message to be sent in its entirety, followed by the full-sized fragments of the next big message. When it came time to send the second small fragment, Nagle would force it to wait for an ACK for the first small fragment. I'm not convinced that the difference is all that important, but who knows.

Here's my attempt at implementing the modified Nagle algorithm described in draft-minshall-nagle-01.txt. It should be applied instead of your patch, not on top of it. If (q->outstanding_bytes % asoc->frag_point) is zero, no delay is introduced. The assumption is that this means that all outstanding packets (if any) are full-sized.

---

net/sctp/output.c +2 -1
@@ -717,7 +717,8 @@ static sctp_xmit_t sctp_packet_append_da
 	 * unacknowledged.
 	 */
 	if (!sp->nodelay && sctp_packet_empty(packet) &&
-	    q->outstanding_bytes && sctp_state(asoc, ESTABLISHED)) {
+	    (q->outstanding_bytes % asoc->frag_point) != 0 &&
+	    sctp_state(asoc, ESTABLISHED)) {
 		unsigned len = datasize + q->out_qlen;
 
 		/* Check whether this chunk and all the rest of pending
#18 lore raw
In reply to Doug Graham
Quoted 79 lines · Doug Graham
Quoted 27 lines · Wei Yongjun
Quoted 9 lines · Doug Graham
 13 2.002632    10.0.0.15   10.0.0.11   DATA (1452 bytes data) 
 14 2.203092    10.0.0.11   10.0.0.15   SACK 
 15 2.203153    10.0.0.15   10.0.0.11   DATA (2 bytes data)
 16 2.203427    10.0.0.11   10.0.0.15   SACK 
 17 2.203808    10.0.0.11   10.0.0.15   DATA (1452 bytes data)
 18 2.403524    10.0.0.15   10.0.0.11   SACK 
 19 2.403686    10.0.0.11   10.0.0.15   DATA (2 bytes data)
 20 2.603285    10.0.0.15   10.0.0.11   SACK 
Quoted 12 lines

What bothers me about this is that Nagle seems to be introducing a delay here. The first DATA packets in both directions are MTU-sized packets, yet both the Linux client and the BSD server wait 200ms until they get the SACK to the first fragment before sending the second fragment. The server can't send its reply until it gets both fragments, and the client can't reassemble the reply until it gets both fragments, so from the application's point of view, the reply doesn't arrive until 400ms after the request is sent. This could probably be fixed by disabling Nagle with SCTP_NODELAY, but that shouldn't be required. Nagle is only supposed to prevent multiple outstanding *small* packets.

I think you hit the point which Nagle's algorithm should be not used.

Can you try the following patch?

Quoted 8 lines

[PATCH] sctp: do not used Nagle algorithm while fragmented data is transmitted

If fragmented data is sent, the Nagle's algorithm should not be used. In special case, if only one large packet is sent, the delay send of fragmented data will cause the receiver wait for more fragmented data to reassembe them and not send SACK, but the sender still wait for SACK before send the last fragment.

[patch deleted]

This patch seems to work quite well, but I think disabling Nagle completely for large messages is not quite the right thing to do. There's a draft-minshall-nagle-01.txt floating around that describes a modified Nagle algorithm for TCP. It appears to have been implemented in Linux TCP even though the draft has expired. The modified algorithm is how I thought Nagle had always worked to begin with. From the draft:

"If a TCP has less than a full-sized packet to transmit, and if any previously transmitted less than full-sized packet has not yet been acknowledged, do not transmit a packet."

so in the case of sending a fragmented SCTP message, all but the last fragment will be full-sized and will be sent without delay. The last fragment will usually not be full-sized, but it too will be sent without delay because there are no outstanding non-full-sized packets.

The difference between this and your method is that yours would allow many small fragments of big messages to be outstanding, whereas this one would only allow the first big message to be sent in its entirety, followed by the full-sized fragments of the next big message. When it came time to send the second small fragment, Nagle would force it to wait for an ACK for the first small fragment. I'm not convinced that the difference is all that important, but who knows.

Here's my attempt at implementing the modified Nagle algorithm described in draft-minshall-nagle-01.txt. It should be applied instead of your patch, not on top of it. If (q->outstanding_bytes % asoc->frag_point) is zero, no delay is introduced. The assumption is that this means that all outstanding packets (if any) are full-sized.

---

net/sctp/output.c static sctp_xmit_t sctp_packet_append_da · line 724

Quoted 9 lines
net/sctp/output.c +2 -1
@@ -717,7 +717,8 @@ static sctp_xmit_t sctp_packet_append_da
 	 * unacknowledged.
 	 */
 	if (!sp->nodelay && sctp_packet_empty(packet) &&
-	    q->outstanding_bytes && sctp_state(asoc, ESTABLISHED)) {
+	    (q->outstanding_bytes % asoc->frag_point) != 0 &&
+	    sctp_state(asoc, ESTABLISHED)) {
 		unsigned len = datasize + q->out_qlen;
 
 		/* Check whether this chunk and all the rest of pending

Seem good! But it may be broken the small packet transmit which can be used Nagle algorithm.
Such as this:

Endpoint A                Endpint B
          <-------------  DATA (size52/2) delay send
          <-------------  DATA (size52/2) send immediately
          <-------------  DATA (size52/2) send immediately ** broken
          <-------------  DATA (size52/2) delay send
          <-------------  DATA (size52/2) send immediately
          <-------------  DATA (size52/2) send immediately ** broken

Can you try this one?

#19 lore raw
In reply to Doug Graham
Quoted 78 lines · Doug Graham
Quoted 27 lines · Wei Yongjun
Quoted 9 lines · Doug Graham
 13 2.002632    10.0.0.15   10.0.0.11   DATA (1452 bytes data) 
 14 2.203092    10.0.0.11   10.0.0.15   SACK 
 15 2.203153    10.0.0.15   10.0.0.11   DATA (2 bytes data)
 16 2.203427    10.0.0.11   10.0.0.15   SACK 
 17 2.203808    10.0.0.11   10.0.0.15   DATA (1452 bytes data)
 18 2.403524    10.0.0.15   10.0.0.11   SACK 
 19 2.403686    10.0.0.11   10.0.0.15   DATA (2 bytes data)
 20 2.603285    10.0.0.15   10.0.0.11   SACK 
Quoted 12 lines

What bothers me about this is that Nagle seems to be introducing a delay here. The first DATA packets in both directions are MTU-sized packets, yet both the Linux client and the BSD server wait 200ms until they get the SACK to the first fragment before sending the second fragment. The server can't send its reply until it gets both fragments, and the client can't reassemble the reply until it gets both fragments, so from the application's point of view, the reply doesn't arrive until 400ms after the request is sent. This could probably be fixed by disabling Nagle with SCTP_NODELAY, but that shouldn't be required. Nagle is only supposed to prevent multiple outstanding *small* packets.

I think you hit the point which Nagle's algorithm should be not used.

Can you try the following patch?

Quoted 8 lines

[PATCH] sctp: do not used Nagle algorithm while fragmented data is transmitted

If fragmented data is sent, the Nagle's algorithm should not be used. In special case, if only one large packet is sent, the delay send of fragmented data will cause the receiver wait for more fragmented data to reassembe them and not send SACK, but the sender still wait for SACK before send the last fragment.

[patch deleted]

This patch seems to work quite well, but I think disabling Nagle completely for large messages is not quite the right thing to do. There's a draft-minshall-nagle-01.txt floating around that describes a modified Nagle algorithm for TCP. It appears to have been implemented in Linux TCP even though the draft has expired. The modified algorithm is how I thought Nagle had always worked to begin with. From the draft:

"If a TCP has less than a full-sized packet to transmit, and if any previously transmitted less than full-sized packet has not yet been acknowledged, do not transmit a packet."

so in the case of sending a fragmented SCTP message, all but the last fragment will be full-sized and will be sent without delay. The last fragment will usually not be full-sized, but it too will be sent without delay because there are no outstanding non-full-sized packets.

The difference between this and your method is that yours would allow many small fragments of big messages to be outstanding, whereas this one would only allow the first big message to be sent in its entirety, followed by the full-sized fragments of the next big message. When it came time to send the second small fragment, Nagle would force it to wait for an ACK for the first small fragment. I'm not convinced that the difference is all that important, but who knows.

Here's my attempt at implementing the modified Nagle algorithm described in draft-minshall-nagle-01.txt. It should be applied instead of your patch, not on top of it. If (q->outstanding_bytes % asoc->frag_point) is zero, no delay is introduced. The assumption is that this means that all outstanding packets (if any) are full-sized.

---

net/sctp/output.c static sctp_xmit_t sctp_packet_append_da · line 721

Quoted 6 lines
net/sctp/output.c +1 -1
@@ -717,7 +717,8 @@ static sctp_xmit_t sctp_packet_append_da
 	 * unacknowledged.
 	 */
 	if (!sp->nodelay && sctp_packet_empty(packet) &&
-	    q->outstanding_bytes && sctp_state(asoc, ESTABLISHED)) {
+	    (q->outstanding_bytes % asoc->frag_point) != 0 &&
 

I think asoc->unack_data can be used for check full-sized packet. change (q->outstanding_bytes % asoc->frag_point) != 0 to:

+ (q->outstanding_bytes != asoc->frag_point * (asoc->unack_data - 1) &&

Quoted 12 lines
net/sctp/output.c +1 -0
+	    sctp_state(asoc, ESTABLISHED)) {
 		unsigned len = datasize + q->out_qlen;
 

/* Check whether this chunk and all the rest of pending
--
To unsubscribe from this list: send the line "unsubscribe linux-sctp" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#20 lore raw
In reply to Doug Graham
Quoted 57 lines · Doug Graham
Quoted 27 lines · Wei Yongjun
Quoted 9 lines · Doug Graham
 13 2.002632    10.0.0.15   10.0.0.11   DATA (1452 bytes data) 
 14 2.203092    10.0.0.11   10.0.0.15   SACK 
 15 2.203153    10.0.0.15   10.0.0.11   DATA (2 bytes data)
 16 2.203427    10.0.0.11   10.0.0.15   SACK 
 17 2.203808    10.0.0.11   10.0.0.15   DATA (1452 bytes data)
 18 2.403524    10.0.0.15   10.0.0.11   SACK 
 19 2.403686    10.0.0.11   10.0.0.15   DATA (2 bytes data)
 20 2.603285    10.0.0.15   10.0.0.11   SACK 
Quoted 12 lines

What bothers me about this is that Nagle seems to be introducing a delay here. The first DATA packets in both directions are MTU-sized packets, yet both the Linux client and the BSD server wait 200ms until they get the SACK to the first fragment before sending the second fragment. The server can't send its reply until it gets both fragments, and the client can't reassemble the reply until it gets both fragments, so from the application's point of view, the reply doesn't arrive until 400ms after the request is sent. This could probably be fixed by disabling Nagle with SCTP_NODELAY, but that shouldn't be required. Nagle is only supposed to prevent multiple outstanding *small* packets.

I think you hit the point which Nagle's algorithm should be not used.

Can you try the following patch?

Quoted 8 lines

[PATCH] sctp: do not used Nagle algorithm while fragmented data is transmitted

If fragmented data is sent, the Nagle's algorithm should not be used. In special case, if only one large packet is sent, the delay send of fragmented data will cause the receiver wait for more fragmented data to reassembe them and not send SACK, but the sender still wait for SACK before send the last fragment.

[patch deleted]

This patch seems to work quite well, but I think disabling Nagle completely for large messages is not quite the right thing to do. There's a draft-minshall-nagle-01.txt floating around that describes a modified Nagle algorithm for TCP. It appears to have been implemented in Linux TCP even though the draft has expired. The modified algorithm is how I thought Nagle had always worked to begin with. From the draft:

"If a TCP has less than a full-sized packet to transmit, and if any previously transmitted less than full-sized packet has not yet been acknowledged, do not transmit a packet."

so in the case of sending a fragmented SCTP message, all but the last fragment will be full-sized and will be sent without delay. The last fragment will usually not be full-sized, but it too will be sent without delay because there are no outstanding non-full-sized packets.

Quoted 7 lines

The difference between this and your method is that yours would allow many small fragments of big messages to be outstanding, whereas this one would only allow the first big message to be sent in its entirety, followed by the full-sized fragments of the next big message. When it came time to send the second small fragment, Nagle would force it to wait for an ACK for the first small fragment.

This case will never happend because when we fragment data, the fragment size
is always be frag_point expect the last fragment. So either the last fragment is
full size or not, we should not use Nagle algorithm.

Nagle algorithm is not adapt to fragment datas.

Quoted 32 lines

I'm not convinced that the difference is all that important, but who knows.

Here's my attempt at implementing the modified Nagle algorithm described in draft-minshall-nagle-01.txt. It should be applied instead of your patch, not on top of it. If (q->outstanding_bytes % asoc->frag_point) is zero, no delay is introduced. The assumption is that this means that all outstanding packets (if any) are full-sized.

---

net/sctp/output.c +2 -2
@@ -717,7 +717,8 @@ static sctp_xmit_t sctp_packet_append_da
 	 * unacknowledged.
 	 */
 	if (!sp->nodelay && sctp_packet_empty(packet) &&
-	    q->outstanding_bytes && sctp_state(asoc, ESTABLISHED)) {
+	    (q->outstanding_bytes % asoc->frag_point) != 0 &&
+	    sctp_state(asoc, ESTABLISHED)) {
 		unsigned len = datasize + q->out_qlen;
 
 		/* Check whether this chunk and all the rest of pending
--

To unsubscribe from this list: send the line "unsubscribe linux-sctp" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#21 lore raw
In reply to Doug Graham

Oops. Sent the last one in HTML, so the mailing list rejected it. Damned GUI email
clients!

Quoted 109 lines · Wei Yongjun
Quoted 83 lines · Doug Graham
Quoted 29 lines · Wei Yongjun
Quoted 9 lines · Doug Graham
 13 2.002632    10.0.0.15   10.0.0.11   DATA (1452 bytes data) 
 14 2.203092    10.0.0.11   10.0.0.15   SACK 
 15 2.203153    10.0.0.15   10.0.0.11   DATA (2 bytes data)
 16 2.203427    10.0.0.11   10.0.0.15   SACK 
 17 2.203808    10.0.0.11   10.0.0.15   DATA (1452 bytes data)
 18 2.403524    10.0.0.15   10.0.0.11   SACK 
 19 2.403686    10.0.0.11   10.0.0.15   DATA (2 bytes data)
 20 2.603285    10.0.0.15   10.0.0.11   SACK 
Quoted 13 lines

What bothers me about this is that Nagle seems to be introducing a delay here. The first DATA packets in both directions are MTU-sized packets, yet both the Linux client and the BSD server wait 200ms until they get the SACK to the first fragment before sending the second fragment. The server can't send its reply until it gets both fragments, and the client can't reassemble the reply until it gets both fragments, so from the application's point of view, the reply doesn't arrive until 400ms after the request is sent. This could probably be fixed by disabling Nagle with SCTP_NODELAY, but that shouldn't be required. Nagle is only supposed to prevent multiple outstanding *small* packets.

I think you hit the point which Nagle's algorithm should be not used.

Can you try the following patch?

Quoted 9 lines

[PATCH] sctp: do not used Nagle algorithm while fragmented data is transmitted

If fragmented data is sent, the Nagle's algorithm should not be used. In special case, if only one large packet is sent, the delay send of fragmented data will cause the receiver wait for more fragmented data to reassembe them and not send SACK, but the sender still wait for SACK before send the last fragment.

[patch deleted]

This patch seems to work quite well, but I think disabling Nagle completely for large messages is not quite the right thing to do. There's a draft-minshall-nagle-01.txt floating around that describes a modified Nagle algorithm for TCP. It appears to have been implemented in Linux TCP even though the draft has expired. The modified algorithm is how I thought Nagle had always worked to begin with. From the draft:

"If a TCP has less than a full-sized packet to transmit, and if any previously transmitted less than full-sized packet has not yet been acknowledged, do not transmit a packet."

so in the case of sending a fragmented SCTP message, all but the last fragment will be full-sized and will be sent without delay. The last fragment will usually not be full-sized, but it too will be sent without delay because there are no outstanding non-full-sized packets.

The difference between this and your method is that yours would allow many small fragments of big messages to be outstanding, whereas this one would only allow the first big message to be sent in its entirety, followed by the full-sized fragments of the next big message. When it came time to send the second small fragment, Nagle would force it to wait for an ACK for the first small fragment. I'm not convinced that the difference is all that important, but who knows.

Here's my attempt at implementing the modified Nagle algorithm described in draft-minshall-nagle-01.txt. It should be applied instead of your patch, not on top of it. If (q->outstanding_bytes % asoc->frag_point) is zero, no delay is introduced. The assumption is that this means that all outstanding packets (if any) are full-sized.

---

net/sctp/output.c static sctp_xmit_t sctp_packet_append_da · line 724

Quoted 9 lines
net/sctp/output.c +2 -1
@@ -717,7 +717,8 @@ static sctp_xmit_t sctp_packet_append_da
 	 * unacknowledged.
 	 */
 	if (!sp->nodelay && sctp_packet_empty(packet) &&
-	    q->outstanding_bytes && sctp_state(asoc, ESTABLISHED)) {
+	    (q->outstanding_bytes % asoc->frag_point) != 0 &&
+	    sctp_state(asoc, ESTABLISHED)) {
 		unsigned len = datasize + q->out_qlen;
 
 		/* Check whether this chunk and all the rest of pending

Seem good! But it may be broken the small packet transmit which can be used Nagle algorithm.
Such as this:

Endpoint A                Endpint B
          <-------------  DATA (size52/2) delay send
          <-------------  DATA (size52/2) send immediately
          <-------------  DATA (size52/2) send immediately ** broken
          <-------------  DATA (size52/2) delay send
          <-------------  DATA (size52/2) send immediately
          <-------------  DATA (size52/2) send immediately ** broken
Quoted 4 lines

Can you try this one?

I would, except I don't understand what you're getting at. Does this mean to send a total of
6 1454 byte messages from B to A? If so, why would the first one be delayed?

Assuming that no SACKs are received by B, this should result in the first 3 packets getting sent
immediately, a 1452 byte fragment, then a 2 byte fragment, then the second 1452 byte fragment.
When it comes time to send the second 2 byte fragment, Nagle kicks in and prevents if from
being sent until a SACK is received.

But I'm pretty sure I missed your point. Can you flesh it out a bit?

--Doug

Quoted 2 lines
#22 lore raw
In reply to Doug Graham
Quoted 122 lines · Doug Graham

Oops. Sent the last one in HTML, so the mailing list rejected it. Damned GUI email
clients!

Quoted 112 lines · Wei Yongjun
Quoted 86 lines · Doug Graham
Quoted 30 lines · Wei Yongjun
Quoted 15 lines · Doug Graham

13 2.002632 10.0.0.15 10.0.0.11 DATA (1452 bytes data) 14

2.203092    10.0.0.11   10.0.0.15   SACK  15 2.203153   
10.0.0.15   10.0.0.11   DATA (2 bytes data)

16 2.203427 10.0.0.11 10.0.0.15 SACK 17 2.203808
10.0.0.11 10.0.0.15 DATA (1452 bytes data)
18 2.403524 10.0.0.15 10.0.0.11 SACK 19 2.403686
10.0.0.11 10.0.0.15 DATA (2 bytes data)
20 2.603285 10.0.0.15 10.0.0.11 SACK
What bothers me about this is that Nagle seems to be introducing a delay
here. The first DATA packets in both directions are MTU-sized packets,
yet both the Linux client and the BSD server wait 200ms until they get
the SACK to the first fragment before sending the second fragment.

Quoted 9 lines

The server can't send its reply until it gets both fragments, and the client can't reassemble the reply until it gets both fragments, so from
the application's point of view, the reply doesn't arrive until 400ms after the request is sent. This could probably be fixed by disabling Nagle with SCTP_NODELAY, but that shouldn't be required. Nagle is only
supposed to prevent multiple outstanding *small* packets.

I think you hit the point which Nagle's algorithm should be not used.

Can you try the following patch?

Quoted 9 lines

[PATCH] sctp: do not used Nagle algorithm while fragmented data is transmitted

If fragmented data is sent, the Nagle's algorithm should not be used. In special case, if only one large packet is sent, the delay send of fragmented data will cause the receiver wait for more fragmented data to reassembe them and not send SACK, but the sender still wait for SACK before send the last fragment.

[patch deleted]

This patch seems to work quite well, but I think disabling Nagle completely for large messages is not quite the right thing to do. There's a draft-minshall-nagle-01.txt floating around that describes a modified Nagle algorithm for TCP. It appears to have been implemented in Linux TCP even though the draft has expired. The modified algorithm is how I thought Nagle had always worked to begin with. From the draft:

"If a TCP has less than a full-sized packet to transmit, and if any previously transmitted less than full-sized packet has not yet been acknowledged, do not transmit a packet."

so in the case of sending a fragmented SCTP message, all but the last fragment will be full-sized and will be sent without delay. The last fragment will usually not be full-sized, but it too will be sent without
delay because there are no outstanding non-full-sized packets.

The difference between this and your method is that yours would allow many small fragments of big messages to be outstanding, whereas this one would only allow the first big message to be sent in its entirety, followed by the full-sized fragments of the next big message. When it came time to send the second small fragment, Nagle would force it to wait for an ACK for the first small fragment. I'm not convinced that the difference is all that important, but who knows.

Here's my attempt at implementing the modified Nagle algorithm described
in draft-minshall-nagle-01.txt. It should be applied instead of your patch, not on top of it. If (q->outstanding_bytes % asoc->frag_point) is zero, no delay is introduced. The assumption is that this means that
all outstanding packets (if any) are full-sized.

---

net/sctp/output.c 2009/08/02 00:51:18 static sctp_xmit_t sctp_packet_append_da · line 724

Quoted 9 lines
net/sctp/output.c 2009/08/02 00:51:18 renamed from net/sctp/output.c 2009/08/02 00:47:44 1.3 +2 -1
@@ -717,7 +717,8 @@ static sctp_xmit_t sctp_packet_append_da
      * unacknowledged.
      */
     if (!sp->nodelay && sctp_packet_empty(packet) &&
-        q->outstanding_bytes && sctp_state(asoc, ESTABLISHED)) {
+        (q->outstanding_bytes % asoc->frag_point) != 0 &&
+        sctp_state(asoc, ESTABLISHED)) {
         unsigned len = datasize + q->out_qlen;
 
         /* Check whether this chunk and all the rest of pending

Seem good! But it may be broken the small packet transmit which can be used Nagle algorithm.
Such as this:

Endpoint A                Endpint B
          <-------------  DATA (size52/2) delay send
          <-------------  DATA (size52/2) send immediately
          <-------------  DATA (size52/2) send immediately ** broken
          <-------------  DATA (size52/2) delay send
          <-------------  DATA (size52/2) send immediately
          <-------------  DATA (size52/2) send immediately ** broken
Quoted 4 lines

Can you try this one?

Quoted 4 lines

I would, except I don't understand what you're getting at. Does this mean to send a total of
6 1454 byte messages from B to A? If so, why would the first one be delayed?

Oh, no, six 726 bytes(1452/2) messages, may be the 1st and 2nd are bundled in one packet,
the 3rd is a single packet, the 4th, 5th are bundled, the 6th is single. I have no test it.

Quoted 17 lines

Assuming that no SACKs are received by B, this should result in the first 3 packets getting sent
immediately, a 1452 byte fragment, then a 2 byte fragment, then the second 1452 byte fragment.
When it comes time to send the second 2 byte fragment, Nagle kicks in and prevents if from
being sent until a SACK is received.

But I'm pretty sure I missed your point. Can you flesh it out a bit?

--Doug

Quoted 2 lines
#23 lore raw
In reply to Doug Graham
Quoted 71 lines · Wei Yongjun
Quoted 60 lines · Doug Graham
Quoted 29 lines · Wei Yongjun
Quoted 9 lines · Doug Graham
 13 2.002632    10.0.0.15   10.0.0.11   DATA (1452 bytes data) 
 14 2.203092    10.0.0.11   10.0.0.15   SACK 
 15 2.203153    10.0.0.15   10.0.0.11   DATA (2 bytes data)
 16 2.203427    10.0.0.11   10.0.0.15   SACK 
 17 2.203808    10.0.0.11   10.0.0.15   DATA (1452 bytes data)
 18 2.403524    10.0.0.15   10.0.0.11   SACK 
 19 2.403686    10.0.0.11   10.0.0.15   DATA (2 bytes data)
 20 2.603285    10.0.0.15   10.0.0.11   SACK 
Quoted 13 lines

What bothers me about this is that Nagle seems to be introducing a delay here. The first DATA packets in both directions are MTU-sized packets, yet both the Linux client and the BSD server wait 200ms until they get the SACK to the first fragment before sending the second fragment. The server can't send its reply until it gets both fragments, and the client can't reassemble the reply until it gets both fragments, so from the application's point of view, the reply doesn't arrive until 400ms after the request is sent. This could probably be fixed by disabling Nagle with SCTP_NODELAY, but that shouldn't be required. Nagle is only supposed to prevent multiple outstanding *small* packets.

I think you hit the point which Nagle's algorithm should be not used.

Can you try the following patch?

Quoted 9 lines

[PATCH] sctp: do not used Nagle algorithm while fragmented data is transmitted

If fragmented data is sent, the Nagle's algorithm should not be used. In special case, if only one large packet is sent, the delay send of fragmented data will cause the receiver wait for more fragmented data to reassembe them and not send SACK, but the sender still wait for SACK before send the last fragment.

[patch deleted]

This patch seems to work quite well, but I think disabling Nagle completely for large messages is not quite the right thing to do. There's a draft-minshall-nagle-01.txt floating around that describes a modified Nagle algorithm for TCP. It appears to have been implemented in Linux TCP even though the draft has expired. The modified algorithm is how I thought Nagle had always worked to begin with. From the draft:

"If a TCP has less than a full-sized packet to transmit, and if any previously transmitted less than full-sized packet has not yet been acknowledged, do not transmit a packet."

so in the case of sending a fragmented SCTP message, all but the last fragment will be full-sized and will be sent without delay. The last fragment will usually not be full-sized, but it too will be sent without delay because there are no outstanding non-full-sized packets.

Quoted 8 lines

The difference between this and your method is that yours would allow many small fragments of big messages to be outstanding, whereas this one would only allow the first big message to be sent in its entirety, followed by the full-sized fragments of the next big message. When it came time to send the second small fragment, Nagle would force it to wait for an ACK for the first small fragment.

Quoted 10 lines

This case will never happend because when we fragment data, the fragment size
is always be frag_point expect the last fragment. So either the last fragment is
full size or not, we should not use Nagle algorithm.

Nagle algorithm is not adapt to fragment datas.

Why can it never happen? If I send a bunch of large messages with small last fragments, your modification will allow all messages to be sent, because it disables Nagle for large messages, right? If so, many small last fragments can be outstanding at any one time (one from each message). Technically, this violates Nagle, which aims to prevent more than one small fragment from ever being outstanding, but I'm not sure that it really violates the spirit of what Nagle is trying to accomplish.

Nagle is really meant to prevent the case of an application like telnet from sending a whole lot of small packets containing only 1 or a few characters. If the receive window is, say, 10000 bytes, Nagle would allow 10000 packets to be outstanding, all clogging up the network. But if the PMTU is, say, 1000 bytes and the user tries to send a bunch of 1001 byte messages, your method (if I understand it correctly) will allow 9 unacknowledged messages to be outstanding. Those 9 messages will be split into 9 full-sized packets and 9 packets carrying only 1 byte of data. 18 outstanding packets isn't all that bad. If the user were instead sending 1000 byte messages, Nagle would have nothing to say about it, and you'd be able to have 10 packets outstanding. The increase from 10 to 19 outstanding packets isn't likely to cause network collapse.

--Doug

#24 lore raw
In reply to Doug Graham
Quoted 135 lines · Wei Yongjun
Quoted 127 lines · Doug Graham

Oops. Sent the last one in HTML, so the mailing list rejected it. Damned GUI email
clients!

Quoted 116 lines · Wei Yongjun
Quoted 91 lines · Doug Graham
Quoted 32 lines · Wei Yongjun
Quoted 15 lines · Doug Graham

13 2.002632 10.0.0.15 10.0.0.11 DATA (1452 bytes data) 14

2.203092    10.0.0.11   10.0.0.15   SACK  15 2.203153   
10.0.0.15   10.0.0.11   DATA (2 bytes data)

16 2.203427 10.0.0.11 10.0.0.15 SACK 17 2.203808
10.0.0.11 10.0.0.15 DATA (1452 bytes data)
18 2.403524 10.0.0.15 10.0.0.11 SACK 19 2.403686
10.0.0.11 10.0.0.15 DATA (2 bytes data)
20 2.603285 10.0.0.15 10.0.0.11 SACK
What bothers me about this is that Nagle seems to be introducing a delay
here. The first DATA packets in both directions are MTU-sized packets,
yet both the Linux client and the BSD server wait 200ms until they get
the SACK to the first fragment before sending the second fragment.

Quoted 10 lines

The server can't send its reply until it gets both fragments, and the client can't reassemble the reply until it gets both fragments, so from
the application's point of view, the reply doesn't arrive until 400ms after the request is sent. This could probably be fixed by disabling Nagle with SCTP_NODELAY, but that shouldn't be required. Nagle is only
supposed to prevent multiple outstanding *small* packets.

I think you hit the point which Nagle's algorithm should be not used.

Can you try the following patch?

Quoted 10 lines

[PATCH] sctp: do not used Nagle algorithm while fragmented data is transmitted

If fragmented data is sent, the Nagle's algorithm should not be used. In special case, if only one large packet is sent, the delay send of fragmented data will cause the receiver wait for more fragmented data to reassembe them and not send SACK, but the sender still wait for SACK before send the last fragment.

[patch deleted]

This patch seems to work quite well, but I think disabling Nagle completely for large messages is not quite the right thing to do. There's a draft-minshall-nagle-01.txt floating around that describes a modified Nagle algorithm for TCP. It appears to have been implemented in Linux TCP even though the draft has expired. The modified algorithm is how I thought Nagle had always worked to begin with. From the draft:

"If a TCP has less than a full-sized packet to transmit, and if any previously transmitted less than full-sized packet has not yet been acknowledged, do not transmit a packet."

so in the case of sending a fragmented SCTP message, all but the last fragment will be full-sized and will be sent without delay. The last fragment will usually not be full-sized, but it too will be sent without
delay because there are no outstanding non-full-sized packets.

The difference between this and your method is that yours would allow many small fragments of big messages to be outstanding, whereas this one would only allow the first big message to be sent in its entirety, followed by the full-sized fragments of the next big message. When it came time to send the second small fragment, Nagle would force it to wait for an ACK for the first small fragment. I'm not convinced that the difference is all that important, but who knows.

Here's my attempt at implementing the modified Nagle algorithm described
in draft-minshall-nagle-01.txt. It should be applied instead of your patch, not on top of it. If (q->outstanding_bytes % asoc->frag_point) is zero, no delay is introduced. The assumption is that this means that
all outstanding packets (if any) are full-sized.

---

net/sctp/output.c 2009/08/02 00:51:18 static sctp_xmit_t sctp_packet_append_da · line 724

Quoted 9 lines
net/sctp/output.c 2009/08/02 00:51:18 renamed from net/sctp/output.c 2009/08/02 00:47:44 1.3 +2 -1
@@ -717,7 +717,8 @@ static sctp_xmit_t sctp_packet_append_da
      * unacknowledged.
      */
     if (!sp->nodelay && sctp_packet_empty(packet) &&
-        q->outstanding_bytes && sctp_state(asoc, ESTABLISHED)) {
+        (q->outstanding_bytes % asoc->frag_point) != 0 &&
+        sctp_state(asoc, ESTABLISHED)) {
         unsigned len = datasize + q->out_qlen;
 
         /* Check whether this chunk and all the rest of pending

Seem good! But it may be broken the small packet transmit which can be used Nagle algorithm.
Such as this:

Endpoint A                Endpint B
          <-------------  DATA (size52/2) delay send
          <-------------  DATA (size52/2) send immediately
          <-------------  DATA (size52/2) send immediately ** broken
          <-------------  DATA (size52/2) delay send
          <-------------  DATA (size52/2) send immediately
          <-------------  DATA (size52/2) send immediately ** broken
Quoted 5 lines

Can you try this one?

Quoted 5 lines

I would, except I don't understand what you're getting at. Does this mean to send a total of
6 1454 byte messages from B to A? If so, why would the first one be delayed?

Quoted 6 lines

Oh, no, six 726 bytes(1452/2) messages, may be the 1st and 2nd are bundled in one packet,
the 3rd is a single packet, the 4th, 5th are bundled, the 6th is single. I have no test it.

Ah, so that / meant *division*! I thought that was your notation meaning that the
packets were fragmented into a 1452 byte chunk and a 2 byte chunk! Makes more sense now :-)

I admit that I didn't study too closely exactly what q->outstanding_bytes represents. I assumed
it meant the number of bytes that had been sent on the wire, but not yet acknowledged.
Any bytes that were delayed because of Nagle would not be counted in outstanding_bytes
(I assume). So the first send of 726 would get sent immediately and counted in
outstanding_bytes. The second one would get delayed by Nagle and not counted
in outstanding_bytes. All the later ones would also get delayed by Nagle because
outstanding_bytes is still 726.

I do think that using outstanding_bytes the way I did is probably an ugly kludge, and
there's hopefully a better way. But the right way will probably involve adding
some more state to each association (the snd.sml variable mentioned in the minshall
draft at the very least). I'm not sure that using asoc->frag_point the way I did is correct
either, because I think the frag_point can change during the lifetime of an association.

--Doug

#25 lore raw
In reply to Doug Graham
Quoted 11 lines
Quoted 2 lines

Why can it never happen? If I send a bunch of large messages with small last fragments, your modification will allow all messages to be sent, because it disables Nagle for large messages, right? If so, many small last fragments can be outstanding at any one time (one from each message). Technically, this violates Nagle, which aims to prevent more than one small fragment from ever being outstanding, but I'm not sure that it really violates the spirit of what Nagle is trying to accomplish.

Quoted 5 lines

Nagle is really meant to prevent the case of an application like telnet from sending a whole lot of small packets containing only 1 or a few characters. If the receive window is, say, 10000 bytes, Nagle would allow 10000 packets to be outstanding, all clogging up the network.

I meant to say that *without* Nagle, you'd be able to have 10000 packets outstanding. This is what Nagle is trying to prevent.

Quoted 13 lines

But if the PMTU is, say, 1000 bytes and the user
tries to send a bunch of 1001 byte messages, your method (if I understand it correctly) will allow 9 unacknowledged messages to be outstanding. Those 9 messages will be split into 9 full-sized packets and 9 packets carrying only 1 byte of data. 18 outstanding packets isn't all that bad. If the user were instead sending 1000 byte messages, Nagle would have nothing to say about it, and you'd be able to have 10 packets outstanding. The increase from 10 to 19 outstanding packets isn't likely to cause network collapse.

--Doug

#26 lore raw
In reply to Doug Graham
Quoted 159 lines · Doug Graham
Quoted 133 lines · Wei Yongjun
Quoted 121 lines · Doug Graham

Oops. Sent the last one in HTML, so the mailing list rejected it. Damned GUI email
clients!

Quoted 115 lines · Wei Yongjun
Quoted 90 lines · Doug Graham
Quoted 33 lines · Wei Yongjun
Quoted 18 lines · Doug Graham

13 2.002632 10.0.0.15 10.0.0.11 DATA (1452 bytes data) 14

2.203092    10.0.0.11   10.0.0.15   SACK  15 2.203153  
10.0.0.15   10.0.0.11   DATA (2 bytes data)

16 2.203427 10.0.0.11 10.0.0.15 SACK 17 2.203808
10.0.0.11 10.0.0.15 DATA (1452 bytes data)
18 2.403524 10.0.0.15 10.0.0.11 SACK 19 2.403686
10.0.0.11 10.0.0.15 DATA (2 bytes data)
20 2.603285 10.0.0.15 10.0.0.11 SACK
What bothers me about this is that Nagle seems to be introducing a delay
here. The first DATA packets in both directions are MTU-sized packets,
yet both the Linux client and the BSD server wait 200ms until they get
the SACK to the first fragment before sending the second fragment. The server can't send its reply until it gets both fragments, and the
client can't reassemble the reply until it gets both fragments, so

Quoted 9 lines

from
the application's point of view, the reply doesn't arrive until 400ms
after the request is sent. This could probably be fixed by disabling
Nagle with SCTP_NODELAY, but that shouldn't be required. Nagle is only
supposed to prevent multiple outstanding *small* packets.

I think you hit the point which Nagle's algorithm should be not used.

Can you try the following patch?

Quoted 9 lines

[PATCH] sctp: do not used Nagle algorithm while fragmented data is transmitted

If fragmented data is sent, the Nagle's algorithm should not be used. In special case, if only one large packet is sent, the delay send of fragmented data will cause the receiver wait for more fragmented data to reassembe them and not send SACK, but the sender still wait for SACK before send the last fragment.

[patch deleted]

This patch seems to work quite well, but I think disabling Nagle completely for large messages is not quite the right thing to do. There's a draft-minshall-nagle-01.txt floating around that describes a modified Nagle algorithm for TCP. It appears to have been implemented in Linux TCP even though the draft has expired. The modified algorithm
is how I thought Nagle had always worked to begin with. From the draft:

"If a TCP has less than a full-sized packet to transmit, and if any previously transmitted less than full-sized packet has not yet been acknowledged, do not transmit a packet."

so in the case of sending a fragmented SCTP message, all but the last fragment will be full-sized and will be sent without delay. The last fragment will usually not be full-sized, but it too will be sent without
delay because there are no outstanding non-full-sized packets.

The difference between this and your method is that yours would allow many small fragments of big messages to be outstanding, whereas this one would only allow the first big message to be sent in its entirety, followed by the full-sized fragments of the next big message. When it came time to send the second small fragment, Nagle would force it to wait for an ACK for the first small fragment. I'm not convinced that the difference is all that important, but who knows.

Here's my attempt at implementing the modified Nagle algorithm described
in draft-minshall-nagle-01.txt. It should be applied instead of your patch, not on top of it. If (q->outstanding_bytes % asoc->frag_point) is zero, no delay is introduced. The assumption is that this means that
all outstanding packets (if any) are full-sized.

---

net/sctp/output.c 2009/08/02 00:51:18 static sctp_xmit_t sctp_packet_append_da · line 724

Quoted 9 lines
net/sctp/output.c 2009/08/02 00:51:18 renamed from net/sctp/output.c 2009/08/02 00:47:44 1.3 +2 -1
@@ -717,7 +717,8 @@ static sctp_xmit_t sctp_packet_append_da
      * unacknowledged.
      */
     if (!sp->nodelay && sctp_packet_empty(packet) &&
-        q->outstanding_bytes && sctp_state(asoc, ESTABLISHED)) {
+        (q->outstanding_bytes % asoc->frag_point) != 0 &&
+        sctp_state(asoc, ESTABLISHED)) {
         unsigned len = datasize + q->out_qlen;
 
         /* Check whether this chunk and all the rest of pending

Seem good! But it may be broken the small packet transmit which can be used Nagle algorithm.
Such as this:

Endpoint A                Endpint B
          <-------------  DATA (size52/2) delay send
          <-------------  DATA (size52/2) send immediately
          <-------------  DATA (size52/2) send immediately ** broken
          <-------------  DATA (size52/2) delay send
          <-------------  DATA (size52/2) send immediately
          <-------------  DATA (size52/2) send immediately ** broken
Quoted 9 lines
Quoted 4 lines

Can you try this one?

I would, except I don't understand what you're getting at. Does this mean to send a total of
6 1454 byte messages from B to A? If so, why would the first one be delayed?

Quoted 6 lines

Oh, no, six 726 bytes(1452/2) messages, may be the 1st and 2nd are bundled in one packet,
the 3rd is a single packet, the 4th, 5th are bundled, the 6th is single. I have no test it.

Ah, so that / meant *division*! I thought that was your notation meaning that the
packets were fragmented into a 1452 byte chunk and a 2 byte chunk! Makes more sense now :-)

I admit that I didn't study too closely exactly what q->outstanding_bytes represents. I assumed
it meant the number of bytes that had been sent on the wire, but not yet acknowledged.
Any bytes that were delayed because of Nagle would not be counted in outstanding_bytes
(I assume). So the first send of 726 would get sent immediately and counted in
outstanding_bytes. The second one would get delayed by Nagle and not counted
in outstanding_bytes. All the later ones would also get delayed by Nagle because
outstanding_bytes is still 726.

Quoted 10 lines

I do think that using outstanding_bytes the way I did is probably an ugly kludge, and
there's hopefully a better way. But the right way will probably involve adding
some more state to each association (the snd.sml variable mentioned in the minshall
draft at the very least). I'm not sure that using asoc->frag_point the way I did is correct
either, because I think the frag_point can change during the lifetime of an association.

Using division in such a hot path is a non-starter to begin with, so we definitely need to find a better way.

Using frag_point is not the right way to do it either since it's effected by MTU and user API.

I think we can add something to sctp_outq structure to properly track this.

-vlad

Quoted 3 lines

--Doug

#27 lore raw
In reply to Doug Graham
Quoted 25 lines · Vlad Yasevich
Quoted 14 lines

I admit that I didn't study too closely exactly what q->outstanding_bytes represents. I assumed
it meant the number of bytes that had been sent on the wire, but not yet acknowledged.
Any bytes that were delayed because of Nagle would not be counted in outstanding_bytes
(I assume). So the first send of 726 would get sent immediately and counted in
outstanding_bytes. The second one would get delayed by Nagle and not counted
in outstanding_bytes. All the later ones would also get delayed by Nagle because
outstanding_bytes is still 726.

Quoted 10 lines

I do think that using outstanding_bytes the way I did is probably an ugly kludge, and
there's hopefully a better way. But the right way will probably involve adding
some more state to each association (the snd.sml variable mentioned in the minshall
draft at the very least). I'm not sure that using asoc->frag_point the way I did is correct
either, because I think the frag_point can change during the lifetime of an association.

Quoted 2 lines

Using division in such a hot path is a non-starter to begin with, so we definitely need to find a better way.

That thought crossed by mind too, although I didn't consider it as much of a show-stopper as you do. 32 bit integer division isn't really all that expensive on modern processors is it? The C compiler is probably doing it in places as a result of pointer arithmetic anyway.

Quoted 4 lines

Using frag_point is not the right way to do it either since it's effected by MTU and user API.

I think we can add something to sctp_outq structure to properly track this.

I've pretty much convinced myself that Wei's original Nagle patch is fine anyway. Just disable Nagle for large messages that need to be fragmented.

--Doug.

#28 lore raw
In reply to Doug Graham
Quoted 33 lines · Doug Graham
Quoted 24 lines · Vlad Yasevich
Quoted 14 lines

I admit that I didn't study too closely exactly what q->outstanding_bytes represents. I assumed
it meant the number of bytes that had been sent on the wire, but not yet acknowledged.
Any bytes that were delayed because of Nagle would not be counted in outstanding_bytes
(I assume). So the first send of 726 would get sent immediately and counted in
outstanding_bytes. The second one would get delayed by Nagle and not counted
in outstanding_bytes. All the later ones would also get delayed by Nagle because
outstanding_bytes is still 726.

Quoted 10 lines

I do think that using outstanding_bytes the way I did is probably an ugly kludge, and
there's hopefully a better way. But the right way will probably involve adding
some more state to each association (the snd.sml variable mentioned in the minshall
draft at the very least). I'm not sure that using asoc->frag_point the way I did is correct
either, because I think the frag_point can change during the lifetime of an association.

Quoted 2 lines

Using division in such a hot path is a non-starter to begin with, so we definitely need to find a better way.

That thought crossed by mind too, although I didn't consider it as much of a show-stopper as you do. 32 bit integer division isn't really all that expensive on modern processors is it? The C compiler is probably doing it in places as a result of pointer arithmetic anyway.

Quoted 8 lines
Quoted 4 lines

Using frag_point is not the right way to do it either since it's effected by MTU and user API.

I think we can add something to sctp_outq structure to properly track this.

I've pretty much convinced myself that Wei's original Nagle patch is fine anyway. Just disable Nagle for large messages that need to be fragmented.

We can't do it blindly since the user may set a fragmentation point such that we'll send sub-MSS (for lack of a better term) segments. I need to look at this more closely.

-vlad

Quoted 6 lines

--Doug.
--
To unsubscribe from this list: send the line "unsubscribe linux-sctp" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html