Skip to contentSkip to conversation

high latency with TCP connections

PATCH In review Alexander Vodomerov 11 participants 43 messages Aug 30, 2006 – Sep 22, 2006 1 message not in the archive

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

Hello!

I'm writing an application that is working over TCP. Total traffic is very low (~ 10 kb/sec), but performance is very bad. I've tried to investigate problem with tcpdump and strace, and it shows that application does multiple writes, but TCP buffers them and send after some delay (about 40 msec). Due to nature of my application, it is essential to send any available data ASAP (decreased bandwidth is not important). I've set TCP_NODELAY option on socket, but it doesn't help.

We've written a simple program to reproduce the effect. It sends 10 small packets, then sleeps for 0.1 sec. Another node tries to receive data. Strace shows that 2 packets are sent immediately and other 8 are grouped together and delayed by 40 msec.

It is interesting that this effect can be seen not only on Ethernet links, but on loopback also (with the same magic constant of 40 msec).

Here is a test run:
server (should be run first):

$ ./a.out 1 5000
Server: begin send_all
Server: total time 14.216441
  client:
$ ./a.out 2 5000 localhost
Client: connected to localhost:5000
Client: begin receive_all
Client: total time 14.223265

Expected time is 10.0 sec (instead of 14.0 sec).
If packets are received more often (DELIM constant is set to 1 or 2) then effect disappear.

Is this a desired behaviour? How can I specify that packets should be sent really immediately after write?

Some people reported that this program runs in 9.997 sec when run on FreeBSD.

Please cc me on replies, as I'm not subscribed to mailing list.

With best regards,
Alexander.

Listing follows:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <assert.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/time.h>
#include <netdb.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
#include <time.h>

int TOTAL_SENDS = 1000;
int DELIM = 10;

int sock = -1;

int init_server(int port)

{
	struct sockaddr_in sin;
	struct sockaddr_in new_sin;
	int new_sock;
	int val;
	int sockaddrlen;

	sock = socket(PF_INET, SOCK_STREAM, 0);
	if (sock == -1) return -1;

	val = 1;
	if (setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &val, sizeof(val)) < 0) return -1;

	memset(&sin,0,sizeof(struct sockaddr_in));
        sin.sin_family = AF_INET;
	sin.sin_port = htons(port);
	
	if (-1 == bind(sock,(struct sockaddr*)&sin,sizeof(sin))) return -2;
	if (-1 == listen(sock,1)) return -3;

	sockaddrlen = sizeof(struct sockaddr_in);
        new_sock = accept(sock,(struct sockaddr*)&new_sin,(socklen_t*)&sockaddrlen);
	if (new_sock == -1) return -4;

	sock = new_sock;
	val = 1;
	if (setsockopt(sock, IPPROTO_TCP, TCP_NODELAY, &val, sizeof(val)) != 0) return -5;
	return 0;
}

int init_client(char* hostname, int port)

{
	int val;
	int res;
	struct sockaddr_in sin;

	sock = socket(PF_INET, SOCK_STREAM, 0);

	memset(&sin, 0, sizeof(sin));
	sin.sin_family = AF_INET;
	sin.sin_port = htons(port);
	memcpy(&sin.sin_addr, gethostbyname(hostname)->h_addr, 4);
  
	val = 1;
	setsockopt(sock, IPPROTO_TCP, TCP_NODELAY, &val, sizeof(val));

	res = connect(sock, (struct sockaddr*)&sin, sizeof(sin));
	printf("Client: connected to %s:%d\n", hostname, port);
	if (res == -1) return -1;
        setsockopt(sock, IPPROTO_TCP, TCP_NODELAY, &val, sizeof(val));
	return 0;
}

void send_all(unsigned long delay)

{
	int i;
	char buf[1024];
	printf("Server: begin send_all\n");
	for (i = 1; i < TOTAL_SENDS; ++i) {
                write(sock, buf, 1);
		if (i % DELIM == 0) read(sock, buf, 1);
		if (i % 10 == 0) usleep(delay);
	}
}

void receive_all(unsigned long delay)

{
	int i;
	char buf[1024];
	printf("Client: begin receive_all\n");
	for (i = 1; i < TOTAL_SENDS; ++i) {
		read(sock, buf, 1);
		if (i % DELIM == 0) write(sock, buf, 1);
		if (i % 10 == 0) usleep(delay);
	}
}

int main(int argc, char* argv[])

{
	int port;
	char* host;
	int me;
	struct timeval tv1, tv2;
	double tt;

	assert(argc > 2);
	me = atoi(argv[1]);
	switch (me) {
		case 1:
			port = atoi(argv[2]);
			if (init_server(port)) {
				printf("Server initialization failed!\n");
				return 1;
			}
			gettimeofday(&tv1, 0);
			send_all(100000);
			gettimeofday(&tv2, 0);
			tt = tv2.tv_sec - tv1.tv_sec + (tv2.tv_usec - tv1.tv_usec) * 0.000001;
			printf("Server: total time %f\n", tt);
			break;
		case 2:
			assert(argc == 4);
			port = atoi(argv[2]);
			host = argv[3];
			if (init_client(host, port)) {
				printf("Client initialization failed!\n");
				return 1;
			}
			gettimeofday(&tv1, 0);
			receive_all(100000);
			gettimeofday(&tv2, 0);
			tt = tv2.tv_sec - tv1.tv_sec + (tv2.tv_usec - tv1.tv_usec) * 0.000001;
			printf("Client: total time %f\n", tt);

			break;
		default:
			printf("Wrong parameter\n");
			return 1;
	}
	shutdown(sock, SHUT_RDWR);
	close(sock);
	return 0;
}
#1 lore raw
Quoted 9 lines · On Wed, 30 Aug 2006 14:07:34 +0400 Alexander Vodomerov <[email protected]>…

Hello!

I'm writing an application that is working over TCP. Total traffic is very low (~ 10 kb/sec), but performance is very bad. I've tried to investigate problem with tcpdump and strace, and it shows that application does multiple writes, but TCP buffers them and send after some delay (about 40 msec). Due to nature of my application, it is essential to send any available data ASAP (decreased bandwidth is not important). I've set TCP_NODELAY option on socket, but it doesn't help.

Linux TCP implements "Appropriate Byte Count" (ABC) and this penalizes applications that do small sends. The problem is that the other side may be delaying acknowledgments. If receiver doesn't acknowledge the sender will limit itself to the congestion window. If the flow is light, then you will be limited to 4 packets.

Quoted 36 lines

We've written a simple program to reproduce the effect. It sends 10 small packets, then sleeps for 0.1 sec. Another node tries to receive data. Strace shows that 2 packets are sent immediately and other 8 are grouped together and delayed by 40 msec.

It is interesting that this effect can be seen not only on Ethernet links, but on loopback also (with the same magic constant of 40 msec).

Here is a test run:
server (should be run first):

$ ./a.out 1 5000
Server: begin send_all
Server: total time 14.216441
  client:
$ ./a.out 2 5000 localhost
Client: connected to localhost:5000
Client: begin receive_all
Client: total time 14.223265

Expected time is 10.0 sec (instead of 14.0 sec).
If packets are received more often (DELIM constant is set to 1 or 2) then effect disappear.

Is this a desired behaviour? How can I specify that packets should be sent really immediately after write?

Some people reported that this program runs in 9.997 sec when run on FreeBSD.

Please cc me on replies, as I'm not subscribed to mailing list.

With best regards,
Alexander.

Quoted 7 lines

void send_all(unsigned long delay)

{
	int i;
	char buf[1024];
	printf("Server: begin send_all\n");
	for (i = 1; i < TOTAL_SENDS; ++i) {
                write(sock, buf, 1);

Expecting any performance with one byte write's is silly.

maintainer
#2 lore raw

From: Stephen Hemminger <[email protected]>
Date: Wed, 30 Aug 2006 10:27:27 -0700

Quoted 5 lines

Linux TCP implements "Appropriate Byte Count" (ABC) and this penalizes applications that do small sends. The problem is that the other side may be delaying acknowledgments. If receiver doesn't acknowledge the sender will limit itself to the congestion window. If the flow is light, then you will be limited to 4 packets.

Right.

However it occured to me the other day that ABC could be made smarter. If we sent small frames, ABC should account for that.

The problem with ABC is that it prevents CWND growth not just during ACK division, but also when we truly are sending smaller sized frames.

In fact, for "chatty" protocols, the real load on a router for the small packets is much less than that for full sized frames. So it is in fact these small frame sending cases for which we can be less conservative, whatever that means here.

So my suggestion is that ABC should go look in the retransmit queue and see how many "real packets" are being fully ACK'd, rather than assuming the send queue is composed of MSS sized frames.

I also think we should seriously consider changing the ABC default to "2" rather than "1".

Quoted 1 line

Expecting any performance with one byte write's is silly.

This is absolutely true. TCP_NODELAY can only save you when you are sending a small amount of data "in aggregate", such as in an SSH or telnet session, whereas in the case being shown here a large amount of data is being sent in small chunks which will always get bad performance.

#3 lore raw
Quoted 22 lines · On Wed, 30 Aug 2006 14:39:55 -0700 (PDT) David Miller <[email protected]>…

From: Stephen Hemminger <[email protected]>
Date: Wed, 30 Aug 2006 10:27:27 -0700

Quoted 5 lines

Linux TCP implements "Appropriate Byte Count" (ABC) and this penalizes applications that do small sends. The problem is that the other side may be delaying acknowledgments. If receiver doesn't acknowledge the sender will limit itself to the congestion window. If the flow is light, then you will be limited to 4 packets.

Right.

However it occured to me the other day that ABC could be made smarter. If we sent small frames, ABC should account for that.

The problem with ABC is that it prevents CWND growth not just during ACK division, but also when we truly are sending smaller sized frames.

In fact, for "chatty" protocols, the real load on a router for the small packets is much less than that for full sized frames. So it is in fact these small frame sending cases for which we can be less conservative, whatever that means here.

Quoted 6 lines

So my suggestion is that ABC should go look in the retransmit queue and see how many "real packets" are being fully ACK'd, rather than assuming the send queue is composed of MSS sized frames.

I also think we should seriously consider changing the ABC default to "2" rather than "1".

That would be a good simple first step. I can't hurt and seems reasonable.

Signature · 1 line
Stephen Hemminger <[email protected]>
#4 lore raw
In reply to David Miller
Quoted 11 lines · David Miller

From: Stephen Hemminger <[email protected]>
Date: Wed, 30 Aug 2006 10:27:27 -0700

Quoted 5 lines

Linux TCP implements "Appropriate Byte Count" (ABC) and this penalizes applications that do small sends. The problem is that the other side may be delaying acknowledgments. If receiver doesn't acknowledge the sender will limit itself to the congestion window. If the flow is light, then you will be limited to 4 packets.

Quoted 4 lines

Right.

However it occured to me the other day that ABC could be made smarter. If we sent small frames, ABC should account for that.

Is that part of the application of a byte-based RFC to packet-counting cwnd?

rick jones

#5 lore raw
In reply to David Miller
Quoted 7 lines · David Miller
Quoted 1 line

Expecting any performance with one byte write's is silly.

This is absolutely true. TCP_NODELAY can only save you when you are sending a small amount of data "in aggregate", such as in an SSH or telnet session, whereas in the case being shown here a large amount of data is being sent in small chunks which will always get bad performance.

Information is sent with one byte write's because it is not available at the moment of sending (it may be read from hardware device or user). If I change 1 to 10 or 100 nothing changes. I'm afraid there is a bit of misunderstanding here. Only very small amount of data is being sent over network. The total traffic for example I sent is only 10 bytes/s. After every 10th packet program does usleep(100000) to simulate pause before next available data.

There are really 3 factors:
1) total size of information is small
2) data for transferring is arrived by small portions from external source
3) it is very important that any portion should be delivered to receiver as soon as possible.
Is TCP is good choice for such transfer or some other protocol is better suited?

With best regards,
Alexander.

#6 lore raw
Quoted 19 lines · Alexander Vodomerov
Quoted 8 lines · David Miller
Quoted 2 lines

Expecting any performance with one byte write's is silly.

This is absolutely true. TCP_NODELAY can only save you when you are sending a small amount of data "in aggregate", such as in an SSH or telnet session, whereas in the case being shown here a large amount of data is being sent in small chunks which will always get bad performance.

Information is sent with one byte write's because it is not available at the moment of sending (it may be read from hardware device or user). If I change 1 to 10 or 100 nothing changes. I'm afraid there is a bit of misunderstanding here. Only very small amount of data is being sent over network. The total traffic for example I sent is only 10 bytes/s. After every 10th packet program does usleep(100000) to simulate pause before next available data.

Quoted 9 lines

There are really 3 factors:
1) total size of information is small
2) data for transferring is arrived by small portions from external source
3) it is very important that any portion should be delivered to receiver as soon as possible.
Is TCP is good choice for such transfer or some other protocol is better suited?

If message boundary preservation is a useful feature for your app, you could try SCTP. You should be able to do this by replacing IPPROTO_TCP with IPPROTO_SCTP and TCP_NODELAY with SCTP_NODELAY.

Thanks
Sridhar

Quoted 7 lines

With best regards,
Alexander.
-
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#7 lore raw
In reply to David Miller
Quoted 8 lines · David Miller

From: Stephen Hemminger <[email protected]>

Quoted 1 line

Expecting any performance with one byte write's is silly.

This is absolutely true. TCP_NODELAY can only save you when you are sending a small amount of data "in aggregate", such as in an SSH or telnet session, whereas in the case being shown here a large amount of data is being sent in small chunks which will always get bad performance.

The word performance in this list seems to always mean 'throughput'. It seems though that there could be some knob to tweak for those of us who don't care so much about throughput but care a great deal about latency.

-K

#8 lore raw
Quoted 13 lines · Kelly Burkhart
Quoted 8 lines · David Miller

From: Stephen Hemminger <[email protected]>

Quoted 1 line

Expecting any performance with one byte write's is silly.

This is absolutely true. TCP_NODELAY can only save you when you are sending a small amount of data "in aggregate", such as in an SSH or telnet session, whereas in the case being shown here a large amount of data is being sent in small chunks which will always get bad performance.

Quoted 4 lines

The word performance in this list seems to always mean 'throughput'. It seems though that there could be some knob to tweak for those of us who don't care so much about throughput but care a great deal about latency.

IIRC Apart from interactions with Nagle (TCP_NODELAY) or the mixing of packet and byte-based congestion control and avoidance heuristics, there really isn't much of anything else to tweak in TCP. If it can send data, it sends data.

Where there _is_ a knob to turn these days might be down with the drivers and their NICs' use of interrupt coalescing:

ftp://ftp.cup.hp.com/dist/networking/briefs/nic_latency_vs_tput.txt

rick jones

#9 lore raw
In reply to Kelly Burkhart
Quoted 5 lines

The word performance in this list seems to always mean 'throughput'. It seems though that there could be some knob to tweak for those of us who don't care so much about throughput but care a great deal about latency.

SCTP has been mentioned. There is also DCCP - http://www.read.cs.ucla.edu/dccp/

Signature · 7 lines
Ian McDonald
Web: http://wand.net.nz/~iam4
Blog: http://imcdnzl.blogspot.com
WAND Network Research Group
Department of Computer Science
University of Waikato
New Zealand
#10 lore raw
In reply to Stephen Hemminger

Hello!

Quoted 1 line

Expecting any performance with one byte write's is silly.

I am not sure why you are so confident about status of ABC. I missed the discussions, when it was implemented. Apparently, it was noticed that ABC in its pure form does not make sense with snd_cwnd counted in packets and there were some reasons, why it still was not adapted.

That's what I am sure: if congestion control causes delack syndrome, it is difficult to advocate this congestion control scheme. It must be repaired, if there is even the smallest possibility to do this.

The first idea:

Behaviour is broken, when we have < 2*MSS in flight. In that case receiver falls to delack timeout and breaks ACK clock.

	thresh = tp->mss_cache;

	if (tp->snd_nxt - tp->snd_una <= tp->mss_cache) {
		thresh = 1;
		if (tp->send_head)
			thresh = tp->send_head->len;
		/* If send_head is empty, it disables ABC.
		 * If send_head is good mss sized segment, ABC is in work.
		 * If send_head is a small segment, it is half-way.
		 */
	}

	if (tp->bytes_acked < thresh)
	       return;

Probably, do this only when TCP_NODELAY is set, but actually it does not matter. If it is nagled/corked, cwnd will not be used/increased in any case.

#11 lore raw
Quoted 9 lines · On Fri, 1 Sep 2006 01:46:35 +0400 Alexey Kuznetsov <[email protected]> wrote:

Hello!

Quoted 1 line

Expecting any performance with one byte write's is silly.

I am not sure why you are so confident about status of ABC. I missed the discussions, when it was implemented. Apparently, it was noticed that ABC in its pure form does not make sense with snd_cwnd counted in packets and there were some reasons, why it still was not adapted.

I implemented it but don't think ABC is the correct thing to be doing in all cases.

If you read the RFC3465, the problem it is trying to address is that of small packets causing growth of congestion window beyond the capacity of the link.

It makes a number of assumptions that may not be true for Linux:
* ABC doesn't take into account congestion window validation RFC2861 already prevents most of the problem of inflated growth.
* ABC assumes that the "true" capacity of the link is limited by byte count not packet count.

Quoted 26 lines

That's what I am sure: if congestion control causes delack syndrome, it is difficult to advocate this congestion control scheme. It must be repaired, if there is even the smallest possibility to do this.

The first idea:

Behaviour is broken, when we have < 2*MSS in flight. In that case receiver falls to delack timeout and breaks ACK clock.

	thresh = tp->mss_cache;

	if (tp->snd_nxt - tp->snd_una <= tp->mss_cache) {
		thresh = 1;
		if (tp->send_head)
			thresh = tp->send_head->len;
		/* If send_head is empty, it disables ABC.
		 * If send_head is good mss sized segment, ABC is in work.
		 * If send_head is a small segment, it is half-way.
		 */
	}

	if (tp->bytes_acked < thresh)
	       return;
Quoted 2 lines

Probably, do this only when TCP_NODELAY is set, but actually it does not matter. If it is nagled/corked, cwnd will not be used/increased in any case.

Maybe it makes better sense to add a bonus to bytes_acked on each ack to try and make small packets get an increase sooner, and try and make the cwnd * mss_cache estimate be more accurate. In the ideal case, during slow start with small packets cwnd should increase at the same rate that it would with larger packets.

maintainer
#12 lore raw

From: Stephen Hemminger <[email protected]>
Date: Thu, 31 Aug 2006 15:14:56 -0700

Quoted 18 lines
Quoted 7 lines · On Fri, 1 Sep 2006 01:46:35 +0400 Alexey Kuznetsov <[email protected]> wrote:
Quoted 1 line

Expecting any performance with one byte write's is silly.

I am not sure why you are so confident about status of ABC. I missed the discussions, when it was implemented. Apparently, it was noticed that ABC in its pure form does not make sense with snd_cwnd counted in packets and there were some reasons, why it still was not adapted.

I implemented it but don't think ABC is the correct thing to be doing in all cases.

If you read the RFC3465, the problem it is trying to address is that of small packets causing growth of congestion window beyond the capacity of the link.

Quoted 5 lines

It makes a number of assumptions that may not be true for Linux:
* ABC doesn't take into account congestion window validation RFC2861 already prevents most of the problem of inflated growth.
* ABC assumes that the "true" capacity of the link is limited by byte count not packet count.

It seems to me that the thing gained by ABC are twofold:

1) protection against ACK division
2) a way to take delayed ACKs into account for cwnd growth

Both of which can be obtained by simply validating the ACK against the retransmit queue, returning number of true packets ACK'd.

I would even go so far as to suggest that we should drop ACKs which do not fall on packetization boundaries. Perhaps only when not in LOSS state, but I doubt that this matters in practice.

Cases where mid-packet ACK is valid are truly marginal ones involving repacketization wrt. MSS/MTU changes, and these would self-correct eventually.

I agree that ABC has some problems. Solution is good, implementation is just horrible :-)

#13 lore raw

Hello!

Quoted 1 line

2) a way to take delayed ACKs into account for cwnd growth

This part is OK now, right?

Quoted 1 line

1) protection against ACK division

But Linux never had this problem... Congestion window was increased only when a whole skb is ACKed, flag FLAG_DATA_ACKED. (TSO could break this, but should not). Otherwise, this ACK just advanced snd_una and nothing more.

This aspect of ABC is crucial for BSD. TCP_NODELAY sockets did not obey congestion control there. From the very beginning, before slow start it can send thousands of 1 byte segments.

The only problem of kind "too-aggressive" with Linux was that we could develop large cwnd sending small segments, and then switch to sending mss-sized segments. It does not look scary, to be honest. :-)

Linux had troubles with slow start even before ABC. Actually, some of applications can suffer of the same syndrome even if ABC disabled. With ABC it becomes TROUBLE, cwnd has no chances to develop at all.

Probably, aspect 1 of ABC just should be disabled. And the first my suggestion looks working too.

Alexey

maintainer
#14 lore raw

From: Alexey Kuznetsov <[email protected]>
Date: Fri, 1 Sep 2006 03:29:23 +0400

Quoted 3 lines
Quoted 1 line

2) a way to take delayed ACKs into account for cwnd growth

This part is OK now, right?

This part of ABC is not on by default, and was broken until last week :-)

Test in tcp_slow_start() used to be:

	tp->bytes_acked > 2*tp->mss_cache

but now it is the correct:

	tp->bytes_acked >= 2*tp->mss_cache

It allows to make two congestion window increases from one ACK, when noticing delayed ACK.

Non-ABC code did not do this, but could figure this kind of thing out while scanning retransmit queue.

Quoted 6 lines
Quoted 1 line

1) protection against ACK division

But Linux never had this problem... Congestion window was increased only when a whole skb is ACKed, flag FLAG_DATA_ACKED. (TSO could break this, but should not). Otherwise, this ACK just advanced snd_una and nothing more.

Ugh, I missed this. :-/

The TSO code is carefuly to only trim TSO skbs on proper boundaries, and this ensures proper setting of FLAG_DATA_ACKED too. So no problems here.

Quoted 3 lines

The only problem of kind "too-aggressive" with Linux was that we could develop large cwnd sending small segments, and then switch to sending mss-sized segments. It does not look scary, to be honest. :-)

Agreed.

Quoted 3 lines

Linux had troubles with slow start even before ABC. Actually, some of applications can suffer of the same syndrome even if ABC disabled. With ABC it becomes TROUBLE, cwnd has no chances to develop at all.

I've discussed that very issue here before, some time ago, with John Heffner. It was in response to a user reporting a similar problem. The problem is really at the receiver because we only ACK every other full sized frame. I had the idea to ACK every 2 frames, regardless of size, but that might have other problems.

There is an assymetry between how we do congestion control on sending (packet counting) and our ACK policy on receive (packet sized based).

Quoted 2 lines

Probably, aspect 1 of ABC just should be disabled. And the first my suggestion looks working too.

I'm ready to rip out ABC entirely, to be honest. Or at least turn it off by default.

#15 lore raw
Quoted 57 lines · On Thu, 31 Aug 2006 16:57:01 -0700 (PDT) David Miller <[email protected]>…

From: Alexey Kuznetsov <[email protected]>
Date: Fri, 1 Sep 2006 03:29:23 +0400

Quoted 3 lines
Quoted 1 line

2) a way to take delayed ACKs into account for cwnd growth

This part is OK now, right?

This part of ABC is not on by default, and was broken until last week :-)

Test in tcp_slow_start() used to be:

	tp->bytes_acked > 2*tp->mss_cache

but now it is the correct:

	tp->bytes_acked >= 2*tp->mss_cache

It allows to make two congestion window increases from one ACK, when noticing delayed ACK.

Non-ABC code did not do this, but could figure this kind of thing out while scanning retransmit queue.

Quoted 6 lines
Quoted 1 line

1) protection against ACK division

But Linux never had this problem... Congestion window was increased only when a whole skb is ACKed, flag FLAG_DATA_ACKED. (TSO could break this, but should not). Otherwise, this ACK just advanced snd_una and nothing more.

Ugh, I missed this. :-/

The TSO code is carefuly to only trim TSO skbs on proper boundaries, and this ensures proper setting of FLAG_DATA_ACKED too. So no problems here.

Quoted 3 lines

The only problem of kind "too-aggressive" with Linux was that we could develop large cwnd sending small segments, and then switch to sending mss-sized segments. It does not look scary, to be honest. :-)

Agreed.

Quoted 3 lines

Linux had troubles with slow start even before ABC. Actually, some of applications can suffer of the same syndrome even if ABC disabled. With ABC it becomes TROUBLE, cwnd has no chances to develop at all.

I've discussed that very issue here before, some time ago, with John Heffner. It was in response to a user reporting a similar problem. The problem is really at the receiver because we only ACK every other full sized frame. I had the idea to ACK every 2 frames, regardless of size, but that might have other problems.

There is an assymetry between how we do congestion control on sending (packet counting) and our ACK policy on receive (packet sized based).

Quoted 5 lines
Quoted 2 lines

Probably, aspect 1 of ABC just should be disabled. And the first my suggestion looks working too.

I'm ready to rip out ABC entirely, to be honest. Or at least turn it off by default.

Turn it off for 2.6.18, by default then evaluate more for 2.6.19

#16 lore raw
Quoted 5 lines
Quoted 2 lines

I'm ready to rip out ABC entirely, to be honest. Or at least turn it off by default.

Turn it off for 2.6.18, by default then evaluate more for 2.6.19

If it goes out in 2.6.18 there could probably be a good argument for going into the stable tree as well... to stop the likes of the JVM type issues that users keep hitting (which is fixed or going to be fixed by Sun).

Signature · 7 lines
Ian McDonald
Web: http://wand.net.nz/~iam4
Blog: http://imcdnzl.blogspot.com
WAND Network Research Group
Department of Computer Science
University of Waikato
New Zealand
maintainer
#17 lore raw

From: "Ian McDonald" <[email protected]>
Date: Fri, 1 Sep 2006 15:39:37 +1200

Quoted 9 lines
Quoted 5 lines
Quoted 2 lines

I'm ready to rip out ABC entirely, to be honest. Or at least turn it off by default.

Turn it off for 2.6.18, by default then evaluate more for 2.6.19

If it goes out in 2.6.18 there could probably be a good argument for going into the stable tree as well... to stop the likes of the JVM type issues that users keep hitting (which is fixed or going to be fixed by Sun).

I completely agree.

#18 lore raw
In reply to David Miller

On Thu, 31 Aug 2006, David Miller wrote:
...

Quoted 5 lines
Quoted 2 lines

Probably, aspect 1 of ABC just should be disabled. And the first my suggestion looks working too.

I'm ready to rip out ABC entirely, to be honest. Or at least turn it off by default.

Just as a curious observer: do you think these issues are due to ABC implementation, or due to ABC specification?

I.e., should the ABC specification be 1) revised to fix specification bugs/omissions, 2) revised to document assumptions better, 3) deprecated due to FOO, 4) recommended for use, or 5) something else?

Signature · 3 lines
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings
maintainer
#19 lore raw

From: Pekka Savola <[email protected]>
Date: Fri, 1 Sep 2006 12:44:48 +0300 (EEST)

Quoted 10 lines · David Miller

...

Quoted 5 lines
Quoted 2 lines

Probably, aspect 1 of ABC just should be disabled. And the first my suggestion looks working too.

I'm ready to rip out ABC entirely, to be honest. Or at least turn it off by default.

Just as a curious observer: do you think these issues are due to ABC implementation, or due to ABC specification?

It simply doesn't apply to us, as Alexey explained, because we prevent ACK divison already when we apply the ACK to the retransmit queue purging loop. If we didn't free any whole packets, we don't advance the congestion window.

The other bit, dealing with delayed ACKs, we could handle another way.

ABC is a very BSD specific algorithm, as Alexey also mentioned.

#20 lore raw
In reply to David Miller

Hello!

Quoted 3 lines

problem. The problem is really at the receiver because we only ACK every other full sized frame. I had the idea to ACK every 2 frames, regardless of size,

This would solve lots of problems.

Quoted 1 line

but that might have other problems.

BSD used to do this, everyone is happy.

It could disclose some new problems in our stack, because we have quite a lot of code to mitigate problems caused by ACK each 2*MSS. But they should not be essential. Actually, we could have it adjustable:
normally each 2 frames, but apply efforts to coarsen ACKs up to 2*MSS, when lots of small frames are ACKed back to back.

Quoted 3 lines

There is an assymetry between how we do congestion control on sending (packet counting) and our ACK policy on receive (packet sized based).

It is curious fact: BSD has exactly opposite assymetry.
ACK are sent each second packet, but congestion avoidance is made by size. :-)

Quoted 1 line

turn it off by default.

Yes.

Alexey

#21 lore raw
Quoted 7 lines · Alexey Kuznetsov

Hello!

Quoted 3 lines

problem. The problem is really at the receiver because we only ACK every other full sized frame. I had the idea to ACK every 2 frames, regardless of size,

This would solve lots of problems.

At least for slow start it is safe, but experiments with atcp for netchannels showed that it is better not to send excessive number of acks when slow start is over, instead we can introduce some tricky ack avoidance scheme and ack at least 2-3-4 packets or full MSS instead of two mss-sized frames.

Quoted 1 line

Alexey

Signature · 1 line
	Evgeniy Polyakov

A message that is not in the archive (sent off-list or not yet received): [email protected]

#23 lore raw

PATCH 1/1 tcp: turn ABC off

Turn Appropriate Byte Count off by default because it unfairly penalizes applications that do small writes.
Add better documentation to describe what it is so users will understand why they might want to turn it on.

---

 Documentation/networking/ip-sysctl.txt |   12 +++++++++---
 net/ipv4/tcp_input.c                   |    2 +-
 2 files changed, 10 insertions(+), 4 deletions(-)

3988eb0b87288c0d30c165742af7fc7a733100ab

Documentation/networking/ip-sysctl.txt +9 -3
index 3e0c017..90ed781 100644
@@ -102,9 +102,15 @@ inet_peer_gc_maxtime - INTEGER
 TCP variables: 
 
 tcp_abc - INTEGER
-	Controls Appropriate Byte Count defined in RFC3465. If set to
-	0 then does congestion avoid once per ack. 1 is conservative
-	value, and 2 is more agressive.
+	Controls Appropriate Byte Count (ABC) defined in RFC3465.
+	ABC is a way of increasing congestion window (cwnd) more slowly
+	in response to partial acknowledgments.
+	Possible values are:
+		0 increase cwnd once per acknowledgment (no ABC)
+		1 increase cwnd once per acknowledgment of full sized segment
+		2 allow increase cwnd by two if acknowledgment is
+		  of two segments to compensate for delayed acknowledgments.
+	Default: 0 (off)
 
 tcp_syn_retries - INTEGER
 	Number of times initial SYNs for an active TCP connection attempt
net/ipv4/tcp_input.c +1 -1
index 111ff39..159fa3f 100644
@@ -89,7 +89,7 @@ int sysctl_tcp_frto;
 int sysctl_tcp_nometrics_save;
 
 int sysctl_tcp_moderate_rcvbuf = 1;
-int sysctl_tcp_abc = 1;
+int sysctl_tcp_abc;
 
 #define FLAG_DATA		0x01 /* Incoming frame contained data.		*/
 #define FLAG_WIN_UPDATE		0x02 /* Incoming ACK was a window update.	*/
Signature · 5 lines
1.2.4


-- 
VGER BF report: U 0.5
#24 lore raw

Re: [PATCH] tcp: turn ABC off

net/ipv4/tcp_input.c int sysctl_tcp_frto; · line 92

Quoted 10 lines · Stephen Hemminger
net/ipv4/tcp_input.c +1 -1
index 111ff39..159fa3f 100644
@@ -89,7 +89,7 @@ int sysctl_tcp_frto;
 int sysctl_tcp_nometrics_save;
 
 int sysctl_tcp_moderate_rcvbuf = 1;
-int sysctl_tcp_abc = 1;
+int sysctl_tcp_abc;

Since it is not static are you sure it will be zero?

Signature · 4 lines
	Evgeniy Polyakov

-- 
VGER BF report: H 8.21099e-10
#25 lore raw

Re: [PATCH] tcp: turn ABC off

Quoted 8 lines · Evgeniy Polyakov
Quoted 7 lines · Stephen Hemminger
net/ipv4/tcp_input.c +0 -0
index 111ff39..159fa3f 100644
@@ -89,7 +89,7 @@ int sysctl_tcp_frto;
 int sysctl_tcp_nometrics_save;
 
Quoted 5 lines

net/ipv4/tcp_input.c int sysctl_tcp_frto; · line 92

Quoted 3 lines
net/ipv4/tcp_input.c +1 -1
 int sysctl_tcp_moderate_rcvbuf = 1;
-int sysctl_tcp_abc = 1;
+int sysctl_tcp_abc;

Since it is not static are you sure it will be zero?

Outside a function the static modifier merely modifies whether the symbol is visible externally. It does not control whether it gets zeroed. And yes this will get zeroed.

Cheers,

Signature · 7 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

-- 
VGER BF report: U 0.49998
#26 lore raw
In reply to Evgeniy Polyakov

Hello!

Quoted 3 lines

At least for slow start it is safe, but experiments with atcp for netchannels showed that it is better not to send excessive number of acks when slow start is over,

If this thing is done from tcp_cleanup_rbuf(), it should not affect performance too much.

Note, that with ABC and another pathological cases, which do not allow to send more than a fixed amount of segments [ we have lots of them, f.e. sending tiny segments, we can hit sndbuf limit ], we deal with case, when slow start is _never_ over.

Quoted 3 lines

instead we can introduce some tricky
ack avoidance scheme and ack at least 2-3-4 packets or full MSS instead of two mss-sized frames.

One smart scheme was used at some stage (2000, probably never merged in this form to mainstream): tcp counted amount of unacked small segments in ack.rcv_small and kept threshold in ack.rcv_thresh.

+
+ /* If we ever saw N>1 small segments from peer, it has

+        * enough of send buffer to send N packets and does not nagle.
+        * Hence, we may delay acks more aggresively.
+        */

+ if (tp->ack.rcv_small > tp->ack.rcv_thresh+1)
+ tp->ack.rcv_thresh = tp->ack.rcv_small-1;
+ tp->ack.rcv_small = 0;

That was too much of trouble for such simple thing. So, eventually it was replaced with much dumber scheme. Look at current tcp_cleanup_rbuf(). It forces ACK, each time when it sees, that some small segment was received. It survived for 6 years, so that I guess it did not hurt anybody. :-)

What I would suggest to do now, is to replace:

	(copied > 0 &&
	(icsk->icsk_ack.pending & ICSK_ACK_PUSHED) &&
	!icsk->icsk_ack.pingpong &&
	!atomic_read(&sk->sk_rmem_alloc)))
		time_to_ack = 1;

with:

	(copied > 0 &&
	(icsk->icsk_ack.unacked > 1 ||
	(icsk->icsk_ack.pending & ICSK_ACK_PUSHED) && !icsk->icsk_ack.pingpong) &&
	!atomic_read(&sk->sk_rmem_alloc)))
		time_to_ack = 1;

I would not hesitate even a minute, if variable "unacked" could be caluclated using some existing state variables.

Alexey

Signature · 1 line
VGER BF report: U 0.500017
#27 lore raw
In reply to Alexander Vodomerov

Hello!

Quoted 2 lines

Some people reported that this program runs in 9.997 sec when run on FreeBSD.

Try enclosed patch. I have no idea why 9.997 sec is so magic, but I get exactly this number on my notebook. :-)

Alexey

=================

This patch enables sending ACKs each 2d received segment.
It does not affect either mss-sized connections (obviously) or connections controlled by Nagle (because there is only one small segment in flight).

The idea is to record the fact that a small segment arrives on a connection, where one small segment has already been received and still not-ACKed. In this case ACK is forced after tcp_recvmsg() drains receive buffer.

In other words, it is a "soft" each-2d-segment ACK, which is enough to preserve ACK clock even when ABC is enabled.

include/net/inet_connection_sock.h +2 -1
index 9bf73fe..de4e83b 100644
@@ -147,7 +147,8 @@ extern struct sock *inet_csk_clone(struc
 enum inet_csk_ack_state_t {
 	ICSK_ACK_SCHED	= 1,
 	ICSK_ACK_TIMER  = 2,
-	ICSK_ACK_PUSHED = 4
+	ICSK_ACK_PUSHED = 4,
+	ICSK_ACK_PUSHED2 = 8
 };
 
 extern void inet_csk_init_xmit_timers(struct sock *sk,
net/ipv4/tcp.c +5 -2
index 934396b..4f3b76f 100644
@@ -955,8 +955,11 @@ #endif
 		     * receive buffer and there was a small segment
 		     * in queue.
 		     */
-		    (copied > 0 && (icsk->icsk_ack.pending & ICSK_ACK_PUSHED) &&
-		     !icsk->icsk_ack.pingpong && !atomic_read(&sk->sk_rmem_alloc)))
+		    (copied > 0 &&
+		     ((icsk->icsk_ack.pending & ICSK_ACK_PUSHED2) ||
+		      ((icsk->icsk_ack.pending & ICSK_ACK_PUSHED) &&
+		       !icsk->icsk_ack.pingpong) &&
+		      !atomic_read(&sk->sk_rmem_alloc)))
 			time_to_ack = 1;
 	}
 
net/ipv4/tcp_input.c +2 -0
index 111ff39..5877920 100644
@@ -156,6 +156,8 @@ static void tcp_measure_rcv_mss(struct s
 				return;
 			}
 		}
+		if (icsk->icsk_ack.pending & ICSK_ACK_PUSHED)
+			icsk->icsk_ack.pending |= ICSK_ACK_PUSHED2;
 		icsk->icsk_ack.pending |= ICSK_ACK_PUSHED;
 	}
 }
#28 lore raw
Quoted 18 lines · Alexey Kuznetsov

Hello!

Quoted 2 lines

Some people reported that this program runs in 9.997 sec when run on FreeBSD.

Try enclosed patch. I have no idea why 9.997 sec is so magic, but I get exactly this number on my notebook. :-)

Alexey

=================

This patch enables sending ACKs each 2d received segment.
It does not affect either mss-sized connections (obviously) or connections controlled by Nagle (because there is only one small segment in flight).

Quoted 7 lines

The idea is to record the fact that a small segment arrives on a connection, where one small segment has already been received and still not-ACKed. In this case ACK is forced after tcp_recvmsg() drains receive buffer.

In other words, it is a "soft" each-2d-segment ACK, which is enough to preserve ACK clock even when ABC is enabled.

Is this really necessary? I thought that the problems with ABC were in trying to apply byte-based heuristics from the RFC(s) to a packet-oritented cwnd in the stack?

rick jones

#29 lore raw

Hello!

Quoted 1 line

Is this really necessary?

No, of course. We lived for ages without this, would live for another age.

Quoted 3 lines

I thought that the problems with ABC were in
trying to apply byte-based heuristics from the RFC(s) to a packet-oritented cwnd in the stack?

It was just the last drop.

Even with disabled ABC, that test shows some gaps in latency summed up to ~300 msec. Almost invisible, but not good.

Too aggressive delack has many other issues. Even without ABC we have quadratically suppressed cwnd on TCP_NODELAY connections comparing to BSD: at sender side we suppress it by counting cwnd in packets, at receiver side by ACKing by byte counter.

Each time when another victim sees artificial latencies introduced by agressive delayed acks, even though he requested TCP_NODELAY, our best argument is "Stupid, you do all wrong, how could you get a decent performance?" :-).

Probably, we stand for a feature which really does not worth to stand for and causes nothing but permanent pain in ass.

Alexey

maintainer
#30 lore raw
In reply to Rick Jones

From: Rick Jones <[email protected]>
Date: Tue, 05 Sep 2006 10:55:16 -0700

Quoted 3 lines

Is this really necessary? I thought that the problems with ABC were in trying to apply byte-based heuristics from the RFC(s) to a packet-oritented cwnd in the stack?

This is receiver side, and helps a sender who does congestion
control based upon packet counting like Linux does. It really
is less related to ABC than Alexey implies, we've always had this kind of problem as I mentioned in previous talks in the past on this issue.

#31 lore raw
Quoted 4 lines · David Miller

From: Rick Jones <[email protected]>
Date: Tue, 05 Sep 2006 10:55:16 -0700

Quoted 8 lines
Quoted 1 line

Is this really necessary? I thought that the problems with ABC were in trying to apply byte-based heuristics from the RFC(s) to a packet-oritented cwnd in the stack?

This is receiver side, and helps a sender who does congestion
control based upon packet counting like Linux does. It really
is less related to ABC than Alexey implies, we've always had this kind of problem as I mentioned in previous talks in the past on this issue.

For a connection receiving nothing but sub-MSS segments this is going to non-trivially increase the number of ACKs sent no? I would expect an unpleasant increase in service demands on something like a "burst enabled" (./configure --enable-burst) netperf TCP_RR test:

netperf -t TCP_RR -H foo -- -b N # N > 1

to increase as a result. Pipelined HTTP would be like that, some NFS over TCP stuff too, maybe X traffic, other "transactional" workloads as well - maybe Tuxeudo.

rick jones

#32 lore raw

Hello!

Of course, number of ACK increases. It is the goal. :-)

Quoted 4 lines

unpleasant increase in service demands on something like a "burst enabled" (./configure --enable-burst) netperf TCP_RR test:

netperf -t TCP_RR -H foo -- -b N # N > 1

foo=localhost

b patched orig
2 105874.83 105143.71
3 114208.53 114023.07
4 120493.99 120851.27
5 128087.48 128573.33
10 151328.48 151056.00

Probably, the test is done wrong. But I see no difference.

Quoted 2 lines

to increase as a result. Pipelined HTTP would be like that, some NFS
over TCP stuff too, maybe X traffic,

X will be excited about better latency.

What's about protocols not interested in latency, they will be a little happier, if transactions are processed asynchronously.

But actually, it is not about increasing/decreasing number of ACKs. It is about killing that pain in ass which we used to have because we pretended to be too smart.

Alexey

#33 lore raw
Quoted 9 lines · Alexey Kuznetsov

Hello!

Of course, number of ACK increases. It is the goal. :-)

Quoted 3 lines

unpleasant increase in service demands on something like a "burst enabled" (./configure --enable-burst) netperf TCP_RR test:

netperf -t TCP_RR -H foo -- -b N # N > 1

foo=localhost

There isn't any sort of clever short-circuiting in loopback is there? I do like the convenience of testing things over loopback, but always fret about not including drivers and actual hardware interrupts etc.

Quoted 8 lines

b patched orig
2 105874.83 105143.71
3 114208.53 114023.07
4 120493.99 120851.27
5 128087.48 128573.33
10 151328.48 151056.00

Probably, the test is done wrong. But I see no difference.

Regardless, kudos for running the test. The only thing missing is the -c and -C options to enable the CPU utilization measurements which will then give the service demand on a CPU time per transaction basis. Or was this a UP system that was taken to CPU saturation?

Quoted 7 lines
Quoted 1 line

to increase as a result. Pipelined HTTP would be like that, some NFS over TCP stuff too, maybe X traffic,

X will be excited about better latency.

What's about protocols not interested in latency, they will be a little happier, if transactions are processed asynchronously.

What i'm thinking about isn't so much about the latency as it is the aggregate throughput a system can do with lots of these protocols/connections going at the same time. Hence the concern about increases in service demand.

Quoted 3 lines

But actually, it is not about increasing/decreasing number of ACKs. It is about killing that pain in ass which we used to have because we pretended to be too smart.

:)

rick jones

#34 lore raw

Hello!

Quoted 1 line

There isn't any sort of clever short-circuiting in loopback is there?

No, from all that I know.

Quoted 3 lines

I
do like the convenience of testing things over loopback, but always fret about not including drivers and actual hardware interrupts etc.

Well, if the test is right, it should show cost of redundant ACKs.

Quoted 4 lines

Regardless, kudos for running the test. The only thing missing is the -c and -C options to enable the CPU utilization measurements which will then give the service demand on a CPU time per transaction basis. Or was this a UP system that was taken to CPU saturation?

It is my notebook. :-) Of course, cpu consumption is 100%. (Actally, netperf shows 100.10 :-))

I will redo test on a real network. What range of -b should I test?

Quoted 1 line

What i'm thinking about isn't so much about the latency

I understand.

Actually, I did those tests ages ago for a pure throughput case, when nothing goes in the opposite direction. I did not find a difference that time. And nobody even noticed that Linux sends ACKs _each_ small segment for unidirectional connections for all those years. :-)

Alexey

#35 lore raw
Quoted 7 lines
Quoted 3 lines

Regardless, kudos for running the test. The only thing missing is the
-c and -C options to enable the CPU utilization measurements which will
then give the service demand on a CPU time per transaction basis. Or was this a UP system that was taken to CPU saturation?

It is my notebook. :-) Of course, cpu consumption is 100%. (Actally, netperf shows 100.10 :-))

Gotta love the accuracy. :)

Quoted 3 lines

I will redo test on a real network. What range of -b should I test?

I suppose that depends on your patience :) In theory, as you increase (eg double) the -b setting you should reach a point of diminishing returns wrt transaction rate. If you see that, and see the service demand flattening-out I'd say it is probably time to stop.

I'm also not quite sure if "abc" needs to be disabled or not.

I do know that I left-out one very important netperf option. The command line should be:

netperf -t TCP_RR -H foo -- -b N -D

where "-D" is added to set TCP_NODELAY. Otherwise, the ratio of transactions to data segments is fubar. That issue is also why I wonder about the setting of tcp_abc.

[I have this quixotic pipedream about being able to --enable-burst, set -D and say that the number of TCP segments exchanged on the network is 2X the transaction count when request and response size are < MSS. The raison d'etre for this pipe dream is maximizing PPS with TCP_RR tests without _having_ to have hundreds if not thousands of simultaneous netperfs/connections - say with just as many netperfs/connections as there are CPUs or threads/strands in the system. It was while trying to make this pipe dream a reality I first noticed that HP-UX 11i, which normally has a very nice ACK avoidance heuristic, would send an immediate ACK if it received back-to-back sub-MSS segments - thus ruining my pipe dream when it came to HP-UX testing. Hapily, I noticed that "linux" didn't seem to be doing the same thing. Hence my tweaking when seeing this patch come along...]

Quoted 9 lines
Quoted 1 line

What i'm thinking about isn't so much about the latency

I understand.

Actually, I did those tests ages ago for a pure throughput case, when nothing goes in the opposite direction. I did not find a difference that time. And nobody even noticed that Linux sends ACKs _each_ small segment for unidirectional connections for all those years. :-)

Not everyone looks very closely (alas, sometimes myself included).

If all anyone does is look at throughput, until they CPU saturate they wouldn't notice. Heck, before netperf and TCP_RR tests, and sadly even still today, most people just look at how fast a single-connection, unidirectional data transfer goes and leave it at that :(

Thankfully, the set of "most people" and "netdev" aren't completely overlapping.

rick jones

#36 lore raw

Hello!

Quoted 2 lines

transactions to data segments is fubar. That issue is also why I wonder about the setting of tcp_abc.

Yes, switching ABC on/off has visible impact on amount of segments. When ABC is off, amount of segments is almost the same as number of transactions. When it is on, ~1.5% are merged. But this is invisible in numbers of throughput/cpu usage.

That' numbers:

1Gig link. The first column is "b". - separates runs of netperf in backward direction.

Run #1. One host is slower.

        old,abc=0
	 new,abc=0
	  new,abc=1
	   old,abc=1
2	23652.00  6.31   21.11  10.665  8.924
	 23622.16  6.47   21.01  10.951  8.893
	  23625.05  6.21   21.01  10.512  8.891
	   23725.12  6.46   20.31  10.898  8.559
	-
	23594.87  21.90  6.44   9.283   10.912
	 23631.52  20.30  6.36   8.592   10.766
	  23609.55  21.00  6.26   8.896   10.599
	   23633.75  21.10  5.44   8.929   9.206
4	36349.11  8.71   31.21  9.584   8.585
	 36461.37  8.65   30.81  9.492   8.449
	  36723.72  8.22   31.31  8.949   8.526
	   35801.24  8.58   30.51  9.589   8.521
	-
	35127.34  33.80  8.43   9.621   9.605
	 36165.50  30.90  8.48   8.545   9.381
	  36201.45  31.10  8.31   8.592   9.185
	   35269.76  30.00  8.58   8.507   9.732
8	41148.23  10.39  42.30  10.101  10.281
	 41270.06  11.04  31.31  10.698  7.585
	  41181.56  5.66   48.61  5.496   11.803
	   40372.37  9.68   56.50  9.591   13.996
	-
	40392.14  47.00  11.89  11.637  11.775
	 40613.80  36.90  9.16   9.086   9.019
	  40504.66  53.60  7.73   13.234  7.639
	   40388.99  48.70  11.93  12.058  11.814

16 67952.27 16.27 43.70 9.576 6.432

	 68031.40  10.56  53.70  6.206   7.894
	  67777.95  12.81  46.90  7.559   6.920
	   67814.41  16.13  46.50  9.517   6.857
	-
	68031.46  51.30  11.53  7.541   6.781
	 68044.57  40.70  8.48   5.982   4.986
	  67808.13  39.60  15.86  5.840   9.355
	   67818.32  52.90  11.51  7.801   6.791

32 90445.09 15.41 99.90 6.817 11.045

	 90210.34  16.11  100.00 7.143   11.085
	  90221.84  17.31  98.90  7.676   10.962
	   90712.78  18.41  99.40  8.120   10.958
	-
	89155.51  99.90  12.89  11.205  5.782
	 90058.54  99.90  16.16  11.093  7.179
	  90092.31  98.60  15.41  10.944  6.840
	   88688.96  99.00  17.59  11.163  7.933

64 89983.76 13.66 100.00 6.071 11.113

	 90504.24  17.54  100.00 7.750   11.049
	  92043.36  17.44  99.70  7.580   10.832
	   90979.29  16.01  99.90  7.038   10.981
	-
	88615.27  99.90  14.91  11.273  6.729
	 89316.13  99.90  17.28  11.185  7.740
	  90622.85  99.90  16.81  11.024  7.420
	   89084.85  99.90  17.51  11.214  7.861

Run #2. Slower host is replaced with better one. ABC=0. No runs in backward directions.

new
old

2	24009.73  8.80   6.49   3.667   10.806
	 24008.43  8.00   6.32   3.334   10.524
4	40012.53  18.30  8.79   4.574   8.783
	 39999.84  19.40  8.86   4.851   8.857
8	60500.29  26.30  12.78  4.348   8.452
	 60397.79  26.30  11.73  4.355   7.769
16	69619.95  39.80  14.03  5.717   8.063
	 70528.72  24.90  14.43  3.531   8.184
32	132522.01  53.20  21.28  4.015   6.424
	 132602.93  57.70  22.59  4.351   6.813
64	145738.83  60.30  25.01  4.138   6.865
	 143129.55  73.20  24.19  5.114   6.759
128	148184.21  69.70  24.96  4.704   6.739
	 148143.47  71.00  25.01  4.793   6.753
256	144798.91  69.40  25.01  4.793   6.908
	 144086.01  73.00  24.61  5.067   6.832

Frankly, I do not see any statistically valid correlations.

Quoted 2 lines

that "linux" didn't seem to be doing the same thing. Hence my tweaking when seeing this patch come along...]

netperf does not catch this. :-)

Even with this patch linux does not ack each second segment dumbly, it waits for some conditions, mostly read() emptying receive queue. To model this it is necessary to insert some gaps between bursted segments or to use slow network.

I have no doubts it is easy to model a situation when we send lots of useless ACKs. F.e. inserting 20ms gaps between requests. To see effect on thoughput/cpu, we could start enough of connections, doing the same thing.

Alexey

#37 lore raw
Quoted 10 lines · Alexey Kuznetsov

Hello!

Quoted 1 line

transactions to data segments is fubar. That issue is also why I wonder about the setting of tcp_abc.

Yes, switching ABC on/off has visible impact on amount of segments. When ABC is off, amount of segments is almost the same as number of transactions. When it is on, ~1.5% are merged. But this is invisible in numbers of throughput/cpu usage.

Hmm, that would seem to suggest that for "new" the netperf/netserver were being fast enough that the code didn't perceive the receipt of back-to-back sub-MSS segments? (Is that even possible once -b is fairly large?) Otherwise, with new I would have expected the segment count to be meaningfully > than the transaction count?

Quoted 96 lines

That' numbers:

1Gig link. The first column is "b". - separates runs of netperf in backward direction.

Run #1. One host is slower.

        old,abc=0
	 new,abc=0
	  new,abc=1
	   old,abc=1
2	23652.00  6.31   21.11  10.665  8.924
	 23622.16  6.47   21.01  10.951  8.893
	  23625.05  6.21   21.01  10.512  8.891
	   23725.12  6.46   20.31  10.898  8.559
	-
	23594.87  21.90  6.44   9.283   10.912
	 23631.52  20.30  6.36   8.592   10.766
	  23609.55  21.00  6.26   8.896   10.599
	   23633.75  21.10  5.44   8.929   9.206
4	36349.11  8.71   31.21  9.584   8.585
	 36461.37  8.65   30.81  9.492   8.449
	  36723.72  8.22   31.31  8.949   8.526
	   35801.24  8.58   30.51  9.589   8.521
	-
	35127.34  33.80  8.43   9.621   9.605
	 36165.50  30.90  8.48   8.545   9.381
	  36201.45  31.10  8.31   8.592   9.185
	   35269.76  30.00  8.58   8.507   9.732
8	41148.23  10.39  42.30  10.101  10.281
	 41270.06  11.04  31.31  10.698  7.585
	  41181.56  5.66   48.61  5.496   11.803
	   40372.37  9.68   56.50  9.591   13.996
	-
	40392.14  47.00  11.89  11.637  11.775
	 40613.80  36.90  9.16   9.086   9.019
	  40504.66  53.60  7.73   13.234  7.639
	   40388.99  48.70  11.93  12.058  11.814

16 67952.27 16.27 43.70 9.576 6.432

	 68031.40  10.56  53.70  6.206   7.894
	  67777.95  12.81  46.90  7.559   6.920
	   67814.41  16.13  46.50  9.517   6.857
	-
	68031.46  51.30  11.53  7.541   6.781
	 68044.57  40.70  8.48   5.982   4.986
	  67808.13  39.60  15.86  5.840   9.355
	   67818.32  52.90  11.51  7.801   6.791

32 90445.09 15.41 99.90 6.817 11.045

	 90210.34  16.11  100.00 7.143   11.085
	  90221.84  17.31  98.90  7.676   10.962
	   90712.78  18.41  99.40  8.120   10.958
	-
	89155.51  99.90  12.89  11.205  5.782
	 90058.54  99.90  16.16  11.093  7.179
	  90092.31  98.60  15.41  10.944  6.840
	   88688.96  99.00  17.59  11.163  7.933

64 89983.76 13.66 100.00 6.071 11.113

	 90504.24  17.54  100.00 7.750   11.049
	  92043.36  17.44  99.70  7.580   10.832
	   90979.29  16.01  99.90  7.038   10.981
	-
	88615.27  99.90  14.91  11.273  6.729
	 89316.13  99.90  17.28  11.185  7.740
	  90622.85  99.90  16.81  11.024  7.420
	   89084.85  99.90  17.51  11.214  7.861

Run #2. Slower host is replaced with better one. ABC=0. No runs in backward directions.

new
old

2	24009.73  8.80   6.49   3.667   10.806
	 24008.43  8.00   6.32   3.334   10.524
4	40012.53  18.30  8.79   4.574   8.783
	 39999.84  19.40  8.86   4.851   8.857
8	60500.29  26.30  12.78  4.348   8.452
	 60397.79  26.30  11.73  4.355   7.769
16	69619.95  39.80  14.03  5.717   8.063
	 70528.72  24.90  14.43  3.531   8.184
32	132522.01  53.20  21.28  4.015   6.424
	 132602.93  57.70  22.59  4.351   6.813
64	145738.83  60.30  25.01  4.138   6.865
	 143129.55  73.20  24.19  5.114   6.759
128	148184.21  69.70  24.96  4.704   6.739
	 148143.47  71.00  25.01  4.793   6.753
256	144798.91  69.40  25.01  4.793   6.908
	 144086.01  73.00  24.61  5.067   6.832
Quoted 1 line

Frankly, I do not see any statistically valid correlations.

Does look like it jumps-around quite a bit - for example the run#2 with -b 16 had the CPU util all over the map on the netperf side. That wasn't by any chance an SMP system?

Quoted 4 lines
Quoted 1 line

that "linux" didn't seem to be doing the same thing. Hence my tweaking when seeing this patch come along...]

netperf does not catch this. :-)

Nope :( One of these days.... I need to teach netperf how to extract TCP statistics from as many platforms as possible. Meantime it relies as always on the kindness of benchmarkers :) (My appologies to Tennesee Williams :)

Quoted 2 lines

Even with this patch linux does not ack each second segment dumbly, it waits for some conditions, mostly read() emptying receive queue.

Good. HP-UX is indeed dumb about this, but I'm assured it will be changing. I forget what Solaris does in this situation - I thought I looked a while ago but cannot recall the result.

Quoted 7 lines

To model this it is necessary to insert some gaps between bursted segments or to use slow network.

I have no doubts it is easy to model a situation when we send lots of useless ACKs. F.e. inserting 20ms gaps between requests. To see effect on thoughput/cpu, we could start enough of connections, doing the same thing.

Adding --enable-intervals might work there. I don't recall how well it gets along with --enable-burst though, and you have already made lots of runs as it is.

rick

maintainer
#38 lore raw
In reply to Alexey Kuznetsov

From: Alexey Kuznetsov <[email protected]>
Date: Mon, 4 Sep 2006 20:00:45 +0400

Quoted 14 lines

Try enclosed patch. I have no idea why 9.997 sec is so magic, but I get exactly this number on my notebook. :-)

=================

This patch enables sending ACKs each 2d received segment.
It does not affect either mss-sized connections (obviously) or connections controlled by Nagle (because there is only one small segment in flight).

The idea is to record the fact that a small segment arrives on a connection, where one small segment has already been received and still not-ACKed. In this case ACK is forced after tcp_recvmsg() drains receive buffer.

Quoted 4 lines

In other words, it is a "soft" each-2d-segment ACK, which is enough to preserve ACK clock even when ABC is enabled.

This looks exactly like the kind of patch I tried to formulate, very unsuccessfully, last time this topic came up a year or so ago.

It looks perfectly fine to me, would you like me to apply it Alexey?

#39 lore raw

Hello!

Quoted 2 lines

It looks perfectly fine to me, would you like me to apply it Alexey?

Yes, I think it is safe.

Theoretically, there is one place where it can be not so good. Good nagling tcp connection, which makes lots of small write()s, will send MSS sized frames due to delayed ACKs. But if we ACK each other segment, more segments will come out incomplete, which could result in some decrease of throughput.

But the trap for this case was set 6 years ago. For unidirectional sessions ACKs were sent not even each second segment, but each small segment. :-) This did not show any problems for those 6 years. I guess it means that the problem does not exist.

Alexey

maintainer
#40 lore raw

From: Alexey Kuznetsov <[email protected]>
Date: Mon, 18 Sep 2006 14:37:05 +0400

Quoted 4 lines
Quoted 2 lines

It looks perfectly fine to me, would you like me to apply it Alexey?

Yes, I think it is safe.

Ok, I'll put this into net-2.6.19 for now. Thanks.

#41 lore raw
Quoted 9 lines · On Mon, 18 Sep 2006 06:56:55 -0700 (PDT) David Miller <[email protected]>…

From: Alexey Kuznetsov <[email protected]>
Date: Mon, 18 Sep 2006 14:37:05 +0400

Quoted 4 lines
Quoted 2 lines

It looks perfectly fine to me, would you like me to apply it Alexey?

Yes, I think it is safe.

Ok, I'll put this into net-2.6.19 for now. Thanks.

Did you try this on a desktop system? Something is wrong with net-2.6.19 basic web browsing seems slower.

Signature · 1 line
Stephen Hemminger <[email protected]>
maintainer
#42 lore raw

From: Stephen Hemminger <[email protected]>
Date: Wed, 20 Sep 2006 15:44:06 -0700

Quoted 7 lines
Quoted 1 line · On Mon, 18 Sep 2006 06:56:55 -0700 (PDT) David Miller <[email protected]>…

Ok, I'll put this into net-2.6.19 for now. Thanks.

Did you try this on a desktop system? Something is wrong with net-2.6.19 basic web browsing seems slower.

It might be due to other changes, please verify that it's truly caused by Alexey's change by backing it out and retesting.

Note that I had to use an updated version of Alexey's change, which he sent me privately, because the first version didn't compile :)

#43 lore raw
Quoted 11 lines · On Wed, 20 Sep 2006 15:47:56 -0700 (PDT) David Miller <[email protected]>…

From: Stephen Hemminger <[email protected]>
Date: Wed, 20 Sep 2006 15:44:06 -0700

Quoted 7 lines
Quoted 1 line · On Mon, 18 Sep 2006 06:56:55 -0700 (PDT) David Miller <[email protected]>…

Ok, I'll put this into net-2.6.19 for now. Thanks.

Did you try this on a desktop system? Something is wrong with net-2.6.19 basic web browsing seems slower.

Quoted 7 lines

It might be due to other changes, please verify that it's truly caused by Alexey's change by backing it out and retesting.

Note that I had to use an updated version of Alexey's change, which he sent me privately, because the first version didn't compile :)

It might be something else.. there are a lot of changes from 2.6.18 to net-2.6.19.

Signature · 1 line
Stephen Hemminger <[email protected]>