Skip to contentSkip to conversation

problems achieving decent throughput with latency.

DISCUSSION Ben Greear 6 participants 14 messages Feb 2, 2003 – Feb 4, 2003

lore
Share & tools
lore

Ben Greear

author
#0 lore raw

I am testing my latency-insertion tool, and I notice that tcp will not use all of the available bandwidth if there is any significant amount of latency on the wire.

For example, with 25ms latency in both directions, I see about 8Mbps bi-directional throughput.

If I lower that to 15ms, I see 12Mbps bi-directional throughput.

I see 27Mbps at 5ms.

Here is the /proc/net/tcp output at 5ms latency.

machine demo2
13: 050302AC:80EB 070302AC:80EB 01 0005900C:0002012E 01:00000016 00000000 0 0 578943 3 c6628a80 22 4 1 45 -1

machine demo1
11: 070302AC:80EB 050302AC:80EB 01 00010DDB:00000000 01:00000014 00000000 0 0 513094 3 c62c5080 21 4 1 45 -1

Any ideas why it is so slow at the higher latencies? Any other info I can gather to help determine the cause?

(UDP does not experience this slowdown, so I believe my latency insertion tool is working as designed, but it's always possible it is to blame...)

Signature · 3 lines
Ben Greear <[email protected]>       <Ben_Greear AT excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear

<div class="moz-text-flowed" style="font-family: -moz-fixed">I am testing my latency-insertion tool, and I notice that tcp will not use all of the available bandwidth if there is any significant amount of latency
on the wire.

For example, with 25ms latency in both directions, I see about 8Mbps bi-directional throughput.

If I lower that to 15ms, I see 12Mbps bi-directional throughput.

I see 27Mbps at 5ms.

Here is the /proc/net/tcp output at 5ms latency.

machine demo2
13: 050302AC:80EB 070302AC:80EB 01 0005900C:0002012E 01:00000016 00000000 0 0 578943 3 c6628a80 22 4 1 45 -1

machine demo1
11: 070302AC:80EB 050302AC:80EB 01 00010DDB:00000000 01:00000014 00000000 0 0 513094 3 c62c5080 21 4 1 45 -1

Any ideas why it is so slow at the higher latencies? Any other info I can gather to help determine the cause?

(UDP does not experience this slowdown, so I believe my latency insertion tool is working as designed, but it's always possible it is to blame...)

Signature · 6 lines
Ben Greear <[email protected]>       <Ben_Greear AT excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear


</div>
#1 lore raw
Quoted 6 lines · Ben Greear

I am testing my latency-insertion tool, and I notice that tcp will not use all of the available bandwidth if there is any significant amount of latency on the wire.

For example, with 25ms latency in both directions, I see about 8Mbps bi-directional throughput.

Check if large windows are being used, if window scaling is enabled.

Regards,

bert

Signature · 3 lines
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO
http://netherlabs.nl                         Consulting
#2 lore raw

TCP can only send into a pipe as fast as it can see the ACKs coming back. That is how TCP clocks its sending rate, and latency thus affects that.

#3 lore raw
Quoted 3 lines · David S. Miller

TCP can only send into a pipe as fast as it can see the ACKs coming back. That is how TCP clocks its sending rate, and latency thus affects that.

Wouldn't you just need larger windows? The problem is latency, not bandwidth.

Chris

Signature · 4 lines
Chris Friesen                    | MailStop: 043/33/F10
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: [email protected]
#4 lore raw
Quoted 6 lines
Quoted 3 lines

TCP can only send into a pipe as fast as it can see the ACKs coming back. That is how TCP clocks its sending rate, and latency thus affects that.

Wouldn't you just need larger windows? The problem is latency, not bandwidth.

Exactly - the original post says that no problems are experienced using UDP, which backs that up.

John.

#5 lore raw
Quoted 9 lines · John Bradford
Quoted 6 lines
Quoted 3 lines

TCP can only send into a pipe as fast as it can see the ACKs coming back. That is how TCP clocks its sending rate, and latency thus affects that.

Wouldn't you just need larger windows? The problem is latency, not bandwidth.

Exactly - the original post says that no problems are experienced using UDP, which backs that up.

My TCP may be getting hazy but with latency, TCP may limit the amount of packets 'in flight'. I think this is called the 'cwnd', congestion window. I also seem to recall that this window is limited to the actually negotiated TCP window size.

But here I leave this discussion as I've swapped out the finer details of congestion windows to the Stevens books which other's have probably read more recently than I did.

Regards,

bert

Signature · 3 lines
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO
http://netherlabs.nl                         Consulting
author
#6 lore raw
In reply to John Bradford
Quoted 9 lines · John Bradford
Quoted 5 lines
Quoted 3 lines

TCP can only send into a pipe as fast as it can see the ACKs coming back. That is how TCP clocks its sending rate, and latency thus affects that.

Wouldn't you just need larger windows? The problem is latency, not bandwidth.

Exactly - the original post says that no problems are experienced using UDP, which backs that up.

I started poking around, and found the tcp_mem, tcp_rmem, and tcp_wmem tunables in /proc/sys/net/ipv4...

If I change the values, I see up to 25Mbps with 25ms of latency. It would go higher, but I have uncovered a performance bug in my code that drops a packet every now and then at those higher rates, so that backs tcp off quickly. I should have that fixed this evening and will continue testing.

Here are the values that I used. The documentation I found is not overly descriptive, so if anyone has any suggestions for improving my tunings, please let me know!

Also, if it's as simple as allocating a few more buffers for tcp, maybe we should consider defaulting to higher in the normal kernel? (I'm not suggesting **my** numbers..)

# See the kernel documentation: Documentation/networking/ip-sysctl.txt
my $tcp_rmem_min = 4096;
my $tcp_rmem_default = 256000; # TCP specific receive memory pool size.
my $tcp_rmem_max = 3000000; # TCP specific receive memory pool size.

my $tcp_wmem_min = 4096;
my $tcp_wmem_default = 256000; # TCP specific receive memory pool size.
my $tcp_wmem_max = 3000000; # TCP specific receive memory pool size.

my $tcp_mem_lo = 20000000; # Below here there is no memory pressure.
my $tcp_mem_pressure = 30000000; # Can use up to 30MB for TCP buffers.
my $tcp_mem_high = 30000000; # Can use up to 30MB for TCP buffers.

Quoted 8 lines

John.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 3 lines
Ben Greear <[email protected]>       <Ben_Greear AT excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear
#7 lore raw

You might want to have a look at
http://www.psc.edu/networking/perf_tune.html
It's a pretty good reference on TCP tuning.

Also, one gotcha: in earlier versions of 2.4 the window scaling factor was set using the DEFAULT window size instead of the MAXIMUM window size; that is, window scaling is set such that neither autotuning nor application level tuning can actually tune to the maximum size (it can't be represented with the default window scale factor). I don't think this has been fixed yet, so you may have to set your default to be the same as your maximum for tests.

-Eric

Signature · 5 lines
------------------------------------------------------------
        Eric H. Weigle -- http://public.lanl.gov/ehw/
"They that can give up essential liberty to obtain a little
temporary safety deserve neither" -- Benjamin Franklin
------------------------------------------------------------
#8 lore raw
In reply to Ben Greear

From: Ben Greear <[email protected]>
Date: Mon, 03 Feb 2003 10:03:48 -0800

Also, if it's as simple as allocating a few more buffers for tcp, maybe we should consider defaulting to higher in the normal kernel? (I'm not suggesting **my** numbers..)

The current values are the only "safe" defaults. Here "safe" means that if you have thousands of web connections, clients cannot force the serve to queue large amounts of traffic per socket.

The attack goes something like: Open N thousand connections to server, ask for large static object, do not ACK any of the data packets. Server must thus hold onto N thousnad * maximum socket write buffer bytes amount of memory.

author
#9 lore raw
Quoted 7 lines · David S. Miller

From: Ben Greear <[email protected]>
Date: Mon, 03 Feb 2003 10:03:48 -0800

Also, if it's as simple as allocating a few more buffers for tcp, maybe we should consider defaulting to higher in the normal kernel? (I'm not suggesting **my** numbers..)

Quoted 8 lines

The current values are the only "safe" defaults. Here "safe" means that if you have thousands of web connections, clients cannot force the serve to queue large amounts of traffic per socket.

The attack goes something like: Open N thousand connections to server, ask for large static object, do not ACK any of the data packets. Server must thus hold onto N thousnad * maximum socket write buffer bytes amount of memory.

Why would it use the maximum socket for a connection with low to no acks, ie low to no throughput? Seems like the connection would have to scale up to full speed/sliding-window, which would require the DoS guy to have large receive bandwidth, and also enough precision to stop acking as soon as the window gets big (but before the object download has completed.) This does not seem like a great DoS to me.

On my system, the default memory seems to be about 80k (docs say it is based on how much memory I have (128MB)). How big can N get? If N is 10k I can be DOS'd for only 800k?

Signature · 3 lines
Ben Greear <[email protected]>       <Ben_Greear AT excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear
#10 lore raw

From: Ben Greear <[email protected]>
Date: Mon, 03 Feb 2003 23:50:05 -0800

Why would it use the maximum socket for a connection with low to no acks, ie low to no throughput?

You open up the congestion window by ACK'ing a few windows worth of data, then you stop ACK'ing.

I'm sorry if that wasn't obvious.

author
#11 lore raw
Quoted 8 lines · David S. Miller

From: Ben Greear <[email protected]>
Date: Mon, 03 Feb 2003 23:50:05 -0800

Why would it use the maximum socket for a connection with low to no acks, ie low to no throughput?

You open up the congestion window by ACK'ing a few windows worth of data, then you stop ACK'ing.

I think I understand, but on my system it seem to take 5-8 seconds for the bandwidth to get up to ~20Mbps (with my larger buffer settings mentioned earlier). This is with 25ms latency. With the default settings I can run about 8Mbps, so it would appear to me that only 3x the current default buffer settings should get a window size enough to go ~20Mbps at 25ms latency.

Am I correct that if I have 10k clients doing their worst tricks, and 3 * (80k, my default according to the kernel) == 240k, then I have at most 2.4MB denial of service? Assuming 60k clients, that is only about 15MB of DoS? If true, that is a fairly small time DoS considering the RAM available on today's machines.

You claim for a very large N that the denial of service can happen. I am just trying to understand the upper bound of N, and thus the upperbound of the memory consumption assuming each connection is using it's maximum buffer size.

Thanks,
Ben

Signature · 3 lines
Ben Greear <[email protected]>       <Ben_Greear AT excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear
author
#12 lore raw

Ben Greear wrote:

[ a bunch of bogus math]

Sorry...I think my math is off by many zeros. I should stick to reading slashdot this time of night!

Ben

Signature · 3 lines
Ben Greear <[email protected]>       <Ben_Greear AT excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear
#13 lore raw
In reply to Ben Greear

From: Ben Greear <[email protected]>
Date: Tue, 04 Feb 2003 00:42:02 -0800

Am I correct that if I have 10k clients doing their worst tricks, and 3 * (80k, my default according to the kernel) == 240k, then I have at most 2.4MB denial of service? Assuming 60k clients, that is only about 15MB of DoS? If true, that is a fairly small time DoS considering the RAM available on today's machines.

Add in the struct sk_buff for each packet as well, which is dependant upon MSS. Thus you could make the clients use a super-small MSS to get more per-packet struct sk_buff overhead. The list goes on and on. At least Linux, unlike BSD, makes an attempt to account for the sk_buff overhead in the limits :-)