Skip to contentSkip to conversation

how to get the latency down on maxed out classes?

LARTC Abraham van der Merwe 5 participants 10 messages Dec 7, 2002 – Dec 9, 2002

lore
Share & tools
lore
#0 lore raw

Hi!

I'm using HTB to shape traffic to/from clients, but one of the problems I have is that once a class utilizes its maximum potential, they latency spirals out of control.

For example:

     .-----.
     | isp |
     `-----'
        |
   .--------.
   | shaper |
   `--------'
        |

.--------.
| client |
`--------'

lets say I want to limit traffic to/from client to 64kbit. now, client opens a tcp connection blasting away at full speed.

If client now pings isp, it gets on average around 7 seconds latency. I tried to improve this by using SFQ on the leaf nodes of my HTB hierarchy, but that does not really improve the situation, only makes it much worse. with SFQ I get anything between 250ms and 13 seconds latency.

I then tried fifos. With small packet fifos the packet loss is just to great to be of any use and even then the latency is quite high (~200ms). If I make the fifos big enough so that unreasonable numbers of packets isn't dropped, it just doesn't do much to the latency/or throughput. This behaviour kind of makes sense, but doesn't help me :P

I'm thinking of using RED, but the number of parameters is daunting and I have no idea how the HTB rate correlates to packet size and burst rates for red.

Does anybody have any idea how to get the latency down and still maintain the correct throughput?

Signature · 11 lines
Regards
 Abraham

Sooner or later you must pay for your sins.
(Those who have already paid may disregard this cookie).

___________________________________________________
 Abraham vd Merwe [ZR1BBQ] - Frogfoot Networks
 P.O. Box 3472, Matieland, Stellenbosch, 7602
 Cell: +27 82 565 4451 Http: http://www.frogfoot.net
 Email: [email protected]
#1 lore raw

Hi

even i have same problem using CBQ.INIT
let me know if you get any solution
thanks
hare
----- Original Message -----
From: "Abraham van der Merwe" <[email protected]>
To: "Linux Advanced Routing & Traffic Control list" <[email protected]>
Sent: Saturday, December 07, 2002 6:45 PM
Subject: [LARTC] how to get the latency down on maxed out classes?

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#2 lore raw
In reply to Abraham van der Merwe
Quoted 40 lines · Abraham van der Merwe

Hi!

I'm using HTB to shape traffic to/from clients, but one of the problems I have is that once a class utilizes its maximum potential, they latency spirals out of control.

For example:

     .-----.

     | isp |

     `-----'
   .--------.

   | shaper |

   `--------'

   .--------.

   | client |

   `--------'

lets say I want to limit traffic to/from client to 64kbit. now, client opens a tcp connection blasting away at full speed.

If client now pings isp, it gets on average around 7 seconds latency. I tried to improve this by using SFQ on the leaf nodes of my HTB hierarchy, but that does not really improve the situation, only makes it much worse. with SFQ I get anything between 250ms and 13 seconds latency.

I then tried fifos. With small packet fifos the packet loss is just to great to be of any use and even then the latency is quite high (~200ms). If I make the fifos big enough so that unreasonable numbers of packets isn't dropped, it just doesn't do much to the latency/or throughput. This behaviour kind of makes sense, but doesn't help me :P

Quoted 6 lines

I'm thinking of using RED, but the number of parameters is daunting and I have no idea how the HTB rate correlates to packet size and burst rates for red.

Does anybody have any idea how to get the latency down and still maintain the correct throughput?

If you want low latency for some traffic (ping, telnet, ssh), then you can create a separate class for it.
And if you have a 64kbit modem, you have to be sure you never send more data then the modem can handle. If you send more data, the hugh modem queue's will be filled so they create a lot of latency. So for a 64kbit link, try to limit ALL traffic to 60kbit so the queue's of the modem are never filled.

Stef

Signature · 8 lines
[email protected]
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#3 lore raw
In reply to Abraham van der Merwe

Hi Stef!

Quoted 8 lines
Quoted 2 lines

Does anybody have any idea how to get the latency down and still maintain the correct throughput?

If you want low latency for some traffic (ping, telnet, ssh), then you can create a separate class for it.
And if you have a 64kbit modem, you have to be sure you never send more data then the modem can handle. If you send more data, the hugh modem queue's will be filled so they create a lot of latency. So for a 64kbit link, try to limit ALL traffic to 60kbit so the queue's of the modem are never filled.

I'm doing all my tests under ideal conditions (over 100mbit lan and shaping the traffic to something low such as 384kbit). The problem is that the latency becomes unnatural.

If you have a normal line, e.g. 64kbit and you saturate the line, the latency still stays within limits, but with HTB, the latency can become very high if you have multiple concurrent tcp sessions going at full steam in a class. I can understand why it does this, but I need a way to get the latency down to acceptable limits.

Signature · 11 lines
Regards
 Abraham

I never killed a man that didn't deserve it.
		-- Mickey Cohen

___________________________________________________
 Abraham vd Merwe [ZR1BBQ] - Frogfoot Networks
 P.O. Box 3472, Matieland, Stellenbosch, 7602
 Cell: +27 82 565 4451 Http: http://www.frogfoot.net
 Email: [email protected]
#4 lore raw
In reply to Abraham van der Merwe
Quoted 17 lines · Abraham van der Merwe

Hi Stef!

Quoted 10 lines
Quoted 2 lines

Does anybody have any idea how to get the latency down and still maintain the correct throughput?

If you want low latency for some traffic (ping, telnet, ssh), then you can create a separate class for it.
And if you have a 64kbit modem, you have to be sure you never send more data then the modem can handle. If you send more data, the hugh modem queue's will be filled so they create a lot of latency. So for a 64kbit link, try to limit ALL traffic to 60kbit so the queue's of the modem are never filled.

I'm doing all my tests under ideal conditions (over 100mbit lan and shaping the traffic to something low such as 384kbit). The problem is that the latency becomes unnatural.

Quoted 5 lines

If you have a normal line, e.g. 64kbit and you saturate the line, the latency still stays within limits, but with HTB, the latency can become very high if you have multiple concurrent tcp sessions going at full steam in a class. I can understand why it does this, but I need a way to get the latency down to acceptable limits.

The only thing where I can think of, is adding a small fifo to each class. But you already tried and it created packets loss. So I can't help you.

Stef

Signature · 8 lines
[email protected]
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#5 lore raw
In reply to Abraham van der Merwe
Quoted 5 lines

If you have a normal line, e.g. 64kbit and you saturate the line, the latency still stays within limits, but with HTB, the latency can become very high if you have multiple concurrent tcp sessions going at full steam in a class. I can understand why it does this, but I need a way to get the latency down to acceptable limits.

Are you sure the latency and packet loss are not the same, when you have the same queue size as the router has ?

Is the traffic locally generated (on the same machine as the HTB is running) ? If so then perhaps that's the difference.

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#6 lore raw
In reply to Abraham van der Merwe
Quoted 7 lines

lets say I want to limit traffic to/from client to 64kbit. now, client opens a tcp connection blasting away at full speed.

If client now pings isp, it gets on average around 7 seconds latency. I tried to improve this by using SFQ on the leaf nodes of my HTB hierarchy, but that does not really improve the situation, only makes it much worse. with SFQ I get anything between 250ms and 13 seconds latency.

You understand what's going on here?
As I recall, both pfifo and sfq default to queues of length 128 packets. If you fill that with 1500 byte packets you have ~200Kbytes which is about 1.6Mbits. At 64Kbit/sec that would take ~30 sec to send so your latency could be as high as 30 sec.
You can limit this latency by reducing the queue size.

On the other hand, the application that fills the queue evidently doesn't mind large latency. Otherwise it wouldn't fill the queue.

I think I posted to this list once a description (maybe even the code?) of another way to limit latency - drop packets that have been in the queue for more than a timeout period (I tend to use 3 sec).

SFQ should have the desirable result that one tcp connection won't slow down another one or a ping.

Quoted 2 lines

I then tried fifos. With small packet fifos the packet loss is just to great to be of any use and even then the latency is quite high (~200ms).

You consider 200ms high? One max size packet = 1500 bytes = 12Kbit which is about 200ms on a 64Kbit link. You can't expect to do better.

Quoted 3 lines

I'm thinking of using RED, but the number of parameters is daunting and I have no idea how the HTB rate correlates to packet size and burst rates for red.

RED should be independent of HTB.
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#7 lore raw
In reply to Abraham van der Merwe

Hi Don!

Quoted 4 lines
Quoted 2 lines

I then tried fifos. With small packet fifos the packet loss is just to great to be of any use and even then the latency is quite high (~200ms).

You consider 200ms high? One max size packet = 1500 bytes = 12Kbit which is about 200ms on a 64Kbit link. You can't expect to do better.

The problem is that with 200ms the packet loss is so much that the link is effectively useless (90% packet loss). As soon as I make the queue big enough to not drop significant amounts of packets, the latency goes way up (>3 secs).

#8 lore raw
In reply to Abraham van der Merwe
Quoted 4 lines · Abraham van der Merwe

Hi Don!

Quoted 2 lines
Quoted 2 lines

I then tried fifos. With small packet fifos the packet loss is just to great to be of any use and even then the latency is quite high (~200ms).

A small detail: what are "small packet fifos"? You mean fifos that can only hold a small number of packets? Or fifos that only hold packets with small numbers of bytes?

Quoted 7 lines
Quoted 2 lines

You consider 200ms high? One max size packet = 1500 bytes = 12Kbit which is about 200ms on a 64Kbit link. You can't expect to do better.

The problem is that with 200ms the packet loss is so much that the link is effectively useless (90% packet loss). As soon as I make the queue big enough to not drop significant amounts of packets, the latency goes way up (>3 secs).

I don't understand the connection between 200ms and packet loss. If you make the queue small (in packet capacity) then worst case latency decreases. Packet loss occurs in either case whenever a packet arrives and the queue is full.

If you try to send at a higher rate than allowed then you will fill the queue in either case (a small queue more quickly, of course), and from then on you will lose packets. If you send packets at twice the allowed rate you lose half of them, if you send at 10 times the allowed rate you lose 90%.

The fact that you're losing lots of packets, though, indicates to me that you're acting like an attacker, and dropping most of that traffic is therefore exactly the right thing to do.

If you were using a correctly working tcp it would not continue to send at 10 times the allowed rate. It would notice that packets were being lost and would slow down until the loss rate became very small.

Similarly, I don't understand the latency issue. An application that cares about latency will not create a large backlog.

What is this application that is sending faster than the link allows and wants a low latency, and why is it misbehaving?
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#9 lore raw
In reply to Abraham van der Merwe
Quoted 2 lines

Similarly, I don't understand the latency issue. An application that cares about latency will not create a large backlog.

I think the problem may be that a TCP flow will create a queue of packets, behind which all other packets queue. Thus one sees 200ms latency via 'ping'. Of course this is exactly what you would see on a real connection of the specified throttled speed (e.g. a DSL line). However it seems to surprise folks that there's a relationship between throughput and latency. They expect to be able to throttle traffic to 100Kbits, but still see 5ms latency. Without special measures that isn't going to happen.

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/