Skip to contentSkip to conversation

TCP Rate Control

LARTC Varun Varma 4 participants 13 messages May 14, 2003 – May 16, 2003

lore
Share & tools
lore
#0 lore raw

Hi!

Take a setup as follows:

Clients---->[eth0]Proxy/Firewall[eth1]---->Internet

The clients are using the proxy/firewall for various client like activities like browsing, email, ftp etc. Most of the traffic is "download" traffic, i.e. clients send very small requests and recieve very large replies.

I want to do the most common thing...configure download limits/fairness of use among the clients.

I understand that I can achieve this by traffic shaping the *outgoing* traffic on eth0, but that is a less than optimal solution. The problem is that the link to the Internet can still get choked, if too many clients start pulling too much data. Traffic shaping on eth0 would limit the bandwidth each client sees and TCPs congestion control mechanisms *might* indeed slow down the download requests from clients, but this does not seem like a elegant way to achieve the result.

I am wondering if there is any implementation [or planned project] for TCP based rate control, like that provided by commercial solutions like PacketShaper and Sitara.

Regards,
-Varun

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#1 lore raw
Quoted 14 lines · Varun Varma

Hi!

Take a setup as follows:

Clients---->[eth0]Proxy/Firewall[eth1]---->Internet

The clients are using the proxy/firewall for various client like activities like browsing, email, ftp etc. Most of the traffic is "download" traffic, i.e. clients send very small requests and recieve very large replies.

I want to do the most common thing...configure download limits/fairness of use among the clients.

Quoted 7 lines

I understand that I can achieve this by traffic shaping the *outgoing* traffic on eth0, but that is a less than optimal solution. The problem is that the link to the Internet can still get choked, if too many clients start pulling too much data. Traffic shaping on eth0 would limit the bandwidth each client sees and TCPs congestion control mechanisms *might* indeed slow down the download requests from clients, but this does not seem like a elegant way to achieve the result.

Why not?
The only problem you will have is the Proxy. If you shape on eth0, you shape cached objects and objects fetched from the internet. But you don't want to limit traffic that's cached. So somehow you have to be able to separate cached and non cached traffic.

Quoted 3 lines

I am wondering if there is any implementation [or planned project] for TCP based rate control, like that provided by commercial solutions like PacketShaper and Sitara.

You can do the same (and even more) with a linux box.

Stef

Signature · 8 lines
[email protected]
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#2 lore raw
In reply to Varun Varma
Quoted 6 lines
Quoted 3 lines

I am wondering if there is any implementation [or planned project] for TCP based rate control, like that provided by commercial solutions like PacketShaper and Sitara.

You can do the same (and even more) with a linux box.

Can you write how to do it ?

�����������������������������������ΛΒ™¨¥�x%�Λ,S ω��Y��χlυ―η–m§��™¨¥™©�vΟZώy™¨¥™©�–+-�wθώV«µΑΞY3�†Ϋi��εj»\ώ�ΰ

#3 lore raw
In reply to Varun Varma
Quoted 7 lines · Brandis Jaroslav
Quoted 5 lines
Quoted 3 lines

I am wondering if there is any implementation [or planned project] for TCP based rate control, like that provided by commercial solutions like PacketShaper and Sitara.

You can do the same (and even more) with a linux box.

Can you write how to do it ?

www.lartc.org
But what exactly do you mean with tcp based rate control?

Stef

Signature · 8 lines
[email protected]
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#4 lore raw
In reply to Varun Varma
Quoted 2 lines · Stef Coene

But what exactly do you mean with tcp based rate control?

I suppose he means the way products like PacketShaper (from Packeteer)
work: instead of using queues to limit traffic, the use waht they call "TCP rate control". It consists of modifiying TCP "packets" on the fly, transparently, adapting TCP window sizes, shortly delaying ACKs, and so on, so the sender and receiver "agree" to send traffic at a lower (configured) rate.

Seems to be a patented "idea" in the USA, but I remember someone on this list talked not long ago about he waaas implementing something like this for Linux, from outside the USA. Check the archives for the message:
Subject: Re: [LARTC] How far can TC go?
From: Patrick McHardy <[email protected]>
Date: Sat, 29 Mar 2003 11:08:30 +0100

Hope it helps.

Signature · 5 lines
Jose Luis Domingo Lopez
Linux Registered User #189436     Debian Linux Sid (Linux 2.5.69)
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#5 lore raw
In reply to Varun Varma
Quoted 11 lines · Jose Luis Domingo Lopez

On Wednesday, 14 May 2003, at 22:09:58 +0200,

Quoted 1 line · Stef Coene

But what exactly do you mean with tcp based rate control?

I suppose he means the way products like PacketShaper (from Packeteer)
work: instead of using queues to limit traffic, the use waht they call "TCP rate control". It consists of modifiying TCP "packets" on the fly, transparently, adapting TCP window sizes, shortly delaying ACKs, and so on, so the sender and receiver "agree" to send traffic at a lower (configured) rate.

You mean something like the gred or red qdisc.
But I wonder if it will be a big different between "TCP rate contorl" and plain old shaping.

Quoted 8 lines

Seems to be a patented "idea" in the USA, but I remember someone on this list talked not long ago about he waaas implementing something like this for Linux, from outside the USA. Check the archives for the message:
Subject: Re: [LARTC] How far can TC go?
From: Patrick McHardy <[email protected]>
Date: Sat, 29 Mar 2003 11:08:30 +0100

Hope it helps.

ok.

Stef

Signature · 8 lines
[email protected]
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#6 lore raw
In reply to Varun Varma
Quoted 11 lines · Varun Varma
Quoted 1 line · Stef Coene

You mean something like the gred or red qdisc.

The difference is RED drops the packets after they arrive basing it on the understanding that the TCP implementations would take this as a hint and slow down the transmission rates.

Rate control on the other hand takes a more "hands on" approach to the problem. As Jose mentioned, ack pacing, window sizing etc...it actually exploits fundamentals in TCP flow control itself.

Quoted 6 lines
Quoted 2 lines

But I wonder if it will be a big different between "TCP rate contorl" and plain old shaping.

There are lot of comparisions on this. I don't want to get into a debate
here, since people from each group [rate control vs. queueing] feel
very stongly about their stand.

But I'm still intested in the comparision :)

Quoted 4 lines

Simply put, there are dozens of queueing disciplines, because some might "behave better" than other in some cases. I *feel* that rate control would work better in some particular cases, so I was interested in knowing if a rate control like implementation is available under Linux.

Not in the default linux kernel.

Quoted 8 lines
Quoted 6 lines
Quoted 6 lines

Seems to be a patented "idea" in the USA, but I remember someone on this list talked not long ago about he waaas implementing something like this for Linux, from outside the USA. Check the archives for the message:
Subject: Re: [LARTC] How far can TC go?
From: Patrick McHardy <[email protected]>
Date: Sat, 29 Mar 2003 11:08:30 +0100

Thanks for the reference...still need to check it out.

I already did :

Packeteer has various patents covering tcp rate control and everything else they do, including the "idea" to look at upper layers to detect the type of traffic.
I live in germany so i don't really care that much about their patents (they had none in europe last time i checked). last summer i started implementing tcp rate control as qdisc for linux. i haven't worked on it for a couple of month now, but if anyone wishes to participate i would be glad to dig out my source again. it is basically working, the remaining problems are mostly how to detect and handle interactive traffic.

Patrick

Quoted 6 lines

About the patent thing...I know for sure that ack pacing has been published in paper before. Note: I am not suggesting how to circumvent this patent in any way. That said, I think it should be possible to work out other mechanisms, based around TCP flow control, which don't work the way Packeteer's patent works.

Quoted 6 lines

If you google for "linux rate control", you come up with a link like:

http://www.postel.org/pipermail/end2end-interest/2001-April/000701.html

Which is a master's thesis, proposing another way to implement rate control.

Stef

Signature · 8 lines
[email protected]
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#7 lore raw
In reply to Varun Varma
Quoted 2 lines · Stef Coene

You mean something like the gred or red qdisc.

The difference is RED drops the packets after they arrive basing it on the understanding that the TCP implementations would take this as a hint and slow down the transmission rates.

Rate control on the other hand takes a more "hands on" approach to the problem. As Jose mentioned, ack pacing, window sizing etc...it actually exploits fundamentals in TCP flow control itself.

Quoted 2 lines

But I wonder if it will be a big different between "TCP rate contorl" and plain old shaping.

There are lot of comparisions on this. I don't want to get into a debate
here, since people from each group [rate control vs. queueing] feel
very stongly about their stand.

Simply put, there are dozens of queueing disciplines, because some might "behave better" than other in some cases. I *feel* that rate control would work better in some particular cases, so I was interested in knowing if a rate control like implementation is available under Linux.

Quoted 7 lines
Quoted 7 lines

Seems to be a patented "idea" in the USA, but I remember someone on this list talked not long ago about he waaas implementing something like this for Linux, from outside the USA. Check the archives for the message:
Subject: Re: [LARTC] How far can TC go?
From: Patrick McHardy <[email protected]>
Date: Sat, 29 Mar 2003 11:08:30 +0100

Thanks for the reference...still need to check it out.

About the patent thing...I know for sure that ack pacing has been published in paper before. Note: I am not suggesting how to circumvent this patent in any way. That said, I think it should be possible to work out other mechanisms, based around TCP flow control, which don't work the way Packeteer's patent works.

If you google for "linux rate control", you come up with a link like:

http://www.postel.org/pipermail/end2end-interest/2001-April/000701.html

Which is a master's thesis, proposing another way to implement rate control.

Regards,
-Varun

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#8 lore raw
In reply to Varun Varma

Stef,

Quoted 5 lines
Quoted 3 lines

There are lot of comparisons on this. I don't want to get into a debate
here, since people from each group [rate control vs. queuing] feel
very strongly about their stand.

But I'm still intested in the comparision :)

It's you quest for the truth that endears you to all of us. :)

Googling for "tcp rate control" would lead you to a lot of material, including documents "proving" queuing is better than rate control.

Here's the test jig I plan to setup...

Windows client running Gozilla, for crude throughput reporting

|
|(1)
|
->"Gateway" m/c with two NICs [RTL8139 based] running nisnet and tbf, 
iptraf and MRTG--
                 |
                 |(2)
                 |
                 ->Linux based FTP/HTTP Server

I will try with sustained downloads, perhaps 1GB. The tbf would be set to limit at 100kbps.

From what I expect, I would only get 100kbps on the windows client m/c,
i.e. at point (1). But between the gateway and server, i.e. point (2), the bandwidth consumption would be higher. How much higher I don't know
- it might be anything from 100kbps to NIC speed. If it is within a 5% tolerance, i.e. from 95kbps to 100kbps, I would say that the world is a great place and I am happy to be here...i.e. queuing would have achieved what I want.

But, I suspect the difference would be much more...

Would most probably do this on this weekend and publish my results. I don't have too much experience with tc, so I might get the values wrong for the tbf.

Has anyone done this kind of test? Where can I get some published results? All the tests I have seen are done from the client perspective...or is it just a case of RTFM?

I am going to go ahead with the test in any case.

This only tests one side of the story. I have tested two commercial solutions and can testify that they give very fine-grained control on the bandwidth, albeit at the cost of huge latency fluctuations. Don't have access to these units right now, so won't be able to include the results from there.

I as see it, if the above test goes as I think it would, then that itself is a strong case for looking into other mechanisms.

Quoted 7 lines
Quoted 5 lines

Simply put, there are dozens of queuing disciplines, because some might "behave better" than other in some cases. I *feel* that rate control would work better in some particular cases, so I was interested in knowing if a rate control like implementation is available under Linux.

Not in the default linux kernel.

I have been tracking the tc development for quite some time, and have been hoping for someone to develop this.

Quoted 22 lines
Quoted 9 lines
Quoted 7 lines
Quoted 7 lines

Seems to be a patented "idea" in the USA, but I remember someone on this list talked not long ago about he waaas implementing something like this for Linux, from outside the USA. Check the archives for the message:
Subject: Re: [LARTC] How far can TC go?
From: Patrick McHardy <[email protected]>
Date: Sat, 29 Mar 2003 11:08:30 +0100

Thanks for the reference...still need to check it out.

I already did :

Packeteer has various patents covering tcp rate control and everything else they do, including the "idea" to look at upper layers to detect the type of traffic.
I live in germany so i don't really care that much about their patents (they had none in europe last time i checked). last summer i started implementing tcp rate control as qdisc for linux. i haven't worked on it for a couple of month now, but if anyone wishes to participate i would be glad to dig out my source again. it is basically working, the remaining problems are mostly how to detect and handle interactive traffic.

Quoted 1 line

Patrick

Like I said, it's you quest for the truth that endears you to us!

Regards,
-Varun

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#9 lore raw
In reply to Varun Varma
Quoted 31 lines
Quoted 1 line

But I'm still intested in the comparision :)

It's you quest for the truth that endears you to all of us. :)

Googling for "tcp rate control" would lead you to a lot of material, including documents "proving" queuing is better than rate control.

Here's the test jig I plan to setup...

Windows client running Gozilla, for crude throughput reporting

|(1)

->"Gateway" m/c with two NICs [RTL8139 based] running nisnet and tbf, iptraf and MRTG--

                 |(2)

                 ->Linux based FTP/HTTP Server

I will try with sustained downloads, perhaps 1GB. The tbf would be set to limit at 100kbps.

From what I expect, I would only get 100kbps on the windows client m/c,
i.e. at point (1). But between the gateway and server, i.e. point (2), the bandwidth consumption would be higher. How much higher I don't know
- it might be anything from 100kbps to NIC speed. If it is within a 5% tolerance, i.e. from 95kbps to 100kbps, I would say that the world is a great place and I am happy to be here...i.e. queuing would have achieved what I want.

Quoted 1 line

But, I suspect the difference would be much more...

Why? What will happen with the packets that are arrived too much? Do you really think they all will be lost?
I'm no tcp specialist, but the traffic at 2 will be exactly 100kbps. I see it like this : client starts download and goes faster and faster. When it reaches 100kbps, all tokens in the bucket are used. After that, the gateway starts delaying packets. From the tbf manpage :

If no tokens are available, packets are queued, up to a configured limit

Packets are not dropped but delayed. At that time, the server noticed that the packets are not arriving at a speed > 100kbps (they are queued in the bucket) at the client and throttle downs. When the server throttles down below 100kbps, the queued packets can be send again because there are tokens available in the bucket. So after some time it will be stable at 100kbps.

Again, I don't know how tbf works. This is just the way I think it works.

Stef

Signature · 8 lines
[email protected]
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#10 lore raw
In reply to Varun Varma
Quoted 16 lines · Michael T. Babcock
Quoted 1 line

Packets are not dropped but delayed. At that time, the server noticed

that

Quoted 3 lines

the packets are not arriving at a speed > 100kbps (they are queued in the bucket) at the client and throttle downs. When the server throttles down below 100kbps, the queued packets can be send again because there are

tokens

Quoted 5 lines

available in the bucket. So after some time it will be stable at 100kbps.

Again, I don't know how tbf works. This is just the way I think it works.

Quoted 7 lines

Its worth reading up on how RED works while you're at it. There are many good
whitepapers on RED/GRED w.r.t. how TCP handles exponential backoff. As I understand it, TBF may not work all the time because the TCP server (sending the packets) may respond incorrectly to the delayed ACKs on the part of the client. Dropping packets in a RED fashion as the queue builds up instead of queuing the packets should get perfectly stable 100kbit/s.

I think we can use some test results to prove this. Varun ??

Stef

Signature · 8 lines
[email protected]
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#11 lore raw
In reply to Varun Varma

Stef, Michael,

Quoted 9 lines
Quoted 7 lines

Its worth reading up on how RED works while you're at it. There are many good
whitepapers on RED/GRED w.r.t. how TCP handles exponential backoff. As I understand it, TBF may not work all the time because the TCP server (sending the packets) may respond incorrectly to the delayed ACKs on the part of the client. Dropping packets in a RED fashion as the queue builds up instead of queuing the packets should get perfectly stable 100kbit/s.

I think we can use some test results to prove this. Varun ??

This mail caught me in the middle of writing a long theoritical explanation of why I think rate control might be better, which included my technical, philosophical, financial and ..er.. sexual reasons.

But then I looked at this mail and realised that the best way to take this forward is tests. I would be game to testing both TBF and RED in the jig I have mentioned in my previous mail. Hopefully, should be able produce comprehensive documentation, including detailed explanation of the test setup, and the results over this weekend.

Regards,
-Varun

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#12 lore raw
In reply to Varun Varma
Quoted 9 lines · Michael T. Babcock
Quoted 5 lines

But then I looked at this mail and realised that the best way to take this forward is tests. I would be game to testing both TBF and RED in the jig I have mentioned in my previous mail. Hopefully, should be able produce comprehensive documentation, including detailed explanation of the test setup, and the results over this weekend.

I would be specifically interested in how an HTB + RED setup compares with the same (effective) configuration performs with just HTB.

Quoted 7 lines

1) HTB class on 100mbit line allocating 1mbit queue.
2) HTB class on 100mbit line allocating 1mbit RED queue.
3) HTB class on 100mbit line allocating 100 1mbit queues.
4) HTB class on 100mbit line allocating 100 1mbit RED queues.

At near-saturation points, RED *should* give better performance (closer to complete saturation of the link).

And also how it reaches the saturation. How many packets are dropped? Does it go straight to a flat line or is "jumping" around the line before it stabelizes? And what happens if you change the rate? How fast does it react? How many packets are dropped?
(I'm planning to create such a setup my self, but I have no spare NIC)

Stef

Signature · 8 lines
[email protected]
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/