Skip to contentSkip to conversation

Split bandwidth equally per IP

LARTC Mihai Vlad 6 participants 15 messages Dec 4, 2003 – Dec 17, 2003

lore
Share & tools
lore

Mihai Vlad

author
#0 lore raw

It's me again, with the same problem.

I want to be able to split the bandwidth per IP, as I have 10 clients in my LAN and I want to be sure they get the same rate when downloading.

I followed all the steps from the HTB manual, but the script can be "beaten" when one uses more connection for downloading ("FlashGet" or "Download Accelerator").

I have the same rates for all the 10 IP-s in my LAN.

Stef Coene told me to decrease the burst. (Indeed, 80k was a huge burst) I did that, but it seams nothing's changed.

Jon Zeeff said that sfq generally creates a queue per connection. (And that is the reason why clients with more connections get more bandwidth)

So I need to change this "per connection" into "per IP".

Everything is clear, but I do not know what to change into my script to do this.

I guess that I have to change "sfq perturb 10" into something else, but I do not know into what :)

Here is my script:

$TC qdisc add dev $LAN_IFACE root handle 1: htb default 30

$TC class add dev $LAN_IFACE parent 1: classid 1:1 htb rate 120kbit ceil 120kbit burst 2k

#--------default class---------

$TC class add dev $LAN_IFACE parent 1:1 classid 1:30 htb rate 1kbit ceil 120kbit burst 2k

$TC qdisc add dev $LAN_IFACE parent 1:30 handle 300: sfq perturb 10

#-----10 client classes-------------

$TC class add dev $LAN_IFACE parent 1:1 classid 1:11 htb rate 12kbit ceil 120kbit burst 2k

$FILTER_IP 192.168.0.121 flowid 1:11

$TC qdisc add dev $LAN_IFACE parent 1:11 handle 110: sfq perturb 10

$TC class add dev $LAN_IFACE parent 1:1 classid 1:12 htb rate 12kbit ceil 120kbit burst 2k

$FILTER_IP 192.168.0.122 flowid 1:12

$TC qdisc add dev $LAN_IFACE parent 1:12 handle 120: sfq perturb 10

etc...

Thanks again for your patience.

#1 lore raw
Quoted 5 lines · Mihai Vlad

It's me again, with the same problem.

I want to be able to split the bandwidth per IP, as I have 10 clients in my LAN and I want to be sure they get the same rate when downloading.

Quoted 8 lines

I followed all the steps from the HTB manual, but the script can be "beaten" when one uses more connection for downloading ("FlashGet" or "Download Accelerator").

I have the same rates for all the 10 IP-s in my LAN.

Stef Coene told me to decrease the burst. (Indeed, 80k was a huge burst) I did that, but it seams nothing's changed.

The burst parameter is not made for hugh bursts. If you don't specify a burst, htb will calculate the smallest one suitable for that class.

Quoted 2 lines

Jon Zeeff said that sfq generally creates a queue per connection. (And that is the reason why clients with more connections get more bandwidth)

Yes and no. Sfq creates 1 queue for a connection, but htb should distribute the bandwidth eaqual between the classes even if 1 class has more connections in it.

Quoted 4 lines

So I need to change this "per connection" into "per IP".

Everything is clear, but I do not know what to change into my script to do this.

Can you post the output of
tc -s -d class show dev eth0

Stef

Signature · 8 lines
[email protected]
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.openprojects.net

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#2 lore raw
In reply to Mihai Vlad

Here are my untested (but simple) changes to SFQ to make it "fair" on a per ip basis. I believe that this is actually what most people want - equal bandwidth among a number of users without being tricked by users using many connections.

In net/sched/sch_sfq.c:

                h = iph->daddr;
                h2 = iph->saddr^iph->protocol;
                if (!(iph->frag_off&htons(IP_MF|IP_OFFSET)) &&
                    (iph->protocol = IPPROTO_TCP ||
                     iph->protocol = IPPROTO_UDP ||
                     iph->protocol = IPPROTO_ESP))
                        h2 ^= *(((u32*)iph) + iph->ihl);
                break;

change to:

h = iph->daddr;
                h2 = iph->saddr^iph->protocol;
                if (!(iph->frag_off&htons(IP_MF|IP_OFFSET)) &&
                    (iph->protocol = IPPROTO_TCP ||
                     iph->protocol = IPPROTO_UDP ||
                     iph->protocol = IPPROTO_ESP))
                        h2 ^= *(((u32*)iph) + iph->ihl);
                break;

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#3 lore raw
In reply to Mihai Vlad

Here are my untested (but simple) changes to SFQ to make it share outgoing bandwidth "fairly" per ip address (roughly, per local user) instead of being susceptible to being tricked by users with many connections. Don't use this on the wrong side of a NAT box where there is only one source ip address in use.

In net/sched/sch_sfq.c:

Change:

                h = iph->daddr;
                h2 = iph->saddr^iph->protocol;
                if (!(iph->frag_off&htons(IP_MF|IP_OFFSET)) &&
                    (iph->protocol = IPPROTO_TCP ||
                     iph->protocol = IPPROTO_UDP ||
                     iph->protocol = IPPROTO_ESP))
                        h2 ^= *(((u32*)iph) + iph->ihl);
                break;

To:

                h = h2 = iph->saddr;
                break;

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

author
#4 lore raw
In reply to Mihai Vlad

Thanks very much for this reply.

In fact letting HTB calculate itself the burst parameter, and hard-coding the quantum parameter for each leaf class (as Stef suggested), made the traffic shaping much more accurate than it was before. (especially the quantum settings > 1500).

I will make the changes in sch_sfq.c and keeep you all informed with the results.

-----Original Message-----
From: [email protected] [mailto:[email protected]]
Sent: Friday, December 05, 2003 5:19 PM
To: Stef Coene
Cc: Mihai Vlad; [email protected]
Subject: Re: [LARTC] Split bandwidth equally per IP

Here are my untested (but simple) changes to SFQ to make it share outgoing bandwidth "fairly" per ip address (roughly, per local user) instead of
being susceptible to being tricked by users with many connections. Don't use
this on the wrong side of a NAT box where there is only one source ip address
in use.

In net/sched/sch_sfq.c:

Change:

                h = iph->daddr;
                h2 = iph->saddr^iph->protocol;
                if (!(iph->frag_off&htons(IP_MF|IP_OFFSET)) &&
                    (iph->protocol = IPPROTO_TCP ||
                     iph->protocol = IPPROTO_UDP ||
                     iph->protocol = IPPROTO_ESP))
                        h2 ^= *(((u32*)iph) + iph->ihl);
                break;

To:

                h = h2 = iph->saddr;
                break;

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#5 lore raw
In reply to Mihai Vlad

Hello,

: In fact letting HTB calculate itself the burst parameter, and : hard-coding the quantum parameter for each leaf class (as Stef : suggested), made the traffic shaping much more accurate than it was : before. (especially the quantum settings > 1500).

That Stef guy! Always making good suggestions.

: I will make the changes in sch_sfq.c and keeep you all informed with : the results.

Achtung! There is already an esfq qdisc [0] which does this! This patch may be a good one, but since esfq already exists, perhaps you could try that instead.

-Martin

[0] http://www.ssi.bg/~alex/esfq/index.html

Signature · 5 lines
Martin A. Brown --- SecurePipe, Inc. --- [email protected]

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#6 lore raw
In reply to Mihai Vlad

Good point, what has to be done to make this part of the standard linux kernel distribution?

Quoted 8 lines

Achtung! There is already an esfq qdisc [0] which does this! This patch may be a good one, but since esfq already exists, perhaps you could try that instead.

-Martin

[0] http://www.ssi.bg/~alex/esfq/index.html

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

author
#7 lore raw
In reply to Mihai Vlad

Hello Martin,

I have recently read your howto from:
http://www.ibiblio.org/pub/Linux/docs/HOWTO/other-formats/html_single/Traffi
c-Control-HOWTO.html

I quote:

"Of the many types of contention for network bandwidth, this is one of the easier types of contention to address in general. By using the SFQ qdisc, traffic in a particular queue can be separated into flows, each of which will be serviced fairly (inside that queue). Well-behaved applications (and users) will find that using SFQ and ESFQ are sufficient for most sharing needs.

The Achilles heel of these fair queuing algorithms is a misbehaving user or application which opens many connections simultaneously (e.g., eMule, eDonkey, Kazaa). By creating a large number of individual flows, the application can dominate slots in the fair queuing algorithm. Restated, the fair queuing algorithm has no idea that a single application is generating the majority of the flows, and cannot penalize the user. Other methods are called for."

Can you post a real script using esfq, that splits the bandwidth equally per IP?

The documentation on esfq is scarce and I have no idea where to start from.

Thanks again for your time.

-----Original Message-----
From: Martin A. Brown [mailto:[email protected]]
Sent: Friday, December 05, 2003 7:03 PM
To: Mihai Vlad
Cc: [email protected]
Subject: RE: [LARTC] Split bandwidth equally per IP

Achtung! There is already an esfq qdisc [0] which does this! This patch may be a good one, but since esfq already exists, perhaps you could try that instead.

-Martin

[0] http://www.ssi.bg/~alex/esfq/index.html

Signature · 7 lines
Martin A. Brown --- SecurePipe, Inc. --- [email protected]



_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#8 lore raw
In reply to Mihai Vlad

ESFQ allows this to be "per ip address", which solves the problem.

I have not seen applications that grab many ip addresses for a single host, although this is possible.

Quoting Mihai Vlad <[email protected]>:

Quoted 51 lines

Hello Martin,

I have recently read your howto from:
http://www.ibiblio.org/pub/Linux/docs/HOWTO/other-formats/html_single/Traffi
c-Control-HOWTO.html

I quote:

"Of the many types of contention for network bandwidth, this is one of the easier types of contention to address in general. By using the SFQ qdisc, traffic in a particular queue can be separated into flows, each of which will be serviced fairly (inside that queue). Well-behaved applications (and users) will find that using SFQ and ESFQ are sufficient for most sharing needs.

The Achilles heel of these fair queuing algorithms is a misbehaving user or application which opens many connections simultaneously (e.g., eMule, eDonkey, Kazaa). By creating a large number of individual flows, the application can dominate slots in the fair queuing algorithm. Restated, the fair queuing algorithm has no idea that a single application is generating the majority of the flows, and cannot penalize the user. Other methods are called for."

Can you post a real script using esfq, that splits the bandwidth equally per
IP?

The documentation on esfq is scarce and I have no idea where to start from.

Thanks again for your time.

-----Original Message-----
From: Martin A. Brown [mailto:[email protected]]
Sent: Friday, December 05, 2003 7:03 PM
To: Mihai Vlad
Cc: [email protected]
Subject: RE: [LARTC] Split bandwidth equally per IP

Quoted 18 lines

Achtung! There is already an esfq qdisc [0] which does this! This patch may be a good one, but since esfq already exists, perhaps you could try that instead.

-Martin

[0] http://www.ssi.bg/~alex/esfq/index.html

Signature · 7 lines
Martin A. Brown --- SecurePipe, Inc. --- [email protected]



_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#9 lore raw
In reply to Mihai Vlad

Hello Mihai,

: Can you post a real script using esfq, that splits the bandwidth : equally per IP?

I wish I had an example script to lend you....

To my recollection your issue was an obnoxious piece of software which was generating many outbound flows. This sort of a technique should prevent any single IP from dominating your bandwidth (without your allowing that domination):

tc qdisc add dev $LAN_DEV parent $HTB_CLASS handle $ESFQ_HANDLE \
esfq perturb 10 depth $UNIQUE_CLIENTS hash dst

This attaches an esfq qdisc to the class which contains competing workstations/users for download bandwidth (in your environment). Reminder! We have already deNATted (is that a word) the inbound packets, so the destination is the real workstation IP, and the

Here's the help string [from the README]....

Usage: ... esfq [ perturb SECS ] [ quantum BYTES ] [ depth FLOWS ] [ divisor HASHBITS ] [ limit PKTS ] [ hash HASHTYPE]

Where:
HASHTYPE := { classic | src | dst }

I'd agree that the easily available documentation for ESFQ is minimal....I would love to remedy that, so let me know how your experimentation with the qdisc goes. In your case, I'd suggest starting with a "hash dst".

When I cautioned about Jon Zeeff's patch [0], I didn't necessarily mean to scare you away from it. It may be a very good patch--I simply wanted to point out that there already exists a tool to solve the problem you appeared to need to solve. Note also the recent list activity (by Chijioke Kalu) with regard to kernel-2.6 and ESFQ [1].

Also, the section of my text (from tldp.org) you quoted was specifically about the very same download accelerators which were giving you fits. These tools simultaneously create a large number of TCP connections for a single application-layer function (usually fetching a large file) in order to (attempt to) defeat per-connection byte-rate limits. The design of SFQ did not anticipate this deliberate flooding attempt. ESFQ takes a step toward correcting this.

Again, I don't know if ESFQ will solve your problem--I hope it does, and I hope to hear a report from you that this was the tool you needed.

Best of luck,

-Martin

[0] http://mailman.ds9a.nl/pipermail/lartc/2003q4/010919.html
[1] http://mailman.ds9a.nl/pipermail/lartc/2003q4/010939.html

Signature · 6 lines
Martin A. Brown --- SecurePipe, Inc. --- [email protected]


_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#10 lore raw
In reply to Mihai Vlad

Hello all (again),

: ESFQ allows this to be "per ip address", which solves the problem. :
: I have not seen applications that grab many ip addresses for a single : host, although this is possible.

Would WRR be a better candidate for Mihai's problem? I have no experience with WRR.

Others?

-Martin

Signature · 5 lines
Martin A. Brown --- SecurePipe, Inc. --- [email protected]

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
author
#11 lore raw
In reply to Mihai Vlad

Thanks very much for your help.

It seems that Jim diGriz already accomplished that, but I did not read the entire page to see that there was a script to download. My mistake :)

For those who don't know yet, here is the link:
http://www.digriz.org.uk/jdg-qos-script/index.htm

I will try to do some testing with it, and I hope it will work. The only thing I an afraid of is the "tiny" rates I have to use for each client (7-8 Kbit).

I will keep you informed with my progress.

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#12 lore raw
In reply to Mihai Vlad

I have used rates of 4 - 5kbits per system and works fine. Also improved surfing by doing a TC+Squid patch.

But lets see how your's goes.

K

Quoted 13 lines

Thanks very much for your help.

It seems that Jim diGriz already accomplished that, but I did not read the entire page to see that there was a script to download. My mistake :)

For those who don't know yet, here is the link:
http://www.digriz.org.uk/jdg-qos-script/index.htm

I will try to do some testing with it, and I hope it will work. The only thing I an afraid of is the "tiny" rates I have to use for each client (7-8 Kbit).

Quoted 8 lines

I will keep you informed with my progress.

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

_________________________________________________________________
Cell phone ‘switch’ rules are taking effect — find out more here.
http://special.msn.com/msnbc/consumeradvocate.armx

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#13 lore raw
In reply to Mihai Vlad

Hello, Martin, list,

Probably WRR would be a nice solution, indeed. But the question was is there anyone on this list that could post a working (part of) script, that shares bandwidth equally by source/destination hashes using ESFQ.

I've posted once or twice here on this subject, but of no avail. Maybe half an year ago I tried to find a working solution myself, but the bandwidth division was not fair, at least not AS fair as I'd like. It is possible the reason for this to be the quite tiny amounts of bandwidth I gave to each connection (around 6 Kbits/s), but still...

I dont know, probably Alexander Atanasov (the author of this qdisc) could enlighen us.

On Sun, 7 Dec 2003 00:23:18 -0600 (CST)
"'Martin A. Brown'" <[email protected]> wrote:

: Hello all (again),
:

 :  : ESFQ allows this to be "per ip address", which solves the problem.
 :  :
 :  : I have not seen applications that grab many ip addresses for a single
 :  : host, although this is possible.

:
: Would WRR be a better candidate for Mihai's problem? I have no experience : with WRR.
:
: Others?
:
: -Martin

________________________________________________________________________

Cheers,
Nickola

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM d? s:- a- C++++ UL++++ P+ L++++ E--- W-- N o K- w-- O- M- V- PS+ PE Y PGP- t 5 X R- tv- b+++ DI-- D+ G e h-- r y**
------END GEEK CODE BLOCK------

author
#14 lore raw
In reply to Mihai Vlad

http://www.digriz.org.uk/jdg-qos-script/

This is in fact the link... My mistake.

I promised I will keep you informed with my progress in splitting the bandwidth equally per IP. Unfortunately, for some odd reason Jim diGriz's script did not work as I expected...

At that moment I was sure that esfq is uselsss, until I created my own script and it worked surprisingly well :) The error as I spotted using iptraf as a monitor was about 0.2 - 0.4 Kbytes which is very very small.

I spotted also a little "malfunction"... Maybe you can tell me what's wrong.

Imagine a 128 kbits (not Kbytes... a verry small) bandwidth that I have to split for 12 clients in my LAN. Using the "default" HTB would result in a r2q error and a small calculated quantum. So, as Stef sugested I hardcoded the quantum parameter for all the 12 leaf classes:

$TC class add dev $LAN_IFACE parent 1:12 classid 1:121 htb rate 8kbit ceil 96kbit quantum 1500

I noticed that changing the rate parameter will not change the guaranteed bandwith. So here are some questions:

1. Is "quantum" the real parameter related with the guaranteed bandwidth?
2. Is the sum of all "leaf class quantums" relevant? (As far as I understood the sum of all rates in the leaf classes must not exceed the parent class rate...)

I went further and changed the quantum from diffent clients from 1500 to 3000 (based on my knowledge this would mean double guaranteed bandwidth)

I set up a liitle test. I started 2 client computers and made them download the same file with the same download manager. Client_1 had a 3000 quantum and Client_2 had a 1500 quantum. The rates observed with iptraf were 10Kbytes for Client_1 and 5Kbytes for Client_2, which was the expected result.

Another test with the quantum of Client_1 set to 1500000*10. The expected result was to have a bandwith for Client_1 10 times greater than the one for Client_2... The real results told me that whatever chenges are made for the quantum, the Client_2 bandwith won't fall below 4Kbytes...

3. Can you tell me why the Cllient_2 bandwidth will not decrease below 4 Kbytes?

Thanks again for your time :)

Mihai Vlad

-----Original Message-----
From: Fabian Gervan [mailto:[email protected]]
Sent: Wednesday, December 17, 2003 4:59 PM
To: Mihai Vlad
Subject: Re[2]: [LARTC] Split bandwidth equally per IP

Hello Mihai,

MV> It seems that Jim diGriz already accomplished that, but I did not read the
MV> entire page to see that there was a script to download. My mistake :) MV> For those who don't know yet, here is the link:
MV> http://www.digriz.org.uk/jdg-qos-script/index.htm

404 error.
Fabian

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/