htb child classes don't borrow form "parent's parent class"
Dear list,
I have the problem that child classes don't seem to issue the parent class to borrow from their parent.
This is a simplified version of my script:
tc qdisc del dev $DEV root
tc qdisc add dev $DEV root handle 1: htb default 10
tc class add dev $DEV parent 1: classid 1:1 htb rate 10mbit tc class add dev $DEV parent 1:1 classid 1:10 htb rate 1kbit ceil 512kbit
#groups
tc class add dev $DEV parent 1: classid 1:f1 htb rate 2048kbit ceil 10mbit burst 50k
tc class add dev $DEV parent 1:f1 classid 1:a1 htb rate 2048kbit ceil 10mbit prio 2 burst 50k
tc class add dev $DEV parent 1:f1 classid 1:b1 htb rate 2048kbit ceil 6mbit prio 3 burst 50k
tc class add dev $DEV parent 1:f1 classid 1:c1 htb rate 2048kbit ceil 5mbit prio 4 burst 50k
tc class add dev $DEV parent 1: classid 1:f2 htb rate 1024kbit ceil 10mbit burst 50k
tc class add dev $DEV parent 1:f2 classid 1:a2 htb rate 1024kbit ceil 10mbit prio 2 burst 50k
tc class add dev $DEV parent 1:f2 classid 1:b2 htb rate 1024kbit ceil 6mbit prio 3 burst 50k
tc class add dev $DEV parent 1:f2 classid 1:c2 htb rate 1024kbit ceil 5mbit prio 4 burst 50k
#users
tc class add dev $DEV parent 1:a1 classid 1:3001 htb rate 64kbit ceil 512kbit burst 50k
tc qdisc add dev $DEV parent 1:3001 sfq perturb 10
tc filter add dev $DEV parent 1:0 protocol ip prio 16 u32 match ip dst 172.30.10.1 flowid 1:3001
tc class add dev $DEV parent 1:b1 classid 1:3002 htb rate 64kbit ceil 1024kbit burst 50k
tc qdisc add dev $DEV parent 1:3002 sfq perturb 10
tc filter add dev $DEV parent 1:0 protocol ip prio 16 u32 match ip dst 172.30.10.2 flowid 1:3002
tc class add dev $DEV parent 1:c1 classid 1:3003 htb rate 64kbit ceil 1024kbit burst 50k
tc qdisc add dev $DEV parent 1:3003 sfq perturb 10
tc filter add dev $DEV parent 1:0 protocol ip prio 16 u32 match ip dst 172.30.10.3 flowid 1:3003
tc class add dev $DEV parent 1:c2 classid 1:3004 htb rate 64kbit ceil 1024kbit burst 50k
tc qdisc add dev $DEV parent 1:3004 sfq perturb 10
tc filter add dev $DEV parent 1:0 protocol ip prio 16 u32 match ip dst 172.30.10.4 flowid 1:3004
.
.
.
The script is dynamically generated, periodically called, and should put the users (distinguished through their IP address) according to their traffic usages into lower priorities (axx,bxx,cxx) respecting different bandwidth needs but also respecting minimum rates for different groups (fxx).
Now the borrowing among user classes (1:3xxx) is working; but for some reason and I just can't figure out why, the group classes (fxx) don't borrow from each other when their is unused bandwidth available. The "ceil 10mbit" just seems to be ignored :(
e.g.:
class htb 1:3003 parent 1:c1 leaf 92f4: prio 0 quantum 1000 rate 64000bit ceil 1024Kbit burst 50Kb/8 mpu 0b overhead 0b cburst 1727b/8 mpu 0b overhead 0b level 0
Sent 22500226 bytes 28784 pkts (dropped 0, overlimits 0) rate 245080bit 39pps
lended: 15306 borrowed: 13478 giants: 0 <------------ user borrowed
from others
tokens: -429574 ctokens: 13314
but:
class htb 1:f1 root rate 2048Kbit ceil 10000Kbit burst 50Kb/8 mpu 0b overhead 0b cburst 2848b/8 mpu 0b overhead 0b level 7
Sent 221521122 bytes 257990 pkts (dropped 0, overlimits 0) rate 1818Kbit 260pps
lended: 0 borrowed: 0 giants: 0 <------------ nothing going on here
tokens: -1118817 ctokens: -274
class htb 1:c1 parent 1:f1 rate 2048Kbit ceil 6000Kbit burst 50Kb/8 mpu 0b overhead 0b cburst 2349b/8 mpu 0b overhead 0b level 6
Sent 22500226 bytes 28784 pkts (dropped 0, overlimits 0) rate 230776bit 37pps
lended: 13478 borrowed: 0 giants: 0 <------------ or here
tokens: 204546 ctokens: 3121
Any help is highly appreciated!
Thanx,
Bernd Froemel
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] htb child classes don't borrow form "parent's parent
Quoted 5 lines · Bernd Froemel
tc class add dev $DEV parent 1: classid 1:1 htb rate 10mbit tc class add dev $DEV parent 1: classid 1:f1 htb rate 2048kbit ceil 10mbit burst 50k
tc class add dev $DEV parent 1: classid 1:f2 htb rate 1024kbit ceil 10mbit burst 50k
That's most likely because you're using different rate/ceil for root classes, which does not make sense at all. A root class does not have parents or siblings to borrow from. It can always use it's full rate.
Quoted 1 line
The "ceil 10mbit" just seems to be ignored :(
It might very well be the case that root classes just ignore the ceil parameter since they can't borrow anyway.
You should rebuild your tree so that you only have one root class.
HTH
Andreas
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] htb child classes don't borrow form "parent's parent
In reply to Bernd FroemelThanks for your input.
Quoted 7 lines
Quoted 2 lines
The "ceil 10mbit" just seems to be ignored :(
It might very well be the case that root classes just ignore the ceil parameter since they can't borrow anyway.
You should rebuild your tree so that you only have one root class.
Could you give me a short example what you mean?
The only other way I can think of would be a tree (respecting the described group and user requirements) with qdiscs on every node down:
#just to be sure that I have the beginning right:
tc qdisc add dev $DEV root handle 1: htb default 20
tc class add dev $DEV parent 1: classid 1:1 htb rate 10mbit #total bandwidth of link
#the two main nodes - approved and non approved traffic:
tc class add dev $DEV parent 1:1 classid 1:10 htb rate 8mbit ceil 10mbit prio 1 tc class add dev $DEV parent 1:1 classid 1:40 htb rate 2mbit ceil 2mbit prio 2
#attach qdiscs:
tc qdisc add dev $DEV parent 1:10 handle 10: htb
tc qdisc add dev $DEV parent 1:20 sfq perturb 10
#groups
tc class add dev $DEV parent 10: classid 10:f1 htb rate 2048kbit ceil 10mbit tc qdisc add dev $DEV parent 10:f1 handle f1: htb
#bandwidth priorities of group f1:
tc class add dev $DEV parent f1: classid f1:a1 htb rate 2048kbit ceil 10mbit prio 1 tc qdisc add dev $DEV parent f1:a1 handle a1: htb
tc class add dev $DEV parent f1: classid f1:b1 htb rate 2048kbit ceil 10mbit prio 2 tc qdisc add dev $DEV parent f1:b1 handle b1: htb
tc class add dev $DEV parent f1: classid f1:c1 htb rate 2048kbit ceil 10mbit prio 3 tc qdisc add dev $DEV parent f1:c1 handle c1: htb
#further groups omitted
#users
tc class add dev $DEV parent a1: classid a1:1 htb rate 64kbit ceil 1024kbit
tc qdisc add dev $DEV parent a1:1 sfq perturb 10
tc filter add dev $DEV parent a1:0 protocol ip prio 16 u32 match ip dst 172.30.10.1 flowid a1:1 tc filter add dev $DEV parent f1:0 protocol ip prio 16 u32 match ip dst 172.30.10.1 flowid f1:a1 tc filter add dev $DEV parent 10:0 protocol ip prio 16 u32 match ip dst 172.30.10.1 flowid 10:f1 tc filter add dev $DEV parent 1:0 protocol ip prio 16 u32 match ip dst 172.30.10.1 flowid 1:10
but this hadn't brought the desired effect either.
I need to schedule the traffic output 4 times:
-for other purposes & unknown traffic [10:,20:]
-for groups (that they have a minimum rate at all times) [fxx:] -for priority (fair use, traffic usage based)[axx:,bxx:,cxx:] -for users (that they don't exceed a given maximum and are fairly treated within their priority) [axx:yyyy],[bxx:yyyy],[cxx:yyyy]
Users should be allowed to borrow from each other, insufficient bandwidth "requests" should be forwarded to the parent class/qdisc which should try to fulfill it by borrowing from other groups.
Am I missing an important point? Unfortunately I couldn't find any examples showing a more complex tree.
Thank you,
Bernd
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] htb child classes don't borrow form "parent's parent
In reply to Bernd FroemelQuoted 2 lines · Bernd Froemel
Am I missing an important point? Unfortunately I couldn't find any examples showing a more complex tree.
Yes, unless I'm completely mistaken, you misunderstood something about the way QDiscs and especially HTB works.
Your old setup looked something like this:
1: HTB Qdisc
| | |
| | \--- 1:1 HTB Root Class (10mbit)
| | |
| | \--- 1:10 HTB Child Class (1kbit/512kbit)
| |
| \--- 1:f1 HTB Root Class (2048kbit)
| |
| \--- 1:a1 HTB Child Class (2048kbit)
| \--- 1:b1 HTB Child Class (2048kbit)
| \--- 1:c1 HTB Child Class (2048kbit)
|
\--- 1:f2 HTB Root Class (1024kbit)
|
\--- 1:a2 HTB Child Class (1024kbit)
\--- 1:b2 HTB Child Class (1024kbit)
\--- 1:c2 HTB Child Class (1024kbit)Root classes are independent from each other. A HTB qdisc by itself has unlimited bandwidth. Limiting is done by the classes, the root class imposing a global limit. You have three root classes, and a total bandwidth capacity of 10mbit+2048kbit+1024kbit on your link. This is assuming that the ceil parameter does not have any effect for root classes, otherwise you'd have 30mbit altogether.
Now, the 1:f1 class has a guaranteed rate of 2048kbit. It's children, 1:a1 1:b1 and 1:c1 also have 2048kbit guaranteed each. This is impossible, since a parent with only 2048kbit guaranteed bandwidth cannot guarantee three times as much bandwidth to it's children. The 1:f2 class has the same problem.
Your new setup isn't any different; instead of using more than one root class, you now use nested HTB qdiscs, which is extremely nasty:
1: HTB QDisc
|
\--- 1:1 HTB Root Class (10MBit)
|
\--- 1:10 HTB Child Class (8MBit/10MBit)
| *
| \--- 10: HTB QDisc
| |
| \--- 10:f1 HTB Root Class (2MBit)
| *
| \--- f1: HTB QDisc
| [...]
|
\--- 1:40 HTB Child Class (2MBit/2MBit)The tree is broken at *. Although it looks like the HTB QDiscs depend on each other, they actually don't really know about each other. There is no borrowing/lending possible across different QDiscs / root classes.
You should build a tree with only one HTB QDisc and only one root class. Something similar to this:
1: HTB QDisc
|
\--- 1:1 HTB Root Class (10MBit) [The total link capacity]
|
\--- 1:10 HTB Child Class (7MBit/10MBit) [High priority stuff]
| |
| \--- 1:a1 HTB Child Class (3MBit/10MBit)
| \--- 1:b1 HTB Child Class (2MBit/10MBit)
| \--- 1:c1 HTB Child Class (2MBit/10MBit)
|
\--- 1:40 HTB Child Class (3MBit/10MBit) [Low priority stuff]
|
\--- 1:a2 HTB Child Class (1MBit/10MBit)
\--- 1:b2 HTB Child Class (1MBit/10MBit)
\--- 1:c2 HTB Child Class (1MBit/10MBit)Make sure your child class rates add up to the parent class rate (7MBit = 3+2+2MBit, 3MBit = 1+1+1MBit). Use smaller units than MBit if you want to divide more accurately.
That's about it. Use only one HTB Qdisc, and only one HTB Root Class, and divide rates properly, then things should start working the way you expect them to.
HTH
Andreas
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
[LARTC] htb child classes don't borrow form "parent's parent class" :(
In reply to Bernd FroemelQuoted 20 lines
You should build a tree with only one HTB QDisc and only one root class. Something similar to this:
1: HTB QDisc
|
\--- 1:1 HTB Root Class (10MBit) [The total link capacity]| \--- 1:10 HTB Child Class (7MBit/10MBit) [High priority stuff] | | | \--- 1:a1 HTB Child Class (3MBit/10MBit) | \--- 1:b1 HTB Child Class (2MBit/10MBit) | \--- 1:c1 HTB Child Class (2MBit/10MBit) | \--- 1:40 HTB Child Class (3MBit/10MBit) [Low priority stuff] | \--- 1:a2 HTB Child Class (1MBit/10MBit) \--- 1:b2 HTB Child Class (1MBit/10MBit) \--- 1:c2 HTB Child Class (1MBit/10MBit)
Quoted 7 lines
Make sure your child class rates add up to the parent class rate (7MBit = 3+2+2MBit, 3MBit = 1+1+1MBit). Use smaller units than MBit if you want to divide more accurately.
That's about it. Use only one HTB Qdisc, and only one HTB Root Class, and divide rates properly, then things should start working the way you expect them to.
First, thank you very much for your effort!
Now everything becomes a lot more clear (especially about of the qdisc roles, and the complete nonsense I tried)!
Just that I don't make any more mistakes, would this tree work as I expect it (with the groups and users and priority stuff)?
1: HTB QDisc
|
\--- 1:1 HTB Root Class (10MBit) [The total link capacity]
|
\--- 1:f1 HTB Child Class (3MBit/10MBit) [group one]
| |
| \--- 1:a1 HTB Child Class (1MBit/10MBit) prio 1
| | |
| | \---1:e1 HTB child Class (120kbit/1Mbit) [user 1]
| | \---1:e2 HTB child Class (120kbit/2Mbit) [user 2]
| | \---1:e3 HTB child Class (120kbit/512kbit) [user 3]
| |
| \--- 1:b1 HTB Child Class (1MBit/10MBit) prio 2
| | |
| | \---1:e4 HTB child Class (120kbit/1Mbit) [user 1]
| | \---1:e5 HTB child Class (120kbit/2Mbit) [user 2]
| | \---1:e6 HTB child Class (120kbit/512kbit) [user 3]
| |
| \--- 1:c1 HTB Child Class (1MBit/10MBit) prio 3
|
\--- 1:f2 HTB Child Class (3MBit/10MBit) [group two]
| |
| \--- 1:a2 HTB Child Class (1MBit/10MBit) prio 1
| \--- 1:b2 HTB Child Class (1MBit/10MBit) prio 2
| \--- 1:c2 HTB Child Class (1MBit/10MBit) prio 3
|
\--- 1:f3 HTB Child Class (3Mbit/10Mbit) [group three]
.
.
.Regards,
Bernd
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] htb child classes don't borrow form "parent's parent
In reply to Bernd FroemelQuoted 2 lines · Bernd Froemel
Just that I don't make any more mistakes, would this tree work as I expect it (with the groups and users and priority stuff)?
Sorry for the late reply, I wasn't at home over the weekend. The tree looks better now, except the class rates still don't add up
Quoted 1 line
\--- 1:1 HTB Root Class (10MBit) [The total link capacity]
Children of this class have 3MBit+3MBit+3MBit = 9MBit, should be 10.
Quoted 1 line
| \--- 1:a1 HTB Child Class (1MBit/10MBit) prio 1
Children of this class have 120kbit+120kbit+120kbit = 360kbit, should be 1MBit.
Quoted 1 line
| \--- 1:b1 HTB Child Class (1MBit/10MBit) prio 2
Same here.
For my own setup, I'm using a script that converts all rates into bps (or whatever is the smallest unit available) and then does the math for the child classes automatically. I recommend you do something similar. Your classes will have a lot more borrowing to do if you don't maximize their guaranteed rates.
About the priority stuff, I'm not sure if it will work the way you expect it, because I don't know exactly what you expect. Classes with low priority won't be allowed to borrow, if there is another class with higher priority that wants to borrow.
As a result, on full links (and links that require shaping are usually full), low priority classes usually don't get to borrow anything at all.
I suggest not using the prio parameter at all in the beginning; logical errors in the shaping setup are much more obvious if you don't have to consider priority also. Later, if you find that you require additional prio for HTB classes, you can start experimenting with it and see if the results fit your requirements.
HTH
Andreas
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] htb child classes don't borrow form "parent's parent
In reply to Bernd FroemelNo problem! You've been a great help, now I have a rather good working shaping system. I expect the last "problems" (shaping adaptation is a bit slow, it takes up to 5-10 minutes until the full link is used after script reinit) will be solved when I make the script implementation "proper". But to sum it up: one programmer (usually, I don't do system administrating stuff) and around 300 users are much happier now :)
Quoted 1 line
Same here.
I see - thanks for pointing it out.
Quoted 5 lines
For my own setup, I'm using a script that converts all rates into bps (or whatever is the smallest unit available) and then does the math for the child classes automatically. I recommend you do something similar. Your classes will have a lot more borrowing to do if you don't maximize their guaranteed rates.
I didn't know it is that strict (right now I guarantee around 90kbit/sec to everyone but allow an up-to bandwidth of the selected subscription (512kbit-2Mbit) through the ceil parameter. I will fix this and will calculate a fixed rate for every user class which sums up to the full bandwidth of the parent class.
Again, thanks for all your help Andreas!
Byebye,
Bernd
Quoted 32 lines · Bernd Froemel
1: HTB QDisc
|
\--- 1:1 HTB Root Class (10MBit) [The total link capacity]| \--- 1:f1 HTB Child Class (3MBit/10MBit) [group one] | | | \--- 1:a1 HTB Child Class (1MBit/10MBit) prio 1 | | | | | \---1:e1 HTB child Class (120kbit/1Mbit) [user 1] | | \---1:e2 HTB child Class (120kbit/2Mbit) [user 2] | | \---1:e3 HTB child Class (120kbit/512kbit) [user 3] | | | \--- 1:b1 HTB Child Class (1MBit/10MBit) prio 2 | | | | | \---1:e4 HTB child Class (120kbit/1Mbit) [user 1] | | \---1:e5 HTB child Class (120kbit/2Mbit) [user 2] | | \---1:e6 HTB child Class (120kbit/512kbit) [user 3] | | | \--- 1:c1 HTB Child Class (1MBit/10MBit) prio 3 | \--- 1:f2 HTB Child Class (3MBit/10MBit) [group two] | | | \--- 1:a2 HTB Child Class (1MBit/10MBit) prio 1 | \--- 1:b2 HTB Child Class (1MBit/10MBit) prio 2 | \--- 1:c2 HTB Child Class (1MBit/10MBit) prio 3 | \--- 1:f3 HTB Child Class (3Mbit/10Mbit) [group three] . . .
Quoted 3 lines
Regards,
Bernd
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc