HTB not working quite the way I'd expect?
I have a 2Mbit connection, and I would like to distribute that 250k/s (upstream bandwidth, of course, because I am adding the qdisc on our internet interface) as follows:
* From our residence students (199.212.55.128/25), min 50k/s, ceil 200k/s
* From our DMZ network (where our webserver is located, 199.212.53.0/24), min 200k/s, ceil 250k/s
This is the configuration I _think_ should work:
tc qdisc add dev eth8 root handle 1: htb default 11
tc class add dev eth8 parent 1: classid 1:1 htb rate 250kbps tc class add dev eth8 parent 1:1 classid 1:10 htb rate 200kbps\
ceil 250kbps burst 10k
tc qdisc add dev eth8 parent 1:10 sfq perturb 5
tc filter add dev eth8 protocol ip parent 1: prio 1 u32 match ip\
src 199.212.53.0/24 flowid 1:10
tc class add dev eth8 parent 1:1 classid 1:11 htb rate 50kbps \
ceil 200kbps burst 2k
tc qdisc add dev eth8 parent 1:11 sfq perturb 10
tc filter add dev eth8 protocol ip parent 1: prio 4 u32 match \
ip src 199.212.55.128/25 flowid 1:11
So, our residence students are assigned to classid 1:11. Notice that 1:10, our DMZ, is given prio 1, because I want it to get any extra bandwidth before our residence. Indeed, residence users are happily throttled at 200k/s (sustaining it, thanks to the proliferation of p2p file sharing programs). However, when I hit our webserver (from 3 outside sites with lots of bandwidth), I am only able to transfer at about 65k/s (peak 85k/s), while the residence students only drop down to about 160k-180k/s. What I would expect to happen is residence dropping to 50k/s, and our webserver serving up at 200k/s. (Incidentally I am monitoring the input/output on the interfaces using iptraf to measure bandwidth usage.) I am sustaining the transfer for 20-30 seconds before measuring bandwidth usage.
If I drop the ceil on 1:11 to 150kbps, I am able to get about 100-120k/s from the web server. And if I again drop the ceil to 100kbps, I get around 175k/s from the webserver. So clearly my remote sites are able to pull down the full 200k/s, which is my goal.
So knowing what I want, and my current configuration, can anyone see what I'm doing wrong?
Thanks,
Jason.
Signature · 9 lines
Academic Computing Support Specialist Algoma University College Sault Ste. Marie, Ontario 705-949-2301 x330 Personal Home Page http://www.auc.ca http://sault.org _______________________________________________ LARTC mailing list / [email protected] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/lartc/
Quoted 11 lines
tc class add dev eth8 parent 1:1 classid 1:10 htb rate 200kbps\ ceil 250kbps burst 10k tc filter add dev eth8 protocol ip parent 1: prio 1 u32 match ip\ src 199.212.53.0/24 flowid 1:10 tc class add dev eth8 parent 1:1 classid 1:11 htb rate 50kbps \ ceil 200kbps burst 2kSo, our residence students are assigned to classid 1:11. Notice that 1:10, our DMZ, is given prio 1, because I want it to get any extra bandwidth before our residence. Indeed, residence users are happily
You did it wrong ! prio 1 attached to filter has to do nothing with prio 1 attached to the class. I have a BIG objectives to the overal filter design in Linux QoS but I can't do anything with it. Simply add prio 1 to the class definition (last tc line above). And let me know ! ;)
devik
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/lartc/
Hi Martin. Thanks for your reply.
Quoted 3 lines · Martin Devera
filter design in Linux QoS but I can't do anything with it. Simply add prio 1 to the class definition (last tc line above). And let me know ! ;)
In fact, there is no change. Here is my current configuration, with the changes you suggested:
tc class add dev eth8 parent 1: classid 1:1 htb rate 250kbps
tc class add dev eth8 parent 1:1 classid 1:10 htb rate 200kbps \
ceil 250kbps burst 10k prio 1
tc qdisc add dev eth8 parent 1:10 sfq perturb 5
tc filter add dev eth8 protocol ip parent 1: u32 match ip \
src 199.212.53.0/24 flowid 1:10
tc class add dev eth8 parent 1:1 classid 1:11 htb rate 50kbps \
ceil 200kbps burst 2k prio 4
tc qdisc add dev eth8 parent 1:11 sfq perturb 5
tc filter add dev eth8 protocol ip parent 1: u32 match ip \
src 199.212.55.128/25 flowid 1:11Monitoring eth8, I see it sending out at 200k/s, all traffic generated from 199.212.55.128/25. This is what I'd expect. When I hit the webserver (on 199.212.53.0/24) hard, eth8 begins to send at 250k/s (it is a 2Mbit uplink), but the portion of that from the webserver is only about 50-90/s, not 200k/s like I'd expect based on id 1:10.
Any other thoughts?
Regards,
Jason.
Signature · 9 lines
Academic Computing Support Specialist Algoma University College Sault Ste. Marie, Ontario 705-949-2301 x330 Personal Home Page http://www.auc.ca http://sault.org _______________________________________________ LARTC mailing list / [email protected] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/lartc/
Quoted 7 lines
Quoted 2 lines
Simply add prio 1 to the class definition (last tc line above). And let me know ! ;)
In fact, there is no change. Here is my current configuration, with the changes you suggested:
ceil 200kbps burst 2k prio 4
default max prio is 3 by the way but it is not problem here ..
Quoted 5 lines
Monitoring eth8, I see it sending out at 200k/s, all traffic generated from 199.212.55.128/25. This is what I'd expect. When I hit the webserver (on 199.212.53.0/24) hard, eth8 begins to send at 250k/s (it is a 2Mbit uplink), but the portion of that from the webserver is only about 50-90/s, not 200k/s like I'd expect based on id 1:10.
very interesting ! I'm interested in "tc -s -d class show dev eth8" output in middle of your run. Please take several snapshots, copy into file along with time (I need to know the time difference between them). I'll look at them and try to deduce the problem. I use configuration like your regulary with no probs so that I'm really interested.
devik
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/lartc/
Quoted 3 lines · Martin Devera
very interesting ! I'm interested in "tc -s -d class show dev eth8" output in middle of your run. Please take several snapshots, copy into file along with time (I need to know the time difference between them).
Okay. Attached is a commented typescript taken. The first invocation of 'tc show' is before I initiated the remote wgets, the next 2 are during, and the last is after I stopped them. I don't quite understand all the data, but strictly looking at the rate field, I think it pretty nicely shows what's happening. (Except on the last reading, which shows a rate of ~40k/s even though the web server's interface was idle -- I assumed it was just an averaging window issue.)
Quoted 2 lines
I'll look at them and try to deduce the problem. I use configuration like your regulary with no probs so that I'm really interested.
Thanks very much for your help on this, Martin. :)
Jason.
Signature · 5 lines
Academic Computing Support Specialist Algoma University College Sault Ste. Marie, Ontario 705-949-2301 x330 Personal Home Page http://www.auc.ca http://sault.org
###
# Before hitting the webserver. This is normal traffic generated by
# our residence. iptraf showed ~200k/s being sent on eth8. ########
[root@hawk tack]# date && tc -s -d class show dev eth8
Sun Feb 3 15:47:22 EST 2002
class htb 1:11 parent 1:1 leaf 8066: prio 3 rate 400Kbit ceil 1600Kbit burst 2Kb/8 mpu 0b cburst 3647b/8 mpu 0b quantum 5120 level 0
Sent 75213499 bytes 106516 pkts (dropped 34, overlimits 70905)
rate 192901bps 236pps backlog 36p
lended: 38257 borrowed: 68223 giants: 0 injects: 0
tokens: -118 ctokens: -3836
class htb 1:1 root prio 0 rate 2000Kbit ceil 2000Kbit burst 4159b/8 mpu 0b cburst 4159b/8 mpu 0b quantum 25600 level 3
Sent 87895315 bytes 121163 pkts (dropped 0, overlimits 0)
rate 191724bps 239pps
lended: 68223 borrowed: 0 giants: 0 injects: 0
tokens: 3968 ctokens: 3968
class htb 1:10 parent 1:1 leaf 8065: prio 1 rate 1600Kbit ceil 2000Kbit burst 10Kb/8 mpu 0b cburst 4159b/8 mpu 0b quantum 20480 level 0
Sent 12717097 bytes 14683 pkts (dropped 0, overlimits 2668)
rate 1380bps 9pps
lended: 14683 borrowed: 0 giants: 0 injects: 0
tokens: 40577 ctokens: 13004
###
# Immediately following the previous command, I began the run by initiating
# wget on 3 separate sites downloading from our webserver (a large file). ########
[root@hawk tack]# date && tc -s -d class show dev eth8
Sun Feb 3 15:47:44 EST 2002
class htb 1:11 parent 1:1 leaf 8066: prio 3 rate 400Kbit ceil 1600Kbit burst 2Kb/8 mpu 0b cburst 3647b/8 mpu 0b quantum 5120 level 0
Sent 78931896 bytes 110921 pkts (dropped 34, overlimits 76204)
rate 182403bps 219pps backlog 18p
lended: 39606 borrowed: 71297 giants: 0 injects: 0
tokens: -7489 ctokens: -4151
class htb 1:1 root prio 0 rate 2000Kbit ceil 2000Kbit burst 4159b/8 mpu 0b cburst 4159b/8 mpu 0b quantum 25600 level 3
Sent 93010578 bytes 126696 pkts (dropped 0, overlimits 0)
rate 1663Kbit 247pps
lended: 71297 borrowed: 0 giants: 0 injects: 0
tokens: -2 ctokens: -2
class htb 1:10 parent 1:1 leaf 8065: prio 1 rate 1600Kbit ceil 2000Kbit burst 10Kb/8 mpu 0b cburst 4159b/8 mpu 0b quantum 20480 level 0
Sent 14099891 bytes 15793 pkts (dropped 0, overlimits 2988)
rate 32109bps 28pps
lended: 15793 borrowed: 0 giants: 0 injects: 0
tokens: 19305 ctokens: 2537
###
# The wgets are still going.
########
[root@hawk tack]# date && tc -s -d class show dev eth8
Sun Feb 3 15:48:00 EST 2002
class htb 1:11 parent 1:1 leaf 8066: prio 3 rate 400Kbit ceil 1600Kbit burst 2Kb/8 mpu 0b cburst 3647b/8 mpu 0b quantum 5120 level 0
Sent 81834500 bytes 114311 pkts (dropped 34, overlimits 80140)
rate 182926bps 217pps backlog 16p
lended: 40580 borrowed: 73715 giants: 0 injects: 0
tokens: -596 ctokens: -1591
class htb 1:1 root prio 0 rate 2000Kbit ceil 2000Kbit burst 4159b/8 mpu 0b cburst 4159b/8 mpu 0b quantum 25600 level 3
Sent 96964624 bytes 130954 pkts (dropped 0, overlimits 0)
rate 230521bps 258pps
lended: 73715 borrowed: 0 giants: 0 injects: 0
tokens: 7272 ctokens: 7272
class htb 1:10 parent 1:1 leaf 8065: prio 1 rate 1600Kbit ceil 2000Kbit burst 10Kb/8 mpu 0b cburst 4159b/8 mpu 0b quantum 20480 level 0
Sent 15150851 bytes 16659 pkts (dropped 0, overlimits 3082)
rate 47974bps 41pps
lended: 16659 borrowed: 0 giants: 0 injects: 0
tokens: 21062 ctokens: 667
###
# Immediately following the last command, I stopped the wgets. Our
# webserver was idle. For whatever reason, it shows the rate of 1:10 at # 62790bps but the web server's interface was clearly idle. I assume it # averages rate over a wide window, like 20-30 seconds? ########
[root@hawk tack]# date && tc -s -d class show dev eth8
Sun Feb 3 15:48:14 EST 2002
class htb 1:11 parent 1:1 leaf 8066: prio 3 rate 400Kbit ceil 1600Kbit burst 2Kb/8 mpu 0b cburst 3647b/8 mpu 0b quantum 5120 level 0
Sent 84370895 bytes 117247 pkts (dropped 34, overlimits 83715)
rate 177121bps 208pps backlog 19p
lended: 41432 borrowed: 75796 giants: 0 injects: 0
tokens: -13591 ctokens: -216
class htb 1:1 root prio 0 rate 2000Kbit ceil 2000Kbit burst 4159b/8 mpu 0b cburst 4159b/8 mpu 0b quantum 25600 level 3
Sent 100567608 bytes 134827 pkts (dropped 0, overlimits 0)
rate 240215bps 261pps
lended: 75796 borrowed: 0 giants: 0 injects: 0
tokens: -3658 ctokens: -3658
class htb 1:10 parent 1:1 leaf 8065: prio 1 rate 1600Kbit ceil 2000Kbit burst 10Kb/8 mpu 0b cburst 4159b/8 mpu 0b quantum 20480 level 0
Sent 16215919 bytes 17599 pkts (dropped 0, overlimits 3251)
rate 62790bps 52pps
lended: 17599 borrowed: 0 giants: 0 injects: 0
tokens: 22819 ctokens: -1203
[root@hawk tack]# date && tc -s -d class show dev eth8
Sun Feb 3 15:48:37 EST 2002
class htb 1:11 parent 1:1 leaf 8066: prio 3 rate 400Kbit ceil 1600Kbit burst 2Kb/8 mpu 0b cburst 3647b/8 mpu 0b quantum 5120 level 0
Sent 88862765 bytes 122423 pkts (dropped 34, overlimits 89207)
rate 187439bps 221pps backlog 25p
lended: 42738 borrowed: 79660 giants: 0 injects: 0
tokens: -17782 ctokens: -2329
class htb 1:1 root prio 0 rate 2000Kbit ceil 2000Kbit burst 4159b/8 mpu 0b cburst 4159b/8 mpu 0b quantum 25600 level 3
Sent 105201282 bytes 140534 pkts (dropped 0, overlimits 0)
rate 231031bps 267pps
lended: 79660 borrowed: 0 giants: 0 injects: 0
tokens: 7044 ctokens: 7044
class htb 1:10 parent 1:1 leaf 8065: prio 1 rate 1600Kbit ceil 2000Kbit burst 10Kb/8 mpu 0b cburst 4159b/8 mpu 0b quantum 20480 level 0
Sent 16367152 bytes 18136 pkts (dropped 0, overlimits 3262)
rate 42641bps 45pps
lended: 18136 borrowed: 0 giants: 0 injects: 0
tokens: 40609 ctokens: 13031
(tc output below cropped)
Quoted 5 lines
class htb 1:11 parent 1:1 leaf 8066: prio 3 rate 400Kbit ceil
1600Kbit burst 2Kb/8 mpu 0b cburst 3647b/8 mpu 0b quantum 5120 ] level 0
rate 192901bps 236pps backlog 36p
Quoted 8 lines
class htb 1:1 root prio 0 rate 2000Kbit ceil 2000Kbit burst 4159b/8
mpu 0b cburst 4159b/8 mpu 0b quantum 25600 level 3
rate 191724bps 239ppsclass htb 1:10 parent 1:1 leaf 8065: prio 1 rate 1600Kbit ceil
2000Kbit burst 10Kb/8 mpu 0b cburst 4159b/8 mpu 0b quantum 20480 level 0
rate 1380bps 9pps
Now that I study this a bit more, how can it be that the parent class, 1:1, has a rate that is LESS than the sum of its children's rates? Either I've uncovered a bug, or I'm greatly misunderstanding something. ;)
Jason.
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/lartc/
Quoted 8 lines
Quoted 3 lines
rate 192901bps 236pps backlog 36p
rate 191724bps 239pps
rate 1380bps 9ppsNow that I study this a bit more, how can it be that the parent class, 1:1, has a rate that is LESS than the sum of its children's rates? Either I've uncovered a bug, or I'm greatly misunderstanding something. ;)
The rate discrepancy is ok. Each class is averaged on its own and there is really possible to overshot parent's rate for a while. It is accounted to the child later.
But is seems to me that problem is at your www server. First, the HTB reports no backlog for the www connection - VERY unlikely ! It means that all data has been sent and no new arrived. The same is shown by tokens. They are positive which means that class is not throtling your data.
I'm not sure what is going here but absence of backlog is alarming. SFQ can hold up to 128 packets and server which is trying to send more data would create some backlog, wouldn't it ?
devik
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/lartc/
Quoted 5 lines · Martin Devera
But is seems to me that problem is at your www server. First, the HTB reports no backlog for the www connection - VERY unlikely ! It means that all data has been sent and no new arrived. The same is shown by tokens. They are positive which means that class is not throtling your data.
Okay, except if I drop the ceil of 1:11 to 50kbps, I can get about 150-200k/s from the webserver, which tells me that there is no bottleneck on the webserver or on the remote sites pulling the data down.
What else could account for the fact that there is no backlog?
Jason.
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/lartc/
Quoted 6 lines
Okay, except if I drop the ceil of 1:11 to 50kbps, I can get about 150-200k/s from the webserver, which tells me that there is no bottleneck on the webserver or on the remote sites pulling the data down.
What else could account for the fact that there is no backlog?
Only bug. :-| Just to eliminate potential bugs please try to add burst 20k to all your classes (1:1,10,11) and send me statistics again - the one during download is enough. It would be nice to do it like:
while true; do tc -s -d class show dev eth8; sleep 1;done > file
to get idea what is done during whole time. There is still "brute force" way: run htb with debug 3333333 and do tcpdump of packets. But it would be time consuming way :(
By the way, what speed is the machine ?
devik
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/lartc/
Quoted 4 lines · Martin Devera
burst 20k to all your classes (1:1,10,11) and send me statistics again - the one during download is enough. It would be nice to do it like:
while true; do tc -s -d class show dev eth8; sleep 1;done > file
Okay. I've accumulated about 30k worth of data, which I will attach to a separate email just to you (so as not to spam the list).
Quoted 1 line
By the way, what speed is the machine ?
It is a Celeron 366, with a load average of about 0.10.
Jason.
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/lartc/