HTB doesn't respect rate values
Subsystems (3)
Hi,
machine: AMD K6 200 MHz Linux distribution: Mandrake 8.1 kernel: compiled 2.4.21
applied this:
#define PSCHED_CLOCK_SOURCE PSCHED_CPU in file linux/include/net/pkt_sched.h bevore compiled the kernel (described on http://www.docum.org/stef.coene/qos/faq/cache/40.html)
bandwitch on eth0: 128kbit
The most simple configuration - 122kbit guaranted for WWW (sport 80) and 6kbit for the rest.
tc qdisc del root dev eth0
tc qdisc add dev eth0 root handle 1:0 htb default 3
tc class add dev eth0 parent 1:0 classid 1:1 htb rate 128kbit ceil 128kbit tc class add dev eth0 parent 1:1 classid 1:2 htb rate 122kbit ceil 128kbit quantum 30500
tc class add dev eth0 parent 1:1 classid 1:3 htb rate 6kbit ceil
128kbit quantum 1500
tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip sport 80 0xffff flowid 1:2
I make a test:
I send an email - it goes to default class 1:3. Then (during email is sent) I get e big file through www. What happen? WWW rate is 30-70kbit. So it doesn't keep his guaranted rate 122kbit. It lends his rate for SMTP. When SMTP stops sending his packets, WWW gets 100%.
In real, I've got the more complicated configuration (64 kbit for www, 40kbit for pop3, 6kbit for smtp 12kbit for others), but the problem
remains: www doesn't get hit minmal rate. Extremally it gets < 30kbit!
It works only "ceil". If I set ceil for a class (for example 1:3) - it is respected. 1:3 doesn't get more than set in ceil.
So, HTB doesn't work for me at all!
WHAT'S UP?
Could it be, that my rates are small? (6kbit). Does HTB work good for bandwitch <8kbit?
Did I something wrong?
Some sugestions?
thanks in advance
Sergiusz
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
1) 6kbit is really too small it should be at least 10 ..
2) it should workeven with 6k:
- look at stats (tc -s class show dev eth0) before and after the test - you are interested in drops. Also try it during the test to look whether queues are build up.
devik
Quoted 52 lines · Sergiusz Brzeziñski
Hi,
machine: AMD K6 200 MHz Linux distribution: Mandrake 8.1 kernel: compiled 2.4.21applied this:
#definePSCHED_CLOCK_SOURCEPSCHED_CPUin filelinux/include/net/pkt_sched.hbevore compiled the kernel (described on http://www.docum.org/stef.coene/qos/faq/cache/40.html)
bandwitch on eth0: 128kbitThe most simple configuration - 122kbit guaranted for WWW (sport 80) and 6kbit for the rest.
tc qdisc del root dev eth0
tc qdisc add dev eth0 root handle 1:0 htb default 3
tc class add dev eth0 parent 1:0 classid 1:1 htb rate 128kbit ceil 128kbit tc class add dev eth0 parent 1:1 classid 1:2 htb rate 122kbit ceil 128kbit quantum 30500
tc class add dev eth0 parent 1:1 classid 1:3 htb rate 6kbit ceil
128kbit quantum 1500tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip sport 80
0xffffflowid 1:2I make a test:
I send an email - it goes to default class 1:3. Then (during email is sent) I get e big file through www. What happen? WWW rate is 30-70kbit. So it doesn't keep his guaranted rate 122kbit. It lends his rate for SMTP. When SMTP stops sending his packets, WWW gets 100%.In real, I've got the more complicated configuration (64 kbit for www, 40kbit for pop3, 6kbit for smtp 12kbit for others), but the problem
remains: www doesn't get hit minmal rate. Extremally it gets < 30kbit!It works only "ceil". If I set ceil for a class (for example 1:3) - it is respected. 1:3 doesn't get more than set in ceil.
So, HTB doesn't work for me at all!
WHAT'S UP?
Could it be, that my rates are small? (6kbit). Does HTB work good for bandwitch <8kbit?
Did I something wrong?
Some sugestions?
Quoted 10 lines
thanks in advance
Sergiusz
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Thak you for your hints!
Quoted 1 line
1) 6kbit is really too small it should be at least 10 ..
I tried with 12, 20 and even with 30kbit for 1:3
I noticed, that it work for some seconds (or 1-2 minutes) but than the 1:3 class gets more then it should get :(.
Quoted 5 lines
2) it should workeven with 6k:
- look at stats (tc -s class show dev eth0) before and after the test - you are interested in drops. Also try it during the test to look whether queues are build up.
I made a test with settings:
---------------------------------
tc qdisc del root dev eth0
tc qdisc add dev eth0 root handle 1:0 htb default 3
tc class add dev eth0 parent 1:0 classid 1:1 htb rate 128kbit ceil 128kbit burst 20kbit
tc class add dev eth0 parent 1:1 classid 1:2 htb rate 98kbit ceil 128kbit quantum 4900 burst 20kbit
tc class add dev eth0 parent 1:1 classid 1:3 htb rate 30kbit ceil 128kbit quantum 1500
tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip sport 80 0xffff flowid 1:2
Bevore test: (reseted htb)
--------------------------------
# tc -s class show dev eth0
class htb 1:1 root rate 128Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 244140 ctokens: 168131
class htb 1:2 parent 1:1 prio 0 rate 98Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 318876 ctokens: 168131
class htb 1:3 parent 1:1 prio 0 rate 30Kbit ceil 128Kbit burst 1637b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 666503 ctokens: 168131
After test:
------------
class htb 1:1 root rate 128Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 5843869 bytes 4715 pkts (dropped 0, overlimits 0) rate 15427bps 12pps
lended: 1461 borrowed: 0 giants: 0
tokens: -21142 ctokens: -97151
class htb 1:2 parent 1:1 prio 0 rate 98Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 2735702 bytes 1811 pkts (dropped 0, overlimits 0) rate 6397bps 4pps
lended: 1802 borrowed: 9 giants: 0
tokens: 312898 ctokens: 163555
class htb 1:3 parent 1:1 prio 0 rate 30Kbit ceil 128Kbit burst 1637b cburst 1762b
Sent 3108167 bytes 2904 pkts (dropped 0, overlimits 0) rate 9488bps 8pps
lended: 1452 borrowed: 1452 giants: 0
tokens: -561135 ctokens: -97151
Description of the test:
------------------------
On the beginning it was everything OK, after 1 min, 1:2 lost his 98kbit. Than he got sometimes his 98kbit again and sometimes he got even 30kbit.
1. Can I do something more to find out what happen?
2. What does mean: "queues are build up" ?
Sergiusz
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Interestingly from what I see HTB didn't come into play. All drop and overlimits counters are zero. It seems that www server haven't managed to send more.
Please try to add pfifo with limit 10 under both classes. Because you are sending from the same computer, your TCP stack uses send queue management which counts packets in qdisc and backs off. It MIGHT cause problems ...
-------------------------------
Martin Devera aka devik
Linux kernel QoS/HTB maintainer
http://luxik.cdi.cz/~devik/
Quoted 86 lines · Sergiusz Brzeziñski
Thak you for your hints!
Quoted 1 line
1) 6kbit is really too small it should be at least 10 ..
I tried with 12, 20 and even with 30kbit for 1:3
I noticed, that it work for some seconds (or 1-2 minutes) but than the 1:3 class gets more then it should get :(.
Quoted 5 lines
2) it should workeven with 6k:
- look at stats (tc -s class show dev eth0) before and after the test - you are interested in drops. Also try it during the test to look whether queues are build up.I made a test with settings:
---------------------------------tc qdisc del root dev eth0
tc qdisc add dev eth0 root handle 1:0 htb default 3tc class add dev eth0 parent 1:0 classid 1:1 htb rate 128kbit ceil 128kbit burst 20kbit
tc class add dev eth0 parent 1:1 classid 1:2 htb rate 98kbit ceil 128kbit quantum 4900 burst 20kbit
tc class add dev eth0 parent 1:1 classid 1:3 htb rate 30kbit ceil 128kbit quantum 1500
tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip sport 80
0xffffflowid 1:2Bevore test: (reseted htb)
--------------------------------# tc -s class show dev eth0class htb 1:1 root rate 128Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 244140 ctokens: 168131class htb 1:2 parent 1:1 prio 0 rate 98Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 318876 ctokens: 168131class htb 1:3 parent 1:1 prio 0 rate 30Kbit ceil 128Kbit burst 1637b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 666503 ctokens: 168131After test:
------------
class htb 1:1 root rate 128Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 5843869 bytes 4715 pkts (dropped 0, overlimits 0) rate 15427bps 12pps
lended: 1461 borrowed: 0 giants: 0
tokens: -21142 ctokens: -97151class htb 1:2 parent 1:1 prio 0 rate 98Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 2735702 bytes 1811 pkts (dropped 0, overlimits 0) rate 6397bps 4pps
lended: 1802 borrowed: 9 giants: 0
tokens: 312898 ctokens: 163555class htb 1:3 parent 1:1 prio 0 rate 30Kbit ceil 128Kbit burst 1637b cburst 1762b
Sent 3108167 bytes 2904 pkts (dropped 0, overlimits 0) rate 9488bps 8pps
lended: 1452 borrowed: 1452 giants: 0
tokens: -561135 ctokens: -97151Description of the test:
------------------------
On the beginning it was everything OK, after 1 min, 1:2 lost his 98kbit. Than he got sometimes his 98kbit again and sometimes he got even 30kbit.1. Can I do something more to find out what happen?
2. What does mean: "queues are build up" ?
Quoted 10 lines
Sergiusz
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
That's it!
I did pfifo with limit 10 and HTB started to work. I noticed drops and rate was OK. Sometimes (for 10-40 seconds) but seldom it worked bad (1:2 got less, than it should) but there wasn't drops during this time. I tried this also with 12kbit at it was similar.
If I good understand, there is something (TCP stack or what) whitch works BEVORE htb and this makes some connections slower or faster, so HTB has later nothing to do.
The question for me is: how can I set it (the mechanism bevore HTB) to give HTB full control over the bandwitch? I don't wont to use pfifo (not everywhere). I would like to use sfq for some classes.
Should I set specific parameter somewhere or set something in network configuration, or maybe recompile the kernel with a specific option?
Sergiusz
U¿ytkownik devik napisa³:
Quoted 8 lines
Interestingly from what I see HTB didn't come into play. All drop and overlimits counters are zero. It seems that www server haven't managed to send more.
Please try to add pfifo with limit 10 under both classes. Because you are sending from the same computer, your TCP stack uses send queue management which counts packets in qdisc and backs off. It MIGHT cause problems ...
Quoted 96 lines
-------------------------------
Martin Devera aka devik
Linux kernel QoS/HTB maintainer
http://luxik.cdi.cz/~devik/Quoted 88 lines · Sergiusz Brzeziñski
Thak you for your hints!
Quoted 1 line
1) 6kbit is really too small it should be at least 10 ..
I tried with 12, 20 and even with 30kbit for 1:3
I noticed, that it work for some seconds (or 1-2 minutes) but than the 1:3 class gets more then it should get :(.
Quoted 5 lines
2) it should workeven with 6k:
- look at stats (tc -s class show dev eth0) before and after the test - you are interested in drops. Also try it during the test to look whether queues are build up.I made a test with settings:
---------------------------------tc qdisc del root dev eth0
tc qdisc add dev eth0 root handle 1:0 htb default 3tc class add dev eth0 parent 1:0 classid 1:1 htb rate 128kbit ceil 128kbit burst 20kbit
tc class add dev eth0 parent 1:1 classid 1:2 htb rate 98kbit ceil 128kbit quantum 4900 burst 20kbit
tc class add dev eth0 parent 1:1 classid 1:3 htb rate 30kbit ceil 128kbit quantum 1500
tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip sport 80
0xffffflowid 1:2Bevore test: (reseted htb)
--------------------------------# tc -s class show dev eth0class htb 1:1 root rate 128Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 244140 ctokens: 168131class htb 1:2 parent 1:1 prio 0 rate 98Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 318876 ctokens: 168131class htb 1:3 parent 1:1 prio 0 rate 30Kbit ceil 128Kbit burst 1637b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 666503 ctokens: 168131After test:
------------
class htb 1:1 root rate 128Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 5843869 bytes 4715 pkts (dropped 0, overlimits 0) rate 15427bps 12pps
lended: 1461 borrowed: 0 giants: 0
tokens: -21142 ctokens: -97151class htb 1:2 parent 1:1 prio 0 rate 98Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 2735702 bytes 1811 pkts (dropped 0, overlimits 0) rate 6397bps 4pps
lended: 1802 borrowed: 9 giants: 0
tokens: 312898 ctokens: 163555class htb 1:3 parent 1:1 prio 0 rate 30Kbit ceil 128Kbit burst 1637b cburst 1762b
Sent 3108167 bytes 2904 pkts (dropped 0, overlimits 0) rate 9488bps 8pps
lended: 1452 borrowed: 1452 giants: 0
tokens: -561135 ctokens: -97151Description of the test:
------------------------
On the beginning it was everything OK, after 1 min, 1:2 lost his 98kbit. Than he got sometimes his 98kbit again and sometimes he got even 30kbit.1. Can I do something more to find out what happen?
2. What does mean: "queues are build up" ?Sergiusz
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
At 01:05 a.m. 06/07/03 +0200, you wrote:
Hi, Sergiusz:
Quoted 5 lines
I make a test:
I send an email - it goes to default class 1:3. Then (during email is sent) I get e big file through www. What happen? WWW rate is 30-70kbit. So it doesn't keep his guaranted rate 122kbit. It lends his rate for SMTP. When SMTP stops sending his packets, WWW gets 100%.
If your HTB configuration is working well (HTB works really very well) you have to wait some time that TCP flows get stable. How long your test last? How heavy is your e-mail? Did you leave enough time to www to reclaim its rights? How strong is your www flow? Are you measuring average or instantaneous rates?
Best regards,
Leonardo Balliache
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Quoted 5 lines
I did pfifo with limit 10 and HTB started to work. I noticed drops and rate was OK. Sometimes (for 10-40 seconds) but seldom it worked bad (1:2 got less, than it should) but there wasn't drops during this time. I tried this also with 12kbit at it was similar.
Quoted 7 lines
If I good understand, there is something (TCP stack or what) whitch works BEVORE htb and this makes some connections slower or faster, so HTB has later nothing to do.
The question for me is: how can I set it (the mechanism bevore HTB) to give HTB full control over the bandwitch? I don't wont to use pfifo (not everywhere). I would like to use sfq for some classes.
you can try to play with /proc/sys/net/ipv4/tcp_{w,r,}mem. If wmem is smaller than space in qdisc (15kB for 10 pfifo, approx 200kB for SFQ) then TCP will back off before it fill the qdisc...
Also with sfq it should work - it is like pfifo with limit 128 - you might need to increase wmem.
Note that the problem occur only when you have qdisc on the same machine as sending app.
devik
Quoted 113 lines
Should I set specific parameter somewhere or set something in network configuration, or maybe recompile the kernel with a specific option?
Sergiusz
U¿ytkownik devik napisa³:
Quoted 104 lines
Interestingly from what I see HTB didn't come into play. All drop and overlimits counters are zero. It seems that www server haven't managed to send more.
Please try to add pfifo with limit 10 under both classes. Because you are sending from the same computer, your TCP stack uses send queue management which counts packets in qdisc and backs off. It MIGHT cause problems ...-------------------------------
Martin Devera aka devik
Linux kernel QoS/HTB maintainer
http://luxik.cdi.cz/~devik/Quoted 88 lines · Sergiusz Brzeziñski
Thak you for your hints!
Quoted 1 line
1) 6kbit is really too small it should be at least 10 ..
I tried with 12, 20 and even with 30kbit for 1:3
I noticed, that it work for some seconds (or 1-2 minutes) but than the 1:3 class gets more then it should get :(.
Quoted 5 lines
2) it should workeven with 6k:
- look at stats (tc -s class show dev eth0) before and after the test - you are interested in drops. Also try it during the test to look whether queues are build up.I made a test with settings:
---------------------------------tc qdisc del root dev eth0
tc qdisc add dev eth0 root handle 1:0 htb default 3tc class add dev eth0 parent 1:0 classid 1:1 htb rate 128kbit ceil 128kbit burst 20kbit
tc class add dev eth0 parent 1:1 classid 1:2 htb rate 98kbit ceil 128kbit quantum 4900 burst 20kbit
tc class add dev eth0 parent 1:1 classid 1:3 htb rate 30kbit ceil 128kbit quantum 1500
tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip sport 80
0xffffflowid 1:2Bevore test: (reseted htb)
--------------------------------# tc -s class show dev eth0class htb 1:1 root rate 128Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 244140 ctokens: 168131class htb 1:2 parent 1:1 prio 0 rate 98Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 318876 ctokens: 168131class htb 1:3 parent 1:1 prio 0 rate 30Kbit ceil 128Kbit burst 1637b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 666503 ctokens: 168131After test:
------------
class htb 1:1 root rate 128Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 5843869 bytes 4715 pkts (dropped 0, overlimits 0) rate 15427bps 12pps
lended: 1461 borrowed: 0 giants: 0
tokens: -21142 ctokens: -97151class htb 1:2 parent 1:1 prio 0 rate 98Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 2735702 bytes 1811 pkts (dropped 0, overlimits 0) rate 6397bps 4pps
lended: 1802 borrowed: 9 giants: 0
tokens: 312898 ctokens: 163555class htb 1:3 parent 1:1 prio 0 rate 30Kbit ceil 128Kbit burst 1637b cburst 1762b
Sent 3108167 bytes 2904 pkts (dropped 0, overlimits 0) rate 9488bps 8pps
lended: 1452 borrowed: 1452 giants: 0
tokens: -561135 ctokens: -97151Description of the test:
------------------------
On the beginning it was everything OK, after 1 min, 1:2 lost his 98kbit. Than he got sometimes his 98kbit again and sometimes he got even 30kbit.1. Can I do something more to find out what happen?
2. What does mean: "queues are build up" ?Sergiusz
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Quoted 9 lines · Leonardo Balliache
At 01:05 a.m. 06/07/03 +0200, you wrote:
Hi, Sergiusz:
Quoted 5 lines
I make a test:
I send an email - it goes to default class 1:3. Then (during email is sent) I get e big file through www. What happen? WWW rate is 30-70kbit. So it doesn't keep his guaranted rate 122kbit. It lends his rate for SMTP. When SMTP stops sending his packets, WWW gets 100%.
Quoted 5 lines
If your HTB configuration is working well (HTB works really very well) you have to wait some time that TCP flows get stable. How long your test last? How heavy is your e-mail? Did you leave enough time to www to reclaim its rights? How strong is your www flow? Are you measuring average or instantaneous rates?
Also, check out my tests on www.docum.org. Just to prove that it's working. You can even use my test scripts and adapt them to your needs.
Stef
Signature · 8 lines
[email protected] "Using Linux as bandwidth manager" http://www.docum.org/ #lartc @ irc.oftc.net _______________________________________________ LARTC mailing list / [email protected] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
U¿ytkownik devik napisa³:
Quoted 14 lines
Quoted 5 lines
I did pfifo with limit 10 and HTB started to work. I noticed drops and rate was OK. Sometimes (for 10-40 seconds) but seldom it worked bad (1:2 got less, than it should) but there wasn't drops during this time. I tried this also with 12kbit at it was similar.
Quoted 7 lines
If I good understand, there is something (TCP stack or what) whitch works BEVORE htb and this makes some connections slower or faster, so HTB has later nothing to do.
The question for me is: how can I set it (the mechanism bevore HTB) to give HTB full control over the bandwitch? I don't wont to use pfifo (not everywhere). I would like to use sfq for some classes.
Quoted 7 lines
you can try to play with
/proc/sys/net/ipv4/tcp_{w,r,}mem. If wmem is smaller than space in qdisc (15kB for 10 pfifo, approx 200kB for SFQ) then TCP will back off before it fill the qdisc...
Also with sfq it should work - it is like pfifo with limit 128 - you might need to increase wmem.
Note that the problem occur only when you have qdisc on the same machine as sending app.
- I did this:
# echo "4096 2048000 5120000" > /proc/sys/net/ipv4/tcp_wmem than I tried if in /proc/sys/net/ipv4/tcp_wmem there are really the new values (i set so high values because i wanted really be sure that the amount of memory will be OK :) - defaults where: 4096 16384 131072)
Well, it helped in 80%. Why only in 80? I repeated my test with SFQ and:
- it worked better than bevore, there where long time periods (15-20sec.) with right rate-values
- but class whitch should become 98kbit still got sometimes only 38kbit; it happen seldom and short but it was a fact!
- what was very strange: there were still no drops and no overlimits (!!!) in stats ("tc -s class show dev eth0"); in a test with "pfifo limit 10" I could see: when there were drops - the rate was OK, when there were no drops - the rate got lower than it should; with SFQ there were NO DROPS AT ALL so the question is: who (or what) really makes the whole work? It doesn't look like HTB-work.(have I right?)
Should I also do something with "/proc/sys/net/ipv4/tcp_mem"? Is the min value in tcp_wmem (4096) OK?
Do you have some more ideas?
I would make some experiments but I'm really not familiar with this theme. So the only thing I can do is to ask YOU or someone else from the group.
Sergiusz
Quoted 104 lines
Quoted 104 lines
U¿ytkownik devik napisa³:
Quoted 102 lines
Interestingly from what I see HTB didn't come into play. All drop and overlimits counters are zero. It seems that www server haven't managed to send more.
Please try to add pfifo with limit 10 under both classes. Because you are sending from the same computer, your TCP stack uses send queue management which counts packets in qdisc and backs off. It MIGHT cause problems ...-------------------------------
Martin Devera aka devik
Linux kernel QoS/HTB maintainer
http://luxik.cdi.cz/~devik/Quoted 85 lines · Sergiusz Brzeziñski
Thak you for your hints!
Quoted 1 line
1) 6kbit is really too small it should be at least 10 ..
I tried with 12, 20 and even with 30kbit for 1:3
I noticed, that it work for some seconds (or 1-2 minutes) but than the 1:3 class gets more then it should get :(.
Quoted 5 lines
2) it should workeven with 6k:
- look at stats (tc -s class show dev eth0) before and after the test - you are interested in drops. Also try it during the test to look whether queues are build up.I made a test with settings:
---------------------------------tc qdisc del root dev eth0
tc qdisc add dev eth0 root handle 1:0 htb default 3tc class add dev eth0 parent 1:0 classid 1:1 htb rate 128kbit ceil 128kbit burst 20kbit
tc class add dev eth0 parent 1:1 classid 1:2 htb rate 98kbit ceil 128kbit quantum 4900 burst 20kbit
tc class add dev eth0 parent 1:1 classid 1:3 htb rate 30kbit ceil 128kbit quantum 1500
tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip sport 80
0xffffflowid 1:2Bevore test: (reseted htb)
--------------------------------# tc -s class show dev eth0class htb 1:1 root rate 128Kbit ceil 128Kbit burst 2559b cburst 1762b Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 244140 ctokens: 168131class htb 1:2 parent 1:1 prio 0 rate 98Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 318876 ctokens: 168131class htb 1:3 parent 1:1 prio 0 rate 30Kbit ceil 128Kbit burst 1637b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 666503 ctokens: 168131After test:
------------
class htb 1:1 root rate 128Kbit ceil 128Kbit burst 2559b cburst 1762b Sent 5843869 bytes 4715 pkts (dropped 0, overlimits 0) rate 15427bps 12pps
lended: 1461 borrowed: 0 giants: 0
tokens: -21142 ctokens: -97151class htb 1:2 parent 1:1 prio 0 rate 98Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 2735702 bytes 1811 pkts (dropped 0, overlimits 0) rate 6397bps 4pps
lended: 1802 borrowed: 9 giants: 0
tokens: 312898 ctokens: 163555class htb 1:3 parent 1:1 prio 0 rate 30Kbit ceil 128Kbit burst 1637b cburst 1762b
Sent 3108167 bytes 2904 pkts (dropped 0, overlimits 0) rate 9488bps 8pps
lended: 1452 borrowed: 1452 giants: 0
tokens: -561135 ctokens: -97151Description of the test:
------------------------
On the beginning it was everything OK, after 1 min, 1:2 lost his 98kbit. Than he got sometimes his 98kbit again and sometimes he got even 30kbit.
Quoted 10 lines
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
1. Can I do something more to find out what happen?
2. What does mean: "queues are build up" ?Sergiusz
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Well. The right way for your case would be to limit single subqueue in SFQ. See line 24 of attached patch - and try patch itself.
devik
Quoted 46 lines · Sergiusz Brzeziński
Quoted 14 lines · Użytkownik devik
Quoted 5 lines
I did pfifo with limit 10 and HTB started to work. I noticed drops and rate was OK. Sometimes (for 10-40 seconds) but seldom it worked bad (1:2 got less, than it should) but there wasn't drops during this time. I tried this also with 12kbit at it was similar.
Quoted 7 lines
If I good understand, there is something (TCP stack or what) whitch works BEVORE htb and this makes some connections slower or faster, so HTB has later nothing to do.
The question for me is: how can I set it (the mechanism bevore HTB) to give HTB full control over the bandwitch? I don't wont to use pfifo (not everywhere). I would like to use sfq for some classes.
Quoted 7 lines
you can try to play with
/proc/sys/net/ipv4/tcp_{w,r,}mem. If wmem is smaller than space in qdisc (15kB for 10 pfifo, approx 200kB for SFQ) then TCP will back off before it fill the qdisc...
Also with sfq it should work - it is like pfifo with limit 128 - you might need to increase wmem.
Note that the problem occur only when you have qdisc on the same machine as sending app.- I did this:
# echo "4096 2048000 5120000" > /proc/sys/net/ipv4/tcp_wmem than I tried if in /proc/sys/net/ipv4/tcp_wmem there are really the new values (i set so high values because i wanted really be sure that the amount of memory will be OK :) - defaults where: 4096 16384 131072)Well, it helped in 80%. Why only in 80? I repeated my test with SFQ and:
- it worked better than bevore, there where long time periods (15-20sec.) with right rate-values
- but class whitch should become 98kbit still got sometimes only 38kbit; it happen seldom and short but it was a fact!
- what was very strange: there were still no drops and no overlimits (!!!) in stats ("tc -s class show dev eth0"); in a test with "pfifo limit 10" I could see: when there were drops - the rate was OK, when there were no drops - the rate got lower than it should; with SFQ there were NO DROPS AT ALL so the question is: who (or what) really makes the whole work? It doesn't look like HTB-work.(have I right?)Should I also do something with "
/proc/sys/net/ipv4/tcp_mem"? Is the min value intcp_wmem(4096) OK?
Do you have some more ideas?
Quoted 124 lines
I would make some experiments but I'm really not familiar with this theme. So the only thing I can do is to ask YOU or someone else from the group.
Sergiusz
Quoted 114 lines
Quoted 112 lines
Quoted 107 lines · Użytkownik devik
Interestingly from what I see HTB didn't come into play. All drop and overlimits counters are zero. It seems that www server haven't managed to send more.
Please try to add pfifo with limit 10 under both classes. Because you are sending from the same computer, your TCP stack uses send queue management which counts packets in qdisc and backs off. It MIGHT cause problems ...-------------------------------
Martin Devera aka devik
Linux kernel QoS/HTB maintainer
http://luxik.cdi.cz/~devik/Quoted 90 lines · Sergiusz Brzeziński
Thak you for your hints!
Quoted 1 line
1) 6kbit is really too small it should be at least 10 ..
I tried with 12, 20 and even with 30kbit for 1:3
I noticed, that it work for some seconds (or 1-2 minutes) but than the 1:3 class gets more then it should get :(.
Quoted 5 lines
2) it should workeven with 6k:
- look at stats (tc -s class show dev eth0) before and after the test - you are interested in drops. Also try it during the test to look whether queues are build up.I made a test with settings:
---------------------------------tc qdisc del root dev eth0
tc qdisc add dev eth0 root handle 1:0 htb default 3tc class add dev eth0 parent 1:0 classid 1:1 htb rate 128kbit ceil 128kbit burst 20kbit
tc class add dev eth0 parent 1:1 classid 1:2 htb rate 98kbit ceil 128kbit quantum 4900 burst 20kbit
tc class add dev eth0 parent 1:1 classid 1:3 htb rate 30kbit ceil 128kbit quantum 1500
tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip sport 80
0xffffflowid 1:2Bevore test: (reseted htb)
--------------------------------# tc -s class show dev eth0class htb 1:1 root rate 128Kbit ceil 128Kbit burst 2559b cburst 1762b Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 244140 ctokens: 168131class htb 1:2 parent 1:1 prio 0 rate 98Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 318876 ctokens: 168131class htb 1:3 parent 1:1 prio 0 rate 30Kbit ceil 128Kbit burst 1637b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 666503 ctokens: 168131After test:
------------
class htb 1:1 root rate 128Kbit ceil 128Kbit burst 2559b cburst 1762b Sent 5843869 bytes 4715 pkts (dropped 0, overlimits 0) rate 15427bps 12pps
lended: 1461 borrowed: 0 giants: 0
tokens: -21142 ctokens: -97151class htb 1:2 parent 1:1 prio 0 rate 98Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 2735702 bytes 1811 pkts (dropped 0, overlimits 0) rate 6397bps 4pps
lended: 1802 borrowed: 9 giants: 0
tokens: 312898 ctokens: 163555class htb 1:3 parent 1:1 prio 0 rate 30Kbit ceil 128Kbit burst 1637b cburst 1762b
Sent 3108167 bytes 2904 pkts (dropped 0, overlimits 0) rate 9488bps 8pps
lended: 1452 borrowed: 1452 giants: 0
tokens: -561135 ctokens: -97151Description of the test:
------------------------
On the beginning it was everything OK, after 1 min, 1:2 lost his 98kbit. Than he got sometimes his 98kbit again and sometimes he got even 30kbit.1. Can I do something more to find out what happen?
2. What does mean: "queues are build up" ?Sergiusz
net/sched/sch_sfq.c renamed from usr/src/linux-2.2.16/net/sched/sch_sfq.c +12 -3
@@ -111,6 +111,8 @@int perturbation;sfq_index tail;/* Index of current slot in round */ sfq_index max_depth;/* Maximal depth */ +short flows;/* Num of flows */ +short max_flows;/* Max num of flows */ sfq_index ht[SFQ_HASH_DIVISOR ];/* Hash table */ sfq_index next[SFQ_DEPTH ];/* Active slots link */
@@ -265,6 +267,7 @@__skb_queue_tail (&q->qs[x], skb);sfq_inc (q, x);if (q->qs[x].qlen ==1 ) {/* The flow is new */ +if (++q->flows > q->max_flows) q->max_flows = q->flows;if (q->tail ==SFQ_DEPTH ) {/* It is the first flow */ q->tail = x; q->next[x] = x;
@@ -276,11 +279,13 @@ } }if (++sch->q.qlen <SFQ_DEPTH -1 ) { +// arbitrary hack to limit maximal flow's backlog +if (q->qs[x].qlen >5 ) goto drop; sch->stats.bytes += skb->len; sch->stats.packets++;return 1 ; } - +drop:sfq_drop (sch);return 0 ; }
@@ -300,6 +305,7 @@__skb_queue_head (&q->qs[x], skb);sfq_inc (q, x);if (q->qs[x].qlen ==1 ) {/* The flow is new */ +if (++q->flows > q->max_flows) q->max_flows = q->flows;if (q->tail ==SFQ_DEPTH ) {/* It is the first flow */ q->tail = x; q->next[x] = x;
@@ -341,6 +347,7 @@/* Is the slot empty? */ if (q->qs[a].qlen ==0 ) { + q->flows--; a = q->next[a];if (a == old_a) { q->tail =SFQ_DEPTH ;
@@ -417,6 +424,8 @@ q->dep[i+SFQ_DEPTH ].prev = i+SFQ_DEPTH ; } q->max_depth =0 ; + q->max_flows =0 ; + q->flows =0 ; q->tail =SFQ_DEPTH ;if (opt ==NULL ) { q->quantum =psched_mtu (sch->dev);
@@ -450,8 +459,8 @@ opt.perturb_period = q->perturb_period/HZ ; opt.limit =SFQ_DEPTH ; - opt.divisor =SFQ_HASH_DIVISOR ; - opt.flows =SFQ_DEPTH ; + opt.divisor = q->max_flows; + opt.flows = q->flows;RTA_PUT (skb,TCA_OPTIONS ,sizeof (opt), &opt);
U¿ytkownik devik napisa³:
Quoted 4 lines
Well. The right way for your case would be to limit single subqueue in SFQ. See line 24 of attached patch - and try patch itself.
devik
hmm.. with pfifo_fast is thesame problem - no drops... and unstable. It doesn't look like a SFQ-specific problem. With fifo (without setting limit) there were only 2 drops. Only "fifo limit 10" generates drops (but it has also periods with no drops and with bad rates). Everything of course with tcp_wmem = "4096 2048000 5120000"
Sergiusz
Quoted 166 lines · Sergiusz Brzeziñski
Quoted 164 lines
U¿ytkownik devik napisa³:
Quoted 14 lines
Quoted 5 lines
I did pfifo with limit 10 and HTB started to work. I noticed drops and rate was OK. Sometimes (for 10-40 seconds) but seldom it worked bad (1:2 got less, than it should) but there wasn't drops during this time. I tried this also with 12kbit at it was similar.
Quoted 7 lines
If I good understand, there is something (TCP stack or what) whitch works BEVORE htb and this makes some connections slower or faster, so HTB has later nothing to do.
The question for me is: how can I set it (the mechanism bevore HTB) to give HTB full control over the bandwitch? I don't wont to use pfifo (not everywhere). I would like to use sfq for some classes.
Quoted 7 lines
you can try to play with
/proc/sys/net/ipv4/tcp_{w,r,}mem. If wmem is smaller than space in qdisc (15kB for 10 pfifo, approx 200kB for SFQ) then TCP will back off before it fill the qdisc...
Also with sfq it should work - it is like pfifo with limit 128 - you might need to increase wmem.
Note that the problem occur only when you have qdisc on the same machine as sending app.- I did this:
# echo "4096 2048000 5120000" > /proc/sys/net/ipv4/tcp_wmem than I tried if in /proc/sys/net/ipv4/tcp_wmem there are really the new values (i set so high values because i wanted really be sure that the amount of memory will be OK :) - defaults where: 4096 16384 131072)Well, it helped in 80%. Why only in 80? I repeated my test with SFQ and:
- it worked better than bevore, there where long time periods (15-20sec.) with right rate-values
- but class whitch should become 98kbit still got sometimes only 38kbit; it happen seldom and short but it was a fact!
- what was very strange: there were still no drops and no overlimits (!!!) in stats ("tc -s class show dev eth0"); in a test with "pfifo limit 10" I could see: when there were drops - the rate was OK, when there were no drops - the rate got lower than it should; with SFQ there were NO DROPS AT ALL so the question is: who (or what) really makes the whole work? It doesn't look like HTB-work.(have I right?)Should I also do something with "
/proc/sys/net/ipv4/tcp_mem"? Is the min value intcp_wmem(4096) OK?
Do you have some more ideas?I would make some experiments but I'm really not familiar with this theme. So the only thing I can do is to ask YOU or someone else from the group.
Sergiusz
Quoted 108 lines
Quoted 108 lines
U¿ytkownik devik napisa³:
Quoted 105 lines
Interestingly from what I see HTB didn't come into play. All drop and overlimits counters are zero. It seems that www server haven't managed to send more.
Please try to add pfifo with limit 10 under both classes. Because you are sending from the same computer, your TCP stack uses send queue management which counts packets in qdisc and backs off. It MIGHT cause problems ...-------------------------------
Martin Devera aka devik
Linux kernel QoS/HTB maintainer
http://luxik.cdi.cz/~devik/Quoted 87 lines · Sergiusz Brzeziñski
Thak you for your hints!
Quoted 1 line
1) 6kbit is really too small it should be at least 10 ..
I tried with 12, 20 and even with 30kbit for 1:3
I noticed, that it work for some seconds (or 1-2 minutes) but than the 1:3 class gets more then it should get :(.
Quoted 5 lines
2) it should workeven with 6k:
- look at stats (tc -s class show dev eth0) before and after the test - you are interested in drops. Also try it during the test to look whether queues are build up.I made a test with settings:
---------------------------------tc qdisc del root dev eth0
tc qdisc add dev eth0 root handle 1:0 htb default 3tc class add dev eth0 parent 1:0 classid 1:1 htb rate 128kbit ceil 128kbit burst 20kbit
tc class add dev eth0 parent 1:1 classid 1:2 htb rate 98kbit ceil 128kbit quantum 4900 burst 20kbit
tc class add dev eth0 parent 1:1 classid 1:3 htb rate 30kbit ceil 128kbit quantum 1500
tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip sport 80
0xffffflowid 1:2Bevore test: (reseted htb)
--------------------------------# tc -s class show dev eth0class htb 1:1 root rate 128Kbit ceil 128Kbit burst 2559b cburst 1762b Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 244140 ctokens: 168131class htb 1:2 parent 1:1 prio 0 rate 98Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 318876 ctokens: 168131class htb 1:3 parent 1:1 prio 0 rate 30Kbit ceil 128Kbit burst 1637b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 666503 ctokens: 168131After test:
------------
class htb 1:1 root rate 128Kbit ceil 128Kbit burst 2559b cburst 1762b Sent 5843869 bytes 4715 pkts (dropped 0, overlimits 0) rate 15427bps 12pps
lended: 1461 borrowed: 0 giants: 0
tokens: -21142 ctokens: -97151class htb 1:2 parent 1:1 prio 0 rate 98Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 2735702 bytes 1811 pkts (dropped 0, overlimits 0) rate 6397bps 4pps
lended: 1802 borrowed: 9 giants: 0
tokens: 312898 ctokens: 163555class htb 1:3 parent 1:1 prio 0 rate 30Kbit ceil 128Kbit burst 1637b cburst 1762b
Sent 3108167 bytes 2904 pkts (dropped 0, overlimits 0) rate 9488bps 8pps
lended: 1452 borrowed: 1452 giants: 0
tokens: -561135 ctokens: -97151Description of the test:
------------------------
On the beginning it was everything OK, after 1 min, 1:2 lost his 98kbit. Than he got sometimes his 98kbit again and sometimes he got even 30kbit.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
1. Can I do something more to find out what happen?
2. What does mean: "queues are build up" ?Sergiusz
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Original Message:
-----------------
From: Sergiusz Brzeziñski [email protected]
Quoted 4 lines
Well, it helped in 80%. Why only in 80? I repeated my test with SFQ and:
- it worked better than bevore, there where long time periods (15-20sec.) with right rate-values
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
I didnt follow this thread from beginning, sorry if i didnt solve the problem. From your last post to this list i assumed you have set correct rate values to each class, but the distribution is still unfair. Am i right ?
At first i applied HTB for my bandwidth manager i came into the same problem. When my users at one class start to open several tcp streams, perhaps you can see such as Kazaa, DAP, FlashGET. These can consume high bandwidth. Also they opened several tcp streams to the same destination host. The fact is those class get high rate while the other class is suffer.
I applied HTB with SFQ as leaf qdisc. I played with the quantum values and it started to work well.
Try it..
Regards,
Rio Martin.
--------------------------------------------------------------------
mail2web - Check your email from the web at
http://mail2web.com/ .
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Quoted 7 lines
U¿ytkownik devik napisa³:
Quoted 4 lines
Well. The right way for your case would be to limit single subqueue in SFQ. See line 24 of attached patch - and try patch itself.
devik
Quoted 5 lines
hmm.. with
pfifo_fastis thesame problem - no drops... and unstable. It doesn't look like a SFQ-specific problem. With fifo (without setting limit) there were only 2 drops. Only "fifo limit 10" generates drops (but it has also periods with no drops and with bad rates). Everything of course withtcp_wmem= "4096 2048000 5120000"
because pfifo_fast uses default queue size for eth = 100. It is almost the same as SFQ uses. Did you try my patch ?
Regarding wmem, try also increasing /proc/sys/net/core/wmem_max. Maybe it is blocking TCP's settings.
devik
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Quoted 9 lines · Sergiusz Brzezi?ski
machine: AMD K6 200 MHz Linux distribution: Mandrake 8.1 kernel: compiled 2.4.21applied this:
#definePSCHED_CLOCK_SOURCEPSCHED_CPUin filelinux/include/net/pkt_sched.hbevore compiled the kernel (described on http://www.docum.org/stef.coene/qos/faq/cache/40.html)
bandwitch on eth0: 128kbit
Could this have something to do with this known bug in linux kernels 2.5.x ?
http://bugme.osdl.org/show_bug.cgi?ide7
Regards.
Signature · 5 lines
Jose Luis Domingo Lopez Linux Registered User #189436 Debian Linux Sid (Linux 2.5.73) _______________________________________________ LARTC mailing list / [email protected] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/