HTB speed
Dear all,
I'm running htb on 2.4.24, following setup:
running on a bridge, ebtables patch, firewall marks with iptables, and an SFQ for every HTB.
Everything works fine, so I don't go into details, there is just one thing which suprises me:
When I configure HTB to shape to about 900 kbit/s, the resulting speed is about 400 kbit/s.
For higher speeds it's more accurate (but stil too slow), for lower speeds it gets worse.
Does anyone here have any idea why? I've just added a "correction factor" to my scripts, so it's kind of allright, but I would like to know if others experience the same thing.
Jeroen.
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Hi,
just putting the answer to my own question here, for those who have the same problem, and read the mailing list archive.
The timing of the P4 based on "jiffies" is hopeless, it's different for every processor, and can be a wrong by a factor 3.
If the tsc (time stamp counter) is used, the htb works fine, the error in speed is now only about 1 %.
It's set by:
in pkt_sched.h:
#define PSCHED_CLOCK_SOURCE PSCHED_CPU
that's all, I wonder why it's not default to do this, or maybe it's an idea to make the packet scheduler detect the presence of tsc when the module is loaded.
Cheers,
Jeroen.
Quoted 10 lines · On Fri, 12 Mar 2004 10:58:40 +0100 Jeroen Vriesman <[email protected]> wrote:
Dear all,
I'm running htb on 2.4.24, following setup:
running on a bridge, ebtables patch, firewall marks with iptables, and an SFQ for every HTB.
Everything works fine, so I don't go into details, there is just one thing which suprises me:
When I configure HTB to shape to about 900 kbit/s, the resulting speed is about 400 kbit/s.
Quoted 9 lines
For higher speeds it's more accurate (but stil too slow), for lower speeds it gets worse.
Does anyone here have any idea why? I've just added a "correction factor" to my scripts, so it's kind of allright, but I would like to know if others experience the same thing.
Jeroen.
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Quoted 6 lines
in
pkt_sched.h:#define
PSCHED_CLOCK_SOURCEPSCHED_CPUthat's all, I wonder why it's not default to do this, or maybe it's an idea to make the packet scheduler detect the presence of tsc when the module is loaded.
Hi,
I think not all processors accept this #define PSCHED_CLOCK_SOURCE PSCHED_CPU. At least, I couldn't do it in my router.
I am not using HTB but TBF, but I think the timing problems are similar. As to tune TBF with higher precision, I had to manually change Hz value from 100 to 1000, in param.h, and actually I am getting better results.
My question is, does anybody tried to do this in an production environment? For instance, in a router that integrates traffic control, NAT, firewall, services gateway, etc., can this change in HZ value harm in any way the rest of the system?
Thanks in advance,
Joana Urbano
Hi,
to use the TSC, the processor has to have a tsc, you can see that in /proc/cpuinfo, for as far as I know, every P4 has it (but I'm not sure), it's in the "flags" of cpuinfo:
On my P4-1800:
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm
The "loop calibrated" jiffies stuff dates back to 386, it's used everywhere in the kernel timing, and it's very inaccurate, I don't know if 2.6 still uses it everwhere, what kind of processor do you have?
I've never tried to change the Hz value, does changing it in param.h really changes the frequency of the clock ticks? If so, why is the default only 100Hz these days? doesn't make sense to me.
I do use the PSCHED_CPU in a production environement at the moment, works fine.
Cheers,
Jeroen.
Quoted 18 lines · On Mon, 15 Mar 2004 11:37:16 +0000 Maria Joana Urbano <[email protected]> wrote:
Quoted 8 lines
in
pkt_sched.h:#define
PSCHED_CLOCK_SOURCEPSCHED_CPUthat's all, I wonder why it's not default to do this, or maybe it's an idea to make the packet scheduler detect the presence of tsc when the module is loaded.
Hi,
I think not all processors accept this #define
PSCHED_CLOCK_SOURCEPSCHED_CPU. At least, I couldn't do it in my router.I am not using HTB but TBF, but I think the timing problems are similar. As to tune TBF with higher precision, I had to manually change Hz value from 100 to 1000, in
param.h, and actually I am getting better results.
Quoted 8 lines
My question is, does anybody tried to do this in an production environment? For instance, in a router that integrates traffic control, NAT, firewall, services gateway, etc., can this change in HZ value harm in any way the rest of the system?
Thanks in advance,
Joana Urbano
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Hi,
Indeed, changing the timer to 1000Hz is possible, it turned out that I have a machine here running with a 1000Hz timer ticker. (I've installed a "realtime" kernel on it for audio recording).
About your previous question, I've noticed that the system with 1000Hz ticker (which has been running for a few months) is UNSTABLE.
might be more to it than just the 1000Hz, but I did notice that some "realtime" applications running in userspace where able to crash the system completely, happened 2 times in 3 months.
Cheers,
Jeroen.
Quoted 18 lines · On Mon, 15 Mar 2004 11:37:16 +0000 Maria Joana Urbano <[email protected]> wrote:
Quoted 8 lines
in
pkt_sched.h:#define
PSCHED_CLOCK_SOURCEPSCHED_CPUthat's all, I wonder why it's not default to do this, or maybe it's an idea to make the packet scheduler detect the presence of tsc when the module is loaded.
Hi,
I think not all processors accept this #define
PSCHED_CLOCK_SOURCEPSCHED_CPU. At least, I couldn't do it in my router.I am not using HTB but TBF, but I think the timing problems are similar. As to tune TBF with higher precision, I had to manually change Hz value from 100 to 1000, in
param.h, and actually I am getting better results.
Quoted 8 lines
My question is, does anybody tried to do this in an production environment? For instance, in a router that integrates traffic control, NAT, firewall, services gateway, etc., can this change in HZ value harm in any way the rest of the system?
Thanks in advance,
Joana Urbano
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Quoted 13 lines · Jeroen Vriesman
Hi,
just putting the answer to my own question here, for those who have the same problem, and read the mailing list archive.
The timing of the P4 based on "jiffies" is hopeless, it's different for every processor, and can be a wrong by a factor 3.
If the tsc (time stamp counter) is used, the htb works fine, the error in speed is now only about 1 %.
It's set by:
Quoted 7 lines
in
pkt_sched.h:#define
PSCHED_CLOCK_SOURCEPSCHED_CPUthat's all, I wonder why it's not default to do this, or maybe it's an idea to make the packet scheduler detect the presence of tsc when the module is loaded.
Hi,
Which pkt_sched.h are you refering to ?/usr/src/linux/include/linux/pkt_sched.h or/usr/src/linux/include/net/pkt_sched.h ?
And after changing it what did you do ? Recompile the kernel ?
Or recompile tc ?
I too see the same problems with htb (very poor accuracy of speed, significantly too slow, also very jerky) while cbq is very accurate and smooth. (But lacks some features I need)
Regards,
Simon
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
You need to recompile the kernel after altering this value in linux/include/net/pkt_sched.h. Also remember that if using SFQ on leaf qdisc, then the queue length may cause delay problems if it's too long (default is 128). Changing this to 32 for rates below 100kb/s, I have found to help things considerably. This change needs to be done in linux/net/sched/sch_sfq.c. which also needs a kernel recompilation.
----- Original Message -----
From: "Simon Byrnand" <[email protected]>
To: "Jeroen Vriesman" <[email protected]>; <[email protected]>
Sent: Thursday, March 25, 2004 4:36 PM
Subject: Re: [LARTC] HTB speed
Quoted 36 lines
Quoted 13 lines · Jeroen Vriesman
Hi,
just putting the answer to my own question here, for those who have the same problem, and read the mailing list archive.
The timing of the P4 based on "jiffies" is hopeless, it's different for every processor, and can be a wrong by a factor 3.
If the tsc (time stamp counter) is used, the htb works fine, the error in speed is now only about 1 %.
It's set by:
Quoted 7 lines
in
pkt_sched.h:#define
PSCHED_CLOCK_SOURCEPSCHED_CPUthat's all, I wonder why it's not default to do this, or maybe it's an idea to make the packet scheduler detect the presence of tsc when the module is loaded.
Hi,
Which
pkt_sched.hare you refering to ?/usr/src/linux/include/linux/pkt_sched.hor/usr/src/linux/include/net/pkt_sched.h?And after changing it what did you do ? Recompile the kernel ?
Or recompile tc ?
I too see the same problems with htb (very poor accuracy of speed, significantly too slow, also very jerky) while cbq is very accurate and smooth. (But lacks some features I need)
Quoted 6 lines
Regards,
Simon_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Would the problem with queue lengths also exist if CONFIG_HZ00 in .config? (it certainly improves latency)
I've configured rates from 10kbit/s...2Mbit/s on one machine, so changing the kernel code "for low rates" would probably effect the high rates too.
Or, in other words, what is exactly the problem with an SFQ length of 128 for low rates?
Cheers,
Jeroen.
Quoted 53 lines · On Thu, 25 Mar 2004 17:17:44 +1100 "Andrew Hall" <[email protected]> wrote:
You need to recompile the kernel after altering this value in
linux/include/net/pkt_sched.h. Also remember that if using SFQ on leaf qdisc, then the queue length may cause delay problems if it's too long (default is 128). Changing this to 32 for rates below 100kb/s, I have found to help things considerably. This change needs to be done inlinux/net/sched/sch_sfq.c. which also needs a kernel recompilation.----- Original Message -----
From: "Simon Byrnand" <[email protected]>
To: "Jeroen Vriesman" <[email protected]>; <[email protected]>
Sent: Thursday, March 25, 2004 4:36 PM
Subject: Re: [LARTC] HTB speedQuoted 39 lines
Quoted 13 lines · Jeroen Vriesman
Hi,
just putting the answer to my own question here, for those who have the same problem, and read the mailing list archive.
The timing of the P4 based on "jiffies" is hopeless, it's different for every processor, and can be a wrong by a factor 3.
If the tsc (time stamp counter) is used, the htb works fine, the error in speed is now only about 1 %.
It's set by:
Quoted 7 lines
in
pkt_sched.h:#define
PSCHED_CLOCK_SOURCEPSCHED_CPUthat's all, I wonder why it's not default to do this, or maybe it's an idea to make the packet scheduler detect the presence of tsc when the module is loaded.
Hi,
Which
pkt_sched.hare you refering to ?/usr/src/linux/include/linux/pkt_sched.hor/usr/src/linux/include/net/pkt_sched.h?And after changing it what did you do ? Recompile the kernel ?
Or recompile tc ?
I too see the same problems with htb (very poor accuracy of speed, significantly too slow, also very jerky) while cbq is very accurate and smooth. (But lacks some features I need)
Regards,
Simon
Quoted 8 lines
Quoted 3 lines
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Thanks, Andrew, it's intresting...
-> (default is 128). Changing this to 32 for rates below 100kb/s, I -> have found
you mean that try change "linux/net/sched/sch_sfq.c" this: ?
#define SFQ_DEPTH 128
#define SFQ_HASH_DIVISOR 1024
for this.
#define SFQ_DEPTH 32
#define SFQ_HASH_DIVISOR 1024
I 've 2 questions about it
1) what is I use various rates (from 8k to 256k)
2) how I must proceed in Esfq ??
(in order to work with those htb classes)
Usage: ... esfq [ perturb SECS ] [ quantum BYTES ] [ depth FLOWS ]
[ divisor HASHBITS ] [ limit PKTS ] [ hash HASHTYPE]
Where:
HASHTYPE := { classic | src | dst }
Thankyou!
andres
->
-> You need to recompile the kernel after altering this value in -> linux/include/net/pkt_sched.h. Also remember that if using SFQ on leaf -> qdisc, then the queue length may cause delay problems if it's too long -> (default is 128). Changing this to 32 for rates below 100kb/s, I -> have found
-> to help things considerably. This change needs to be done in -> linux/net/sched/sch_sfq.c. which also needs a kernel recompilation. ->
-> ----- Original Message -----
-> From: "Simon Byrnand" <[email protected]>
-> To: "Jeroen Vriesman" <[email protected]>; <[email protected]> -> Sent: Thursday, March 25, 2004 4:36 PM
-> Subject: Re: [LARTC] HTB speed
->
->
-> > At 11:14 14/03/2004, Jeroen Vriesman wrote:
-> > >Hi,
-> > >
-> > >just putting the answer to my own question here, for those -> who have the
-> > >same problem, and read the mailing list archive.
-> > >
-> > >The timing of the P4 based on "jiffies" is hopeless, it's -> different for
-> > >every processor, and can be a wrong by a factor 3.
-> > >
-> > >If the tsc (time stamp counter) is used, the htb works fine, -> the error in
-> > >speed is now only about 1 %.
-> > >
-> > >It's set by:
-> > >
-> > >in pkt_sched.h:
-> > >
-> > >#define PSCHED_CLOCK_SOURCE PSCHED_CPU
-> > >
-> > >that's all, I wonder why it's not default to do this, or maybe it's an -> > >idea to make the packet scheduler detect the presence of tsc when the -> > >module is loaded.
-> >
-> > Hi,
-> >
-> > Which pkt_sched.h are you refering to ?
-> > /usr/src/linux/include/linux/pkt_sched.h or
-> > /usr/src/linux/include/net/pkt_sched.h ?
-> >
-> > And after changing it what did you do ? Recompile the kernel ? -> >
-> > Or recompile tc ?
-> >
-> > I too see the same problems with htb (very poor accuracy of speed, -> > significantly too slow, also very jerky) while cbq is very accurate and -> > smooth. (But lacks some features I need)
-> >
-> > Regards,
-> > Simon
-> >
-> > _______________________________________________
-> > LARTC mailing list / [email protected]
-> > http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
->
-> _______________________________________________
-> LARTC mailing list / [email protected]
-> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
->
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Quoted 6 lines · Andrew Hall
You need to recompile the kernel after altering this value in
linux/include/net/pkt_sched.h. Also remember that if using SFQ on leaf qdisc, then the queue length may cause delay problems if it's too long (default is 128). Changing this to 32 for rates below 100kb/s, I have found to help things considerably. This change needs to be done inlinux/net/sched/sch_sfq.c. which also needs a kernel recompilation.
Hmm,
When I use sfq with cbq at speeds like 256Kbit there is no problem at all, runs very sweetly, but with HTB and sfq, it is very jerky and poor. I'll try the change in pkt_sched.h first and see how I go...
Regards,
Simon
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Quoted 8 lines · Simon Byrnand
Quoted 6 lines · Andrew Hall
You need to recompile the kernel after altering this value in
linux/include/net/pkt_sched.h. Also remember that if using SFQ on leaf qdisc, then the queue length may cause delay problems if it's too long (default is 128). Changing this to 32 for rates below 100kb/s, I have found to help things considerably. This change needs to be done inlinux/net/sched/sch_sfq.c. which also needs a kernel recompilation.
Quoted 5 lines
Hmm,
When I use sfq with cbq at speeds like 256Kbit there is no problem at all, runs very sweetly, but with HTB and sfq, it is very jerky and poor. I'll try the change in
pkt_sched.hfirst and see how I go...
Ok, I tried the change in pkt_sched.h and didn't notice any difference whatsoever. Any other ideas ? cbq is still fine but htb for the same speed is very jerky and the speed fluctuates around 60-80% of the wanted speed, while cbq gives a steady 99% of the wanted speed...
Regards,
Simon
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Quoted 20 lines · Simon Byrnand
Quoted 11 lines · Simon Byrnand
Quoted 7 lines · Andrew Hall
You need to recompile the kernel after altering this value in
linux/include/net/pkt_sched.h. Also remember that if using SFQ on leaf qdisc, then the queue length may cause delay problems if it's too long (default is 128). Changing this to 32 for rates below 100kb/s, I have found
to help things considerably. This change needs to be done inlinux/net/sched/sch_sfq.c. which also needs a kernel recompilation.Quoted 5 lines
Hmm,
When I use sfq with cbq at speeds like 256Kbit there is no problem at all, runs very sweetly, but with HTB and sfq, it is very jerky and poor. I'll try the change in
pkt_sched.hfirst and see how I go...
Quoted 4 lines
Ok, I tried the change in
pkt_sched.hand didn't notice any difference whatsoever. Any other ideas ? cbq is still fine but htb for the same speed is very jerky and the speed fluctuates around 60-80% of the wanted speed, while cbq gives a steady 99% of the wanted speed...
You can make HTB more accurate by setting HTB_HYSTERESIS to 0 in net/sched/sch_htb.c.
To save time - if you built HTB as a module, you can probably (well it worked for me) get away with editing htb.c and do
make SUBDIRS=net/sched modules
and replacing /lib/modules/[kversion]/kernel/net/sched/htb.o with the new htb.o from your source tree.
If you are doing it live stop shaping and check with lsmod that modprobe -r gets rid (do it again if it's still there) of the old htb.o and reload shaping scripts.
Make sure the quantum is your mtu, set 0 burst for bulk classes and don't set perturb too low on sfq (I use 20 as it causes packet reordering).
Are you shaping egress or ingress and how are you measuring speed?
Andy.
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Strange, must be somthing else going on.
I had the same 40 to 50 percent too slow, which was completely fixed bij using PSCHED_CPU in pkt_sched.h.
Using kernel 2.4.24, I measure the speed with the iptables byte counters.
Quoted 20 lines · On Fri, 26 Mar 2004 16:27:42 +1200 Simon Byrnand <[email protected]> wrote:
Quoted 8 lines · Simon Byrnand
Quoted 6 lines · Andrew Hall
You need to recompile the kernel after altering this value in
linux/include/net/pkt_sched.h. Also remember that if using SFQ on leaf qdisc, then the queue length may cause delay problems if it's too long (default is 128). Changing this to 32 for rates below 100kb/s, I have found to help things considerably. This change needs to be done inlinux/net/sched/sch_sfq.c. which also needs a kernel recompilation.Quoted 5 lines
Hmm,
When I use sfq with cbq at speeds like 256Kbit there is no problem at all, runs very sweetly, but with HTB and sfq, it is very jerky and poor. I'll try the change in
pkt_sched.hfirst and see how I go...Ok, I tried the change in
pkt_sched.hand didn't notice any difference whatsoever. Any other ideas ? cbq is still fine but htb for the same speed is very jerky and the speed fluctuates around 60-80% of the wanted speed, while cbq gives a steady 99% of the wanted speed...
Quoted 7 lines
Regards,
Simon_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/