Skip to contentSkip to conversation

HTB speed

LARTC Jeroen Vriesman 6 participants 13 messages Mar 12, 2004 – Mar 26, 2004

lore
Share & tools
lore
#0 lore raw

Dear all,

I'm running htb on 2.4.24, following setup:

running on a bridge, ebtables patch, firewall marks with iptables, and an SFQ for every HTB.

Everything works fine, so I don't go into details, there is just one thing which suprises me:

When I configure HTB to shape to about 900 kbit/s, the resulting speed is about 400 kbit/s.

For higher speeds it's more accurate (but stil too slow), for lower speeds it gets worse.

Does anyone here have any idea why? I've just added a "correction factor" to my scripts, so it's kind of allright, but I would like to know if others experience the same thing.

Jeroen.
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#1 lore raw

Hi,

just putting the answer to my own question here, for those who have the same problem, and read the mailing list archive.

The timing of the P4 based on "jiffies" is hopeless, it's different for every processor, and can be a wrong by a factor 3.

If the tsc (time stamp counter) is used, the htb works fine, the error in speed is now only about 1 %.

It's set by:

in pkt_sched.h:

#define PSCHED_CLOCK_SOURCE PSCHED_CPU

that's all, I wonder why it's not default to do this, or maybe it's an idea to make the packet scheduler detect the presence of tsc when the module is loaded.

Cheers,
Jeroen.

Quoted 10 lines · On Fri, 12 Mar 2004 10:58:40 +0100 Jeroen Vriesman <[email protected]> wrote:

Dear all,

I'm running htb on 2.4.24, following setup:

running on a bridge, ebtables patch, firewall marks with iptables, and an SFQ for every HTB.

Everything works fine, so I don't go into details, there is just one thing which suprises me:

When I configure HTB to shape to about 900 kbit/s, the resulting speed is about 400 kbit/s.

Quoted 9 lines

For higher speeds it's more accurate (but stil too slow), for lower speeds it gets worse.

Does anyone here have any idea why? I've just added a "correction factor" to my scripts, so it's kind of allright, but I would like to know if others experience the same thing.

Jeroen.
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#2 lore raw
In reply to Jeroen Vriesman
Quoted 6 lines

in pkt_sched.h:

#define PSCHED_CLOCK_SOURCE PSCHED_CPU

that's all, I wonder why it's not default to do this, or maybe it's an idea to make the packet scheduler detect the presence of tsc when the module is loaded.

Hi,

I think not all processors accept this #define PSCHED_CLOCK_SOURCE PSCHED_CPU. At least, I couldn't do it in my router.

I am not using HTB but TBF, but I think the timing problems are similar. As to tune TBF with higher precision, I had to manually change Hz value from 100 to 1000, in param.h, and actually I am getting better results.

My question is, does anybody tried to do this in an production environment? For instance, in a router that integrates traffic control, NAT, firewall, services gateway, etc., can this change in HZ value harm in any way the rest of the system?

Thanks in advance,
Joana Urbano

#3 lore raw
In reply to Jeroen Vriesman

Hi,

to use the TSC, the processor has to have a tsc, you can see that in /proc/cpuinfo, for as far as I know, every P4 has it (but I'm not sure), it's in the "flags" of cpuinfo:

On my P4-1800:

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm

The "loop calibrated" jiffies stuff dates back to 386, it's used everywhere in the kernel timing, and it's very inaccurate, I don't know if 2.6 still uses it everwhere, what kind of processor do you have?

I've never tried to change the Hz value, does changing it in param.h really changes the frequency of the clock ticks? If so, why is the default only 100Hz these days? doesn't make sense to me.

I do use the PSCHED_CPU in a production environement at the moment, works fine.

Cheers,
Jeroen.

Quoted 18 lines · On Mon, 15 Mar 2004 11:37:16 +0000 Maria Joana Urbano <[email protected]> wrote:
Quoted 8 lines

in pkt_sched.h:

#define PSCHED_CLOCK_SOURCE PSCHED_CPU

that's all, I wonder why it's not default to do this, or maybe it's an idea to make the packet scheduler detect the presence of tsc when the module is loaded.

Hi,

I think not all processors accept this #define PSCHED_CLOCK_SOURCE PSCHED_CPU. At least, I couldn't do it in my router.

I am not using HTB but TBF, but I think the timing problems are similar. As to tune TBF with higher precision, I had to manually change Hz value from 100 to 1000, in param.h, and actually I am getting better results.

Quoted 8 lines

My question is, does anybody tried to do this in an production environment? For instance, in a router that integrates traffic control, NAT, firewall, services gateway, etc., can this change in HZ value harm in any way the rest of the system?

Thanks in advance,
Joana Urbano

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#4 lore raw
In reply to Jeroen Vriesman

Hi,

Indeed, changing the timer to 1000Hz is possible, it turned out that I have a machine here running with a 1000Hz timer ticker. (I've installed a "realtime" kernel on it for audio recording).

About your previous question, I've noticed that the system with 1000Hz ticker (which has been running for a few months) is UNSTABLE.

might be more to it than just the 1000Hz, but I did notice that some "realtime" applications running in userspace where able to crash the system completely, happened 2 times in 3 months.

Cheers,
Jeroen.

Quoted 18 lines · On Mon, 15 Mar 2004 11:37:16 +0000 Maria Joana Urbano <[email protected]> wrote:
Quoted 8 lines

in pkt_sched.h:

#define PSCHED_CLOCK_SOURCE PSCHED_CPU

that's all, I wonder why it's not default to do this, or maybe it's an idea to make the packet scheduler detect the presence of tsc when the module is loaded.

Hi,

I think not all processors accept this #define PSCHED_CLOCK_SOURCE PSCHED_CPU. At least, I couldn't do it in my router.

I am not using HTB but TBF, but I think the timing problems are similar. As to tune TBF with higher precision, I had to manually change Hz value from 100 to 1000, in param.h, and actually I am getting better results.

Quoted 8 lines

My question is, does anybody tried to do this in an production environment? For instance, in a router that integrates traffic control, NAT, firewall, services gateway, etc., can this change in HZ value harm in any way the rest of the system?

Thanks in advance,
Joana Urbano

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#5 lore raw
In reply to Jeroen Vriesman
Quoted 13 lines · Jeroen Vriesman

Hi,

just putting the answer to my own question here, for those who have the same problem, and read the mailing list archive.

The timing of the P4 based on "jiffies" is hopeless, it's different for every processor, and can be a wrong by a factor 3.

If the tsc (time stamp counter) is used, the htb works fine, the error in speed is now only about 1 %.

It's set by:

Quoted 7 lines

in pkt_sched.h:

#define PSCHED_CLOCK_SOURCE PSCHED_CPU

that's all, I wonder why it's not default to do this, or maybe it's an idea to make the packet scheduler detect the presence of tsc when the module is loaded.

Hi,

Which pkt_sched.h are you refering to ?
/usr/src/linux/include/linux/pkt_sched.h or
/usr/src/linux/include/net/pkt_sched.h ?

And after changing it what did you do ? Recompile the kernel ?

Or recompile tc ?

I too see the same problems with htb (very poor accuracy of speed, significantly too slow, also very jerky) while cbq is very accurate and smooth. (But lacks some features I need)

Regards,
Simon

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#6 lore raw
In reply to Jeroen Vriesman

You need to recompile the kernel after altering this value in linux/include/net/pkt_sched.h. Also remember that if using SFQ on leaf qdisc, then the queue length may cause delay problems if it's too long (default is 128). Changing this to 32 for rates below 100kb/s, I have found to help things considerably. This change needs to be done in linux/net/sched/sch_sfq.c. which also needs a kernel recompilation.

----- Original Message -----
From: "Simon Byrnand" <[email protected]>
To: "Jeroen Vriesman" <[email protected]>; <[email protected]>
Sent: Thursday, March 25, 2004 4:36 PM
Subject: Re: [LARTC] HTB speed

Quoted 36 lines
Quoted 13 lines · Jeroen Vriesman

Hi,

just putting the answer to my own question here, for those who have the same problem, and read the mailing list archive.

The timing of the P4 based on "jiffies" is hopeless, it's different for every processor, and can be a wrong by a factor 3.

If the tsc (time stamp counter) is used, the htb works fine, the error in speed is now only about 1 %.

It's set by:

Quoted 7 lines

in pkt_sched.h:

#define PSCHED_CLOCK_SOURCE PSCHED_CPU

that's all, I wonder why it's not default to do this, or maybe it's an idea to make the packet scheduler detect the presence of tsc when the module is loaded.

Hi,

Which pkt_sched.h are you refering to ?
/usr/src/linux/include/linux/pkt_sched.h or
/usr/src/linux/include/net/pkt_sched.h ?

And after changing it what did you do ? Recompile the kernel ?

Or recompile tc ?

I too see the same problems with htb (very poor accuracy of speed, significantly too slow, also very jerky) while cbq is very accurate and smooth. (But lacks some features I need)

Quoted 6 lines

Regards,
Simon

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#7 lore raw
In reply to Jeroen Vriesman

Would the problem with queue lengths also exist if CONFIG_HZ00 in .config? (it certainly improves latency)

I've configured rates from 10kbit/s...2Mbit/s on one machine, so changing the kernel code "for low rates" would probably effect the high rates too.

Or, in other words, what is exactly the problem with an SFQ length of 128 for low rates?

Cheers,
Jeroen.

Quoted 53 lines · On Thu, 25 Mar 2004 17:17:44 +1100 "Andrew Hall" <[email protected]> wrote:

You need to recompile the kernel after altering this value in linux/include/net/pkt_sched.h. Also remember that if using SFQ on leaf qdisc, then the queue length may cause delay problems if it's too long (default is 128). Changing this to 32 for rates below 100kb/s, I have found to help things considerably. This change needs to be done in linux/net/sched/sch_sfq.c. which also needs a kernel recompilation.

----- Original Message -----
From: "Simon Byrnand" <[email protected]>
To: "Jeroen Vriesman" <[email protected]>; <[email protected]>
Sent: Thursday, March 25, 2004 4:36 PM
Subject: Re: [LARTC] HTB speed

Quoted 39 lines
Quoted 13 lines · Jeroen Vriesman

Hi,

just putting the answer to my own question here, for those who have the same problem, and read the mailing list archive.

The timing of the P4 based on "jiffies" is hopeless, it's different for every processor, and can be a wrong by a factor 3.

If the tsc (time stamp counter) is used, the htb works fine, the error in speed is now only about 1 %.

It's set by:

Quoted 7 lines

in pkt_sched.h:

#define PSCHED_CLOCK_SOURCE PSCHED_CPU

that's all, I wonder why it's not default to do this, or maybe it's an idea to make the packet scheduler detect the presence of tsc when the module is loaded.

Hi,

Which pkt_sched.h are you refering to ?
/usr/src/linux/include/linux/pkt_sched.h or
/usr/src/linux/include/net/pkt_sched.h ?

And after changing it what did you do ? Recompile the kernel ?

Or recompile tc ?

I too see the same problems with htb (very poor accuracy of speed, significantly too slow, also very jerky) while cbq is very accurate and smooth. (But lacks some features I need)

Regards,
Simon

Quoted 8 lines
Quoted 3 lines

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#8 lore raw
In reply to Jeroen Vriesman

Thanks, Andrew, it's intresting...

-> (default is 128). Changing this to 32 for rates below 100kb/s, I -> have found

you mean that try change "linux/net/sched/sch_sfq.c" this: ?

#define SFQ_DEPTH 128
#define SFQ_HASH_DIVISOR 1024

for this.

#define SFQ_DEPTH 32
#define SFQ_HASH_DIVISOR 1024

I 've 2 questions about it

1) what is I use various rates (from 8k to 256k)

2) how I must proceed in Esfq ??
(in order to work with those htb classes)

Usage: ... esfq [ perturb SECS ] [ quantum BYTES ] [ depth FLOWS ]
[ divisor HASHBITS ] [ limit PKTS ] [ hash HASHTYPE]
Where:
HASHTYPE := { classic | src | dst }

Thankyou!
andres

->
-> You need to recompile the kernel after altering this value in -> linux/include/net/pkt_sched.h. Also remember that if using SFQ on leaf -> qdisc, then the queue length may cause delay problems if it's too long -> (default is 128). Changing this to 32 for rates below 100kb/s, I -> have found
-> to help things considerably. This change needs to be done in -> linux/net/sched/sch_sfq.c. which also needs a kernel recompilation. ->
-> ----- Original Message -----
-> From: "Simon Byrnand" <[email protected]>
-> To: "Jeroen Vriesman" <[email protected]>; <[email protected]> -> Sent: Thursday, March 25, 2004 4:36 PM
-> Subject: Re: [LARTC] HTB speed
->
->
-> > At 11:14 14/03/2004, Jeroen Vriesman wrote:
-> > >Hi,
-> > >
-> > >just putting the answer to my own question here, for those -> who have the
-> > >same problem, and read the mailing list archive.
-> > >
-> > >The timing of the P4 based on "jiffies" is hopeless, it's -> different for
-> > >every processor, and can be a wrong by a factor 3.
-> > >
-> > >If the tsc (time stamp counter) is used, the htb works fine, -> the error in
-> > >speed is now only about 1 %.
-> > >
-> > >It's set by:
-> > >
-> > >in pkt_sched.h:
-> > >
-> > >#define PSCHED_CLOCK_SOURCE PSCHED_CPU
-> > >
-> > >that's all, I wonder why it's not default to do this, or maybe it's an -> > >idea to make the packet scheduler detect the presence of tsc when the -> > >module is loaded.
-> >
-> > Hi,
-> >
-> > Which pkt_sched.h are you refering to ?
-> > /usr/src/linux/include/linux/pkt_sched.h or
-> > /usr/src/linux/include/net/pkt_sched.h ?
-> >
-> > And after changing it what did you do ? Recompile the kernel ? -> >
-> > Or recompile tc ?
-> >
-> > I too see the same problems with htb (very poor accuracy of speed, -> > significantly too slow, also very jerky) while cbq is very accurate and -> > smooth. (But lacks some features I need)
-> >
-> > Regards,
-> > Simon
-> >
-> > _______________________________________________
-> > LARTC mailing list / [email protected]
-> > http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
->
-> _______________________________________________
-> LARTC mailing list / [email protected]
-> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
->

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#9 lore raw
In reply to Jeroen Vriesman
Quoted 6 lines · Andrew Hall

You need to recompile the kernel after altering this value in linux/include/net/pkt_sched.h. Also remember that if using SFQ on leaf qdisc, then the queue length may cause delay problems if it's too long (default is 128). Changing this to 32 for rates below 100kb/s, I have found to help things considerably. This change needs to be done in linux/net/sched/sch_sfq.c. which also needs a kernel recompilation.

Hmm,

When I use sfq with cbq at speeds like 256Kbit there is no problem at all, runs very sweetly, but with HTB and sfq, it is very jerky and poor. I'll try the change in pkt_sched.h first and see how I go...

Regards,
Simon

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#10 lore raw
In reply to Jeroen Vriesman
Quoted 8 lines · Simon Byrnand
Quoted 6 lines · Andrew Hall

You need to recompile the kernel after altering this value in linux/include/net/pkt_sched.h. Also remember that if using SFQ on leaf qdisc, then the queue length may cause delay problems if it's too long (default is 128). Changing this to 32 for rates below 100kb/s, I have found to help things considerably. This change needs to be done in linux/net/sched/sch_sfq.c. which also needs a kernel recompilation.

Quoted 5 lines

Hmm,

When I use sfq with cbq at speeds like 256Kbit there is no problem at all, runs very sweetly, but with HTB and sfq, it is very jerky and poor. I'll try the change in pkt_sched.h first and see how I go...

Ok, I tried the change in pkt_sched.h and didn't notice any difference whatsoever. Any other ideas ? cbq is still fine but htb for the same speed is very jerky and the speed fluctuates around 60-80% of the wanted speed, while cbq gives a steady 99% of the wanted speed...

Regards,
Simon

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#11 lore raw
In reply to Jeroen Vriesman
Quoted 20 lines · Simon Byrnand
Quoted 11 lines · Simon Byrnand
Quoted 7 lines · Andrew Hall

You need to recompile the kernel after altering this value in linux/include/net/pkt_sched.h. Also remember that if using SFQ on leaf qdisc, then the queue length may cause delay problems if it's too long (default is 128). Changing this to 32 for rates below 100kb/s, I have found
to help things considerably. This change needs to be done in linux/net/sched/sch_sfq.c. which also needs a kernel recompilation.

Quoted 5 lines

Hmm,

When I use sfq with cbq at speeds like 256Kbit there is no problem at all, runs very sweetly, but with HTB and sfq, it is very jerky and poor. I'll try the change in pkt_sched.h first and see how I go...

Quoted 4 lines

Ok, I tried the change in pkt_sched.h and didn't notice any difference whatsoever. Any other ideas ? cbq is still fine but htb for the same speed is very jerky and the speed fluctuates around 60-80% of the wanted speed, while cbq gives a steady 99% of the wanted speed...

You can make HTB more accurate by setting HTB_HYSTERESIS to 0 in net/sched/sch_htb.c.

To save time - if you built HTB as a module, you can probably (well it worked for me) get away with editing htb.c and do

make SUBDIRS=net/sched modules
and replacing /lib/modules/[kversion]/kernel/net/sched/htb.o with the new htb.o from your source tree.

If you are doing it live stop shaping and check with lsmod that modprobe -r gets rid (do it again if it's still there) of the old htb.o and reload shaping scripts.

Make sure the quantum is your mtu, set 0 burst for bulk classes and don't set perturb too low on sfq (I use 20 as it causes packet reordering).

Are you shaping egress or ingress and how are you measuring speed?

Andy.

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#12 lore raw
In reply to Jeroen Vriesman

Strange, must be somthing else going on.

I had the same 40 to 50 percent too slow, which was completely fixed bij using PSCHED_CPU in pkt_sched.h.

Using kernel 2.4.24, I measure the speed with the iptables byte counters.

Quoted 20 lines · On Fri, 26 Mar 2004 16:27:42 +1200 Simon Byrnand <[email protected]> wrote:
Quoted 8 lines · Simon Byrnand
Quoted 6 lines · Andrew Hall

You need to recompile the kernel after altering this value in linux/include/net/pkt_sched.h. Also remember that if using SFQ on leaf qdisc, then the queue length may cause delay problems if it's too long (default is 128). Changing this to 32 for rates below 100kb/s, I have found to help things considerably. This change needs to be done in linux/net/sched/sch_sfq.c. which also needs a kernel recompilation.

Quoted 5 lines

Hmm,

When I use sfq with cbq at speeds like 256Kbit there is no problem at all, runs very sweetly, but with HTB and sfq, it is very jerky and poor. I'll try the change in pkt_sched.h first and see how I go...

Ok, I tried the change in pkt_sched.h and didn't notice any difference whatsoever. Any other ideas ? cbq is still fine but htb for the same speed is very jerky and the speed fluctuates around 60-80% of the wanted speed, while cbq gives a steady 99% of the wanted speed...

Quoted 7 lines

Regards,
Simon

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/