HTB stalling
Hi
Couple months ago I started to have a strange problem with HTB. My setup is Fedora Core 2 + Pentium 2 233 + 128 MB of ram and its serving as a router.
For some time since going to kernel 2.6 my HTB QoS Stalls for couple seconds, every couple minutes. If the connection load is bigger the stalling is more frequent and takes longer.
I isolated the problem to be with HTB (CBQ works fine).
The script I use now worked for a year without any problems, but since fedora changed to 2.6 the problems started.
The problem is with the stock kernel and with a self-compiled one.
Signature · 2 lines
Miłego Dnia Krystian Antoni
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Quoted 1 line
Hi
Hi!
Quoted 7 lines
Couple months ago I started to have a strange problem with HTB. My setup is Fedora Core 2 + Pentium 2 233 + 128 MB of ram and its serving as a router.
For some time since going to kernel 2.6 my HTB QoS Stalls for couple seconds, every couple minutes. If the connection load is bigger the stalling is more frequent and takes longer.
I experienced this too. I use 2.6.11 with qnet patch. I've a bigger
machine with lots of users.
Quoted 1 line
I isolated the problem to be with HTB (CBQ works fine).
I use HTB too. I can't use CBQ with some reasons so I cant test it. My
HTB setup stands about 2500 classes and a total of approx 15000 tc objects.
Quoted 2 lines
The script I use now worked for a year without any problems, but since fedora changed to 2.6 the problems started.
Me too. I've think the problem started after I use kernel 2.6.8.1. If
I good remember I doesn't have this problem with 2.6.7. The new thing in 2.6.8.1 was the QoS clock source (or similar). I use here CPU cycle counter because I have a fast uplink (1 Gbps). I think maybe this can be the source of the problem. Or the HTB has a bug from 2.6.8.1.
If anybody has idea please write to here.
Arpad Kunszt
PS: Sorry for my terribly bad english :-(
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Quoted 12 lines · Krystian Antoni
Hi
Couple months ago I started to have a strange problem with HTB. My setup is Fedora Core 2 + Pentium 2 233 + 128 MB of ram and its serving as a router.
For some time since going to kernel 2.6 my HTB QoS Stalls for couple seconds, every couple minutes. If the connection load is bigger the stalling is more frequent and takes longer.
I isolated the problem to be with HTB (CBQ works fine).
The script I use now worked for a year without any problems, but since fedora changed to 2.6 the problems started.
The problem is with the stock kernel and with a self-compiled one.
What sort of load, number of rules/classes and bandwidth do you have?
Andy.
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Connection is 256kbit/2mbit, but the down interface is used for SMB also so its shaped from 100mbits. The interfaces are both ethernet.
Number of rules in Iptables is around 300, and I have a new kernel (from
kernel org) and new iptables.
Number of qdisc/class per interface is 7-8. Leaf classes are SFQ. I tryed:
- running only one part of QOS (down or up)
- puting only minimal set of rules in iptables
- getting kernels from fedora/kernel.org
- reinstalling fedora.
- changing clock from standart to gettimeofday
- changed leaves from SFQ to PRIO
- changed speeds of QOS.
- played with my outside interface from HALF to FULL duplex and reverse way. and still I had this stalls on the interface which has HTB QoS setup. Load of my internet connection is around 10%. Propably you're thinking that
I set up my connections parameters too high in QOS, but that is not the case (I think ;-) since I did a little bit of testing and found out the setting which alowed me to keep LAN pings through my internet connection even under heavy load.
CBQ works without any problems.
Computer is 233 MMX with 128 of RAM (36 MB used for system), NIC's are
3com's 905B and 905C. I dont remember if I tryed changing them :-)
And I dont really now what is that I screw up :-) Wonna see my qos scripts?
:D
Quoted 21 lines · Andy Furniss
Quoted 4 lines · Krystian Antoni
Hi
Couple months ago I started to have a strange problem with HTB. My setup is Fedora Core 2 + Pentium 2 233 + 128 MB of ram and its
serving as
Quoted 3 lines
a router.
For some time since going to kernel 2.6 my HTB QoS Stalls for couple seconds, every couple minutes. If the connection load is bigger thestalling
Quoted 5 lines
is more frequent and takes longer.
I isolated the problem to be with HTB (CBQ works fine).
The script I use now worked for a year without any problems, but since fedora changed to 2.6 the problems started.
The problem is with the stock kernel and with a self-compiled one.What sort of load, number of rules/classes and bandwidth do you have?
Andy.
Signature · 2 lines
Miłego Dnia Krystian Antoni
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Quoted 17 lines · Krystian Antoni
Connection is 256kbit/2mbit, but the down interface is used for SMB also so its shaped from 100mbits. The interfaces are both ethernet.
Number of rules in Iptables is around 300, and I have a new kernel (from
kernel org) and new iptables.
Number of qdisc/class per interface is 7-8. Leaf classes are SFQ. I tryed:
- running only one part of QOS (down or up)
- puting only minimal set of rules in iptables
- getting kernels from fedora/kernel.org
- reinstalling fedora.
- changing clock from standart to gettimeofday
- changed leaves from SFQ to PRIO
- changed speeds of QOS.
- played with my outside interface from HALF to FULL duplex and reverse way. and still I had this stalls on the interface which has HTB QoS setup. Load of my internet connection is around 10%. Propably you're thinking that
I set up my connections parameters too high in QOS, but that is not the case
Quoted 8 lines
(I think ;-) since I did a little bit of testing and found out the setting which alowed me to keep LAN pings through my internet connection even under heavy load.
CBQ works without any problems.
Computer is 233 MMX with 128 of RAM (36 MB used for system), NIC's are
3com's 905B and 905C. I dont remember if I tryed changing them :-)
And I dont really now what is that I screw up :-) Wonna see my qos scripts?
:D
I use a P200 for a gateway but it's only got one eth card, so I can't really recreate your setup properly - I assume the SMB traffic is forwarded and not local traffic.
I can't even do 100Mbit when actually running anything on mine with a rtl8139 - I run out of CPU - doesn't seem to affect internet shaping/throughput.
I'll have a go at shaping on eth and see if I can recreate it when I have time.
When you say stalls, what happens eg just no throughput or dropped packets or just high latency - is swap being used etc.
Andy.
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Hi all
iam also facing the same problem what Mr Antoni have
even i have done many kind of experment, but i could not resolve is this bug in FC3, but when i does the FC1 its working fine
I found difference from FC1 to FC3 is
FC1 iptables 1.2.8 HTB 3.12
FC3 iptable 1.2.11 htb 3.16
can some experts coment on this
hare
----- Original Message -----
From: Krystian Antoni
To: Andy Furniss
Cc: [email protected]
Sent: Wednesday, April 20, 2005 11:43 AM
Subject: Re: [LARTC] HTB stalling
Connection is 256kbit/2mbit, but the down interface is used for SMB also so its shaped from 100mbits. The interfaces are both ethernet.
Number of rules in Iptables is around 300, and I have a new kernel (from kernel org) and new iptables.
Number of qdisc/class per interface is 7-8. Leaf classes are SFQ.
I tryed:
- running only one part of QOS (down or up)
- puting only minimal set of rules in iptables
- getting kernels from fedora/kernel.org
- reinstalling fedora.
- changing clock from standart to gettimeofday
- changed leaves from SFQ to PRIO
- changed speeds of QOS.
- played with my outside interface from HALF to FULL duplex and reverse way. and still I had this stalls on the interface which has HTB QoS setup.
Load of my internet connection is around 10%. Propably you're thinking that I set up my connections parameters too high in QOS, but that is not the case (I think ;-) since I did a little bit of testing and found out the setting which alowed me to keep LAN pings through my internet connection even under heavy load.
CBQ works without any problems.
Computer is 233 MMX with 128 of RAM (36 MB used for system), NIC's are 3com's 905B and 905C. I dont remember if I tryed changing them :-)
And I dont really now what is that I screw up :-) Wonna see my qos scripts? :D
On 4/19/05, Andy Furniss <[email protected]> wrote:
Krystian Antoni wrote:
> Hi
>
> Couple months ago I started to have a strange problem with HTB. > My setup is Fedora Core 2 + Pentium 2 233 + 128 MB of ram and its serving as > a router.
> For some time since going to kernel 2.6 my HTB QoS Stalls for couple > seconds, every couple minutes. If the connection load is bigger the stalling > is more frequent and takes longer.
> I isolated the problem to be with HTB (CBQ works fine).
> The script I use now worked for a year without any problems, but since > fedora changed to 2.6 the problems started.
> The problem is with the stock kernel and with a self-compiled one.
What sort of load, number of rules/classes and bandwidth do you have?
Andy.
--
Miłego Dnia
Krystian Antoni
------------------------------------------------------------------------------
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Quoted 2 lines
Hi all
Hi!
Quoted 10 lines
iam also facing the same problem what Mr Antoni have even i have done many kind of experment, but i could not resolve is this bug in FC3,
but when i does the FC1 its working fineI found difference from FC1 to FC3 is
FC1 iptables 1.2.8 HTB 3.12
FC3 iptable 1.2.11 htb 3.16
I experience the problem with iptables 1.3.1 patched with qnet patch.
Are you using a 2.6 series kernel with FC1? IMHO in FC1 the default kernel was 2.4.
I think the problem is in the kernel somewhere.
Bye,
Arpad Kunszt
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Quoted 9 lines
hareram wrote:
Hi all
iam also facing the same problem what Mr Antoni have even i have done many kind of experment, but i could not resolve is this bug in FC3,
but when i does the FC1 its working fine
Quoted 7 lines
I found difference from FC1 to FC3 is
FC1 iptables 1.2.8 HTB 3.12
FC3 iptable 1.2.11 htb 3.16can some experts coment on this
hare
Although this stalling issue may still be a timer problem rather than HTB, there are bugs in versions of HTB < 3.17 so you MUST patch the kernel so HTB is at least 3.17. I don't know about iptables.
Do remember that FC3 contains much BETA; you really should be using something less "bleeding edge" for a server.
--
gypsy
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
The problem started with arrival of 2.6 kernels.
Quoted 31 lines · Kunszt Arpad
Quoted 3 lines
Hi all
Hi!
Quoted 10 lines
iam also facing the same problem what Mr Antoni have even i have done many kind of experment, but i could not resolve is this bug in FC3,
but when i does the FC1 its working fineI found difference from FC1 to FC3 is
FC1 iptables 1.2.8 HTB 3.12
FC3 iptable 1.2.11 htb 3.16I experience the problem with iptables 1.3.1 patched with qnet patch. Are you using a 2.6 series kernel with FC1? IMHO in FC1 the default kernel was 2.4.
I think the problem is in the kernel somewhere.
Bye,Arpad Kunszt
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Signature · 2 lines
Miłego Dnia Krystian Antoni
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
How to check version of HTB??
I have standart one which came with kernel 2.6.11.7 <http://2.6.11.7>, my tc says I have "TC HTB version 3.3". Is this my HTB version? :-)
Im not using FC3 kernel but a vanilla one so there is no much beta in it :) besides I cut most of the stuff out just for experimentation and still no luck :)
Quoted 31 lines · gypsy
Quoted 10 lines
hareram wrote:
Hi all
iam also facing the same problem what Mr Antoni have even i have done many kind of experment, but i could not resolve is this bug in FC3,
but when i does the FC1 its working fineQuoted 7 lines
I found difference from FC1 to FC3 is
FC1 iptables 1.2.8 HTB 3.12
FC3 iptable 1.2.11 htb 3.16can some experts coment on this
hareAlthough this stalling issue may still be a timer problem rather than HTB, there are bugs in versions of HTB < 3.17 so you MUST patch the kernel so HTB is at least 3.17. I don't know about iptables.
Do remember that FC3 contains much BETA; you really should be using something less "bleeding edge" for a server.
--
gypsy
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Signature · 2 lines
Miłego Dnia Krystian Antoni
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Quoted 4 lines · Krystian Antoni
How to check version of HTB??
I have standart one which came with kernel 2.6.11.7, my tc says I have "TC HTB version 3.3". Is this my HTB version? :-)
No, it isn't. 3.3 is the TC version.
The only ways I know to find out the HTB version are:
1) if it loads as a module, it will record in your logs its version.
2) read the source. 'grep HTB_VER /usr/src/linux/net/sched/sch_htb.c' should return something like
0x30011
That "11" at the end is hex, so in decimal it = 17, making the example above 3.17
Quoted 3 lines
Im not using FC3 kernel but a vanilla one so there is no much beta in it :) besides I cut most of the stuff out just for experimentation and still no luck :)
But it is still a 2.6 kernel, not a 2.4 kernel? In 2.4.x, linux/include/net/pkt_sched.h specifies the timer. (I can't help you with 2.6. Until it appears as standard in Slackware current, it is not stable enough for my production servers.).
--
gypsy
Quoted 37 lines · gypsy
> hareram wrote:
>
> Hi all
>
> iam also facing the same problem what Mr Antoni have > even i have done many kind of experment, but i could not resolve is
> this bug in FC3,
> but when i does the FC1 its working fine
>
> I found difference from FC1 to FC3 is
>
> FC1 iptables 1.2.8 HTB 3.12
> FC3 iptable 1.2.11 htb 3.16
>
> can some experts coment on this
> hareAlthough this stalling issue may still be a timer problem rather than
HTB, there are bugs in versions of HTB < 3.17 so you MUST patch the
kernel so HTB is at least 3.17. I don't know about iptables.Do remember that FC3 contains much BETA; you really should be using
something less "bleeding edge" for a server.
--
gypsy
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Quoted 3 lines
--
Mi³ego Dnia
Krystian Antoni
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
so i checked using your way: "HTB init, kernel part version 3.17".
Quoted 70 lines · gypsy
Quoted 3 lines · Krystian Antoni
How to check version of HTB??
I have standart one which came with kernel 2.6.11.7 <http://2.6.11.7>,my tc says I have
Quoted 1 line
"TC HTB version 3.3". Is this my HTB version? :-)
No, it isn't. 3.3 is the TC version.
The only ways I know to find out the HTB version are:
1) if it loads as a module, it will record in your logs its version.
2) read the source. 'grepHTB_VER/usr/src/linux/net/sched/sch_htb.c' should return something like0x30011
That "11" at the end is hex, so in decimal it = 17, making the example above 3.17Quoted 3 lines
Im not using FC3 kernel but a vanilla one so there is no much beta in it :) besides I cut most of the stuff out just for experimentation and still no luck :)
But it is still a 2.6 kernel, not a 2.4 kernel? In 2.4.x,
linux/include/net/pkt_sched.hspecifies the timer. (I can't help you with 2.6. Until it appears as standard in Slackware current, it is not stable enough for my production servers.).
--
gypsyQuoted 40 lines · gypsy
Quoted 7 lines
hareram wrote:
Hi all
iam also facing the same problem what Mr Antoni have even i have done many kind of experment, but i could not
resolve is
Quoted 10 lines
this bug in FC3,
but when i does the FC1 its working fineI found difference from FC1 to FC3 is
FC1 iptables 1.2.8 HTB 3.12
FC3 iptable 1.2.11 htb 3.16can some experts coment on this
hareAlthough this stalling issue may still be a timer problem rather than
HTB, there are bugs in versions of HTB < 3.17 so you MUST patch the
kernel so HTB is at least 3.17. I don't know about iptables.Do remember that FC3 contains much BETA; you really should be using
something less "bleeding edge" for a server.
--
gypsy
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc--
Mi³ego Dnia
Krystian Antoni
Signature · 2 lines
Miłego Dnia Krystian Antoni
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Quoted 2 lines · Andy Furniss
I'll have a go at shaping on eth and see if I can recreate it when I have time.
I tried and can't recreate it on my p200 with CPU as timer.
The test probably could have been better though - I just ran 7 netperfs each shaped to its own class and tweaked the rates until I was using 98% CPU. Played DoD with the netgraph all the way across the screen and could notice nothing strange happening.
Is anyone who gets the stalls using CPU as timer?
The class my internet traffic went to had way more rate than it needed and I had pfifos on the leaves.
Do the stalls literally last 2 seconds, should I be able to easily see it just pinging or polling packet counts every second?
Andy
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Quoted 1 line · Andy Furniss
Is anyone who gets the stalls using CPU as timer?
I see Arpad is - another difference possibly - do you have smp as kernel option but have one processor. I don't know if it's anything to do with this, but last time I left it on as an option I got a panic - so it's always off for me now.
Andy.
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Quoted 10 lines · Andy Furniss
Quoted 1 line · Andy Furniss
Is anyone who gets the stalls using CPU as timer?
I see Arpad is - another difference possibly - do you have smp as kernel option but have one processor. I don't know if it's anything to do with this, but last time I left it on as an option I got a panic - so it's always off for me now.
I use the CPU as timer and I have a dual Xeon box. So I use the SMP but I have phisically 2 CPUs. I don't use HyperThreading.
When I ran into this problem the CPU usage was not too high. The load was low and I had 25-50% idle CPU time (top). If I make some load on the macine the ping times grow and none else change. I checked that the ksoftirqd is not eating my CPU.
In the next 2-3 weeks I'll try to make a setup with HFSC not HTB maybe it will help (I cant use CBQ on this machine). But there are about 20-25000 tc objects on this machine and it's not a 5 minutes job :-( I haven't any else idea yet :-(
Arpad Kunszt
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Quoted 2 lines · Kunszt Arpad
I use the CPU as timer and I have a dual Xeon box. So I use the SMP but I have phisically 2 CPUs. I don't use HyperThreading.
The TSCs on an SMP box may drift apart. Can you reproduce the problem with gettimeofday as time source?
Regards
Patrick
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
I have a normal kernel + vanilla one with SMP turned off.
Quoted 12 lines · Andy Furniss
Quoted 1 line · Andy Furniss
Is anyone who gets the stalls using CPU as timer?
I see Arpad is - another difference possibly - do you have smp as kernel option but have one processor. I don't know if it's anything to do with this, but last time I left it on as an option I got a panic - so it's always off for me now.
Andy.
Signature · 2 lines
Miłego Dnia Krystian Antoni
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
the stalls might last as long as 5 seconds and when they happen, everything including a web browser stops working.
ill be back tuesday evening so then i'll try to look at my system.
Quoted 24 lines · Andy Furniss
Quoted 2 lines · Andy Furniss
I'll have a go at shaping on eth and see if I can recreate it when I have time.
I tried and can't recreate it on my p200 with CPU as timer.
The test probably could have been better though - I just ran 7 netperfs each shaped to its own class and tweaked the rates until I was using 98% CPU. Played DoD with the netgraph all the way across the screen and could notice nothing strange happening.
Is anyone who gets the stalls using CPU as timer?
The class my internet traffic went to had way more rate than it needed and I had pfifos on the leaves.
Do the stalls literally last 2 seconds, should I be able to easily see it just pinging or polling packet counts every second?
Andy
Signature · 2 lines
Miłego Dnia Krystian Antoni
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Quoted 7 lines
Quoted 2 lines · Kunszt Arpad
I use the CPU as timer and I have a dual Xeon box. So I use the SMP but I have phisically 2 CPUs. I don't use HyperThreading.
The TSCs on an SMP box may drift apart. Can you reproduce the problem with gettimeofday as time source?
I'll make a try in the next 2-3 days.
Arpad Kunszt
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Quoted 4 lines · Krystian Antoni
the stalls might last as long as 5 seconds and when they happen, everything including a web browser stops working.
ill be back tuesday evening so then i'll try to look at my system.
Maybe I should try with a script more like you are using if you don't mind posting.
One thought I have is that htb seems to have far longer default queues than it used to. If you have set a default class other than 0 with a low rate then your arp packets may be getting stuck in a very long queue.
Andy.
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc