From: "Pablo Fernandes Yahoo" Date: Sat, 26 May 2007 09:54:03 +0000 Subject: [LARTC] big problem with HTB/CBQ and CPU for more than 1.700 Message-Id: <20070526135435.C96F540DB@outpost.ds9a.nl> MIME-Version: 1 Content-Type: multipart/mixed; boundary="===============1619405892==" List-Id: To: lartc@vger.kernel.org This is a multi-part message in MIME format. --===============1619405892== Content-Type: multipart/alternative; boundary="----=_NextPart_000_0009_01C79F84.2F6E0F70" This is a multi-part message in MIME format. ------=_NextPart_000_0009_01C79F84.2F6E0F70 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Hello, have HTB "rules" in 4 different ISPs and i control for each customer this way: Flush and 1:0 class tc qdisc del dev eth0 root tc qdisc add dev eth0 root handle 1:0 htb tc class add dev eth0 parent 1:0 classid 1:1 htb rate 100mbit tc qdisc del dev eth1 root tc qdisc add dev eth1 root handle 1:0 htb tc class add dev eth1 parent 1:0 classid 1:1 htb rate 100mbit Upload and Download: user1 tc class add dev eth0 parent 1:1 classid 1:5 htb rate 150kbit ceil 150kbit tc qdisc add dev eth0 parent 1:5 handle 5: sfq perturb 10 tc class add dev eth1 parent 1:1 classid 1:5 htb rate 50kbit ceil 50kbit tc qdisc add dev eth1 parent 1:5 handle 5: sfq perturb 10 iptables -t mangle -A POSTROUTING --dest x.x.x.x -o eth0 -j CLASSIFY --set-class 1:5 iptables -t mangle -A FORWARD --src x.x.x.x -o eth1 -j CLASSIFY --set-class 1:5 Upload and Download: user2 tc class add dev eth0 parent 1:1 classid 1:8 htb rate 150kbit ceil 150kbit tc qdisc add dev eth0 parent 1:8 handle 8: sfq perturb 10 tc class add dev eth1 parent 1:1 classid 1:8 htb rate 50kbit ceil 50kbit tc qdisc add dev eth1 parent 1:8 handle 8: sfq perturb 10 iptables -t mangle -A POSTROUTING --dest y.y.y.y -o eth0 -j CLASSIFY --set-class 1:8 iptables -t mangle -A FORWARD --src y.y.y.y -o eth1 -j CLASSIFY --set-class 1:8 (.) This rules works fine, but just for less than 1.700 customers. More than 1.700 customers, i have my load avarage in the sky and Ksoftirqd process (top information) in 100% fulltime. I don't know why. I used to use CBQ instead HTB because i had the same problem and Ron (a guy in this list) gave this rules and told me that he uses this for more than 3.000 customers. I tested it in more than 7 different computers (but the same hadware specifications) and i had the same problem with either CBQ or HTB rules. The computers that i have are all of them DELL PowerEdge 1850. I will put some hardware iformations here: top PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3 root 39 19 0 0 0 R 100 0.0 5316:20 ksoftirqd/0 [root@fw ~]# uptime 10:38:11 up 161 days, 17:21, 3 users, load average: 1.58, 1.65, 1.51 (unfortunately when i took this, the load average was "pretty good", but minutes ago, it was more than 11.0 [root@fw ~]# lspci 00:00.0 Host bridge: Intel Corporation E7520 Memory Controller Hub (rev 09) 00:02.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express Port A (rev 09) 00:04.0 PCI bridge: Intel Corporation E7525/E7520 PCI Express Port B (rev 09) 00:05.0 PCI bridge: Intel Corporation E7520 PCI Express Port B1 (rev 09) 00:06.0 PCI bridge: Intel Corporation E7520 PCI Express Port C (rev 09) 00:1d.0 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #1 (rev 02) 00:1d.1 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #2 (rev 02) 00:1d.2 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #3 (rev 02) 00:1d.7 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB2 EHCI Controller (rev 02) 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c2) 00:1f.0 ISA bridge: Intel Corporation 82801EB/ER (ICH5/ICH5R) LPC Interface Bridge (rev 02) 00:1f.1 IDE interface: Intel Corporation 82801EB/ER (ICH5/ICH5R) IDE Controller (rev 02) 01:00.0 PCI bridge: Intel Corporation 80332 [Dobson] I/O processor (A-Segment Bridge) (rev 06) 01:00.2 PCI bridge: Intel Corporation 80332 [Dobson] I/O processor (B-Segment Bridge) (rev 06) 02:0c.0 Ethernet controller: Intel Corporation 82545GM Gigabit Ethernet Controller (rev 04) 02:0e.0 RAID bus controller: Dell PowerEdge Expandable RAID controller 4 (rev 06) 03:0b.0 Ethernet controller: Intel Corporation 82545GM Gigabit Ethernet Controller (rev 04) 05:00.0 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge A (rev 09) 05:00.2 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge B (rev 09) 06:07.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit Ethernet Controller (rev 05) 07:08.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit Ethernet Controller (rev 05) 09:0d.0 VGA compatible controller: ATI Technologies Inc Radeon RV100 QY [Radeon 7000/VE] [root@fw ~]# free -m total used free shared buffers cached Mem: 2021 1479 542 0 400 654 -/+ buffers/cache: 424 1597 Swap: 1027 0 1027 [root@fw ~]# cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 15 model : 4 model name : Intel(R) Xeon(TM) CPU 3.00GHz stepping : 3 cpu MHz : 2992.674 cache size : 2048 KB physical id : 0 siblings : 2 core id : 0 cpu cores : 1 fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 5 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pni monitor ds_cpl cid cx16 xtpr bogomips : 5990.78 processor : 1 vendor_id : GenuineIntel cpu family : 15 model : 4 model name : Intel(R) Xeon(TM) CPU 3.00GHz stepping : 3 cpu MHz : 2992.674 cache size : 2048 KB physical id : 0 siblings : 2 core id : 0 cpu cores : 1 fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 5 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pni monitor ds_cpl cid cx16 xtpr bogomips : 5985.13 Any help/Tipp/hint will be very welcome. Thanks in Advance! Pablo Fernandes ------=_NextPart_000_0009_01C79F84.2F6E0F70 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hello,

 

have HTB „rules“ in 4 different ISPs and = i control for each customer this way:

 

Flush and 1:0 = class

tc qdisc del dev eth0 = root

tc qdisc add dev eth0 root handle 1:0 = htb

tc class add dev eth0 parent 1:0 classid 1:1 = htb rate 100mbit

tc qdisc del dev eth1 = root

tc qdisc add dev eth1 root handle 1:0 = htb

tc class add dev eth1 parent 1:0 classid 1:1 = htb rate 100mbit

 

Upload and Download: = user1

tc class add dev eth0 parent 1:1 classid 1:5 = htb rate 150kbit ceil 150kbit

tc qdisc add dev eth0 parent 1:5 handle 5: sfq = perturb 10

tc class add dev eth1 parent 1:1 classid 1:5 htb rate = 50kbit ceil 50kbit

tc qdisc add dev eth1 parent 1:5 handle 5: sfq = perturb 10

iptables -t mangle -A POSTROUTING --dest x.x.x.x -o = eth0 -j CLASSIFY --set-class 1:5

iptables -t mangle -A FORWARD --src x.x.x.x -o eth1 = -j CLASSIFY --set-class 1:5

 

Upload and Download: = user2

tc class add dev eth0 parent 1:1 classid 1:8 htb rate 150kbit ceil 150kbit

tc qdisc add dev eth0 parent 1:8 handle 8: sfq = perturb 10

tc class add dev eth1 parent 1:1 classid 1:8 htb rate = 50kbit ceil 50kbit

tc qdisc add dev eth1 parent 1:8 handle 8: sfq = perturb 10

iptables -t mangle -A POSTROUTING --dest y.y.y.y -o = eth0 -j CLASSIFY --set-class 1:8

iptables -t mangle -A FORWARD --src y.y.y.y -o eth1 = -j CLASSIFY --set-class 1:8

 

(…)

 

This rules works fine, but just for less than 1.700 customers. More than 1.700 customers, i have my load avarage in the sky = and Ksoftirqd process (top information) in 100% fulltime. I don’t know = why. I used to use CBQ instead HTB because i had the same problem and Ron (a = guy in this list) gave this rules and told me that he uses this for more than = 3.000 customers. I tested it in more than 7 different computers (but the same = hadware specifications) and i had the same problem with either CBQ or HTB rules. = The computers that i have are all of them DELL PowerEdge 1850. I will put = some hardware iformations here:

 

top

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

    3 root      39  19     0    0    0 R  100  = 0.0   5316:20 ksoftirqd/0

 

[root@fw ~]# uptime

 10:38:11 up 161 days, 17:21,  3 = users,  load average: 1.58, 1.65, 1.51   (unfortunately when i took this, = the load average was „pretty good“, but minutes ago, it was more than = 11.0

 

[root@fw ~]# = lspci

00:00.0 Host bridge: Intel Corporation E7520 = Memory Controller Hub (rev 09)

00:02.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express Port A (rev = 09)

00:04.0 PCI bridge: Intel Corporation = E7525/E7520 PCI Express Port B (rev 09)

00:05.0 PCI bridge: Intel Corporation E7520 = PCI Express Port B1 (rev 09)

00:06.0 PCI bridge: Intel Corporation E7520 = PCI Express Port C (rev 09)

00:1d.0 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #1 (rev = 02)

00:1d.1 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #2 (rev = 02)

00:1d.2 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #3 (rev = 02)

00:1d.7 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB2 EHCI Controller (rev 02)

00:1e.0 PCI bridge: Intel Corporation 82801 PCI = Bridge (rev c2)

00:1f.0 ISA bridge: Intel Corporation 82801EB/ER (ICH5/ICH5R) LPC Interface Bridge (rev 02)

00:1f.1 IDE interface: Intel Corporation 82801EB/ER (ICH5/ICH5R) IDE Controller (rev 02)

01:00.0 PCI bridge: Intel Corporation 80332 = [Dobson] I/O processor (A-Segment Bridge) (rev 06)

01:00.2 PCI bridge: Intel Corporation 80332 = [Dobson] I/O processor (B-Segment Bridge) (rev 06)

02:0c.0 Ethernet controller: Intel Corporation = 82545GM Gigabit Ethernet Controller (rev 04)

02:0e.0 RAID bus controller: Dell PowerEdge = Expandable RAID controller 4 (rev 06)

03:0b.0 Ethernet controller: Intel Corporation = 82545GM Gigabit Ethernet Controller (rev 04)

05:00.0 PCI bridge: Intel Corporation 6700PXH = PCI Express-to-PCI Bridge A (rev 09)

05:00.2 PCI bridge: Intel Corporation 6700PXH = PCI Express-to-PCI Bridge B (rev 09)

06:07.0 Ethernet controller: Intel Corporation = 82541GI/PI Gigabit Ethernet Controller (rev 05)

07:08.0 Ethernet controller: Intel Corporation = 82541GI/PI Gigabit Ethernet Controller (rev 05)

09:0d.0 VGA compatible controller: ATI Technologies = Inc Radeon RV100 QY [Radeon 7000/VE]

 

[root@fw ~]# free -m

         =     total       used       free     shared    buffers     = cached

Mem:        &n= bsp; 2021       1479        542          0        = 400        654

-/+ = buffers/cache:        424       = 1597

Swap:         1027          0       1027

 

[root@fw ~]# cat = /proc/cpuinfo

processor       = : 0

vendor_id       = : GenuineIntel

cpu family      : = 15

model        &= nbsp;  : 4

model name      : Intel(R) = Xeon(TM) CPU 3.00GHz

stepping        : = 3

cpu = MHz         : 2992.674

cache size      : 2048 = KB

physical id     : = 0

siblings      &nb= sp; : 2

core id         : = 0

cpu cores       = : 1

fdiv_bug      &nb= sp; : no

hlt_bug      &nbs= p;  : no

f00f_bug      &nb= sp; : no

coma_bug      &nb= sp; : no

fpu       &n= bsp;     : yes

fpu_exception   : = yes

cpuid level     : = 5

wp        &nbs= p;     : yes

flags        &= nbsp;  : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat = pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pni = monitor ds_cpl cid cx16 xtpr

bogomips      &nb= sp; : 5990.78

 

processor       = : 1

vendor_id       = : GenuineIntel

cpu family      : = 15

model        &= nbsp;  : 4

model name      : Intel(R) = Xeon(TM) CPU 3.00GHz

stepping        : = 3

cpu = MHz         : 2992.674

cache size      : 2048 = KB

physical id     : = 0

siblings      &nb= sp; : 2

core id         : = 0

cpu cores       = : 1

fdiv_bug      =   : no

hlt_bug      &nbs= p;  : no

f00f_bug      &nb= sp; : no

coma_bug      &nb= sp; : no

fpu       &n= bsp;     : yes

fpu_exception   : = yes

cpuid level     : = 5

wp       &nb= sp;      : yes

flags       =     : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat = pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pni = monitor ds_cpl cid cx16 xtpr

bogomips        : = 5985.13

 

 

Any help/Tipp/hint will be very = welcome.

 

Thanks in Advance!

 

Pablo Fernandes

 

------=_NextPart_000_0009_01C79F84.2F6E0F70-- _______________________________________________________ Yahoo! Mail - Sempre a melhor opgco para vocj! Experimente ja e veja as novidades. http://br.yahoo.com/mailbeta/tudonovo/ --===============1619405892== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc --===============1619405892==-- _______________________________________________________ Yahoo! Mail - Sempre a melhor opgco para vocj! Experimente ja e veja as novidades. http://br.yahoo.com/mailbeta/tudonovo/