Skip to contentSkip to conversation

big problem with HTB/CBQ and CPU for more than 1.700

LARTC Pablo Fernandes Yahoo 7 participants 13 messages May 26, 2007 – Jun 1, 2007

lore
Share & tools
lore
#0 lore raw

Hello,

have HTB "rules" in 4 different ISPs and i control for each customer this way:

Flush and 1:0 class

tc qdisc del dev eth0 root

tc qdisc add dev eth0 root handle 1:0 htb

tc class add dev eth0 parent 1:0 classid 1:1 htb rate 100mbit

tc qdisc del dev eth1 root

tc qdisc add dev eth1 root handle 1:0 htb

tc class add dev eth1 parent 1:0 classid 1:1 htb rate 100mbit

Upload and Download: user1

tc class add dev eth0 parent 1:1 classid 1:5 htb rate 150kbit ceil 150kbit

tc qdisc add dev eth0 parent 1:5 handle 5: sfq perturb 10

tc class add dev eth1 parent 1:1 classid 1:5 htb rate 50kbit ceil 50kbit

tc qdisc add dev eth1 parent 1:5 handle 5: sfq perturb 10

iptables -t mangle -A POSTROUTING --dest x.x.x.x -o eth0 -j CLASSIFY --set-class 1:5

iptables -t mangle -A FORWARD --src x.x.x.x -o eth1 -j CLASSIFY --set-class 1:5

Upload and Download: user2

tc class add dev eth0 parent 1:1 classid 1:8 htb rate 150kbit ceil 150kbit

tc qdisc add dev eth0 parent 1:8 handle 8: sfq perturb 10

tc class add dev eth1 parent 1:1 classid 1:8 htb rate 50kbit ceil 50kbit

tc qdisc add dev eth1 parent 1:8 handle 8: sfq perturb 10

iptables -t mangle -A POSTROUTING --dest y.y.y.y -o eth0 -j CLASSIFY --set-class 1:8

iptables -t mangle -A FORWARD --src y.y.y.y -o eth1 -j CLASSIFY --set-class 1:8

(.)

This rules works fine, but just for less than 1.700 customers. More than 1.700 customers, i have my load avarage in the sky and Ksoftirqd process (top information) in 100% fulltime. I don't know why. I used to use CBQ instead HTB because i had the same problem and Ron (a guy in this list) gave this rules and told me that he uses this for more than 3.000 customers. I tested it in more than 7 different computers (but the same hadware specifications) and i had the same problem with either CBQ or HTB rules. The computers that i have are all of them DELL PowerEdge 1850. I will put some hardware iformations here:

top

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

    3 root      39  19     0    0    0 R  100  0.0   5316:20 ksoftirqd/0

[root@fw ~]# uptime

10:38:11 up 161 days, 17:21, 3 users, load average: 1.58, 1.65, 1.51
(unfortunately when i took this, the load average was "pretty good", but minutes ago, it was more than 11.0

[root@fw ~]# lspci

00:00.0 Host bridge: Intel Corporation E7520 Memory Controller Hub (rev 09)

00:02.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express Port A (rev 09)

00:04.0 PCI bridge: Intel Corporation E7525/E7520 PCI Express Port B (rev
09)

00:05.0 PCI bridge: Intel Corporation E7520 PCI Express Port B1 (rev 09)

00:06.0 PCI bridge: Intel Corporation E7520 PCI Express Port C (rev 09)

00:1d.0 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #1 (rev 02)

00:1d.1 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #2 (rev 02)

00:1d.2 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #3 (rev 02)

00:1d.7 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB2 EHCI Controller (rev 02)

00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c2)

00:1f.0 ISA bridge: Intel Corporation 82801EB/ER (ICH5/ICH5R) LPC Interface Bridge (rev 02)

00:1f.1 IDE interface: Intel Corporation 82801EB/ER (ICH5/ICH5R) IDE Controller (rev 02)

01:00.0 PCI bridge: Intel Corporation 80332 [Dobson] I/O processor (A-Segment Bridge) (rev 06)

01:00.2 PCI bridge: Intel Corporation 80332 [Dobson] I/O processor (B-Segment Bridge) (rev 06)

02:0c.0 Ethernet controller: Intel Corporation 82545GM Gigabit Ethernet Controller (rev 04)

02:0e.0 RAID bus controller: Dell PowerEdge Expandable RAID controller 4 (rev 06)

03:0b.0 Ethernet controller: Intel Corporation 82545GM Gigabit Ethernet Controller (rev 04)

05:00.0 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge A (rev 09)

05:00.2 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge B (rev 09)

06:07.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit Ethernet Controller (rev 05)

07:08.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit Ethernet Controller (rev 05)

09:0d.0 VGA compatible controller: ATI Technologies Inc Radeon RV100 QY [Radeon 7000/VE]

[root@fw ~]# free -m

total used free shared buffers cached

Mem: 2021 1479 542 0 400 654

-/+ buffers/cache: 424 1597

Swap: 1027 0 1027

[root@fw ~]# cat /proc/cpuinfo

processor : 0

vendor_id : GenuineIntel

cpu family : 15

model : 4

model name : Intel(R) Xeon(TM) CPU 3.00GHz

stepping : 3

cpu MHz : 2992.674

cache size : 2048 KB

physical id : 0

siblings : 2

core id : 0

cpu cores : 1

fdiv_bug : no

hlt_bug : no

f00f_bug : no

coma_bug : no

fpu : yes

fpu_exception : yes

cpuid level : 5

wp : yes

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pni monitor ds_cpl cid cx16 xtpr

bogomips : 5990.78

processor : 1

vendor_id : GenuineIntel

cpu family : 15

model : 4

model name : Intel(R) Xeon(TM) CPU 3.00GHz

stepping : 3

cpu MHz : 2992.674

cache size : 2048 KB

physical id : 0

siblings : 2

core id : 0

cpu cores : 1

fdiv_bug : no

hlt_bug : no

f00f_bug : no

coma_bug : no

fpu : yes

fpu_exception : yes

cpuid level : 5

wp : yes

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pni monitor ds_cpl cid cx16 xtpr

bogomips : 5985.13

Any help/Tipp/hint will be very welcome.

Thanks in Advance!

Pablo Fernandes

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#1 lore raw
Quoted 1 line

Hello,

Hi there!

Quoted 4 lines

iptables -t mangle -A POSTROUTING --dest x.x.x.x -o eth0 -j CLASSIFY --set-class 1:5
iptables -t mangle -A FORWARD --src x.x.x.x -o eth1 -j CLASSIFY --set-class 1:5

3k iptables rules strike me as something suicidaly slow.

Try using tc hashing filters for traffic classification as described here:
http://lartc.org/howto/lartc.adv-filter.hashing.html

If you use private addresses and NAT then you'll need IFB
(http://linux-net.osdl.org/index.php/IFB) to shape upload per client with u32 hashing filters.

Hope that helps.

pozdrawiam,
Marek Kierdelewicz
KoBa ISP
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#2 lore raw
In reply to Pablo Fernandes Yahoo

Pablo Fernandes Yahoo написа:

Quoted 249 lines

Hello,

have HTB „rules“ in 4 different ISPs and i control for each customer this way:

Flush and 1:0 class

tc qdisc del dev eth0 root

tc qdisc add dev eth0 root handle 1:0 htb

tc class add dev eth0 parent 1:0 classid 1:1 htb rate 100mbit

tc qdisc del dev eth1 root

tc qdisc add dev eth1 root handle 1:0 htb

tc class add dev eth1 parent 1:0 classid 1:1 htb rate 100mbit

Upload and Download: user1

tc class add dev eth0 parent 1:1 classid 1:5 htb rate 150kbit ceil 150kbit

tc qdisc add dev eth0 parent 1:5 handle 5: sfq perturb 10

tc class add dev eth1 parent 1:1 classid 1:5 htb rate 50kbit ceil 50kbit

tc qdisc add dev eth1 parent 1:5 handle 5: sfq perturb 10

iptables -t mangle -A POSTROUTING --dest x.x.x.x -o eth0 -j CLASSIFY --set-class 1:5

iptables -t mangle -A FORWARD --src x.x.x.x -o eth1 -j CLASSIFY --set-class 1:5

Upload and Download: user2

tc class add dev eth0 parent 1:1 classid 1:8 htb rate 150kbit ceil 150kbit

tc qdisc add dev eth0 parent 1:8 handle 8: sfq perturb 10

tc class add dev eth1 parent 1:1 classid 1:8 htb rate 50kbit ceil 50kbit

tc qdisc add dev eth1 parent 1:8 handle 8: sfq perturb 10

iptables -t mangle -A POSTROUTING --dest y.y.y.y -o eth0 -j CLASSIFY --set-class 1:8

iptables -t mangle -A FORWARD --src y.y.y.y -o eth1 -j CLASSIFY --set-class 1:8

(…)

This rules works fine, but just for less than 1.700 customers. More than 1.700 customers, i have my load avarage in the sky and Ksoftirqd process (top information) in 100% fulltime. I don’t know why. I used to use CBQ instead HTB because i had the same problem and Ron (a guy in this list) gave this rules and told me that he uses this for more than 3.000 customers. I tested it in more than 7 different computers (but the same hadware specifications) and i had the same problem with either CBQ or HTB rules. The computers that i have are all of them DELL PowerEdge 1850. I will put some hardware iformations here:

top

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

3 root 39 19 0 0 0 R 100 0.0 5316:20 ksoftirqd/0

[root@fw ~]# uptime

10:38:11 up 161 days, 17:21, 3 users, load average: 1.58, 1.65, 1.51 (unfortunately when i took this, the load average was „pretty good“, but minutes ago, it was more than 11.0

[root@fw ~]# lspci

00:00.0 Host bridge: Intel Corporation E7520 Memory Controller Hub (rev 09)

00:02.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express Port A (rev 09)

00:04.0 PCI bridge: Intel Corporation E7525/E7520 PCI Express Port B (rev 09)

00:05.0 PCI bridge: Intel Corporation E7520 PCI Express Port B1 (rev 09)

00:06.0 PCI bridge: Intel Corporation E7520 PCI Express Port C (rev 09)

00:1d.0 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #1 (rev 02)

00:1d.1 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #2 (rev 02)

00:1d.2 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #3 (rev 02)

00:1d.7 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB2 EHCI Controller (rev 02)

00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c2)

00:1f.0 ISA bridge: Intel Corporation 82801EB/ER (ICH5/ICH5R) LPC Interface Bridge (rev 02)

00:1f.1 IDE interface: Intel Corporation 82801EB/ER (ICH5/ICH5R) IDE Controller (rev 02)

01:00.0 PCI bridge: Intel Corporation 80332 [Dobson] I/O processor (A-Segment Bridge) (rev 06)

01:00.2 PCI bridge: Intel Corporation 80332 [Dobson] I/O processor (B-Segment Bridge) (rev 06)

02:0c.0 Ethernet controller: Intel Corporation 82545GM Gigabit Ethernet Controller (rev 04)

02:0e.0 RAID bus controller: Dell PowerEdge Expandable RAID controller 4 (rev 06)

03:0b.0 Ethernet controller: Intel Corporation 82545GM Gigabit Ethernet Controller (rev 04)

05:00.0 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge A (rev 09)

05:00.2 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge B (rev 09)

06:07.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit Ethernet Controller (rev 05)

07:08.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit Ethernet Controller (rev 05)

09:0d.0 VGA compatible controller: ATI Technologies Inc Radeon RV100 QY [Radeon 7000/VE]

[root@fw ~]# free -m

total used free shared buffers cached

Mem: 2021 1479 542 0 400 654

-/+ buffers/cache: 424 1597

Swap: 1027 0 1027

[root@fw ~]# cat /proc/cpuinfo

processor : 0

vendor_id : GenuineIntel

cpu family : 15

model : 4

model name : Intel(R) Xeon(TM) CPU 3.00GHz

stepping : 3

cpu MHz : 2992.674

cache size : 2048 KB

physical id : 0

siblings : 2

core id : 0

cpu cores : 1

fdiv_bug : no

hlt_bug : no

f00f_bug : no

coma_bug : no

fpu : yes

fpu_exception : yes

cpuid level : 5

wp : yes

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pni monitor ds_cpl cid cx16 xtpr

bogomips : 5990.78

processor : 1

vendor_id : GenuineIntel

cpu family : 15

model : 4

model name : Intel(R) Xeon(TM) CPU 3.00GHz

stepping : 3

cpu MHz : 2992.674

cache size : 2048 KB

physical id : 0

siblings : 2

core id : 0

cpu cores : 1

fdiv_bug : no

hlt_bug : no

f00f_bug : no

coma_bug : no

fpu : yes

fpu_exception : yes

cpuid level : 5

wp : yes

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pni monitor ds_cpl cid cx16 xtpr

bogomips : 5985.13

Any help/Tipp/hint will be very welcome.

Thanks in Advance!

Quoted 9 lines

Pablo Fernandes

------------------------------------------------------------------------

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

You may find this:
http://openfmi.net/frs/download.php/410/IPCLASSIFY.zip
useful
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#3 lore raw
In reply to Pablo Fernandes Yahoo

Pablo,

Here we have HTB being used for more than 10.000 customers. The difference, is that we use tc and u32 filters to classify the packets..

I use the same Dell PE 1850, but I have two Quad-Core Xeon (1.86GHz) on it :)

# uptime
13:18:08 up 16 days, 12:32,  1 user,  load average: 0.02, 0.02, 0.00

mpstat says:

01:19:11 PM CPU %user %nice %sys %iowait %irq %soft %
steal %idle intr/s
01:19:13 PM all 0.00 0.00 0.00 0.00 0.57 13.81
0.00 85.61 10568.88

And as you can see.. the use of cpu is not that big..

Quoted 290 lines · Pablo Fernandes Yahoo

Hello,

have HTB „rules“ in 4 different ISPs and i control for each customer this way:

Flush and 1:0 class

tc qdisc del dev eth0 root

tc qdisc add dev eth0 root handle 1:0 htb

tc class add dev eth0 parent 1:0 classid 1:1 htb rate 100mbit

tc qdisc del dev eth1 root

tc qdisc add dev eth1 root handle 1:0 htb

tc class add dev eth1 parent 1:0 classid 1:1 htb rate 100mbit

Upload and Download: user1

tc class add dev eth0 parent 1:1 classid 1:5 htb rate 150kbit ceil 150kbit

tc qdisc add dev eth0 parent 1:5 handle 5: sfq perturb 10

tc class add dev eth1 parent 1:1 classid 1:5 htb rate 50kbit ceil 50kbit

tc qdisc add dev eth1 parent 1:5 handle 5: sfq perturb 10

iptables -t mangle -A POSTROUTING --dest x.x.x.x -o eth0 -j CLASSIFY --set-class 1:5

iptables -t mangle -A FORWARD --src x.x.x.x -o eth1 -j CLASSIFY -- set-class 1:5

Upload and Download: user2

tc class add dev eth0 parent 1:1 classid 1:8 htb rate 150kbit ceil 150kbit

tc qdisc add dev eth0 parent 1:8 handle 8: sfq perturb 10

tc class add dev eth1 parent 1:1 classid 1:8 htb rate 50kbit ceil 50kbit

tc qdisc add dev eth1 parent 1:8 handle 8: sfq perturb 10

iptables -t mangle -A POSTROUTING --dest y.y.y.y -o eth0 -j CLASSIFY --set-class 1:8

iptables -t mangle -A FORWARD --src y.y.y.y -o eth1 -j CLASSIFY -- set-class 1:8

(…)

This rules works fine, but just for less than 1.700 customers. More than 1.700 customers, i have my load avarage in the sky and Ksoftirqd process (top information) in 100% fulltime. I don’t know why. I used to use CBQ instead HTB because i had the same problem and Ron (a guy in this list) gave this rules and told me that he uses this for more than 3.000 customers. I tested it in more than 7 different computers (but the same hadware specifications) and i had the same problem with either CBQ or HTB rules. The computers that i have are all of them DELL PowerEdge 1850. I will put some hardware iformations here:

top

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

    3 root      39  19     0    0    0 R  100  0.0   5316:20  

ksoftirqd/0

[root@fw ~]# uptime

10:38:11 up 161 days, 17:21, 3 users, load average: 1.58, 1.65,
1.51 (unfortunately when i took this, the load average was
„pretty good“, but minutes ago, it was more than 11.0

[root@fw ~]# lspci

00:00.0 Host bridge: Intel Corporation E7520 Memory Controller Hub (rev 09)

00:02.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express Port A (rev 09)

00:04.0 PCI bridge: Intel Corporation E7525/E7520 PCI Express Port B (rev 09)

00:05.0 PCI bridge: Intel Corporation E7520 PCI Express Port B1 (rev 09)

00:06.0 PCI bridge: Intel Corporation E7520 PCI Express Port C (rev
09)

00:1d.0 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #1 (rev 02)

00:1d.1 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #2 (rev 02)

00:1d.2 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #3 (rev 02)

00:1d.7 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB2 EHCI Controller (rev 02)

00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c2)

00:1f.0 ISA bridge: Intel Corporation 82801EB/ER (ICH5/ICH5R) LPC Interface Bridge (rev 02)

00:1f.1 IDE interface: Intel Corporation 82801EB/ER (ICH5/ICH5R) IDE Controller (rev 02)

01:00.0 PCI bridge: Intel Corporation 80332 [Dobson] I/O processor (A-Segment Bridge) (rev 06)

01:00.2 PCI bridge: Intel Corporation 80332 [Dobson] I/O processor (B-Segment Bridge) (rev 06)

02:0c.0 Ethernet controller: Intel Corporation 82545GM Gigabit Ethernet Controller (rev 04)

02:0e.0 RAID bus controller: Dell PowerEdge Expandable RAID controller 4 (rev 06)

03:0b.0 Ethernet controller: Intel Corporation 82545GM Gigabit Ethernet Controller (rev 04)

05:00.0 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge A (rev 09)

05:00.2 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge B (rev 09)

06:07.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit Ethernet Controller (rev 05)

07:08.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit Ethernet Controller (rev 05)

09:0d.0 VGA compatible controller: ATI Technologies Inc Radeon RV100 QY [Radeon 7000/VE]

[root@fw ~]# free -m

total used free shared buffers
cached

Mem: 2021 1479 542 0
400 654

-/+ buffers/cache: 424 1597

Swap: 1027 0 1027

[root@fw ~]# cat /proc/cpuinfo

processor : 0

vendor_id : GenuineIntel

cpu family : 15

model : 4

model name : Intel(R) Xeon(TM) CPU 3.00GHz

stepping : 3

cpu MHz : 2992.674

cache size : 2048 KB

physical id : 0

siblings : 2

core id : 0

cpu cores : 1

fdiv_bug : no

hlt_bug : no

f00f_bug : no

coma_bug : no

fpu : yes

fpu_exception : yes

cpuid level : 5

wp : yes

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr
pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pni monitor ds_cpl cid cx16 xtpr

bogomips : 5990.78

processor : 1

vendor_id : GenuineIntel

cpu family : 15

model : 4

model name : Intel(R) Xeon(TM) CPU 3.00GHz

stepping : 3

cpu MHz : 2992.674

cache size : 2048 KB

physical id : 0

siblings : 2

core id : 0

cpu cores : 1

fdiv_bug : no

hlt_bug : no

f00f_bug : no

coma_bug : no

fpu : yes

fpu_exception : yes

cpuid level : 5

wp : yes

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr
pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pni monitor ds_cpl cid cx16 xtpr

bogomips : 5985.13

Any help/Tipp/hint will be very welcome.

Thanks in Advance!

Quoted 8 lines

Pablo Fernandes

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

--
Acácio Alves dos Santos
Administração de redes
Diginet Brasil
[email protected]
(+55) 84 4008-9000

Esta mensagem, incluindo seus anexos, pode conter informação confidencial e/ou
privilegiada. Se você não for o destinatário ou a pessoa autorizada a receber
esta mensagem, não pode usar, copiar ou divulgar as informações nela contidas ou tomar qualquer ação baseada nessas informações. Se você recebeu esta mensagem por engano, por favor avise imediatamente o remetente, respondendo o e-mail e em seguida apague-o. Agradecemos sua cooperação.

This message, including its attatchments, may contain confidential and/or
privileged information. If you are not the recipient or authorized person to
receive this message, you must not use, copy, disclose or take any action based
on this message or any information herein. If you received this message by mistake, please advise the sender immediately by replying the e- mail and
deleting this message. Thank you for your cooperation.

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#4 lore raw
In reply to Pablo Fernandes Yahoo

Hello everybody,

I have a problem which is similar:

450 customers with 450 HTB classes and their corresponding 450 filters in a subclass (1:3) of the root qdisc (PRIO) of the interface.

The problem is manifesting at times, when I try to ping a host behind the router from a host before the router, the latency becomes 1-1,5 msec.

On the machine is running also iptables firewall with a bunch of rules for dropping/accepting/natting specific traffic, plus routing about 30 Mbits/sec.

When I remove the HTB qdisc, the latency is normal, 0,3-0,4 msec.

Anyone has an idea what could cause this?

Any input much appreciated!

Regards,
Stoimen

--------------
Pablo,

Here we have HTB being used for more than 10.000 customers. The difference, is that we use tc and u32 filters to classify the packets..

I use the same Dell PE 1850, but I have two Quad-Core Xeon (1.86GHz) on it :)

# uptime
13:18:08 up 16 days, 12:32,  1 user,  load average: 0.02, 0.02, 0.00

mpstat says:

01:19:11 PM CPU %user %nice %sys %iowait %irq %soft %
steal %idle intr/s
01:19:13 PM all 0.00 0.00 0.00 0.00 0.57 13.81
0.00 85.61 10568.88

And as you can see.. the use of cpu is not that big..

Quoted 286 lines · Pablo Fernandes Yahoo

Hello,

have HTB „rules“ in 4 different ISPs and i control for each customer this way:

Flush and 1:0 class

tc qdisc del dev eth0 root

tc qdisc add dev eth0 root handle 1:0 htb

tc class add dev eth0 parent 1:0 classid 1:1 htb rate 100mbit

tc qdisc del dev eth1 root

tc qdisc add dev eth1 root handle 1:0 htb

tc class add dev eth1 parent 1:0 classid 1:1 htb rate 100mbit

Upload and Download: user1

tc class add dev eth0 parent 1:1 classid 1:5 htb rate 150kbit ceil 150kbit

tc qdisc add dev eth0 parent 1:5 handle 5: sfq perturb 10

tc class add dev eth1 parent 1:1 classid 1:5 htb rate 50kbit ceil 50kbit

tc qdisc add dev eth1 parent 1:5 handle 5: sfq perturb 10

iptables -t mangle -A POSTROUTING --dest x.x.x.x -o eth0 -j CLASSIFY --set-class 1:5

iptables -t mangle -A FORWARD --src x.x.x.x -o eth1 -j CLASSIFY -- set-class 1:5

Upload and Download: user2

tc class add dev eth0 parent 1:1 classid 1:8 htb rate 150kbit ceil 150kbit

tc qdisc add dev eth0 parent 1:8 handle 8: sfq perturb 10

tc class add dev eth1 parent 1:1 classid 1:8 htb rate 50kbit ceil 50kbit

tc qdisc add dev eth1 parent 1:8 handle 8: sfq perturb 10

iptables -t mangle -A POSTROUTING --dest y.y.y.y -o eth0 -j CLASSIFY --set-class 1:8

iptables -t mangle -A FORWARD --src y.y.y.y -o eth1 -j CLASSIFY -- set-class 1:8

(…)

This rules works fine, but just for less than 1.700 customers. More than 1.700 customers, i have my load avarage in the sky and Ksoftirqd process (top information) in 100% fulltime. I don’t know why. I used to use CBQ instead HTB because i had the same problem and Ron (a guy in this list) gave this rules and told me that he uses this for more than 3.000 customers. I tested it in more than 7 different computers (but the same hadware specifications) and i had the same problem with either CBQ or HTB rules. The computers that i have are all of them DELL PowerEdge 1850. I will put some hardware iformations here:

top

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

    3 root      39  19     0    0    0 R  100  0.0   5316:20  

ksoftirqd/0

[root at fw ~]# uptime

10:38:11 up 161 days, 17:21, 3 users, load average: 1.58, 1.65,
1.51 (unfortunately when i took this, the load average was
„pretty good“, but minutes ago, it was more than 11.0

[root at fw ~]# lspci

00:00.0 Host bridge: Intel Corporation E7520 Memory Controller Hub (rev 09)

00:02.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express Port A (rev 09)

00:04.0 PCI bridge: Intel Corporation E7525/E7520 PCI Express Port B (rev 09)

00:05.0 PCI bridge: Intel Corporation E7520 PCI Express Port B1 (rev 09)

00:06.0 PCI bridge: Intel Corporation E7520 PCI Express Port C (rev
09)

00:1d.0 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #1 (rev 02)

00:1d.1 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #2 (rev 02)

00:1d.2 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #3 (rev 02)

00:1d.7 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB2 EHCI Controller (rev 02)

00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c2)

00:1f.0 ISA bridge: Intel Corporation 82801EB/ER (ICH5/ICH5R) LPC Interface Bridge (rev 02)

00:1f.1 IDE interface: Intel Corporation 82801EB/ER (ICH5/ICH5R) IDE Controller (rev 02)

01:00.0 PCI bridge: Intel Corporation 80332 [Dobson] I/O processor (A-Segment Bridge) (rev 06)

01:00.2 PCI bridge: Intel Corporation 80332 [Dobson] I/O processor (B-Segment Bridge) (rev 06)

02:0c.0 Ethernet controller: Intel Corporation 82545GM Gigabit Ethernet Controller (rev 04)

02:0e.0 RAID bus controller: Dell PowerEdge Expandable RAID controller 4 (rev 06)

03:0b.0 Ethernet controller: Intel Corporation 82545GM Gigabit Ethernet Controller (rev 04)

05:00.0 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge A (rev 09)

05:00.2 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge B (rev 09)

06:07.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit Ethernet Controller (rev 05)

07:08.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit Ethernet Controller (rev 05)

09:0d.0 VGA compatible controller: ATI Technologies Inc Radeon RV100 QY [Radeon 7000/VE]

[root at fw ~]# free -m

total used free shared buffers
cached

Mem: 2021 1479 542 0
400 654

-/+ buffers/cache: 424 1597

Swap: 1027 0 1027

[root at fw ~]# cat /proc/cpuinfo

processor : 0

vendor_id : GenuineIntel

cpu family : 15

model : 4

model name : Intel(R) Xeon(TM) CPU 3.00GHz

stepping : 3

cpu MHz : 2992.674

cache size : 2048 KB

physical id : 0

siblings : 2

core id : 0

cpu cores : 1

fdiv_bug : no

hlt_bug : no

f00f_bug : no

coma_bug : no

fpu : yes

fpu_exception : yes

cpuid level : 5

wp : yes

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr
pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pni monitor ds_cpl cid cx16 xtpr

bogomips : 5990.78

processor : 1

vendor_id : GenuineIntel

cpu family : 15

model : 4

model name : Intel(R) Xeon(TM) CPU 3.00GHz

stepping : 3

cpu MHz : 2992.674

cache size : 2048 KB

physical id : 0

siblings : 2

core id : 0

cpu cores : 1

fdiv_bug : no

hlt_bug : no

f00f_bug : no

coma_bug : no

fpu : yes

fpu_exception : yes

cpuid level : 5

wp : yes

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr
pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pni monitor ds_cpl cid cx16 xtpr

bogomips : 5985.13

Any help/Tipp/hint will be very welcome.

Quoted 5 lines

Thanks in Advance!

Pablo Fernandes

---------------------------------------------------
Webmail of Bulsat Ltd. at http://mail.bulsattv.com/

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#5 lore raw
In reply to Pablo Fernandes Yahoo

As i told before, i tryed to shape my traffic CBQ and i did use u32 filters. the results were exactly the same as HTB using u32 filters (i tryed before too) or not.

Do you have in your HTB setup a class per customer? I've seen different setups, but all of them shaping the traffic either based in protocols or/and IP ranges (it isn't our reality whilst we have at least a class per each single IP within the network).

After some readings, i'm starting to suspect about my NIC driver (e1000). Do you have in your Dell PE 1850 interfaces using the e1000 driver? Is the entire traffic passing by this server? I suppose that problem is something about the hardware or software interruptions. Are you using the default parameters for the e1000 kernel module?

Regards

Quoted 5 lines

Pablo,

Here we have HTB being used for more than 10.000 customers. The difference,

is that we use tc and u32 filters to classify >the packets..

Quoted 3 lines

I use the same Dell PE 1850, but I have two Quad-Core Xeon (1.86GHz) on it

:)

Quoted 16 lines

# uptime

13:18:08 up 16 days, 12:32, 1 user, load average: 0.02, 0.02, 0.00

mpstat says:

01:19:11 PM CPU %user %nice %sys %iowait %irq %soft %

steal %idle intr/s

Quoted 7 lines

01:19:13 PM all 0.00 0.00 0.00 0.00 0.57 13.81

0.00 85.61 10568.88

And as you can see.. the use of cpu is not that big..

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#6 lore raw
In reply to Pablo Fernandes Yahoo

Hey,

I'm definately glad because i can see that someone else knows what is happening here. Thank for all the help and also i'm here to help anyone as much as i can.

So, refreshing my current setup, i have this rules for each customer:

tc qdisc del dev eth0 root
tc qdisc add dev eth0 root handle 1:0 htb
tc class add dev eth0 parent 1:0 classid 1:1 htb rate 100mbit

tc qdisc del dev eth1 root
tc qdisc add dev eth1 root handle 1:0 htb
tc class add dev eth1 parent 1:0 classid 1:1 htb rate 100mbit

user 1
tc class add dev eth0 parent 1:1 classid 1:5 htb rate 150kbit ceil 150kbit tc qdisc add dev eth0 parent 1:5 handle 5: sfq perturb 10

tc class add dev eth1 parent 1:1 classid 1:5 htb rate 50kbit ceil 50kbit tc qdisc add dev eth1 parent 1:5 handle 5: sfq perturb 10
iptables -t mangle -A POSTROUTING --dest 10.30.0.54 -o eth0 -j CLASSIFY --set-class 1:5 iptables -t mangle -A FORWARD --src 10.30.0.54 -o eth1 -j CLASSIFY --set-class 1:5

user n
tc class add dev eth0 parent 1:1 classid 1:8 htb rate 150kbit ceil 150kbit tc qdisc add dev eth0 parent 1:8 handle 8: sfq perturb 10

tc class add dev eth1 parent 1:1 classid 1:8 htb rate 50kbit ceil 50kbit tc qdisc add dev eth1 parent 1:8 handle 8: sfq perturb 10
iptables -t mangle -A POSTROUTING --dest 10.20.0.43 -o eth0 -j CLASSIFY --set-class 1:8 iptables -t mangle -A FORWARD --src 10.20.0.43 -o eth1 -j CLASSIFY --set-class 1:8

what u32 rules could replace these iptables rules? I would like to try u32 filters and see if them will solve the problem, if i had no success, i will try the IPCLASSIFY patch.

Thanks again in Advance.

Regards

Pablo Fernandes

-----Ursprüngliche Nachricht-----
Von: VladSun [mailto:[email protected]]
Gesendet: segunda-feira, 28 de maio de 2007 14:39
An: Alexandru Dragoi
Cc: Pablo Fernandes Yahoo; [email protected]
Betreff: Re: [LARTC] big problem with HTB/CBQ and CPU for more than 1.700 customers

Alexandru Dragoi напи�а:

Quoted 15 lines

u32 hash filters is the key, as somebody pointed. You can also tune your iptables setup, like this

#192.168.1.0/24
iptables -t mangle -N 192-168-1-0-24
iptables -t mangle -A FORWARD -s 192.168.1.0/24 -j 192-168-1-0-24 iptables -t mangle -N 192-168-1-0-25
iptables -t mangle -N 192-168-1-128-25
iptables -t mangle -A 192-168-1-0-24 -s 192.168.1.0/25 -j 192-168-1-0-25 iptables -t mangle -A 192-168-1-0-24 -s 192.168.128.0/25 -j 192-168-1-128-25 .
.
and so on, until (ip 192.168.1.11, which is called in chain created for 192.168.1.10/31)

Quoted 9 lines

iptables -t mangle -A 192-168-1-10-31 -s 192.168.1.10 -j CLASSIFY --set-class 1:10
iptables -t mangle -A 192-168-1-10-31 -s 192.168.1.11 -j CLASSIFY --set-class 1:11

.. I guess you got the ideea, it requires some RAM, which i belive is not such a big problem. Similar rules should be made for download.

Or you can use my patch - IPCLASSIFY. Then the rules above would be substituted by a signle rule per direction:

iptables -t mangle -A FORWARD -s 192.168.1.0/24 -j IPCLASSIFY --addr=src --and-mask=0xff --or-mask=0x11000
iptables -t mangle -A FORWARD -d 192.168.1.0/24 -j IPCLASSIFY --addr=dst --and-mask=0xff --or-mask=0x12000

This is equal to applying CLASSIFY target to each packet with --set-class (srcIP & 0xFF | 0x1100 ) and --set-class (dstIP & 0xFF | 0x1200 ).
It is very similar to IPMARK, but it uses skb->priority field instead mark. So no tc filters are needed.

_______________________________________________________
Yahoo! Mail - Sempre a melhor opção para você!
Experimente já e veja as novidades.
http://br.yahoo.com/mailbeta/tudonovo/
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#7 lore raw
In reply to Pablo Fernandes Yahoo

I have classes configured by protocol..

My NIC is the e1000 too (default parameters), but I'm using off-board cards (with scalable i/o support - good for multi-core setups). The main problem I had was the number of interruptions (~ 10.000/s)... With IRQ balance activated, each NIC got used with a specific processor core, and the use of CPU on these cores was always 100%.

I've solved this problem in my setup (2 Quad-Cores), deactivating the IRQ balance, that caused the interruptions to be processed by all the 8 cores.

Are you doing p2p control on this server? This is usually what takes more CPU usage..

Quoted 13 lines · Pablo Fernandes Yahoo

As i told before, i tryed to shape my traffic CBQ and i did use u32 filters. the results were exactly the same as HTB using u32 filters (i tryed before too) or not.

Do you have in your HTB setup a class per customer? I’ve seen different setups, but all of them shaping the traffic either based in protocols or/and IP ranges (it isn’t our reality whilst we have at least a class per each single IP within the network).

Quoted 6 lines

After some readings, i’m starting to suspect about my NIC driver (e1000). Do you have in your Dell PE 1850 interfaces using the e1000 driver? Is the entire traffic passing by this server? I suppose that problem is something about the hardware or software interruptions. Are you using the default parameters for the e1000 kernel module?

--
Acácio Alves dos Santos
Administração de redes
Diginet Brasil
[email protected]
(+55) 84 4008-9000

Esta mensagem, incluindo seus anexos, pode conter informação confidencial e/ou
privilegiada. Se você não for o destinatário ou a pessoa autorizada a receber
esta mensagem, não pode usar, copiar ou divulgar as informações nela contidas ou tomar qualquer ação baseada nessas informações. Se você recebeu esta mensagem por engano, por favor avise imediatamente o remetente, respondendo o e-mail e em seguida apague-o. Agradecemos sua cooperação.

This message, including its attatchments, may contain confidential and/or
privileged information. If you are not the recipient or authorized person to
receive this message, you must not use, copy, disclose or take any action based
on this message or any information herein. If you received this message by mistake, please advise the sender immediately by replying the e- mail and
deleting this message. Thank you for your cooperation.

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#8 lore raw
In reply to Pablo Fernandes Yahoo

u32 hash filters is the key, as somebody pointed. You can also tune your iptables setup, like this

#192.168.1.0/24
iptables -t mangle -N 192-168-1-0-24
iptables -t mangle -A FORWARD -s 192.168.1.0/24 -j 192-168-1-0-24 iptables -t mangle -N 192-168-1-0-25
iptables -t mangle -N 192-168-1-128-25
iptables -t mangle -A 192-168-1-0-24 -s 192.168.1.0/25 -j 192-168-1-0-25 iptables -t mangle -A 192-168-1-0-24 -s 192.168.128.0/25 -j 192-168-1-128-25 .
.
and so on, until (ip 192.168.1.11, which is called in chain created for 192.168.1.10/31)

iptables -t mangle -A 192-168-1-10-31 -s 192.168.1.10 -j CLASSIFY --set-class 1:10
iptables -t mangle -A 192-168-1-10-31 -s 192.168.1.11 -j CLASSIFY --set-class 1:11

.. I guess you got the ideea, it requires some RAM, which i belive is not such a big problem. Similar rules should be made for download.

Quoted 249 lines · Pablo Fernandes Yahoo

Hello,

have HTB „rules“ in 4 different ISPs and i control for each customer this way:

Flush and 1:0 class

tc qdisc del dev eth0 root

tc qdisc add dev eth0 root handle 1:0 htb

tc class add dev eth0 parent 1:0 classid 1:1 htb rate 100mbit

tc qdisc del dev eth1 root

tc qdisc add dev eth1 root handle 1:0 htb

tc class add dev eth1 parent 1:0 classid 1:1 htb rate 100mbit

Upload and Download: user1

tc class add dev eth0 parent 1:1 classid 1:5 htb rate 150kbit ceil 150kbit

tc qdisc add dev eth0 parent 1:5 handle 5: sfq perturb 10

tc class add dev eth1 parent 1:1 classid 1:5 htb rate 50kbit ceil 50kbit

tc qdisc add dev eth1 parent 1:5 handle 5: sfq perturb 10

iptables -t mangle -A POSTROUTING --dest x.x.x.x -o eth0 -j CLASSIFY --set-class 1:5

iptables -t mangle -A FORWARD --src x.x.x.x -o eth1 -j CLASSIFY --set-class 1:5

Upload and Download: user2

tc class add dev eth0 parent 1:1 classid 1:8 htb rate 150kbit ceil 150kbit

tc qdisc add dev eth0 parent 1:8 handle 8: sfq perturb 10

tc class add dev eth1 parent 1:1 classid 1:8 htb rate 50kbit ceil 50kbit

tc qdisc add dev eth1 parent 1:8 handle 8: sfq perturb 10

iptables -t mangle -A POSTROUTING --dest y.y.y.y -o eth0 -j CLASSIFY --set-class 1:8

iptables -t mangle -A FORWARD --src y.y.y.y -o eth1 -j CLASSIFY --set-class 1:8

(…)

This rules works fine, but just for less than 1.700 customers. More than 1.700 customers, i have my load avarage in the sky and Ksoftirqd process (top information) in 100% fulltime. I don’t know why. I used to use CBQ instead HTB because i had the same problem and Ron (a guy in this list) gave this rules and told me that he uses this for more than 3.000 customers. I tested it in more than 7 different computers (but the same hadware specifications) and i had the same problem with either CBQ or HTB rules. The computers that i have are all of them DELL PowerEdge 1850. I will put some hardware iformations here:

top

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

3 root 39 19 0 0 0 R 100 0.0 5316:20 ksoftirqd/0

[root@fw ~]# uptime

10:38:11 up 161 days, 17:21, 3 users, load average: 1.58, 1.65, 1.51 (unfortunately when i took this, the load average was „pretty good“, but minutes ago, it was more than 11.0

[root@fw ~]# lspci

00:00.0 Host bridge: Intel Corporation E7520 Memory Controller Hub (rev 09)

00:02.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express Port A (rev 09)

00:04.0 PCI bridge: Intel Corporation E7525/E7520 PCI Express Port B (rev 09)

00:05.0 PCI bridge: Intel Corporation E7520 PCI Express Port B1 (rev 09)

00:06.0 PCI bridge: Intel Corporation E7520 PCI Express Port C (rev 09)

00:1d.0 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #1 (rev 02)

00:1d.1 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #2 (rev 02)

00:1d.2 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #3 (rev 02)

00:1d.7 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB2 EHCI Controller (rev 02)

00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c2)

00:1f.0 ISA bridge: Intel Corporation 82801EB/ER (ICH5/ICH5R) LPC Interface Bridge (rev 02)

00:1f.1 IDE interface: Intel Corporation 82801EB/ER (ICH5/ICH5R) IDE Controller (rev 02)

01:00.0 PCI bridge: Intel Corporation 80332 [Dobson] I/O processor (A-Segment Bridge) (rev 06)

01:00.2 PCI bridge: Intel Corporation 80332 [Dobson] I/O processor (B-Segment Bridge) (rev 06)

02:0c.0 Ethernet controller: Intel Corporation 82545GM Gigabit Ethernet Controller (rev 04)

02:0e.0 RAID bus controller: Dell PowerEdge Expandable RAID controller 4 (rev 06)

03:0b.0 Ethernet controller: Intel Corporation 82545GM Gigabit Ethernet Controller (rev 04)

05:00.0 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge A (rev 09)

05:00.2 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge B (rev 09)

06:07.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit Ethernet Controller (rev 05)

07:08.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit Ethernet Controller (rev 05)

09:0d.0 VGA compatible controller: ATI Technologies Inc Radeon RV100 QY [Radeon 7000/VE]

[root@fw ~]# free -m

total used free shared buffers cached

Mem: 2021 1479 542 0 400 654

-/+ buffers/cache: 424 1597

Swap: 1027 0 1027

[root@fw ~]# cat /proc/cpuinfo

processor : 0

vendor_id : GenuineIntel

cpu family : 15

model : 4

model name : Intel(R) Xeon(TM) CPU 3.00GHz

stepping : 3

cpu MHz : 2992.674

cache size : 2048 KB

physical id : 0

siblings : 2

core id : 0

cpu cores : 1

fdiv_bug : no

hlt_bug : no

f00f_bug : no

coma_bug : no

fpu : yes

fpu_exception : yes

cpuid level : 5

wp : yes

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pni monitor ds_cpl cid cx16 xtpr

bogomips : 5990.78

processor : 1

vendor_id : GenuineIntel

cpu family : 15

model : 4

model name : Intel(R) Xeon(TM) CPU 3.00GHz

stepping : 3

cpu MHz : 2992.674

cache size : 2048 KB

physical id : 0

siblings : 2

core id : 0

cpu cores : 1

fdiv_bug : no

hlt_bug : no

f00f_bug : no

coma_bug : no

fpu : yes

fpu_exception : yes

cpuid level : 5

wp : yes

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pni monitor ds_cpl cid cx16 xtpr

bogomips : 5985.13

Any help/Tipp/hint will be very welcome.

Thanks in Advance!

Quoted 9 lines

Pablo Fernandes

------------------------------------------------------------------------

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#9 lore raw
In reply to Pablo Fernandes Yahoo

Alexandru Dragoi написа:

Quoted 15 lines

u32 hash filters is the key, as somebody pointed. You can also tune your iptables setup, like this

#192.168.1.0/24
iptables -t mangle -N 192-168-1-0-24
iptables -t mangle -A FORWARD -s 192.168.1.0/24 -j 192-168-1-0-24 iptables -t mangle -N 192-168-1-0-25
iptables -t mangle -N 192-168-1-128-25
iptables -t mangle -A 192-168-1-0-24 -s 192.168.1.0/25 -j 192-168-1-0-25 iptables -t mangle -A 192-168-1-0-24 -s 192.168.128.0/25 -j 192-168-1-128-25 .
.
and so on, until (ip 192.168.1.11, which is called in chain created for 192.168.1.10/31)

Quoted 9 lines

iptables -t mangle -A 192-168-1-10-31 -s 192.168.1.10 -j CLASSIFY --set-class 1:10
iptables -t mangle -A 192-168-1-10-31 -s 192.168.1.11 -j CLASSIFY --set-class 1:11

.. I guess you got the ideea, it requires some RAM, which i belive is not such a big problem. Similar rules should be made for download.

Or you can use my patch - IPCLASSIFY. Then the rules above would be substituted by a signle rule per direction:

iptables -t mangle -A FORWARD -s 192.168.1.0/24 -j IPCLASSIFY --addr=src --and-mask=0xff --or-mask=0x11000
iptables -t mangle -A FORWARD -d 192.168.1.0/24 -j IPCLASSIFY --addr=dst --and-mask=0xff --or-mask=0x12000

This is equal to applying CLASSIFY target to each packet with --set-class (srcIP & 0xFF | 0x1100 ) and --set-class (dstIP & 0xFF | 0x1200 ).
It is very similar to IPMARK, but it uses skb->priority field instead mark. So no tc filters are needed.

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#10 lore raw
In reply to Pablo Fernandes Yahoo
Quoted 35 lines · VladSun

Alexandru Dragoi написа:

Quoted 16 lines

u32 hash filters is the key, as somebody pointed. You can also tune your iptables setup, like this

#192.168.1.0/24
iptables -t mangle -N 192-168-1-0-24
iptables -t mangle -A FORWARD -s 192.168.1.0/24 -j 192-168-1-0-24 iptables -t mangle -N 192-168-1-0-25
iptables -t mangle -N 192-168-1-128-25
iptables -t mangle -A 192-168-1-0-24 -s 192.168.1.0/25 -j 192-168-1-0-25 iptables -t mangle -A 192-168-1-0-24 -s 192.168.128.0/25 -j 192-168-1-128-25
.
.
and so on, until (ip 192.168.1.11, which is called in chain created for 192.168.1.10/31)

Quoted 9 lines

iptables -t mangle -A 192-168-1-10-31 -s 192.168.1.10 -j CLASSIFY --set-class 1:10
iptables -t mangle -A 192-168-1-10-31 -s 192.168.1.11 -j CLASSIFY --set-class 1:11

.. I guess you got the ideea, it requires some RAM, which i belive is not such a big problem. Similar rules should be made for download.

Or you can use my patch - IPCLASSIFY. Then the rules above would be substituted by a signle rule per direction:

iptables -t mangle -A FORWARD -s 192.168.1.0/24 -j IPCLASSIFY --addr=src --and-mask=0xff --or-mask=0x11000 iptables -t mangle -A FORWARD -d 192.168.1.0/24 -j IPCLASSIFY --addr=dst --and-mask=0xff --or-mask=0x12000

Quoted 6 lines

This is equal to applying CLASSIFY target to each packet with --set-class (srcIP & 0xFF | 0x1100 ) and --set-class (dstIP & 0xFF | 0x1200 ).
It is very similar to IPMARK, but it uses skb->priority field instead mark. So no tc filters are needed.

Cool, I remember I red about this a little while ago. Now, another thing to tune would be some htb paches for massive hashing on classid lookup. I must say I haven't use it so far, I hope I will do it soon.

http://www.mail-archive.com/[email protected]/msg16279.html
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#11 lore raw
In reply to Pablo Fernandes Yahoo
Quoted 24 lines · On Monday 28 May 2007 10:39:11 VladSun wrote:

Alexandru Dragoi написа:

Quoted 15 lines

u32 hash filters is the key, as somebody pointed. You can also tune your iptables setup, like this

#192.168.1.0/24
iptables -t mangle -N 192-168-1-0-24
iptables -t mangle -A FORWARD -s 192.168.1.0/24 -j 192-168-1-0-24 iptables -t mangle -N 192-168-1-0-25
iptables -t mangle -N 192-168-1-128-25
iptables -t mangle -A 192-168-1-0-24 -s 192.168.1.0/25 -j 192-168-1-0-25 iptables -t mangle -A 192-168-1-0-24 -s 192.168.128.0/25 -j 192-168-1-128-25 .
.
and so on, until (ip 192.168.1.11, which is called in chain created for 192.168.1.10/31)

Quoted 7 lines

iptables -t mangle -A 192-168-1-10-31 -s 192.168.1.10 -j CLASSIFY --set-class 1:10
iptables -t mangle -A 192-168-1-10-31 -s 192.168.1.11 -j CLASSIFY --set-class 1:11

.. I guess you got the ideea, it requires some RAM, which i belive is not such a big problem. Similar rules should be made for download.

Quoted 8 lines

Or you can use my patch - IPCLASSIFY. Then the rules above would be substituted by a signle rule per direction:

iptables -t mangle -A FORWARD -s 192.168.1.0/24 -j IPCLASSIFY --addr=src --and-mask=0xff --or-mask=0x11000
iptables -t mangle -A FORWARD -d 192.168.1.0/24 -j IPCLASSIFY --addr=dst --and-mask=0xff --or-mask=0x12000

Wow! now i get it, this patch is amazing, now i have a pendient hack that is to merge this with htb-gen. Any chances that this get into mainline, have you mailed netfilter-dev list?

Signature · 5 lines
Luciano
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
#12 lore raw
In reply to Pablo Fernandes Yahoo

Luciano Ruete написа:

Quoted 10 lines
Quoted 9 lines

Or you can use my patch - IPCLASSIFY. Then the rules above would be substituted by a signle rule per direction:

iptables -t mangle -A FORWARD -s 192.168.1.0/24 -j IPCLASSIFY --addr=src --and-mask=0xff --or-mask=0x11000
iptables -t mangle -A FORWARD -d 192.168.1.0/24 -j IPCLASSIFY --addr=dst --and-mask=0xff --or-mask=0x12000

Quoted 5 lines

Wow! now i get it, this patch is amazing, now i have a pendient hack that is to merge this with htb-gen. Any chances that this get into mainline, have you mailed netfilter-dev list?

:)

Thank you! You should thank Grzegorz Janoszka also - he wrote the original IPMARK patch. My patch is just a slight modification of it.

As far as I know netfilter team refused to include the IPMARK in the official P-o-M. So I don't think IPCLASSIFY would be accepted either.

Regards, Vladimir Mirchev.
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc