Skip to contentSkip to conversation

Simple question about network stack

DISCUSSION Badalian Vyacheslav 2 participants 3 messages Dec 24, 2007 – Dec 25, 2007 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi all.
Sorry for offtopic.
Have problems with balance CPU load in networking.

Have 2 Ethernet adapters e1000. Have 8 CPU (4 real).
Computer work as Shaper. Use only TC rules to shape and IPTABLES to drop.

rx on eth0 go to CPU0. traffic above 400mbs do 90% SI. rx on eth1 go to CPU1. traffic above 400mbs do 90% SI. All other CPUS 100%idle.

question:
1. I may balance load to other cpu? I understand that i can't balance polling place, but find in TC and IPTABLES hash may do different cpu?
2. If SI on 1 cpu more then 100% (600mbs traffic) i see strange. SOFTIRQ process do 100%. Traffic bandwidth go from 400mbs to 100 mbs. pings trough computer go from 0.5ms to 100ms. 1 cpu use 100%. All other cpu 100%idle. If traffic down - after some time cpu load again go to different cpu.

P.S. Very strange that computer with 4(8) CPU and 1 CPU HT do some network performance.
P.P.S Sorry for my English

Thanks for answers.
Slavon

#2 lore raw
In reply to a message not in the archive

Marek Kierdelewicz:

Quoted 11 lines

Hi,

Quoted 7 lines

Have 2 Ethernet adapters e1000. Have 8 CPU (4 real). Computer work as Shaper. Use only TC rules to shape and IPTABLES to drop.
question:
1. I may balance load to other cpu? I understand that i can't balance polling place, but find in TC and IPTABLES hash may do different cpu?

Quoted 4 lines

You need as many nics as cpus to effectivelu use all your processing power. You can pair-up nics and cpus by configuring appropriate irq smp affinity. Read in [1] from line 350 on.

Interesting. Sorry if my questions will be stupid. "smp affinity" its was i need, but it not work for me, and never work if i remember =( Maybe Guru of Network ask me where i have simple stupid mistake?

In theory
#cat ffffffff > /proc/irq/ID/smp_affinity
set mask that all cpus must get interrupts RR

but i see in cat /proc/interrupts that all interrupts get only CPU0 On CPU1 0 interrupts

i do
#cat 2 > /proc/irq/16/smp_affinity
#cat 2 > /proc/irq/17/smp_affinity
#echo /proc/irq/1[67]/smp_affinity
00000002
00000002

Great. Interrupts go to CPU1

#cat 1 > /proc/irq/16/smp_affinity
#cat 1 > /proc/irq/17/smp_affinity
#echo /proc/irq/1[67]/smp_affinity
00000001
00000001

Great. Interrupts go to CPU0

#cat 3 > /proc/irq/16/smp_affinity
#cat 3 > /proc/irq/17/smp_affinity
#echo /proc/irq/1[67]/smp_affinity
00000003
00000003

Strange. Interrupts go to CPU0 only.

Where i have mistake? Why i not have RR? Or i mistake in SMP Affinity idea? Thanks for answers!

Slavon

Quoted 11 lines

Another option is to use recent e1000 nics with multiqueue capability. Read in [2]. Google for more information. I'm not sure but you'll probably need non-kerel recent e1000 drivers from sourceforge.

[1]http://www.mjmwired.net/kernel/Documentation/filesystems/proc.txt [2]http://www.mjmwired.net/kernel/Documentation/networking/multiqueue.txt

cheers,
Marek Kierdelewicz

#3 lore raw

Probably you have enabled in kernel "Enable kernel irq balancing" or CONFIG_IRQBALANCE

It is wrong. It has to be disabled.

On Tue, 25 Dec 2007 11:52:48 +0300, Badalian Vyacheslav wrote

Quoted 69 lines

Marek Kierdelewicz:

Quoted 11 lines

Hi,

Quoted 7 lines

Have 2 Ethernet adapters e1000. Have 8 CPU (4 real). Computer work as Shaper. Use only TC rules to shape and IPTABLES to drop.
question:
1. I may balance load to other cpu? I understand that i can't balance polling place, but find in TC and IPTABLES hash may do different cpu?

Quoted 4 lines

You need as many nics as cpus to effectivelu use all your processing power. You can pair-up nics and cpus by configuring appropriate irq smp affinity. Read in [1] from line 350 on.

Interesting. Sorry if my questions will be stupid. "smp affinity" its was i need, but it not work for me, and never work if i remember =( Maybe Guru of Network ask me where i have simple stupid mistake?

In theory
#cat ffffffff > /proc/irq/ID/smp_affinity
set mask that all cpus must get interrupts RR

but i see in cat /proc/interrupts that all interrupts get only CPU0 On CPU1 0 interrupts

i do
#cat 2 > /proc/irq/16/smp_affinity
#cat 2 > /proc/irq/17/smp_affinity
#echo /proc/irq/1[67]/smp_affinity
00000002
00000002

Great. Interrupts go to CPU1

#cat 1 > /proc/irq/16/smp_affinity
#cat 1 > /proc/irq/17/smp_affinity
#echo /proc/irq/1[67]/smp_affinity
00000001
00000001

Great. Interrupts go to CPU0

#cat 3 > /proc/irq/16/smp_affinity
#cat 3 > /proc/irq/17/smp_affinity
#echo /proc/irq/1[67]/smp_affinity
00000003
00000003

Strange. Interrupts go to CPU0 only.

Where i have mistake? Why i not have RR? Or i mistake in SMP Affinity idea? Thanks for answers!

Slavon

Quoted 11 lines

Another option is to use recent e1000 nics with multiqueue capability. Read in [2]. Google for more information. I'm not sure but you'll probably need non-kerel recent e1000 drivers from sourceforge.

[1]http://www.mjmwired.net/kernel/Documentation/filesystems/proc.txt [2]http://www.mjmwired.net/kernel/Documentation/networking/multiqueue.txt

cheers,
Marek Kierdelewicz

Quoted 4 lines

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
Denys Fedoryshchenko
Technical Manager
Virtual ISP S.A.L.