Bad network performance over 2Gbps
Subsystems (3)
I use Linux for serving a huge amount of static web on few servers. When network traffic goes above 2Gbit/sec ksoftirqd/5 (not every time 5, but every time just one) starts using exactly 100% CPU time and packet packet loss starts preventing traffic from going up. When the network traffic is lower than 1.9Gbit ksoftirqds use 0% CPU according to top.
Uplink is 6 gigabit Intel cards bonded together using 802.3ad algorithm with xmit_hash_policy set to layer3+4. On the other side is Cisco 2960 switch. Machine is with two quad core Intel Xeons @2.33GHz.
Here goes a screen snapshot of "top" command. The described behavior have nothing to do with 13% io-wait. It happens even if it is 0% io-wait.
http://www.titov.net/misc/top-snap.png
kernel configuration:
http://www.titov.net/misc/config.gz
/proc/interrupts, lspci, dmesg (nothing intresting there), ifconfig, uname -a:
http://www.titov.net/misc/misc.txt.gz
Is it a Linux bug or some hardware limitation?
Regards,
Anton Titov
[Changed mail list]
Quoted 33 lines · Anton Titov
I use Linux for serving a huge amount of static web on few servers. When
network traffic goes above 2Gbit/sec ksoftirqd/5 (not every time 5, but every time just one) starts using exactly 100% CPU time and packet packet loss starts preventing traffic from going up. When the network traffic is lower than 1.9Gbit ksoftirqds use 0% CPU according to top.Uplink is 6 gigabit Intel cards bonded together using 802.3ad algorithm with
xmit_hash_policyset to layer3+4. On the other side is Cisco 2960 switch. Machine is with two quad core Intel Xeons @2.33GHz.Here goes a screen snapshot of "top" command. The described behavior have nothing to do with 13% io-wait. It happens even if it is 0% io-wait.
http://www.titov.net/misc/top-snap.pngkernel configuration:
http://www.titov.net/misc/config.gz
/proc/interrupts, lspci, dmesg (nothing intresting there), ifconfig, uname -a:
http://www.titov.net/misc/misc.txt.gzIs it a Linux bug or some hardware limitation?
Regards,
Anton Titov--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 26 lines · H. Willstrand
[Changed mail list]
Quoted 23 lines · Anton Titov
I use Linux for serving a huge amount of static web on few servers. When
network traffic goes above 2Gbit/sec ksoftirqd/5 (not every time 5, but every time just one) starts using exactly 100% CPU time and packet packet loss starts preventing traffic from going up. When the network traffic is lower than 1.9Gbit ksoftirqds use 0% CPU according to top.Uplink is 6 gigabit Intel cards bonded together using 802.3ad algorithm with
xmit_hash_policyset to layer3+4. On the other side is Cisco 2960 switch. Machine is with two quad core Intel Xeons @2.33GHz.Here goes a screen snapshot of "top" command. The described behavior have nothing to do with 13% io-wait. It happens even if it is 0% io-wait.
http://www.titov.net/misc/top-snap.pngkernel configuration:
http://www.titov.net/misc/config.gz
/proc/interrupts, lspci, dmesg (nothing intresting there), ifconfig, uname -a:
http://www.titov.net/misc/misc.txt.gzIs it a Linux bug or some hardware limitation?
I'm wondering if this is not a classical demonstration of the NAPI-irq trap where after migration all the interrupts from the various cards are migrated to a single CPU, and because of NAPI once they're busy polling won't ever migrate away from that CPU again.
Have you looked at cat /proc/interrupts before and after this happens?
My guess is that your specific situation can benefit from setting smp_affinity and forcing the NIC irq's so that you're at least occupying the load over multiple CPU's (but preferably ones that use the same cache!) will help relieve the situation.
alternatively you might even see an improvement by disabling NAPI. depending on the driver that you're using this might be possible.
I actually don't know much about bonding and how this affects everything, but my guess is that that's a less important factor in this issue.
Cheers,
Auke
Quoted 42 lines · Auke
Quoted 26 lines · H. Willstrand
[Changed mail list]
Quoted 23 lines · Anton Titov
I use Linux for serving a huge amount of static web on few servers. When
network traffic goes above 2Gbit/sec ksoftirqd/5 (not every time 5, but every time just one) starts using exactly 100% CPU time and packet packet loss starts preventing traffic from going up. When the network traffic is lower than 1.9Gbit ksoftirqds use 0% CPU according to top.Uplink is 6 gigabit Intel cards bonded together using 802.3ad algorithm with
xmit_hash_policyset to layer3+4. On the other side is Cisco 2960 switch. Machine is with two quad core Intel Xeons @2.33GHz.Here goes a screen snapshot of "top" command. The described behavior have nothing to do with 13% io-wait. It happens even if it is 0% io-wait.
http://www.titov.net/misc/top-snap.pngkernel configuration:
http://www.titov.net/misc/config.gz
/proc/interrupts, lspci, dmesg (nothing intresting there), ifconfig, uname -a:
http://www.titov.net/misc/misc.txt.gzIs it a Linux bug or some hardware limitation?
I'm wondering if this is not a classical demonstration of the NAPI-irq trap where after migration all the interrupts from the various cards are migrated to a single CPU, and because of NAPI once they're busy polling won't ever migrate away from that CPU again.
Have you looked at
cat /proc/interruptsbefore and after this happens?My guess is that your specific situation can benefit from setting
smp_affinityand forcing the NIC irq's so that you're at least occupying the load over multiple CPU's (but preferably ones that use the same cache!) will help relieve the situation.alternatively you might even see an improvement by disabling NAPI. depending on the driver that you're using this might be possible.
Quoted 6 lines
I actually don't know much about bonding and how this affects everything, but my guess is that that's a less important factor in this issue.
Cheers,
Auke
I'm not sure that spreading IRQs out completely is necessarily a good idea, due to cache line ping-pong. I suspect you'll get optimal performance by assigning the six IRQs to two cores that share an L2 cache.
Still, I think you're on to something here. Disabling NAPI and instead tuning the cards' interrupt coalescing settings might allow irqbalance to do a better job than it is currently.
-- Chris
Quoted 61 lines · Chris Snook
Quoted 52 lines · Auke
Quoted 29 lines · H. Willstrand
[Changed mail list]
Quoted 26 lines · Anton Titov
I use Linux for serving a huge amount of static web on few servers. When
network traffic goes above 2Gbit/sec ksoftirqd/5 (not every time 5,
but
every time just one) starts using exactly 100% CPU time and packet packet loss starts preventing traffic from going up. When the network traffic is lower than 1.9Gbit ksoftirqds use 0% CPU according to top.Uplink is 6 gigabit Intel cards bonded together using 802.3ad
algorithm
withxmit_hash_policyset to layer3+4. On the other side is Cisco 2960 switch. Machine is with two quad core Intel Xeons @2.33GHz.Here goes a screen snapshot of "top" command. The described behavior have nothing to do with 13% io-wait. It happens even if it is 0% io-wait.
http://www.titov.net/misc/top-snap.pngkernel configuration:
http://www.titov.net/misc/config.gz
/proc/interrupts, lspci, dmesg (nothing intresting there), ifconfig, uname -a:
http://www.titov.net/misc/misc.txt.gzIs it a Linux bug or some hardware limitation?
I'm wondering if this is not a classical demonstration of the NAPI-irq trap where
after migration all the interrupts from the various cards are migrated to a single
CPU, and because of NAPI once they're busy polling won't ever migrate away from
that CPU again.Have you looked at
cat /proc/interruptsbefore and after this happens?My guess is that your specific situation can benefit from setting
smp_affinityand
forcing the NIC irq's so that you're at least occupying the load over multiple
CPU's (but preferably ones that use the same cache!) will help relieve the situation.alternatively you might even see an improvement by disabling NAPI. depending on
the driver that you're using this might be possible.Quoted 7 lines
I actually don't know much about bonding and how this affects everything, but my
guess is that that's a less important factor in this issue.Cheers,
Auke
Quoted 7 lines
I'm not sure that spreading IRQs out completely is necessarily a good idea, due to cache line ping-pong. I suspect you'll get optimal performance by assigning the six IRQs to two cores that share an L2 cache.
Still, I think you're on to something here. Disabling NAPI and instead tuning the cards' interrupt coalescing settings might allow irqbalance to do a better job than it is currently.
well I posted another reply to him after I looked at the debug output he posted and it appeats that the in-kernel irqbalance is the culprit and the 100% softirqd is because his interrupts are being balanced across all cores, therefore pretty much guaranteeing him full cache misses on every single receive, not to mention unneeded migration of tasks.
I definately think we should disable the in-kernel irqbalance option by default :)
as to what the best solution is with 6 adapters is not clear as they're all on the same bridge, but hard setting the affinity to two cores that share an L2 seems to give the best results quickly.
Cheers,
Auke
Quoted 3 lines · Chris Snook
Still, I think you're on to something here. Disabling NAPI and instead tuning the cards' interrupt coalescing settings might allow irqbalance to do a better job than it is currently.
Disabling NAPI allowed me to push as much as 3.5Gbit out of the same server with ~ 20% of time CPUs doing software interrupts.
Regards,
Anton Titov
Quoted 7 lines · Anton Titov
Quoted 3 lines · Chris Snook
Still, I think you're on to something here. Disabling NAPI and instead tuning the cards' interrupt coalescing settings might allow irqbalance to do a better job than it is currently.
Disabling NAPI allowed me to push as much as 3.5Gbit out of the same server with ~ 20% of time CPUs doing software interrupts.
yes, I really don't see this is such an amazing discovery - the in-kernel irqbalance code is totally wrong for network interrupts (and probably for most interrupts).
on your system with 6 network interrupts it blows chunks and it's not NAPI that is the issue - NAPI will work just fine on it's own. By disabling NAPI and reverting to the in-driver irq moderation code you've effectively put the in-kernel irqbalance code to the sideline and this is what makes it work again.
It's not the right solution.
We keep seing this exact issue pop up everywhere - especially with e1000(e) datacenter users - this code _has_ to go or be fixed. Since there is a perfectly viable solution, I strongly suggest disabling it.
This is not the first time I've sent this patch out in some form...
Auke
---
[X86] IRQBALANCE: Mark as BROKEN and disable by default
The IRQBALANCE option causes interrupts to bounce all around on SMP systems quickly burying the CPU in migration cost and cache misses. Mainly affected are network interrupts and this results in one CPU pegged in softirqd completely.
Disable this option and provide documentation to a better solution (userspace irqbalance daemon does overall the best job to begin with and only manual setting of smp_affinity will beat it).
---
arch/x86/Kconfig +6 -2
@@ -1026,13 +1026,17 @@ config EFI platforms. config IRQBALANCE - def_bool y + def_bool n prompt "Enable kernel irq balancing" - depends on X86_32 && SMP && X86_IO_APIC + depends on X86_32 && SMP && X86_IO_APIC && BROKEN help The default yes will allow the kernel to do irq load balancing. Saying no will keep the kernel from doing irq load balancing. + This option is known to cause performance issues on SMP + systems. The preferred method is to use the userspace + 'irqbalance' daemon instead. See http://irqbalance.org/. + config SECCOMP def_bool y prompt "Enable seccomp to safely compute untrusted bytecode"
By default also without IRQBALANCE enabled in kernel, APIC or someone else distributing interrupts over processors too. There is no irqbalance daemon or whatever.
For example:
Router-KARAM ~ # cat /proc/interrupts
CPU0 CPU1
0: 87956938 1403052485 IO-APIC-edge timer 1: 0 2 IO-APIC-edge i8042 9: 0 0 IO-APIC-fasteoi acpi 19: 140 5714 IO-APIC-fasteoi ohci_hcd:usb1, ohci_hcd:usb2 24: 675673280 1186506694 IO-APIC-fasteoi eth2 26: 717865662 2201633562 IO-APIC-fasteoi eth0 27: 1869190 23075556 IO-APIC-fasteoi eth1 NMI: 0 0 Non-maskable interrupts LOC: 1403052485 87956683 Local timer interrupts RES: 75059 25408 Rescheduling interrupts CAL: 99542 83 function call interrupts TLB: 616 200 TLB shootdowns TRM: 0 0 Thermal event interrupts SPU: 0 0 Spurious interrupts ERR: 0 MIS: 0
sunfire-1 ~ # cat config|grep -i irqCONFIG_GENERIC_HARDIRQS=yCONFIG_GENERIC_IRQ_PROBE=yCONFIG_GENERIC_PENDING_IRQ=y
# CONFIG_IRQBALANCE is not setCONFIG_HT_IRQ=y
# CONFIG_HPET_RTC_IRQ is not setCONFIG_TRACE_IRQFLAGS_SUPPORT=y
# CONFIG_DEBUG_SHIRQ is not set
Is it harmful too?
Quoted 73 lines · Auke
Quoted 7 lines · Anton Titov
Quoted 3 lines · Chris Snook
Still, I think you're on to something here. Disabling NAPI and instead tuning the cards' interrupt coalescing settings might allow irqbalance to do a better job than it is currently.
Disabling NAPI allowed me to push as much as 3.5Gbit out of the same server with ~ 20% of time CPUs doing software interrupts.
yes, I really don't see this is such an amazing discovery - the in-kernel irqbalance code is totally wrong for network interrupts (and probably for most interrupts).
on your system with 6 network interrupts it blows chunks and it's not NAPI that is the issue - NAPI will work just fine on it's own. By disabling NAPI and reverting to the in-driver irq moderation code you've effectively put the in-kernel irqbalance code to the sideline and this is what makes it work again.
It's not the right solution.
We keep seing this exact issue pop up everywhere - especially with e1000(e) datacenter users - this code _has_ to go or be fixed. Since there is a perfectly viable solution, I strongly suggest disabling it.
This is not the first time I've sent this patch out in some form...
Auke
---
[X86] IRQBALANCE: Mark as BROKEN and disable by defaultThe IRQBALANCE option causes interrupts to bounce all around on SMP systems quickly burying the CPU in migration cost and cache misses. Mainly affected are network interrupts and this results in one CPU pegged in softirqd completely.
Disable this option and provide documentation to a better solution (userspace irqbalance daemon does overall the best job to begin with and only manual setting of
smp_affinitywill beat it).Signed-off-by Auke Kok <[email protected]>---
arch/x86/Kconfig +6
-3@@ -1026,13 +1026,17 @@ config EFI platforms. config IRQBALANCE - def_bool y + def_bool n prompt "Enable kernel irq balancing" - depends on X86_32 && SMP && X86_IO_APIC + depends on X86_32 && SMP && X86_IO_APIC && BROKEN help The default yes will allow the kernel to do irq load balancing. Saying no will keep the kernel from doing irq load balancing. + This option is known to cause performance issues on SMP + systems. The preferred method is to use the userspace + 'irqbalance' daemon instead. See http://irqbalance.org/. + config SECCOMP def_bool y prompt "Enable seccomp to safely compute untrusted bytecode" --To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
------ Technical Manager Virtual ISP S.A.L. Lebanon
Hi!
Quoted 19 lines
[X86] IRQBALANCE: Mark as BROKEN and disable by default
The IRQBALANCE option causes interrupts to bounce all around on SMP systems quickly burying the CPU in migration cost and cache misses. Mainly affected are network interrupts and this results in one CPU pegged in softirqd completely.
Disable this option and provide documentation to a better solution (userspace irqbalance daemon does overall the best job to begin with and only manual setting of
smp_affinitywill beat it).Signed-off-by Auke Kok <[email protected]>---
arch/x86/Kconfig +0
-0
arch/x86/Kconfig config EFI · line 1029
Quoted 5 lines
arch/x86/Kconfig +1
-1@@ -1026,13 +1026,17 @@ config EFI platforms. config IRQBALANCE - def_bool y + def_bool n
ACK.
arch/x86/Kconfig config EFI · line 1031
Quoted 3 lines
arch/x86/Kconfig +1
-1prompt "Enable kernel irq balancing" - depends on X86_32 && SMP && X86_IO_APIC + depends on X86_32 && SMP && X86_IO_APIC && BROKEN
This is wrong. irqbalance works, there's nothing wrong with it; but it has nasty sideffects.
arch/x86/Kconfig config EFI · line 1039
Quoted 8 lines
arch/x86/Kconfig +4
-0help The default yes will allow the kernel to do irq load balancing. Saying no will keep the kernel from doing irq load balancing. + This option is known to cause performance issues on SMP + systems. The preferred method is to use the userspace + 'irqbalance' daemon instead. See http://irqbalance.org/. +
ACK.
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Quoted 28 lines · Pavel Machek
Hi!
Quoted 19 lines
[X86] IRQBALANCE: Mark as BROKEN and disable by default
The IRQBALANCE option causes interrupts to bounce all around on SMP systems quickly burying the CPU in migration cost and cache misses. Mainly affected are network interrupts and this results in one CPU pegged in softirqd completely.
Disable this option and provide documentation to a better solution (userspace irqbalance daemon does overall the best job to begin with and only manual setting of
smp_affinitywill beat it).Signed-off-by Auke Kok <[email protected]>---
arch/x86/Kconfig +0
-0arch/x86/Kconfig config EFI · line 1029
Quoted 5 lines
arch/x86/Kconfig +1
-1@@ -1026,13 +1026,17 @@ config EFI platforms. config IRQBALANCE - def_bool y + def_bool nACK.
Quoted 6 lines
arch/x86/Kconfig config EFI · line 1031
Quoted 3 lines
arch/x86/Kconfig +1
-1prompt "Enable kernel irq balancing" - depends on X86_32 && SMP && X86_IO_APIC + depends on X86_32 && SMP && X86_IO_APIC && BROKENThis is wrong. irqbalance works, there's nothing wrong with it; but it has nasty sideffects.
ok, I'm fine with taking that part out of the patch.
Ingo, want me to send an updated patch?
Quoted 12 lines
arch/x86/Kconfig config EFI · line 1039
Quoted 9 lines
arch/x86/Kconfig +4
-0help The default yes will allow the kernel to do irq load balancing. Saying no will keep the kernel from doing irq load balancing. + This option is known to cause performance issues on SMP + systems. The preferred method is to use the userspace + 'irqbalance' daemon instead. See http://irqbalance.org/. +ACK.
Quoted 3 lines · Kok, Auke
We keep seing this exact issue pop up everywhere - especially with e1000(e) datacenter users - this code _has_ to go or be fixed. Since there is a perfectly viable solution, I strongly suggest disabling it.
strongly agreed. Thanks Auke, applied.
Ingo
Quoted 9 lines · Ingo Molnar
Quoted 3 lines · Kok, Auke
We keep seing this exact issue pop up everywhere - especially with e1000(e) datacenter users - this code _has_ to go or be fixed. Since there is a perfectly viable solution, I strongly suggest disabling it.
strongly agreed. Thanks Auke, applied.
Ingo
excellent, ignore my other reply to Pavel - I didn't see this reply yet :)
Thanks Ingo
Auke
Quoted 7 lines · Auke
Quoted 6 lines · Ingo Molnar
Quoted 3 lines · Kok, Auke
We keep seing this exact issue pop up everywhere - especially with e1000(e) datacenter users - this code _has_ to go or be fixed. Since there is a perfectly viable solution, I strongly suggest disabling it.
Quoted 6 lines
Quoted 3 lines
strongly agreed. Thanks Auke, applied.
Ingo
excellent, ignore my other reply to Pavel - I didn't see this reply yet :)
Shouldn't you just add it to the FeatureRemoval list too and remove it then quickly? No need to keep disabled and known to be wrong code around.
-Andi
Quoted 20 lines · Auke
[X86] IRQBALANCE: Mark as BROKEN and disable by default
The IRQBALANCE option causes interrupts to bounce all around on SMP systems quickly burying the CPU in migration cost and cache misses. Mainly affected are network interrupts and this results in one CPU pegged in softirqd completely.
Disable this option and provide documentation to a better solution (userspace irqbalance daemon does overall the best job to begin with and only manual setting of
smp_affinitywill beat it).Signed-off-by Auke Kok <[email protected]>---
arch/x86/Kconfig +0
-0@@ -1026,13 +1026,17 @@ config EFI platforms.
arch/x86/Kconfig config EFI · line 1034
Quoted 10 lines
arch/x86/Kconfig +2
-2config IRQBALANCE - def_bool y + def_bool n prompt "Enable kernel irq balancing" - depends on X86_32 && SMP && X86_IO_APIC + depends on X86_32 && SMP && X86_IO_APIC && BROKEN help The default yes will allow the kernel to do irq load balancing. Saying no will keep the kernel from doing irq load balancing.
Since you're changing the default setting, shouldn't the above be changed to:
Saying yes will allow the kernel to do irq load balancing. The default no will keep the kernel from doing irq load balancing.
arch/x86/Kconfig config EFI · line 1042
Quoted 7 lines
arch/x86/Kconfig +4
-0+ This option is known to cause performance issues on SMP + systems. The preferred method is to use the userspace + 'irqbalance' daemon instead. See http://irqbalance.org/. + config SECCOMP def_bool y prompt "Enable seccomp to safely compute untrusted bytecode"
-Bill
Quoted 15 lines · Anton Titov
I use Linux for serving a huge amount of static web on few servers. When network traffic goes above 2Gbit/sec ksoftirqd/5 (not every time 5, but every time just one) starts using exactly 100% CPU time and packet packet loss starts preventing traffic from going up. When the network traffic is lower than 1.9Gbit ksoftirqds use 0% CPU according to top.
Uplink is 6 gigabit Intel cards bonded together using 802.3ad algorithm with
xmit_hash_policyset to layer3+4. On the other side is Cisco 2960 switch. Machine is with two quad core Intel Xeons @2.33GHz.Here goes a screen snapshot of "top" command. The described behavior have nothing to do with 13% io-wait. It happens even if it is 0% io-wait.
http://www.titov.net/misc/top-snap.png
Quoted 8 lines
kernel configuration:
http://www.titov.net/misc/config.gz
/proc/interrupts, lspci, dmesg (nothing intresting there), ifconfig, uname -a:
http://www.titov.net/misc/misc.txt.gzIs it a Linux bug or some hardware limitation?
possibly some missing parameters when loading your e1000 drivers. e1000 NICs support interrupt rate limitation, which proves very efficient in cases such as yours. I'm used to limit them to about 5k ints/s. Do a "modinfo e1000" to get the parameter name, I don't have it quite right in mind.
Also, I've CCed linux-net.
Regards,
Willy
Quoted 25 lines · Willy Tarreau
Quoted 15 lines · Anton Titov
I use Linux for serving a huge amount of static web on few servers. When network traffic goes above 2Gbit/sec ksoftirqd/5 (not every time 5, but every time just one) starts using exactly 100% CPU time and packet packet loss starts preventing traffic from going up. When the network traffic is lower than 1.9Gbit ksoftirqds use 0% CPU according to top.
Uplink is 6 gigabit Intel cards bonded together using 802.3ad algorithm with
xmit_hash_policyset to layer3+4. On the other side is Cisco 2960 switch. Machine is with two quad core Intel Xeons @2.33GHz.Here goes a screen snapshot of "top" command. The described behavior have nothing to do with 13% io-wait. It happens even if it is 0% io-wait.
http://www.titov.net/misc/top-snap.pngQuoted 8 lines
kernel configuration:
http://www.titov.net/misc/config.gz
/proc/interrupts, lspci, dmesg (nothing intresting there), ifconfig, uname -a:
http://www.titov.net/misc/misc.txt.gzIs it a Linux bug or some hardware limitation?
Quoted 7 lines
possibly some missing parameters when loading your e1000 drivers. e1000 NICs support interrupt rate limitation, which proves very efficient in cases such as yours. I'm used to limit them to about 5k ints/s. Do a "modinfo e1000" to get the parameter name, I don't have it quite right in mind.
Also, I've CCed linux-net.
# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6
CPU7
0: 342 261 258 278 271 253 264
283 IO-APIC-edge timer
1: 0 0 1 0 1 0 0
0 IO-APIC-edge i8042
6: 0 1 0 1 0 0 1
0 IO-APIC-edge floppy
9: 0 0 0 0 0 0 0
0 IO-APIC-fasteoi acpi
12: 1 1 0 0 0 1 1
0 IO-APIC-edge i8042
17: 180 190 178 183 182 186 186
188 IO-APIC-fasteoi uhci_hcd:usb1, ehci_hcd:usb4
18: 843504 842514 843653 842033 842416 842742 841903
842960 IO-APIC-fasteoi 3w-9xxx, uhci_hcd:usb3
19: 0 0 0 0 0 0 0
0 IO-APIC-fasteoi uhci_hcd:usb2
498: 534642903 534635899 534726883 534732377 534701710 534708588 534730550
534742730 PCI-MSI-edge eth5
499: 531832274 531846609 531917849 531942676 531855140 531850692 531885565
531863468 PCI-MSI-edge eth4
500: 487251627 487279206 487248030 487220044 487239637 487231454 487281672
487227202 PCI-MSI-edge eth3
501: 486083953 486062203 486109925 486075793 486036977 486035152 486097551
486117164 PCI-MSI-edge eth2
502: 528889380 528863624 528760188 528798619 528891886 528890760 528807939
528822746 PCI-MSI-edge eth1
503: 529043135 529056706 528980250 528975209 529018995 529027386 528941583
528970472 PCI-MSI-edge eth0
NMI: 0 0 0 0 0 0 0
0 Non-maskable interrupts
LOC: 62893699 62809502 62744208 62746035 62708815 62709055 62739182
62620363 Local timer interrupts
RES: 15454866 15827970 16235695 15386970 15761053 16097167 16190851
16159843 Rescheduling interrupts
CAL: 85 98 85 84 98 93 94
91 function call interrupts
TLB: 3565361 3561798 3570271 3566272 3556996 3555866 3578257
3564557 TLB shootdowns
TRM: 0 0 0 0 0 0 0
0 Thermal event interrupts
THR: 0 0 0 0 0 0 0
0 Threshold APIC interrupts
SPU: 0 0 0 0 0 0 0
0 Spurious interruptsYikes! all wrong!
the network irq's are being ping-ponged around all the cores! bad!
1) turn the in-kernel IRQBALANCE option off !
2) use either the userspace irqbalance daemon or
3) set smp_affinity manually
Auke
Quoted 9 lines
Regards,
Willy--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 2 lines · Auke
Willy Tarreau wrote:
1) turn the in-kernel IRQBALANCE option off !
Actually it may be already removed. I remember it being under "Processor type and features" and I currently cannot find it there for x86_64
Quoted 2 lines
2) use either the userspace
irqbalancedaemon or
3) setsmp_affinitymanually
I tried echoing 3 (assuming that CPU0 and CPU1 will share their cache, as advised in other mails) into smp_affinity of all ethX interrupts and no positive result was observed.
I will try disabling NAPI and limiting e1000 interrupts tomorrow.
Quoted 13 lines
Auke
Quoted 9 lines
Regards,
Willy--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 6 lines · Anton Titov
Quoted 2 lines · Auke
Willy Tarreau wrote:
1) turn the in-kernel IRQBALANCE option off !Actually it may be already removed. I remember it being under "Processor type and features" and I currently cannot find it there for
x86_64
Quoted 6 lines
Quoted 2 lines
2) use either the userspace
irqbalancedaemon or
3) setsmp_affinitymanuallyI tried echoing 3 (assuming that CPU0 and CPU1 will share their cache, as advised in other mails) into
smp_affinityof all ethX interrupts and no positive result was observed.
But have you disabled irqbalance before doing this ? (you must reboot and pass "noirqbalance" on the command line for this).
Also, if you are running on quad-core intel CPUs, I'm told that they're simply two standard dual-core CPUs in the same case, so there is no shared cache between any core. You should try to assign all irqs to CPU0 for a test. It *must* make a difference, in either direction.
Quoted 1 line
I will try disabling NAPI and limiting e1000 interrupts tomorrow.
I found the parameter name I was speaking about : InterruptThrottleRate. Beware it's an array with one entry per NIC, so you have to set as many values as you have NICs. I have always observed huge performance boosts when using the tunables the driver provides.
Willy