Ethernet bridge performance
Subsystems (2)
Hi,
I'm evaluating a performance of a dual port ethernet bridge, and the results are a bit disappointing. I would appreciate any hints on improving the results.
I'm using a Dual Xeon 2.66 GHz box based on Intel 5701 chipset with 1G of RAM. NICs are e1000 82546 connected to PCIX bus. Kernel is 2.4.22-pre8, e1000 driver version 5.1.13-k1 with NAPI support. NICs' interrupts are bound to CPU0.
The test consists of sending 200 byte UDP packets from 2 ports of Gigabit IXIA traffic generator to 2 bridge ports. The bridge is capable to sustain the rate of ~170000 pps from each IXIA port without drops. I was expecting it to be able to do at least 250000 pps (our own bridge code based on 2.2.x kernel sustains ~266000 pps on the same hardware). e1000 driver drops 0 packets, all drops occur at higher level.
The output of oprofile attached. I'd be happy to provide any info you may need.
Thanks in advance for your help.
Felix.
CPU: P4 / Xeon, speed 2666.83 MHz (estimated)
Counted GLOBAL_POWER_EVENTS events (time during which processor is not stopped) with a unit mask of 0x01 (count cycles when processor is active) count 1333415
vma samples % symbol name c01a13ac 7983 12.5065 eth_type_trans c01a1590 7629 11.9519 qdisc_restart c0197720 7365 11.5383 skb_release_data c010c170 5962 9.3403 do_gettimeofday c019a9b4 5207 8.1575 dev_queue_xmit c012df98 4243 6.6472 free_block c019ad44 3404 5.3328 netif_rx c01974e0 3266 5.1166 alloc_skb c012e1b4 2600 4.0733 kmalloc c012e3b8 2448 3.8351 kfree c019b284 2401 3.7615 process_backlog c012debc 1941 3.0408 kmem_cache_alloc_batch c0197790 1555 2.4361 kfree_skbmem c01977f8 1296 2.0304 __kfree_skb c012e07c 1203 1.8847 kmem_cache_alloc c019afa4 1079 1.6904 net_tx_action c019b0bc 932 1.4601 netif_receive_skb c01a1a04 750 1.1750 pfifo_fast_dequeue c019accc 445 0.6972 get_sample_stats c012e338 431 0.6752 kmem_cache_free c01a1990 222 0.3478 pfifo_fast_enqueue c01087f0 180 0.2820 do_IRQ c01142a8 152 0.2381 schedule c010c434 131 0.2052 timer_interrupt c010b04c 123 0.1927 IRQ0x31_interrupt c010b040 115 0.1802 IRQ0x30_interrupt c011c400 82 0.1285 cpu_raise_softirq c019b3ac 62 0.0971 net_rx_action c01052a0 57 0.0893 default_idle c011be40 55 0.0862 do_softirq c0112b28 47 0.0736 end_level_ioapic_irq c016b210 39 0.0611 add_timer_randomness c016afb0 38 0.0595 add_entropy_words c016b0a4 37 0.0580 batch_entropy_store c0105944 35 0.0548 __switch_to c0115778 32 0.0501 wake_up_process c0112de0 23 0.0360 do_check_pgt_cache c010b760 22 0.0345 apic_timer_interrupt c010ae10 21 0.0329 common_interrupt c01da960 13 0.0204 __rdtsc_delay c0106f3d 11 0.0172 restore_all c0111d0c 11 0.0172 smp_apic_timer_interrupt c01245a8 11 0.0172 check_pgt_cache c016b158 11 0.0172 batch_entropy_process c0107138 9 0.0141 page_fault c0108654 9 0.0141 handle_IRQ_event c010ae30 9 0.0141 IRQ0x00_interrupt c0125414 9 0.0141 do_wp_page c011f1b0 8 0.0125 update_one_process c011f36c 8 0.0125 timer_bh c011c33c 7 0.0110 ksoftirqd c0113eb8 6 0.0094 reschedule_idle c011c0cc 5 0.0078 tasklet_hi_action c011c204 5 0.0078 bh_action c010ae23 4 0.0063 call_do_IRQ c011eb90 4 0.0063 add_timer c011f284 4 0.0063 update_process_times c0125970 4 0.0063 do_anonymous_page c016b320 4 0.0063 add_interrupt_randomness c0106ef4 3 0.0047 system_call c010c78c 3 0.0047 inc_new_microsec_time c0124850 3 0.0047 zap_page_range c01daa70 3 0.0047 __generic_copy_to_user c010b778 2 0.0031 call_apic_timer_interrupt c0112ae8 2 0.0031 ack_edge_ioapic_irq c011327c 2 0.0031 do_page_fault c01147c8 2 0.0031 __wake_up c011ee88 2 0.0031 tqueue_bh c0124698 2 0.0031 copy_page_range c0125a5c 2 0.0031 do_no_page c01282d8 2 0.0031 __find_get_page c0128e1c 2 0.0031 file_read_actor c01d1880 2 0.0031 fn_hash_lookup c01db1b0 2 0.0031 number c01052f4 1 0.0016 cpu_idle c0106628 1 0.0016 setup_frame c0106fb0 1 0.0016 ret_from_intr c0106fb7 1 0.0016 ret_from_exception c0106fe4 1 0.0016 error_code c01114f8 1 0.0016 flush_tlb_page c01198ec 1 0.0016 get_module_list c0119d30 1 0.0016 s_show c011c2d0 1 0.0016 __run_task_queue c011f73c 1 0.0016 do_timer c012093c 1 0.0016 sys_rt_sigprocmask c0125bf8 1 0.0016 handle_mm_fault c0125cc8 1 0.0016 pte_alloc c0125fd8 1 0.0016 lock_vma_mappings c0126348 1 0.0016 do_mmap_pgoff c012ca90 1 0.0016 vmfree_area_pages c012dcac 1 0.0016 kmem_cache_grow c012f000 1 0.0016 __lru_cache_del c0130288 1 0.0016 rmqueue c01308a8 1 0.0016 __free_pages c0130acf 1 0.0016 .text.lock.page_alloc c0134460 1 0.0016 fd_install c013fb70 1 0.0016 link_path_walk c01557f8 1 0.0016 proc_pid_statm c016b078 1 0.0016 credit_entropy_store c01804dc 1 0.0016 ide_inb c0180524 1 0.0016 ide_outb c0180a60 1 0.0016 drive_is_ready c0198d3c 1 0.0016 skb_headerinit c01a58e4 1 0.0016 ip_route_input_slow c01a8324 1 0.0016 ip_rcv c01d0c68 1 0.0016 fib_semantic_match
What kernel version? 2.6 should be faster.
Quoted 27 lines · On Thu, 07 Aug 2003 12:12:07 +0300 Felix Radensky <[email protected]> wrote:
Hi,
I'm evaluating a performance of a dual port ethernet bridge, and the results are a bit disappointing. I would appreciate any hints on improving the results.
I'm using a Dual Xeon 2.66 GHz box based on Intel 5701 chipset with 1G of RAM. NICs are e1000 82546 connected to PCIX bus. Kernel is 2.4.22-pre8, e1000 driver version 5.1.13-k1 with NAPI support. NICs' interrupts are bound
to CPU0.The test consists of sending 200 byte UDP packets from 2 ports of Gigabit IXIA traffic generator to 2 bridge ports. The bridge is capable to sustain the rate of ~170000 pps from each IXIA port without drops. I was expecting it to be able to do at least 250000 pps (our own bridge code
based
on 2.2.x kernel sustains ~266000 pps on the same hardware). e1000 driver drops 0 packets, all drops occur at higher level.The output of oprofile attached. I'd be happy to provide any info you may need.
Thanks in advance for your help.
Felix.
It's 2.4.22-pre8. I have to work with 2.4, 2.6 is not an option at this point.
Felix.
Quoted 37 lines · Stephen Hemminger
What kernel version? 2.6 should be faster.
Quoted 27 lines · On Thu, 07 Aug 2003 12:12:07 +0300 Felix Radensky <[email protected]> wrote:
Hi,
I'm evaluating a performance of a dual port ethernet bridge, and the results are a bit disappointing. I would appreciate any hints on improving the results.
I'm using a Dual Xeon 2.66 GHz box based on Intel 5701 chipset with 1G of RAM. NICs are e1000 82546 connected to PCIX bus. Kernel is 2.4.22-pre8, e1000 driver version 5.1.13-k1 with NAPI support. NICs' interrupts are bound to CPU0.
The test consists of sending 200 byte UDP packets from 2 ports of Gigabit IXIA traffic generator to 2 bridge ports. The bridge is capable to sustain the rate of ~170000 pps from each IXIA port without drops. I was expecting it to be able to do at least 250000 pps (our own bridge code based on 2.2.x kernel sustains ~266000 pps on the same hardware). e1000 driver drops 0 packets, all drops occur at higher level.
The output of oprofile attached. I'd be happy to provide any info you may need.
Thanks in advance for your help.
Felix.
Quoted 10 lines · Felix Radensky
Hi,
I'm evaluating a performance of a dual port ethernet bridge, and the results are a bit disappointing. I would appreciate any hints on improving the results.
c01a13ac 7983 12.5065 eth_type_trans c01a1590 7629 11.9519 qdisc_restart c0197720 7365 11.5383 skb_release_data c010c170 5962 9.3403 do_gettimeofday
If that do_gettimeofday is happening in the skb rx code, then you could gain ~10% by disabling it somehow..as it should not matter for a bridge. I bet Robert's skb-recycle patch would help here too, especially if you allowed the NICs to save up a large number of skbs so that alloc was less likely to fail.
Btw, I've considered saving, say, 10k skbs on a list in my module, allocated by GFP_KERNEL at module load time, and using them when GFP_ATOMIC skb_alloc fails in the IRQ handling portion of the code....
Anyone think that's a good idea? :)
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Thanks for your help, Ben. What is skb-recycle patch and where can I find it ?
Felix.
Quoted 29 lines · Ben Greear
Quoted 11 lines · Felix Radensky
Hi,
I'm evaluating a performance of a dual port ethernet bridge, and the results are a bit disappointing. I would appreciate any hints on improving the results.
c01a13ac 7983 12.5065 eth_type_trans c01a1590 7629 11.9519 qdisc_restart c0197720 7365 11.5383 skb_release_data c010c170 5962 9.3403 do_gettimeofdayIf that
do_gettimeofdayis happening in the skb rx code, then you could gain ~10% by disabling it somehow..as it should not matter for a bridge. I bet Robert's skb-recycle patch would help here too, especially if you allowed the NICs to save up a large number of skbs so that alloc was less likely to fail.Btw, I've considered saving, say, 10k skbs on a list in my module, allocated by
GFP_KERNELat module load time, and using them whenGFP_ATOMICskb_allocfails in the IRQ handling portion of the code....Anyone think that's a good idea? :)
Ben
Quoted 2 lines · Felix Radensky
Thanks for your help, Ben. What is skb-recycle patch and where can I find it ?
It's experimental and not updated for almost a year and current implementation does not add anything to SMP. Got some idea how to improve this... but try to keep to slab as long as possible it has been improved.
Routing/bridging on SMP has affinty problem. If you are passing skb's say from eth0 to eth1 and they are bound on different CPU's you get cache boucing since the TX-interrupts come on another CPU.
In a recent test with pktgen:
300 kpps with TX interrupts on same CPU as sender. 198 kpps with TX intr on different CPU as sender.
Recycling tries to address this but current implementation fails as said.
But you are probably hit by something else... Check were the drops happens qdisc?. NIC ring RX/TX size, Number of interrupts. ksoftird priority, link HW_FLOW control, checksumming, affinity etc.
Cheers.
--ro
Actually seems his biggest problem is he is not running the NAPI driver
cheers,
jamal
Quoted 29 lines · Robert Olsson
Quoted 2 lines · Felix Radensky
Thanks for your help, Ben. What is skb-recycle patch and where can I find it ?
It's experimental and not updated for almost a year and current implementation does not add anything to SMP. Got some idea how to improve this... but try to keep to slab as long as possible it has been improved.
Routing/bridging on SMP has affinty problem. If you are passing skb's say from eth0 to eth1 and they are bound on different CPU's you get cache boucing since the TX-interrupts come on another CPU.
In a recent test with pktgen:
300 kpps with TX interrupts on same CPU as sender. 198 kpps with TX intr on different CPU as sender.Recycling tries to address this but current implementation fails as said.
But you are probably hit by something else... Check were the drops happens qdisc?. NIC ring RX/TX size, Number of interrupts. ksoftird priority, link
HW_FLOWcontrol, checksumming, affinity etc.Cheers.
--ro
Quoted 2 lines · jamal
Actually seems his biggest problem is he is not running the NAPI driver
Oh! I missed this.
Cheers.
--ro
Hi, Jamal
I guess you were not reading my first posting
very carefully :)
2.4.22 has NAPI capable e1000 driver and I've
compiled the driver with NAPI support.
So running non-NAPI driver is not my problem.
Felix.
Quoted 39 lines · jamal
Actually seems his biggest problem is he is not running the NAPI driver
cheers,
jamalQuoted 27 lines · Robert Olsson
Quoted 2 lines · Felix Radensky
Thanks for your help, Ben. What is skb-recycle patch and where can I find it ?
It's experimental and not updated for almost a year and current implementation does not add anything to SMP. Got some idea how to improve this... but try to keep to slab as long as possible it has been improved.
Routing/bridging on SMP has affinty problem. If you are passing skb's say from eth0 to eth1 and they are bound on different CPU's you get cache boucing since the TX-interrupts come on another CPU.
In a recent test with pktgen:
300 kpps with TX interrupts on same CPU as sender. 198 kpps with TX intr on different CPU as sender.Recycling tries to address this but current implementation fails as said.
But you are probably hit by something else... Check were the drops happens qdisc?. NIC ring RX/TX size, Number of interrupts. ksoftird priority, link
HW_FLOWcontrol, checksumming, affinity etc.Cheers.
--ro
Felix,
Actually i based my comments on the profiles you posted ;-> Are you running any other nics? If not that profile does look strange.
cheers,
jamal
Quoted 54 lines · Felix Radensky
Hi, Jamal
I guess you were not reading my first posting
very carefully :)2.4.22 has NAPI capable e1000 driver and I've
compiled the driver with NAPI support.So running non-NAPI driver is not my problem.
Felix.
Quoted 40 lines · jamal
Actually seems his biggest problem is he is not running the NAPI driver
cheers,
jamalQuoted 30 lines · Robert Olsson
Quoted 2 lines · Felix Radensky
Thanks for your help, Ben. What is skb-recycle patch and where can I find it ?
It's experimental and not updated for almost a year and current implementation does not add anything to SMP. Got some idea how to improve this... but try to keep to slab as long as possible it has been improved.
Routing/bridging on SMP has affinty problem. If you are passing skb's say from eth0 to eth1 and they are bound on different CPU's you get cache boucing since the TX-interrupts come on another CPU.
In a recent test with pktgen:
300 kpps with TX interrupts on same CPU as sender. 198 kpps with TX intr on different CPU as sender.Recycling tries to address this but current implementation fails as said.
But you are probably hit by something else... Check were the drops happens qdisc?. NIC ring RX/TX size, Number of interrupts. ksoftird priority, link
HW_FLOWcontrol, checksumming, affinity etc.Cheers.
--ro
Quoted 5 lines · jamal
Felix,
Actually i based my comments on the profiles you posted ;-> Are you running any other nics? If not that profile does look strange.
Yes netif_rx the non-NAPI entrance to the upper layers is in the profiles.
c019ad44 3404 5.3328 netif_rx
Cheers.
--ro
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 4 lines · Felix Radensky
I've mentioned in my first post that there are zero drops at driver level (as shown by ifconfig). I'm also using IRQ affinity feature, binding interrupts of eth0 and eth1 to CPU0, so I guess NIC ring RX/TX size, link
HW_FLOWcontrol and affinity are not my problems.
Maybe try un-binding the cpus, or binding them to different procs?
Quoted 4 lines
Speaking of checksumming. This can be a problem indeed. ethtool -S shows a lot of
rx_csum_offloaderrors. Scott, what could possibly be a problem ? The NIC is dual port 82546, driver is 5.1.13-k1.
Why would a bridge be checksumming anything?
Quoted 3 lines
I've failed to find a discussion about ksoftird priority. Can someone please provide a link.
I've increased my performance by setting softirqd priority to -18, but since you seem to be dropping someplace other than the driver, I'm not sure that will help.
Quoted 5 lines
Thanks a lot.
Felix.
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 7 lines
Quoted 5 lines
Speaking of checksumming. This can be a problem indeed. ethtool -S shows a lot
ofrx_csum_offloaderrors. Scott, what could possibly be a problem ? The NIC
is dual port 82546, driver is 5.1.13-k1.Why would a bridge be checksumming anything?
A NIC normally checksums every incoming packet when hardware checksumming is enabled. But of course the bridge doesn't use that information.
-Andi
Quoted 4 lines · Felix Radensky
Is slab good enough in 2.4 ? I was thinking that one of the goals of skb-recycle patch was to avoid skb allocations and deallocations which consume quite a lot of CPU time (as profile shows). Are you saying that your patch is not hepling to reduce CPU load ?
Try to understand why/where packets get dropped in your setup to start with. Bridging shouldn't be different from routing which I experiments with.
Check /proc/interrupts, /proc/net/softnet_stat and check for drops at qdisc ( "tc -s qdisc" you might have to readd the qdisc just get the stats)
Possible your TX-side cannot keep up with RX. Often the TX ring is not cleard aggressivly enough at high rates due intr. mitigation etc or possibly HW_FLOWCTRL from sink device. Disable it in your for testing.
When you cure the cause in-kernel-drop you should have packets drops on the DMA-ring (in the driver) with NAPI drivers and no unnecssary skb allocation/CPU use or DMA's.
Quoted 2 lines
echo 1 >
/proc/irq/48/smp_affinity
to avoid this kind of problem, or something else is required ?
If you have both both incoming and outgoing on same CPU there will be no cache bouncing of course and a UP kernel would be faster if this all your job.
Cheers.
--ro
Quoted 5 lines · On Thu, 07 Aug 2003 09:57:14 -0700 Ben Greear <[email protected]> wrote:
Btw, I've considered saving, say, 10k skbs on a list in my module, allocated by
GFP_KERNELat module load time, and using them whenGFP_ATOMICskb_allocfails in the IRQ handling portion of the code....Anyone think that's a good idea? :)
Not really.
GFP_ATOMIC should not fail regularly under normal (even heavy load) operation. If it does, it means the amount of reserved pages the kernel keeps around is not set correctly for your system.
In 2.6.x, play with /proc/sys/vm/min_free_kbytes
Quoted 11 lines · David S. Miller
Quoted 5 lines · On Thu, 07 Aug 2003 09:57:14 -0700 Ben Greear <[email protected]> wrote:
Btw, I've considered saving, say, 10k skbs on a list in my module, allocated by
GFP_KERNELat module load time, and using them whenGFP_ATOMICskb_allocfails in the IRQ handling portion of the code....Anyone think that's a good idea? :)
Quoted 7 lines
Not really.
GFP_ATOMICshould not fail regularly under normal (even heavy load) operation. If it does, it means the amount of reserved pages the kernel keeps around is not set correctly for your system.In 2.6.x, play with
/proc/sys/vm/min_free_kbytes
Anything to set for 2.4? I've looked for how to tune the 2.4 VM for some time, but never found anything.
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 5 lines · On Thu, 07 Aug 2003 12:50:53 -0700 Ben Greear <[email protected]> wrote:
Quoted 1 line · David S. Miller
In 2.6.x, play with
/proc/sys/vm/min_free_kbytesAnything to set for 2.4? I've looked for how to tune the 2.4 VM for some time, but never found anything.
Good question. Nothing exists there.
The per-zone ->pages_min value is what is controlled by this settting. It should be easy to backport the 2.6.x sysctl to 2.4.x, even for an amateur :-)