Skip to contentSkip to conversation

_High_ CPU usage while routing (mostly) small UDP packets

DISCUSSION Karsten Desler 8 participants 32 messages Dec 6, 2004 – Dec 8, 2004

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

Hi,

I'm running a dual Opteron 244 router with two active e1000 interfaces, that mostly deals with small (<200b) udp packets.
I'm using Linux 2.6.10-rc3 (32bit), but I tried 64bit with early 2.6.9-rc versions, and it didn't make much of a difference.

The irq of eth0 is bound to cpu1 while eth1 is bound to cpu0. NAPI is enabled.
Current packetload on eth0 (and reversed on eth1):
115kpps tx
135kpps rx

There are about 200 iptables rules, but the common packet only has to traverse about 20.
conntrack is not loaded.

eth0 and eth1 are running on the same 66MHz/64bit PCI bus.

Kernel-Profiling is running, I don't know how much that contributes to the overall load.

I don't know if that has anything to do with it, but the systemclock is getting out of sync _fast_ (openntpd can't keep up).
ntpdate -b ntp.soohrt.org; sleep 60; ntpdate ntp.soohrt.org:
6 Dec 21:40:39 ntpdate[30146]: adjust time server 134.100.177.5 offset 0.000092 sec 6 Dec 21:41:39 ntpdate[30218]: adjust time server 134.100.177.5 offset 0.006971 sec

Is that the expected cpu usage?
I'd appreciate _any_ pointers, thanks in advance,
Karsten

Current cpu usage:

 Cpu0 :  0.0% us,  0.0% sy,  0.0% ni, 46.3% id,  0.0% wa,  0.0% hi, 53.7% si
 Cpu1 :  0.0% us,  0.0% sy,  0.0% ni, 21.4% id,  0.0% wa,  0.0% hi, 78.6% si

vmstat 5:
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0      0 1804612 100488 104464    0    0     0    32 5965   167  3 68 30  0
 0  1      0 1804548 100488 104464    0    0     0   157 5994    18  0 67 33  0
 1  1      0 1804684 100488 104464    0    0     0    63 5998    19  0 67 33  0
 0  1      0 1804684 100492 104460    0    0     0     4 5985    10  0 68 33  0
 0  1      0 1804620 100492 104460    0    0     0    12 6032    15  0 68 32  0

lspci -vt:
-[00]-+-06.0-[03]--+-00.0 Advanced Micro Devices [AMD] AMD-8111 USB
[...]
+-0a.1 Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC +-0b.0-[01]--+-01.0 Intel Corp. 82545EM Gigabit Ethernet Controller (Fiber) <- eth0
| +-03.0 Intel Corp. 82546GB Gigabit Ethernet Controller <- eth1
| \-03.1 Intel Corp. 82546GB Gigabit Ethernet Controller

arp -n|grep -c incomplete:
73

arp -n|grep -vc incomplete:
778

iptables -nL|grep -v Chain|grep -vc source:
199

readprofile -r; sleep 60; readprofile|sort -n +2:

    76 __do_softirq                               0.3654
    73 dst_alloc                                  0.4148
   489 ip_rcv                                     0.4187
    96 fib_semantic_match                         0.4615
    15 memset                                     0.4688
     8 _write_lock                                0.5000
    37 match                                      0.5781
    72 handle_IRQ_event                           0.6429
   138 fn_hash_lookup                             0.7188
   401 qdisc_restart                              0.7371
    13 _read_unlock                               0.8125
    28 _spin_lock_bh                              0.8750
    70 e1000_rx_checksum                          0.8750
   536 ip_rcv_finish                              0.9306
    33 kfree_skbmem                               1.0312
   248 e1000_intr                                 1.0333
    30 _read_lock                                 1.8750
  1198 ip_forward                                 1.9199
   910 ip_finish_output2                          1.9612
   282 rt_hash_code                               2.2031
   286 pfifo_fast_enqueue                         2.2344
    36 _spin_unlock                               2.2500
  1657 dev_queue_xmit                             2.3014
   230 ip_output                                  2.3958
   905 netif_receive_skb                          2.4592
  2852 e1000_clean_rx_irq                         2.6213
   697 nf_hook_slow                               2.7227
   674 e1000_alloc_rx_buffers                     2.8083
   228 ip_forward_finish                          2.8500
   187 ipt_hook                                   3.8958
   346 kmem_cache_free                            4.3250
   357 pfifo_fast_dequeue                         4.4625
   626 local_bh_enable                            4.8906
   250 e1000_irq_enable                           5.2083
   425 kmem_cache_alloc                           5.3125
  3100 e1000_clean_tx_irq                         5.6985
  1014 nf_iterate                                 5.7614
   284 ipt_route_hook                             5.9167
   701 kfree                                      6.2589
  1063 __kmalloc                                  8.3047
  1605 skb_release_data                          10.0312
  2692 eth_type_trans                            11.2167
  4013 __kfree_skb                               15.6758
  4554 alloc_skb                                 18.9750
 20017 ipt_do_table                              24.0589
 10700 ip_route_input                            30.3977
  1341 _spin_lock                                83.8125
  1483 _read_unlock_bh                           92.6875
  3345 _read_lock_bh                            104.5312
  3185 _spin_unlock_irqrestore                  199.0625
 44402 default_idle                             693.7812
maintainer
#1 lore raw
Quoted 1 line · On Mon, 6 Dec 2004 21:53:05 +0100 Karsten Desler <[email protected]> wrote:

20017 ipt_do_table 24.0589

It's spending nearly half of it's time in iptables. Try to consolidate your rules if possible. This is the part of netfilter that really doesn't scale well at all.

#2 lore raw
Quoted 4 lines · David S. Miller

It's spending nearly half of it's time in iptables. Try to consolidate your rules if possible. This is the part of netfilter that really doesn't scale well at all.

Removing the iptables rules helps reducing the load a little, but the majority of time is still spent somewhere else.

50kpps rx and 43kpps tx.
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0      0 1802956 101032 104464    0    0     0    18   74    26  0 21 79  0
 0  0      0 1802956 101032 104464    0    0     0    61 8810    28  0 25 75  0
 0  0      0 1802956 101032 104464    0    0     0   233 8867    17  0 24 76  0
 0  0      0 1802892 101032 104464    0    0     0     0 8865    16  0 21 79  0
 0  0      0 1802892 101032 104464    0    0     0     0 8772     8  0 18 82  0 <- iptables -F
 0  0      0 1802892 101032 104464    0    0     0    36 8863    18  0 19 81  0
 0  0      0 1802892 101032 104464    0    0     0    80 8700    18  0 18 82  0
 0  0      0 1802956 101032 104464    0    0     0     0 8779     7  0 17 83  0
 0  0      0 1802948 101032 104464    0    0     0   223 8716   278  4 19 76  0

- Karsten

#3 lore raw
Quoted 2 lines · In article <[email protected]> you wrote:

Removing the iptables rules helps reducing the load a little, but the majority of time is still spent somewhere else.

In handling Interrupts. Are those equally sidtributed on eth0 and eth1?

Gruss
Bernd

#14 lore raw
In reply to Karsten Desler
Quoted 8 lines · Karsten Desler
Quoted 4 lines · David S. Miller

It's spending nearly half of it's time in iptables. Try to consolidate your rules if possible. This is the part of netfilter that really doesn't scale well at all.

Removing the iptables rules helps reducing the load a little, but the majority of time is still spent somewhere else.

I had a similar scenario recently with a very low spec box and found it to be the QoS. Disabling traffic shaping and removing the QoS modules made it much faster. I don't know if you're using them but it's worth pointing out.

Cheers,
Con

#15 lore raw
Quoted 3 lines · Con Kolivas

I had a similar scenario recently with a very low spec box and found it to be the QoS. Disabling traffic shaping and removing the QoS modules made it much faster. I don't know if you're using them but it's worth pointing out.

No QoS loaded.
.config is attached.

Thanks,
Karsten

CONFIG_X86=y
CONFIG_MMU=y
CONFIG_UID16=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_EXPERIMENTAL=y
CONFIG_CLEAN_COMPILE=y
CONFIG_LOCK_KERNEL=y
CONFIG_LOCALVERSION=""
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
CONFIG_SYSCTL=y
CONFIG_LOG_BUF_SHIFT=15
CONFIG_KOBJECT_UEVENT=y
CONFIG_EMBEDDED=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SHMEM=y
CONFIG_CC_ALIGN_FUNCTIONS=0
CONFIG_CC_ALIGN_LABELS=0
CONFIG_CC_ALIGN_LOOPS=0
CONFIG_CC_ALIGN_JUMPS=0
CONFIG_X86_PC=y
CONFIG_MK8=y
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_SMP=y
CONFIG_NR_CPUS=2
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_TSC=y
CONFIG_X86_MCE=y
CONFIG_HIGHMEM4G=y
CONFIG_HIGHMEM=y
CONFIG_MTRR=y
CONFIG_HAVE_DEC_LOCK=y
CONFIG_REGPARM=y
CONFIG_ACPI=y
CONFIG_ACPI_BOOT=y
CONFIG_ACPI_INTERPRETER=y
CONFIG_ACPI_BLACKLIST_YEAR=0
CONFIG_ACPI_BUS=y
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_PCI=y
CONFIG_ACPI_SYSTEM=y
CONFIG_PCI=y
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_MSI=y
CONFIG_PCI_NAMES=y
CONFIG_BINFMT_ELF=y
CONFIG_STANDALONE=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_INITRAMFS_SOURCE=""
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_CFQ=y
CONFIG_SCSI=y
CONFIG_BLK_DEV_SD=y
CONFIG_SCSI_SATA=y
CONFIG_SCSI_SATA_SIL=y
CONFIG_SCSI_QLA2XXX=y
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_RAID1=y
CONFIG_NET=y
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_UNIX=y
CONFIG_INET=y
CONFIG_IP_TCPDIAG=y
CONFIG_NETFILTER=y
CONFIG_IP_NF_IPTABLES=y
CONFIG_IP_NF_MATCH_LIMIT=y
CONFIG_IP_NF_MATCH_IPRANGE=y
CONFIG_IP_NF_MATCH_MULTIPORT=y
CONFIG_IP_NF_MATCH_LENGTH=y
CONFIG_IP_NF_MATCH_HASHLIMIT=y
CONFIG_IP_NF_FILTER=y
CONFIG_IP_NF_TARGET_REJECT=y
CONFIG_IP_NF_TARGET_LOG=y
CONFIG_IP_NF_TARGET_ULOG=y
CONFIG_IP_NF_MANGLE=y
CONFIG_NETDEVICES=y
CONFIG_NET_ETHERNET=y
CONFIG_MII=y
CONFIG_NET_PCI=y
CONFIG_E100=y
CONFIG_E100_NAPI=y
CONFIG_E1000=y
CONFIG_E1000_NAPI=y
CONFIG_TIGON3=y
CONFIG_INPUT=y
CONFIG_SOUND_GAMEPORT=y
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_SERIAL_8250_ACPI=y
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_UNIX98_PTYS=y
CONFIG_WATCHDOG=y
CONFIG_SOFT_WATCHDOG=y
CONFIG_RTC=y
CONFIG_HANGCHECK_TIMER=y
CONFIG_VGA_CONSOLE=y
CONFIG_DUMMY_CONSOLE=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_EXT2_FS=y
CONFIG_EXT3_FS=y
CONFIG_JBD=y
CONFIG_DNOTIFY=y
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_RAMFS=y
CONFIG_PARTITION_ADVANCED=y
CONFIG_MSDOS_PARTITION=y
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_ISO8859_1=y
CONFIG_4KSTACKS=y
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_X86_SMP=y
CONFIG_X86_HT=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_X86_TRAMPOLINE=y

#4 lore raw
In reply to Bernd Eckenfels
Quoted 5 lines · Bernd Eckenfels
Quoted 2 lines · In article <[email protected]> you wrote:

Removing the iptables rules helps reducing the load a little, but the majority of time is still spent somewhere else.

In handling Interrupts. Are those equally sidtributed on eth0 and eth1?

Yes they are.

Thanks,
Karsten

CPU0 CPU1

  0:  117199776  133677244    IO-APIC-edge  timer
  1:          0          9    IO-APIC-edge  i8042
  8:          0          4    IO-APIC-edge  rtc
  9:          0          0   IO-APIC-level  acpi
169:        139  893669684   IO-APIC-level  eth0
177:  919803109      30665   IO-APIC-level  eth1
209:     414257     413316   IO-APIC-level  libata
NMI:          0          0 
LOC:  250918849  250918819 
ERR:          0
MIS:          0
#5 lore raw

Your numbers are very suspect. You may be having other issues in the box. You should be able to do much higher packet rates even with iptables compiled in.
Some numbers at:

http://www.suug.ch/sucon/04/slides/pkt_cls.pdf

If all you need is std filtering then consider using tc actions. I do have a suspicion that your problem has to do with your machine more than it does with Linux.

cheers,
jamal

Quoted 26 lines · Karsten Desler
Quoted 5 lines · Bernd Eckenfels
Quoted 2 lines · In article <[email protected]> you wrote:

Removing the iptables rules helps reducing the load a little, but the majority of time is still spent somewhere else.

In handling Interrupts. Are those equally sidtributed on eth0 and eth1?

Yes they are.

Thanks,
Karsten

CPU0 CPU1

  0:  117199776  133677244    IO-APIC-edge  timer
  1:          0          9    IO-APIC-edge  i8042
  8:          0          4    IO-APIC-edge  rtc
  9:          0          0   IO-APIC-level  acpi
169:        139  893669684   IO-APIC-level  eth0
177:  919803109      30665   IO-APIC-level  eth1
209:     414257     413316   IO-APIC-level  libata
NMI:          0          0 
LOC:  250918849  250918819 
ERR:          0
MIS:          0
#6 lore raw
Quoted 4 lines · jamal

Your numbers are very suspect. You may be having other issues in the box. You should be able to do much higher packet rates even with iptables compiled in.

I know, and I have no idea why I'm not.

Quoted 5 lines

Some numbers at:

http://www.suug.ch/sucon/04/slides/pkt_cls.pdf

If all you need is std filtering then consider using tc actions.

Thanks, I'll look into it.

Quoted 2 lines

I do have a suspicion that your problem has to do with your machine more than it does with Linux.

But what could be the reason? I'm really out of ideas.
The only thing I can think off is the 66/64 PCI bus and the disadvantageous placement of the PCI cards, but neither should cause a higher CPU usage. If the bus couldn't keep up, I'd get packetloss.

- Karsten

#7 lore raw
Quoted 4 lines · Karsten Desler

The only thing I can think off is the 66/64 PCI bus and the disadvantageous placement of the PCI cards, but neither should cause a higher CPU usage. If the bus couldn't keep up, I'd get packetloss.

cant tell offhand; it looks like a modern piece of hardware. Are you sure you are using NAPI? This is an e1000, correct?

cheers,
jamal

#8 lore raw
Quoted 8 lines · jamal
Quoted 4 lines

The only thing I can think off is the 66/64 PCI bus and the disadvantageous placement of the PCI cards, but neither should cause a higher CPU usage. If the bus couldn't keep up, I'd get packetloss.

cant tell offhand; it looks like a modern piece of hardware. Are you sure you are using NAPI? This is an e1000, correct?

Yes and yes.

0000:01:01.0 Ethernet controller: Intel Corp. 82545EM Gigabit Ethernet Controller (Fiber) (rev 01) 0000:01:03.0 Ethernet controller: Intel Corp. 82546GB Gigabit Ethernet Controller (rev 03)

driver: e1000
version: 5.5.4-k2-NAPI
firmware-version: N/A
bus-info: 0000:01:03.0

driver: e1000
version: 5.5.4-k2-NAPI
firmware-version: N/A
bus-info: 0000:01:01.0

#9 lore raw
Quoted 11 lines · Karsten Desler

roller (rev 03)

driver: e1000
version: 5.5.4-k2-NAPI
firmware-version: N/A
bus-info: 0000:01:03.0

driver: e1000
version: 5.5.4-k2-NAPI
firmware-version: N/A
bus-info: 0000:01:01.0

Beats me. Make sure it boots NAPI. Also if you can turn off ITR; intel loves to turn on that silly feature.

cheers,
jamal

#10 lore raw
Quoted 2 lines · jamal

Beats me. Make sure it boots NAPI. Also if you can turn off ITR; intel loves to turn on that silly feature.

ITR was in fact activated. I think i've disabled it now (e1000.InterruptThrottleRate=0 in the kernel cmdline). And as I'm reading the e1000 code, there is no way to enable/disable NAPI without a recompile. So the fact that ethtool spat out -NAPI in the version string means that NAPI is actually used.

- Karsten

#11 lore raw
Quoted 9 lines · Karsten Desler
Quoted 2 lines · jamal

Beats me. Make sure it boots NAPI. Also if you can turn off ITR; intel loves to turn on that silly feature.

ITR was in fact activated. I think i've disabled it now (e1000.InterruptThrottleRate=0 in the kernel cmdline). And as I'm reading the e1000 code, there is no way to enable/disable NAPI without a recompile. So the fact that ethtool spat out -NAPI in the version string means that NAPI is actually used.

But looking and the int/s number, I'm not so sure anymore. Is there any other way to find out?

# ethtool -i eth0|grep ^vers
version: 5.5.4-k2-NAPI
# ethtool -i eth1|grep ^vers
version: 5.5.4-k2-NAPI

CPU0 CPU1

169:          5  115554253   IO-APIC-level  eth0
177:   78998347       5568   IO-APIC-level  eth1
# sar -I 169 5 5
11:20:05         INTR    intr/s
11:20:10          169  10401.40
11:20:15          169  10579.80
11:20:20          169  10965.20
11:20:25          169  10768.20
11:20:30          169  10460.60
Average:          169  10635.04
# sar -I 177 5 5
11:18:50         INTR    intr/s
11:18:55          177   4769.74
11:19:00          177   4780.80
11:19:05          177   4669.74
11:19:10          177   4724.55
11:19:15          177   4748.50
Average:          177   4738.67

Cheers,
Karsten

#16 lore raw
In reply to Karsten Desler
Quoted 9 lines · Karsten Desler
Quoted 4 lines · David S. Miller

It's spending nearly half of it's time in iptables. Try to consolidate your rules if possible. This is the part of netfilter that really doesn't scale well at all.

Removing the iptables rules helps reducing the load a little, but the majority of time is still spent somewhere else.

Well with NAPI it can be hard to tell CPU usage. You may need to use something like cyclesoak to get a true idea of CPU left.

Also have a look at http://www.hipac.org/ as netfilter has silly scalability properties.

I also notice that a lot of time is spent allocating and freeing the packet buffers (and possible hidden time due to cache misses due to allocating on one CPU and freeing on another?).
How many [RT]xDescriptors do you have configured by the way?

Anyway attached is a small patch that I used to make the e1000 "own" the packet buffers, and hence it does not alloc/free per packet at all. Now this has only been tested in one configuration where I was just sniffing the packets, so definitely YMMV.

Signature · 2 lines
Pádraig Brady - http://www.pixelbeat.org
--
drivers/net/e1000/e1000_main.c +28 -14
diff -Naur linux-2.4.20-5.2.52/drivers/net/e1000/e1000_main.c linux-2.4.20-pb/drivers/net/e1000/e1000_main.c
@@ -2319,9 +2323,9 @@
 			E1000_DBG("%s: Receive packet consumed multiple buffers\n",
 				netdev->name);
 
-			dev_kfree_skb_irq(skb);
+			//dev_kfree_skb_irq(skb); //PB
 			rx_desc->status = 0;
-			buffer_info->skb = NULL;
+			//buffer_info->skb = NULL; //PB
 
 			if(++i == rx_ring->count) i = 0;
 
@@ -2347,9 +2351,9 @@
 				length--;
 			} else {
 
-				dev_kfree_skb_irq(skb);
+				//dev_kfree_skb_irq(skb); //PB
 				rx_desc->status = 0;
-				buffer_info->skb = NULL;
+				//buffer_info->skb = NULL; //PB
 
 				if(++i == rx_ring->count) i = 0;
 
@@ -2393,7 +2398,7 @@
 		netdev->last_rx = jiffies;
 
 		rx_desc->status = 0;
-		buffer_info->skb = NULL;
+		//buffer_info->skb = NULL; //PB: E1000 doesn't reallocate packets
 
 		if(++i == rx_ring->count) i = 0;
 
@@ -2421,20 +2426,37 @@
 	struct e1000_rx_desc *rx_desc;
 	struct e1000_buffer *buffer_info;
 	struct sk_buff *skb;
-	int reserve_len = 2;
+	int reserve_len = 18;
 	unsigned int i;
 
 	i = rx_ring->next_to_use;
-	buffer_info = &rx_ring->buffer_info[i];
 
-	while(!buffer_info->skb) {
+	while(1) {
+		buffer_info = &rx_ring->buffer_info[i];
+		if (!buffer_info->skb) {
+			; /* try to allocate new buf */
+		} else {
+			if (!skb_shared(buffer_info->skb)) {
+				; /* try to reallocate unused buf */
+			} else {
+				break; /* Better luck next round */
+			}
+		}
+
 		rx_desc = E1000_RX_DESC(*rx_ring, i);
 
-		skb = dev_alloc_skb(adapter->rx_buffer_len + reserve_len);
+		if (!buffer_info->skb) {
+			/* TODO: optimize this alloc size */
+			skb = alloc_skb(adapter->rx_buffer_len + reserve_len, GFP_ATOMIC);
+		} else {
+			skb = realloc_skb(buffer_info->skb, adapter->rx_buffer_len + reserve_len, GFP_ATOMIC);
+		}
 
 		if(!skb) {
-			/* Better luck next round */
-			break;
+			break; /* Better luck next round */
+		} else {
+			skb->dev = netdev;
+			skb_get(skb); /* It's ours. Don't let others deallocate */
 		}
 
 		/* Make buffer alignment 2 beyond a 16 byte boundary
@@ -2443,8 +2465,6 @@
 		 */
 		skb_reserve(skb, reserve_len);
 
-		skb->dev = netdev;
-
 		buffer_info->skb = skb;
 		buffer_info->length = adapter->rx_buffer_len;
 		buffer_info->dma =
@@ -2466,7 +2486,6 @@
 		}
 
 		if(++i == rx_ring->count) i = 0;
-		buffer_info = &rx_ring->buffer_info[i];
 	}
 
 	rx_ring->next_to_use = i;
include/linux/skbuff.h +22 -0
diff -Naur linux-2.4.20-5.2.52/include/linux/skbuff.h linux-2.4.20-pb/include/linux/skbuff.h
@@ -230,6 +230,7 @@
 
 extern void			__kfree_skb(struct sk_buff *skb);
 extern struct sk_buff *		alloc_skb(unsigned int size, int priority);
+extern struct sk_buff *		realloc_skb(struct sk_buff *skb, unsigned int size, int priority);
 extern void			kfree_skbmem(struct sk_buff *skb);
 extern struct sk_buff *		skb_clone(struct sk_buff *skb, int priority);
 extern struct sk_buff *		skb_copy(const struct sk_buff *skb, int priority);
@@ -240,6 +241,7 @@
 						int newheadroom,
 						int newtailroom,
 						int priority);
+extern struct sk_buff *         skb_pad(struct sk_buff *skb, int pad);
 #define dev_kfree_skb(a)	kfree_skb(a)
 extern void	skb_over_panic(struct sk_buff *skb, int len, void *here);
 extern void	skb_under_panic(struct sk_buff *skb, int len, void *here);
@@ -1082,6 +1084,26 @@
 }
 
 /**
+ *      skb_padto       - pad an skbuff up to a minimal size
+ *      @skb: buffer to pad
+ *      @len: minimal length
+ *
+ *      Pads up a buffer to ensure the trailing bytes exist and are
+ *      blanked. If the buffer already contains sufficient data it
+ *      is untouched. Returns the buffer, which may be a replacement
+ *      for the original, or NULL for out of memory - in which case
+ *      the original buffer is still freed.
+ */
+
+static inline struct sk_buff *skb_padto(struct sk_buff *skb, unsigned int len)
+{
+        unsigned int size = skb->len;
+        if(likely(size >= len))
+                return skb;
+        return skb_pad(skb, len-size);
+}
+
+/**
  *	skb_linearize - convert paged skb to linear one
  *	@skb: buffer to linarize
  *	@gfp: allocation mode
net/core/skbuff.c +83 -1

102 lines not on this page · raw message

#17 lore raw
Quoted 8 lines · * [email protected] wrote:
Quoted 7 lines · Karsten Desler
Quoted 4 lines · David S. Miller

It's spending nearly half of it's time in iptables. Try to consolidate your rules if possible. This is the part of netfilter that really doesn't scale well at all.

Quoted 6 lines
Quoted 2 lines

Removing the iptables rules helps reducing the load a little, but the majority of time is still spent somewhere else.

Well with NAPI it can be hard to tell CPU usage. You may need to use something like cyclesoak to get a true idea of CPU left.

Thanks, I'll look into it.

Quoted 2 lines

Also have a look at http://www.hipac.org/ as netfilter has silly scalability properties.

I did before, but I read on Harald Weltes' weblog that 2.4 gives slightly worse pps results than 2.6, and since the cpu usage is as high as it is, I didn't want to take any more performance hits. I'll try to see what performance impact the netfilter rules have during peak load.

Quoted 5 lines

I also notice that a lot of time is spent allocating and freeing the packet buffers (and possible hidden time due to cache misses due to allocating on one CPU and freeing on another?).
How many [RT]xDescriptors do you have configured by the way?

256. I increased them to 1024 shortly after the profiling run, but didn't notice any change in the cpu usage (will try again with cyclesoak).

Quoted 5 lines

Anyway attached is a small patch that I used to make the e1000 "own" the packet buffers, and hence it does not alloc/free per packet at all. Now this has only been tested in one configuration where I was just sniffing the packets, so definitely YMMV.

Thanks, I'll give it a spin.

Cheers,
Karsten

#12 lore raw
In reply to Karsten Desler
Quoted 7 lines · Karsten Desler

But looking and the int/s number, I'm not so sure anymore. Is there any other way to find out?

# sar -I 169 5 5
11:20:05         INTR    intr/s
11:20:10          169  10401.40

That doesnt seem to be too high.
You have a dual opteron 244. You are supposed to be kicking ass with that machine - not 200Kpps+ you are getting with all that CPU overload. Something is wrong with your setup. Unfortunately i cant afford such a machine so i cant see it right off the bat. I know Robert has at least one similar machine; maybe he could help. Robert?

cheers,
jamal

#18 lore raw
In reply to Karsten Desler

Hello!

Well my experience is that it very hard not to say almost impossible to extrapolate idle cpu into any network system capacity. I guess this is what you are trying to do?

Rather load and overload the system with traffic having the characteristics you expect as a bonus you will get some kind proof of robustness and responsiveness a max load. There are tools for this type of tests.

Pádraig! Very funny... I started hacking 2 hours ago on idea I had for long time, This to do a light version of skb recycling based skb->users (the pktgen trick) with very minimal kernel change..

Quoted 5 lines

Anyway attached is a small patch that I used to make the e1000 "own" the packet buffers, and hence it does not alloc/free per packet at all. Now this has only been tested in one configuration where I was just sniffing the packets, so definitely YMMV.

						--ro
#19 lore raw
Quoted 3 lines · Robert Olsson

Well my experience is that it very hard not to say almost impossible to extrapolate idle cpu into any network system capacity. I guess this is what you are trying to do?

Kinda, yes.
I'm trying to evaluate if the behaviour I'm seeing is expected, which would heavily influence my choice of hardware/software for future projects (and of course to optimize the current setup).

Currently I'm having problems capturing packets with tcpdump (lots of "packets dropped by kernel") which indicates to me that there's genuinely not much (enough) idle time sitting around.

Quoted 3 lines

Rather load and overload the system with traffic having the characteristics you expect as a bonus you will get some kind proof of robustness and responsiveness a max load. There are tools for this type of tests.

Will do, that could take a couple of days though.

Cheers,
Karsten

#20 lore raw
Quoted 4 lines · Karsten Desler

Currently I'm having problems capturing packets with tcpdump (lots of "packets dropped by kernel") which indicates to me that there's genuinely not much (enough) idle time sitting around.

Ah, more hints. So you are not trying to forward - rather just packet capturing?
Are you using a tcpdump patched with mmaped packet socket?

The 230-240Kpps you are reporting as a capture dont seem as unreasonable as i thought then. Neither would the CPU use.

cheers,
jamal

#21 lore raw
Quoted 9 lines · jamal
Quoted 4 lines · Karsten Desler

Currently I'm having problems capturing packets with tcpdump (lots of "packets dropped by kernel") which indicates to me that there's genuinely not much (enough) idle time sitting around.

Ah, more hints. So you are not trying to forward - rather just packet capturing?

forward/routing is the goal.

I was just trying to capture a tcpdump to analyze the traffic to generate something that could emulate the trafficpattern for further testing in a non-production environment.

Cheers,
Karsten

#13 lore raw
In reply to jamal
Quoted 5 lines · jamal
Quoted 3 lines
# sar -I 169 5 5
11:20:05         INTR    intr/s
11:20:10          169  10401.40

That doesnt seem to be too high.

To recap: 169 is a fibre e1000, eth1 is one of two ports on a dualport e1000 copper nic. eth1 is still running at about 4k int/s.

14:12:18 INTR intr/s

14:12:23          169  34012.80
14:12:28          169  33977.60
14:12:33          169  34218.16
14:12:38          169  34060.60
14:12:43          169  34252.60
Average:          169  34104.40

Cheers,
Karsten

#22 lore raw
In reply to jamal
Quoted 8 lines · jamal
Quoted 4 lines · Karsten Desler

Currently I'm having problems capturing packets with tcpdump (lots of "packets dropped by kernel") which indicates to me that there's genuinely not much (enough) idle time sitting around.

Quoted 6 lines

Ah, more hints. So you are not trying to forward - rather just packet capturing?
Are you using a tcpdump patched with mmaped packet socket?

The 230-240Kpps you are reporting as a capture dont seem as unreasonable as i thought then. Neither would the CPU use.

Yes this is vital Karsten, otherwise tcpdump
will do 2 syscalls per packet, which is the
bottleneck in my experience.

You may want to try a simpler capture program that uses the kernel PACKET_MMAP feature directly:
http://www.scaramanga.co.uk/code-fu/lincap.c

Signature · 2 lines
Pádraig Brady - http://www.pixelbeat.org
--
#23 lore raw
In reply to Karsten Desler
Quoted 8 lines · Karsten Desler
Quoted 2 lines

Also have a look at http://www.hipac.org/ as netfilter has silly scalability properties.

I did before, but I read on Harald Weltes' weblog that 2.4 gives slightly worse pps results than 2.6, and since the cpu usage is as high as it is, I didn't want to take any more performance hits. I'll try to see what performance impact the netfilter rules have during peak load.

using 2 CPUs
System load: 61.4% || Free: 51.0%(0) 26.3%(1) System load: 59.6% || Free: 53.6%(0) 27.3%(1) System load: 59.6% || Free: 53.6%(0) 27.3%(1) System load: 59.7% || Free: 53.6%(0) 27.0%(1) System load: 60.3% || Free: 53.0%(0) 26.4%(1) System load: 51.9% || Free: 60.4%(0) 35.8%(1) <- iptables -F System load: 50.1% || Free: 62.1%(0) 37.7%(1) System load: 50.1% || Free: 62.0%(0) 37.8%(1) System load: 50.6% || Free: 61.6%(0) 37.2%(1) System load: 50.5% || Free: 61.7%(0) 37.3%(1)

Quoted 8 lines
Quoted 5 lines

I also notice that a lot of time is spent allocating and freeing the packet buffers (and possible hidden time due to cache misses due to allocating on one CPU and freeing on another?).
How many [RT]xDescriptors do you have configured by the way?

256. I increased them to 1024 shortly after the profiling run, but didn't notice any change in the cpu usage (will try again with cyclesoak).

Again, no effect.

Cheers,
Karsten

#27 lore raw
In reply to Karsten Desler
Quoted 3 lines · Karsten Desler

Current packetload on eth0 (and reversed on eth1):
115kpps tx
135kpps rx

I totally forgot to mention: There are approximately 100k concurrent flows.

Quoted 1 line

From dmesg:

IP: routing cache hash table of 16384 buckets, 128Kbytes

Maybe there is some contention on the rt_hash_table spinlocks? Is the attached patch enough to increase the size?

- Karsten

net/ipv4/route.c renamed from net/ipv4/route.c~old +1 -1
@@ -2728,7 +2728,7 @@
 	if (!ipv4_dst_ops.kmem_cachep)
 		panic("IP: failed to allocate ip_dst_cache\n");
 
-	goal = num_physpages >> (26 - PAGE_SHIFT);
+	goal = num_physpages >> (23 - PAGE_SHIFT);
 	if (rhash_entries)
 		goal = (rhash_entries * sizeof(struct rt_hash_bucket)) >> PAGE_SHIFT;
 	for (order = 0; (1UL << order) < goal; order++)
#28 lore raw
Quoted 5 lines · Karsten Desler

I totally forgot to mention: There are approximately 100k concurrent flows.

Quoted 1 line

From dmesg:

IP: routing cache hash table of 16384 buckets, 128Kbytes

You can take a looks at stats w. rtstat. Hash spinning and how many new entires create and how many warm you hit.

Quoted 2 lines

Maybe there is some contention on the rt_hash_table spinlocks? Is the attached patch enough to increase the size?

There is boot option for this now

rhash_entries= [KNL,NET]
Set number of hash buckets for route cache

--ro

#30 lore raw
In reply to Karsten Desler
Quoted 7 lines · Karsten Desler
Quoted 3 lines · Karsten Desler

Current packetload on eth0 (and reversed on eth1):
115kpps tx
135kpps rx

I totally forgot to mention: There are approximately 100k concurrent flows.

;-> Aha. That would make a huge difference. I know of noone who has actually done this level of testing. I have tried upto about 50K flows myself in early 2.6.x and was eventually compute bound. Really try compiling out totaly iptables/netfilter - it will make a difference.
You may also want to try something like LC trie algorithm that Robert and co are playing with to see if it makes a difference with this many flows.

cheers,
jamal

#24 lore raw
In reply to Karsten Desler
Quoted 8 lines · Karsten Desler
Quoted 5 lines

I also notice that a lot of time is spent allocating and freeing the packet buffers (and possible hidden time due to cache misses due to allocating on one CPU and freeing on another?).
How many [RT]xDescriptors do you have configured by the way?

256. I increased them to 1024 shortly after the profiling run, but didn't notice any change in the cpu usage (will try again with cyclesoak).

Have you checked the interrupts rate ? I had an e1000 eating many CPU cycles because it would generate 50000 interrupts/s. Passing the module InterruptThrottleRate=5000 definitely calmed it down, and more than doubled the data rate.

Regards
Willy

#25 lore raw
Quoted 8 lines · Willy Tarreau
Quoted 6 lines · Karsten Desler
Quoted 5 lines

I also notice that a lot of time is spent allocating and freeing the packet buffers (and possible hidden time due to cache misses due to allocating on one CPU and freeing on another?).
How many [RT]xDescriptors do you have configured by the way?

Quoted 7 lines
Quoted 2 lines

256. I increased them to 1024 shortly after the profiling run, but didn't notice any change in the cpu usage (will try again with cyclesoak).

Have you checked the interrupts rate ? I had an e1000 eating many CPU cycles because it would generate 50000 interrupts/s. Passing the module InterruptThrottleRate=5000 definitely calmed it down, and more than doubled the data rate.

I was running mit ITR=3000, but as a test to see if NAPI works, I disabled ITR on eth0 bringing the int/s rate up to 50k.
Is that normal? I always though NAPI was supposed to kick in way earlier. Anyways, I'm going to try different ITR settings to see if they make any difference.

Cheers,
Karsten

#31 lore raw
In reply to jamal
Quoted 9 lines · jamal
Quoted 2 lines · Karsten Desler

I totally forgot to mention: There are approximately 100k concurrent flows.

;-> Aha. That would make a huge difference. I know of noone who has actually done this level of testing. I have tried upto about 50K flows myself in early 2.6.x and was eventually compute bound. Really try compiling out totaly iptables/netfilter - it will make a difference.

Unfortunately I need netfilter (for now, I haven't had time to look into replacing the rules with tc yet).

Quoted 3 lines

You may also want to try something like LC trie algorithm that Robert and co are playing with to see if it makes a difference with this many flows.

On a scale of one to ten, one being "will crash on the first packet", five being "will allow moderate load, but is probably going to crash under high load" and ten being "rock stable" how would the patch be rated? The announcement doesn't look too promising:
| Locking.
| Not yet done.

Also when looking at the profiles, fib_* isn't showing up at (not even near) the top. Testworthy none the less?

ip route|wc -l:
40

profile:
4 fib_validate_source 0.0064
39 fib_semantic_match 0.1875
[...]

    76 ipt_route_hook                             1.5833
   219 __kmalloc                                  1.7109
   985 e1000_clean_tx_irq                         1.8107
   157 kmem_cache_alloc                           1.9625
   405 skb_release_data                           2.5312
   633 eth_type_trans                             2.6375
   394 handle_IRQ_event                           3.5179
  1017 __kfree_skb                                3.9727
   989 alloc_skb                                  4.1208
  1645 ip_route_input                             4.6733
  5128 ipt_do_table                               6.1635
  1678 e1000_intr                                 6.9917
   289 _read_unlock_bh                           18.0625
   616 _read_lock_bh                             19.2500
   418 _spin_lock                                26.1250
  2076 e1000_irq_enable                          43.2500
   881 _spin_unlock_irqrestore                   55.0625

96895 default_idle 1513.9844

Cheers,
Karsten

#26 lore raw
In reply to Karsten Desler
Quoted 6 lines · Karsten Desler

I was running mit ITR=3000, but as a test to see if NAPI works, I disabled ITR on eth0 bringing the int/s rate up to 50k.
Is that normal? I always though NAPI was supposed to kick in way earlier. Anyways, I'm going to try different ITR settings to see if they make any difference.

The one time you would need this ITR crap is when you are running low traffic (relatively speaking: which could mean anything below 100Kpps depending on h/ware). NAPI will consume a little more CPU otherwise - given it does an extra IO (if it kicks in and out on every packet or two). Granted you are doing some new types of tests, so you may be seeing some things we havent experienced before.

Can you put a printk in ->open() of e1000 with
#if napi is defined
printk("%s: NAPI is on\n",dev->name);
#endif

This should print on ifconfig up and indicate whether NAPI is on or not.

cheers,
jamal

#29 lore raw
In reply to Robert Olsson
Quoted 2 lines · Robert Olsson

rhash_entries= [KNL,NET]
Set number of hash buckets for route cache

IP: routing cache hash table of 131072 buckets, 1024Kbytes A 3 percent improvement over the average cpu usage in the last 8 hours compared to the same time yesterday.
Still in the noise.

I'm going to try a day without netfilter next and then the realloc_skb patch.

Cheers,
Karsten