Skip to contentSkip to conversation

10GBE performance drop with net.ipv4.tcp_timestamps=0

DISCUSSION Stefan Priebe 2 participants 17 messages Jun 19, 2012 – Jun 20, 2012

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hello List,

i'm testing 10GBE speed with tweo servers. One with 3.5-rc3 nd thoe other one whith RHEL 6 (2.6.32 kernel).

I noticed that setting
net.ipv4.tcp_timestamps=0

descreased the performance from 9,7 Full Duplex to 3-4Gb/s.

Is this bahviour fine? What should / could i tet?

Greets
Stefan

#1 lore raw
Quoted 5 lines · Stefan Priebe

Hello List,

i'm testing 10GBE speed with tweo servers. One with 3.5-rc3 nd thoe other one whith RHEL 6 (2.6.32 kernel).

Quoted 7 lines

I noticed that setting
net.ipv4.tcp_timestamps=0

descreased the performance from 9,7 Full Duplex to 3-4Gb/s.

Is this bahviour fine? What should / could i tet?

Really, you should provide more input than that, if you really want us to help.

#2 lore raw

Am 19.06.2012 23:31, schrieb Eric Dumazet:

Quoted 7 lines · Stefan Priebe
Quoted 6 lines

i'm testing 10GBE speed with tweo servers. One with 3.5-rc3 nd thoe other one whith RHEL 6 (2.6.32 kernel).

I noticed that setting
net.ipv4.tcp_timestamps=0

Quoted 7 lines
Quoted 4 lines

descreased the performance from 9,7 Full Duplex to 3-4Gb/s.

Is this bahviour fine? What should / could i tet?

Really, you should provide more input than that, if you really want us to help.

*arg* forgot to add the pastebin links. Sorry. Speed degraded in this case from 9,88Gbit/s to 2,45Gbit/s. When i turn on timestamps it's perfect again. Server A has 3.5.0-rc3 and server B has an RHEL6 2.6.32 kernel.

Before:
http://pastebin.com/raw.php?i=1gVraWVc

After:
http://pastebin.com/raw.php?i=NSh8Y29s

Thanks,
Stefan

#3 lore raw
Quoted 21 lines · Stefan Priebe - Profihost AG

Am 19.06.2012 23:31, schrieb Eric Dumazet:

Quoted 7 lines · Stefan Priebe
Quoted 6 lines

i'm testing 10GBE speed with tweo servers. One with 3.5-rc3 nd thoe other one whith RHEL 6 (2.6.32 kernel).

I noticed that setting
net.ipv4.tcp_timestamps=0

Quoted 7 lines
Quoted 4 lines

descreased the performance from 9,7 Full Duplex to 3-4Gb/s.

Is this bahviour fine? What should / could i tet?

Really, you should provide more input than that, if you really want us to help.

*arg* forgot to add the pastebin links. Sorry. Speed degraded in this case from 9,88Gbit/s to 2,45Gbit/s. When i turn on timestamps it's perfect again. Server A has 3.5.0-rc3 and server B has an RHEL6 2.6.32 kernel.

Quoted 5 lines

Before:
http://pastebin.com/raw.php?i=1gVraWVc

After:
http://pastebin.com/raw.php?i=NSh8Y29s

You have a lot of packet losses

add "tc -s -d qdisc" , "ifconfig -a " and "ethtool -S ethX" outputs for both servers

#4 lore raw

Am 20.06.2012 09:22, schrieb Eric Dumazet:

Quoted 9 lines · Stefan Priebe - Profihost AG
Quoted 6 lines

Am 19.06.2012 23:31, schrieb Eric Dumazet:
Before:
http://pastebin.com/raw.php?i=1gVraWVc

After:
http://pastebin.com/raw.php?i=NSh8Y29s

You have a lot of packet losses

But this ONLY happens with tcp_timestamps=0.

Quoted 2 lines

add "tc -s -d qdisc" , "ifconfig -a " and "ethtool -S ethX" outputs for both servers

eth2 is the 10Gb device on both systems server a has kernel 3.5 server b has rhel 6 kernel

Server A:

# tc -s -d qdisc
RTNETLINK answers: Operation not supported
Dump terminated

Server B (eth2 is the 10GB/s device):

# tc -s -d qdisc
qdisc mq 0: dev eth0 root
 Sent 55151 bytes 555 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0
qdisc mq 0: dev eth2 root
 Sent 38374148475 bytes 2774405 pkt (dropped 0, overlimits 0 requeues 4)
 rate 0bit 0pps backlog 0b 0p requeues 4

ifconfig -a:
http://pastebin.com/raw.php?i=M3QHQjSU

ethtool -S:
http://pastebin.com/raw.php?i=Eap05xKc

Thanks again,
Stefan

#5 lore raw
Quoted 6 lines · Stefan Priebe - Profihost AG

Am 20.06.2012 09:22, schrieb Eric Dumazet:

Quoted 5 lines · Stefan Priebe - Profihost AG
Quoted 4 lines

Am 19.06.2012 23:31, schrieb Eric Dumazet:
Before:
http://pastebin.com/raw.php?i=1gVraWVc

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

After:
http://pastebin.com/raw.php?i=NSh8Y29s

You have a lot of packet losses

But this ONLY happens with tcp_timestamps=0.

Yes, you already told that in subject line.

single tcp flow ?

You seem to have a switch or something that drops packets in this case.

You could try to rate limit to 9Gb/s and see if it is better.

Here, I roughly have same bandwidth with tcp_timestamps on or off, with ixgbe cards and net-next kernels.

#6 lore raw

Am 20.06.2012 10:41, schrieb Eric Dumazet:

Quoted 5 lines · Stefan Priebe - Profihost AG
Quoted 1 line

Am 20.06.2012 09:22, schrieb Eric Dumazet:

Yes, you already told that in subject line.

single tcp flow ?

I use iperf - i think it uses just a single tcp flow. But i'm not sure.

Quoted 2 lines

You seem to have a switch or something that drops packets in this case. You could try to rate limit to 9Gb/s and see if it is better.

Sadly i can't rate limit to 9Gbit/s on the switch.

Quoted 2 lines

Here, I roughly have same bandwidth with tcp_timestamps on or off, with ixgbe cards and net-next kernels.

Mhm strange. Do you have any vague idea what could cause this? Any wrong reordering of the packets without tcp_timestamps?

Stefan

#7 lore raw

Am 20.06.2012 11:06, schrieb Stefan Priebe - Profihost AG:

Quoted 8 lines
Quoted 2 lines

You seem to have a switch or something that drops packets in this case. You could try to rate limit to 9Gb/s and see if it is better.

Sadly i can't rate limit to 9Gbit/s on the switch.

Quoted 2 lines

Here, I roughly have same bandwidth with tcp_timestamps on or off, with ixgbe cards and net-next kernels.

Mhm strange. Do you have any vague idea what could cause this? Any wrong reordering of the packets without tcp_timestamps?

I've now done another test without the switch. So both systems where direct attached and still the same. Without tcp_timstamps speed drops to 2-4Gbit/s.

Stefan

#16 lore raw
In reply to Stefan Priebe - Profihost AG
Quoted 11 lines · Stefan Priebe - Profihost AG

Am 20.06.2012 10:41, schrieb Eric Dumazet:

Quoted 5 lines · Stefan Priebe - Profihost AG
Quoted 1 line

Am 20.06.2012 09:22, schrieb Eric Dumazet:

Yes, you already told that in subject line.

single tcp flow ?

I use iperf - i think it uses just a single tcp flow. But i'm not sure.

Quoted 2 lines

You seem to have a switch or something that drops packets in this case. You could try to rate limit to 9Gb/s and see if it is better.

Sadly i can't rate limit to 9Gbit/s on the switch.

I was suggesting rate limiting on your linux sender machine, just to verify if its indeed a problem on the path.

Its a matter of a "tc qdisc ..." command ;)

Or maybe iperf has an option for rate limiting.

#8 lore raw
In reply to Stefan Priebe - Profihost AG
Quoted 7 lines · Stefan Priebe - Profihost AG

Am 20.06.2012 11:06, schrieb Stefan Priebe - Profihost AG:

Quoted 6 lines
Quoted 2 lines

You seem to have a switch or something that drops packets in this case. You could try to rate limit to 9Gb/s and see if it is better.

Sadly i can't rate limit to 9Gbit/s on the switch.

Quoted 2 lines

Here, I roughly have same bandwidth with tcp_timestamps on or off, with ixgbe cards and net-next kernels.

Quoted 8 lines
Quoted 2 lines

Mhm strange. Do you have any vague idea what could cause this? Any wrong reordering of the packets without tcp_timestamps?

I've now done another test without the switch. So both systems where direct attached and still the same. Without tcp_timstamps speed drops to 2-4Gbit/s.

Stefan

If you exchange sender/receiver role between linux kernel versions, is it the same problem ?

#9 lore raw

Am 20.06.2012 11:17, schrieb Eric Dumazet:

Quoted 8 lines · Stefan Priebe - Profihost AG
Quoted 4 lines

Am 20.06.2012 11:06, schrieb Stefan Priebe - Profihost AG:

Quoted 3 lines
Quoted 2 lines

You seem to have a switch or something that drops packets in this case. You could try to rate limit to 9Gb/s and see if it is better.

Sadly i can't rate limit to 9Gbit/s on the switch.

If you exchange sender/receiver role between linux kernel versions, is it the same problem ?

I'm testing in both directions. So both are sending and receiving.

I've now made tests with only one sending an the other receiving.

When server B is the sender i get 4Gbit/s. When server A is the sender i get full 9,9Gbit/s.

Stefan

#10 lore raw
Quoted 9 lines · Stefan Priebe - Profihost AG

Am 20.06.2012 11:17, schrieb Eric Dumazet:

Quoted 8 lines · Stefan Priebe - Profihost AG
Quoted 4 lines

Am 20.06.2012 11:06, schrieb Stefan Priebe - Profihost AG:

Quoted 3 lines
Quoted 2 lines

You seem to have a switch or something that drops packets in this case. You could try to rate limit to 9Gb/s and see if it is better.

Sadly i can't rate limit to 9Gbit/s on the switch.

If you exchange sender/receiver role between linux kernel versions, is it the same problem ?

Quoted 8 lines

I'm testing in both directions. So both are sending and receiving.

I've now made tests with only one sending an the other receiving.

When server B is the sender i get 4Gbit/s. When server A is the sender i get full 9,9Gbit/s.

Stefan

To rule out bad hardware, could you try a 3.5-rc3 kernel on B ?

#11 lore raw

Am 20.06.2012 11:28, schrieb Eric Dumazet:

Quoted 10 lines · Stefan Priebe - Profihost AG
Quoted 9 lines

Am 20.06.2012 11:17, schrieb Eric Dumazet:

Quoted 8 lines · Stefan Priebe - Profihost AG
Quoted 4 lines

Am 20.06.2012 11:06, schrieb Stefan Priebe - Profihost AG:

Quoted 3 lines
Quoted 2 lines

You seem to have a switch or something that drops packets in this case. You could try to rate limit to 9Gb/s and see if it is better.

Sadly i can't rate limit to 9Gbit/s on the switch.

If you exchange sender/receiver role between linux kernel versions, is it the same problem ?

Quoted 8 lines
Quoted 7 lines

I'm testing in both directions. So both are sending and receiving.

I've now made tests with only one sending an the other receiving.

When server B is the sender i get 4Gbit/s. When server A is the sender i get full 9,9Gbit/s.

To rule out bad hardware, could you try a 3.5-rc3 kernel on B ?

Sure. In that case i get 4Gbit/s in both variants. I also tried two other different machines same result.

Stefan

#12 lore raw
Quoted 3 lines · Stefan Priebe - Profihost AG

Sure. In that case i get 4Gbit/s in both variants. I also tried two other different machines same result.

So 3.5 on receiver is the problem, it seems ?

And you checked all the stuff about irq affinities, i presume, since a lot of things might have changed between 2.6.32 and 3.5 ?

cat /proc/interrupts

what kind of NIC it is ?

#13 lore raw

Am 20.06.2012 11:47, schrieb Eric Dumazet:

Quoted 7 lines · Stefan Priebe - Profihost AG
Quoted 4 lines

Sure. In that case i get 4Gbit/s in both variants. I also tried two other different machines same result.

So 3.5 on receiver is the problem, it seems ?

Yes.

Quoted 2 lines

And you checked all the stuff about irq affinities, i presume, since a lot of things might have changed between 2.6.32 and 3.5 ?

It is a single core E5 Xeon - i've set the affinity like this:

eth2 mask=1 for /proc/irq/83/smp_affinity
eth2 mask=2 for /proc/irq/84/smp_affinity
eth2 mask=4 for /proc/irq/85/smp_affinity
eth2 mask=8 for /proc/irq/86/smp_affinity
eth2 mask=10 for /proc/irq/87/smp_affinity
eth2 mask=20 for /proc/irq/88/smp_affinity
eth2 mask=40 for /proc/irq/89/smp_affinity
eth2 mask=80 for /proc/irq/90/smp_affinity

Quoted 1 line

cat /proc/interrupts

# cat /proc/interrupts
            CPU0       CPU1       CPU2       CPU3       CPU4       CPU5 
      CPU6       CPU7
   0:        141          0          0          0          0          0 
         0          0   IO-APIC-edge      timer
   1:          1          8          0          0          0          0 
         0          0   IO-APIC-edge      i8042
   9:          0          0          0          0          0          0 
         0          0   IO-APIC-fasteoi   acpi
  12:          0          3          0          0          0          0 
         0          0   IO-APIC-edge      i8042
  14:          0          0          0          0          0          0 
         0          0   IO-APIC-edge      ide0
  15:          0          0          0          0          0          0 
         0          0   IO-APIC-edge      ide1
  16:          0          0         26          0          0          0 
         0          0   IO-APIC-fasteoi   ehci_hcd:usb1
  23:          0          0         30          0          0          0 
         0          0   IO-APIC-fasteoi   ehci_hcd:usb2
  64:          0          0          0      81979          0          0 
         0          0   PCI-MSI-edge      ahci
  65:          0          0          0          1          0          0 
         0          0   PCI-MSI-edge      eth0
  66:          0          0          0          0       1090          0 
         0          0   PCI-MSI-edge      eth0-TxRx-0
  67:          0          0          0          0        411          0 
         0          0   PCI-MSI-edge      eth0-TxRx-1
  68:          0          0          0          0        592          0 
         0          0   PCI-MSI-edge      eth0-TxRx-2
  69:          0          0          0          0        472          0 
         0          0   PCI-MSI-edge      eth0-TxRx-3
  70:          0          0          0          0          0       1196 
         0          0   PCI-MSI-edge      eth0-TxRx-4
  71:          0          0          0          0          0        374 
         0          0   PCI-MSI-edge      eth0-TxRx-5
  72:          0          0          0          0          0        405 
         0          0   PCI-MSI-edge      eth0-TxRx-6
  73:          0          0          0          0          0        468 
         0          0   PCI-MSI-edge      eth0-TxRx-7
  83:      31278          0          0         65          0          0 
         0          0   PCI-MSI-edge      eth2-TxRx-0
  84:          0      36311          0          0         61          0 
         0          0   PCI-MSI-edge      eth2-TxRx-1
  85:          0          0      46189          0         61          0 
         0          0   PCI-MSI-edge      eth2-TxRx-2
  86:          0          0          0      28712         67          0 
         0          0   PCI-MSI-edge      eth2-TxRx-3
  87:          0          0          0          0      28089          0 
         0          0   PCI-MSI-edge      eth2-TxRx-4
  88:          0          0          0          0          0      34982 
         0          0   PCI-MSI-edge      eth2-TxRx-5
  89:          0          0          0          0          0         61 
     32420          0   PCI-MSI-edge      eth2-TxRx-6
  90:          0          0          0          0          0         61 
         0      25922   PCI-MSI-edge      eth2-TxRx-7
  91:          0          0          0          0          0          3 
         0          0   PCI-MSI-edge      eth2
 NMI:         13         12         15         22          5          5 
         5          5   Non-maskable interrupts
 LOC:      58919      61420      65519      82647      35519      40489 
     27141      30228   Local timer interrupts
 SPU:          0          0          0          0          0          0 
         0          0   Spurious interrupts
 PMI:         13         12         15         22          5          5 
         5          5   Performance monitoring interrupts
 IWI:          0          0          0          0          0          0 
         0          0   IRQ work interrupts
 RTR:          6          0          0          0          0          0 
         0          0   APIC ICR read retries
 RES:      15116       4521       2418       1814       2375       1615 
      1488       1367   Rescheduling interrupts
 CAL:        134        148        100        162        170        172 
       172        172   Function call interrupts
 TLB:        422        486        415        483        460        460 
       476        398   TLB shootdowns
 TRM:          0          0          0          0          0          0 
         0          0   Thermal event interrupts
 THR:          0          0          0          0          0          0 
         0          0   Threshold APIC interrupts
 MCE:          0          0          0          0          0          0 
         0          0   Machine check exceptions
 MCP:          4          4          4          4          4          4 
         4          4   Machine check polls
Quoted 1 line

what kind of NIC it is ?

# lspci | grep 10-Giga
06:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
06:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)

Stefan

#14 lore raw
Quoted 7 lines · Stefan Priebe - Profihost AG

Am 20.06.2012 11:47, schrieb Eric Dumazet:

Quoted 6 lines · Stefan Priebe - Profihost AG
Quoted 4 lines

Sure. In that case i get 4Gbit/s in both variants. I also tried two other different machines same result.

Quoted 7 lines
Quoted 1 line

So 3.5 on receiver is the problem, it seems ?

Yes.

Quoted 2 lines

And you checked all the stuff about irq affinities, i presume, since a lot of things might have changed between 2.6.32 and 3.5 ?

It is a single core E5 Xeon - i've set the affinity like this:

And you still have the retransmits in "netstat -s" output ?

Might be a firmware or pci issue, I have same cards but no problem here.

Check LRO is on ?

ethtool -k eth2

#15 lore raw
Quoted 16 lines · Eric Dumazet
Quoted 7 lines · Stefan Priebe - Profihost AG

Am 20.06.2012 11:47, schrieb Eric Dumazet:

Quoted 6 lines · Stefan Priebe - Profihost AG
Quoted 4 lines

Sure. In that case i get 4Gbit/s in both variants. I also tried two other different machines same result.

Quoted 7 lines
Quoted 1 line

So 3.5 on receiver is the problem, it seems ?

Yes.

Quoted 2 lines

And you checked all the stuff about irq affinities, i presume, since a lot of things might have changed between 2.6.32 and 3.5 ?

It is a single core E5 Xeon - i've set the affinity like this:

Quoted 8 lines

And you still have the retransmits in "netstat -s" output ?

Might be a firmware or pci issue, I have same cards but no problem here.

Check LRO is on ?

ethtool -k eth2

Ah, your ethtool -S gives strange fdir_miss counts, you should ask Intel guys help maybe...