10GBE performance drop with net.ipv4.tcp_timestamps=0
Subsystems (2)
Hello List,
i'm testing 10GBE speed with tweo servers. One with 3.5-rc3 nd thoe other one whith RHEL 6 (2.6.32 kernel).
I noticed that setting
net.ipv4.tcp_timestamps=0
descreased the performance from 9,7 Full Duplex to 3-4Gb/s.
Is this bahviour fine? What should / could i tet?
Greets
Stefan
Quoted 5 lines · Stefan Priebe
Hello List,
i'm testing 10GBE speed with tweo servers. One with 3.5-rc3 nd thoe other one whith RHEL 6 (2.6.32 kernel).
Quoted 7 lines
I noticed that setting
net.ipv4.tcp_timestamps=0descreased the performance from 9,7 Full Duplex to 3-4Gb/s.
Is this bahviour fine? What should / could i tet?
Really, you should provide more input than that, if you really want us to help.
Am 19.06.2012 23:31, schrieb Eric Dumazet:
Quoted 7 lines · Stefan Priebe
Quoted 6 lines
i'm testing 10GBE speed with tweo servers. One with 3.5-rc3 nd thoe other one whith RHEL 6 (2.6.32 kernel).
I noticed that setting
net.ipv4.tcp_timestamps=0
Quoted 7 lines
Quoted 4 lines
descreased the performance from 9,7 Full Duplex to 3-4Gb/s.
Is this bahviour fine? What should / could i tet?
Really, you should provide more input than that, if you really want us to help.
*arg* forgot to add the pastebin links. Sorry. Speed degraded in this case from 9,88Gbit/s to 2,45Gbit/s. When i turn on timestamps it's perfect again. Server A has 3.5.0-rc3 and server B has an RHEL6 2.6.32 kernel.
Before:
http://pastebin.com/raw.php?i=1gVraWVc
After:
http://pastebin.com/raw.php?i=NSh8Y29s
Thanks,
Stefan
Quoted 21 lines · Stefan Priebe - Profihost AG
Am 19.06.2012 23:31, schrieb Eric Dumazet:
Quoted 7 lines · Stefan Priebe
Quoted 6 lines
i'm testing 10GBE speed with tweo servers. One with 3.5-rc3 nd thoe other one whith RHEL 6 (2.6.32 kernel).
I noticed that setting
net.ipv4.tcp_timestamps=0Quoted 7 lines
Quoted 4 lines
descreased the performance from 9,7 Full Duplex to 3-4Gb/s.
Is this bahviour fine? What should / could i tet?
Really, you should provide more input than that, if you really want us to help.
*arg* forgot to add the pastebin links. Sorry. Speed degraded in this case from 9,88Gbit/s to 2,45Gbit/s. When i turn on timestamps it's perfect again. Server A has 3.5.0-rc3 and server B has an RHEL6 2.6.32 kernel.
Quoted 5 lines
You have a lot of packet losses
add "tc -s -d qdisc" , "ifconfig -a " and "ethtool -S ethX" outputs for both servers
Am 20.06.2012 09:22, schrieb Eric Dumazet:
Quoted 9 lines · Stefan Priebe - Profihost AG
Quoted 6 lines
Am 19.06.2012 23:31, schrieb Eric Dumazet:
Before:
http://pastebin.com/raw.php?i=1gVraWVcYou have a lot of packet losses
But this ONLY happens with tcp_timestamps=0.
Quoted 2 lines
add "tc -s -d qdisc" , "ifconfig -a " and "ethtool -S ethX" outputs for both servers
eth2 is the 10Gb device on both systems server a has kernel 3.5 server b has rhel 6 kernel
Server A:
# tc -s -d qdisc RTNETLINK answers: Operation not supported Dump terminated
Server B (eth2 is the 10GB/s device):
# tc -s -d qdisc qdisc mq 0: dev eth0 root Sent 55151 bytes 555 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 qdisc mq 0: dev eth2 root Sent 38374148475 bytes 2774405 pkt (dropped 0, overlimits 0 requeues 4) rate 0bit 0pps backlog 0b 0p requeues 4
ifconfig -a:
http://pastebin.com/raw.php?i=M3QHQjSU
ethtool -S:
http://pastebin.com/raw.php?i=Eap05xKc
Thanks again,
Stefan
Quoted 6 lines · Stefan Priebe - Profihost AG
Am 20.06.2012 09:22, schrieb Eric Dumazet:
Quoted 5 lines · Stefan Priebe - Profihost AG
Quoted 4 lines
Am 19.06.2012 23:31, schrieb Eric Dumazet:
Before:
http://pastebin.com/raw.php?i=1gVraWVc
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
You have a lot of packet losses
But this ONLY happens with tcp_timestamps=0.
Yes, you already told that in subject line.
single tcp flow ?
You seem to have a switch or something that drops packets in this case.
You could try to rate limit to 9Gb/s and see if it is better.
Here, I roughly have same bandwidth with tcp_timestamps on or off, with ixgbe cards and net-next kernels.
Am 20.06.2012 10:41, schrieb Eric Dumazet:
Quoted 5 lines · Stefan Priebe - Profihost AG
Quoted 1 line
Am 20.06.2012 09:22, schrieb Eric Dumazet:
Yes, you already told that in subject line.
single tcp flow ?
I use iperf - i think it uses just a single tcp flow. But i'm not sure.
Quoted 2 lines
You seem to have a switch or something that drops packets in this case. You could try to rate limit to 9Gb/s and see if it is better.
Sadly i can't rate limit to 9Gbit/s on the switch.
Quoted 2 lines
Here, I roughly have same bandwidth with
tcp_timestampson or off, with ixgbe cards and net-next kernels.
Mhm strange. Do you have any vague idea what could cause this? Any wrong reordering of the packets without tcp_timestamps?
Stefan
Am 20.06.2012 11:06, schrieb Stefan Priebe - Profihost AG:
Quoted 8 lines
Quoted 2 lines
You seem to have a switch or something that drops packets in this case. You could try to rate limit to 9Gb/s and see if it is better.
Sadly i can't rate limit to 9Gbit/s on the switch.
Quoted 2 lines
Here, I roughly have same bandwidth with
tcp_timestampson or off, with ixgbe cards and net-next kernels.Mhm strange. Do you have any vague idea what could cause this? Any wrong reordering of the packets without
tcp_timestamps?
I've now done another test without the switch. So both systems where direct attached and still the same. Without tcp_timstamps speed drops to 2-4Gbit/s.
Stefan
Quoted 11 lines · Stefan Priebe - Profihost AG
Am 20.06.2012 10:41, schrieb Eric Dumazet:
Quoted 5 lines · Stefan Priebe - Profihost AG
Quoted 1 line
Am 20.06.2012 09:22, schrieb Eric Dumazet:
Yes, you already told that in subject line.
single tcp flow ?
I use iperf - i think it uses just a single tcp flow. But i'm not sure.
Quoted 2 lines
You seem to have a switch or something that drops packets in this case. You could try to rate limit to 9Gb/s and see if it is better.
Sadly i can't rate limit to 9Gbit/s on the switch.
I was suggesting rate limiting on your linux sender machine, just to verify if its indeed a problem on the path.
Its a matter of a "tc qdisc ..." command ;)
Or maybe iperf has an option for rate limiting.
Quoted 7 lines · Stefan Priebe - Profihost AG
Am 20.06.2012 11:06, schrieb Stefan Priebe - Profihost AG:
Quoted 6 lines
Quoted 2 lines
You seem to have a switch or something that drops packets in this case. You could try to rate limit to 9Gb/s and see if it is better.
Sadly i can't rate limit to 9Gbit/s on the switch.
Quoted 2 lines
Here, I roughly have same bandwidth with
tcp_timestampson or off, with ixgbe cards and net-next kernels.
Quoted 8 lines
Quoted 2 lines
Mhm strange. Do you have any vague idea what could cause this? Any wrong reordering of the packets without
tcp_timestamps?I've now done another test without the switch. So both systems where direct attached and still the same. Without
tcp_timstampsspeed drops to 2-4Gbit/s.Stefan
If you exchange sender/receiver role between linux kernel versions, is it the same problem ?
Am 20.06.2012 11:17, schrieb Eric Dumazet:
Quoted 8 lines · Stefan Priebe - Profihost AG
Quoted 4 lines
Am 20.06.2012 11:06, schrieb Stefan Priebe - Profihost AG:
Quoted 3 lines
Quoted 2 lines
You seem to have a switch or something that drops packets in this case. You could try to rate limit to 9Gb/s and see if it is better.
Sadly i can't rate limit to 9Gbit/s on the switch.
If you exchange sender/receiver role between linux kernel versions, is it the same problem ?
I'm testing in both directions. So both are sending and receiving.
I've now made tests with only one sending an the other receiving.
When server B is the sender i get 4Gbit/s. When server A is the sender i get full 9,9Gbit/s.
Stefan
Quoted 9 lines · Stefan Priebe - Profihost AG
Am 20.06.2012 11:17, schrieb Eric Dumazet:
Quoted 8 lines · Stefan Priebe - Profihost AG
Quoted 4 lines
Am 20.06.2012 11:06, schrieb Stefan Priebe - Profihost AG:
Quoted 3 lines
Quoted 2 lines
You seem to have a switch or something that drops packets in this case. You could try to rate limit to 9Gb/s and see if it is better.
Sadly i can't rate limit to 9Gbit/s on the switch.
If you exchange sender/receiver role between linux kernel versions, is it the same problem ?
Quoted 8 lines
I'm testing in both directions. So both are sending and receiving.
I've now made tests with only one sending an the other receiving.
When server B is the sender i get 4Gbit/s. When server A is the sender i get full 9,9Gbit/s.
Stefan
To rule out bad hardware, could you try a 3.5-rc3 kernel on B ?
Am 20.06.2012 11:28, schrieb Eric Dumazet:
Quoted 10 lines · Stefan Priebe - Profihost AG
Quoted 9 lines
Am 20.06.2012 11:17, schrieb Eric Dumazet:
Quoted 8 lines · Stefan Priebe - Profihost AG
Quoted 4 lines
Am 20.06.2012 11:06, schrieb Stefan Priebe - Profihost AG:
Quoted 3 lines
Quoted 2 lines
You seem to have a switch or something that drops packets in this case. You could try to rate limit to 9Gb/s and see if it is better.
Sadly i can't rate limit to 9Gbit/s on the switch.
If you exchange sender/receiver role between linux kernel versions, is it the same problem ?
Quoted 8 lines
Quoted 7 lines
I'm testing in both directions. So both are sending and receiving.
I've now made tests with only one sending an the other receiving.
When server B is the sender i get 4Gbit/s. When server A is the sender i get full 9,9Gbit/s.
To rule out bad hardware, could you try a 3.5-rc3 kernel on B ?
Sure. In that case i get 4Gbit/s in both variants. I also tried two other different machines same result.
Stefan
Quoted 3 lines · Stefan Priebe - Profihost AG
Sure. In that case i get 4Gbit/s in both variants. I also tried two other different machines same result.
So 3.5 on receiver is the problem, it seems ?
And you checked all the stuff about irq affinities, i presume, since a lot of things might have changed between 2.6.32 and 3.5 ?
cat /proc/interrupts
what kind of NIC it is ?
Am 20.06.2012 11:47, schrieb Eric Dumazet:
Quoted 7 lines · Stefan Priebe - Profihost AG
Quoted 4 lines
Sure. In that case i get 4Gbit/s in both variants. I also tried two other different machines same result.
So 3.5 on receiver is the problem, it seems ?
Yes.
Quoted 2 lines
And you checked all the stuff about irq affinities, i presume, since a lot of things might have changed between 2.6.32 and 3.5 ?
It is a single core E5 Xeon - i've set the affinity like this:
eth2 mask=1 for /proc/irq/83/smp_affinity
eth2 mask=2 for /proc/irq/84/smp_affinity
eth2 mask=4 for /proc/irq/85/smp_affinity
eth2 mask=8 for /proc/irq/86/smp_affinity
eth2 mask=10 for /proc/irq/87/smp_affinity
eth2 mask=20 for /proc/irq/88/smp_affinity
eth2 mask=40 for /proc/irq/89/smp_affinity
eth2 mask=80 for /proc/irq/90/smp_affinity
Quoted 1 line
cat
/proc/interrupts
# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5
CPU6 CPU7
0: 141 0 0 0 0 0
0 0 IO-APIC-edge timer
1: 1 8 0 0 0 0
0 0 IO-APIC-edge i8042
9: 0 0 0 0 0 0
0 0 IO-APIC-fasteoi acpi
12: 0 3 0 0 0 0
0 0 IO-APIC-edge i8042
14: 0 0 0 0 0 0
0 0 IO-APIC-edge ide0
15: 0 0 0 0 0 0
0 0 IO-APIC-edge ide1
16: 0 0 26 0 0 0
0 0 IO-APIC-fasteoi ehci_hcd:usb1
23: 0 0 30 0 0 0
0 0 IO-APIC-fasteoi ehci_hcd:usb2
64: 0 0 0 81979 0 0
0 0 PCI-MSI-edge ahci
65: 0 0 0 1 0 0
0 0 PCI-MSI-edge eth0
66: 0 0 0 0 1090 0
0 0 PCI-MSI-edge eth0-TxRx-0
67: 0 0 0 0 411 0
0 0 PCI-MSI-edge eth0-TxRx-1
68: 0 0 0 0 592 0
0 0 PCI-MSI-edge eth0-TxRx-2
69: 0 0 0 0 472 0
0 0 PCI-MSI-edge eth0-TxRx-3
70: 0 0 0 0 0 1196
0 0 PCI-MSI-edge eth0-TxRx-4
71: 0 0 0 0 0 374
0 0 PCI-MSI-edge eth0-TxRx-5
72: 0 0 0 0 0 405
0 0 PCI-MSI-edge eth0-TxRx-6
73: 0 0 0 0 0 468
0 0 PCI-MSI-edge eth0-TxRx-7
83: 31278 0 0 65 0 0
0 0 PCI-MSI-edge eth2-TxRx-0
84: 0 36311 0 0 61 0
0 0 PCI-MSI-edge eth2-TxRx-1
85: 0 0 46189 0 61 0
0 0 PCI-MSI-edge eth2-TxRx-2
86: 0 0 0 28712 67 0
0 0 PCI-MSI-edge eth2-TxRx-3
87: 0 0 0 0 28089 0
0 0 PCI-MSI-edge eth2-TxRx-4
88: 0 0 0 0 0 34982
0 0 PCI-MSI-edge eth2-TxRx-5
89: 0 0 0 0 0 61
32420 0 PCI-MSI-edge eth2-TxRx-6
90: 0 0 0 0 0 61
0 25922 PCI-MSI-edge eth2-TxRx-7
91: 0 0 0 0 0 3
0 0 PCI-MSI-edge eth2
NMI: 13 12 15 22 5 5
5 5 Non-maskable interrupts
LOC: 58919 61420 65519 82647 35519 40489
27141 30228 Local timer interrupts
SPU: 0 0 0 0 0 0
0 0 Spurious interrupts
PMI: 13 12 15 22 5 5
5 5 Performance monitoring interrupts
IWI: 0 0 0 0 0 0
0 0 IRQ work interrupts
RTR: 6 0 0 0 0 0
0 0 APIC ICR read retries
RES: 15116 4521 2418 1814 2375 1615
1488 1367 Rescheduling interrupts
CAL: 134 148 100 162 170 172
172 172 Function call interrupts
TLB: 422 486 415 483 460 460
476 398 TLB shootdowns
TRM: 0 0 0 0 0 0
0 0 Thermal event interrupts
THR: 0 0 0 0 0 0
0 0 Threshold APIC interrupts
MCE: 0 0 0 0 0 0
0 0 Machine check exceptions
MCP: 4 4 4 4 4 4
4 4 Machine check pollsQuoted 1 line
what kind of NIC it is ?
# lspci | grep 10-Giga 06:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) 06:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
Stefan
Quoted 7 lines · Stefan Priebe - Profihost AG
Am 20.06.2012 11:47, schrieb Eric Dumazet:
Quoted 6 lines · Stefan Priebe - Profihost AG
Quoted 4 lines
Sure. In that case i get 4Gbit/s in both variants. I also tried two other different machines same result.
Quoted 7 lines
Quoted 1 line
So 3.5 on receiver is the problem, it seems ?
Yes.
Quoted 2 lines
And you checked all the stuff about irq affinities, i presume, since a lot of things might have changed between 2.6.32 and 3.5 ?
It is a single core E5 Xeon - i've set the affinity like this:
And you still have the retransmits in "netstat -s" output ?
Might be a firmware or pci issue, I have same cards but no problem here.
Check LRO is on ?
ethtool -k eth2
Quoted 16 lines · Eric Dumazet
Quoted 7 lines · Stefan Priebe - Profihost AG
Am 20.06.2012 11:47, schrieb Eric Dumazet:
Quoted 6 lines · Stefan Priebe - Profihost AG
Quoted 4 lines
Sure. In that case i get 4Gbit/s in both variants. I also tried two other different machines same result.
Quoted 7 lines
Quoted 1 line
So 3.5 on receiver is the problem, it seems ?
Yes.
Quoted 2 lines
And you checked all the stuff about irq affinities, i presume, since a lot of things might have changed between 2.6.32 and 3.5 ?
It is a single core E5 Xeon - i've set the affinity like this:
Quoted 8 lines
And you still have the retransmits in "netstat -s" output ?
Might be a firmware or pci issue, I have same cards but no problem here.
Check LRO is on ?
ethtool -k eth2
Ah, your ethtool -S gives strange fdir_miss counts, you should ask Intel guys help maybe...