unstable 10GBE performance with recent kernels (> 3.0.X)

Hello list,

i've discovered very unstable 10GBE performance using recent kernels. I'm using some optimized settings mentioned by intel here (part:Improving Performance):
http://downloadmirror.intel.com/5874/eng/README.txt

I'm using Intel X520 cards (ixgbe driver version: 3.9.17-NAPI in all tests).

I'm measuring the performance with iperf.

With 3.0.32 i get constant 9,90 Gbit/s in both directions simultaneously.

With 3.4.2 or 3.5.0-rc2 it get sometimes 9,9gbit/s - sometimes 3gbit/s or even sometimes only 1gbit/s throughput.

I also tried to change the tcp_congestion_control from cubic to reno, bic and highspeed but no change.

I also tried to bisect the issue but there are so many changes in the net kernel part that i'm unable to identify the problem as with some commits i only get 0-300kb/s performance.

Any ideas?

Quoted 7 lines

Hello list,

i've discovered very unstable 10GBE performance using recent kernels. I'm using some optimized settings mentioned by intel here (part:Improving Performance):
http://downloadmirror.intel.com/5874/eng/README.txt

like what settings ?

Quoted 1 line

Any ideas?

ethtool -S eth0

ethtool -k eth0

netstat -s (on sender, on receiver)

single flow , multiple flows ?

Am 18.06.2012 11:45, schrieb Eric Dumazet:

Quoted 8 lines · Stefan Priebe - Profihost AG

Quoted 7 lines

Hello list,

i've discovered very unstable 10GBE performance using recent kernels. I'm using some optimized settings mentioned by intel here (part:Improving Performance):
http://downloadmirror.intel.com/5874/eng/README.txt

Quoted 1 line

like what settings ?

Sorry these ones:
http://pastebin.com/raw.php?i=J2XRFAjD

I would remove all this (pretty old and obsolete) stuff and use standard params.

Only thing you could do is :
ethtool -K eth2 ntuple on

on both machines

Am 18.06.2012 12:25, schrieb Eric Dumazet:

Quoted 2 lines

I would remove all this (pretty old and obsolete) stuff and use standard params.

OK, thanks. done.

I've one RHEL6 system (using default 2.6.32 kernel) where only using "ntuple on" results in just 4-5Gbit/s while the others using 3.5.0-rc2 are working fine.

Quoted 4 lines

Only thing you could do is :
ethtool -K eth2 ntuple on

on both machines

Thanks that works great and boosts the performance a lot. What does it do? man ethtool doesn't show anything useful.

Is this also recommanded for 1Gb/s?

Could you recommend a tcp_congestion_control module? RHEL6 uses cubic by default. Some others use reno or bic.

Quoted 5 lines

Am 18.06.2012 12:25, schrieb Eric Dumazet:

Quoted 2 lines

I would remove all this (pretty old and obsolete) stuff and use standard params.

OK, thanks. done.

Quoted 4 lines

I've one RHEL6 system (using default 2.6.32 kernel) where only using "ntuple on" results in just 4-5Gbit/s while the others using 3.5.0-rc2 are working fine.

OK, then you might play with affinities, that's the only thing that might need sysadmin tuning (irqbalance needs to be disabled)

Am 18.06.2012 14:51, schrieb Eric Dumazet:

Quoted 3 lines · Stefan Priebe - Profihost AG

Quoted 2 lines

Am 18.06.2012 12:25, schrieb Eric Dumazet:

Quoted 1 line

I would remove all this (pretty old and obsolete) stuff and use

standard

Quoted 10 lines

Quoted 7 lines

Quoted 1 line

params.

OK, thanks. done.

I've one RHEL6 system (using default 2.6.32 kernel) where only using "ntuple on" results in just 4-5Gbit/s while the others using 3.5.0-rc2 are working fine.

OK, then you might play with affinities, that's the only thing that might need sysadmin tuning (irqbalance needs to be disabled)

I've now used this one:

# script provided by intel
/root/set_irq_affinity.sh eth2
ifconfig eth2 mtu 9000
ethtool -K eth2 ntuple on

irqbalance is off

But with the latest long term stable vanilla kernel 3.0.34.

# iperf -c HOST -t 60

works fine but with -d for using both directions i see around 9.8Gbit/s in one direction and only 300kb/s in the other direction.

Stefan

Quoted 6 lines

But with the latest long term stable vanilla kernel 3.0.34.

# iperf -c HOST -t 60

works fine but with -d for using both directions i see around 9.8Gbit/s in one direction and only 300kb/s in the other direction.

I would try using regular MTU.

And post "netstat -s" results before/after your iperf.