Skip to contentSkip to conversation

MVEBU and MVNETA driver

DISCUSSION Greg 3 participants 10 messages Apr 19, 2013 – Apr 22, 2013

lore
Share & tools
lore

Greg

author
#0 lore raw

Hello All,

This is my first post there, let me introduce myself: I am an embedded systems designer working in a French ISP. My goal is to design devices with a minimalistic software package installed that system/software engineers can use as a base for their work. Said in a few words : I'm not the kernel expert but I have some knowledge of it.

That being said, I'm currently working with a Marvell ArmadaXP SoC and I'm performing the transition from Marvell "Linux Support Package" to MVEBU in the 3.8.8 Vanilla.
Having the system up and running was actually quite easy, Thomas Petazzoni and his team did a very good job!

I'm facing a problem with ethernet transmission (reception -seems- to be fine), at best is it slow (~90Mbps), at worst the kernel OOPSes at various locations dealing with memory (tcp_v4_destroy_sock, free_block and skb_put seen so far) and ends up crashing right after. I'm simply using iperf to generate traffic and the system has been stable for a while with Marvell's LSP so I can say with confidence the hardware is pretty stable.

My question is fairly simple : before complaining about the network driver (which might not even be involved in this), what is the starting point to gather more information about this problem and what data should I provide you to help you advising me ?

Thank you very much!

#1 lore raw

Hi Greg,

Quoted 22 lines · Greg

Hello All,

This is my first post there, let me introduce myself: I am an embedded systems designer working in a French ISP. My goal is to design devices with a minimalistic software package installed that system/software engineers can use as a base for their work. Said in a few words : I'm not the kernel expert but I have some knowledge of it.

That being said, I'm currently working with a Marvell ArmadaXP SoC and I'm performing the transition from Marvell "Linux Support Package" to MVEBU in the 3.8.8 Vanilla.
Having the system up and running was actually quite easy, Thomas Petazzoni and his team did a very good job!

I'm facing a problem with ethernet transmission (reception -seems- to be fine), at best is it slow (~90Mbps), at worst the kernel OOPSes at various locations dealing with memory (tcp_v4_destroy_sock, free_block and skb_put seen so far) and ends up crashing right after. I'm simply using iperf to generate traffic and the system has been stable for a while with Marvell's LSP so I can say with confidence the hardware is pretty stable.

Quoted 4 lines

My question is fairly simple : before complaining about the network driver (which might not even be involved in this), what is the starting point to gather more information about this problem and what data should I provide you to help you advising me ?

You need to get the following commit from mainline to fix this issue :

ee40a116ebf139f900c3d2e6febb8388738e96d0

Cheers,
Willy

PS: is it indiscrete to ask what board this is ?

author
#2 lore raw

Le 19/04/2013 15:12, Willy Tarreau a ?crit :

Quoted 11 lines

Hi Greg,

Quoted 8 lines · Greg

I'm facing a problem with ethernet transmission (reception -seems- to be fine), at best is it slow (~90Mbps), at worst the kernel OOPSes at various locations dealing with memory (tcp_v4_destroy_sock, free_block and skb_put seen so far) and ends up crashing right after. I'm simply using iperf to generate traffic and the system has been stable for a while with Marvell's LSP so I can say with confidence the hardware is pretty stable.

Quoted 4 lines

You need to get the following commit from mainline to fix this issue :

    ee40a116ebf139f900c3d2e6febb8388738e96d0

Willy, thanks for your fast an accurate answer, this seems to solve the stability issue but not the performance issue.

Please look at this strange behavior : 1 iperf will only transmit at ~100Mbps, 100 parallel iperf will transmit at ~950Mb. Reception is not an issue.
I'm thinking of a socket <-> txQ relationship that would limit transmission rate, but I have no idea on how this is implemented.

Quoted 1 line

PS: is it indiscrete to ask what board this is ?

I designed the board myself, if you already heard about it I'd be glad to know you told you about it ;-)

Cheers,

#3 lore raw
Quoted 2 lines · Greg

Willy, thanks for your fast an accurate answer, this seems to solve the stability issue but not the performance issue.

OK.

Quoted 5 lines

Please look at this strange behavior : 1 iperf will only transmit at ~100Mbps, 100 parallel iperf will transmit at ~950Mb. Reception is not an issue.
I'm thinking of a socket <-> txQ relationship that would limit transmission rate, but I have no idea on how this is implemented.

Strange, I have not experienced this. Yes a socket size limit could be the reason. Have you tried with netcat or any simple TCP server instead ?

Quoted 3 lines
Quoted 1 line

PS: is it indiscrete to ask what board this is ?

I designed the board myself, if you already heard about it I'd be glad to know you told you about it ;-)

Wow, impressed !

Willy

#4 lore raw
Quoted 6 lines · Willy Tarreau
Quoted 2 lines · Greg

Willy, thanks for your fast an accurate answer, this seems to solve the stability issue but not the performance issue.

OK.

Quoted 9 lines
Quoted 5 lines

Please look at this strange behavior : 1 iperf will only transmit at ~100Mbps, 100 parallel iperf will transmit at ~950Mb. Reception is not an issue.
I'm thinking of a socket <-> txQ relationship that would limit transmission rate, but I have no idea on how this is implemented.

Strange, I have not experienced this. Yes a socket size limit could be the reason. Have you tried with netcat or any simple TCP server instead ?

I also have this behavior:
iperf -c 192.168.0.19 -d
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.0.19, TCP port 5001
TCP window size: 20.7 KByte (default)
------------------------------------------------------------
[ 5] local 192.168.0.37 port 48473 connected with 192.168.0.19 port 5001 [ 4] local 192.168.0.37 port 5001 connected with 192.168.0.19 port 36987
[ ID] Interval Transfer Bandwidth

[  5]  0.0-10.0 sec   128 MBytes   107 Mbits/sec
[  4]  0.0-10.0 sec  1.07 GBytes   919 Mbits/sec

but I only need 10 thread in iperf to be close to the Gb
------------------------------------------------------------
Client connecting to 192.168.0.19, TCP port 5001 TCP window size: 20.7 KByte (default)
------------------------------------------------------------

[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec   117 MBytes  98.2 Mbits/sec
[ 12]  0.0-10.0 sec  42.5 MBytes  35.6 Mbits/sec
[  8]  0.0-10.0 sec   131 MBytes   110 Mbits/sec
[  5]  0.0-10.0 sec   127 MBytes   107 Mbits/sec
[  3]  0.0-10.0 sec   118 MBytes  98.4 Mbits/sec
[ 10]  0.0-10.0 sec   130 MBytes   109 Mbits/sec
[  7]  0.0-10.0 sec   128 MBytes   107 Mbits/sec
[  9]  0.0-10.0 sec  83.2 MBytes  69.7 Mbits/sec
[  6]  0.0-10.0 sec   110 MBytes  92.5 Mbits/sec
[ 11]  0.0-10.0 sec   132 MBytes   110 Mbits/sec
[SUM]  0.0-10.0 sec  1.09 GBytes   936 Mbits/sec

I am not a network expert and until now we didn't try to have the best performance the hardware can offer. We mainly focused on features.

Regards,

Signature · 4 lines
Gregory Clement, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
author
#5 lore raw

Le 19/04/2013 16:59, Gregory CLEMENT a ?crit :

Quoted 48 lines · Willy Tarreau
Quoted 5 lines · Greg
Quoted 2 lines

Willy, thanks for your fast an accurate answer, this seems to solve the stability issue but not the performance issue.

OK.

Quoted 8 lines
Quoted 5 lines

Please look at this strange behavior : 1 iperf will only transmit at ~100Mbps, 100 parallel iperf will transmit at ~950Mb. Reception is not an issue.
I'm thinking of a socket <-> txQ relationship that would limit transmission rate, but I have no idea on how this is implemented.

Strange, I have not experienced this. Yes a socket size limit could be the reason. Have you tried with netcat or any simple TCP server instead ?

I also have this behavior:
iperf -c 192.168.0.19 -d
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.0.19, TCP port 5001
TCP window size: 20.7 KByte (default)
------------------------------------------------------------
[ 5] local 192.168.0.37 port 48473 connected with 192.168.0.19 port 5001 [ 4] local 192.168.0.37 port 5001 connected with 192.168.0.19 port 36987
[ ID] Interval Transfer Bandwidth

[  5]  0.0-10.0 sec   128 MBytes   107 Mbits/sec
[  4]  0.0-10.0 sec  1.07 GBytes   919 Mbits/sec

but I only need 10 thread in iperf to be close to the Gb
------------------------------------------------------------
Client connecting to 192.168.0.19, TCP port 5001 TCP window size: 20.7 KByte (default)
------------------------------------------------------------

[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec   117 MBytes  98.2 Mbits/sec
[ 12]  0.0-10.0 sec  42.5 MBytes  35.6 Mbits/sec
[  8]  0.0-10.0 sec   131 MBytes   110 Mbits/sec
[  5]  0.0-10.0 sec   127 MBytes   107 Mbits/sec
[  3]  0.0-10.0 sec   118 MBytes  98.4 Mbits/sec
[ 10]  0.0-10.0 sec   130 MBytes   109 Mbits/sec
[  7]  0.0-10.0 sec   128 MBytes   107 Mbits/sec
[  9]  0.0-10.0 sec  83.2 MBytes  69.7 Mbits/sec
[  6]  0.0-10.0 sec   110 MBytes  92.5 Mbits/sec
[ 11]  0.0-10.0 sec   132 MBytes   110 Mbits/sec
[SUM]  0.0-10.0 sec  1.09 GBytes   936 Mbits/sec
Quoted 4 lines

I am not a network expert and until now we didn't try to have the best performance the hardware can offer. We mainly focused on features.

I've got the same numbers here.
A simple tool like netcat shows ~470Mbps but this limited by the CPU. Iperf isn't CPU limited

The same test with Marvell LSP shows ~940Mbps in both directions.

Regards,

#6 lore raw
Quoted 4 lines · Greg

A simple tool like netcat shows ~470Mbps but this limited by the CPU. Iperf isn't CPU limited

The same test with Marvell LSP shows ~940Mbps in both directions.

I've retested here on my mirabox using httpterm as the tcp server. Using the normal send() syscall, it emits 839 Mbps on a single TCP connection. Using splice(), it emits 969 Mbps in the same conditions.

Here's what ethtool is saying about the NIC's tuning :

# ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: off
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: off
tx-vlan-offload: off
ntuple-filters: off
receive-hashing: off

I've tried changing other settings such as the interrupt delays (rx-usecs, rx-frames, tx-frames) but this has no effect. So in short, I cannot reproduce this behaviour. Are you using the SLOB allocator ? I'm asking because I remember experiencing very poor performance with it in the past.

Regards,
Willy

author
#7 lore raw

Le 22/04/2013 08:20, Willy Tarreau a ?crit :

Quoted 25 lines · Greg
Quoted 4 lines

A simple tool like netcat shows ~470Mbps but this limited by the CPU. Iperf isn't CPU limited

The same test with Marvell LSP shows ~940Mbps in both directions.

I've retested here on my mirabox using httpterm as the tcp server. Using the normal send() syscall, it emits 839 Mbps on a single TCP connection. Using splice(), it emits 969 Mbps in the same conditions.

Here's what ethtool is saying about the NIC's tuning :

# ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: off
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: off
tx-vlan-offload: off
ntuple-filters: off
receive-hashing: off
Quoted 6 lines

I've tried changing other settings such as the interrupt delays (rx-usecs, rx-frames, tx-frames) but this has no effect. So in short, I cannot reproduce this behaviour. Are you using the SLOB allocator ? I'm asking because I remember experiencing very poor performance with it in the past.

Willy,

I'm using SLAB allocator.
As for offloading, I can't seem to be able to turn it on :

Quoted 5 lines
$ ethtool -K eth0 gro on
Cannot set device GRO settings: Operation not supported
$ ethtool -K eth0 gso on
Cannot set device generic segmentation offload settings: Operation not 
supported

Default values :

Quoted 6 lines
$ ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: off
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: off
Quoted 8 lines
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
rx-vlan-offload: off
tx-vlan-offload: off
ntuple-filters: off
receive-hashing: off

I can see 1 difference between our setups, Mirabox has an Armada370, I have an ArmadaXP, it would be interesting to know what Gregory is using.

Another difference on my setup, and something I wanted to report you :
there is no PHY on my system. I had to instantiate a dummy PHY in the device tree blob and bind it to the mvneta driver because it would fail at init without PHY.

Regards,

#8 lore raw
Quoted 23 lines · Greg

I'm using SLAB allocator.
As for offloading, I can't seem to be able to turn it on :

Quoted 5 lines
$ ethtool -K eth0 gro on
Cannot set device GRO settings: Operation not supported
$ ethtool -K eth0 gso on
Cannot set device generic segmentation offload settings: Operation not 
supported

Default values :

Quoted 6 lines
$ ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: off
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: off
Quoted 8 lines
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
rx-vlan-offload: off
tx-vlan-offload: off
ntuple-filters: off
receive-hashing: off
Quoted 2 lines

I can see 1 difference between our setups, Mirabox has an Armada370, I have an ArmadaXP, it would be interesting to know what Gregory is using.

OK so I guess you have a kernel version prior to 3.9-rc7. There is a bug in the driver which causes it to report tx cksum and sg but they're not enabled. You need to disable then enable them :

ethtool -K eth0 tx off
ethtool -K eth0 sg off
ethtool -K eth0 tx on
ethtool -K eth0 sg on

Then you can enable gso/gro. And that clearly explains a significant performance difference.

Quoted 4 lines

Another difference on my setup, and something I wanted to report you :
there is no PHY on my system. I had to instantiate a dummy PHY in the device tree blob and bind it to the mvneta driver because it would fail at init without PHY.

I think this is more something for Thomas then.

Regards,
Willy

author
#9 lore raw

Le 22/04/2013 11:08, Willy Tarreau a ?crit :

Quoted 29 lines · Greg
Quoted 22 lines

I'm using SLAB allocator.
As for offloading, I can't seem to be able to turn it on :

Quoted 5 lines
$ ethtool -K eth0 gro on
Cannot set device GRO settings: Operation not supported
$ ethtool -K eth0 gso on
Cannot set device generic segmentation offload settings: Operation not
supported

Default values :

Quoted 6 lines
$ ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: off
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: off
Quoted 8 lines
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
rx-vlan-offload: off
tx-vlan-offload: off
ntuple-filters: off
receive-hashing: off
Quoted 2 lines

I can see 1 difference between our setups, Mirabox has an Armada370, I have an ArmadaXP, it would be interesting to know what Gregory is using.

OK so I guess you have a kernel version prior to 3.9-rc7. There is a bug in the driver which causes it to report tx cksum and sg but they're not enabled. You need to disable then enable them :

Quoted 7 lines

ethtool -K eth0 tx off
ethtool -K eth0 sg off
ethtool -K eth0 tx on
ethtool -K eth0 sg on

Then you can enable gso/gro. And that clearly explains a significant performance difference.

Yes, as explain in my original mail, I'm using 3.8.8
I followed your recommendations and performance is now ~930-940Mbits in both directions which is fine.
Thanks !

Quoted 6 lines
Quoted 4 lines

Another difference on my setup, and something I wanted to report you :
there is no PHY on my system. I had to instantiate a dummy PHY in the device tree blob and bind it to the mvneta driver because it would fail at init without PHY.

I think this is more something for Thomas then.

The "failure" is located in mvneta_mdio_probe but I guess this is more like an architectural problem.
The MVNETA driver assumes a PHY is present, but this is not a requirement to establish link between 2 MACs.
I guess using a dummy PHY is a not-so-bad idea for my situation at this time.

Regards,