Link stops working under heavy traffic load connected to a mv88e6176
Subsystems (2)
We are working in an ARMv7 embedded system running kernel 4.9 (LEDE build). It is an imx6 board with 2 ethernet interfaces. One of them is connected to a Marvell switch.
The schema of the system is the following:
+-------------------+ eth0
| +--+
| | |
| Embedded system +--+
| |
| ARMv7 |
| | Marvell 88E8057(sky2) +-------------+
| +--+ +--+ +--+ eth1
| | +---------------------+ | | +------+
| +--+ CPU port +--+ mv88e6176 +--+
+------+--+---------+ | |
emulated| | | |
GPIO +--+ +--+ +--+ eth2
MDIO +-----------------------------------+ | | +------+
MDIO +--+ +--+
+-------------+There is a bridge (br-lan) which includes eth0/eth1/eth2
If I connect the eth1/eth2, the link is up and I can do ping through it. But, once I start sending a heavy traffic load the link fails and the kernel sends the following messages:
[ 48.557140] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.564964] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.572110] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.579263] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.586417] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.593573] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.600718] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 54.877567] net_ratelimit: 6 callbacks suppressed [ 54.882293] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 61.413552] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518
I have a modified device-tree of imx6 which includes the mdio and dsa nodes. This device-tree works in a kernel 4.1.6, but I know that these parts of the kernel have a lot of changes. The changes included for mdio and dsa in the device-tree are the following (diff arch/arm/boot/dts/imx6qdl-gw53xx.dtsi arch/arm/boot/dts/imx6qdl-gw53xx-mdio.dtsi):
16a17,18
Quoted 2 lines
can0 = &can1;
ethernet0 = &fec;21a24
Quoted 1 line
sky2 = ð1;
24a28,29
Quoted 2 lines
usdhc2 = &usdhc3;
mdio-gpio0 = &mdio0;62a68,125
Quoted 51 lines
mdio0: mdio { compatible = "virtual,mdio-gpio"; #address-cells = <1>; #size-cells = <0>; /* MDC = gpio-17, MDIO = gpio-20 */ gpios = <&gpio1 17 1 &gpio1 20 0>; ethernet-phy@0 { compatible = "marvell,dsa"; }; }; dsa { compatible = "marvell,dsa"; #address-cells = <2>; #size-cells = <0>;interrupts = <10>; dsa,ethernet = <ð1>; dsa,mii-bus = <&mdio0>; switch@0 { #address-cells = <1>; #size-cells = <0>; reg = <0 0>; /* MDIO address 0, switch 0 in tree */ port@0 { reg = <0>; label = "cpu"; }; port@1 { reg = <1>; label = "eth1"; }; port@2 { reg = <2>; label = "eth2"; }; port@3 { reg = <3>; label = "eth3"; }; port@4 { reg = <4>; label = "eth4"; }; };
Quoted 2 lines
};
361a425,430
Quoted 6 lines
&mdio0 {
pinctrl-names = "default";
pinctrl-0 = <&pinctrl_mdio>;
status = "okay";
};
363c432
< imx6qdl-gw53xx {
---
Quoted 1 line
imx6qdl-gw53xx-mdio {448a518,524
Quoted 7 lines
>;
};
pinctrl_mdio: mdiogrp {
fsl,pins = <
MX6QDL_PAD_SD1_DAT1__GPIO1_IO17 0x1b0b9
MX6QDL_PAD_SD1_CLK__GPIO1_IO20 0x1b0b9Do you know of any possible reason why this could be happening?
Thanks in advance.
Rafa
I resend the mail with the schema fixed. Sorry for the inconvenience.
We are working in an ARMv7 embedded system running kernel 4.9 (LEDE build). It is an imx6 board with 2 ethernet interfaces. One of them is connected to a Marvell switch.
The schema of the system is the following:
+-------------------+ eth0
| +--+
| | |
| Embedded system +--+
| |
| ARMv7 |
| | Marvell 88E8057(sky2) +-------------+
| +--+ +--+ +--+ eth1
| | +---------------------+ | | +------+
| +--+ CPU port +--+ mv88e6176 +--+
+------+--+---------+ | |
emulated| | | |
GPIO +--+ +--+ +--+ eth2
MDIO +-----------------------------------+ | | +------+
MDIO +--+ +--+
+-------------+There is a bridge (br-lan) which includes eth0/eth1/eth2
If I connect the eth1/eth2, the link is up and I can do ping through it. But, once I start sending a heavy traffic load the link fails and the kernel sends the following messages:
[ 48.557140] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.564964] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.572110] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.579263] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.586417] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.593573] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.600718] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 54.877567] net_ratelimit: 6 callbacks suppressed [ 54.882293] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 61.413552] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518
I have a modified device-tree of imx6 which includes the mdio and dsa nodes. This device-tree works in a kernel 4.1.6, but I know that these parts of the kernel have a lot of changes. The changes included for mdio and dsa in the device-tree are the following (diff arch/arm/boot/dts/imx6qdl-gw53xx.dtsi arch/arm/boot/dts/imx6qdl-gw53xx-mdio.dtsi):
16a17,18
Quoted 2 lines
can0 = &can1;
ethernet0 = &fec;21a24
Quoted 1 line
sky2 = ð1;
24a28,29
Quoted 2 lines
usdhc2 = &usdhc3;
mdio-gpio0 = &mdio0;62a68,125
Quoted 51 lines
mdio0: mdio { compatible = "virtual,mdio-gpio"; #address-cells = <1>; #size-cells = <0>; /* MDC = gpio-17, MDIO = gpio-20 */ gpios = <&gpio1 17 1 &gpio1 20 0>; ethernet-phy@0 { compatible = "marvell,dsa"; }; }; dsa { compatible = "marvell,dsa"; #address-cells = <2>; #size-cells = <0>;interrupts = <10>; dsa,ethernet = <ð1>; dsa,mii-bus = <&mdio0>; switch@0 { #address-cells = <1>; #size-cells = <0>; reg = <0 0>; /* MDIO address 0, switch 0 in tree */ port@0 { reg = <0>; label = "cpu"; }; port@1 { reg = <1>; label = "eth1"; }; port@2 { reg = <2>; label = "eth2"; }; port@3 { reg = <3>; label = "eth3"; }; port@4 { reg = <4>; label = "eth4"; }; };
Quoted 2 lines
};
361a425,430
Quoted 6 lines
&mdio0 {
pinctrl-names = "default";
pinctrl-0 = <&pinctrl_mdio>;
status = "okay";
};
363c432
< imx6qdl-gw53xx {
---
Quoted 1 line
imx6qdl-gw53xx-mdio {448a518,524
Quoted 7 lines
>;
};
pinctrl_mdio: mdiogrp {
fsl,pins = <
MX6QDL_PAD_SD1_DAT1__GPIO1_IO17 0x1b0b9
MX6QDL_PAD_SD1_CLK__GPIO1_IO20 0x1b0b9Do you know of any possible reason why this could be happening?
Thanks in advance.
Rafa
Quoted 48 lines · On Fri, 21 Apr 2017 14:39:00 +0200 Rafa Corvillo <[email protected]>…
We are working in an ARMv7 embedded system running kernel 4.9 (LEDE build). It is an imx6 board with 2 ethernet interfaces. One of them is connected to a Marvell switch.
The schema of the system is the following:
+-------------------+ eth0 | +--+ | | | | Embedded system +--+ | | | ARMv7 | | | Marvell 88E8057(sky2) +-------------+ | +--+ +--+ +--+ eth1 | | +---------------------+ | | +------+ | +--+ CPU port +--+ mv88e6176 +--+ +------+--+---------+ | | emulated| | | | GPIO +--+ +--+ +--+ eth2 MDIO +-----------------------------------+ | | +------+ MDIO +--+ +--+ +-------------+There is a bridge (br-lan) which includes eth0/eth1/eth2
If I connect the eth1/eth2, the link is up and I can do ping through it. But, once
I start sending a heavy traffic load the link fails and the kernel sends the following messages:[ 48.557140] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.564964] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.572110] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.579263] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.586417] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.593573] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.600718] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 54.877567] net_ratelimit: 6 callbacks suppressed [ 54.882293] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 61.413552] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010
Quoted 1 line
length 1518
The status error bits are in sky2.h0x5f20010 is
05f2 frame length => 1522
0010 Too long err
That means the packet was longer than the configured MTU. You are probably getting packets with VLAN tag but have not configured a VLAN.
Quoted 53 lines · Stephen Hemminger
Quoted 48 lines · On Fri, 21 Apr 2017 14:39:00 +0200 Rafa Corvillo <[email protected]>…
We are working in an ARMv7 embedded system running kernel 4.9 (LEDE build). It is an imx6 board with 2 ethernet interfaces. One of them is connected to a Marvell switch.
The schema of the system is the following:
+-------------------+ eth0 | +--+ | | | | Embedded system +--+ | | | ARMv7 | | | Marvell 88E8057(sky2) +-------------+ | +--+ +--+ +--+ eth1 | | +---------------------+ | | +------+ | +--+ CPU port +--+ mv88e6176 +--+ +------+--+---------+ | | emulated| | | | GPIO +--+ +--+ +--+ eth2 MDIO +-----------------------------------+ | | +------+ MDIO +--+ +--+ +-------------+There is a bridge (br-lan) which includes eth0/eth1/eth2
If I connect the eth1/eth2, the link is up and I can do ping through it. But, once
I start sending a heavy traffic load the link fails and the kernel sends the following messages:[ 48.557140] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.564964] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.572110] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.579263] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.586417] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.593573] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.600718] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 54.877567] net_ratelimit: 6 callbacks suppressed [ 54.882293] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 61.413552] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010Quoted 1 line
length 1518
Quoted 11 lines
The status error bits are in
sky2.h0x5f20010is
05f2 frame length => 1522
0010 Too long errThat means the packet was longer than the configured MTU. You are probably getting packets with VLAN tag but have not configured a VLAN.
Thanks for the information. I have increased the MTU value to 1550 (workaround) and it works if sends traffic (with iperf) from my computer to the unit. But, if I send traffic outside the unit, I get a new error message and link goes down:
[ 4901.032989] sky2 0000:04:00.0 marvell: tx timeout [ 4904.722670] sky2 0000:04:00.0 marvell: Link is up at 1000 Mbps, full duplex, flow control both
Rafa
Quoted 10 lines · Rafa Corvillo
Quoted 9 lines · Stephen Hemminger
Quoted 6 lines · On Fri, 21 Apr 2017 14:39:00 +0200 Rafa Corvillo <[email protected]>…
We are working in an ARMv7 embedded system running kernel 4.9 (LEDE build). It is an imx6 board with 2 ethernet interfaces. One of them is connected to a Marvell switch.
The schema of the system is the following:
Hi Rafa
Your ASCII art got messed up somewhere. Is this the correct reconstruction?
+-------------------+ eth0
| +--+
| | |
| Embedded system +--+
| |
| ARMv7 |
| | Marvell 88E8057(sky2) +-------------+
| +--+ +--+ +--+ eth1
| | +---------------------+ | | +------+
| +--+ CPU port +--+ mv88e6176 +--+
+------+--+---------+ | |
emulated | | | |
GPIO +--+ +--+ +--+ eth2
MDIO +-----------------------------------+ | | +------+
MDIO +--+ +--+
+-------------+I assume you are using DSA? Since this is LEDE, it could be swconfig, but the bridge configuration you mentioned would not make sense for swconfig.
Quoted 25 lines
Quoted 25 lines
Quoted 24 lines
If I connect the eth1/eth2, the link is up and I can do ping through it. But, once
I start sending a heavy traffic load the link fails and the kernel sends the following messages:[ 48.557140] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.564964] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.572110] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.579263] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.586417] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.593573] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.600718] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 54.877567] net_ratelimit: 6 callbacks suppressed [ 54.882293] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 61.413552] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010length 1518
Quoted 8 lines
Quoted 8 lines
The status error bits are in
sky2.h0x5f20010is
05f2 frame length => 1522
0010 Too long errThat means the packet was longer than the configured MTU. You are probably getting packets with VLAN tag but have not configured a VLAN.
Since you are using DSA, you will have DSA tags enabled on frames to/from the switch. This adds an extra 8 byte header in the frame. My guess is, it is this header, not the VLAN tag which is causing you MTU issues.
I think this is the first time i've seen sky2 used in a DSA setup. mv643xx or mvneta is generally what is used, when using Marvell chipsets. These drivers are more lenient about MTU, and are happy to pass frames with additional headers.
Quoted 4 lines
Thanks for the information. I have increased the MTU value to 1550 (workaround) and it works if sends traffic (with iperf) from my computer to the unit. But, if I send traffic outside the unit, I get a new error message and link goes down:
Changing the MTU like this is not a good fix. It will allow you to receive frames which are bigger, but it also means the local network stack will generate bigger frames to be transmitted. You probably need to modify the sky2 driver to allow it to receive frames bigger than the interface MTU, by about 8 bytes.
Quoted 3 lines
[ 4901.032989] sky2 0000:04:00.0 marvell: tx timeout [ 4904.722670] sky2 0000:04:00.0 marvell: Link is up at 1000 Mbps,full duplex, flow control both
Between the sky2 and the switch, do you have two back-to-back PHYs or are you connecting the RGMII interfaces together?
Andrew
Quoted 9 lines · Andrew Lunn
Quoted 8 lines · Rafa Corvillo
Quoted 7 lines · Stephen Hemminger
Quoted 4 lines · On Fri, 21 Apr 2017 14:39:00 +0200 Rafa Corvillo <[email protected]>…
We are working in an ARMv7 embedded system running kernel 4.9 (LEDE build). It is an imx6 board with 2 ethernet interfaces. One of them is connected to a Marvell switch.
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The schema of the system is the following:
Hi Rafa
Your ASCII art got messed up somewhere. Is this the correct reconstruction?
Yes, this is the schema.
Quoted 18 lines
+-------------------+ eth0
| +--+
| | |
| Embedded system +--+
| |
| ARMv7 |
| | Marvell 88E8057(sky2) +-------------+
| +--+ +--+ +--+ eth1
| | +---------------------+ | | +------+
| +--+ CPU port +--+ mv88e6176 +--+
+------+--+---------+ | |
emulated | | | |
GPIO +--+ +--+ +--+ eth2
MDIO +-----------------------------------+ | | +------+
MDIO +--+ +--+
+-------------+Quoted 3 lines
I assume you are using DSA? Since this is LEDE, it could be swconfig, but the bridge configuration you mentioned would not make sense for swconfig.
Yes, we use DSA driver. We don't use swconfig to configure the Marvell switch. Our board has two ethernet interfaces (eth0 and marvell) using sky2 driver. The marvell interface is connected to an external Marvell switch (mv88e6176) with four ethernet ports (but we only use two of them, eth1 and eth2). The Marvell switch is configured with the MDIO protocol, that we emulate through GPIOS (mdio-gpio kernel module), and the DSA driver is used to works with the Marvell switch.
We have the ethernet interfaces in the same bridge:
config interface 'lan'
option type 'bridge'
option ifname 'eth0 eth1 eth2'
option proto 'static'
option ipaddr '192.168.1.100'
option netmask '255.255.255.0'
option ip6assign '60'
root@LEDE:/# brctl show
bridge name bridge id STP enabled interfaces
br-lan 7fff.00d01274f069 no eth0
eth1
eth2
root@LEDE:/# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel master br-lan state DOWN group default qlen 1000
link/ether 00:d0:12:74:f0:69 brd ff:ff:ff:ff:ff:ff
3: ifb0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 32
link/ether be:80:bc:5e:63:c3 brd ff:ff:ff:ff:ff:ff
4: ifb1: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 32
link/ether 0a:1d:8d:06:e3:5d brd ff:ff:ff:ff:ff:ff
5: gre0@NONE: <NOARP> mtu 1476 qdisc noop state DOWN group default qlen 1
link/gre 0.0.0.0 brd 0.0.0.0
6: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN group default qlen 1000
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
7: bond0: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether e2:0b:10:b8:b7:b0 brd ff:ff:ff:ff:ff:ff
8: teql0: <NOARP> mtu 1500 qdisc noop state DOWN group default qlen 100
link/void
9: can0: <NOARP,ECHO> mtu 16 qdisc noop state DOWN group default qlen 10
link/can
10: marvell: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether aa:64:73:91:09:a9 brd ff:ff:ff:ff:ff:ff
inet6 fe80::a864:73ff:fe91:9a9/64 scope link
valid_lft forever preferred_lft forever
11: eth1@marvell: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-lan switchid 00000000 state LOWERLAYERDOWN group default qlen 1000
link/ether aa:64:73:91:09:a9 brd ff:ff:ff:ff:ff:ff
12: eth2@marvell: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan switchid 00000000 state UP group default qlen 1000
link/ether aa:64:73:91:09:a9 brd ff:ff:ff:ff:ff:ff
13: eth3@marvell: <BROADCAST,MULTICAST> mtu 1500 qdisc noop switchid 00000000 state DOWN group default qlen 1000
link/ether aa:64:73:91:09:a9 brd ff:ff:ff:ff:ff:ff
14: eth4@marvell: <BROADCAST,MULTICAST> mtu 1500 qdisc noop switchid 00000000 state DOWN group default qlen 1000
link/ether aa:64:73:91:09:a9 brd ff:ff:ff:ff:ff:ff
15: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:d0:12:74:f0:69 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.100/24 brd 192.168.1.255 scope global br-lan
valid_lft forever preferred_lft forever
inet6 fd7b:a43b:e93e::1/60 scope global noprefixroute
valid_lft forever preferred_lft forever
inet6 fe80::2d0:12ff:fe74:f069/64 scope link
valid_lft forever preferred_lft foreverWe have this configuration working on a kernel 4.1 and including patches to upgrade dsa/mv88e6xxx to kernel version 4.3 (5acf4d0, Wed, 27 May 2015 15:32:15 -0700) "[PATCH] blk: rq_data_dir() should not return a boolean."
Quoted 31 lines
Quoted 31 lines
Quoted 31 lines
Quoted 24 lines
If I connect the eth1/eth2, the link is up and I can do ping through it. But, once
I start sending a heavy traffic load the link fails and the kernel sends the following messages:[ 48.557140] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.564964] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.572110] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.579263] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.586417] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.593573] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 48.600718] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 54.877567] net_ratelimit: 6 callbacks suppressed [ 54.882293] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010 length 1518 [ 61.413552] sky2 0000:04:00.0 marvell: rx error, status 0x5f20010Quoted 1 line
length 1518
The status error bits are in
sky2.h0x5f20010is
05f2 frame length => 1522
0010 Too long err
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
That means the packet was longer than the configured MTU. You are probably getting packets with VLAN tag but have not configured a VLAN.
Since you are using DSA, you will have DSA tags enabled on frames to/from the switch. This adds an extra 8 byte header in the frame. My guess is, it is this header, not the VLAN tag which is causing you MTU issues.
But it is strange because, as I have said above, we have the same configuration working properly on a kernel 4.1 (with OpenWrt), and we have the MTU set to 1500.
Quoted 6 lines
I think this is the first time i've seen sky2 used in a DSA setup. mv643xx or mvneta is generally what is used, when using Marvell chipsets. These drivers are more lenient about MTU, and are happy to pass frames with additional headers.
We use the mv88e6xxx (as our switch is mv88e6176) and it depends on DSA driver in the kernel (isn't it?).
Quoted 10 lines
Quoted 4 lines
Thanks for the information. I have increased the MTU value to 1550 (workaround) and it works if sends traffic (with iperf) from my computer to the unit. But, if I send traffic outside the unit, I get a new error message and link goes down:
Changing the MTU like this is not a good fix. It will allow you to receive frames which are bigger, but it also means the local network stack will generate bigger frames to be transmitted. You probably need to modify the sky2 driver to allow it to receive frames bigger than the interface MTU, by about 8 bytes.
Should the DSA driver remove the DSA tags before pass the frames to sky2 interface?
Quoted 7 lines
Quoted 4 lines
[ 4901.032989] sky2 0000:04:00.0 marvell: tx timeout [ 4904.722670] sky2 0000:04:00.0 marvell: Link is up at 1000 Mbps,full duplex, flow control both
Between the sky2 and the switch, do you have two back-to-back PHYs or are you connecting the RGMII interfaces together?
I think that we have two back-to-back PHYs, but I am going to double check this with the hardware team.
Thanks,
Rafa
Quoted 3 lines
Andrew
Quoted 8 lines
Quoted 4 lines
Since you are using DSA, you will have DSA tags enabled on frames to/from the switch. This adds an extra 8 byte header in the frame. My guess is, it is this header, not the VLAN tag which is causing you MTU issues.
But it is strange because, as I have said above, we have the same configuration working properly on a kernel 4.1 (with OpenWrt), and we have the MTU set to 1500.
If you look at sky2.c:
static unsigned sky2_get_rx_threshold(struct sky2_port *sky2)
{
unsigned size;
/* Space needed for frame data + headers rounded up */
size = roundup(sky2->netdev->mtu + ETH_HLEN + VLAN_HLEN, 8);
/* Stopping point for hardware truncation */
return (size - 8) / sizeof(u32);
}This is not going to be big enough for a frame with a DSA header.
Quoted 8 lines
Quoted 5 lines
I think this is the first time i've seen sky2 used in a DSA setup. mv643xx or mvneta is generally what is used, when using Marvell chipsets. These drivers are more lenient about MTU, and are happy to pass frames with additional headers.
We use the mv88e6xxx (as our switch is mv88e6176) and it depends on DSA driver in the kernel (isn't it?).
That is correct. But i was talking about the Ethernet interface. All the designs i've seen use an mv643xxx Ethernet interface, or an mvneta interface. This is the first time i've seen a sky2 used, which is why i'm not too surprised you have issues.
Quoted 8 lines
Quoted 5 lines
Changing the MTU like this is not a good fix. It will allow you to receive frames which are bigger, but it also means the local network stack will generate bigger frames to be transmitted. You probably need to modify the sky2 driver to allow it to receive frames bigger than the interface MTU, by about 8 bytes.
Should the DSA driver remove the DSA tags before pass the frames to sky2 interface?
The DSA driver is adding the DSA tags to the frame and passing these tagged frames to the sky2 interface. Frames going to/from the switch will always have such tags.
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
[ 4901.032989] sky2 0000:04:00.0 marvell: tx timeout [ 4904.722670] sky2 0000:04:00.0 marvell: Link is up at 1000 Mbps,full duplex, flow control both
Between the sky2 and the switch, do you have two back-to-back PHYs or are you connecting the RGMII interfaces together?
I think that we have two back-to-back PHYs, but I am going to double check this with the hardware team.
This could be your problem them. The mv88e6xxx switch driver assumes there is a straight rgmii-rgmii connection, no PHYs. So it hard configures the 'CPU' port to its fastest speed, with the link forced up. If you actually have a PHY there, this might not work so well. I don't know if the switch PHY is going to do autoneg correctly. Try using ethtool to look at the sky2 PHY and see what state it is in.
Andrew
Quoted 8 lines · Andrew Lunn
Quoted 8 lines
Quoted 4 lines
Since you are using DSA, you will have DSA tags enabled on frames to/from the switch. This adds an extra 8 byte header in the frame. My guess is, it is this header, not the VLAN tag which is causing you MTU issues.
But it is strange because, as I have said above, we have the same configuration working properly on a kernel 4.1 (with OpenWrt), and we have the MTU set to 1500.
Hi Andrew,
Sorry for the delay in my answer, I was out of the office.
Quoted 14 lines
If you look at
sky2.c:static unsigned
sky2_get_rx_threshold(struct sky2_port *sky2){ unsigned size; /* Space needed for frame data + headers rounded up */ size = roundup(sky2->netdev->mtu + ETH_HLEN + VLAN_HLEN, 8); /* Stopping point for hardware truncation */ return (size - 8) / sizeof(u32); }
Quoted 2 lines
This is not going to be big enough for a frame with a DSA header.
Then, would be a good fix add 8 bytes to the size variable in this function?
Quoted 37 lines
Quoted 8 lines
Quoted 5 lines
I think this is the first time i've seen sky2 used in a DSA setup. mv643xx or mvneta is generally what is used, when using Marvell chipsets. These drivers are more lenient about MTU, and are happy to pass frames with additional headers.
We use the mv88e6xxx (as our switch is mv88e6176) and it depends on DSA driver in the kernel (isn't it?).
That is correct. But i was talking about the Ethernet interface. All the designs i've seen use an mv643xxx Ethernet interface, or an mvneta interface. This is the first time i've seen a sky2 used, which is why i'm not too surprised you have issues.
Quoted 8 lines
Quoted 5 lines
Changing the MTU like this is not a good fix. It will allow you to receive frames which are bigger, but it also means the local network stack will generate bigger frames to be transmitted. You probably need to modify the sky2 driver to allow it to receive frames bigger than the interface MTU, by about 8 bytes.
Should the DSA driver remove the DSA tags before pass the frames to sky2 interface?
The DSA driver is adding the DSA tags to the frame and passing these tagged frames to the sky2 interface. Frames going to/from the switch will always have such tags.
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
[ 4901.032989] sky2 0000:04:00.0 marvell: tx timeout [ 4904.722670] sky2 0000:04:00.0 marvell: Link is up at 1000 Mbps,full duplex, flow control both
Between the sky2 and the switch, do you have two back-to-back PHYs or are you connecting the RGMII interfaces together?
I think that we have two back-to-back PHYs, but I am going to double check this with the hardware team.
Quoted 9 lines
This could be your problem them. The mv88e6xxx switch driver assumes there is a straight rgmii-rgmii connection, no PHYs. So it hard configures the 'CPU' port to its fastest speed, with the link forced up. If you actually have a PHY there, this might not work so well. I don't know if the switch PHY is going to do autoneg correctly. Try using ethtool to look at the sky2 PHY and see what state it is in.
Andrew
The output of ethtool of sky2 interface is the following:
Settings for marvell:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: No
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
MDI-X: Unknown
Supports Wake-on: pg
Wake-on: d
Current message level: 0x000000ff (255)
drv probe link timer ifdown ifup rx_errtx_err
Link detected: yes
And the output of ethtool of eth2@marvell (interface that I have connected):
Settings for eth2:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Link partner advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Link partner advertised pause frame use: No
Link partner advertised auto-negotiation: No
Speed: 100Mb/s
Duplex: Full
Port: MII
PHYAD: 2
Transceiver: external
Auto-negotiation: on
Supports Wake-on: d
Wake-on: d
Link detected: yes
Do you see something strange in these outputs?
Thanks,
Rafa
Quoted 7 lines
Quoted 7 lines
static unsigned
sky2_get_rx_threshold(struct sky2_port *sky2){ unsigned size; /* Space needed for frame data + headers rounded up */ size = roundup(sky2->netdev->mtu + ETH_HLEN + VLAN_HLEN, 8);
Quoted 8 lines
Quoted 6 lines
/* Stopping point for hardware truncation */ return (size - 8) / sizeof(u32); }This is not going to be big enough for a frame with a DSA header.
Then, would be a good fix add 8 bytes to the size variable in this function?
Yes. Also look at the transmit code, is there again a limit based on the MTU.
Quoted 17 lines
Settings for marvell:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: No
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Quoted 9 lines
Auto-negotiation: on
MDI-X: Unknown
Supports Wake-on: pg
Wake-on: d
Current message level:0x000000ff(255)
drv probe link timer ifdown ifuprx_errtx_err
Link detected: yes
So this suggests there is a real PHY there, and it is auto-negotiating.
What we cannot see is the status for the PHY it connects to. But since this PHY has established a link, the other PHY is probably O.K. It is just a bit unsafe, since you are relying on reset behaviour. There is nothing in software configuring the second PHY to make it auto-negotiate.
Andrew
Quoted 11 lines · Andrew Lunn
Quoted 11 lines
Quoted 11 lines
static unsigned
sky2_get_rx_threshold(struct sky2_port *sky2){ unsigned size; /* Space needed for frame data + headers rounded up */ size = roundup(sky2->netdev->mtu + ETH_HLEN + VLAN_HLEN, 8); /* Stopping point for hardware truncation */ return (size - 8) / sizeof(u32); }
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
This is not going to be big enough for a frame with a DSA header.
Then, would be a good fix add 8 bytes to the size variable in this function?
Yes. Also look at the transmit code, is there again a limit based on the MTU.
Hi Andrew,
Adding 8 bytes (sky2->netdev->mtu + ETH_HLEN + VLAN_HLEN + 8 (EDSA_HLEN)) does not fix the error, because the interface keep having a maximum length of 1518 bytes (sky2->netdev->mtu + ETH_HLEN + VLAN_HLEN).
The polling function of sky2 driver (sky2_poll) calls to the function sky2_status_intr (with the parameter struct sky2_hw *hw). The sky2_status_intr function gets the status of the list elements (struct sky2_status_le) from the sky2_hw parameter and, from the sky2_status_le, gets the maximum length (1518) and the status code (0x5f20010). When the latter function (sky2_status_intr) calls to the sky2_receive function with the parameters length and status, it reports about the error (rx error, status 0x5f20010 length 1518).
I don't know who sets the maximum length (1518) and the status code (0x5f20010) of the packets. Is it possible that these values to be set outside the sky2 code?
Thanks,
Rafa
Quoted 40 lines
Quoted 18 lines
Settings for marvell:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: No
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internalQuoted 9 lines
Auto-negotiation: on
MDI-X: Unknown
Supports Wake-on: pg
Wake-on: d
Current message level:0x000000ff(255)
drv probe link timer ifdown ifuprx_errtx_err
Link detected: yesSo this suggests there is a real PHY there, and it is auto-negotiating.
What we cannot see is the status for the PHY it connects to. But since this PHY has established a link, the other PHY is probably O.K. It is just a bit unsafe, since you are relying on reset behaviour. There is nothing in software configuring the second PHY to make it auto-negotiate.
Andrew
Quoted 4 lines
Adding 8 bytes (
sky2->netdev->mtu+ETH_HLEN+VLAN_HLEN+ 8 (EDSA_HLEN)) does not fix the error, because the interface keep having a maximum length of 1518 bytes (sky2->netdev->mtu+ETH_HLEN
+VLAN_HLEN).
Did you check the value being written to here:
/*
* The receiver hangs if it receives frames larger than the
* packet buffer. As a workaround, truncate oversize frames, but
* the register is limited to 9 bits, so if you do frames > 2052
* you better get the MTU right!
*/
thresh = sky2_get_rx_threshold(sky2);
if (thresh > 0x1ff)
sky2_write32(hw, SK_REG(sky2->port, RX_GMF_CTRL_T), RX_TRUNC_OFF);
else {
sky2_write16(hw, SK_REG(sky2->port, RX_GMF_TR_THR), thresh);
sky2_write32(hw, SK_REG(sky2->port, RX_GMF_CTRL_T), RX_TRUNC_ON);
}What is thresh?
Andrew
Quoted 22 lines
Quoted 4 lines
Adding 8 bytes (
sky2->netdev->mtu+ETH_HLEN+VLAN_HLEN+ 8 (EDSA_HLEN)) does not fix the error, because the interface keep having a maximum length of 1518 bytes (sky2->netdev->mtu+ETH_HLEN
+VLAN_HLEN).Did you check the value being written to here:
/* * The receiver hangs if it receives frames larger than the * packet buffer. As a workaround, truncate oversize frames, but * the register is limited to 9 bits, so if you do frames > 2052 * you better get the MTU right! */ thresh = sky2_get_rx_threshold(sky2); if (thresh > 0x1ff) sky2_write32(hw, SK_REG(sky2->port, RX_GMF_CTRL_T), RX_TRUNC_OFF); else { sky2_write16(hw, SK_REG(sky2->port, RX_GMF_TR_THR), thresh); sky2_write32(hw, SK_REG(sky2->port, RX_GMF_CTRL_T), RX_TRUNC_ON); }
Quoted 1 line
What is thresh?
The value of thresh is 380.
Rafa
Quoted 3 lines
Andrew
Quoted 22 lines · Rafa Corvillo
Quoted 22 lines
Quoted 4 lines
Adding 8 bytes (
sky2->netdev->mtu+ETH_HLEN+VLAN_HLEN+ 8 (EDSA_HLEN)) does not fix the error, because the interface keep having a maximum length of 1518 bytes (sky2->netdev->mtu+ETH_HLEN
+VLAN_HLEN).Did you check the value being written to here:
/* * The receiver hangs if it receives frames larger than the * packet buffer. As a workaround, truncate oversize frames, but * the register is limited to 9 bits, so if you do frames > 2052 * you better get the MTU right! */ thresh = sky2_get_rx_threshold(sky2); if (thresh > 0x1ff) sky2_write32(hw, SK_REG(sky2->port, RX_GMF_CTRL_T), RX_TRUNC_OFF); else { sky2_write16(hw, SK_REG(sky2->port, RX_GMF_TR_THR), thresh); sky2_write32(hw, SK_REG(sky2->port, RX_GMF_CTRL_T), RX_TRUNC_ON); }
Quoted 3 lines
Quoted 1 line
What is thresh?
The value of thresh is 380.
So that is 1528.
You could hack it and try 0x1ff.
Also, check that in sky2_rx_add(), le->length is set to 4K.
Andrew
Quoted 23 lines · Rafa Corvillo
Quoted 22 lines
Quoted 22 lines
Quoted 4 lines
Adding 8 bytes (
sky2->netdev->mtu+ETH_HLEN+VLAN_HLEN+ 8 (EDSA_HLEN)) does not fix the error, because the interface keep having a maximum length of 1518 bytes (sky2->netdev->mtu+ETH_HLEN
+VLAN_HLEN).Did you check the value being written to here:
/* * The receiver hangs if it receives frames larger than the * packet buffer. As a workaround, truncate oversize frames, but * the register is limited to 9 bits, so if you do frames > 2052 * you better get the MTU right! */ thresh = sky2_get_rx_threshold(sky2); if (thresh > 0x1ff) sky2_write32(hw, SK_REG(sky2->port, RX_GMF_CTRL_T), RX_TRUNC_OFF); else { sky2_write16(hw, SK_REG(sky2->port, RX_GMF_TR_THR), thresh); sky2_write32(hw, SK_REG(sky2->port, RX_GMF_CTRL_T), RX_TRUNC_ON); }
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
What is thresh?
The value of thresh is 380.
So that is 1528.
Yes, this is the result of the roundup function.
Quoted 2 lines
You could hack it and try
0x1ff.
I have forced the value of thresh to 0x1ff and the rx error still appears.
Quoted 3 lines
Also, check that in
sky2_rx_add(),le->lengthis set to 4K.
The value of le->length is set to 1520.
Rafa
Quoted 6 lines
Quoted 2 lines
Also, check that in
sky2_rx_add(),le->lengthis set to 4K.The value of
le->lengthis set to 1520.Rafa
Ah.
You probably need to change sky2_get_rx_data_size() as well, to add in the extra 8 bytes.
Andrew
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines
Also, check that in
sky2_rx_add(),le->lengthis set to 4K.The value of
le->lengthis set to 1520.
Quoted 9 lines
Quoted 1 line
Rafa
Ah.
You probably need to change
sky2_get_rx_data_size()as well, to add in the extra 8 bytes.Andrew
If I add the extra 8 byte in the function sky2_get_rx_data_size(), the value of le->length is set to 1528. But the rx error still appears. Because of that, I think it is possible the maximum length could be set outside the sky2 code.
Rafa
As modifying sky2 code I have not could get any solution, then I have modified some parameters on the ethernet interface and using a MTU = 1503 and disabling TSO (TCP Segmentation Offload) mechanism all communication errors disappear, both the rx errors due to too long packets and tx timeout errors. I have performed iperf tests during 1 hour (downstream and upstream) and the performance of the ethernet is stable. Furthermore, disabling SG (scatter-gather) improves the throughput of the ethernet interface with traffic upstream.
Is it possible that these offload mechanism don't work fine with the sky2 driver?
Thanks,
Rafa
Quoted 5 lines · Rafa Corvillo
As modifying sky2 code I have not could get any solution, then I have modified some parameters on the ethernet interface and using a MTU = 1503 and disabling TSO (TCP Segmentation Offload) mechanism all communication errors disappear, both the rx errors due to too long packets and tx timeout errors.
Hi Rafa
That at least makes some sense. It would be interesting to sniff the frames going from, the sky2 to the switch. I suspect you will find that the first frame TSO sends for a big segment has the DSA header and the following frames don't. You might want to look at how TSO is setup on transmit, if you can tell it which parts of the header needs to be placed on each segment, so that it includes the DSA header.
Andrew