mvpp2: incorrect max mtu?
Subsystems (2)
Hi all,
While testing out the 10G speeds on my Macchiatobin platforms, the first thing I notice is that they only manage about 1Gbps at a MTU of 1500. As expected, this increases when the MTU is increased - a MTU of 9000 works, and gives a useful performance boost.
Then comes the obvious question - what is the maximum MTU.
#define MVPP2_BM_JUMBO_FRAME_SIZE 10432 /* frame size 9856 */
So, one may assume that 9856 is the maximum. However:
# ip li set dev eth0 mtu 9888 # ip li set dev eth0 mtu 9889 Error: mtu greater than device maximum.
So, the maximum that userspace can set appears to be 9888. If this is set, then, while running iperf3, we get:
mvpp2 f2000000.ethernet eth0: bad rx status 9202e510 (resource error), size=9888
So clearly this is too large, and we should not be allowing userspace to set this large a MTU.
At this point, it seems to be impossible to regain the previous speed of the interface by lowering the MTU. Here is a MTU of 9000:
[ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.37 MBytes 11.5 Mbits/sec 40 17.5 KBytes [ 5] 1.00-2.00 sec 1.25 MBytes 10.5 Mbits/sec 39 8.74 KBytes [ 5] 2.00-3.00 sec 1.13 MBytes 9.45 Mbits/sec 36 17.5 KBytes [ 5] 3.00-4.00 sec 1.13 MBytes 9.45 Mbits/sec 39 8.74 KBytes [ 5] 4.00-5.00 sec 1.13 MBytes 9.45 Mbits/sec 36 17.5 KBytes [ 5] 5.00-6.00 sec 1.28 MBytes 10.7 Mbits/sec 39 8.74 KBytes [ 5] 6.00-7.00 sec 1.13 MBytes 9.45 Mbits/sec 36 17.5 KBytes [ 5] 7.00-8.00 sec 1.25 MBytes 10.5 Mbits/sec 39 8.74 KBytes [ 5] 8.00-9.00 sec 1.13 MBytes 9.45 Mbits/sec 36 17.5 KBytes [ 5] 9.00-10.00 sec 1.13 MBytes 9.45 Mbits/sec 39 8.74 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 11.9 MBytes 9.99 Mbits/sec 379 sender [ 5] 0.00-10.00 sec 11.7 MBytes 9.80 Mbits/sec receiver
Whereas before the test, it was:
[ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 729 MBytes 6.11 Gbits/sec [ 5] 1.00-2.00 sec 719 MBytes 6.03 Gbits/sec [ 5] 2.00-3.00 sec 773 MBytes 6.49 Gbits/sec [ 5] 3.00-4.00 sec 769 MBytes 6.45 Gbits/sec [ 5] 4.00-5.00 sec 779 MBytes 6.54 Gbits/sec [ 5] 5.00-6.00 sec 784 MBytes 6.58 Gbits/sec [ 5] 6.00-7.00 sec 777 MBytes 6.52 Gbits/sec [ 5] 7.00-8.00 sec 774 MBytes 6.50 Gbits/sec [ 5] 8.00-9.00 sec 769 MBytes 6.45 Gbits/sec [ 5] 9.00-10.00 sec 774 MBytes 6.49 Gbits/sec [ 5] 10.00-10.00 sec 3.07 MBytes 5.37 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 7.47 GBytes 6.41 Gbits/sec receiver
(this is on the server end of iperf3, the others are the client end, but the results were pretty very similar to that.)
So, clearly something bad has happened to the buffer management as a result of raising the MTU so high.
As the end which has suffered this issue is the mcbin VM host, I'm not currently in a position I can reboot it without cause major disruption to my network. However, thoughts on this (and... can others reproduce it) would be useful.
Thanks.
Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
Hi Russell,
pt., 14 maj 2021 o 15:00 Russell King (Oracle) <[email protected]> napisał(a):
Quoted 66 lines
Hi all,
While testing out the 10G speeds on my Macchiatobin platforms, the first thing I notice is that they only manage about 1Gbps at a MTU of 1500. As expected, this increases when the MTU is increased - a MTU of 9000 works, and gives a useful performance boost.
Then comes the obvious question - what is the maximum MTU.
#define
MVPP2_BM_JUMBO_FRAME_SIZE10432 /* frame size 9856 */So, one may assume that 9856 is the maximum. However:
# ip li set dev eth0 mtu 9888 # ip li set dev eth0 mtu 9889 Error: mtu greater than device maximum.So, the maximum that userspace can set appears to be 9888. If this is set, then, while running iperf3, we get:
mvpp2 f2000000.ethernet eth0: bad rx status
9202e510(resource error), size=9888So clearly this is too large, and we should not be allowing userspace to set this large a MTU.
At this point, it seems to be impossible to regain the previous speed of the interface by lowering the MTU. Here is a MTU of 9000:
[ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.37 MBytes 11.5 Mbits/sec 40 17.5 KBytes [ 5] 1.00-2.00 sec 1.25 MBytes 10.5 Mbits/sec 39 8.74 KBytes [ 5] 2.00-3.00 sec 1.13 MBytes 9.45 Mbits/sec 36 17.5 KBytes [ 5] 3.00-4.00 sec 1.13 MBytes 9.45 Mbits/sec 39 8.74 KBytes [ 5] 4.00-5.00 sec 1.13 MBytes 9.45 Mbits/sec 36 17.5 KBytes [ 5] 5.00-6.00 sec 1.28 MBytes 10.7 Mbits/sec 39 8.74 KBytes [ 5] 6.00-7.00 sec 1.13 MBytes 9.45 Mbits/sec 36 17.5 KBytes [ 5] 7.00-8.00 sec 1.25 MBytes 10.5 Mbits/sec 39 8.74 KBytes [ 5] 8.00-9.00 sec 1.13 MBytes 9.45 Mbits/sec 36 17.5 KBytes [ 5] 9.00-10.00 sec 1.13 MBytes 9.45 Mbits/sec 39 8.74 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 11.9 MBytes 9.99 Mbits/sec 379 sender [ 5] 0.00-10.00 sec 11.7 MBytes 9.80 Mbits/sec receiverWhereas before the test, it was:
[ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 729 MBytes 6.11 Gbits/sec [ 5] 1.00-2.00 sec 719 MBytes 6.03 Gbits/sec [ 5] 2.00-3.00 sec 773 MBytes 6.49 Gbits/sec [ 5] 3.00-4.00 sec 769 MBytes 6.45 Gbits/sec [ 5] 4.00-5.00 sec 779 MBytes 6.54 Gbits/sec [ 5] 5.00-6.00 sec 784 MBytes 6.58 Gbits/sec [ 5] 6.00-7.00 sec 777 MBytes 6.52 Gbits/sec [ 5] 7.00-8.00 sec 774 MBytes 6.50 Gbits/sec [ 5] 8.00-9.00 sec 769 MBytes 6.45 Gbits/sec [ 5] 9.00-10.00 sec 774 MBytes 6.49 Gbits/sec [ 5] 10.00-10.00 sec 3.07 MBytes 5.37 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 7.47 GBytes 6.41 Gbits/sec receiver(this is on the server end of iperf3, the others are the client end, but the results were pretty very similar to that.)
Quoted 8 lines
So, clearly something bad has happened to the buffer management as a result of raising the MTU so high.
As the end which has suffered this issue is the mcbin VM host, I'm not currently in a position I can reboot it without cause major disruption to my network. However, thoughts on this (and... can others reproduce it) would be useful.
Thank your for the information. I will take a look after the weekend. To be aligned - what exactly kernel baseline are you using?
Best regards,
Marcin
Quoted 9 lines · Russell King (Oracle)
Hi all,
While testing out the 10G speeds on my Macchiatobin platforms, the first thing I notice is that they only manage about 1Gbps at a MTU of 1500. As expected, this increases when the MTU is increased - a MTU of 9000 works, and gives a useful performance boost.
Then comes the obvious question - what is the maximum MTU.
Quoted 7 lines
#define
MVPP2_BM_JUMBO_FRAME_SIZE10432 /* frame size 9856 */So, one may assume that 9856 is the maximum. However:
# ip li set dev eth0 mtu 9888 # ip li set dev eth0 mtu 9889 Error: mtu greater than device maximum.
Hi Russell
It all seems inconsistent:
https://elixir.bootlin.com/linux/latest/source/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c#L6879
/* MTU range: 68 - 9704 */ dev->min_mtu = ETH_MIN_MTU; /* 9704 == 9728 - 20 and rounding to 8 */ dev->max_mtu = MVPP2_BM_JUMBO_PKT_SIZE;
Maybe this comment is correct, the code is now wrong, and the MAX MTU should be 9704?
Andrew
Quoted 2 lines · Marcin Wojtas
Thank your for the information. I will take a look after the weekend. To be aligned - what exactly kernel baseline are you using?
That was with 5.11 with these additional mvpp2 patches:
net: mvpp2: add TX FC firmware check
net: mvpp2: set 802.3x GoP Flow Control mode
net: mvpp2: add PPv23 RX FIFO flow control
net: mvpp2: add BM protection underrun feature support
net: mvpp2: add ethtool flow control configuration support
net: mvpp2: add RXQ flow control configurations
net: mvpp2: enable global flow control
net: mvpp2: add FCA RXQ non occupied descriptor threshold
net: mvpp2: add FCA periodic timer configurations
net: mvpp2: increase BM pool and RXQ size
net: mvpp2: add PPv23 version definition
net: mvpp2: always compare hw-version vs MVPP21
net: mvpp2: add CM3 SRAM memory map
dts: marvell: add CM3 SRAM memory to cp11x ethernet device tree
doc: marvell: add CM3 address space and PPv2.3 description
net: marvell: Fixed two spellings,controling to controlling and oen to one
net: mvpp2: prs: improve ipv4 parse flow
I'll also try to work out what's happening, but I think we need to find out what the correct value for dev->max_mtu should be. That's all rather convoluted:
dev->max_mtu = MVPP2_BM_JUMBO_PKT_SIZE;
#define MVPP2_BM_JUMBO_PKT_SIZE MVPP2_RX_MAX_PKT_SIZE(MVPP2_BM_JUMBO_FRAME_SIZE)
#define MVPP2_BM_JUMBO_FRAME_SIZE 10432 /* frame size 9856 */
#define MVPP2_RX_MAX_PKT_SIZE(total_size) \
((total_size) - MVPP2_SKB_HEADROOM - MVPP2_SKB_SHINFO_SIZE)
The maximum settable MTU on eth0 (9888) disagrees with the comment "frame size 9856" by 32 bytes.
I haven't checked to see whether 9856 works yet, because that will first need me to reboot the machine... which I'll do over the weekend.
Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
Quoted 24 lines · Andrew Lunn
Quoted 9 lines · Russell King (Oracle)
Hi all,
While testing out the 10G speeds on my Macchiatobin platforms, the first thing I notice is that they only manage about 1Gbps at a MTU of 1500. As expected, this increases when the MTU is increased - a MTU of 9000 works, and gives a useful performance boost.
Then comes the obvious question - what is the maximum MTU.
Quoted 7 lines
#define
MVPP2_BM_JUMBO_FRAME_SIZE10432 /* frame size 9856 */So, one may assume that 9856 is the maximum. However:
# ip li set dev eth0 mtu 9888 # ip li set dev eth0 mtu 9889 Error: mtu greater than device maximum.Hi Russell
It all seems inconsistent:
https://elixir.bootlin.com/linux/latest/source/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c#L6879
Quoted 7 lines
/* MTU range: 68 - 9704 */ dev->min_mtu = ETH_MIN_MTU; /* 9704 == 9728 - 20 and rounding to 8 */ dev->max_mtu = MVPP2_BM_JUMBO_PKT_SIZE;Maybe this comment is correct, the code is now wrong, and the MAX MTU should be 9704?
Oh, there's more values given elsewhere that disagree, see my reply to Marcin. I would not be surprised if this was all "confused" about what the proper value is. Certainly the comment you mention above disagrees with what is in mvpp2.h
Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
Quoted 81 lines
-----Original Message-----
From: Russell King <[email protected]>
Sent: Friday, May 14, 2021 4:00 PM
To: Stefan Chulski <[email protected]>
Cc: Marcin Wojtas <[email protected]>; [email protected]
Subject: [EXT] mvpp2: incorrect max mtu?External Email
----------------------------------------------------------------------
Hi all,While testing out the 10G speeds on my Macchiatobin platforms, the first thing I notice is that they only manage about 1Gbps at a MTU of 1500. As expected, this increases when the MTU is increased - a MTU of 9000 works, and gives a useful performance boost.
Then comes the obvious question - what is the maximum MTU.
#define
MVPP2_BM_JUMBO_FRAME_SIZE10432 /* frame size 9856 */So, one may assume that 9856 is the maximum. However:
# ip li set dev eth0 mtu 9888 # ip li set dev eth0 mtu 9889 Error: mtu greater than device maximum.So, the maximum that userspace can set appears to be 9888. If this is set, then, while running iperf3, we get:
mvpp2 f2000000.ethernet eth0: bad rx status
9202e510(resource error), size=9888So clearly this is too large, and we should not be allowing userspace to set this large a MTU.
At this point, it seems to be impossible to regain the previous speed of the interface by lowering the MTU. Here is a MTU of 9000:
[ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.37 MBytes 11.5 Mbits/sec 40 17.5 KBytes [ 5] 1.00-2.00 sec 1.25 MBytes 10.5 Mbits/sec 39 8.74 KBytes [ 5] 2.00-3.00 sec 1.13 MBytes 9.45 Mbits/sec 36 17.5 KBytes [ 5] 3.00-4.00 sec 1.13 MBytes 9.45 Mbits/sec 39 8.74 KBytes [ 5] 4.00-5.00 sec 1.13 MBytes 9.45 Mbits/sec 36 17.5 KBytes [ 5] 5.00-6.00 sec 1.28 MBytes 10.7 Mbits/sec 39 8.74 KBytes [ 5] 6.00-7.00 sec 1.13 MBytes 9.45 Mbits/sec 36 17.5 KBytes [ 5] 7.00-8.00 sec 1.25 MBytes 10.5 Mbits/sec 39 8.74 KBytes [ 5] 8.00-9.00 sec 1.13 MBytes 9.45 Mbits/sec 36 17.5 KBytes [ 5] 9.00-10.00 sec 1.13 MBytes 9.45 Mbits/sec 39 8.74 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 11.9 MBytes 9.99 Mbits/sec 379 sender [ 5] 0.00-10.00 sec 11.7 MBytes 9.80 Mbits/sec receiverWhereas before the test, it was:
[ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 729 MBytes 6.11 Gbits/sec [ 5] 1.00-2.00 sec 719 MBytes 6.03 Gbits/sec [ 5] 2.00-3.00 sec 773 MBytes 6.49 Gbits/sec [ 5] 3.00-4.00 sec 769 MBytes 6.45 Gbits/sec [ 5] 4.00-5.00 sec 779 MBytes 6.54 Gbits/sec [ 5] 5.00-6.00 sec 784 MBytes 6.58 Gbits/sec [ 5] 6.00-7.00 sec 777 MBytes 6.52 Gbits/sec [ 5] 7.00-8.00 sec 774 MBytes 6.50 Gbits/sec [ 5] 8.00-9.00 sec 769 MBytes 6.45 Gbits/sec [ 5] 9.00-10.00 sec 774 MBytes 6.49 Gbits/sec [ 5] 10.00-10.00 sec 3.07 MBytes 5.37 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 7.47 GBytes 6.41 Gbits/sec receiver(this is on the server end of iperf3, the others are the client end, but the results were pretty very similar to that.)
So, clearly something bad has happened to the buffer management as a result of raising the MTU so high.
Quoted 6 lines
As the end which has suffered this issue is the mcbin VM host, I'm not currently in a position I can reboot it without cause major disruption to my network. However, thoughts on this (and... can others reproduce it) would be useful.
Thanks.
Look like PPv2 tried scatter frame since it was larger than Jumbo buffer size and it drained buffer pool(Buffers never released). Received packet should be less than value set in MVPP2_POOL_BUF_SIZE_REG for long pool.
Stefan.
Quoted 2 lines · Stefan Chulski
Look like PPv2 tried scatter frame since it was larger than Jumbo buffer size and it drained buffer pool(Buffers never released). Received packet should be less than value set in
MVPP2_POOL_BUF_SIZE_REGfor long pool.
So this must mean that setting dev->max_mtu is incorrect.
From what I can see, the value programmed into that register would be MVPP2_BM_JUMBO_PKT_SIZE which I believe is 9888. This is currently the same value that dev->max_mtu is set to, but max_mtu is the data payload size in the ethernet frame, which doesn't include the hardware ethernet header.
So, should max_mtu be set to 14 bytes less? Or should it be set to 9856? Less 14 bytes? Or what?
It is really confusing that we have these definitions that state e.g. that JUMBO_FRAME_SIZE is 10432 but the frame size comment says 9856. It's not clear why it's different like that - why the additional 576 octets.
All of this could do with some explanation in the driver - would it be possible to add some kind of documentation, or at least make the definitions around packet and frame size more understandable please?
Thanks.
Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
Quoted 15 lines
-----Original Message-----
From: Russell King <[email protected]>
Sent: Tuesday, May 18, 2021 12:42 PM
To: Stefan Chulski <[email protected]>
Cc: Marcin Wojtas <[email protected]>; [email protected]
Subject: Re: [EXT] mvpp2: incorrect max mtu?Quoted 1 line · Stefan Chulski
Look like PPv2 tried scatter frame since it was larger than Jumbo buffer size
and it drained buffer pool(Buffers never released).
Quoted 1 line
Received packet should be less than value set in
MVPP2_POOL_BUF_SIZE_REGfor long pool.So this must mean that setting
dev->max_mtuis incorrect.
Quoted 8 lines
From what I can see, the value programmed into that register would be
MVPP2_BM_JUMBO_PKT_SIZEwhich I believe is 9888. This is currently the same value thatdev->max_mtuis set to, butmax_mtuis the data payload size in the ethernet frame, which doesn't include the hardware ethernet header.So, should
max_mtube set to 14 bytes less? Or should it be set to 9856? Less 14 bytes? Or what?
Yes, look like dev->max_mtuis incorrect. It should be 9856, MVPP2_POOL_BUF_SIZE_REG, Pool Buffer Size is in 32 units. But before changing it I prefer to test different sizes. Currently, I don't have a connected board.
I will connect board and perform some tests. JUMBO_FRAME_SIZE is the size of the buffer, but packet offset should be taken into account.
Quoted 9 lines
It is really confusing that we have these definitions that state e.g. that
JUMBO_FRAME_SIZEis 10432 but the frame size comment says 9856. It's not clear why it's different like that - why the additional 576 octets.All of this could do with some explanation in the driver - would it be possible to add some kind of documentation, or at least make the definitions around packet and frame size more understandable please?
Thanks.
Yes, definition or comments should be improved.
Regards,
Stefan.
Quoted 41 lines
Quoted 22 lines
-----Original Message-----
From: Russell King <[email protected]>
Sent: Tuesday, May 18, 2021 12:42 PM
To: Stefan Chulski <[email protected]>
Cc: Marcin Wojtas <[email protected]>; [email protected]
Subject: Re: [EXT] mvpp2: incorrect max mtu?Quoted 2 lines · Stefan Chulski
Look like PPv2 tried scatter frame since it was larger than Jumbo buffer size
and it drained buffer pool(Buffers never released).
Quoted 1 line
Received packet should be less than value set in
MVPP2_POOL_BUF_SIZE_REGfor long pool.So this must mean that setting
dev->max_mtuis incorrect.From what I can see, the value programmed into that register would be
MVPP2_BM_JUMBO_PKT_SIZEwhich I believe is 9888. This is currently the same value thatdev->max_mtuis set to, butmax_mtuis the data payload size in the ethernet frame, which doesn't include the hardware ethernet header.Quoted 3 lines
So, should
max_mtube set to 14 bytes less? Or should it be set to 9856? Less
14 bytes? Or what?Yes, look like
dev->max_mtuisincorrect. It should be 9856,MVPP2_POOL_BUF_SIZE_REG, Pool Buffer Size is in 32 units.
But before changing it I prefer to test different sizes. Currently, I don't have a connected board.
I will connect board and perform some tests.JUMBO_FRAME_SIZEis the size of the buffer, but packet offset should be taken into account.Quoted 8 lines
It is really confusing that we have these definitions that state e.g. that
JUMBO_FRAME_SIZEis 10432 but the frame size comment says 9856. It's not clear why it's different like that - why the additional 576 octets.All of this could do with some explanation in the driver - would it be possible to add some kind of documentation, or at least make the definitions around packet and frame size more understandable please?
Quoted 6 lines
Quoted 1 line
Thanks.
Yes, definition or comments should be improved.
Regards,
Stefan.
We probably should split this to 3 issues.
1. BM drain issue. I can post patch witch should fix it. So after lowering the MTU you won't have issue.
2. Regarding MTU max, here we actually talking about max MRU not MTU.
MRU set by mvpp2_xlg_max_rx_size_set/mvpp2_gmac_max_rx_size_set port->pkt_size - MVPP2_MH_SIZE(2 Bytes).port->pkt_size = MTU + MVPP2_MH_SIZE(2 Bytes) + VLAN tag (4 Bytes) + ETH_HLEN(14 Bytes) + ETH_FCS_LEN(4 Bytes) and from some reason aligned to cache_line_size(). So if we set MTU = 9888, we actually get MRU 9918.
3. Fix all max MTU comments in code.
Regards,
Stefan.