net: mvpp2: Classifier updates, RSS
Subsystems (3)
cover net: mvpp2: Classifier updates, RSS
Hello everyone,
Here is a set of updates for the PPv2 classifier, the main feature being the support for steering to RSS contexts, to leverage all the available RSS tables in the controller.
The first two patches are non-critical fixes for the classifier, the first one prevents us from allocating too much room to store the classification rules, the second one configuring the C2 engine as suggested by the PPv2 functionnal specs.
Patches 3 to 5 introduce support for RSS contexts in mvpp2, allowing us to steer traffic to dedicated RSS tables.
Thanks,
Maxime
Maxime Chevallier (5):
net: mvpp2: cls: Use the correct number of rules in various places
net: mvpp2: cls: Bypass C2 internals FIFOs at init
net: mvpp2: cls: Use RSS contexts to handle RSS tables
net: mvpp2: cls: Extract the RSS context when parsing the ethtool rule
net: mvpp2: cls: Support steering to RSS contexts
drivers/net/ethernet/marvell/mvpp2/mvpp2.h | 20 +- .../net/ethernet/marvell/mvpp2/mvpp2_cls.c | 272 ++++++++++++++++-- .../net/ethernet/marvell/mvpp2/mvpp2_cls.h | 15 +- .../net/ethernet/marvell/mvpp2/mvpp2_main.c | 67 ++++- 4 files changed, 326 insertions(+), 48 deletions(-)
Signature · 1 line
2.20.1
PATCH 1/5 net: mvpp2: cls: Use the correct number of rules in various places
As of today, the classification offload implementation only supports 4 different rules to be offloaded. This number has been hardcoded in the rule insertion function, and the wrong define is being used elsewhere.
Use the correct #define everywhere to make sure we always check for the correct number of rules.
--- drivers/net/ethernet/marvell/mvpp2/mvpp2.h | 2 +- drivers/net/ethernet/marvell/mvpp2/mvpp2_cls.c | 4 ++-- drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-)
drivers/net/ethernet/marvell/mvpp2/mvpp2.h +1 -1
@@ -923,7 +923,7 @@ struct mvpp2_port {u32 indir[MVPP22_RSS_TABLE_ENTRIES ];/* List of steering rules active on that port */ -struct mvpp2_ethtool_fs *rfs_rules[MVPP2_N_RFS_RULES ]; +struct mvpp2_ethtool_fs *rfs_rules[MVPP2_N_RFS_ENTRIES_PER_FLOW ];int n_rfs_rules; };
drivers/net/ethernet/marvell/mvpp2/mvpp2_cls.c +2 -2
@@ -1212,7 +1212,7 @@ int mvpp2_ethtool_cls_rule_get(struct mvpp2_port *port, {struct mvpp2_ethtool_fs *efs; -if (rxnfc->fs.location >=MVPP2_N_RFS_RULES ) +if (rxnfc->fs.location >=MVPP2_N_RFS_ENTRIES_PER_FLOW )return -EINVAL ; efs = port->rfs_rules[rxnfc->fs.location];
@@ -1232,7 +1232,7 @@ int mvpp2_ethtool_cls_rule_ins(struct mvpp2_port *port,struct mvpp2_ethtool_fs *efs, *old_efs;int ret =0 ; -if (info->fs.location >=4 || +if (info->fs.location >=MVPP2_N_RFS_ENTRIES_PER_FLOW || info->fs.location <0 )return -EINVAL ;
drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c +1 -1
@@ -3956,7 +3956,7 @@ static int mvpp2_ethtool_get_rxnfc(struct net_device *dev, ret =mvpp2_ethtool_cls_rule_get (port, info);break ;case ETHTOOL_GRXCLSRLALL : -for (i =0 ; i <MVPP2_N_RFS_RULES ; i++) { +for (i =0 ; i <MVPP2_N_RFS_ENTRIES_PER_FLOW ; i++) {if (port->rfs_rules[i]) rules[loc++] = i; }
Signature · 1 line
2.20.1
PATCH 2/5 net: mvpp2: cls: Bypass C2 internals FIFOs at init
In reply to Maxime ChevallierThe C2 TCAM has internal FIFOs that are only useful for the built-in self-tests. Disable these FIFOS at init, as recommended in the functionnal specs.
--- drivers/net/ethernet/marvell/mvpp2/mvpp2.h | 2 ++ drivers/net/ethernet/marvell/mvpp2/mvpp2_cls.c | 6 ++++++ 2 files changed, 8 insertions(+)
drivers/net/ethernet/marvell/mvpp2/mvpp2.h +2 -0
@@ -148,6 +148,8 @@#define MVPP22_CLS_C2_ATTR2 0x1b6c#define MVPP22_CLS_C2_ATTR2_RSS_EN BIT(30)#define MVPP22_CLS_C2_ATTR3 0x1b70 +#define MVPP22_CLS_C2_TCAM_CTRL 0x1b90 +#define MVPP22_CLS_C2_TCAM_BYPASS_FIFO BIT(0)/* Descriptor Manager Top Registers */ #define MVPP2_RXQ_NUM_REG 0x2040
drivers/net/ethernet/marvell/mvpp2/mvpp2_cls.c +6 -0
@@ -923,6 +923,12 @@ void mvpp2_cls_init(struct mvpp2 *priv)mvpp2_cls_c2_write (priv, &c2); } +/* Disable the FIFO stages in C2 engine, which are only used in BIST +* mode +*/ +mvpp2_write (priv,MVPP22_CLS_C2_TCAM_CTRL , +MVPP22_CLS_C2_TCAM_BYPASS_FIFO ); +mvpp2_cls_port_init_flows (priv); }
Signature · 1 line
2.20.1
PATCH 3/5 net: mvpp2: cls: Use RSS contexts to handle RSS tables
In reply to Maxime ChevallierThe PPv2 controller has 8 RSS tables that are shared across all ports on a given PPv2 instance. The previous implementation allocated one table per port, leaving others unused.
By using RSS contexts, we can make use of multiple RSS tables per port, one being the default table (always id 0), the other ones being used as destinations for flow steering, in the same way as rx rings.
This commit introduces RSS contexts management in the PPv2 driver. We always reserve one table per port, allocated when the port is probed.
The global table list is stored in the struct mvpp2, as it's a global resource. Each port then maintains a list of indices in that global table, that way each port can have it's own numbering scheme starting from 0.
One limitation that seems unavoidable is that the hashing parameters are shared across all RSS contexts for a given port. Hashing parameters for ctx 0 will be applied to all contexts.
--- drivers/net/ethernet/marvell/mvpp2/mvpp2.h | 16 +- .../net/ethernet/marvell/mvpp2/mvpp2_cls.c | 200 +++++++++++++++--- .../net/ethernet/marvell/mvpp2/mvpp2_cls.h | 15 +- .../net/ethernet/marvell/mvpp2/mvpp2_main.c | 65 +++++- 4 files changed, 255 insertions(+), 41 deletions(-)
drivers/net/ethernet/marvell/mvpp2/mvpp2.h +13 -3
@@ -626,6 +626,7 @@#define MVPP2_N_RFS_RULES (MVPP2_N_RFS_ENTRIES_PER_FLOW * 7)/* RSS constants */ +#define MVPP22_N_RSS_TABLES 8#define MVPP22_RSS_TABLE_ENTRIES 32/* IPv6 max L3 address size */
@@ -727,6 +728,10 @@ enum mvpp2_prs_l3_cast {/* Definitions */ struct mvpp2_dbgfs_entries ; +struct mvpp2_rss_table { +u32 indir[MVPP22_RSS_TABLE_ENTRIES ]; +}; +/* Shared Packet Processor resources */ struct mvpp2 {/* Shared registers' base addresses */
@@ -790,6 +795,9 @@ struct mvpp2 {/* Debugfs entries private data */ struct mvpp2_dbgfs_entries *dbgfs_entries; + +/* RSS Indirection tables */ +struct mvpp2_rss_table *rss_tables[MVPP22_N_RSS_TABLES ]; };struct mvpp2_pcpu_stats {
@@ -921,12 +929,14 @@ struct mvpp2_port {u32 tx_time_coal; -/* RSS indirection table */ -u32 indir[MVPP22_RSS_TABLE_ENTRIES ]; -/* List of steering rules active on that port */ struct mvpp2_ethtool_fs *rfs_rules[MVPP2_N_RFS_ENTRIES_PER_FLOW ];int n_rfs_rules; + +/* Each port has its own view of the rss contexts, so that it can number +* them from 0 +*/ +int rss_ctx[MVPP22_N_RSS_TABLES ]; };/* The mvpp2_tx_desc and mvpp2_rx_desc structures describe the
drivers/net/ethernet/marvell/mvpp2/mvpp2_cls.c +176 -24
265 lines not on this page · raw message
drivers/net/ethernet/marvell/mvpp2/mvpp2_cls.h +11 -4
@@ -249,11 +249,18 @@ struct mvpp2_cls_lookup_entry {u32 data; }; -void mvpp22_rss_fill_table (struct mvpp2_port *port,u32 table); -void mvpp22_port_rss_init (struct mvpp2_port *port); +int mvpp22_port_rss_init (struct mvpp2_port *port); -void mvpp22_port_rss_enable (struct mvpp2_port *port); -void mvpp22_port_rss_disable (struct mvpp2_port *port); +int mvpp22_port_rss_enable (struct mvpp2_port *port); +int mvpp22_port_rss_disable (struct mvpp2_port *port); + +int mvpp22_port_rss_ctx_create (struct mvpp2_port *port,u32 *rss_ctx); +int mvpp22_port_rss_ctx_delete (struct mvpp2_port *port,u32 rss_ctx); + +int mvpp22_port_rss_ctx_indir_set (struct mvpp2_port *port,u32 rss_ctx, +const u32 *indir); +int mvpp22_port_rss_ctx_indir_get (struct mvpp2_port *port,u32 rss_ctx, +u32 *indir);int mvpp2_ethtool_rxfh_get (struct mvpp2_port *port,struct ethtool_rxnfc *info);int mvpp2_ethtool_rxfh_set (struct mvpp2_port *port,struct ethtool_rxnfc *info);
drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c +55 -10
101 lines not on this page · raw message
Signature · 1 line
2.20.1
Re: [PATCH net-next 3/5] net: mvpp2: cls: Use RSS contexts to handle RSS tables
Quoted 17 lines · Maxime Chevallier
The PPv2 controller has 8 RSS tables that are shared across all ports on a given PPv2 instance. The previous implementation allocated one table per port, leaving others unused.
By using RSS contexts, we can make use of multiple RSS tables per port, one being the default table (always id 0), the other ones being used as destinations for flow steering, in the same way as rx rings.
This commit introduces RSS contexts management in the PPv2 driver. We always reserve one table per port, allocated when the port is probed.
The global table list is stored in the struct mvpp2, as it's a global resource. Each port then maintains a list of indices in that global table, that way each port can have it's own numbering scheme starting from 0.
Quoted 5 lines
One limitation that seems unavoidable is that the hashing parameters are shared across all RSS contexts for a given port. Hashing parameters for ctx 0 will be applied to all contexts.
Signed-off-by Maxime Chevallier <[email protected]>
Hi all,
I noticed that enabling rxhash blocks the RX on my Macchiatobin. It works fine with the 10G ports (the RX rate goes 4x up) but it completely kills the gigabit interface.
# 10G port
root@macchiatobin:~# iperf3 -c 192.168.0.2 Connecting to host 192.168.0.2, port 5201 [ 5] local 192.168.0.1 port 42394 connected to 192.168.0.2 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 941 MBytes 7.89 Gbits/sec 4030 250 KBytes [ 5] 1.00-2.00 sec 933 MBytes 7.82 Gbits/sec 4393 240 KBytes root@macchiatobin:~# ethtool -K eth0 rxhash on root@macchiatobin:~# iperf3 -c 192.168.0.2 Connecting to host 192.168.0.2, port 5201 [ 5] local 192.168.0.1 port 42398 connected to 192.168.0.2 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 860 MBytes 7.21 Gbits/sec 428 410 KBytes [ 5] 1.00-2.00 sec 859 MBytes 7.20 Gbits/sec 185 563 KBytes
# gigabit port root@macchiatobin:~# iperf3 -c turbo Connecting to host turbo, port 5201 [ 5] local 192.168.85.42 port 45144 connected to 192.168.85.6 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 113 MBytes 948 Mbits/sec 0 407 KBytes [ 5] 1.00-2.00 sec 112 MBytes 942 Mbits/sec 0 428 KBytes root@macchiatobin:~# ethtool -K eth2 rxhash on root@macchiatobin:~# iperf3 -c turbo iperf3: error - unable to connect to server: Resource temporarily unavailable
I've bisected and it seems that this commit causes the issue. I tried to revert it on nex-next as a second test, but the code has changed a lot much since, generating too much conflicts.
Can you have a look into this?
Thanks,
Signature · 2 lines
Matteo Croce per aspera ad upstream
Russell King - ARM Linux admin
maintainerRe: [PATCH net-next 3/5] net: mvpp2: cls: Use RSS contexts to handle RSS tables
Quoted 57 lines · Matteo Croce
Quoted 17 lines · Maxime Chevallier
The PPv2 controller has 8 RSS tables that are shared across all ports on a given PPv2 instance. The previous implementation allocated one table per port, leaving others unused.
By using RSS contexts, we can make use of multiple RSS tables per port, one being the default table (always id 0), the other ones being used as destinations for flow steering, in the same way as rx rings.
This commit introduces RSS contexts management in the PPv2 driver. We always reserve one table per port, allocated when the port is probed.
The global table list is stored in the struct mvpp2, as it's a global resource. Each port then maintains a list of indices in that global table, that way each port can have it's own numbering scheme starting from 0.
Quoted 5 lines
One limitation that seems unavoidable is that the hashing parameters are shared across all RSS contexts for a given port. Hashing parameters for ctx 0 will be applied to all contexts.
Signed-off-by Maxime Chevallier <[email protected]>Hi all,
I noticed that enabling rxhash blocks the RX on my Macchiatobin. It works fine with the 10G ports (the RX rate goes 4x up) but it completely kills the gigabit interface.
# 10G port
root@macchiatobin:~# iperf3 -c 192.168.0.2 Connecting to host 192.168.0.2, port 5201 [ 5] local 192.168.0.1 port 42394 connected to 192.168.0.2 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 941 MBytes 7.89 Gbits/sec 4030 250 KBytes [ 5] 1.00-2.00 sec 933 MBytes 7.82 Gbits/sec 4393 240 KBytes root@macchiatobin:~# ethtool -K eth0 rxhash on root@macchiatobin:~# iperf3 -c 192.168.0.2 Connecting to host 192.168.0.2, port 5201 [ 5] local 192.168.0.1 port 42398 connected to 192.168.0.2 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 860 MBytes 7.21 Gbits/sec 428 410 KBytes [ 5] 1.00-2.00 sec 859 MBytes 7.20 Gbits/sec 185 563 KBytes# gigabit port root@macchiatobin:~# iperf3 -c turbo Connecting to host turbo, port 5201 [ 5] local 192.168.85.42 port 45144 connected to 192.168.85.6 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 113 MBytes 948 Mbits/sec 0 407 KBytes [ 5] 1.00-2.00 sec 112 MBytes 942 Mbits/sec 0 428 KBytes root@macchiatobin:~# ethtool -K eth2 rxhash on root@macchiatobin:~# iperf3 -c turbo iperf3: error - unable to connect to server: Resource temporarily unavailable
Quoted 4 lines
I've bisected and it seems that this commit causes the issue. I tried to revert it on nex-next as a second test, but the code has changed a lot much since, generating too much conflicts.
Can you have a look into this?
This behaviour on eth2 is confirmed here on v5.6. Turning on rxhash appears to prevent eth2 working.
Maxime, please look into this regression, thanks.
Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTC broadband for 0.8mile line in suburbia: sync at 10.2Mbps down 587kbps up
Re: [PATCH net-next 3/5] net: mvpp2: cls: Use RSS contexts to handle RSS tables
Quoted 64 lines · Russell King - ARM Linux admin
Quoted 57 lines · Matteo Croce
Quoted 17 lines · Maxime Chevallier
The PPv2 controller has 8 RSS tables that are shared across all ports on a given PPv2 instance. The previous implementation allocated one table per port, leaving others unused.
By using RSS contexts, we can make use of multiple RSS tables per port, one being the default table (always id 0), the other ones being used as destinations for flow steering, in the same way as rx rings.
This commit introduces RSS contexts management in the PPv2 driver. We always reserve one table per port, allocated when the port is probed.
The global table list is stored in the struct mvpp2, as it's a global resource. Each port then maintains a list of indices in that global table, that way each port can have it's own numbering scheme starting from 0.
Quoted 5 lines
One limitation that seems unavoidable is that the hashing parameters are shared across all RSS contexts for a given port. Hashing parameters for ctx 0 will be applied to all contexts.
Signed-off-by Maxime Chevallier <[email protected]>Hi all,
I noticed that enabling rxhash blocks the RX on my Macchiatobin. It works fine with the 10G ports (the RX rate goes 4x up) but it completely kills the gigabit interface.
# 10G port
root@macchiatobin:~# iperf3 -c 192.168.0.2 Connecting to host 192.168.0.2, port 5201 [ 5] local 192.168.0.1 port 42394 connected to 192.168.0.2 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 941 MBytes 7.89 Gbits/sec 4030 250 KBytes [ 5] 1.00-2.00 sec 933 MBytes 7.82 Gbits/sec 4393 240 KBytes root@macchiatobin:~# ethtool -K eth0 rxhash on root@macchiatobin:~# iperf3 -c 192.168.0.2 Connecting to host 192.168.0.2, port 5201 [ 5] local 192.168.0.1 port 42398 connected to 192.168.0.2 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 860 MBytes 7.21 Gbits/sec 428 410 KBytes [ 5] 1.00-2.00 sec 859 MBytes 7.20 Gbits/sec 185 563 KBytes# gigabit port root@macchiatobin:~# iperf3 -c turbo Connecting to host turbo, port 5201 [ 5] local 192.168.85.42 port 45144 connected to 192.168.85.6 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 113 MBytes 948 Mbits/sec 0 407 KBytes [ 5] 1.00-2.00 sec 112 MBytes 942 Mbits/sec 0 428 KBytes root@macchiatobin:~# ethtool -K eth2 rxhash on root@macchiatobin:~# iperf3 -c turbo iperf3: error - unable to connect to server: Resource temporarily unavailableQuoted 4 lines
I've bisected and it seems that this commit causes the issue. I tried to revert it on nex-next as a second test, but the code has changed a lot much since, generating too much conflicts.
Can you have a look into this?
Quoted 9 lines
This behaviour on eth2 is confirmed here on v5.6. Turning on rxhash appears to prevent eth2 working.
Maxime, please look into this regression, thanks.
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 10.2Mbps down 587kbps up
Hi,
What do you think about temporarily disabling it like this?
drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c +2 -1
@@ -5775,7 +5775,8 @@ static int mvpp2_port_probe(struct platform_device *pdev,NETIF_F_HW_VLAN_CTAG_FILTER ;if (mvpp22_rss_is_supported ()) { - dev->hw_features |=NETIF_F_RXHASH ; +if (port->phy_interface !=PHY_INTERFACE_MODE_SGMII ) + dev->hw_features |=NETIF_F_RXHASH ; dev->features |=NETIF_F_NTUPLE ; }
David, is this "workaround" too bad to get accepted?
Bye,
--
Matteo Croce
per aspera ad upstream
RE: [EXT] Re: [PATCH net-next 3/5] net: mvpp2: cls: Use RSS contexts to handle RSS tables
Quoted 122 lines
-----Original Message-----
From: Matteo Croce <[email protected]>
Sent: Saturday, May 9, 2020 3:13 AM
To: David S . Miller <[email protected]>
Cc: Maxime Chevallier <[email protected]>; netdev <[email protected]>; LKML <[email protected]>; Antoine Tenart <[email protected]>; Thomas Petazzoni <[email protected]>; [email protected]; [email protected]; Nadav Haklai <[email protected]>; Stefan Chulski <[email protected]>; Marcin Wojtas <[email protected]>; Linux ARM <[email protected]>; Russell King - ARM Linux admin <[email protected]>
Subject: [EXT] Re: [PATCH net-next 3/5] net: mvpp2: cls: Use RSS contexts to handle RSS tablesExternal Email
----------------------------------------------------------------------
Quoted 64 lines · Russell King - ARM Linux admin
Quoted 57 lines · Matteo Croce
Quoted 17 lines · Maxime Chevallier
The PPv2 controller has 8 RSS tables that are shared across all ports on a given PPv2 instance. The previous implementation allocated one table per port, leaving others unused.
By using RSS contexts, we can make use of multiple RSS tables per port, one being the default table (always id 0), the other ones being used as destinations for flow steering, in the same way as rx rings.
This commit introduces RSS contexts management in the PPv2 driver. We always reserve one table per port, allocated when the port is probed.
The global table list is stored in the struct mvpp2, as it's a global resource. Each port then maintains a list of indices in that global table, that way each port can have it's own numbering scheme starting from 0.
Quoted 5 lines
One limitation that seems unavoidable is that the hashing parameters are shared across all RSS contexts for a given port. Hashing parameters for ctx 0 will be applied to all contexts.
Signed-off-by Maxime Chevallier <[email protected]>Hi all,
I noticed that enabling rxhash blocks the RX on my Macchiatobin. It works fine with the 10G ports (the RX rate goes 4x up) but it completely kills the gigabit interface.
# 10G port
root@macchiatobin:~# iperf3 -c 192.168.0.2 Connecting to host 192.168.0.2, port 5201 [ 5] local 192.168.0.1 port 42394 connected to 192.168.0.2 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 941 MBytes 7.89 Gbits/sec 4030 250 KBytes [ 5] 1.00-2.00 sec 933 MBytes 7.82 Gbits/sec 4393 240 KBytes root@macchiatobin:~# ethtool -K eth0 rxhash on root@macchiatobin:~# iperf3 -c 192.168.0.2 Connecting to host 192.168.0.2, port 5201 [ 5] local 192.168.0.1 port 42398 connected to 192.168.0.2 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 860 MBytes 7.21 Gbits/sec 428 410 KBytes [ 5] 1.00-2.00 sec 859 MBytes 7.20 Gbits/sec 185 563 KBytes# gigabit port root@macchiatobin:~# iperf3 -c turbo Connecting to host turbo, port 5201 [ 5] local 192.168.85.42 port 45144 connected to 192.168.85.6 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 113 MBytes 948 Mbits/sec 0 407 KBytes [ 5] 1.00-2.00 sec 112 MBytes 942 Mbits/sec 0 428 KBytes root@macchiatobin:~# ethtool -K eth2 rxhash on root@macchiatobin:~# iperf3 -c turbo iperf3: error - unable to connect to server: Resource temporarily unavailableQuoted 4 lines
I've bisected and it seems that this commit causes the issue. I tried to revert it on nex-next as a second test, but the code has changed a lot much since, generating too much conflicts. Can you have a look into this?
Quoted 9 lines
This behaviour on eth2 is confirmed here on v5.6. Turning on rxhash appears to prevent eth2 working.
Maxime, please look into this regression, thanks.
--
RMK's Patch system:
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.armlinux.org.uk_developer_patches_&d=DwIBaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=DDQ3dK wkTIxK
Quoted 1 line
Al6_Bs7GMx4zhJArrXKN2mDMOXGh7lg&m=ntT7WKmzla65VWVPZMCr2-
8bTGq4cXdJ1RRL
Quoted 4 lines
gqFkmUc&s=jhKRohlyU0XtX0U0Rjt6XvJgMKLy_HedaFVSJwGYuD8&e= FTTC broadband for 0.8mile line in suburbia: sync at 10.2Mbps down 587kbps up
Hi,
What do you think about temporarily disabling it like this?
drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c +2
-1@@ -5775,7 +5775,8 @@ static int mvpp2_port_probe(struct platform_device *pdev,NETIF_F_HW_VLAN_CTAG_FILTER ;if (mvpp22_rss_is_supported ()) { - dev->hw_features |=NETIF_F_RXHASH ; +if (port->phy_interface !=PHY_INTERFACE_MODE_SGMII ) + dev->hw_features |=NETIF_F_RXHASH ; dev->features |=NETIF_F_NTUPLE ; }
Quoted 1 line
David, is this "workaround" too bad to get accepted?
Not sure that RSS related to physical interface(SGMII), better just remove NETIF_F_RXHASH as "workaround".
Stefan.
Russell King - ARM Linux admin
maintainerRe: [EXT] Re: [PATCH net-next 3/5] net: mvpp2: cls: Use RSS contexts to handle RSS tables
Quoted 35 lines · Stefan Chulski
Quoted 35 lines
-----Original Message-----
From: Matteo Croce <[email protected]>
Sent: Saturday, May 9, 2020 3:13 AM
To: David S . Miller <[email protected]>
Cc: Maxime Chevallier <[email protected]>; netdev <[email protected]>; LKML <[email protected]>; Antoine Tenart <[email protected]>; Thomas Petazzoni <[email protected]>; [email protected]; [email protected]; Nadav Haklai <[email protected]>; Stefan Chulski <[email protected]>; Marcin Wojtas <[email protected]>; Linux ARM <[email protected]>; Russell King - ARM Linux admin <[email protected]>
Subject: [EXT] Re: [PATCH net-next 3/5] net: mvpp2: cls: Use RSS contexts to handle RSS tablesHi,
What do you think about temporarily disabling it like this?
drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c +2
-1@@ -5775,7 +5775,8 @@ static int mvpp2_port_probe(struct platform_device *pdev,NETIF_F_HW_VLAN_CTAG_FILTER ;if (mvpp22_rss_is_supported ()) { - dev->hw_features |=NETIF_F_RXHASH ; +if (port->phy_interface !=PHY_INTERFACE_MODE_SGMII ) + dev->hw_features |=NETIF_F_RXHASH ; dev->features |=NETIF_F_NTUPLE ; }
Quoted 3 lines
Quoted 1 line
David, is this "workaround" too bad to get accepted?
Not sure that RSS related to physical interface(SGMII), better just remove
NETIF_F_RXHASHas "workaround".
Hmm, I'm not sure this is the right way forward. This patch has the effect of disabling:
d33ec4525007 ("net: mvpp2: add an RSS classification step for each flow")
but the commit you're pointing at which caused the regression is:
895586d5dc32 ("net: mvpp2: cls: Use RSS contexts to handle RSS tables")
Looking at the timeline here, it looks like Matteo raised the issue very quickly after the patch was sent on the 14th April, and despite following up on it, despite me following up on it, bootlin have remained quiet. For a regression, that's not particularly good, and doesn't leave many options but to ask davem to revert a commit, or if possible fix it (which there doesn't seem to be any willingness for either - maybe it's a feature no one uses on this platform?)
Would reverting the commit you point to as the cause (895586d5dc32) resolve the problem, and have any advantage over entirely disabling RSS?
Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTC broadband for 0.8mile line in suburbia: sync at 10.2Mbps down 587kbps up
Re: [EXT] Re: [PATCH net-next 3/5] net: mvpp2: cls: Use RSS contexts to handle RSS tables
Hello,
Quoted 4 lines · On Sat, 9 May 2020 12:45:18 +0100 Russell King - ARM Linux admin…
Looking at the timeline here, it looks like Matteo raised the issue very quickly after the patch was sent on the 14th April, and despite following up on it, despite me following up on it, bootlin have remained quiet.
Unfortunately, we are no longer actively working on Marvell platform support at the moment. We might have a look on a best effort basis, but this is potentially a non-trivial issue, so I'm not sure when we will have the chance to investigate and fix this.
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Russell King - ARM Linux admin
maintainerRe: [EXT] Re: [PATCH net-next 3/5] net: mvpp2: cls: Use RSS contexts to handle RSS tables
Quoted 10 lines · Thomas Petazzoni
Hello,
Quoted 4 lines · On Sat, 9 May 2020 12:45:18 +0100 Russell King - ARM Linux admin…
Looking at the timeline here, it looks like Matteo raised the issue very quickly after the patch was sent on the 14th April, and despite following up on it, despite me following up on it, bootlin have remained quiet.
Quoted 4 lines
Unfortunately, we are no longer actively working on Marvell platform support at the moment. We might have a look on a best effort basis, but this is potentially a non-trivial issue, so I'm not sure when we will have the chance to investigate and fix this.
That may be the case, but that doesn't excuse the fact that we have a regression and we need to do something.
Please can you suggest how we resolve this regression prior to 5.7-final?
Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTC broadband for 0.8mile line in suburbia: sync at 10.2Mbps down 587kbps up
Re: [EXT] Re: [PATCH net-next 3/5] net: mvpp2: cls: Use RSS contexts to handle RSS tables
Quoted 7 lines · On Sat, 9 May 2020 13:48:43 +0100 Russell King - ARM Linux admin…
Quoted 4 lines
Unfortunately, we are no longer actively working on Marvell platform support at the moment. We might have a look on a best effort basis, but this is potentially a non-trivial issue, so I'm not sure when we will have the chance to investigate and fix this.
That may be the case, but that doesn't excuse the fact that we have a regression and we need to do something.
Absolutely.
Quoted 2 lines
Please can you suggest how we resolve this regression prior to 5.7-final?
Since 5.7 is really close, I would suggest to disable the functionality.
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Re: [EXT] Re: [PATCH net-next 3/5] net: mvpp2: cls: Use RSS contexts to handle RSS tables
In reply to Russell King - ARM Linux adminQuoted 41 lines · Russell King - ARM Linux admin
Quoted 35 lines · Stefan Chulski
Quoted 35 lines
-----Original Message-----
From: Matteo Croce <[email protected]>
Sent: Saturday, May 9, 2020 3:13 AM
To: David S . Miller <[email protected]>
Cc: Maxime Chevallier <[email protected]>; netdev <[email protected]>; LKML <[email protected]>; Antoine Tenart <[email protected]>; Thomas Petazzoni <[email protected]>; [email protected]; [email protected]; Nadav Haklai <[email protected]>; Stefan Chulski <[email protected]>; Marcin Wojtas <[email protected]>; Linux ARM <[email protected]>; Russell King - ARM Linux admin <[email protected]>
Subject: [EXT] Re: [PATCH net-next 3/5] net: mvpp2: cls: Use RSS contexts to handle RSS tablesHi,
What do you think about temporarily disabling it like this?
drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c +2
-1@@ -5775,7 +5775,8 @@ static int mvpp2_port_probe(struct platform_device *pdev,NETIF_F_HW_VLAN_CTAG_FILTER ;if (mvpp22_rss_is_supported ()) { - dev->hw_features |=NETIF_F_RXHASH ; +if (port->phy_interface !=PHY_INTERFACE_MODE_SGMII ) + dev->hw_features |=NETIF_F_RXHASH ; dev->features |=NETIF_F_NTUPLE ; }Quoted 3 lines
Quoted 1 line
David, is this "workaround" too bad to get accepted?
Not sure that RSS related to physical interface(SGMII), better just remove
NETIF_F_RXHASHas "workaround".
Quoted 10 lines
Hmm, I'm not sure this is the right way forward. This patch has the effect of disabling:
d33ec4525007("net: mvpp2: add an RSS classification step for each flow")but the commit you're pointing at which caused the regression is:
895586d5dc32("net: mvpp2: cls: Use RSS contexts to handle RSS tables")
Hi,
When git bisect pointed to 895586d5dc32 ("net: mvpp2: cls: Use RSS contexts to handle RSS tables"), which was merged
almost an year after d33ec4525007 ("net: mvpp2: add an RSS classification step for each flow"), so I assume that between these two commits either the feature was working or it was disable and we didn't notice
Without knowing what was happening, which commit should my Fixes tag point to?
Regards,
Signature · 2 lines
Matteo Croce per aspera ad upstream
Russell King - ARM Linux admin
maintainerRe: [EXT] Re: [PATCH net-next 3/5] net: mvpp2: cls: Use RSS contexts to handle RSS tables
Quoted 56 lines · Matteo Croce
Quoted 41 lines · Russell King - ARM Linux admin
Quoted 35 lines · Stefan Chulski
Quoted 35 lines
-----Original Message-----
From: Matteo Croce <[email protected]>
Sent: Saturday, May 9, 2020 3:13 AM
To: David S . Miller <[email protected]>
Cc: Maxime Chevallier <[email protected]>; netdev <[email protected]>; LKML <[email protected]>; Antoine Tenart <[email protected]>; Thomas Petazzoni <[email protected]>; [email protected]; [email protected]; Nadav Haklai <[email protected]>; Stefan Chulski <[email protected]>; Marcin Wojtas <[email protected]>; Linux ARM <[email protected]>; Russell King - ARM Linux admin <[email protected]>
Subject: [EXT] Re: [PATCH net-next 3/5] net: mvpp2: cls: Use RSS contexts to handle RSS tablesHi,
What do you think about temporarily disabling it like this?
drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c +2
-1@@ -5775,7 +5775,8 @@ static int mvpp2_port_probe(struct platform_device *pdev,NETIF_F_HW_VLAN_CTAG_FILTER ;if (mvpp22_rss_is_supported ()) { - dev->hw_features |=NETIF_F_RXHASH ; +if (port->phy_interface !=PHY_INTERFACE_MODE_SGMII ) + dev->hw_features |=NETIF_F_RXHASH ; dev->features |=NETIF_F_NTUPLE ; }Quoted 3 lines
Quoted 1 line
David, is this "workaround" too bad to get accepted?
Not sure that RSS related to physical interface(SGMII), better just remove
NETIF_F_RXHASHas "workaround".Quoted 10 lines
Hmm, I'm not sure this is the right way forward. This patch has the effect of disabling:
d33ec4525007("net: mvpp2: add an RSS classification step for each flow")but the commit you're pointing at which caused the regression is:
895586d5dc32("net: mvpp2: cls: Use RSS contexts to handle RSS tables")Hi,
Quoted 8 lines
When git bisect pointed to
895586d5dc32("net: mvpp2: cls: Use RSS contexts to handle RSS tables"), which was merged
almost an year afterd33ec4525007("net: mvpp2: add an RSS classification step for each flow"), so I assume that between these two commits either the feature was working or it was disable and we didn't noticeWithout knowing what was happening, which commit should my Fixes tag point to?
Let me make sure that I get this clear:
- Prior to 895586d5dc32, you can turn on and off rxhash without issue on any port.
- After 895586d5dc32, turning rxhash on eth2 prevents reception.
Prior to 895586d5dc32, with rxhash on, it looks like hashing using CRC32 is supported but only one context. So, if it's possible to enable rxhash on any port on the mcbin without 895586d5dc32, and the port continues to work, I'd say the bug was introduced by 895586d5dc32.
Of course, that would be reinforced if there was a measurable difference in performance due to rxhash on each port.
Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTC broadband for 0.8mile line in suburbia: sync at 10.2Mbps down 587kbps up
Russell King - ARM Linux admin
maintainerRe: [EXT] Re: [PATCH net-next 3/5] net: mvpp2: cls: Use RSS contexts to handle RSS tables
Quoted 72 lines · Russell King - ARM Linux admin
Quoted 56 lines · Matteo Croce
Quoted 41 lines · Russell King - ARM Linux admin
Quoted 35 lines · Stefan Chulski
Quoted 35 lines
-----Original Message-----
From: Matteo Croce <[email protected]>
Sent: Saturday, May 9, 2020 3:13 AM
To: David S . Miller <[email protected]>
Cc: Maxime Chevallier <[email protected]>; netdev <[email protected]>; LKML <[email protected]>; Antoine Tenart <[email protected]>; Thomas Petazzoni <[email protected]>; [email protected]; [email protected]; Nadav Haklai <[email protected]>; Stefan Chulski <[email protected]>; Marcin Wojtas <[email protected]>; Linux ARM <[email protected]>; Russell King - ARM Linux admin <[email protected]>
Subject: [EXT] Re: [PATCH net-next 3/5] net: mvpp2: cls: Use RSS contexts to handle RSS tablesHi,
What do you think about temporarily disabling it like this?
drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c +2
-1@@ -5775,7 +5775,8 @@ static int mvpp2_port_probe(struct platform_device *pdev,NETIF_F_HW_VLAN_CTAG_FILTER ;if (mvpp22_rss_is_supported ()) { - dev->hw_features |=NETIF_F_RXHASH ; +if (port->phy_interface !=PHY_INTERFACE_MODE_SGMII ) + dev->hw_features |=NETIF_F_RXHASH ; dev->features |=NETIF_F_NTUPLE ; }Quoted 3 lines
Quoted 1 line
David, is this "workaround" too bad to get accepted?
Not sure that RSS related to physical interface(SGMII), better just remove
NETIF_F_RXHASHas "workaround".Quoted 10 lines
Hmm, I'm not sure this is the right way forward. This patch has the effect of disabling:
d33ec4525007("net: mvpp2: add an RSS classification step for each flow")but the commit you're pointing at which caused the regression is:
895586d5dc32("net: mvpp2: cls: Use RSS contexts to handle RSS tables")Hi,
Quoted 8 lines
When git bisect pointed to
895586d5dc32("net: mvpp2: cls: Use RSS contexts to handle RSS tables"), which was merged
almost an year afterd33ec4525007("net: mvpp2: add an RSS classification step for each flow"), so I assume that between these two commits either the feature was working or it was disable and we didn't noticeWithout knowing what was happening, which commit should my Fixes tag point to?
Let me make sure that I get this clear:
- Prior to
895586d5dc32, you can turn on and off rxhash without issue on any port.
- After895586d5dc32, turning rxhash on eth2 prevents reception.
Quoted 8 lines
Prior to
895586d5dc32, with rxhash on, it looks like hashing using CRC32 is supported but only one context. So, if it's possible to enable rxhash on any port on the mcbin without895586d5dc32, and the port continues to work, I'd say the bug was introduced by895586d5dc32.Of course, that would be reinforced if there was a measurable difference in performance due to rxhash on each port.
I've just run this test, but I can detect no difference in performance with or without 895586d5dc32 on eth0 or eth2 on the mcbin (apart from eth2 stopping working with 895586d5dc32 applied.) I tested this by reverting almost all changes to the mvpp2 driver between 5.6 and that commit.
That's not too surprising; I'm using my cex7 platform with the Mellanox card in for one end of the 10G link, and that platform doesn't seem to be able to saturdate a 10G link - it only seems to manage around 4Gbps.
Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTC broadband for 0.8mile line in suburbia: sync at 10.2Mbps down 587kbps up
Re: [EXT] Re: [PATCH net-next 3/5] net: mvpp2: cls: Use RSS contexts to handle RSS tables
Quoted 89 lines · Russell King - ARM Linux admin
Quoted 72 lines · Russell King - ARM Linux admin
Quoted 56 lines · Matteo Croce
Quoted 41 lines · Russell King - ARM Linux admin
Quoted 35 lines · Stefan Chulski
Quoted 35 lines
-----Original Message-----
From: Matteo Croce <[email protected]>
Sent: Saturday, May 9, 2020 3:13 AM
To: David S . Miller <[email protected]>
Cc: Maxime Chevallier <[email protected]>; netdev <[email protected]>; LKML <[email protected]>; Antoine Tenart <[email protected]>; Thomas Petazzoni <[email protected]>; [email protected]; [email protected]; Nadav Haklai <[email protected]>; Stefan Chulski <[email protected]>; Marcin Wojtas <[email protected]>; Linux ARM <[email protected]>; Russell King - ARM Linux admin <[email protected]>
Subject: [EXT] Re: [PATCH net-next 3/5] net: mvpp2: cls: Use RSS contexts to handle RSS tablesHi,
What do you think about temporarily disabling it like this?
drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c +2
-1@@ -5775,7 +5775,8 @@ static int mvpp2_port_probe(struct platform_device *pdev,NETIF_F_HW_VLAN_CTAG_FILTER ;if (mvpp22_rss_is_supported ()) { - dev->hw_features |=NETIF_F_RXHASH ; +if (port->phy_interface !=PHY_INTERFACE_MODE_SGMII ) + dev->hw_features |=NETIF_F_RXHASH ; dev->features |=NETIF_F_NTUPLE ; }Quoted 3 lines
Quoted 1 line
David, is this "workaround" too bad to get accepted?
Not sure that RSS related to physical interface(SGMII), better just remove
NETIF_F_RXHASHas "workaround".Quoted 10 lines
Hmm, I'm not sure this is the right way forward. This patch has the effect of disabling:
d33ec4525007("net: mvpp2: add an RSS classification step for each flow")but the commit you're pointing at which caused the regression is:
895586d5dc32("net: mvpp2: cls: Use RSS contexts to handle RSS tables")Hi,
Quoted 8 lines
When git bisect pointed to
895586d5dc32("net: mvpp2: cls: Use RSS contexts to handle RSS tables"), which was merged
almost an year afterd33ec4525007("net: mvpp2: add an RSS classification step for each flow"), so I assume that between these two commits either the feature was working or it was disable and we didn't noticeWithout knowing what was happening, which commit should my Fixes tag point to?
Let me make sure that I get this clear:
- Prior to
895586d5dc32, you can turn on and off rxhash without issue on any port.
- After895586d5dc32, turning rxhash on eth2 prevents reception.Quoted 8 lines
Prior to
895586d5dc32, with rxhash on, it looks like hashing using CRC32 is supported but only one context. So, if it's possible to enable rxhash on any port on the mcbin without895586d5dc32, and the port continues to work, I'd say the bug was introduced by895586d5dc32.Of course, that would be reinforced if there was a measurable difference in performance due to rxhash on each port.
I've just run this test, but I can detect no difference in performance with or without
895586d5dc32on eth0 or eth2 on the mcbin (apart from eth2 stopping working with895586d5dc32applied.) I tested this by reverting almost all changes to the mvpp2 driver between 5.6 and that commit.
Quoted 8 lines
That's not too surprising; I'm using my cex7 platform with the Mellanox card in for one end of the 10G link, and that platform doesn't seem to be able to saturdate a 10G link - it only seems to manage around 4Gbps.
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 10.2Mbps down 587kbps up
Well it depends on the traffic type. I used to generate 5k flows with T-Rex and an Intel X710 card.
This way t-rex changes the UDP port of every packet:
root@macchiatobin:~# tcpdump -tnni eth0 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes IP 16.0.0.18.9874 > 48.0.0.81.2001: UDP, length 18 IP 16.0.0.248.56289 > 48.0.192.56.2001: UDP, length 18 IP 16.0.0.154.44965 > 48.0.128.26.2001: UDP, length 18 IP 16.0.0.23.31363 > 48.0.0.86.2001: UDP, length 18 IP 16.0.0.192.1674 > 48.0.192.63.2001: UDP, length 18 IP 16.0.0.155.62370 > 48.0.128.27.2001: UDP, length 18 IP 16.0.0.30.22126 > 48.0.0.93.2001: UDP, length 18 IP 16.0.0.195.51329 > 48.0.192.66.2001: UDP, length 18 IP 16.0.0.160.18323 > 48.0.128.32.2001: UDP, length 18 IP 16.0.0.199.55413 > 48.0.192.70.2001: UDP, length 18
And here RX hash gives a huge performance gain.
root@macchiatobin:~# utraf eth0 tx: 0 bps 0 pps rx: 425.5 Mbps 886.5 Kpps tx: 0 bps 0 pps rx: 426.0 Mbps 887.6 Kpps tx: 0 bps 0 pps rx: 425.3 Mbps 886.1 Kpps tx: 0 bps 0 pps rx: 425.2 Mbps 885.8 Kpps root@macchiatobin:~# ethtool -K eth0 rxhash on root@macchiatobin:~# utraf eth0 tx: 0 bps 0 pps rx: 1595 Mbps 3323 Kpps tx: 0 bps 0 pps rx: 1593 Mbps 3319 Kpps tx: 0 bps 0 pps rx: 1595 Mbps 3323 Kpps tx: 0 bps 0 pps rx: 1594 Mbps 3320 Kpps
utraf is just a tool which reads netlink statistics, packets are dropped with a tc rule.
Regards,
Signature · 2 lines
Matteo Croce per aspera ad upstream
Russell King - ARM Linux admin
maintainerRe: [EXT] Re: [PATCH net-next 3/5] net: mvpp2: cls: Use RSS contexts to handle RSS tables
In reply to Matteo CroceQuoted 10 lines · Matteo Croce
Hi,
When git bisect pointed to
895586d5dc32("net: mvpp2: cls: Use RSS contexts to handle RSS tables"), which was merged
almost an year afterd33ec4525007("net: mvpp2: add an RSS classification step for each flow"), so I assume that between these two commits either the feature was working or it was disable and we didn't noticeWithout knowing what was happening, which commit should my Fixes tag point to?
It is highly likely that 895586d5dc32 is responsible for this breakage. I've been investigating this afternoon, and what I've found, comparing a kernel without 895586d5dc32 and with 895586d5dc32 applied is:
- The table programmed into the hardware via mvpp22_rss_fill_table() appears to be identical with or without the commit.
- When rxhash is enabled on eth2, mvpp2_rss_port_c2_enable() reports that c2.attr[0] and c2.attr[2] are written back containing:
- with 895586d5dc32, failing: 00200000 40000000
- without 895586d5dc32, working: 04000000 40000000
- When disabling rxhash, c2.attr[0] and c2.attr[2] are written back as:
04000000 00000000
The second value represents the MVPP22_CLS_C2_ATTR2_RSS_EN bit, the first value is the queue number, which comprises two fields. The high 5 bits are 24:29 and the low three are 21:23 inclusive. This comes from:
c2.attr[0] = MVPP22_CLS_C2_ATTR0_QHIGH(qh) |
MVPP22_CLS_C2_ATTR0_QLOW(ql);
#define MVPP22_CLS_C2_ATTR0_QHIGH(qh) (((qh) & 0x1f) << 24)
#define MVPP22_CLS_C2_ATTR0_QLOW(ql) (((ql) & 0x7) << 21)So, the working case gives eth2 a queue id of 4.0, or 32 as per port->first_rxq, and the non-working case a queue id of 0.1, or 1.
The allocation of queue IDs seems to be in mvpp2_port_probe():
if (priv->hw_version == MVPP21)
port->first_rxq = port->id * port->nrxqs;
else
port->first_rxq = port->id * priv->max_port_rxqs;Where:
if (priv->hw_version == MVPP21)
priv->max_port_rxqs = 8;
else
priv->max_port_rxqs = 32;Making the port 0 (eth0 / eth1) have port->first_rxq = 0, and port 1 (eth2) be 32. It seems the idea is that the first 32 queues belong to port 0, the second 32 queues belong to port 1, etc.
mvpp2_rss_port_c2_enable() gets the queue number from it's parameter, 'ctx', which comes from mvpp22_rss_ctx(port, 0). This returns port->rss_ctx[0].
mvpp22_rss_context_create() is responsible for allocating that, which it does by looking for an unallocated priv->rss_tables[] pointer. This table is shared amongst all ports on the CP silicon.
When we write the tables in mvpp22_rss_fill_table(), the RSS table entry is defined by:
u32 sel = MVPP22_RSS_INDEX_TABLE(rss_ctx) |
MVPP22_RSS_INDEX_TABLE_ENTRY(i);where rss_ctx is the context ID (queue number) and i is the index in the table.
#define MVPP22_RSS_INDEX_TABLE_ENTRY(idx) (idx) #define MVPP22_RSS_INDEX_TABLE(idx) ((idx) << 8) #define MVPP22_RSS_INDEX_QUEUE(idx) ((idx) << 16)
If we look at what is written:
- The first table to be written has "sel" values of 00000000..0000001f, containing values 0..3. This appears to be for eth1. This is table 0, RX queue number 0.
- The second table has "sel" values of 00000100..0000011f, and appears to be for eth2. These contain values 0x20..0x23. This is table 1, RX queue number 0.
- The third table has "sel" values of 00000200..0000021f, and appears to be for eth3. These contain values 0x40..0x43. This is table 2, RX queue number 0.
Okay, so how do queue numbers translate to the RSS table? There is another table - the RXQ2RSS table, indexed by the MVPP22_RSS_INDEX_QUEUE field of MVPP22_RSS_INDEX and accessed through the MVPP22_RXQ2RSS_TABLE register. Before 895586d5dc32, it was:
mvpp2_write(priv, MVPP22_RSS_INDEX,
MVPP22_RSS_INDEX_QUEUE(port->first_rxq));
mvpp2_write(priv, MVPP22_RXQ2RSS_TABLE,
MVPP22_RSS_TABLE_POINTER(port->id));and after:
mvpp2_write(priv, MVPP22_RSS_INDEX, MVPP22_RSS_INDEX_QUEUE(ctx));
mvpp2_write(priv, MVPP22_RXQ2RSS_TABLE, MVPP22_RSS_TABLE_POINTER(ctx));So, before the commit, for eth2, that would've contained '32' for the index and '1' for the table pointer - mapping queue 32 to table 1. Remember that this is queue-high.queue-low of 4.0.
After the commit, we appear to map queue 1 to table 1. That again looks fine on the face of it.
Section 9.3.1 of the A8040 manual seems indicate the reason that the queue number is separated. queue-low seems to always come from the classifier, whereas queue-high can be from the ingress physical port number or the classifier depending on the MVPP2_CLS_SWFWD_PCTRL_REG.
We set the port bit in MVPP2_CLS_SWFWD_PCTRL_REG, meaning that queue-high comes from the MVPP2_CLS_SWFWD_P2HQ_REG() register... and this seems to be where our bug comes from.
mvpp2_cls_oversize_rxq_set() sets this up as:
mvpp2_write(port->priv, MVPP2_CLS_SWFWD_P2HQ_REG(port->id),
(port->first_rxq >> MVPP2_CLS_OVERSIZE_RXQ_LOW_BITS));
val = mvpp2_read(port->priv, MVPP2_CLS_SWFWD_PCTRL_REG);
val |= MVPP2_CLS_SWFWD_PCTRL_MASK(port->id);
mvpp2_write(port->priv, MVPP2_CLS_SWFWD_PCTRL_REG, val);so, the queue-high for eth2 is _always_ 4, meaning that only queues 32 through 39 inclusive are available to eth2. Yet, we're trying to tell the classifier to set queue-high, which will be ignored, to zero.
So we end up directing traffic from eth2 not to queue 1, but to queue 33, and then we tell it to look up queue 33 in the RSS table. However, RSS table has not been programmed for queue 33, and so it ends up (presumably) dropping the packets.
It seems that mvpp22_rss_context_create() doesn't take account of the fact that the upper 5 bits of the queue ID can't actually be changed due to the settings in mvpp2_cls_oversize_rxq_set(), _or_ it seems that mvpp2_cls_oversize_rxq_set() has been missed in this commit. Either way, these two functions mutually disagree with what queue number should be used.
So, 895586d5dc32 is indeed the cause of this problem.
Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTC broadband for 0.8mile line in suburbia: sync at 10.2Mbps down 587kbps up
Russell King - ARM Linux admin
maintainerRe: [EXT] Re: [PATCH net-next 3/5] net: mvpp2: cls: Use RSS contexts to handle RSS tables
Quoted 142 lines · Russell King - ARM Linux admin
Quoted 10 lines · Matteo Croce
Hi,
When git bisect pointed to
895586d5dc32("net: mvpp2: cls: Use RSS contexts to handle RSS tables"), which was merged
almost an year afterd33ec4525007("net: mvpp2: add an RSS classification step for each flow"), so I assume that between these two commits either the feature was working or it was disable and we didn't noticeWithout knowing what was happening, which commit should my Fixes tag point to?
It is highly likely that
895586d5dc32is responsible for this breakage. I've been investigating this afternoon, and what I've found, comparing a kernel without895586d5dc32and with895586d5dc32applied is:- The table programmed into the hardware via
mvpp22_rss_fill_table()appears to be identical with or without the commit.- When rxhash is enabled on eth2,
mvpp2_rss_port_c2_enable()reports that c2.attr[0] and c2.attr[2] are written back containing:- with
895586d5dc32, failing: 00200000 40000000
- without895586d5dc32, working: 04000000 40000000- When disabling rxhash, c2.attr[0] and c2.attr[2] are written back as:
04000000 00000000The second value represents the
MVPP22_CLS_C2_ATTR2_RSS_ENbit, the first value is the queue number, which comprises two fields. The high 5 bits are 24:29 and the low three are 21:23 inclusive. This comes from:c2.attr[0] = MVPP22_CLS_C2_ATTR0_QHIGH(qh) | MVPP22_CLS_C2_ATTR0_QLOW(ql); #define MVPP22_CLS_C2_ATTR0_QHIGH(qh) (((qh) & 0x1f) << 24) #define MVPP22_CLS_C2_ATTR0_QLOW(ql) (((ql) & 0x7) << 21)So, the working case gives eth2 a queue id of 4.0, or 32 as per
port->first_rxq, and the non-working case a queue id of 0.1, or 1.The allocation of queue IDs seems to be in
mvpp2_port_probe():if (priv->hw_version == MVPP21) port->first_rxq = port->id * port->nrxqs; else port->first_rxq = port->id * priv->max_port_rxqs;Where:
if (priv->hw_version == MVPP21) priv->max_port_rxqs = 8; else priv->max_port_rxqs = 32;Making the port 0 (eth0 / eth1) have
port->first_rxq= 0, and port 1 (eth2) be 32. It seems the idea is that the first 32 queues belong to port 0, the second 32 queues belong to port 1, etc.
mvpp2_rss_port_c2_enable()gets the queue number from it's parameter, 'ctx', which comes frommvpp22_rss_ctx(port, 0). This returns port->rss_ctx[0].
mvpp22_rss_context_create()is responsible for allocating that, which it does by looking for an unallocated priv->rss_tables[] pointer. This table is shared amongst all ports on the CP silicon.When we write the tables in
mvpp22_rss_fill_table(), the RSS table entry is defined by:u32 sel = MVPP22_RSS_INDEX_TABLE(rss_ctx) | MVPP22_RSS_INDEX_TABLE_ENTRY(i);where
rss_ctxis the context ID (queue number) and i is the index in the table.#define MVPP22_RSS_INDEX_TABLE_ENTRY(idx) (idx) #define MVPP22_RSS_INDEX_TABLE(idx) ((idx) << 8) #define MVPP22_RSS_INDEX_QUEUE(idx) ((idx) << 16)If we look at what is written:
- The first table to be written has "sel" values of 00000000..0000001f, containing values 0..3. This appears to be for eth1. This is table 0, RX queue number 0.
- The second table has "sel" values of 00000100..0000011f, and appears to be for eth2. These contain values 0x20..0x23. This is table 1, RX queue number 0.
- The third table has "sel" values of 00000200..0000021f, and appears to be for eth3. These contain values 0x40..0x43. This is table 2, RX queue number 0.Okay, so how do queue numbers translate to the RSS table? There is another table - the RXQ2RSS table, indexed by the
MVPP22_RSS_INDEX_QUEUEfield ofMVPP22_RSS_INDEXand accessed through theMVPP22_RXQ2RSS_TABLEregister. Before895586d5dc32, it was:mvpp2_write(priv, MVPP22_RSS_INDEX, MVPP22_RSS_INDEX_QUEUE(port->first_rxq)); mvpp2_write(priv, MVPP22_RXQ2RSS_TABLE, MVPP22_RSS_TABLE_POINTER(port->id));and after:
mvpp2_write(priv, MVPP22_RSS_INDEX, MVPP22_RSS_INDEX_QUEUE(ctx)); mvpp2_write(priv, MVPP22_RXQ2RSS_TABLE, MVPP22_RSS_TABLE_POINTER(ctx));So, before the commit, for eth2, that would've contained '32' for the index and '1' for the table pointer - mapping queue 32 to table 1. Remember that this is queue-high.queue-low of 4.0.
After the commit, we appear to map queue 1 to table 1. That again looks fine on the face of it.
Section 9.3.1 of the A8040 manual seems indicate the reason that the queue number is separated. queue-low seems to always come from the classifier, whereas queue-high can be from the ingress physical port number or the classifier depending on the
MVPP2_CLS_SWFWD_PCTRL_REG.We set the port bit in
MVPP2_CLS_SWFWD_PCTRL_REG, meaning that queue-high comes from theMVPP2_CLS_SWFWD_P2HQ_REG()register... and this seems to be where our bug comes from.
mvpp2_cls_oversize_rxq_set()sets this up as:mvpp2_write(port->priv, MVPP2_CLS_SWFWD_P2HQ_REG(port->id), (port->first_rxq >> MVPP2_CLS_OVERSIZE_RXQ_LOW_BITS)); val = mvpp2_read(port->priv, MVPP2_CLS_SWFWD_PCTRL_REG); val |= MVPP2_CLS_SWFWD_PCTRL_MASK(port->id); mvpp2_write(port->priv, MVPP2_CLS_SWFWD_PCTRL_REG, val);so, the queue-high for eth2 is _always_ 4, meaning that only queues 32 through 39 inclusive are available to eth2. Yet, we're trying to tell the classifier to set queue-high, which will be ignored, to zero.
So we end up directing traffic from eth2 not to queue 1, but to queue 33, and then we tell it to look up queue 33 in the RSS table. However, RSS table has not been programmed for queue 33, and so it ends up (presumably) dropping the packets.
Quoted 8 lines
It seems that
mvpp22_rss_context_create()doesn't take account of the fact that the upper 5 bits of the queue ID can't actually be changed due to the settings inmvpp2_cls_oversize_rxq_set(), _or_ it seems thatmvpp2_cls_oversize_rxq_set()has been missed in this commit. Either way, these two functions mutually disagree with what queue number should be used.So,
895586d5dc32is indeed the cause of this problem.
Looking deeper into what mvpp2_cls_oversize_rxq_set() and the MTU validation is doing, it seems that MVPP2_CLS_SWFWD_P2HQ_REG() is used for at least a couple of things.
So, with the classifier having had RSS enabled and directing eth2 traffic to queue 1, we can not ignore the fact that we may have packets appearing on queue 32 for this port.
One of the things that queue 32 will be used for is if an over-sized packet attempts to egress through eth2 - it seems that the A8040 has the ability to forward frames between its ports. However, afaik we don't support that feature, and the kernel restricts the packet size, so we should never violate the MTU validator and end up with such a packet. In any case, _if_ we were to attempt to transmit an oversized packet, we have no support in the kernel to deal with that appearing in the port's receive queue.
Maybe it would be safe to clear the MVPP2_CLS_SWFWD_PCTRL_MASK() bit?
My testing seems to confirm my findings above - clearing this bit means that if I enable rxhash on eth2, the interface can then pass traffic, as we are now directing traffic to RX queue 1 rather than queue 33. Traffic still seems to work with rxhash off as well.
So, I think it's clear where the problem lies, but not what the correct solution is; someone with more experience of packet classifiers (this one?) needs to look at this - this is my first venture into these things, and certainly the first time I've traced through how this is trying to work (or not)...
Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTC broadband for 0.8mile line in suburbia: sync at 10.2Mbps down 587kbps up
Russell King - ARM Linux admin
maintainerRe: [EXT] Re: [PATCH net-next 3/5] net: mvpp2: cls: Use RSS contexts to handle RSS tables
Quoted 164 lines · Russell King - ARM Linux admin
Quoted 130 lines · Russell King - ARM Linux admin
It is highly likely that
895586d5dc32is responsible for this breakage. I've been investigating this afternoon, and what I've found, comparing a kernel without895586d5dc32and with895586d5dc32applied is:- The table programmed into the hardware via
mvpp22_rss_fill_table()appears to be identical with or without the commit.- When rxhash is enabled on eth2,
mvpp2_rss_port_c2_enable()reports that c2.attr[0] and c2.attr[2] are written back containing:- with
895586d5dc32, failing: 00200000 40000000
- without895586d5dc32, working: 04000000 40000000- When disabling rxhash, c2.attr[0] and c2.attr[2] are written back as:
04000000 00000000The second value represents the
MVPP22_CLS_C2_ATTR2_RSS_ENbit, the first value is the queue number, which comprises two fields. The high 5 bits are 24:29 and the low three are 21:23 inclusive. This comes from:c2.attr[0] = MVPP22_CLS_C2_ATTR0_QHIGH(qh) | MVPP22_CLS_C2_ATTR0_QLOW(ql); #define MVPP22_CLS_C2_ATTR0_QHIGH(qh) (((qh) & 0x1f) << 24) #define MVPP22_CLS_C2_ATTR0_QLOW(ql) (((ql) & 0x7) << 21)So, the working case gives eth2 a queue id of 4.0, or 32 as per
port->first_rxq, and the non-working case a queue id of 0.1, or 1.The allocation of queue IDs seems to be in
mvpp2_port_probe():if (priv->hw_version == MVPP21) port->first_rxq = port->id * port->nrxqs; else port->first_rxq = port->id * priv->max_port_rxqs;Where:
if (priv->hw_version == MVPP21) priv->max_port_rxqs = 8; else priv->max_port_rxqs = 32;Making the port 0 (eth0 / eth1) have
port->first_rxq= 0, and port 1 (eth2) be 32. It seems the idea is that the first 32 queues belong to port 0, the second 32 queues belong to port 1, etc.
mvpp2_rss_port_c2_enable()gets the queue number from it's parameter, 'ctx', which comes frommvpp22_rss_ctx(port, 0). This returns port->rss_ctx[0].
mvpp22_rss_context_create()is responsible for allocating that, which it does by looking for an unallocated priv->rss_tables[] pointer. This table is shared amongst all ports on the CP silicon.When we write the tables in
mvpp22_rss_fill_table(), the RSS table entry is defined by:u32 sel = MVPP22_RSS_INDEX_TABLE(rss_ctx) | MVPP22_RSS_INDEX_TABLE_ENTRY(i);where
rss_ctxis the context ID (queue number) and i is the index in the table.#define MVPP22_RSS_INDEX_TABLE_ENTRY(idx) (idx) #define MVPP22_RSS_INDEX_TABLE(idx) ((idx) << 8) #define MVPP22_RSS_INDEX_QUEUE(idx) ((idx) << 16)If we look at what is written:
- The first table to be written has "sel" values of 00000000..0000001f, containing values 0..3. This appears to be for eth1. This is table 0, RX queue number 0.
- The second table has "sel" values of 00000100..0000011f, and appears to be for eth2. These contain values 0x20..0x23. This is table 1, RX queue number 0.
- The third table has "sel" values of 00000200..0000021f, and appears to be for eth3. These contain values 0x40..0x43. This is table 2, RX queue number 0.Okay, so how do queue numbers translate to the RSS table? There is another table - the RXQ2RSS table, indexed by the
MVPP22_RSS_INDEX_QUEUEfield ofMVPP22_RSS_INDEXand accessed through theMVPP22_RXQ2RSS_TABLEregister. Before895586d5dc32, it was:mvpp2_write(priv, MVPP22_RSS_INDEX, MVPP22_RSS_INDEX_QUEUE(port->first_rxq)); mvpp2_write(priv, MVPP22_RXQ2RSS_TABLE, MVPP22_RSS_TABLE_POINTER(port->id));and after:
mvpp2_write(priv, MVPP22_RSS_INDEX, MVPP22_RSS_INDEX_QUEUE(ctx)); mvpp2_write(priv, MVPP22_RXQ2RSS_TABLE, MVPP22_RSS_TABLE_POINTER(ctx));So, before the commit, for eth2, that would've contained '32' for the index and '1' for the table pointer - mapping queue 32 to table 1. Remember that this is queue-high.queue-low of 4.0.
After the commit, we appear to map queue 1 to table 1. That again looks fine on the face of it.
Section 9.3.1 of the A8040 manual seems indicate the reason that the queue number is separated. queue-low seems to always come from the classifier, whereas queue-high can be from the ingress physical port number or the classifier depending on the
MVPP2_CLS_SWFWD_PCTRL_REG.We set the port bit in
MVPP2_CLS_SWFWD_PCTRL_REG, meaning that queue-high comes from theMVPP2_CLS_SWFWD_P2HQ_REG()register... and this seems to be where our bug comes from.
mvpp2_cls_oversize_rxq_set()sets this up as:mvpp2_write(port->priv, MVPP2_CLS_SWFWD_P2HQ_REG(port->id), (port->first_rxq >> MVPP2_CLS_OVERSIZE_RXQ_LOW_BITS)); val = mvpp2_read(port->priv, MVPP2_CLS_SWFWD_PCTRL_REG); val |= MVPP2_CLS_SWFWD_PCTRL_MASK(port->id); mvpp2_write(port->priv, MVPP2_CLS_SWFWD_PCTRL_REG, val);so, the queue-high for eth2 is _always_ 4, meaning that only queues 32 through 39 inclusive are available to eth2. Yet, we're trying to tell the classifier to set queue-high, which will be ignored, to zero.
So we end up directing traffic from eth2 not to queue 1, but to queue 33, and then we tell it to look up queue 33 in the RSS table. However, RSS table has not been programmed for queue 33, and so it ends up (presumably) dropping the packets.
Quoted 8 lines
It seems that
mvpp22_rss_context_create()doesn't take account of the fact that the upper 5 bits of the queue ID can't actually be changed due to the settings inmvpp2_cls_oversize_rxq_set(), _or_ it seems thatmvpp2_cls_oversize_rxq_set()has been missed in this commit. Either way, these two functions mutually disagree with what queue number should be used.So,
895586d5dc32is indeed the cause of this problem.Looking deeper into what
mvpp2_cls_oversize_rxq_set()and the MTU validation is doing, it seems thatMVPP2_CLS_SWFWD_P2HQ_REG()is used for at least a couple of things.So, with the classifier having had RSS enabled and directing eth2 traffic to queue 1, we can not ignore the fact that we may have packets appearing on queue 32 for this port.
One of the things that queue 32 will be used for is if an over-sized packet attempts to egress through eth2 - it seems that the A8040 has the ability to forward frames between its ports. However, afaik we don't support that feature, and the kernel restricts the packet size, so we should never violate the MTU validator and end up with such a packet. In any case, _if_ we were to attempt to transmit an oversized packet, we have no support in the kernel to deal with that appearing in the port's receive queue.
Maybe it would be safe to clear the
MVPP2_CLS_SWFWD_PCTRL_MASK()bit?My testing seems to confirm my findings above - clearing this bit means that if I enable rxhash on eth2, the interface can then pass traffic, as we are now directing traffic to RX queue 1 rather than queue 33. Traffic still seems to work with rxhash off as well.
Quoted 5 lines
So, I think it's clear where the problem lies, but not what the correct solution is; someone with more experience of packet classifiers (this one?) needs to look at this - this is my first venture into these things, and certainly the first time I've traced through how this is trying to work (or not)...
This is what I was using here to work around the problem, and what I mentioned above.
drivers/net/ethernet/marvell/mvpp2/mvpp2_cls.c +1 -1
@@ -1058,7 +1058,7 @@ void mvpp2_cls_oversize_rxq_set(struct mvpp2_port *port) (port->first_rxq >> MVPP2_CLS_OVERSIZE_RXQ_LOW_BITS)); val =mvpp2_read (port->priv,MVPP2_CLS_SWFWD_PCTRL_REG ); - val |=MVPP2_CLS_SWFWD_PCTRL_MASK (port->id); + val &= ~MVPP2_CLS_SWFWD_PCTRL_MASK (port->id);mvpp2_write (port->priv,MVPP2_CLS_SWFWD_PCTRL_REG , val); }
Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTC for 0.8m (est. 1762m) line in suburbia: sync at 13.1Mbps down 424kbps up
Re: [EXT] Re: [PATCH net-next 3/5] net: mvpp2: cls: Use RSS contexts to handle RSS tables
Quoted 181 lines · Russell King - ARM Linux admin
Quoted 164 lines · Russell King - ARM Linux admin
Quoted 130 lines · Russell King - ARM Linux admin
It is highly likely that
895586d5dc32is responsible for this breakage. I've been investigating this afternoon, and what I've found, comparing a kernel without895586d5dc32and with895586d5dc32applied is:- The table programmed into the hardware via
mvpp22_rss_fill_table()appears to be identical with or without the commit.- When rxhash is enabled on eth2,
mvpp2_rss_port_c2_enable()reports that c2.attr[0] and c2.attr[2] are written back containing:- with
895586d5dc32, failing: 00200000 40000000
- without895586d5dc32, working: 04000000 40000000- When disabling rxhash, c2.attr[0] and c2.attr[2] are written back as:
04000000 00000000The second value represents the
MVPP22_CLS_C2_ATTR2_RSS_ENbit, the first value is the queue number, which comprises two fields. The high 5 bits are 24:29 and the low three are 21:23 inclusive. This comes from:c2.attr[0] = MVPP22_CLS_C2_ATTR0_QHIGH(qh) | MVPP22_CLS_C2_ATTR0_QLOW(ql); #define MVPP22_CLS_C2_ATTR0_QHIGH(qh) (((qh) & 0x1f) << 24) #define MVPP22_CLS_C2_ATTR0_QLOW(ql) (((ql) & 0x7) << 21)So, the working case gives eth2 a queue id of 4.0, or 32 as per
port->first_rxq, and the non-working case a queue id of 0.1, or 1.The allocation of queue IDs seems to be in
mvpp2_port_probe():if (priv->hw_version == MVPP21) port->first_rxq = port->id * port->nrxqs; else port->first_rxq = port->id * priv->max_port_rxqs;Where:
if (priv->hw_version == MVPP21) priv->max_port_rxqs = 8; else priv->max_port_rxqs = 32;Making the port 0 (eth0 / eth1) have
port->first_rxq= 0, and port 1 (eth2) be 32. It seems the idea is that the first 32 queues belong to port 0, the second 32 queues belong to port 1, etc.
mvpp2_rss_port_c2_enable()gets the queue number from it's parameter, 'ctx', which comes frommvpp22_rss_ctx(port, 0). This returns port->rss_ctx[0].
mvpp22_rss_context_create()is responsible for allocating that, which it does by looking for an unallocated priv->rss_tables[] pointer. This table is shared amongst all ports on the CP silicon.When we write the tables in
mvpp22_rss_fill_table(), the RSS table entry is defined by:u32 sel = MVPP22_RSS_INDEX_TABLE(rss_ctx) | MVPP22_RSS_INDEX_TABLE_ENTRY(i);where
rss_ctxis the context ID (queue number) and i is the index in the table.#define MVPP22_RSS_INDEX_TABLE_ENTRY(idx) (idx) #define MVPP22_RSS_INDEX_TABLE(idx) ((idx) << 8) #define MVPP22_RSS_INDEX_QUEUE(idx) ((idx) << 16)If we look at what is written:
- The first table to be written has "sel" values of 00000000..0000001f, containing values 0..3. This appears to be for eth1. This is table 0, RX queue number 0.
- The second table has "sel" values of 00000100..0000011f, and appears to be for eth2. These contain values 0x20..0x23. This is table 1, RX queue number 0.
- The third table has "sel" values of 00000200..0000021f, and appears to be for eth3. These contain values 0x40..0x43. This is table 2, RX queue number 0.Okay, so how do queue numbers translate to the RSS table? There is another table - the RXQ2RSS table, indexed by the
MVPP22_RSS_INDEX_QUEUEfield ofMVPP22_RSS_INDEXand accessed through theMVPP22_RXQ2RSS_TABLEregister. Before895586d5dc32, it was:mvpp2_write(priv, MVPP22_RSS_INDEX, MVPP22_RSS_INDEX_QUEUE(port->first_rxq)); mvpp2_write(priv, MVPP22_RXQ2RSS_TABLE, MVPP22_RSS_TABLE_POINTER(port->id));and after:
mvpp2_write(priv, MVPP22_RSS_INDEX, MVPP22_RSS_INDEX_QUEUE(ctx)); mvpp2_write(priv, MVPP22_RXQ2RSS_TABLE, MVPP22_RSS_TABLE_POINTER(ctx));So, before the commit, for eth2, that would've contained '32' for the index and '1' for the table pointer - mapping queue 32 to table 1. Remember that this is queue-high.queue-low of 4.0.
After the commit, we appear to map queue 1 to table 1. That again looks fine on the face of it.
Section 9.3.1 of the A8040 manual seems indicate the reason that the queue number is separated. queue-low seems to always come from the classifier, whereas queue-high can be from the ingress physical port number or the classifier depending on the
MVPP2_CLS_SWFWD_PCTRL_REG.We set the port bit in
MVPP2_CLS_SWFWD_PCTRL_REG, meaning that queue-high comes from theMVPP2_CLS_SWFWD_P2HQ_REG()register... and this seems to be where our bug comes from.
mvpp2_cls_oversize_rxq_set()sets this up as:mvpp2_write(port->priv, MVPP2_CLS_SWFWD_P2HQ_REG(port->id), (port->first_rxq >> MVPP2_CLS_OVERSIZE_RXQ_LOW_BITS)); val = mvpp2_read(port->priv, MVPP2_CLS_SWFWD_PCTRL_REG); val |= MVPP2_CLS_SWFWD_PCTRL_MASK(port->id); mvpp2_write(port->priv, MVPP2_CLS_SWFWD_PCTRL_REG, val);so, the queue-high for eth2 is _always_ 4, meaning that only queues 32 through 39 inclusive are available to eth2. Yet, we're trying to tell the classifier to set queue-high, which will be ignored, to zero.
So we end up directing traffic from eth2 not to queue 1, but to queue 33, and then we tell it to look up queue 33 in the RSS table. However, RSS table has not been programmed for queue 33, and so it ends up (presumably) dropping the packets.
Quoted 8 lines
It seems that
mvpp22_rss_context_create()doesn't take account of the fact that the upper 5 bits of the queue ID can't actually be changed due to the settings inmvpp2_cls_oversize_rxq_set(), _or_ it seems thatmvpp2_cls_oversize_rxq_set()has been missed in this commit. Either way, these two functions mutually disagree with what queue number should be used.So,
895586d5dc32is indeed the cause of this problem.Looking deeper into what
mvpp2_cls_oversize_rxq_set()and the MTU validation is doing, it seems thatMVPP2_CLS_SWFWD_P2HQ_REG()is used for at least a couple of things.So, with the classifier having had RSS enabled and directing eth2 traffic to queue 1, we can not ignore the fact that we may have packets appearing on queue 32 for this port.
One of the things that queue 32 will be used for is if an over-sized packet attempts to egress through eth2 - it seems that the A8040 has the ability to forward frames between its ports. However, afaik we don't support that feature, and the kernel restricts the packet size, so we should never violate the MTU validator and end up with such a packet. In any case, _if_ we were to attempt to transmit an oversized packet, we have no support in the kernel to deal with that appearing in the port's receive queue.
Maybe it would be safe to clear the
MVPP2_CLS_SWFWD_PCTRL_MASK()bit?My testing seems to confirm my findings above - clearing this bit means that if I enable rxhash on eth2, the interface can then pass traffic, as we are now directing traffic to RX queue 1 rather than queue 33. Traffic still seems to work with rxhash off as well.
Quoted 5 lines
So, I think it's clear where the problem lies, but not what the correct solution is; someone with more experience of packet classifiers (this one?) needs to look at this - this is my first venture into these things, and certainly the first time I've traced through how this is trying to work (or not)...
This is what I was using here to work around the problem, and what I mentioned above.
drivers/net/ethernet/marvell/mvpp2/mvpp2_cls.c +0
-0
drivers/net/ethernet/marvell/mvpp2/mvpp2_cls.c void mvpp2_cls_oversize_rxq_set(struct mvpp2_port *port) · line 1064
Quoted 8 lines
drivers/net/ethernet/marvell/mvpp2/mvpp2_cls.c +1
-1@@ -1058,7 +1058,7 @@ void mvpp2_cls_oversize_rxq_set(struct mvpp2_port *port) (port->first_rxq >> MVPP2_CLS_OVERSIZE_RXQ_LOW_BITS)); val =mvpp2_read (port->priv,MVPP2_CLS_SWFWD_PCTRL_REG ); - val |=MVPP2_CLS_SWFWD_PCTRL_MASK (port->id); + val &= ~MVPP2_CLS_SWFWD_PCTRL_MASK (port->id);mvpp2_write (port->priv,MVPP2_CLS_SWFWD_PCTRL_REG , val); }
Hi,
I will try this change and let you know if it works.
Thanks
Signature · 2 lines
Matteo Croce per aspera ad upstream
Re: [EXT] Re: [PATCH net-next 3/5] net: mvpp2: cls: Use RSS contexts to handle RSS tables
Quoted 214 lines · On Tue, 19 May 2020 12:05:20 +0200 Matteo Croce <[email protected]> wrote:
Quoted 205 lines · Russell King - ARM Linux admin
Quoted 182 lines · Russell King - ARM Linux admin
Quoted 152 lines · Russell King - ARM Linux admin
It is highly likely that
895586d5dc32is responsible for this breakage. I've been investigating this afternoon, and what I've found, comparing a kernel without895586d5dc32and with895586d5dc32applied is:- The table programmed into the hardware via
mvpp22_rss_fill_table()appears to be identical with or without the commit.- When rxhash is enabled on eth2,
mvpp2_rss_port_c2_enable()reports that c2.attr[0] and c2.attr[2] are written back containing:- with
895586d5dc32, failing: 00200000 40000000
- without895586d5dc32, working: 04000000 40000000- When disabling rxhash, c2.attr[0] and c2.attr[2] are written back as:
04000000 00000000The second value represents the
MVPP22_CLS_C2_ATTR2_RSS_ENbit, the first value is the queue number, which comprises two fields. The high 5 bits are 24:29 and the low three are 21:23 inclusive. This comes from:c2.attr[0] = MVPP22_CLS_C2_ATTR0_QHIGH(qh) | MVPP22_CLS_C2_ATTR0_QLOW(ql);#define
MVPP22_CLS_C2_ATTR0_QHIGH(qh)(((qh) &0x1f)
<< 24) #defineMVPP22_CLS_C2_ATTR0_QLOW(ql)(((ql) &0x7) << 21)So, the working case gives eth2 a queue id of 4.0, or 32 as per
port->first_rxq, and the non-working case a queue id of 0.1, or 1.The allocation of queue IDs seems to be in
mvpp2_port_probe():if (priv->hw_version == MVPP21) port->first_rxq = port->id * port->nrxqs; else port->first_rxq = port->id *
priv->max_port_rxqs;Where:
if (priv->hw_version == MVPP21) priv->max_port_rxqs = 8; else priv->max_port_rxqs = 32;Making the port 0 (eth0 / eth1) have
port->first_rxq= 0, and port 1 (eth2) be 32. It seems the idea is that the first 32 queues belong to port 0, the second 32 queues belong to port 1, etc.
mvpp2_rss_port_c2_enable()gets the queue number from it's parameter, 'ctx', which comes frommvpp22_rss_ctx(port, 0). This returns port->rss_ctx[0].
mvpp22_rss_context_create()is responsible for allocating that, which it does by looking for an unallocated priv->rss_tables[] pointer. This table is shared amongst all ports on the CP silicon.When we write the tables in
mvpp22_rss_fill_table(), the RSS table entry is defined by:u32 sel = MVPP22_RSS_INDEX_TABLE(rss_ctx) | MVPP22_RSS_INDEX_TABLE_ENTRY(i);where
rss_ctxis the context ID (queue number) and i is the index in the table.#define MVPP22_RSS_INDEX_TABLE_ENTRY(idx) (idx) #define MVPP22_RSS_INDEX_TABLE(idx) ((idx) << 8) #define MVPP22_RSS_INDEX_QUEUE(idx) ((idx) << 16)If we look at what is written:
- The first table to be written has "sel" values of 00000000..0000001f, containing values 0..3. This appears to be for eth1. This is table 0, RX queue number 0.
- The second table has "sel" values of 00000100..0000011f, and appears to be for eth2. These contain values 0x20..0x23. This is table 1, RX queue number 0.
- The third table has "sel" values of 00000200..0000021f, and appears to be for eth3. These contain values 0x40..0x43. This is table 2, RX queue number 0.Okay, so how do queue numbers translate to the RSS table? There is another table - the RXQ2RSS table, indexed by the
MVPP22_RSS_INDEX_QUEUEfield ofMVPP22_RSS_INDEXand accessed through theMVPP22_RXQ2RSS_TABLEregister. Before895586d5dc32, it was:mvpp2_write(priv, MVPP22_RSS_INDEX, MVPP22_RSS_INDEX_QUEUE(port->first_rxq)); mvpp2_write(priv, MVPP22_RXQ2RSS_TABLE, MVPP22_RSS_TABLE_POINTER(port->id));and after:
mvpp2_write(priv, MVPP22_RSS_INDEX,
MVPP22_RSS_INDEX_QUEUE(ctx));mvpp2_write(priv, MVPP22_RXQ2RSS_TABLE, MVPP22_RSS_TABLE_POINTER(ctx));So, before the commit, for eth2, that would've contained '32' for the index and '1' for the table pointer - mapping queue 32 to table 1. Remember that this is queue-high.queue-low of 4.0.
After the commit, we appear to map queue 1 to table 1. That again looks fine on the face of it.
Section 9.3.1 of the A8040 manual seems indicate the reason that the queue number is separated. queue-low seems to always come from the classifier, whereas queue-high can be from the ingress physical port number or the classifier depending on the
MVPP2_CLS_SWFWD_PCTRL_REG.We set the port bit in
MVPP2_CLS_SWFWD_PCTRL_REG, meaning that queue-high comes from theMVPP2_CLS_SWFWD_P2HQ_REG()register... and this seems to be where our bug comes from.
mvpp2_cls_oversize_rxq_set()sets this up as:mvpp2_write(port->priv,
MVPP2_CLS_SWFWD_P2HQ_REG(port->id), (port->first_rxq>>MVPP2_CLS_OVERSIZE_RXQ_LOW_BITS));val = mvpp2_read(port->priv, MVPP2_CLS_SWFWD_PCTRL_REG); val |= MVPP2_CLS_SWFWD_PCTRL_MASK(port->id); mvpp2_write(port->priv, MVPP2_CLS_SWFWD_PCTRL_REG, val);so, the queue-high for eth2 is _always_ 4, meaning that only queues 32 through 39 inclusive are available to eth2. Yet, we're trying to tell the classifier to set queue-high, which will be ignored, to zero.
So we end up directing traffic from eth2 not to queue 1, but to queue 33, and then we tell it to look up queue 33 in the RSS table. However, RSS table has not been programmed for queue 33, and so it ends up (presumably) dropping the packets.
It seems that
mvpp22_rss_context_create()doesn't take account of the fact that the upper 5 bits of the queue ID can't actually be changed due to the settings inmvpp2_cls_oversize_rxq_set(), _or_ it seems thatmvpp2_cls_oversize_rxq_set()has been missed in this commit. Either way, these two functions mutually disagree with what queue number should be used.Quoted 1 line
So,
895586d5dc32is indeed the cause of this problem.Looking deeper into what
mvpp2_cls_oversize_rxq_set()and the MTU validation is doing, it seems thatMVPP2_CLS_SWFWD_P2HQ_REG()is used for at least a couple of things.So, with the classifier having had RSS enabled and directing eth2 traffic to queue 1, we can not ignore the fact that we may have packets appearing on queue 32 for this port.
One of the things that queue 32 will be used for is if an over-sized packet attempts to egress through eth2 - it seems that the A8040 has the ability to forward frames between its ports. However, afaik we don't support that feature, and the kernel restricts the packet size, so we should never violate the MTU validator and end up with such a packet. In any case, _if_ we were to attempt to transmit an oversized packet, we have no support in the kernel to deal with that appearing in the port's receive queue.
Maybe it would be safe to clear the
MVPP2_CLS_SWFWD_PCTRL_MASK()bit?My testing seems to confirm my findings above - clearing this bit means that if I enable rxhash on eth2, the interface can then pass traffic, as we are now directing traffic to RX queue 1 rather than queue 33. Traffic still seems to work with rxhash off as well.
Quoted 5 lines
So, I think it's clear where the problem lies, but not what the correct solution is; someone with more experience of packet classifiers (this one?) needs to look at this - this is my first venture into these things, and certainly the first time I've traced through how this is trying to work (or not)...
This is what I was using here to work around the problem, and what I mentioned above.
(quoted diff) +0
-0
b/drivers/net/ethernet/marvell/mvpp2/mvpp2_cls.cindexfd221d88811e..0dd3b65822dd100644 ---a/drivers/net/ethernet/marvell/mvpp2/mvpp2_cls.c+++b/drivers/net/ethernet/marvell/mvpp2/mvpp2_cls.c@@ -1058,7 +1058,7 @@ voidmvpp2_cls_oversize_rxq_set(struct mvpp2_port *port)(port->first_rxq>>MVPP2_CLS_OVERSIZE_RXQ_LOW_BITS));val = mvpp2_read(port->priv, MVPP2_CLS_SWFWD_PCTRL_REG);drivers/net/ethernet/marvell/mvpp2/mvpp2_cls.c void mvpp2_cls_oversize_rxq_set(struct mvpp2_port *port) · line 1063
Quoted 4 lines
drivers/net/ethernet/marvell/mvpp2/mvpp2_cls.c +1
-1- val |=MVPP2_CLS_SWFWD_PCTRL_MASK (port->id); + val &= ~MVPP2_CLS_SWFWD_PCTRL_MASK (port->id);mvpp2_write (port->priv,MVPP2_CLS_SWFWD_PCTRL_REG , val); }Hi,
Quoted 4 lines
I will try this change and let you know if it works.
Thanks
Hi,
The patch seems to work. I'm generating traffic with random MAC and IP addresses, to have many flows:
# tcpdump -tenni eth2 9a:a9:b1:3a:b1:6b > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.4.0 > 192.168.0.4.0: UDP, length 12 9e:92:fd:f8:7f:0a > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.4.0 > 192.168.0.4.0: UDP, length 12 66:b7:11:8a:c2:1f > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.1.0 > 192.168.0.1.0: UDP, length 12 7a:ba:58:bd:9a:62 > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.1.0 > 192.168.0.1.0: UDP, length 12 7e:78:a9:97:70:3a > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.2.0 > 192.168.0.2.0: UDP, length 12 b2:81:91:34:ce:42 > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.2.0 > 192.168.0.2.0: UDP, length 12 2a:05:52:d0:d9:3f > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.3.0 > 192.168.0.3.0: UDP, length 12 ee:ee:47:35:fa:81 > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.3.0 > 192.168.0.3.0: UDP, length 12
This is the default rate, with rxhash off:
# utraf eth2 tx: 0 bps 0 pps rx: 397.4 Mbps 827.9 Kpps tx: 0 bps 0 pps rx: 396.3 Mbps 825.7 Kpps tx: 0 bps 0 pps rx: 396.6 Mbps 826.3 Kpps tx: 0 bps 0 pps rx: 396.5 Mbps 826.1 Kpps
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9 root 20 0 0 0 0 R 99.7 0.0 7:02.58 ksoftirqd/0
15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1
20 root 20 0 0 0 0 S 0.0 0.0 2:01.48 ksoftirqd/2
25 root 20 0 0 0 0 S 0.0 0.0 0:32.86 ksoftirqd/3and this with rx hashing enabled:
# ethtool -K eth2 rxhash on # utraf eth2 tx: 0 bps 0 pps rx: 456.4 Mbps 950.8 Kpps tx: 0 bps 0 pps rx: 458.4 Mbps 955.0 Kpps tx: 0 bps 0 pps rx: 457.6 Mbps 953.3 Kpps tx: 0 bps 0 pps rx: 462.2 Mbps 962.9 Kpps
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20 root 20 0 0 0 0 R 0.7 0.0 2:02.34 ksoftirqd/2
25 root 20 0 0 0 0 S 0.3 0.0 0:33.25 ksoftirqd/3
9 root 20 0 0 0 0 S 0.0 0.0 7:52.57 ksoftirqd/0
15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1The throughput doesn't increase so much, maybe we hit an HW limit of the gigabit port. The interesting thing is how the global CPU usage drops from 25% to 1%.
I can't explain this, it could be due to the reduced contention?
Regards,
Signature · 2 lines
Matteo Croce per aspera ad upstream
Russell King - ARM Linux admin
maintainerRe: [EXT] Re: [PATCH net-next 3/5] net: mvpp2: cls: Use RSS contexts to handle RSS tables
Quoted 48 lines · Matteo Croce
On Tue, 19 May 2020 12:05:20 +0200
Matteo Croce <[email protected]> wrote:Hi,
The patch seems to work. I'm generating traffic with random MAC and IP addresses, to have many flows:
# tcpdump -tenni eth2 9a:a9:b1:3a:b1:6b > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.4.0 > 192.168.0.4.0: UDP, length 12 9e:92:fd:f8:7f:0a > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.4.0 > 192.168.0.4.0: UDP, length 12 66:b7:11:8a:c2:1f > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.1.0 > 192.168.0.1.0: UDP, length 12 7a:ba:58:bd:9a:62 > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.1.0 > 192.168.0.1.0: UDP, length 12 7e:78:a9:97:70:3a > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.2.0 > 192.168.0.2.0: UDP, length 12 b2:81:91:34:ce:42 > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.2.0 > 192.168.0.2.0: UDP, length 12 2a:05:52:d0:d9:3f > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.3.0 > 192.168.0.3.0: UDP, length 12 ee:ee:47:35:fa:81 > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.3.0 > 192.168.0.3.0: UDP, length 12This is the default rate, with rxhash off:
# utraf eth2 tx: 0 bps 0 pps rx: 397.4 Mbps 827.9 Kpps tx: 0 bps 0 pps rx: 396.3 Mbps 825.7 Kpps tx: 0 bps 0 pps rx: 396.6 Mbps 826.3 Kpps tx: 0 bps 0 pps rx: 396.5 Mbps 826.1 KppsPID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 9 root 20 0 0 0 0 R 99.7 0.0 7:02.58 ksoftirqd/0 15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1 20 root 20 0 0 0 0 S 0.0 0.0 2:01.48 ksoftirqd/2 25 root 20 0 0 0 0 S 0.0 0.0 0:32.86 ksoftirqd/3and this with rx hashing enabled:
# ethtool -K eth2 rxhash on # utraf eth2 tx: 0 bps 0 pps rx: 456.4 Mbps 950.8 Kpps tx: 0 bps 0 pps rx: 458.4 Mbps 955.0 Kpps tx: 0 bps 0 pps rx: 457.6 Mbps 953.3 Kpps tx: 0 bps 0 pps rx: 462.2 Mbps 962.9 KppsPID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 20 root 20 0 0 0 0 R 0.7 0.0 2:02.34 ksoftirqd/2 25 root 20 0 0 0 0 S 0.3 0.0 0:33.25 ksoftirqd/3 9 root 20 0 0 0 0 S 0.0 0.0 7:52.57 ksoftirqd/0 15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1
Quoted 4 lines
The throughput doesn't increase so much, maybe we hit an HW limit of the gigabit port. The interesting thing is how the global CPU usage drops from 25% to 1%.
I can't explain this, it could be due to the reduced contention?
Hi Matteo,
Can I take that as a Tested-by ?
Thanks.
Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTC for 0.8m (est. 1762m) line in suburbia: sync at 13.1Mbps down 424kbps up
Matteo Croce
Tested-byRe: [EXT] Re: [PATCH net-next 3/5] net: mvpp2: cls: Use RSS contexts to handle RSS tables
Quoted 57 lines · Russell King - ARM Linux admin
Quoted 48 lines · Matteo Croce
On Tue, 19 May 2020 12:05:20 +0200
Matteo Croce <[email protected]> wrote:Hi,
The patch seems to work. I'm generating traffic with random MAC and IP addresses, to have many flows:
# tcpdump -tenni eth2 9a:a9:b1:3a:b1:6b > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.4.0 > 192.168.0.4.0: UDP, length 12 9e:92:fd:f8:7f:0a > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.4.0 > 192.168.0.4.0: UDP, length 12 66:b7:11:8a:c2:1f > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.1.0 > 192.168.0.1.0: UDP, length 12 7a:ba:58:bd:9a:62 > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.1.0 > 192.168.0.1.0: UDP, length 12 7e:78:a9:97:70:3a > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.2.0 > 192.168.0.2.0: UDP, length 12 b2:81:91:34:ce:42 > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.2.0 > 192.168.0.2.0: UDP, length 12 2a:05:52:d0:d9:3f > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.3.0 > 192.168.0.3.0: UDP, length 12 ee:ee:47:35:fa:81 > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.3.0 > 192.168.0.3.0: UDP, length 12This is the default rate, with rxhash off:
# utraf eth2 tx: 0 bps 0 pps rx: 397.4 Mbps 827.9 Kpps tx: 0 bps 0 pps rx: 396.3 Mbps 825.7 Kpps tx: 0 bps 0 pps rx: 396.6 Mbps 826.3 Kpps tx: 0 bps 0 pps rx: 396.5 Mbps 826.1 KppsPID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 9 root 20 0 0 0 0 R 99.7 0.0 7:02.58 ksoftirqd/0 15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1 20 root 20 0 0 0 0 S 0.0 0.0 2:01.48 ksoftirqd/2 25 root 20 0 0 0 0 S 0.0 0.0 0:32.86 ksoftirqd/3and this with rx hashing enabled:
# ethtool -K eth2 rxhash on # utraf eth2 tx: 0 bps 0 pps rx: 456.4 Mbps 950.8 Kpps tx: 0 bps 0 pps rx: 458.4 Mbps 955.0 Kpps tx: 0 bps 0 pps rx: 457.6 Mbps 953.3 Kpps tx: 0 bps 0 pps rx: 462.2 Mbps 962.9 KppsPID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 20 root 20 0 0 0 0 R 0.7 0.0 2:02.34 ksoftirqd/2 25 root 20 0 0 0 0 S 0.3 0.0 0:33.25 ksoftirqd/3 9 root 20 0 0 0 0 S 0.0 0.0 7:52.57 ksoftirqd/0 15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1Quoted 4 lines
The throughput doesn't increase so much, maybe we hit an HW limit of the gigabit port. The interesting thing is how the global CPU usage drops from 25% to 1%.
I can't explain this, it could be due to the reduced contention?Hi Matteo,
Quoted 8 lines
Can I take that as a Tested-by ?
Thanks.
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC for 0.8m (est. 1762m) line in suburbia: sync at 13.1Mbps down 424kbps up
probably also:
Thanks,
Signature · 2 lines
Matteo Croce per aspera ad upstream
Russell King - ARM Linux admin
maintainerRe: [EXT] Re: [PATCH net-next 3/5] net: mvpp2: cls: Use RSS contexts to handle RSS tables
Quoted 68 lines · Matteo Croce
Quoted 57 lines · Russell King - ARM Linux admin
Quoted 48 lines · Matteo Croce
On Tue, 19 May 2020 12:05:20 +0200
Matteo Croce <[email protected]> wrote:Hi,
The patch seems to work. I'm generating traffic with random MAC and IP addresses, to have many flows:
# tcpdump -tenni eth2 9a:a9:b1:3a:b1:6b > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.4.0 > 192.168.0.4.0: UDP, length 12 9e:92:fd:f8:7f:0a > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.4.0 > 192.168.0.4.0: UDP, length 12 66:b7:11:8a:c2:1f > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.1.0 > 192.168.0.1.0: UDP, length 12 7a:ba:58:bd:9a:62 > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.1.0 > 192.168.0.1.0: UDP, length 12 7e:78:a9:97:70:3a > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.2.0 > 192.168.0.2.0: UDP, length 12 b2:81:91:34:ce:42 > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.2.0 > 192.168.0.2.0: UDP, length 12 2a:05:52:d0:d9:3f > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.3.0 > 192.168.0.3.0: UDP, length 12 ee:ee:47:35:fa:81 > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.3.0 > 192.168.0.3.0: UDP, length 12This is the default rate, with rxhash off:
# utraf eth2 tx: 0 bps 0 pps rx: 397.4 Mbps 827.9 Kpps tx: 0 bps 0 pps rx: 396.3 Mbps 825.7 Kpps tx: 0 bps 0 pps rx: 396.6 Mbps 826.3 Kpps tx: 0 bps 0 pps rx: 396.5 Mbps 826.1 KppsPID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 9 root 20 0 0 0 0 R 99.7 0.0 7:02.58 ksoftirqd/0 15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1 20 root 20 0 0 0 0 S 0.0 0.0 2:01.48 ksoftirqd/2 25 root 20 0 0 0 0 S 0.0 0.0 0:32.86 ksoftirqd/3and this with rx hashing enabled:
# ethtool -K eth2 rxhash on # utraf eth2 tx: 0 bps 0 pps rx: 456.4 Mbps 950.8 Kpps tx: 0 bps 0 pps rx: 458.4 Mbps 955.0 Kpps tx: 0 bps 0 pps rx: 457.6 Mbps 953.3 Kpps tx: 0 bps 0 pps rx: 462.2 Mbps 962.9 KppsPID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 20 root 20 0 0 0 0 R 0.7 0.0 2:02.34 ksoftirqd/2 25 root 20 0 0 0 0 S 0.3 0.0 0:33.25 ksoftirqd/3 9 root 20 0 0 0 0 S 0.0 0.0 7:52.57 ksoftirqd/0 15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1Quoted 4 lines
The throughput doesn't increase so much, maybe we hit an HW limit of the gigabit port. The interesting thing is how the global CPU usage drops from 25% to 1%.
I can't explain this, it could be due to the reduced contention?Hi Matteo,
Quoted 8 lines
Can I take that as a Tested-by ?
Thanks.
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC for 0.8m (est. 1762m) line in suburbia: sync at 13.1Mbps down 424kbps up
Quoted 5 lines
Tested-by Matteo Croce <[email protected]>probably also:
Reported-by Matteo Croce <[email protected]>
Thanks!
Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTC for 0.8m (est. 1762m) line in suburbia: sync at 13.1Mbps down 424kbps up
Re: [EXT] Re: [PATCH net-next 3/5] net: mvpp2: cls: Use RSS contexts to handle RSS tables
In reply to Stefan ChulskiQuoted 18 lines · Stefan Chulski
Quoted 18 lines
Hi,
What do you think about temporarily disabling it like this?
drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c +2
-1@@ -5775,7 +5775,8 @@ static int mvpp2_port_probe(struct platform_device *pdev,NETIF_F_HW_VLAN_CTAG_FILTER ;if (mvpp22_rss_is_supported ()) { - dev->hw_features |=NETIF_F_RXHASH ; +if (port->phy_interface !=PHY_INTERFACE_MODE_SGMII ) + dev->hw_features |=NETIF_F_RXHASH ; dev->features |=NETIF_F_NTUPLE ; }
Quoted 5 lines
Quoted 1 line
David, is this "workaround" too bad to get accepted?
Not sure that RSS related to physical interface(SGMII), better just remove
NETIF_F_RXHASHas "workaround".Stefan.
Hi,
The point is that RXHASH works fine on all interfaces, but on the gigabit one (eth2 usually).
And on the 10 gbit interface is very very effective, the throughput goes 4x when enabled, so it would be a big drawback to disable it on all interfaces.
Honestly I don't have any 2.5 gbit hardware to test it on eth3, so I don't know if rxhash actually only works on the first interface of a unit (so eth0 and eth1),
or if it just doesn't work on the gigabit one.
If someone could test it on the 2.5 gbit port, this will be helpful.
Regards,
Signature · 2 lines
Matteo Croce per aspera ad upstream
RE: [EXT] Re: [PATCH net-next 3/5] net: mvpp2: cls: Use RSS contexts to handle RSS tables
Quoted 51 lines
-----Original Message-----
From: Matteo Croce <[email protected]>
Sent: Saturday, May 9, 2020 3:16 PM
To: Stefan Chulski <[email protected]>
Cc: David S . Miller <[email protected]>; Maxime Chevallier <[email protected]>; netdev <[email protected]>; LKML <[email protected]>; Antoine Tenart <[email protected]>; Thomas Petazzoni <[email protected]>; [email protected]; [email protected]; Nadav Haklai <[email protected]>; Marcin Wojtas <[email protected]>; Linux ARM <linux-arm- [email protected]>; Russell King - ARM Linux admin <[email protected]>
Subject: Re: [EXT] Re: [PATCH net-next 3/5] net: mvpp2: cls: Use RSS contexts to handle RSS tablesQuoted 18 lines · Stefan Chulski
Quoted 18 lines
Hi,
What do you think about temporarily disabling it like this?
drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c +2
-1@@ -5775,7 +5775,8 @@ static int mvpp2_port_probe(structplatform_device *pdev,NETIF_F_HW_VLAN_CTAG_FILTER ;if (mvpp22_rss_is_supported ()) { - dev->hw_features |=NETIF_F_RXHASH ; +if (port->phy_interface !=PHY_INTERFACE_MODE_SGMII ) + dev->hw_features |=NETIF_F_RXHASH ; dev->features |=NETIF_F_NTUPLE ; }Quoted 3 lines
Quoted 1 line
David, is this "workaround" too bad to get accepted?
Not sure that RSS related to physical interface(SGMII), better just remove
NETIF_F_RXHASHas "workaround".Quoted 2 lines
Stefan.
Hi,
The point is that RXHASH works fine on all interfaces, but on the gigabit one (eth2 usually).
And on the 10 gbit interface is very very effective, the throughput goes 4x when enabled, so it would be a big drawback to disable it on all interfaces.
Quoted 5 lines
Honestly I don't have any 2.5 gbit hardware to test it on eth3, so I don't know if rxhash actually only works on the first interface of a unit (so eth0 and eth1), or if it just doesn't work on the gigabit one.
If someone could test it on the 2.5 gbit port, this will be helpful.
RSS tables is part of Packet Processor IP, not MAC(so it's not related to specific speed). Probably issue exist on specific packet processor ports. Since RSS work fine on first port of the CP, we can do the following:
if (port-> id == 0)
dev->hw_features |= NETIF_F_RXHASH;
Regards.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Russell King - ARM Linux admin
maintainerRe: [EXT] Re: [PATCH net-next 3/5] net: mvpp2: cls: Use RSS contexts to handle RSS tables
Quoted 27 lines · Stefan Chulski
Quoted 27 lines
-----Original Message-----
From: Matteo Croce <[email protected]>
Sent: Saturday, May 9, 2020 3:16 PM
To: Stefan Chulski <[email protected]>
Cc: David S . Miller <[email protected]>; Maxime Chevallier <[email protected]>; netdev <[email protected]>; LKML <[email protected]>; Antoine Tenart <[email protected]>; Thomas Petazzoni <[email protected]>; [email protected]; [email protected]; Nadav Haklai <[email protected]>; Marcin Wojtas <[email protected]>; Linux ARM <linux-arm- [email protected]>; Russell King - ARM Linux admin <[email protected]>
Subject: Re: [EXT] Re: [PATCH net-next 3/5] net: mvpp2: cls: Use RSS contexts to handle RSS tablesHi,
The point is that RXHASH works fine on all interfaces, but on the gigabit one (eth2 usually).
And on the 10 gbit interface is very very effective, the throughput goes 4x when enabled, so it would be a big drawback to disable it on all interfaces.Honestly I don't have any 2.5 gbit hardware to test it on eth3, so I don't know if rxhash actually only works on the first interface of a unit (so eth0 and eth1), or if it just doesn't work on the gigabit one.
Quoted 6 lines
Quoted 1 line
If someone could test it on the 2.5 gbit port, this will be helpful.
RSS tables is part of Packet Processor IP, not MAC(so it's not related to specific speed). Probably issue exist on specific packet processor ports. Since RSS work fine on first port of the CP, we can do the following:
if (port-> id == 0)dev->hw_features |= NETIF_F_RXHASH;
I can confirm that Macchiatobin Single Shot eth0 port works with a 1G Fibre SFP or 10G DA SFP with or without rxhash on.
So it seems Stefan's hunch that it is port related is correct.
Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTC broadband for 0.8mile line in suburbia: sync at 10.2Mbps down 587kbps up
PATCH 4/5 net: mvpp2: cls: Extract the RSS context when parsing the ethtool rule
In reply to Maxime Chevallierethtool_rx_flow_rule_create takes into parameter the ethtool flow spec, which doesn't contain the rss context id. We therefore need to extract it ourself before parsing the ethtool rule.
The FLOW_RSS flag is only set in info->fs.flow_type, and not info->flow_type.
--- drivers/net/ethernet/marvell/mvpp2/mvpp2_cls.c | 6 ++++++ 1 file changed, 6 insertions(+)
drivers/net/ethernet/marvell/mvpp2/mvpp2_cls.c +6 -0
@@ -1281,6 +1281,12 @@ int mvpp2_ethtool_cls_rule_ins(struct mvpp2_port *port, input.fs = &info->fs; +/* We need to manually set the rss_ctx, since this info isn't present +* in info->fs +*/ +if (info->fs.flow_type &FLOW_RSS ) + input.rss_ctx = info->rss_context; + ethtool_rule =ethtool_rx_flow_rule_create (&input);if (IS_ERR (ethtool_rule)) { ret =PTR_ERR (ethtool_rule);
Signature · 1 line
2.20.1
PATCH 5/5 net: mvpp2: cls: Support steering to RSS contexts
In reply to Maxime ChevallierWhen steering to an RXQ, we can perform an extra RSS step to assign a queue from an RSS table.
This is done by setting the RSS_EN attribute in the C2 engine. In that case, the RXQ that is assigned is the global RSS context id, that is then translated to an RSS table using the RXQ2RSS table.
An example using ethtool to steer to RXQ 2 and 3 would be :
ethtool -X eth0 weight 0 0 1 1 context new
(This would print the allocated context id, let's say it's 1)
ethtool -N eth0 flow-type udp4 dst-port 1234 context 1 loc 0
The hash parameters are the ones that are globally configured for RSS :
ethtool -N eth0 rx-flow-hash udp4 sdfn
When an RSS context is removed while there are active classification rules using this context, these rules are removed.
--- .../net/ethernet/marvell/mvpp2/mvpp2_cls.c | 58 +++++++++++++++++-- 1 file changed, 54 insertions(+), 4 deletions(-)
drivers/net/ethernet/marvell/mvpp2/mvpp2_cls.c +54 -4
@@ -1068,7 +1068,7 @@ static int mvpp2_port_c2_tcam_rule_add(struct mvpp2_port *port,struct flow_action_entry *act;struct mvpp2_cls_c2_entry c2;u8 qh, ql, pmap; -int index; +int index, ctx;memset (&c2,0 ,sizeof (c2));
@@ -1108,14 +1108,36 @@ static int mvpp2_port_c2_tcam_rule_add(struct mvpp2_port *port,*/ c2.act =MVPP22_CLS_C2_ACT_COLOR (MVPP22_C2_COL_NO_UPD_LOCK ); +/* Update RSS status after matching this entry */ +if (act->queue.ctx) + c2.attr[2 ] |=MVPP22_CLS_C2_ATTR2_RSS_EN ; + +/* Always lock the RSS_EN decision. We might have high prio +* rules steering to an RXQ, and a lower one steering to RSS, +* we don't want the low prio RSS rule overwriting this flag. +*/ + c2.act =MVPP22_CLS_C2_ACT_RSS_EN (MVPP22_C2_UPD_LOCK ); +/* Mark packet as "forwarded to software", needed for RSS */ c2.act |=MVPP22_CLS_C2_ACT_FWD (MVPP22_C2_FWD_SW_LOCK ); c2.act |=MVPP22_CLS_C2_ACT_QHIGH (MVPP22_C2_UPD_LOCK ) |MVPP22_CLS_C2_ACT_QLOW (MVPP22_C2_UPD_LOCK ); - qh = ((act->queue.index + port->first_rxq) >>3 ) &MVPP22_CLS_C2_ATTR0_QHIGH_MASK ; - ql = (act->queue.index + port->first_rxq) &MVPP22_CLS_C2_ATTR0_QLOW_MASK ; +if (act->queue.ctx) { +/* Get the global ctx number */ + ctx =mvpp22_rss_ctx (port, act->queue.ctx); +if (ctx <0 ) +return -EINVAL ; + + qh = (ctx >>3 ) &MVPP22_CLS_C2_ATTR0_QHIGH_MASK ; + ql = ctx &MVPP22_CLS_C2_ATTR0_QLOW_MASK ; + }else { + qh = ((act->queue.index + port->first_rxq) >>3 ) & +MVPP22_CLS_C2_ATTR0_QHIGH_MASK ; + ql = (act->queue.index + port->first_rxq) & +MVPP22_CLS_C2_ATTR0_QLOW_MASK ; + } c2.attr[0 ] =MVPP22_CLS_C2_ATTR0_QHIGH (qh) |MVPP22_CLS_C2_ATTR0_QLOW (ql);
@@ -1235,6 +1257,13 @@ static int mvpp2_cls_rfs_parse_rule(struct mvpp2_rfs_rule *rule)if (act->id !=FLOW_ACTION_QUEUE && act->id !=FLOW_ACTION_DROP )return -EOPNOTSUPP ; +/* When both an RSS context and an queue index are set, the index +* is considered as an offset to be added to the indirection table +* entries. We don't support this, so reject this rule. +*/ +if (act->queue.ctx && act->queue.index) +return -EOPNOTSUPP ; +/* For now, only use the C2 engine which has a HEK size limited to 64 * bits for TCAM matching. */
@@ -1455,11 +1484,32 @@ static struct mvpp2_rss_table *mvpp22_rss_table_get(struct mvpp2 *priv,int mvpp22_port_rss_ctx_delete (struct mvpp2_port *port,u32 port_ctx) {struct mvpp2 *priv = port->priv; -int rss_ctx =mvpp22_rss_ctx (port, port_ctx); +struct ethtool_rxnfc *rxnfc; +int i, rss_ctx, ret; + + rss_ctx =mvpp22_rss_ctx (port, port_ctx);if (rss_ctx <0 || rss_ctx >=MVPP22_N_RSS_TABLES )return -EINVAL ; +/* Invalidate any active classification rule that use this context */ +for (i =0 ; i <MVPP2_N_RFS_ENTRIES_PER_FLOW ; i++) { +if (!port->rfs_rules[i]) +continue ; + + rxnfc = &port->rfs_rules[i]->rxnfc; +if (!(rxnfc->fs.flow_type &FLOW_RSS ) || + rxnfc->rss_context != port_ctx) +continue ; + + ret =mvpp2_ethtool_cls_rule_del (port, rxnfc); +if (ret) { +netdev_warn (port->dev, +"couldn't remove classification rule %d associated to this context" , + rxnfc->fs.location); + } + } +kfree (priv->rss_tables[rss_ctx]); priv->rss_tables[rss_ctx] =NULL ;
Signature · 1 line
2.20.1
David Miller
maintainerFrom: Maxime Chevallier <[email protected]>
Date: Fri, 24 May 2019 12:05:49 +0200
Quoted 11 lines
Here is a set of updates for the PPv2 classifier, the main feature being the support for steering to RSS contexts, to leverage all the available RSS tables in the controller.
The first two patches are non-critical fixes for the classifier, the first one prevents us from allocating too much room to store the classification rules, the second one configuring the C2 engine as suggested by the PPv2 functionnal specs.
Patches 3 to 5 introduce support for RSS contexts in mvpp2, allowing us to steer traffic to dedicated RSS tables.
Series applied, thanks.