net: ethtool: support including Flow Label in the flow hash for RSS
Subsystems (7)
Jakub Kicinski
maintainerv3 cover net: ethtool: support including Flow Label in the flow hash for RSS
Add support for using IPv6 Flow Label in Rx hash computation and therefore RSS queue selection.
v3:
- change the bnxt driver, bits are now exclusive
- check for RPS/RFS in the test
v2: https://lore.kernel.org/[email protected] RFC: https://lore.kernel.org/[email protected]
Jakub Kicinski (4):
net: ethtool: support including Flow Label in the flow hash for RSS
eth: fbnic: support RSS on IPv6 Flow Label
eth: bnxt: support RSS on IPv6 Flow Label
selftests: drv-net: add test for RSS on flow label
Documentation/netlink/specs/ethtool.yaml | 3 + .../testing/selftests/drivers/net/hw/Makefile | 1 + drivers/net/ethernet/broadcom/bnxt/bnxt.h | 1 + include/uapi/linux/ethtool.h | 1 + drivers/net/ethernet/broadcom/bnxt/bnxt.c | 2 + .../net/ethernet/broadcom/bnxt/bnxt_ethtool.c | 20 ++- .../net/ethernet/meta/fbnic/fbnic_ethtool.c | 2 +- drivers/net/ethernet/meta/fbnic/fbnic_rpc.c | 2 + net/ethtool/ioctl.c | 25 +++ net/ethtool/rss.c | 27 +-- .../drivers/net/hw/rss_flow_label.py | 167 ++++++++++++++++++ 11 files changed, 233 insertions(+), 18 deletions(-) create mode 100755 tools/testing/selftests/drivers/net/hw/rss_flow_label.py
Signature · 1 line
2.50.1
Jakub Kicinski
maintainerPATCH v3 1/4 net: ethtool: support including Flow Label in the flow hash for RSS
Some modern NICs support including the IPv6 Flow Label in the flow hash for RSS queue selection. This is outside the old "Microsoft spec", but was included in the OCP NIC spec:
[ ] RSS include ow label in the hash (configurable)
https://www.opencompute.org/w/index.php?title=Core_Offloads#Receive_Side_Scaling
RSS Flow Label hashing allows TCP Protective Load Balancing (PLB) to recover from receiver congestion / overload.
Rx CPU/queue hotspots are relatively common for data ingest workloads, and so far we had to try to detect the condition at the RPC layer and reopen the connection. PLB lets us change the Flow Label and therefore Rx CPU on RTO, with minimal packet reordering. PLB reaction times are much faster, and can happen at any point in the connection, not just at RPC boundaries.
Due to the nature of host processing (relatively long queues, other kernel subsystems masking IRQs for 100s of msecs) the risk of reordering within the host is higher than in the network. But for applications which need it - it is far preferable to potentially persistent overload of subset of queues.
It is expected that the hash communicated to the host may change if the Flow Label changes. This may be surprising to some host software, but I don't expect the devices can compute two Toeplitz hashes, one with the Flow Label for queue selection and one without for the rx hash communicated to the host. Besides, changing the hash may potentially help to change the path thru host queues. User can disable NETIF_F_RXHASH if they require a stable flow hash.
The name RXH_IP6_FL was chosen based on what we call Flow Label variables in IPv6 processing (fl). I prefer fl_lbl but that appears to be an fbnic-only spelling. We could spell out RXH_IP6_FLOW_LABEL but existing RXH_ defines are a lot more terse.
Willem notes [1] that Flow Label is defined as identifying the flow and therefore including both the flow label _and_ the L4 header fields is not generally necessary. But it should not hurt so it's not explicitly prevented if the driver supports hashing on both at the same time.
---
CC: [email protected]
CC: [email protected]
--- Documentation/netlink/specs/ethtool.yaml | 3 +++ include/uapi/linux/ethtool.h | 1 + net/ethtool/ioctl.c | 25 ++++++++++++++++++++++ net/ethtool/rss.c | 27 ++++++++++++------------ 4 files changed, 43 insertions(+), 13 deletions(-)
Documentation/netlink/specs/ethtool.yaml +3 -0
@@ -204,6 +204,9 @@ c-version-name: ethtool-genl-version doc: dst port in case of TCP/UDP/SCTP - name: gtp-teid + - + name: ip6-fl + doc: IPv6 Flow Label - name: discard value: 31
include/uapi/linux/ethtool.h +1 -0
@@ -2380,6 +2380,7 @@ enum {#define RXH_L4_B_0_1 (1 << 6)/* src port in case of TCP/UDP/SCTP */ #define RXH_L4_B_2_3 (1 << 7)/* dst port in case of TCP/UDP/SCTP */ #define RXH_GTP_TEID (1 << 8)/* teid in case of GTP */ +#define RXH_IP6_FL (1 << 9)/* IPv6 flow label */ #define RXH_DISCARD (1 << 31)#define RX_CLS_FLOW_DISC 0xffffffffffffffffULL
net/ethtool/ioctl.c +25 -0
@@ -1014,6 +1014,28 @@ static bool flow_type_hashable(u32 flow_type)return false; } +static bool flow_type_v6 (u32 flow_type) +{ +switch (flow_type) { +case TCP_V6_FLOW : +case UDP_V6_FLOW : +case SCTP_V6_FLOW : +case AH_ESP_V6_FLOW : +case AH_V6_FLOW : +case ESP_V6_FLOW : +case IPV6_FLOW : +case GTPU_V6_FLOW : +case GTPC_V6_FLOW : +case GTPC_TEID_V6_FLOW : +case GTPU_EH_V6_FLOW : +case GTPU_UL_V6_FLOW : +case GTPU_DL_V6_FLOW : +return true; + } + +return false; +} +/* When adding a new type, update the assert and, if it's hashable, add it to * the flow_type_hashable switch case. */
@@ -1077,6 +1099,9 @@ ethtool_set_rxfh_fields(struct net_device *dev, u32 cmd, void __user *useraddr)if (rc)return rc; +if (info.data &RXH_IP6_FL && !flow_type_v6 (info.flow_type)) +return -EINVAL ; +if (info.flow_type &FLOW_RSS && info.rss_context && !ops->rxfh_per_ctx_fields)return -EINVAL ;
net/ethtool/rss.c +14 -13
@@ -536,35 +536,36 @@ void ethtool_rss_notify(struct net_device *dev, u32 type, u32 rss_context)#define RFH_MASK (RXH_L2DA | RXH_VLAN | RXH_IP_SRC | RXH_IP_DST | \ RXH_L3_PROTO | RXH_L4_B_0_1 | RXH_L4_B_2_3 | \ RXH_GTP_TEID | RXH_DISCARD) +#define RFH_MASKv6 (RFH_MASK | RXH_IP6_FL)static const struct nla_policy ethnl_rss_flows_policy[] = { [ETHTOOL_A_FLOW_ETHER ] =NLA_POLICY_MASK (NLA_UINT ,RFH_MASK ), [ETHTOOL_A_FLOW_IP4 ] =NLA_POLICY_MASK (NLA_UINT ,RFH_MASK ), - [ETHTOOL_A_FLOW_IP6 ] =NLA_POLICY_MASK (NLA_UINT ,RFH_MASK ), + [ETHTOOL_A_FLOW_IP6 ] =NLA_POLICY_MASK (NLA_UINT , RFH_MASKv6), [ETHTOOL_A_FLOW_TCP4 ] =NLA_POLICY_MASK (NLA_UINT ,RFH_MASK ), [ETHTOOL_A_FLOW_UDP4 ] =NLA_POLICY_MASK (NLA_UINT ,RFH_MASK ), [ETHTOOL_A_FLOW_SCTP4 ] =NLA_POLICY_MASK (NLA_UINT ,RFH_MASK ), [ETHTOOL_A_FLOW_AH_ESP4 ] =NLA_POLICY_MASK (NLA_UINT ,RFH_MASK ), - [ETHTOOL_A_FLOW_TCP6 ] =NLA_POLICY_MASK (NLA_UINT ,RFH_MASK ), - [ETHTOOL_A_FLOW_UDP6 ] =NLA_POLICY_MASK (NLA_UINT ,RFH_MASK ), - [ETHTOOL_A_FLOW_SCTP6 ] =NLA_POLICY_MASK (NLA_UINT ,RFH_MASK ), - [ETHTOOL_A_FLOW_AH_ESP6 ] =NLA_POLICY_MASK (NLA_UINT ,RFH_MASK ), + [ETHTOOL_A_FLOW_TCP6 ] =NLA_POLICY_MASK (NLA_UINT , RFH_MASKv6), + [ETHTOOL_A_FLOW_UDP6 ] =NLA_POLICY_MASK (NLA_UINT , RFH_MASKv6), + [ETHTOOL_A_FLOW_SCTP6 ] =NLA_POLICY_MASK (NLA_UINT , RFH_MASKv6), + [ETHTOOL_A_FLOW_AH_ESP6 ] =NLA_POLICY_MASK (NLA_UINT , RFH_MASKv6), [ETHTOOL_A_FLOW_AH4 ] =NLA_POLICY_MASK (NLA_UINT ,RFH_MASK ), [ETHTOOL_A_FLOW_ESP4 ] =NLA_POLICY_MASK (NLA_UINT ,RFH_MASK ), - [ETHTOOL_A_FLOW_AH6 ] =NLA_POLICY_MASK (NLA_UINT ,RFH_MASK ), - [ETHTOOL_A_FLOW_ESP6 ] =NLA_POLICY_MASK (NLA_UINT ,RFH_MASK ), + [ETHTOOL_A_FLOW_AH6 ] =NLA_POLICY_MASK (NLA_UINT , RFH_MASKv6), + [ETHTOOL_A_FLOW_ESP6 ] =NLA_POLICY_MASK (NLA_UINT , RFH_MASKv6), [ETHTOOL_A_FLOW_GTPU4 ] =NLA_POLICY_MASK (NLA_UINT ,RFH_MASK ), - [ETHTOOL_A_FLOW_GTPU6 ] =NLA_POLICY_MASK (NLA_UINT ,RFH_MASK ), + [ETHTOOL_A_FLOW_GTPU6 ] =NLA_POLICY_MASK (NLA_UINT , RFH_MASKv6), [ETHTOOL_A_FLOW_GTPC4 ] =NLA_POLICY_MASK (NLA_UINT ,RFH_MASK ), - [ETHTOOL_A_FLOW_GTPC6 ] =NLA_POLICY_MASK (NLA_UINT ,RFH_MASK ), + [ETHTOOL_A_FLOW_GTPC6 ] =NLA_POLICY_MASK (NLA_UINT , RFH_MASKv6), [ETHTOOL_A_FLOW_GTPC_TEID4 ] =NLA_POLICY_MASK (NLA_UINT ,RFH_MASK ), - [ETHTOOL_A_FLOW_GTPC_TEID6 ] =NLA_POLICY_MASK (NLA_UINT ,RFH_MASK ), + [ETHTOOL_A_FLOW_GTPC_TEID6 ] =NLA_POLICY_MASK (NLA_UINT , RFH_MASKv6), [ETHTOOL_A_FLOW_GTPU_EH4 ] =NLA_POLICY_MASK (NLA_UINT ,RFH_MASK ), - [ETHTOOL_A_FLOW_GTPU_EH6 ] =NLA_POLICY_MASK (NLA_UINT ,RFH_MASK ), + [ETHTOOL_A_FLOW_GTPU_EH6 ] =NLA_POLICY_MASK (NLA_UINT , RFH_MASKv6), [ETHTOOL_A_FLOW_GTPU_UL4 ] =NLA_POLICY_MASK (NLA_UINT ,RFH_MASK ), - [ETHTOOL_A_FLOW_GTPU_UL6 ] =NLA_POLICY_MASK (NLA_UINT ,RFH_MASK ), + [ETHTOOL_A_FLOW_GTPU_UL6 ] =NLA_POLICY_MASK (NLA_UINT , RFH_MASKv6), [ETHTOOL_A_FLOW_GTPU_DL4 ] =NLA_POLICY_MASK (NLA_UINT ,RFH_MASK ), - [ETHTOOL_A_FLOW_GTPU_DL6 ] =NLA_POLICY_MASK (NLA_UINT ,RFH_MASK ), + [ETHTOOL_A_FLOW_GTPU_DL6 ] =NLA_POLICY_MASK (NLA_UINT , RFH_MASKv6), };const struct nla_policy ethnl_rss_set_policy[ETHTOOL_A_RSS_FLOW_HASH +1 ] = {
Signature · 1 line
2.50.1
Jakub Kicinski
maintainerPATCH v3 2/4 eth: fbnic: support RSS on IPv6 Flow Label
In reply to Jakub KicinskiSupport IPv6 Flow Label hashing. Use both inner and outer IPv6 header's Flow Label if both headers are detected. Flow Label is unlike normal header fields, by enabling it user accepts the unstable hash and possible reordering. Because of that I think it's reasonable to hash over all Flow Labels we can find, even tho we don't hash over all L3 addresses.
---
CC: [email protected]
CC: [email protected]
--- drivers/net/ethernet/meta/fbnic/fbnic_ethtool.c | 2 +- drivers/net/ethernet/meta/fbnic/fbnic_rpc.c | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-)
drivers/net/ethernet/meta/fbnic/fbnic_ethtool.c +1 -1
@@ -1310,7 +1310,7 @@ fbnic_get_rss_hash_opts(struct net_device *netdev,#define FBNIC_L2_HASH_OPTIONS \ (RXH_L2DA | RXH_DISCARD)#define FBNIC_L3_HASH_OPTIONS \ - (FBNIC_L2_HASH_OPTIONS | RXH_IP_SRC | RXH_IP_DST) + (FBNIC_L2_HASH_OPTIONS | RXH_IP_SRC | RXH_IP_DST | RXH_IP6_FL)#define FBNIC_L4_HASH_OPTIONS \ (FBNIC_L3_HASH_OPTIONS | RXH_L4_B_0_1 | RXH_L4_B_2_3)
drivers/net/ethernet/meta/fbnic/fbnic_rpc.c +2 -0
@@ -71,6 +71,8 @@ u16 fbnic_flow_hash_2_rss_en_mask(struct fbnic_net *fbn, int flow_type) rss_en_mask |=FBNIC_FH_2_RSSEM_BIT (IP_DST ,IP_DST , flow_hash); rss_en_mask |=FBNIC_FH_2_RSSEM_BIT (L4_B_0_1 ,L4_SRC , flow_hash); rss_en_mask |=FBNIC_FH_2_RSSEM_BIT (L4_B_2_3 ,L4_DST , flow_hash); + rss_en_mask |=FBNIC_FH_2_RSSEM_BIT (IP6_FL ,OV6_FL_LBL , flow_hash); + rss_en_mask |=FBNIC_FH_2_RSSEM_BIT (IP6_FL ,IV6_FL_LBL , flow_hash);return rss_en_mask; }
Signature · 1 line
2.50.1
Jakub Kicinski
maintainerPATCH v3 3/4 eth: bnxt: support RSS on IPv6 Flow Label
In reply to Jakub KicinskiIt appears that the bnxt FW API has the relevant bit for Flow Label hashing. Plumb in the support. Obey the capability bit.
---
v2:
- update the logic to pick the 2 tuple *OR* the FL bit
v1: https://lore.kernel.org/[email protected]
--- drivers/net/ethernet/broadcom/bnxt/bnxt.h | 1 + drivers/net/ethernet/broadcom/bnxt/bnxt.c | 2 ++ .../net/ethernet/broadcom/bnxt/bnxt_ethtool.c | 20 +++++++++++++++---- 3 files changed, 19 insertions(+), 4 deletions(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.h +1 -0
@@ -2407,6 +2407,7 @@ struct bnxt {#define BNXT_RSS_CAP_ESP_V4_RSS_CAP BIT(6)#define BNXT_RSS_CAP_ESP_V6_RSS_CAP BIT(7)#define BNXT_RSS_CAP_MULTI_RSS_CTX BIT(8) +#define BNXT_RSS_CAP_IPV6_FLOW_LABEL_RSS_CAP BIT(9)u8 rss_hash_key[HW_HASH_KEY_SIZE ]; u8 rss_hash_key_valid:1;
drivers/net/ethernet/broadcom/bnxt/bnxt.c +2 -0
@@ -6957,6 +6957,8 @@ static int bnxt_hwrm_vnic_qcaps(struct bnxt *bp) bp->rss_cap |=BNXT_RSS_CAP_ESP_V4_RSS_CAP ;if (flags &VNIC_QCAPS_RESP_FLAGS_RSS_IPSEC_ESP_SPI_IPV6_CAP ) bp->rss_cap |=BNXT_RSS_CAP_ESP_V6_RSS_CAP ; +if (flags &VNIC_QCAPS_RESP_FLAGS_RSS_IPV6_FLOW_LABEL_CAP ) + bp->rss_cap |=BNXT_RSS_CAP_IPV6_FLOW_LABEL_RSS_CAP ;if (flags &VNIC_QCAPS_RESP_FLAGS_RE_FLUSH_CAP ) bp->fw_cap |=BNXT_FW_CAP_VNIC_RE_FLUSH ; }
drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c +16 -4
@@ -1584,6 +1584,8 @@ static u64 get_ethtool_ipv6_rss(struct bnxt *bp) {if (bp->rss_hash_cfg &VNIC_RSS_CFG_REQ_HASH_TYPE_IPV6 )return RXH_IP_SRC |RXH_IP_DST ; +if (bp->rss_hash_cfg &VNIC_RSS_CFG_REQ_HASH_TYPE_IPV6_FLOW_LABEL ) +return RXH_IP_SRC |RXH_IP_DST |RXH_IP6_FL ;return 0 ; }
@@ -1662,13 +1664,18 @@ static int bnxt_set_rxfh_fields(struct net_device *dev,if (cmd->data ==RXH_4TUPLE ) tuple =4 ; -else if (cmd->data ==RXH_2TUPLE ) +else if (cmd->data ==RXH_2TUPLE || + cmd->data == (RXH_2TUPLE |RXH_IP6_FL )) tuple =2 ;else if (!cmd->data) tuple =0 ;else return -EINVAL ; +if (cmd->data &RXH_IP6_FL && + !(bp->rss_cap &BNXT_RSS_CAP_IPV6_FLOW_LABEL_RSS_CAP )) +return -EINVAL ; +if (cmd->flow_type ==TCP_V4_FLOW ) { rss_hash_cfg &= ~VNIC_RSS_CFG_REQ_HASH_TYPE_TCP_IPV4 ;if (tuple ==4 )
@@ -1732,10 +1739,15 @@ static int bnxt_set_rxfh_fields(struct net_device *dev,case AH_V6_FLOW :case ESP_V6_FLOW :case IPV6_FLOW : -if (tuple ==2 ) + rss_hash_cfg &= ~(VNIC_RSS_CFG_REQ_HASH_TYPE_IPV6 | +VNIC_RSS_CFG_REQ_HASH_TYPE_IPV6_FLOW_LABEL ); +if (!tuple) +break ; +if (cmd->data &RXH_IP6_FL ) + rss_hash_cfg |= +VNIC_RSS_CFG_REQ_HASH_TYPE_IPV6_FLOW_LABEL ; +else rss_hash_cfg |=VNIC_RSS_CFG_REQ_HASH_TYPE_IPV6 ; -else if (!tuple) - rss_hash_cfg &= ~VNIC_RSS_CFG_REQ_HASH_TYPE_IPV6 ;break ; }
Signature · 1 line
2.50.1
Michael Chan
maintainerRe: [PATCH net-next v3 3/4] eth: bnxt: support RSS on IPv6 Flow Label
Quoted 36 lines · Jakub Kicinski
It appears that the bnxt FW API has the relevant bit for Flow Label hashing. Plumb in the support. Obey the capability bit.
Signed-off-by Jakub Kicinski <[email protected]>---
v2:
- update the logic to pick the 2 tuple *OR* the FL bit
v1: https://lore.kernel.org/[email protected]drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c +6
-1@@ -1662,13 +1664,18 @@ static int bnxt_set_rxfh_fields(struct net_device *dev,if (cmd->data ==RXH_4TUPLE ) tuple =4 ; -else if (cmd->data ==RXH_2TUPLE ) +else if (cmd->data ==RXH_2TUPLE || + cmd->data == (RXH_2TUPLE |RXH_IP6_FL )) tuple =2 ;else if (!cmd->data) tuple =0 ;else return -EINVAL ; +if (cmd->data &RXH_IP6_FL && + !(bp->rss_cap &BNXT_RSS_CAP_IPV6_FLOW_LABEL_RSS_CAP )) +return -EINVAL ; +if (cmd->flow_type ==TCP_V4_FLOW ) { rss_hash_cfg &= ~VNIC_RSS_CFG_REQ_HASH_TYPE_TCP_IPV4 ;if (tuple ==4 )@@ -1732,10 +1739,15 @@ static int bnxt_set_rxfh_fields(struct net_device *dev,case AH_V6_FLOW :case ESP_V6_FLOW :
drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c static int bnxt_set_rxfh_fields(struct net_device *dev, · line 1749
Quoted 10 lines
drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c +8
-1case IPV6_FLOW : -if (tuple ==2 ) + rss_hash_cfg &= ~(VNIC_RSS_CFG_REQ_HASH_TYPE_IPV6 | +VNIC_RSS_CFG_REQ_HASH_TYPE_IPV6_FLOW_LABEL ); +if (!tuple) +break ; +if (cmd->data &RXH_IP6_FL ) + rss_hash_cfg |= +VNIC_RSS_CFG_REQ_HASH_TYPE_IPV6_FLOW_LABEL ; +else
I think else if (tuple == 2) here is more correct as you were asking earlier. Here tuple can be 2 or 4 and I think we only want to set this flag when it is 2 tuple. This will preserve the existing behavior.
Quoted 9 lines
drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c +0
-3rss_hash_cfg |=VNIC_RSS_CFG_REQ_HASH_TYPE_IPV6 ; - else if (!tuple) - rss_hash_cfg &= ~VNIC_RSS_CFG_REQ_HASH_TYPE_IPV6;break ; } --2.50.1
Jakub Kicinski
maintainerPATCH v3 4/4 selftests: drv-net: add test for RSS on flow label
In reply to Jakub KicinskiAdd a simple test for checking that RSS on flow label works, and that its rejected for IPv4 flows.
# ./tools/testing/selftests/drivers/net/hw/rss_flow_label.py TAP version 13 1..2 ok 1 rss_flow_label.test_rss_flow_label ok 2 rss_flow_label.test_rss_flow_label_6only # Totals: pass:2 fail:0 xfail:0 xpass:0 skip:0 error:0
---
v2:
- check for RPS / RFS
v1: https://lore.kernel.org/[email protected]
--- .../testing/selftests/drivers/net/hw/Makefile | 1 + .../drivers/net/hw/rss_flow_label.py | 167 ++++++++++++++++++ 2 files changed, 168 insertions(+) create mode 100755 tools/testing/selftests/drivers/net/hw/rss_flow_label.py
tools/testing/selftests/drivers/net/hw/Makefile +1 -0
@@ -18,6 +18,7 @@ TEST_PROGS = \ pp_alloc_fail.py \ rss_api.py \ rss_ctx.py \ + rss_flow_label.py \ rss_input_xfrm.py \ tso.py \ xsk_reconfig.py \
tools/testing/selftests/drivers/net/hw/rss_flow_label.py new file +167 -0
167 lines not on this page · raw message
Signature · 1 line
2.50.1