net/mlx5: HWS, Complex Matchers and rehash mechanism fixes
Subsystems (4)
Tariq Toukan
maintainercover net/mlx5: HWS, Complex Matchers and rehash mechanism fixes
Hi,
This series by Yevgeny adds Hardware Steering support for Complex Matchers/Rules (patches 1-5), and rehash mechanism fixes (patches 6-10).
See detailed descriptions by Yevgeny below [1][2].
Regards,
Tariq
[1]
Motivation:
----------
A matcher can match a certain set of match parameters. However, the number and size of match params for a single matcher are limited — all the parameters must fit within a single definer.
A common example of this limitation is IPv6 address matching, where matching both source and destination IPs requires more bits than a single definer can support.
SW Steering addresses this limitation by chaining multiple Steering Table Entries (STEs) within the same matcher, where each STE matches on a subset of the parameters.
In HW Steering, such chaining is not possible — the matcher's STEs are managed in a hash table, and a single definer is used to calculate the hash index for STEs.
Overview:
--------
To address this limitation in HW Steering, we introduce *Complex Matchers*, which consist of two chained matchers. This allows matching on twice as many parameters. Complex Matchers are filled with *Complex Rules* — rules that are split into two parts and inserted into their respective matchers.
The first half of the Complex Matcher is a regular matcher and points to the second half, which is an *Isolated Matcher*. An Isolated Matcher has its own isolated table and is accessible only by traffic coming from the first half of the Complex Matcher.
This splitting of matchers/rules into multiple parts is transparent to users. It is hidden behind the BWC HWS API. It becomes visible only when dumping steering debug information, where the Complex Matcher appears as two separate matchers: one in the user-created table and another in its isolated table.
Implementation Details:
----------------------
All user actions are performed on the second part of the rules only. The first part handles matching and applies two actions: modify header (set metadata, see details below) and go-to-table (directing traffic to the isolated table containing the isolated matcher).
Rule updates (updating rule actions) are applied to the second part of the rule since user-provided actions are not executed in the first matcher.
We use REG_C_6 metadata register to set and match on unique per-rule tag (see details below).
Splitting rules into two parts introduces new challenges:
1. Invalid Combinations
Consider two rules with different matching values:
- Rule 1: A+B
- Rule 2: C+D
Let's split the rules into two parts as follows:
|-----Complex Matcher-------| | | | 1st matcher 2nd matcher | | |---| |---| | | | A | | B | | | |---| -----> |---| | | | C | | D | | | |---| |---| | | | |---------------------------|
Splitting these rules results in invalid combinations: A+D and C+B:
any packet that matched on A will be forwarded to the 2nd matcher, where it will try to match on B (which is legal, and it is what the user asked for), but it will also try to match on D (which is not what the user asked for). To resolve this, we assign unique tags to each rule on the first matcher and match on these tags on the second matcher:
|----------| |---------| | A | | B, TagA | | action: | | | | set TagA | | | |----------| --> |---------| | C | | D, TagB | | action: | | | | set TagB | | | |----------| |---------|
2. Duplicated Entries:
Consider two rules with overlapping values:
- Rule 1: A+B
- Rule 2: A+D
Let's split the rules into two parts as follows:
|---| |---|
| A | | B |
|---| --> |---|
| | | D |
|---| |---|This leads to the duplicated entries on the first matcher, which HWS doesn't allow: subsequent delete of either of the rules will delete the only entry in the first matcher, leaving the remaining rule broken. To address this, we use a reference count for entries in the first matcher and delete STEs only when their refcount reaches zero.
Both challenges are resolved by having a per-matcher data structure (implemented with rhashtable) that manages refcounts for the first part of the rules and holds unique tags (managed via IDA) for these rules to set and to match on the second matcher.
Limitations:
-----------
We utilize metadata register REG_C_6 in this implementation, so its usage anywhere along the flow that might include the need for Complex Matcher is prohibited.
The number and size of match parameters remain limited — now constrained by what can be represented by two definers instead of one. This architectural limitation arises from the structure of Complex Matchers. If future requirements demand more parameters, Complex Matchers can be extended beyond two matchers.
Additionally, there is an implementation limit of 32 match parameters per matcher (disregarding parameter size). This limit can be lifted if needed.
Patches:
-------
- Patches 1-3: small additions/refactoring in preparation for Complex Matcher: exposed mlx5hws_table_ft_set_next_ft() in header, added definer function to convert field name enum to string, expose the polling function mlx5hws_bwc_queue_poll() in a header.
- Patch 4: in preparation for Complex Matcher, this patch adds support for Isolated Matcher.
- Patch 5: the main patch - Complex Matchers implementation.
[2]
Patch 6: fixing the usecase where rule insertion was failing, but rehash couldn't be initiated if the number of rules in the table is below the rehash threshold.
Patch 7: fixing the usecase where many rules in parallel would require rehash, due to the way the counting of rules was done.
Patch 8: fixing the case where rules were requiring action template extension in parallel, leading to unneeded extensions with the same templates.
Patch 9: refactor and simplify the rehash loop.
Patch 10: dump error completion details, which helps a lot in trying to understand what went wrong, especially during rehash.
Yevgeny Kliteynik (10):
net/mlx5: HWS, expose function mlx5hws_table_ft_set_next_ft in header net/mlx5: HWS, add definer function to get field name str net/mlx5: HWS, expose polling function in header file net/mlx5: HWS, introduce isolated matchers
net/mlx5: HWS, support complex matchers
net/mlx5: HWS, force rehash when rule insertion failed net/mlx5: HWS, fix counting of rules in the matcher net/mlx5: HWS, fix redundant extension of action templates net/mlx5: HWS, rework rehash loop
net/mlx5: HWS, dump bad completion details
.../mellanox/mlx5/core/steering/hws/bwc.c | 330 ++-- .../mellanox/mlx5/core/steering/hws/bwc.h | 11 + .../mlx5/core/steering/hws/bwc_complex.c | 1348 ++++++++++++++++- .../mlx5/core/steering/hws/bwc_complex.h | 21 + .../mellanox/mlx5/core/steering/hws/definer.c | 212 +++ .../mellanox/mlx5/core/steering/hws/definer.h | 2 + .../mellanox/mlx5/core/steering/hws/matcher.c | 278 +++- .../mellanox/mlx5/core/steering/hws/matcher.h | 9 + .../mellanox/mlx5/core/steering/hws/mlx5hws.h | 2 + .../mellanox/mlx5/core/steering/hws/send.c | 122 +- .../mellanox/mlx5/core/steering/hws/send.h | 1 + .../mellanox/mlx5/core/steering/hws/table.c | 16 +- .../mellanox/mlx5/core/steering/hws/table.h | 5 + 13 files changed, 2178 insertions(+), 179 deletions(-)
base-commit: 0b28182c73a3d013bcabbb890dc1070a8388f55a
Signature · 1 line
2.31.1
Tariq Toukan
maintainerPATCH 1/10 net/mlx5: HWS, expose function mlx5hws_table_ft_set_next_ft in header
From: Yevgeny Kliteynik <[email protected]>
In preparation for complex matcher support, make function mlx5hws_table_ft_set_next_ft() non-static and expose it in header.
--- .../mellanox/mlx5/core/steering/hws/table.c | 16 ++++++++-------- .../mellanox/mlx5/core/steering/hws/table.h | 5 +++++ 2 files changed, 13 insertions(+), 8 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/table.c +8 -8
@@ -342,10 +342,10 @@ int mlx5hws_table_ft_set_next_rtc(struct mlx5hws_context *ctx,return mlx5hws_cmd_flow_table_modify (ctx->mdev, &ft_attr, ft_id); } -static int hws_table_ft_set_next_ft (struct mlx5hws_context *ctx, -u32 ft_id, -u32 fw_ft_type, -u32 next_ft_id) +int mlx5hws_table_ft_set_next_ft (struct mlx5hws_context *ctx, +u32 ft_id, +u32 fw_ft_type, +u32 next_ft_id) {struct mlx5hws_cmd_ft_modify_attr ft_attr = {0 };
@@ -389,10 +389,10 @@ int mlx5hws_table_connect_to_miss_table(struct mlx5hws_table *src_tbl,if (dst_tbl) {if (list_empty (&dst_tbl->matchers_list)) {/* Connect src_tbl last_ft to dst_tbl start anchor */ - ret =hws_table_ft_set_next_ft (src_tbl->ctx, - last_ft_id, - src_tbl->fw_ft_type, - dst_tbl->ft_id); + ret =mlx5hws_table_ft_set_next_ft (src_tbl->ctx, + last_ft_id, + src_tbl->fw_ft_type, + dst_tbl->ft_id);if (ret)return ret;
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/table.h +5 -0
@@ -65,4 +65,9 @@ int mlx5hws_table_ft_set_next_rtc(struct mlx5hws_context *ctx,u32 rtc_0_id, u32 rtc_1_id); +int mlx5hws_table_ft_set_next_ft (struct mlx5hws_context *ctx, +u32 ft_id, +u32 fw_ft_type, +u32 next_ft_id); +#endif /* MLX5HWS_TABLE_H_ */
Signature · 1 line
2.31.1
Tariq Toukan
maintainerPATCH 2/10 net/mlx5: HWS, add definer function to get field name str
In reply to Tariq ToukanFrom: Yevgeny Kliteynik <[email protected]>
In preparation for complex matcher support, add function for converting definer fname to str, which will be used in following patches.
--- .../mellanox/mlx5/core/steering/hws/definer.c | 212 ++++++++++++++++++ .../mellanox/mlx5/core/steering/hws/definer.h | 2 + 2 files changed, 214 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/definer.c +212 -0
@@ -158,6 +158,218 @@ struct mlx5hws_definer_conv_data {u32 match_flags; }; +#define HWS_DEFINER_ENTRY (name)[MLX5HWS_DEFINER_FNAME_##name] = #name + +static const char *const hws_definer_fname_to_str[] = { +HWS_DEFINER_ENTRY (ETH_SMAC_47_16_O ), +HWS_DEFINER_ENTRY (ETH_SMAC_47_16_I ), +HWS_DEFINER_ENTRY (ETH_SMAC_15_0_O ), +HWS_DEFINER_ENTRY (ETH_SMAC_15_0_I ), +HWS_DEFINER_ENTRY (ETH_DMAC_47_16_O ), +HWS_DEFINER_ENTRY (ETH_DMAC_47_16_I ), +HWS_DEFINER_ENTRY (ETH_DMAC_15_0_O ), +HWS_DEFINER_ENTRY (ETH_DMAC_15_0_I ), +HWS_DEFINER_ENTRY (ETH_TYPE_O ), +HWS_DEFINER_ENTRY (ETH_TYPE_I ), +HWS_DEFINER_ENTRY (ETH_L3_TYPE_O ), +HWS_DEFINER_ENTRY (ETH_L3_TYPE_I ), +HWS_DEFINER_ENTRY (VLAN_TYPE_O ), +HWS_DEFINER_ENTRY (VLAN_TYPE_I ), +HWS_DEFINER_ENTRY (VLAN_FIRST_PRIO_O ), +HWS_DEFINER_ENTRY (VLAN_FIRST_PRIO_I ), +HWS_DEFINER_ENTRY (VLAN_CFI_O ), +HWS_DEFINER_ENTRY (VLAN_CFI_I ), +HWS_DEFINER_ENTRY (VLAN_ID_O ), +HWS_DEFINER_ENTRY (VLAN_ID_I ), +HWS_DEFINER_ENTRY (VLAN_SECOND_TYPE_O ), +HWS_DEFINER_ENTRY (VLAN_SECOND_TYPE_I ), +HWS_DEFINER_ENTRY (VLAN_SECOND_PRIO_O ), +HWS_DEFINER_ENTRY (VLAN_SECOND_PRIO_I ), +HWS_DEFINER_ENTRY (VLAN_SECOND_CFI_O ), +HWS_DEFINER_ENTRY (VLAN_SECOND_CFI_I ), +HWS_DEFINER_ENTRY (VLAN_SECOND_ID_O ), +HWS_DEFINER_ENTRY (VLAN_SECOND_ID_I ), +HWS_DEFINER_ENTRY (IPV4_IHL_O ), +HWS_DEFINER_ENTRY (IPV4_IHL_I ), +HWS_DEFINER_ENTRY (IP_DSCP_O ), +HWS_DEFINER_ENTRY (IP_DSCP_I ), +HWS_DEFINER_ENTRY (IP_ECN_O ), +HWS_DEFINER_ENTRY (IP_ECN_I ), +HWS_DEFINER_ENTRY (IP_TTL_O ), +HWS_DEFINER_ENTRY (IP_TTL_I ), +HWS_DEFINER_ENTRY (IPV4_DST_O ), +HWS_DEFINER_ENTRY (IPV4_DST_I ), +HWS_DEFINER_ENTRY (IPV4_SRC_O ), +HWS_DEFINER_ENTRY (IPV4_SRC_I ), +HWS_DEFINER_ENTRY (IP_VERSION_O ), +HWS_DEFINER_ENTRY (IP_VERSION_I ), +HWS_DEFINER_ENTRY (IP_FRAG_O ), +HWS_DEFINER_ENTRY (IP_FRAG_I ), +HWS_DEFINER_ENTRY (IP_LEN_O ), +HWS_DEFINER_ENTRY (IP_LEN_I ), +HWS_DEFINER_ENTRY (IP_TOS_O ), +HWS_DEFINER_ENTRY (IP_TOS_I ), +HWS_DEFINER_ENTRY (IPV6_FLOW_LABEL_O ), +HWS_DEFINER_ENTRY (IPV6_FLOW_LABEL_I ), +HWS_DEFINER_ENTRY (IPV6_DST_127_96_O ), +HWS_DEFINER_ENTRY (IPV6_DST_95_64_O ), +HWS_DEFINER_ENTRY (IPV6_DST_63_32_O ), +HWS_DEFINER_ENTRY (IPV6_DST_31_0_O ), +HWS_DEFINER_ENTRY (IPV6_DST_127_96_I ), +HWS_DEFINER_ENTRY (IPV6_DST_95_64_I ), +HWS_DEFINER_ENTRY (IPV6_DST_63_32_I ), +HWS_DEFINER_ENTRY (IPV6_DST_31_0_I ), +HWS_DEFINER_ENTRY (IPV6_SRC_127_96_O ), +HWS_DEFINER_ENTRY (IPV6_SRC_95_64_O ), +HWS_DEFINER_ENTRY (IPV6_SRC_63_32_O ), +HWS_DEFINER_ENTRY (IPV6_SRC_31_0_O ), +HWS_DEFINER_ENTRY (IPV6_SRC_127_96_I ), +HWS_DEFINER_ENTRY (IPV6_SRC_95_64_I ), +HWS_DEFINER_ENTRY (IPV6_SRC_63_32_I ), +HWS_DEFINER_ENTRY (IPV6_SRC_31_0_I ), +HWS_DEFINER_ENTRY (IP_PROTOCOL_O ), +HWS_DEFINER_ENTRY (IP_PROTOCOL_I ), +HWS_DEFINER_ENTRY (L4_SPORT_O ), +HWS_DEFINER_ENTRY (L4_SPORT_I ), +HWS_DEFINER_ENTRY (L4_DPORT_O ), +HWS_DEFINER_ENTRY (L4_DPORT_I ), +HWS_DEFINER_ENTRY (TCP_FLAGS_I ), +HWS_DEFINER_ENTRY (TCP_FLAGS_O ), +HWS_DEFINER_ENTRY (TCP_SEQ_NUM ), +HWS_DEFINER_ENTRY (TCP_ACK_NUM ), +HWS_DEFINER_ENTRY (GTP_TEID ), +HWS_DEFINER_ENTRY (GTP_MSG_TYPE ), +HWS_DEFINER_ENTRY (GTP_EXT_FLAG ), +HWS_DEFINER_ENTRY (GTP_NEXT_EXT_HDR ), +HWS_DEFINER_ENTRY (GTP_EXT_HDR_PDU ), +HWS_DEFINER_ENTRY (GTP_EXT_HDR_QFI ), +HWS_DEFINER_ENTRY (GTPU_DW0 ), +HWS_DEFINER_ENTRY (GTPU_FIRST_EXT_DW0 ), +HWS_DEFINER_ENTRY (GTPU_DW2 ), +HWS_DEFINER_ENTRY (FLEX_PARSER_0 ), +HWS_DEFINER_ENTRY (FLEX_PARSER_1 ), +HWS_DEFINER_ENTRY (FLEX_PARSER_2 ), +HWS_DEFINER_ENTRY (FLEX_PARSER_3 ), +HWS_DEFINER_ENTRY (FLEX_PARSER_4 ), +HWS_DEFINER_ENTRY (FLEX_PARSER_5 ), +HWS_DEFINER_ENTRY (FLEX_PARSER_6 ), +HWS_DEFINER_ENTRY (FLEX_PARSER_7 ), +HWS_DEFINER_ENTRY (VPORT_REG_C_0 ), +HWS_DEFINER_ENTRY (VXLAN_FLAGS ), +HWS_DEFINER_ENTRY (VXLAN_VNI ), +HWS_DEFINER_ENTRY (VXLAN_GPE_FLAGS ), +HWS_DEFINER_ENTRY (VXLAN_GPE_RSVD0 ), +HWS_DEFINER_ENTRY (VXLAN_GPE_PROTO ), +HWS_DEFINER_ENTRY (VXLAN_GPE_VNI ), +HWS_DEFINER_ENTRY (VXLAN_GPE_RSVD1 ), +HWS_DEFINER_ENTRY (GENEVE_OPT_LEN ), +HWS_DEFINER_ENTRY (GENEVE_OAM ), +HWS_DEFINER_ENTRY (GENEVE_PROTO ), +HWS_DEFINER_ENTRY (GENEVE_VNI ), +HWS_DEFINER_ENTRY (SOURCE_QP ), +HWS_DEFINER_ENTRY (SOURCE_GVMI ), +HWS_DEFINER_ENTRY (REG_0 ), +HWS_DEFINER_ENTRY (REG_1 ), +HWS_DEFINER_ENTRY (REG_2 ), +HWS_DEFINER_ENTRY (REG_3 ), +HWS_DEFINER_ENTRY (REG_4 ), +HWS_DEFINER_ENTRY (REG_5 ), +HWS_DEFINER_ENTRY (REG_6 ), +HWS_DEFINER_ENTRY (REG_7 ), +HWS_DEFINER_ENTRY (REG_8 ), +HWS_DEFINER_ENTRY (REG_9 ), +HWS_DEFINER_ENTRY (REG_10 ), +HWS_DEFINER_ENTRY (REG_11 ), +HWS_DEFINER_ENTRY (REG_A ), +HWS_DEFINER_ENTRY (REG_B ), +HWS_DEFINER_ENTRY (GRE_KEY_PRESENT ), +HWS_DEFINER_ENTRY (GRE_C ), +HWS_DEFINER_ENTRY (GRE_K ), +HWS_DEFINER_ENTRY (GRE_S ), +HWS_DEFINER_ENTRY (GRE_PROTOCOL ), +HWS_DEFINER_ENTRY (GRE_OPT_KEY ), +HWS_DEFINER_ENTRY (GRE_OPT_SEQ ), +HWS_DEFINER_ENTRY (GRE_OPT_CHECKSUM ), +HWS_DEFINER_ENTRY (INTEGRITY_O ), +HWS_DEFINER_ENTRY (INTEGRITY_I ), +HWS_DEFINER_ENTRY (ICMP_DW1 ), +HWS_DEFINER_ENTRY (ICMP_DW2 ), +HWS_DEFINER_ENTRY (ICMP_DW3 ), +HWS_DEFINER_ENTRY (IPSEC_SPI ), +HWS_DEFINER_ENTRY (IPSEC_SEQUENCE_NUMBER ), +HWS_DEFINER_ENTRY (IPSEC_SYNDROME ), +HWS_DEFINER_ENTRY (MPLS0_O ), +HWS_DEFINER_ENTRY (MPLS1_O ), +HWS_DEFINER_ENTRY (MPLS2_O ), +HWS_DEFINER_ENTRY (MPLS3_O ), +HWS_DEFINER_ENTRY (MPLS4_O ), +HWS_DEFINER_ENTRY (MPLS0_I ), +HWS_DEFINER_ENTRY (MPLS1_I ), +HWS_DEFINER_ENTRY (MPLS2_I ), +HWS_DEFINER_ENTRY (MPLS3_I ), +HWS_DEFINER_ENTRY (MPLS4_I ), +HWS_DEFINER_ENTRY (FLEX_PARSER0_OK ), +HWS_DEFINER_ENTRY (FLEX_PARSER1_OK ), +HWS_DEFINER_ENTRY (FLEX_PARSER2_OK ), +HWS_DEFINER_ENTRY (FLEX_PARSER3_OK ), +HWS_DEFINER_ENTRY (FLEX_PARSER4_OK ), +HWS_DEFINER_ENTRY (FLEX_PARSER5_OK ), +HWS_DEFINER_ENTRY (FLEX_PARSER6_OK ), +HWS_DEFINER_ENTRY (FLEX_PARSER7_OK ), +HWS_DEFINER_ENTRY (OKS2_MPLS0_O ), +HWS_DEFINER_ENTRY (OKS2_MPLS1_O ), +HWS_DEFINER_ENTRY (OKS2_MPLS2_O ), +HWS_DEFINER_ENTRY (OKS2_MPLS3_O ), +HWS_DEFINER_ENTRY (OKS2_MPLS4_O ), +HWS_DEFINER_ENTRY (OKS2_MPLS0_I ), +HWS_DEFINER_ENTRY (OKS2_MPLS1_I ), +HWS_DEFINER_ENTRY (OKS2_MPLS2_I ), +HWS_DEFINER_ENTRY (OKS2_MPLS3_I ), +HWS_DEFINER_ENTRY (OKS2_MPLS4_I ), +HWS_DEFINER_ENTRY (GENEVE_OPT_OK_0 ), +HWS_DEFINER_ENTRY (GENEVE_OPT_OK_1 ), +HWS_DEFINER_ENTRY (GENEVE_OPT_OK_2 ), +HWS_DEFINER_ENTRY (GENEVE_OPT_OK_3 ), +HWS_DEFINER_ENTRY (GENEVE_OPT_OK_4 ), +HWS_DEFINER_ENTRY (GENEVE_OPT_OK_5 ), +HWS_DEFINER_ENTRY (GENEVE_OPT_OK_6 ), +HWS_DEFINER_ENTRY (GENEVE_OPT_OK_7 ), +HWS_DEFINER_ENTRY (GENEVE_OPT_DW_0 ), +HWS_DEFINER_ENTRY (GENEVE_OPT_DW_1 ), +HWS_DEFINER_ENTRY (GENEVE_OPT_DW_2 ), +HWS_DEFINER_ENTRY (GENEVE_OPT_DW_3 ), +HWS_DEFINER_ENTRY (GENEVE_OPT_DW_4 ), +HWS_DEFINER_ENTRY (GENEVE_OPT_DW_5 ), +HWS_DEFINER_ENTRY (GENEVE_OPT_DW_6 ), +HWS_DEFINER_ENTRY (GENEVE_OPT_DW_7 ), +HWS_DEFINER_ENTRY (IB_L4_OPCODE ), +HWS_DEFINER_ENTRY (IB_L4_QPN ), +HWS_DEFINER_ENTRY (IB_L4_A ), +HWS_DEFINER_ENTRY (RANDOM_NUM ), +HWS_DEFINER_ENTRY (PTYPE_L2_O ), +HWS_DEFINER_ENTRY (PTYPE_L2_I ), +HWS_DEFINER_ENTRY (PTYPE_L3_O ), +HWS_DEFINER_ENTRY (PTYPE_L3_I ), +HWS_DEFINER_ENTRY (PTYPE_L4_O ), +HWS_DEFINER_ENTRY (PTYPE_L4_I ), +HWS_DEFINER_ENTRY (PTYPE_L4_EXT_O ), +HWS_DEFINER_ENTRY (PTYPE_L4_EXT_I ), +HWS_DEFINER_ENTRY (PTYPE_FRAG_O ), +HWS_DEFINER_ENTRY (PTYPE_FRAG_I ), +HWS_DEFINER_ENTRY (TNL_HDR_0 ), +HWS_DEFINER_ENTRY (TNL_HDR_1 ), +HWS_DEFINER_ENTRY (TNL_HDR_2 ), +HWS_DEFINER_ENTRY (TNL_HDR_3 ), + [MLX5HWS_DEFINER_FNAME_MAX ] ="DEFINER_FNAME_UNKNOWN" , +}; + +const char *mlx5hws_definer_fname_to_str (enum mlx5hws_definer_fname fname) +{ +if (fname >MLX5HWS_DEFINER_FNAME_MAX ) + fname =MLX5HWS_DEFINER_FNAME_MAX ; +return hws_definer_fname_to_str[fname]; +} + static void hws_definer_ones_set(struct mlx5hws_definer_fc *fc, void *match_param,
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/definer.h +2 -0
@@ -831,4 +831,6 @@ mlx5hws_definer_conv_match_params_to_compressed_fc(struct mlx5hws_context *ctx, u32 *match_param, int *fc_sz); +const char *mlx5hws_definer_fname_to_str (enum mlx5hws_definer_fname fname); +#endif /* HWS_DEFINER_H_ */
Signature · 1 line
2.31.1
Tariq Toukan
maintainerPATCH 3/10 net/mlx5: HWS, expose polling function in header file
In reply to Tariq ToukanFrom: Yevgeny Kliteynik <[email protected]>
In preparation for complex matcher, expose the function that is polling queue for completion (mlx5hws_bwc_queue_poll) in header file, so that it will be used by complex matcher code.
--- .../mellanox/mlx5/core/steering/hws/bwc.c | 29 ++++++++++--------- .../mellanox/mlx5/core/steering/hws/bwc.h | 5 ++++ 2 files changed, 21 insertions(+), 13 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/bwc.c +16 -13
@@ -223,10 +223,10 @@ int mlx5hws_bwc_matcher_destroy(struct mlx5hws_bwc_matcher *bwc_matcher)return 0 ; } -static int hws_bwc_queue_poll (struct mlx5hws_context *ctx, -u16 queue_id, -u32 *pending_rules, -bool drain) +int mlx5hws_bwc_queue_poll (struct mlx5hws_context *ctx, +u16 queue_id, +u32 *pending_rules, +bool drain) {unsigned long timeout = jiffies +secs_to_jiffies (MLX5HWS_BWC_POLLING_TIMEOUT );
@@ -361,7 +361,8 @@ hws_bwc_rule_destroy_hws_sync(struct mlx5hws_bwc_rule *bwc_rule,if (unlikely (ret))return ret; - ret =hws_bwc_queue_poll (ctx, rule_attr->queue_id, &expected_completions, true); + ret =mlx5hws_bwc_queue_poll (ctx, rule_attr->queue_id, + &expected_completions, true);if (unlikely (ret))return ret;
@@ -442,9 +443,8 @@ hws_bwc_rule_create_sync(struct mlx5hws_bwc_rule *bwc_rule,if (unlikely (ret))return ret; - ret =hws_bwc_queue_poll (ctx, rule_attr->queue_id, &expected_completions, true); - -return ret; +return mlx5hws_bwc_queue_poll (ctx, rule_attr->queue_id, + &expected_completions, true); } static int
@@ -465,7 +465,8 @@ hws_bwc_rule_update_sync(struct mlx5hws_bwc_rule *bwc_rule,if (unlikely (ret))return ret; - ret =hws_bwc_queue_poll (ctx, rule_attr->queue_id, &expected_completions, true); + ret =mlx5hws_bwc_queue_poll (ctx, rule_attr->queue_id, + &expected_completions, true);if (unlikely (ret))mlx5hws_err (ctx,"Failed updating BWC rule (%d)\n" , ret);
@@ -651,8 +652,10 @@ static int hws_bwc_matcher_move_all_simple(struct mlx5hws_bwc_matcher *bwc_match &bwc_matcher->rules[i]) ?NULL :list_next_entry (bwc_rules[i], list_node); - ret =hws_bwc_queue_poll (ctx, rule_attr.queue_id, - &pending_rules[i], false); + ret =mlx5hws_bwc_queue_poll (ctx, + rule_attr.queue_id, + &pending_rules[i], + false);if (unlikely (ret)) { mlx5hws_err(ctx, "Moving BWC rule failed during rehash (%d)\n",
@@ -669,8 +672,8 @@ static int hws_bwc_matcher_move_all_simple(struct mlx5hws_bwc_matcher *bwc_matchu16 queue_id =mlx5hws_bwc_get_queue_id (ctx, i);mlx5hws_send_engine_flush_queue (&ctx->send_queue[queue_id]); - ret =hws_bwc_queue_poll (ctx, queue_id, - &pending_rules[i], true); + ret =mlx5hws_bwc_queue_poll (ctx, queue_id, + &pending_rules[i], true);if (unlikely (ret)) {mlx5hws_err (ctx,"Moving BWC rule failed during rehash (%d)\n" , ret);
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/bwc.h +5 -0
@@ -64,6 +64,11 @@ void mlx5hws_bwc_rule_fill_attr(struct mlx5hws_bwc_matcher *bwc_matcher,u32 flow_source, struct mlx5hws_rule_attr *rule_attr); +int mlx5hws_bwc_queue_poll (struct mlx5hws_context *ctx, +u16 queue_id, +u32 *pending_rules, +bool drain); +static inline u16 mlx5hws_bwc_queues (struct mlx5hws_context *ctx) {/* Besides the control queue, half of the queues are
Signature · 1 line
2.31.1
Tariq Toukan
maintainerPATCH 4/10 net/mlx5: HWS, introduce isolated matchers
In reply to Tariq ToukanFrom: Yevgeny Kliteynik <[email protected]>
In preparation for complex matcher support, introduce the isolated matcher.
Isolated matcher is a matcher that has its own isolated table. It is used as the second half of the complex matcher: when the rule is split into two parts (complex rule), then matching on the first part will send the packet to the isolated matcher that will try to match on the second part. In case of miss, the packet goes back to the matcher's end flow table.
--- .../mellanox/mlx5/core/steering/hws/matcher.c | 278 +++++++++++++++++- .../mellanox/mlx5/core/steering/hws/matcher.h | 9 + .../mellanox/mlx5/core/steering/hws/mlx5hws.h | 2 + 3 files changed, 288 insertions(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/matcher.c +277 -1
@@ -23,19 +23,199 @@ static void hws_matcher_destroy_end_ft(struct mlx5hws_matcher *matcher)mlx5hws_table_destroy_default_ft (matcher->tbl, matcher->end_ft_id); } +int mlx5hws_matcher_update_end_ft_isolated (struct mlx5hws_table *tbl, +u32 miss_ft_id) +{ +struct mlx5hws_matcher *tmp_matcher; + +if (list_empty (&tbl->matchers_list)) +return -EINVAL ; + +/* Update isolated_matcher_end_ft_id attribute for all +* the matchers in isolated table. +*/ +list_for_each_entry (tmp_matcher, &tbl->matchers_list, list_node) + tmp_matcher->attr.isolated_matcher_end_ft_id = miss_ft_id; + + tmp_matcher =list_last_entry (&tbl->matchers_list, + struct mlx5hws_matcher, + list_node); + +return mlx5hws_table_ft_set_next_ft (tbl->ctx, + tmp_matcher->end_ft_id, + tbl->fw_ft_type, + miss_ft_id); +} + +static int hws_matcher_connect_end_ft_isolated (struct mlx5hws_matcher *matcher) +{ +struct mlx5hws_table *tbl = matcher->tbl; +u32 end_ft_id; +int ret; + +/* Reset end_ft next RTCs */ + ret =mlx5hws_table_ft_set_next_rtc (tbl->ctx, + matcher->end_ft_id, + matcher->tbl->fw_ft_type, +0 ,0 ); +if (ret) { +mlx5hws_err (tbl->ctx,"Isolated matcher: failed to reset FT's next RTCs\n" ); +return ret; + } + +/* Connect isolated matcher's end_ft to the complex matcher's end FT */ + end_ft_id = matcher->attr.isolated_matcher_end_ft_id; + ret =mlx5hws_table_ft_set_next_ft (tbl->ctx, + matcher->end_ft_id, + matcher->tbl->fw_ft_type, + end_ft_id); + +if (ret) { +mlx5hws_err (tbl->ctx,"Isolated matcher: failed to set FT's miss_ft_id\n" ); +return ret; + } + +return 0 ; +} + +static int hws_matcher_create_end_ft_isolated (struct mlx5hws_matcher *matcher) +{ +struct mlx5hws_table *tbl = matcher->tbl; +int ret; + + ret =mlx5hws_table_create_default_ft (tbl->ctx->mdev, + tbl, + &matcher->end_ft_id); +if (ret) { +mlx5hws_err (tbl->ctx,"Isolated matcher: failed to create end flow table\n" ); +return ret; + } + + ret =hws_matcher_connect_end_ft_isolated (matcher); +if (ret) { +mlx5hws_err (tbl->ctx,"Isolated matcher: failed to connect end FT\n" ); + goto destroy_default_ft; + } + +return 0 ; + +destroy_default_ft: +mlx5hws_table_destroy_default_ft (tbl, matcher->end_ft_id); +return ret; +} +static int hws_matcher_create_end_ft (struct mlx5hws_matcher *matcher) {struct mlx5hws_table *tbl = matcher->tbl;int ret; - ret =mlx5hws_table_create_default_ft (tbl->ctx->mdev, tbl, &matcher->end_ft_id); +if (mlx5hws_matcher_is_isolated (matcher)) + ret =hws_matcher_create_end_ft_isolated (matcher); +else + ret =mlx5hws_table_create_default_ft (tbl->ctx->mdev, tbl, + &matcher->end_ft_id); +if (ret) {mlx5hws_err (tbl->ctx,"Failed to create matcher end flow table\n" );return ret; } + +return 0 ; +} + +static int hws_matcher_connect_isolated_first (struct mlx5hws_matcher *matcher) +{ +struct mlx5hws_table *tbl = matcher->tbl; +struct mlx5hws_context *ctx = tbl->ctx; +int ret; + +/* Isolated matcher's end_ft is already pointing to the end_ft +* of the complex matcher - it was set at creation of end_ft, +* so no need to connect it. +* We still need to connect the isolated table's start FT to +* this matcher's RTC. +*/ + ret =mlx5hws_table_ft_set_next_rtc (ctx, + tbl->ft_id, + tbl->fw_ft_type, + matcher->match_ste.rtc_0_id, + matcher->match_ste.rtc_1_id); +if (ret) { +mlx5hws_err (ctx,"Isolated matcher: failed to connect start FT to match RTC\n" ); +return ret; + } + +/* Reset table's FT default miss (drop refcount) */ + ret =mlx5hws_table_ft_set_default_next_ft (tbl, tbl->ft_id); +if (ret) { +mlx5hws_err (ctx,"Isolated matcher: failed to reset table ft default miss\n" ); +return ret; + } + +list_add (&matcher->list_node, &tbl->matchers_list); + +return ret; +} + +static int hws_matcher_connect_isolated_last (struct mlx5hws_matcher *matcher) +{ +struct mlx5hws_table *tbl = matcher->tbl; +struct mlx5hws_context *ctx = tbl->ctx; +struct mlx5hws_matcher *last; +int ret; + + last =list_last_entry (&tbl->matchers_list, + struct mlx5hws_matcher, + list_node); + +/* New matcher's end_ft is already pointing to the end_ft of +* the complex matcher. +* Connect previous matcher's end_ft to this new matcher RTC. +*/ + ret =mlx5hws_table_ft_set_next_rtc (ctx, + last->end_ft_id, + tbl->fw_ft_type, + matcher->match_ste.rtc_0_id, + matcher->match_ste.rtc_1_id); +if (ret) { +mlx5hws_err (ctx, +"Isolated matcher: failed to connect matcher end_ft to new match RTC\n" ); +return ret; + } + +/* Reset prev matcher FT default miss (drop refcount) */ + ret =mlx5hws_table_ft_set_default_next_ft (tbl, last->end_ft_id); +if (ret) { +mlx5hws_err (ctx,"Isolated matcher: failed to reset matcher ft default miss\n" ); +return ret; + } + +/* Insert after the last matcher */ +list_add (&matcher->list_node, &last->list_node); +return 0 ; } +static int hws_matcher_connect_isolated (struct mlx5hws_matcher *matcher) +{ +/* Isolated matcher is expected to be the only one in its table. +* However, it can have a collision matcher, and it can go through +* rehash process, in which case we will temporary have both old and +* new matchers in the isolated table. +* Check if this is the first matcher in the isolated table. +*/ +if (list_empty (&matcher->tbl->matchers_list)) +return hws_matcher_connect_isolated_first (matcher); + +/* If this wasn't the first matcher, then we have 3 possible cases: +* - this is a collision matcher for the first matcher +* - this is a new rehash dest matcher +* - this is a collision matcher for the new rehash dest matcher +* The logic to add new matcher is the same for all these cases. +*/ +return hws_matcher_connect_isolated_last (matcher); +} +static int hws_matcher_connect (struct mlx5hws_matcher *matcher) {struct mlx5hws_table *tbl = matcher->tbl;
@@ -45,6 +225,9 @@ static int hws_matcher_connect(struct mlx5hws_matcher *matcher)struct mlx5hws_matcher *tmp_matcher;int ret; +if (mlx5hws_matcher_is_isolated (matcher)) +return hws_matcher_connect_isolated (matcher); +/* Find location in matcher list */ if (list_empty (&tbl->matchers_list)) {list_add (&matcher->list_node, &tbl->matchers_list);
@@ -121,6 +304,92 @@ static int hws_matcher_connect(struct mlx5hws_matcher *matcher)return ret; } +static int hws_matcher_disconnect_isolated (struct mlx5hws_matcher *matcher) +{ +struct mlx5hws_matcher *first, *last, *prev, *next; +struct mlx5hws_table *tbl = matcher->tbl; +struct mlx5hws_context *ctx = tbl->ctx; +u32 end_ft_id; +int ret; + + first =list_first_entry (&tbl->matchers_list, + struct mlx5hws_matcher, + list_node); + last =list_last_entry (&tbl->matchers_list, + struct mlx5hws_matcher, + list_node); + prev =list_prev_entry (matcher, list_node); + next =list_next_entry (matcher, list_node); + +list_del_init (&matcher->list_node); + +if (first == last) { +/* This was the only matcher in the list. +* Reset isolated table FT next RTCs and connect it +* to the whole complex matcher end FT instead. +*/ + ret =mlx5hws_table_ft_set_next_rtc (ctx, + tbl->ft_id, + tbl->fw_ft_type, +0 ,0 ); +if (ret) { +mlx5hws_err (tbl->ctx,"Isolated matcher: failed to reset FT's next RTCs\n" ); +return ret; + } + + end_ft_id = matcher->attr.isolated_matcher_end_ft_id; + ret =mlx5hws_table_ft_set_next_ft (tbl->ctx, + tbl->ft_id, + tbl->fw_ft_type, + end_ft_id); +if (ret) { +mlx5hws_err (tbl->ctx,"Isolated matcher: failed to set FT's miss_ft_id\n" ); +return ret; + } + +return 0 ; + } + +/* At this point we know that there are more matchers in the list */ + +if (matcher == first) { +/* We've disconnected the first matcher. +* Now update isolated table default FT. +*/ +if (!next) +return -EINVAL ; +return mlx5hws_table_ft_set_next_rtc (ctx, + tbl->ft_id, + tbl->fw_ft_type, + next->match_ste.rtc_0_id, + next->match_ste.rtc_1_id); + } + +if (matcher == last) { +/* If we've disconnected the last matcher - update prev +* matcher's end_ft to point to the complex matcher end_ft. +*/ +if (!prev) +return -EINVAL ; +return hws_matcher_connect_end_ft_isolated (prev); + } + +/* This wasn't the first or the last matcher, which means that it has +* both prev and next matchers. Note that this only happens if we're +* disconnecting collision matcher of the old matcher during rehash. +*/ +if (!prev || !next || + !(matcher->flags &MLX5HWS_MATCHER_FLAGS_COLLISION )) +return -EINVAL ; + +/* Update prev end FT to point to next match RTC */ +return mlx5hws_table_ft_set_next_rtc (ctx, + prev->end_ft_id, + tbl->fw_ft_type, + next->match_ste.rtc_0_id, + next->match_ste.rtc_1_id); +} +static int hws_matcher_disconnect (struct mlx5hws_matcher *matcher) {struct mlx5hws_matcher *next =NULL , *prev =NULL ;
@@ -128,6 +397,9 @@ static int hws_matcher_disconnect(struct mlx5hws_matcher *matcher)u32 prev_ft_id = tbl->ft_id;int ret; +if (mlx5hws_matcher_is_isolated (matcher)) +return hws_matcher_disconnect_isolated (matcher); +if (!list_is_first (&matcher->list_node, &tbl->matchers_list)) { prev =list_prev_entry (matcher, list_node); prev_ft_id = prev->end_ft_id;
@@ -531,6 +803,8 @@ hws_matcher_process_attr(struct mlx5hws_cmd_query_caps *caps, attr->table.sz_col_log =hws_matcher_rules_to_tbl_depth (attr->rule.num_log); matcher->flags |= attr->resizable ?MLX5HWS_MATCHER_FLAGS_RESIZABLE :0 ; + matcher->flags |= attr->isolated_matcher_end_ft_id ? +MLX5HWS_MATCHER_FLAGS_ISOLATED :0 ;return hws_matcher_check_attr_sz (caps, matcher); }
@@ -617,6 +891,8 @@ hws_matcher_create_col_matcher(struct mlx5hws_matcher *matcher) col_matcher->attr.table.sz_row_log -=MLX5HWS_MATCHER_ASSURED_ROW_RATIO ; col_matcher->attr.max_num_of_at_attach = matcher->attr.max_num_of_at_attach; + col_matcher->attr.isolated_matcher_end_ft_id = + matcher->attr.isolated_matcher_end_ft_id; ret =hws_matcher_process_attr (ctx->caps, col_matcher);if (ret)
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/matcher.h +9 -0
@@ -34,6 +34,7 @@ enum mlx5hws_matcher_offset {enum mlx5hws_matcher_flags {MLX5HWS_MATCHER_FLAGS_COLLISION =1 <<2 ,MLX5HWS_MATCHER_FLAGS_RESIZABLE =1 <<3 , +MLX5HWS_MATCHER_FLAGS_ISOLATED =1 <<4 , };struct mlx5hws_match_template {
@@ -96,9 +97,17 @@ static inline bool mlx5hws_matcher_is_in_resize(struct mlx5hws_matcher *matcher) return !!matcher->resize_dst; } +static inline bool mlx5hws_matcher_is_isolated (struct mlx5hws_matcher *matcher) +{ +return !!(matcher->flags &MLX5HWS_MATCHER_FLAGS_ISOLATED ); +} +static inline bool mlx5hws_matcher_is_insert_by_idx (struct mlx5hws_matcher *matcher) {return matcher->attr.insert_mode ==MLX5HWS_MATCHER_INSERT_BY_INDEX ; } +int mlx5hws_matcher_update_end_ft_isolated (struct mlx5hws_table *tbl, +u32 miss_ft_id); +#endif /* HWS_MATCHER_H_ */
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws.h +2 -0
@@ -119,6 +119,8 @@ struct mlx5hws_matcher_attr { };/* Optional AT attach configuration - Max number of additional AT */ u8 max_num_of_at_attach; +/* Optional end FT (miss FT ID) for match RTC (for isolated matcher) */ +u32 isolated_matcher_end_ft_id; };struct mlx5hws_rule_attr {
Signature · 1 line
2.31.1
Tariq Toukan
maintainerPATCH 5/10 net/mlx5: HWS, support complex matchers
In reply to Tariq ToukanFrom: Yevgeny Kliteynik <[email protected]>
This patch adds support for Complex Matchers/Rules
Overview:
--------
A matcher can match on a certain set of match parameters. However, the number and size of match params for a single matcher are limited: all the parameters must fit within a single definer.
A common example of this limitation is IPv6 address matching, where matching both source and destination IPs requires more bits than a single definer can support.
SW Steering addresses this limitation by chaining multiple Steering Table Entries (STEs) within the same matcher, where each STE matches on a subset of the parameters.
In HW Steering, such chaining is not possible — the matcher's STEs are managed in a hash table, and a single definer is used to calculate the hash index for STEs.
To address this limitation in HW Steering, we introduce Complex Matchers, which consist of two chained matchers. This allows matching on twice as many parameters. Complex Matchers are filled with Complex Rules — rules that are split into two parts and inserted into their respective matchers.
The first half of the Complex Matcher is a regular matcher and points to the second half, which is an Isolated Matcher. An Isolated Matcher has its own isolated table and is accessible only by traffic coming from the first half of the Complex Matcher.
This splitting of matchers/rules into multiple parts is transparent to users. It is hidden under the BWC HWS API. It becomes visible only when dumping steering debug information, where the Complex Matcher appears as two separate matchers: one in the user-created table and another in its isolated table.
Some implementation details:
---------------------------
All user actions are performed on the second part of the rules only. The first part handles matching and applies two actions: modify header (set metadata, see details below) and go-to-table (directing traffic to the isolated table containing the isolated matcher).
Rule updates (updating rule actions) are applied to the second part of the rule since user-provided actions are not executed in the first matcher.
We use REG_C_6 metadata register to set and match on unique per-rule tag (see details below).
Splitting rules into two parts introduces new challenges:
1. Invalid Combinations
Consider two rules with different matching values:
- Rule 1: A+B
- Rule 2: C+D
Let's split the rules into two parts as follows:
|---| |---| | A | | B | |---| --> |---| | C | | D | |---| |---|
Splitting these rules results in invalid combinations like A+D and C+B.
To resolve this, we assign unique tags to each rule on the first matcher and match these tags on the second matcher (the tag is implemented through modify_hdr action that sets value to metadata register REG_C_6):
|----------| |---------| | A | | B, TagA | | action: | | | | set TagA | | | |----------| --> |---------| | C | | D, TagB | | action: | | | | set TagB | | | |----------| |---------|
2. Duplicated Entries:
Consider two rules with overlapping values:
- Rule 1: A+B
- Rule 2: A+D
Let's split the rules into two parts as follows:
|---| |---|
| A | | B |
|---| --> |---|
| | | D |
|---| |---|This leads to the duplicated entries on the first matcher, which HWS doesn't allow: subsequent delete of either of the rules will delete the only entry in the first matcher, leaving the remaining rule broken.
To address this, we use a reference count for entries in the first matcher and delete STEs only when their refcount reaches zero.
Both challenges are resolved by having a per-matcher data structure (implemented with rhashtable) that manages refcounts for the first part of the rules and holds unique tags (managed via IDA) for these rules to set and to match on the second matcher.
Limitations:
-----------
We utilize metadata register REG_C_6 in this implementation, so its usage anywhere along the steering of the flow that might include the need for Complex Matcher is prohibited.
The number and size of match parameters remain limited — now it is constrained by what can be represented by two definers instead of one. This architectural limitation arises from the structure of Complex Matchers. If future requirements demand more parameters, Complex Matchers can be extended beyond two matchers.
Additionally, there is an implementation limit of 32 match parameters per rule (disregarding parameter size). This limit can be lifted if needed.
--- .../mellanox/mlx5/core/steering/hws/bwc.c | 63 +- .../mellanox/mlx5/core/steering/hws/bwc.h | 5 + .../mlx5/core/steering/hws/bwc_complex.c | 1348 ++++++++++++++++- .../mlx5/core/steering/hws/bwc_complex.h | 21 + 4 files changed, 1410 insertions(+), 27 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/bwc.c +47 -16
@@ -46,10 +46,14 @@ static void hws_bwc_unlock_all_queues(struct mlx5hws_context *ctx) } } -static void hws_bwc_matcher_init_attr (struct mlx5hws_matcher_attr *attr, +static void hws_bwc_matcher_init_attr (struct mlx5hws_bwc_matcher *bwc_matcher,u32 priority, -u8 size_log) +u8 size_log, +struct mlx5hws_matcher_attr *attr) { +struct mlx5hws_bwc_matcher *first_matcher = + bwc_matcher->complex_first_bwc_matcher; +memset (attr,0 ,sizeof (*attr)); attr->priority = priority;
@@ -61,6 +65,9 @@ static void hws_bwc_matcher_init_attr(struct mlx5hws_matcher_attr *attr, attr->rule.num_log = size_log; attr->resizable = true; attr->max_num_of_at_attach =MLX5HWS_BWC_MATCHER_ATTACH_AT_NUM ; + + attr->isolated_matcher_end_ft_id = + first_matcher ? first_matcher->matcher->end_ft_id :0 ; } int mlx5hws_bwc_matcher_create_simple(struct mlx5hws_bwc_matcher *bwc_matcher,
@@ -83,9 +90,10 @@ int mlx5hws_bwc_matcher_create_simple(struct mlx5hws_bwc_matcher *bwc_matcher,for (i =0 ; i < bwc_queues; i++)INIT_LIST_HEAD (&bwc_matcher->rules[i]); -hws_bwc_matcher_init_attr (&attr, +hws_bwc_matcher_init_attr (bwc_matcher, priority, -MLX5HWS_BWC_MATCHER_INIT_SIZE_LOG ); +MLX5HWS_BWC_MATCHER_INIT_SIZE_LOG , + &attr); bwc_matcher->priority = priority; bwc_matcher->size_log =MLX5HWS_BWC_MATCHER_INIT_SIZE_LOG ;
@@ -217,7 +225,10 @@ int mlx5hws_bwc_matcher_destroy(struct mlx5hws_bwc_matcher *bwc_matcher)"BWC matcher destroy: matcher still has %d rules\n" , num_of_rules); -mlx5hws_bwc_matcher_destroy_simple (bwc_matcher); +if (bwc_matcher->complex) +mlx5hws_bwc_matcher_destroy_complex (bwc_matcher); +else +mlx5hws_bwc_matcher_destroy_simple (bwc_matcher);kfree (bwc_matcher);return 0 ;
@@ -401,9 +412,13 @@ int mlx5hws_bwc_rule_destroy_simple(struct mlx5hws_bwc_rule *bwc_rule)int mlx5hws_bwc_rule_destroy (struct mlx5hws_bwc_rule *bwc_rule) { -int ret; +bool is_complex = !!bwc_rule->bwc_matcher->complex; +int ret =0 ; - ret =mlx5hws_bwc_rule_destroy_simple (bwc_rule); +if (is_complex) + ret =mlx5hws_bwc_rule_destroy_complex (bwc_rule); +else + ret =mlx5hws_bwc_rule_destroy_simple (bwc_rule);mlx5hws_bwc_rule_free (bwc_rule);return ret;
@@ -692,7 +707,10 @@ static int hws_bwc_matcher_move_all_simple(struct mlx5hws_bwc_matcher *bwc_matchstatic int hws_bwc_matcher_move_all (struct mlx5hws_bwc_matcher *bwc_matcher) { -return hws_bwc_matcher_move_all_simple (bwc_matcher); +if (!bwc_matcher->complex) +return hws_bwc_matcher_move_all_simple (bwc_matcher); + +return mlx5hws_bwc_matcher_move_all_complex (bwc_matcher); }static int hws_bwc_matcher_move (struct mlx5hws_bwc_matcher *bwc_matcher)
@@ -703,9 +721,10 @@ static int hws_bwc_matcher_move(struct mlx5hws_bwc_matcher *bwc_matcher)struct mlx5hws_matcher *new_matcher;int ret; -hws_bwc_matcher_init_attr (&matcher_attr, +hws_bwc_matcher_init_attr (bwc_matcher, bwc_matcher->priority, - bwc_matcher->size_log); + bwc_matcher->size_log, + &matcher_attr); old_matcher = bwc_matcher->matcher; new_matcher = mlx5hws_matcher_create(old_matcher->tbl,
@@ -910,11 +929,18 @@ mlx5hws_bwc_rule_create(struct mlx5hws_bwc_matcher *bwc_matcher, bwc_queue_idx =hws_bwc_gen_queue_idx (ctx); - ret =mlx5hws_bwc_rule_create_simple (bwc_rule, - params->match_buf, - rule_actions, - flow_source, - bwc_queue_idx); +if (bwc_matcher->complex) + ret =mlx5hws_bwc_rule_create_complex (bwc_rule, + params, + flow_source, + rule_actions, + bwc_queue_idx); +else + ret =mlx5hws_bwc_rule_create_simple (bwc_rule, + params->match_buf, + rule_actions, + flow_source, + bwc_queue_idx);if (unlikely (ret)) {mlx5hws_bwc_rule_free (bwc_rule);return NULL ;
@@ -996,5 +1022,10 @@ int mlx5hws_bwc_rule_action_update(struct mlx5hws_bwc_rule *bwc_rule,return -EINVAL ; } -return hws_bwc_rule_action_update (bwc_rule, rule_actions); +/* For complex rule, the update should happen on the second matcher */ +if (bwc_rule->isolated_bwc_rule) +return hws_bwc_rule_action_update (bwc_rule->isolated_bwc_rule, + rule_actions); +else +return hws_bwc_rule_action_update (bwc_rule, rule_actions); }
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/bwc.h +5 -0
@@ -18,10 +18,13 @@#define MLX5HWS_BWC_POLLING_TIMEOUT 60 +struct mlx5hws_bwc_matcher_complex_data ;struct mlx5hws_bwc_matcher {struct mlx5hws_matcher *matcher;struct mlx5hws_match_template *mt;struct mlx5hws_action_template **at; +struct mlx5hws_bwc_matcher_complex_data *complex; +struct mlx5hws_bwc_matcher *complex_first_bwc_matcher;u8 num_of_at;u8 size_of_at_array;u8 size_log;
@@ -33,6 +36,8 @@ struct mlx5hws_bwc_matcher {struct mlx5hws_bwc_rule {struct mlx5hws_bwc_matcher *bwc_matcher;struct mlx5hws_rule *rule; +struct mlx5hws_bwc_rule *isolated_bwc_rule; +struct mlx5hws_bwc_complex_rule_hash_node *complex_hash_node;u16 bwc_queue_idx;struct list_head list_node; };
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/bwc_complex.c +1337 -11
1,382 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/bwc_complex.h +21 -0
@@ -4,6 +4,27 @@#ifndef HWS_BWC_COMPLEX_H_ #define HWS_BWC_COMPLEX_H_ +struct mlx5hws_bwc_complex_rule_hash_node { +u32 match_buf[MLX5_ST_SZ_DW_MATCH_PARAM ]; +u32 tag; +refcount_t refcount; +bool rtc_valid; +u32 rtc_0; +u32 rtc_1; +struct rhash_head hash_node; +}; + +struct mlx5hws_bwc_matcher_complex_data { +struct mlx5hws_table *isolated_tbl; +struct mlx5hws_bwc_matcher *isolated_bwc_matcher; +struct mlx5hws_action *action_metadata; +struct mlx5hws_action *action_go_to_tbl; +struct mlx5hws_action *action_last; +struct rhashtable refcount_hash; +struct mutex hash_lock;/* Protect the refcount rhashtable */ +struct ida metadata_ida; +}; +bool mlx5hws_bwc_match_params_is_complex (struct mlx5hws_context *ctx,u8 match_criteria_enable,struct mlx5hws_match_parameters *mask);
Signature · 1 line
2.31.1
Tariq Toukan
maintainerPATCH 6/10 net/mlx5: HWS, force rehash when rule insertion failed
In reply to Tariq ToukanFrom: Yevgeny Kliteynik <[email protected]>
Rules are inserted into hash table in accordance with their hash index. When a certain number of rules is reached, the table is rehashed:
a bigger new table is allocated and all the rules are moved there. But sometimes a new rule can't be inserted into the hash table because its index is full, even though the number of rules in the table is well below the threshold. The hash function is not perfect, so such cases are not rare. When that happens, we want to do the same rehash, in order to increase the table size and lower the probability for such cases.
This patch fixes the usecase where rule insertion was failing, but rehash couldn't be initiated due to low number of rules: it adds flag that denotes that rehash is required, even if the number of rules in the table is below the rehash threshold.
--- .../net/ethernet/mellanox/mlx5/core/steering/hws/bwc.c | 8 ++++++-- .../net/ethernet/mellanox/mlx5/core/steering/hws/bwc.h | 1 + 2 files changed, 7 insertions(+), 2 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/bwc.c +6 -2
@@ -169,6 +169,7 @@ mlx5hws_bwc_matcher_create(struct mlx5hws_table *table,return NULL ;atomic_set (&bwc_matcher->num_of_rules,0 ); +atomic_set (&bwc_matcher->rehash_required, false);/* Check if the required match params can be all matched * in single STE, otherwise complex matcher is needed.
@@ -769,9 +770,9 @@ hws_bwc_matcher_rehash_size(struct mlx5hws_bwc_matcher *bwc_matcher)/* It is possible that other rule has already performed rehash. * Need to check again if we really need rehash. -* If the reason for rehash was size, but not any more - skip rehash. */ -if (!hws_bwc_matcher_rehash_size_needed (bwc_matcher, +if (!atomic_read (&bwc_matcher->rehash_required) && + !hws_bwc_matcher_rehash_size_needed (bwc_matcher,atomic_read (&bwc_matcher->num_of_rules)))return 0 ;
@@ -782,6 +783,8 @@ hws_bwc_matcher_rehash_size(struct mlx5hws_bwc_matcher *bwc_matcher)* - destroy the old matcher */ +atomic_set (&bwc_matcher->rehash_required, false); + ret =hws_bwc_matcher_extend_size (bwc_matcher);if (ret)return ret;
@@ -875,6 +878,7 @@ int mlx5hws_bwc_rule_create_simple(struct mlx5hws_bwc_rule *bwc_rule,* Try rehash by size and insert rule again - last chance. */ +atomic_set (&bwc_matcher->rehash_required, true);mutex_unlock (queue_lock);hws_bwc_lock_all_queues (ctx);
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/bwc.h +1 -0
@@ -30,6 +30,7 @@ struct mlx5hws_bwc_matcher {u8 size_log;u32 priority;atomic_t num_of_rules; +atomic_t rehash_required;struct list_head *rules; };
Signature · 1 line
2.31.1
Tariq Toukan
maintainerPATCH 7/10 net/mlx5: HWS, fix counting of rules in the matcher
In reply to Tariq ToukanFrom: Yevgeny Kliteynik <[email protected]>
Currently the counter that counts number of rules in a matcher is increased only when rule insertion is completed. In a multi-threaded usecase this can lead to a scenario that many rules can be in process of insertion in the same matcher, while none of them has completed the insertion and the rule counter is not updated. This results in a rule insertion failure for many of them at first attempt, which leads to all of them requiring rehash and requiring locking of all the queue locks.
This patch fixes the case by increasing the rule counter in the beginning of insertion process and decreasing in case of any failure.
--- .../net/ethernet/mellanox/mlx5/core/steering/hws/bwc.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/bwc.c +5 -5
@@ -341,16 +341,12 @@ static void hws_bwc_rule_list_add(struct mlx5hws_bwc_rule *bwc_rule, u16 idx) {struct mlx5hws_bwc_matcher *bwc_matcher = bwc_rule->bwc_matcher; -atomic_inc (&bwc_matcher->num_of_rules); bwc_rule->bwc_queue_idx = idx;list_add (&bwc_rule->list_node, &bwc_matcher->rules[idx]); }static void hws_bwc_rule_list_remove (struct mlx5hws_bwc_rule *bwc_rule) { -struct mlx5hws_bwc_matcher *bwc_matcher = bwc_rule->bwc_matcher; - -atomic_dec (&bwc_matcher->num_of_rules);list_del_init (&bwc_rule->list_node); }
@@ -404,6 +400,7 @@ int mlx5hws_bwc_rule_destroy_simple(struct mlx5hws_bwc_rule *bwc_rule)mutex_lock (queue_lock); ret =hws_bwc_rule_destroy_hws_sync (bwc_rule, &attr); +atomic_dec (&bwc_matcher->num_of_rules);hws_bwc_rule_list_remove (bwc_rule);mutex_unlock (queue_lock);
@@ -840,7 +837,7 @@ int mlx5hws_bwc_rule_create_simple(struct mlx5hws_bwc_rule *bwc_rule, }/* check if number of rules require rehash */ - num_of_rules =atomic_read (&bwc_matcher->num_of_rules); + num_of_rules =atomic_inc_return (&bwc_matcher->num_of_rules);if (unlikely (hws_bwc_matcher_rehash_size_needed (bwc_matcher, num_of_rules))) {mutex_unlock (queue_lock);
@@ -854,6 +851,7 @@ int mlx5hws_bwc_rule_create_simple(struct mlx5hws_bwc_rule *bwc_rule, bwc_matcher->size_log -MLX5HWS_BWC_MATCHER_SIZE_LOG_STEP , bwc_matcher->size_log, ret); +atomic_dec (&bwc_matcher->num_of_rules);return ret; }
@@ -887,6 +885,7 @@ int mlx5hws_bwc_rule_create_simple(struct mlx5hws_bwc_rule *bwc_rule,if (ret) {mlx5hws_err (ctx,"BWC rule insertion: rehash failed (%d)\n" , ret); +atomic_dec (&bwc_matcher->num_of_rules);return ret; }
@@ -902,6 +901,7 @@ int mlx5hws_bwc_rule_create_simple(struct mlx5hws_bwc_rule *bwc_rule,if (unlikely (ret)) {mutex_unlock (queue_lock);mlx5hws_err (ctx,"BWC rule insertion failed (%d)\n" , ret); +atomic_dec (&bwc_matcher->num_of_rules);return ret; }
Signature · 1 line
2.31.1
Tariq Toukan
maintainerPATCH 8/10 net/mlx5: HWS, fix redundant extension of action templates
In reply to Tariq ToukanFrom: Yevgeny Kliteynik <[email protected]>
When a rule is inserted into a matcher, we search for the suitable action template. If such template is not found, action template array is extended with the new template. However, when several threads are performing this in parallel, there is a race - we can end up with extending the action templates array with the same template.
This patch is doing the following:
- refactor the code to find action template index in rule create and update, have the common code in an auxiliary function
- after locking all the queues, check again if the action template array still needs to be extended
--- .../mellanox/mlx5/core/steering/hws/bwc.c | 105 +++++++++--------- 1 file changed, 54 insertions(+), 51 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/bwc.c +54 -51
@@ -789,6 +789,53 @@ hws_bwc_matcher_rehash_size(struct mlx5hws_bwc_matcher *bwc_matcher)return hws_bwc_matcher_move (bwc_matcher); } +static int hws_bwc_rule_get_at_idx (struct mlx5hws_bwc_rule *bwc_rule, +struct mlx5hws_rule_action rule_actions[], +u16 bwc_queue_idx) +{ +struct mlx5hws_bwc_matcher *bwc_matcher = bwc_rule->bwc_matcher; +struct mlx5hws_context *ctx = bwc_matcher->matcher->tbl->ctx; +struct mutex *queue_lock;/* Protect the queue */ +int at_idx, ret; + +/* check if rehash needed due to missing action template */ + at_idx =hws_bwc_matcher_find_at (bwc_matcher, rule_actions); +if (likely (at_idx >=0 )) +return at_idx; + +/* we need to extend BWC matcher action templates array */ + queue_lock =hws_bwc_get_queue_lock (ctx, bwc_queue_idx); +mutex_unlock (queue_lock); +hws_bwc_lock_all_queues (ctx); + +/* check again - perhaps other thread already did extend_at */ + at_idx =hws_bwc_matcher_find_at (bwc_matcher, rule_actions); +if (at_idx >=0 ) + goto out; + + ret =hws_bwc_matcher_extend_at (bwc_matcher, rule_actions); +if (unlikely (ret)) { +mlx5hws_err (ctx,"BWC rule: failed extending AT (%d)" , ret); + at_idx = -EINVAL ; + goto out; + } + +/* action templates array was extended, we need the last idx */ + at_idx = bwc_matcher->num_of_at -1 ; + ret =mlx5hws_matcher_attach_at (bwc_matcher->matcher, + bwc_matcher->at[at_idx]); +if (unlikely (ret)) { +mlx5hws_err (ctx,"BWC rule: failed attaching new AT (%d)" , ret); + at_idx = -EINVAL ; + goto out; + } + +out: +hws_bwc_unlock_all_queues (ctx); +mutex_lock (queue_lock); +return at_idx; +} + int mlx5hws_bwc_rule_create_simple(struct mlx5hws_bwc_rule *bwc_rule, u32 *match_param, struct mlx5hws_rule_action rule_actions[],
@@ -809,31 +856,12 @@ int mlx5hws_bwc_rule_create_simple(struct mlx5hws_bwc_rule *bwc_rule,mutex_lock (queue_lock); -/* check if rehash needed due to missing action template */ - at_idx =hws_bwc_matcher_find_at (bwc_matcher, rule_actions); + at_idx =hws_bwc_rule_get_at_idx (bwc_rule, rule_actions, bwc_queue_idx);if (unlikely (at_idx <0 )) { -/* we need to extend BWC matcher action templates array */ mutex_unlock (queue_lock); -hws_bwc_lock_all_queues (ctx); - - ret =hws_bwc_matcher_extend_at (bwc_matcher, rule_actions); -if (unlikely (ret)) { -hws_bwc_unlock_all_queues (ctx); -return ret; - } - -/* action templates array was extended, we need the last idx */ - at_idx = bwc_matcher->num_of_at -1 ; - - ret =mlx5hws_matcher_attach_at (bwc_matcher->matcher, - bwc_matcher->at[at_idx]); -if (unlikely (ret)) { -hws_bwc_unlock_all_queues (ctx); -return ret; - } - -hws_bwc_unlock_all_queues (ctx); -mutex_lock (queue_lock); +mlx5hws_err (ctx,"BWC rule create: failed getting AT (%d)" , + ret); +return -EINVAL ; }/* check if number of rules require rehash */
@@ -971,36 +999,11 @@ hws_bwc_rule_action_update(struct mlx5hws_bwc_rule *bwc_rule,mutex_lock (queue_lock); -/* check if rehash needed due to missing action template */ - at_idx =hws_bwc_matcher_find_at (bwc_matcher, rule_actions); + at_idx =hws_bwc_rule_get_at_idx (bwc_rule, rule_actions, idx);if (unlikely (at_idx <0 )) { -/* we need to extend BWC matcher action templates array */ mutex_unlock (queue_lock); -hws_bwc_lock_all_queues (ctx); - -/* check again - perhaps other thread already did extend_at */ - at_idx =hws_bwc_matcher_find_at (bwc_matcher, rule_actions); -if (likely (at_idx <0 )) { - ret =hws_bwc_matcher_extend_at (bwc_matcher, rule_actions); -if (unlikely (ret)) { -hws_bwc_unlock_all_queues (ctx); -mlx5hws_err (ctx,"BWC rule update: failed extending AT (%d)" , ret); -return -EINVAL ; - } - -/* action templates array was extended, we need the last idx */ - at_idx = bwc_matcher->num_of_at -1 ; - - ret =mlx5hws_matcher_attach_at (bwc_matcher->matcher, - bwc_matcher->at[at_idx]); -if (unlikely (ret)) { -hws_bwc_unlock_all_queues (ctx); -return ret; - } - } - -hws_bwc_unlock_all_queues (ctx); -mutex_lock (queue_lock); +mlx5hws_err (ctx,"BWC rule update: failed getting AT\n" ); +return -EINVAL ; } ret = hws_bwc_rule_update_sync(bwc_rule,
Signature · 1 line
2.31.1
Tariq Toukan
maintainerPATCH 9/10 net/mlx5: HWS, rework rehash loop
In reply to Tariq ToukanFrom: Yevgeny Kliteynik <[email protected]>
Reworking the rehash loop - simplifying the code and making it less error prone:
- Instead of doing round-robin on all the queues with batch of rules in each cycle, just go over all the queues and move all the rules that belong to this queue.
- If at some stage of moving the rule we get a failure (which should not happen), this can't be rolled back. So instead of aborting rehash and leaving the matcher in a broken state, allow the loop to continue: attempt to move the rest of the rules and delete the old matcher. A rule that failed to move to a new matcher will loose its match STE once the rehash is completed and the old matcher is deleted, so the rule won't match any traffic any more. This rule's packets will fall back to the steering pipeline w/o HW offload. Rehash procedure will return an error, which will cause the rule insertion to fail for the rule that started this whole rehash.
--- .../mellanox/mlx5/core/steering/hws/bwc.c | 127 +++++++----------- 1 file changed, 52 insertions(+), 75 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/bwc.c +52 -75
@@ -610,95 +610,69 @@ hws_bwc_matcher_find_at(struct mlx5hws_bwc_matcher *bwc_matcher,static int hws_bwc_matcher_move_all_simple (struct mlx5hws_bwc_matcher *bwc_matcher) { +bool move_error = false, poll_error = false, drain_error = false;struct mlx5hws_context *ctx = bwc_matcher->matcher->tbl->ctx; +struct mlx5hws_matcher *matcher = bwc_matcher->matcher;u16 bwc_queues =mlx5hws_bwc_queues (ctx); -struct mlx5hws_bwc_rule **bwc_rules;struct mlx5hws_rule_attr rule_attr; -u32 *pending_rules; -int i, j, ret =0 ; -bool all_done; -u16 burst_th; +struct mlx5hws_bwc_rule *bwc_rule; +struct mlx5hws_send_engine *queue; +struct list_head *rules_list; +u32 pending_rules; +int i, ret =0 ;mlx5hws_bwc_rule_fill_attr (bwc_matcher,0 ,0 , &rule_attr); - pending_rules =kcalloc (bwc_queues,sizeof (*pending_rules),GFP_KERNEL ); -if (!pending_rules) -return -ENOMEM ; - - bwc_rules =kcalloc (bwc_queues,sizeof (*bwc_rules),GFP_KERNEL ); -if (!bwc_rules) { - ret = -ENOMEM ; - goto free_pending_rules; - } -for (i =0 ; i < bwc_queues; i++) {if (list_empty (&bwc_matcher->rules[i])) - bwc_rules[i] =NULL ; -else - bwc_rules[i] =list_first_entry (&bwc_matcher->rules[i], - struct mlx5hws_bwc_rule, - list_node); - } +continue ; -do { - all_done = true; + pending_rules =0 ; + rule_attr.queue_id =mlx5hws_bwc_get_queue_id (ctx, i); + rules_list = &bwc_matcher->rules[i]; -for (i =0 ; i < bwc_queues; i++) { - rule_attr.queue_id =mlx5hws_bwc_get_queue_id (ctx, i); - burst_th =hws_bwc_get_burst_th (ctx, rule_attr.queue_id); - -for (j =0 ; j < burst_th && bwc_rules[i]; j++) { - rule_attr.burst = !!((j +1 ) % burst_th); - ret =mlx5hws_matcher_resize_rule_move (bwc_matcher->matcher, - bwc_rules[i]->rule, - &rule_attr); -if (unlikely (ret)) { -mlx5hws_err (ctx, -"Moving BWC rule failed during rehash (%d)\n" , - ret); - goto free_bwc_rules; - } +list_for_each_entry (bwc_rule, rules_list, list_node) { + ret =mlx5hws_matcher_resize_rule_move (matcher, + bwc_rule->rule, + &rule_attr); +if (unlikely (ret && !move_error)) { +mlx5hws_err (ctx, +"Moving BWC rule: move failed (%d), attempting to move rest of the rules\n" , + ret); + move_error = true; + } - all_done = false; - pending_rules[i]++; - bwc_rules[i] =list_is_last (&bwc_rules[i]->list_node, - &bwc_matcher->rules[i]) ? -NULL :list_next_entry (bwc_rules[i], list_node); - - ret =mlx5hws_bwc_queue_poll (ctx, - rule_attr.queue_id, - &pending_rules[i], - false); -if (unlikely (ret)) { -mlx5hws_err (ctx, -"Moving BWC rule failed during rehash (%d)\n" , - ret); - goto free_bwc_rules; - } + pending_rules++; + ret =mlx5hws_bwc_queue_poll (ctx, + rule_attr.queue_id, + &pending_rules, + false); +if (unlikely (ret && !poll_error)) { +mlx5hws_err (ctx, +"Moving BWC rule: poll failed (%d), attempting to move rest of the rules\n" , + ret); + poll_error = true; } } - }while (!all_done); - -/* drain all the bwc queues */ -for (i =0 ; i < bwc_queues; i++) { -if (pending_rules[i]) { -u16 queue_id =mlx5hws_bwc_get_queue_id (ctx, i); -mlx5hws_send_engine_flush_queue (&ctx->send_queue[queue_id]); - ret =mlx5hws_bwc_queue_poll (ctx, queue_id, - &pending_rules[i], true); -if (unlikely (ret)) { +if (pending_rules) { + queue = &ctx->send_queue[rule_attr.queue_id]; +mlx5hws_send_engine_flush_queue (queue); + ret =mlx5hws_bwc_queue_poll (ctx, + rule_attr.queue_id, + &pending_rules, + true); +if (unlikely (ret && !drain_error)) {mlx5hws_err (ctx, -"Moving BWC rule failed during rehash (%d)\n" , ret); - goto free_bwc_rules; +"Moving BWC rule: drain failed (%d), attempting to move rest of the rules\n" , + ret); + drain_error = true; } } } -free_bwc_rules: -kfree (bwc_rules); -free_pending_rules: -kfree (pending_rules); +if (move_error || poll_error || drain_error) + ret = -EINVAL ;return ret; }
@@ -742,15 +716,18 @@ static int hws_bwc_matcher_move(struct mlx5hws_bwc_matcher *bwc_matcher) } ret =hws_bwc_matcher_move_all (bwc_matcher); -if (ret) { -mlx5hws_err (ctx,"Rehash error: moving rules failed\n" ); -return -ENOMEM ; - } +if (ret) +mlx5hws_err (ctx,"Rehash error: moving rules failed, attempting to remove the old matcher\n" ); + +/* Error during rehash can't be rolled back. +* The best option here is to allow the rehash to complete and remove +* the old matcher - can't leave the matcher in the 'in_resize' state. +*/ bwc_matcher->matcher = new_matcher;mlx5hws_matcher_destroy (old_matcher); -return 0 ; +return ret; } static int
Signature · 1 line
2.31.1
Tariq Toukan
maintainerPATCH 10/10 net/mlx5: HWS, dump bad completion details
In reply to Tariq ToukanFrom: Yevgeny Kliteynik <[email protected]>
Failing to insert/delete a rule should not happen. If it does happen, it would be good to know at which stage it happened and what was the failure. This patch adds printing of bad CQE details.
--- .../mellanox/mlx5/core/steering/hws/send.c | 122 +++++++++++++++++- .../mellanox/mlx5/core/steering/hws/send.h | 1 + 2 files changed, 120 insertions(+), 3 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/send.c +119 -3
@@ -344,18 +344,133 @@ hws_send_engine_update_rule_resize(struct mlx5hws_send_engine *queue, } } +static void hws_send_engine_dump_error_cqe (struct mlx5hws_send_engine *queue, +struct mlx5hws_send_ring_priv *priv, +struct mlx5_cqe64 *cqe) +{ +u8 wqe_opcode = cqe ?be32_to_cpu (cqe->sop_drop_qpn) >>24 :0 ; +struct mlx5hws_context *ctx = priv->rule->matcher->tbl->ctx; +u32 opcode = cqe ?get_cqe_opcode (cqe) :0 ; +struct mlx5hws_rule *rule = priv->rule; + +/* If something bad happens and lots of rules are failing, we don't +* want to pollute dmesg. Print only the first bad cqe per engine, +* the one that started the avalanche. +*/ +if (queue->error_cqe_printed) +return ; + + queue->error_cqe_printed = true; + +if (mlx5hws_rule_move_in_progress (rule)) +mlx5hws_err (ctx, +"--- rule 0x%08llx: error completion moving rule: phase %s, wqes left %d\n" , +HWS_PTR_TO_ID (rule), + rule->resize_info->state == +MLX5HWS_RULE_RESIZE_STATE_WRITING ?"WRITING" : + rule->resize_info->state == +MLX5HWS_RULE_RESIZE_STATE_DELETING ?"DELETING" : +"UNKNOWN" , + rule->pending_wqes); +else +mlx5hws_err (ctx, +"--- rule 0x%08llx: error completion %s (%d), wqes left %d\n" , +HWS_PTR_TO_ID (rule), + rule->status == +MLX5HWS_RULE_STATUS_CREATING ?"CREATING" : + rule->status == +MLX5HWS_RULE_STATUS_DELETING ?"DELETING" : + rule->status == +MLX5HWS_RULE_STATUS_FAILING ?"FAILING" : + rule->status == +MLX5HWS_RULE_STATUS_UPDATING ?"UPDATING" :"NA" , + rule->status, + rule->pending_wqes); + +mlx5hws_err (ctx," rule 0x%08llx: matcher 0x%llx %s\n" , +HWS_PTR_TO_ID (rule), +HWS_PTR_TO_ID (rule->matcher), + (rule->matcher->flags &MLX5HWS_MATCHER_FLAGS_ISOLATED ) ? +"(isolated)" :"" ); + +if (!cqe) { +mlx5hws_err (ctx," rule 0x%08llx: no CQE\n" , +HWS_PTR_TO_ID (rule)); +return ; + } + +mlx5hws_err (ctx," rule 0x%08llx: cqe->opcode = %d %s\n" , +HWS_PTR_TO_ID (rule), opcode, + opcode ==MLX5_CQE_REQ ?"(MLX5_CQE_REQ)" : + opcode ==MLX5_CQE_REQ_ERR ?"(MLX5_CQE_REQ_ERR)" :" " ); + +if (opcode ==MLX5_CQE_REQ_ERR ) { +struct mlx5_err_cqe *err_cqe = (struct mlx5_err_cqe *)cqe; + +mlx5hws_err (ctx, +" rule 0x%08llx: |--- hw_error_syndrome = 0x%x\n" , +HWS_PTR_TO_ID (rule), + err_cqe->rsvd1[16 ]); +mlx5hws_err (ctx, +" rule 0x%08llx: |--- hw_syndrome_type = 0x%x\n" , +HWS_PTR_TO_ID (rule), + err_cqe->rsvd1[17 ] >>4 ); +mlx5hws_err (ctx, +" rule 0x%08llx: |--- vendor_err_synd = 0x%x\n" , +HWS_PTR_TO_ID (rule), + err_cqe->vendor_err_synd); +mlx5hws_err (ctx, +" rule 0x%08llx: |--- syndrome = 0x%x\n" , +HWS_PTR_TO_ID (rule), + err_cqe->syndrome); + } + +mlx5hws_err (ctx, +" rule 0x%08llx: cqe->byte_cnt = 0x%08x\n" , +HWS_PTR_TO_ID (rule),be32_to_cpu (cqe->byte_cnt)); +mlx5hws_err (ctx, +" rule 0x%08llx: |-- UPDATE STATUS = %s\n" , +HWS_PTR_TO_ID (rule), + (be32_to_cpu (cqe->byte_cnt) &0x80000000 ) ? +"FAILURE" :"SUCCESS" ); +mlx5hws_err (ctx, +" rule 0x%08llx: |------- SYNDROME = %s\n" , +HWS_PTR_TO_ID (rule), + ((be32_to_cpu (cqe->byte_cnt) &0x00000003 ) ==1 ) ? +"SET_FLOW_FAIL" : + ((be32_to_cpu (cqe->byte_cnt) &0x00000003 ) ==2 ) ? +"DISABLE_FLOW_FAIL" :"UNKNOWN" ); +mlx5hws_err (ctx, +" rule 0x%08llx: cqe->sop_drop_qpn = 0x%08x\n" , +HWS_PTR_TO_ID (rule),be32_to_cpu (cqe->sop_drop_qpn)); +mlx5hws_err (ctx, +" rule 0x%08llx: |-send wqe opcode = 0x%02x %s\n" , +HWS_PTR_TO_ID (rule), wqe_opcode, + wqe_opcode ==MLX5HWS_WQE_OPCODE_TBL_ACCESS ? +"(MLX5HWS_WQE_OPCODE_TBL_ACCESS)" :"(UNKNOWN)" ); +mlx5hws_err (ctx, +" rule 0x%08llx: |------------ qpn = 0x%06x\n" , +HWS_PTR_TO_ID (rule), +be32_to_cpu (cqe->sop_drop_qpn) &0xffffff ); +} +static void hws_send_engine_update_rule (struct mlx5hws_send_engine *queue,struct mlx5hws_send_ring_priv *priv,u16 wqe_cnt, -enum mlx5hws_flow_op_status *status) +enum mlx5hws_flow_op_status *status, +struct mlx5_cqe64 *cqe) { priv->rule->pending_wqes--; -if (*status ==MLX5HWS_FLOW_OP_ERROR ) { +if (unlikely (*status ==MLX5HWS_FLOW_OP_ERROR )) {if (priv->retry_id) { +/* If there is a retry_id, then it's not an error yet, +* retry to insert this rule in the collision RTC. +*/ hws_send_engine_retry_post_send (queue, priv, wqe_cnt);return ; } +hws_send_engine_dump_error_cqe (queue, priv, cqe);/* Some part of the rule failed */ priv->rule->status =MLX5HWS_RULE_STATUS_FAILING ; *priv->used_id =0 ;
@@ -420,7 +535,8 @@ static void hws_send_engine_update(struct mlx5hws_send_engine *queue,if (priv->user_data) {if (priv->rule) { -hws_send_engine_update_rule (queue, priv, wqe_cnt, &status); +hws_send_engine_update_rule (queue, priv, wqe_cnt, + &status, cqe);/* Completion is provided on the last rule WQE */ if (priv->rule->pending_wqes)return ;
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/send.h +1 -0
@@ -140,6 +140,7 @@ struct mlx5hws_send_engine {u16 used_entries;u16 num_entries;bool err; +bool error_cqe_printed;struct mutex lock;/* Protects the send engine */ };
Signature · 1 line
2.31.1
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Quoted 8 lines · On Sun, 11 May 2025 22:38:00 +0300 you wrote:
Hi,
This series by Yevgeny adds Hardware Steering support for Complex Matchers/Rules (patches 1-5), and rehash mechanism fixes (patches 6-10).
See detailed descriptions by Yevgeny below [1][2].
[...]
Here is the summary with links:
- [net-next,01/10] net/mlx5: HWS, expose function mlx5hws_table_ft_set_next_ft in header
https://git.kernel.org/netdev/net-next/c/d2338a27fcee - [net-next,02/10] net/mlx5: HWS, add definer function to get field name str https://git.kernel.org/netdev/net-next/c/fed5f4831281 - [net-next,03/10] net/mlx5: HWS, expose polling function in header file https://git.kernel.org/netdev/net-next/c/3c739d1624e3 - [net-next,04/10] net/mlx5: HWS, introduce isolated matchers https://git.kernel.org/netdev/net-next/c/b816743a182f - [net-next,05/10] net/mlx5: HWS, support complex matchers https://git.kernel.org/netdev/net-next/c/17e0accac577 - [net-next,06/10] net/mlx5: HWS, force rehash when rule insertion failed https://git.kernel.org/netdev/net-next/c/9d4024edce10 - [net-next,07/10] net/mlx5: HWS, fix counting of rules in the matcher https://git.kernel.org/netdev/net-next/c/4c56b5cbc323 - [net-next,08/10] net/mlx5: HWS, fix redundant extension of action templates https://git.kernel.org/netdev/net-next/c/041861b40f59 - [net-next,09/10] net/mlx5: HWS, rework rehash loop https://git.kernel.org/netdev/net-next/c/ef94799a8741 - [net-next,10/10] net/mlx5: HWS, dump bad completion details https://git.kernel.org/netdev/net-next/c/578b856b5e72
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html