net/mlx5: HWS, Optimize matchers ICM usage
Subsystems (4)
Mark Bloch
maintainerv3 cover net/mlx5: HWS, Optimize matchers ICM usage
This series optimizes ICM usage for unidirectional rules and empty matchers and with the last patch we make hardware steering the default FDB steering provider for NICs that don't support software steering.
Hardware steering (HWS) uses a type of rule table container (RTC) that is unidirectional, so matchers consist of two RTCs to accommodate bidirectional rules.
This small series enables resizing the two RTCs independently by tracking the number of rules separately. For extreme cases where all rules are unidirectional, this results in saving close to half the memory footprint.
Results for inserting 1M unidirectional rules using a simple module:
Pages Memory
Before this patch: 300k 1.5GiB
After this patch: 160k 900MiB
The 'Pages' column measures the number of 4KiB pages the device requests for itself (the ICM).
The 'Memory' column is the difference between peak usage and baseline usage (before starting the test) as reported by free -h.
In addition, second to last patch of the series handles a case where all the matcher's rules were deleted: the large RTCs of the matcher are no longer required, and we can save some more ICM by shrinking the matcher to its initial size.
Finally the last patch makes hardware steering the default mode when in swichdev for NICs that don't have software steering support.
Changelog
=========
Changes from v1:
- Fixed author on patches 5 and 6.
Changes from v2:
- Added patch 4 that does only code refactoring.
- Patch 5: this patch now contains the functional change only, w/o refactoring.
- Patch 7: tracking flow_source in the BWC rule to resolve issue with rule_update of a unidirectional rule.
- Added patch 8 that does code rearranging to prevent forward declaration in the following patch.
- Patch 9: removed forward declaration of function.
Moshe Shemesh (1):
net/mlx5: Add HWS as secondary steering mode
Vlad Dogaru (6):
net/mlx5: HWS, remove unused create_dest_array parameter net/mlx5: HWS, Export rule skip logic
net/mlx5: HWS, Refactor rule skip logic net/mlx5: HWS, Create STEs directly from matcher net/mlx5: HWS, Decouple matcher RX and TX sizes net/mlx5: HWS, Track matcher sizes individually
Yevgeny Kliteynik (3):
net/mlx5: HWS, remove incorrect comment net/mlx5: HWS, Rearrange to prevent forward declaration net/mlx5: HWS, Shrink empty matchers
.../net/ethernet/mellanox/mlx5/core/fs_core.c | 2 + .../mellanox/mlx5/core/steering/hws/action.c | 7 +- .../mellanox/mlx5/core/steering/hws/bwc.c | 529 ++++++++++++------ .../mellanox/mlx5/core/steering/hws/bwc.h | 15 +- .../mellanox/mlx5/core/steering/hws/debug.c | 20 +- .../mellanox/mlx5/core/steering/hws/fs_hws.c | 15 +- .../mellanox/mlx5/core/steering/hws/matcher.c | 166 ++++-- .../mellanox/mlx5/core/steering/hws/matcher.h | 3 +- .../mellanox/mlx5/core/steering/hws/mlx5hws.h | 36 +- .../mellanox/mlx5/core/steering/hws/rule.c | 34 +- .../mellanox/mlx5/core/steering/hws/rule.h | 3 + 11 files changed, 526 insertions(+), 304 deletions(-)
base-commit: 5f712c3877f99d5b5e4d011955c6467ae0e535a6
Signature · 1 line
2.34.1
Mark Bloch
maintainerPATCH v3 1/10 net/mlx5: HWS, remove unused create_dest_array parameter
From: Vlad Dogaru <[email protected]>
flow_source is not used anywhere in mlx5hws_action_create_dest_array.
--- .../mellanox/mlx5/core/steering/hws/action.c | 7 ++----- .../mellanox/mlx5/core/steering/hws/fs_hws.c | 15 ++++++--------- .../mellanox/mlx5/core/steering/hws/mlx5hws.h | 8 ++------ 3 files changed, 10 insertions(+), 20 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/action.c +2 -5
@@ -1358,12 +1358,9 @@ mlx5hws_action_create_modify_header(struct mlx5hws_context *ctx, }struct mlx5hws_action * -mlx5hws_action_create_dest_array (struct mlx5hws_context *ctx, -size_t num_dest, +mlx5hws_action_create_dest_array (struct mlx5hws_context *ctx,size_t num_dest,struct mlx5hws_action_dest_attr *dests, -bool ignore_flow_level, -u32 flow_source, -u32 flags) +bool ignore_flow_level,u32 flags) {struct mlx5hws_cmd_set_fte_dest *dest_list =NULL ;struct mlx5hws_cmd_ft_create_attr ft_attr = {0 };
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/fs_hws.c +6 -9
@@ -571,14 +571,12 @@ static void mlx5_fs_put_dest_action_sampler(struct mlx5_fs_hws_context *fs_ctx,static struct mlx5hws_action *mlx5_fs_create_action_dest_array (struct mlx5hws_context *ctx,struct mlx5hws_action_dest_attr *dests, -u32 num_of_dests,bool ignore_flow_level, -u32 flow_source) +u32 num_of_dests,bool ignore_flow_level) {u32 flags =MLX5HWS_ACTION_FLAG_HWS_FDB |MLX5HWS_ACTION_FLAG_SHARED ;return mlx5hws_action_create_dest_array (ctx, num_of_dests, dests, - ignore_flow_level, - flow_source, flags); + ignore_flow_level, flags); } static struct mlx5hws_action *
@@ -1015,7 +1013,6 @@ static int mlx5_fs_fte_get_hws_actions(struct mlx5_flow_root_namespace *ns, } (*ractions)[num_actions++].action = dest_actions->dest; }else if (num_dest_actions >1 ) { -u32 flow_source = fte->act_dests.flow_context.flow_source;bool ignore_flow_level; if (num_actions == MLX5_FLOW_CONTEXT_ACTION_MAX ||
@@ -1025,10 +1022,10 @@ static int mlx5_fs_fte_get_hws_actions(struct mlx5_flow_root_namespace *ns, } ignore_flow_level = !!(fte_action->flags &FLOW_ACT_IGNORE_FLOW_LEVEL ); - tmp_action =mlx5_fs_create_action_dest_array (ctx, dest_actions, - num_dest_actions, - ignore_flow_level, - flow_source); + tmp_action = +mlx5_fs_create_action_dest_array (ctx, dest_actions, + num_dest_actions, + ignore_flow_level);if (!tmp_action) { err = -EOPNOTSUPP ; goto free_actions;
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws.h +2 -6
@@ -727,18 +727,14 @@ mlx5hws_action_create_push_vlan(struct mlx5hws_context *ctx, u32 flags);* @dests: The destination array. Each contains a destination action and can * have additional actions. * @ignore_flow_level: Whether to turn on 'ignore_flow_level' for this dest. -* @flow_source: Source port of the traffic for this actions. * @flags: Action creation flags (enum mlx5hws_action_flags). * * Return: pointer to mlx5hws_action on success NULL otherwise. */ struct mlx5hws_action * -mlx5hws_action_create_dest_array (struct mlx5hws_context *ctx, -size_t num_dest, +mlx5hws_action_create_dest_array (struct mlx5hws_context *ctx,size_t num_dest,struct mlx5hws_action_dest_attr *dests, -bool ignore_flow_level, -u32 flow_source, -u32 flags); +bool ignore_flow_level,u32 flags);/** * mlx5hws_action_create_insert_header - Create insert header action.
Signature · 1 line
2.34.1
Mark Bloch
maintainerPATCH v3 2/10 net/mlx5: HWS, remove incorrect comment
In reply to Mark BlochFrom: Yevgeny Kliteynik <[email protected]>
Removing incorrect comment section that is probably some copy-paste artifact.
--- drivers/net/ethernet/mellanox/mlx5/core/steering/hws/bwc.c | 2 -- 1 file changed, 2 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/bwc.c +0 -2
@@ -876,8 +876,6 @@ int mlx5hws_bwc_rule_create_simple(struct mlx5hws_bwc_rule *bwc_rule,/* At this point the rule wasn't added. * It could be because there was collision, or some other problem. -* If we don't dive deeper than API, the only thing we know is that -* the status of completion is RTE_FLOW_OP_ERROR. * Try rehash by size and insert rule again - last chance. */
Signature · 1 line
2.34.1
Mark Bloch
maintainerPATCH v3 3/10 net/mlx5: HWS, Export rule skip logic
In reply to Mark BlochFrom: Vlad Dogaru <[email protected]>
The bwc layer will use mlx5hws_rule_skip to keep track of numbers of RX and TX rules individually, so export this function for future usage.
--- .../net/ethernet/mellanox/mlx5/core/steering/hws/rule.c | 9 ++++----- .../net/ethernet/mellanox/mlx5/core/steering/hws/rule.h | 3 +++ 2 files changed, 7 insertions(+), 5 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/rule.c +4 -5
@@ -3,10 +3,8 @@#include "internal.h" -static void hws_rule_skip (struct mlx5hws_matcher *matcher, -struct mlx5hws_match_template *mt, -u32 flow_source, -bool *skip_rx,bool *skip_tx) +void mlx5hws_rule_skip (struct mlx5hws_matcher *matcher,u32 flow_source, +bool *skip_rx,bool *skip_tx) {/* By default FDB rules are added to both RX and TX */ *skip_rx = false;
@@ -66,7 +64,8 @@ static void hws_rule_init_dep_wqe(struct mlx5hws_send_ring_dep_wqe *dep_wqe, attr->rule_idx : 0;if (tbl->type ==MLX5HWS_TABLE_TYPE_FDB ) { -hws_rule_skip (matcher, mt, attr->flow_source, &skip_rx, &skip_tx); +mlx5hws_rule_skip (matcher, attr->flow_source, + &skip_rx, &skip_tx);if (!skip_rx) { dep_wqe->rtc_0 = matcher->match_ste.rtc_0_id;
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/rule.h +3 -0
@@ -69,6 +69,9 @@ struct mlx5hws_rule {*/ }; +void mlx5hws_rule_skip (struct mlx5hws_matcher *matcher,u32 flow_source, +bool *skip_rx,bool *skip_tx); +void mlx5hws_rule_free_action_ste (struct mlx5hws_action_ste_chunk *action_ste); int mlx5hws_rule_move_hws_remove(struct mlx5hws_rule *rule,
Signature · 1 line
2.34.1
Re: [PATCH net-next v3 03/10] net/mlx5: HWS, Export rule skip logic
Quoted 8 lines · Mark Bloch
From: Vlad Dogaru <[email protected]>
The bwc layer will use
mlx5hws_rule_skipto keep track of numbers of RX and TX rules individually, so export this function for future usage.Signed-off-by Vlad Dogaru <[email protected]>Reviewed-by Yevgeny Kliteynik <[email protected]>Signed-off-by Mark Bloch <[email protected]>
Mark Bloch
maintainerPATCH v3 4/10 net/mlx5: HWS, Refactor rule skip logic
In reply to Mark BlochFrom: Vlad Dogaru <[email protected]>
Reduce nesting by adding a couple of early return statements.
--- .../mellanox/mlx5/core/steering/hws/rule.c | 25 ++++++++++--------- 1 file changed, 13 insertions(+), 12 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/rule.c +13 -12
@@ -12,20 +12,21 @@ void mlx5hws_rule_skip(struct mlx5hws_matcher *matcher, u32 flow_source,if (flow_source ==MLX5_FLOW_CONTEXT_FLOW_SOURCE_LOCAL_VPORT ) { *skip_rx = true; - }else if (flow_source ==MLX5_FLOW_CONTEXT_FLOW_SOURCE_UPLINK ) { +return ; + } + +if (flow_source ==MLX5_FLOW_CONTEXT_FLOW_SOURCE_UPLINK ) { *skip_tx = true; - }else { -/* If no flow source was set for current rule, -* check for flow source in matcher attributes. -*/ -if (matcher->attr.optimize_flow_src) { - *skip_tx = - matcher->attr.optimize_flow_src ==MLX5HWS_MATCHER_FLOW_SRC_WIRE ; - *skip_rx = - matcher->attr.optimize_flow_src ==MLX5HWS_MATCHER_FLOW_SRC_VPORT ; -return ; - } +return ; } + +/* If no flow source was set for current rule, +* check for flow source in matcher attributes. +*/ + *skip_tx = matcher->attr.optimize_flow_src == +MLX5HWS_MATCHER_FLOW_SRC_WIRE ; + *skip_rx = matcher->attr.optimize_flow_src == +MLX5HWS_MATCHER_FLOW_SRC_VPORT ; } static void
Signature · 1 line
2.34.1
Re: [PATCH net-next v3 04/10] net/mlx5: HWS, Refactor rule skip logic
Quoted 7 lines · Mark Bloch
From: Vlad Dogaru <[email protected]>
Reduce nesting by adding a couple of early return statements.
Signed-off-by Vlad Dogaru <[email protected]>Reviewed-by Yevgeny Kliteynik <[email protected]>Signed-off-by Mark Bloch <[email protected]>
Mark Bloch
maintainerPATCH v3 5/10 net/mlx5: HWS, Create STEs directly from matcher
In reply to Mark BlochFrom: Vlad Dogaru <[email protected]>
Matchers were using the pool abstraction solely as a convenience to allocate two STE ranges. The pool's core functionality, that of allocating individual items from the range, was unused. Matchers rely either on the hardware to hash rules into a table, or on a user-provided index.
Remove the STE pool from the matcher and allocate the STE ranges manually instead.
--- .../mellanox/mlx5/core/steering/hws/debug.c | 10 +-- .../mellanox/mlx5/core/steering/hws/matcher.c | 71 ++++++++++--------- .../mellanox/mlx5/core/steering/hws/matcher.h | 3 +- 3 files changed, 41 insertions(+), 43 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/debug.c +3 -7
@@ -118,7 +118,6 @@ static int hws_debug_dump_matcher(struct seq_file *f, struct mlx5hws_matcher *ma {enum mlx5hws_table_type tbl_type = matcher->tbl->type;struct mlx5hws_cmd_ft_query_attr ft_attr = {0 }; -struct mlx5hws_pool *ste_pool;u64 icm_addr_0 =0 ;u64 icm_addr_1 =0 ;u32 ste_0_id =-1 ;
@@ -133,12 +132,9 @@ static int hws_debug_dump_matcher(struct seq_file *f, struct mlx5hws_matcher *ma matcher->end_ft_id, matcher->col_matcher ? HWS_PTR_TO_ID(matcher->col_matcher) : 0); - ste_pool = matcher->match_ste.pool; -if (ste_pool) { - ste_0_id =mlx5hws_pool_get_base_id (ste_pool); -if (tbl_type ==MLX5HWS_TABLE_TYPE_FDB ) - ste_1_id =mlx5hws_pool_get_base_mirror_id (ste_pool); - } + ste_0_id = matcher->match_ste.ste_0_base; +if (tbl_type ==MLX5HWS_TABLE_TYPE_FDB ) + ste_1_id = matcher->match_ste.ste_1_base; seq_printf(f, ",%d,%d,%d,%d", matcher->match_ste.rtc_0_id,
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/matcher.c +36 -35
115 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/matcher.h +2 -1
@@ -48,7 +48,8 @@ struct mlx5hws_match_template {struct mlx5hws_matcher_match_ste {u32 rtc_0_id;u32 rtc_1_id; -struct mlx5hws_pool *pool; +u32 ste_0_base; +u32 ste_1_base; };enum {
Signature · 1 line
2.34.1
Mark Bloch
maintainerPATCH v3 6/10 net/mlx5: HWS, Decouple matcher RX and TX sizes
In reply to Mark BlochFrom: Vlad Dogaru <[email protected]>
Kernel HWS only uses FDB tables and, as such, creates two lower level containers (RTCs) for each matcher: one for RX and one for TX. Allow these RTCs to differ in size by converting the size part of the matcher attribute to a two element array.
--- .../mellanox/mlx5/core/steering/hws/bwc.c | 7 +- .../mellanox/mlx5/core/steering/hws/debug.c | 10 +- .../mellanox/mlx5/core/steering/hws/matcher.c | 107 ++++++++++++------ .../mellanox/mlx5/core/steering/hws/mlx5hws.h | 28 +++-- 4 files changed, 104 insertions(+), 48 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/bwc.c +5 -2
@@ -48,7 +48,7 @@ static void hws_bwc_unlock_all_queues(struct mlx5hws_context *ctx)static void hws_bwc_matcher_init_attr (struct mlx5hws_bwc_matcher *bwc_matcher,u32 priority, -u8 size_log, +u8 size_log_rx,u8 size_log_tx,struct mlx5hws_matcher_attr *attr) { struct mlx5hws_bwc_matcher *first_matcher =
@@ -62,7 +62,8 @@ static void hws_bwc_matcher_init_attr(struct mlx5hws_bwc_matcher *bwc_matcher, attr->optimize_flow_src =MLX5HWS_MATCHER_FLOW_SRC_ANY ; attr->insert_mode =MLX5HWS_MATCHER_INSERT_BY_HASH ; attr->distribute_mode =MLX5HWS_MATCHER_DISTRIBUTE_BY_HASH ; - attr->rule.num_log = size_log; + attr->size[MLX5HWS_MATCHER_SIZE_TYPE_RX ].rule.num_log = size_log_rx; + attr->size[MLX5HWS_MATCHER_SIZE_TYPE_TX ].rule.num_log = size_log_tx; attr->resizable = true; attr->max_num_of_at_attach =MLX5HWS_BWC_MATCHER_ATTACH_AT_NUM ;
@@ -93,6 +94,7 @@ int mlx5hws_bwc_matcher_create_simple(struct mlx5hws_bwc_matcher *bwc_matcher,hws_bwc_matcher_init_attr (bwc_matcher, priority,MLX5HWS_BWC_MATCHER_INIT_SIZE_LOG , +MLX5HWS_BWC_MATCHER_INIT_SIZE_LOG , &attr); bwc_matcher->priority = priority;
@@ -696,6 +698,7 @@ static int hws_bwc_matcher_move(struct mlx5hws_bwc_matcher *bwc_matcher)hws_bwc_matcher_init_attr (bwc_matcher, bwc_matcher->priority, bwc_matcher->size_log, + bwc_matcher->size_log, &matcher_attr); old_matcher = bwc_matcher->matcher;
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/debug.c +6 -4
@@ -99,17 +99,19 @@ hws_debug_dump_matcher_attr(struct seq_file *f, struct mlx5hws_matcher *matcher) {struct mlx5hws_matcher_attr *attr = &matcher->attr; -seq_printf (f,"%d,0x%llx,%d,%d,%d,%d,%d,%d,%d,%d\n" , +seq_printf (f,"%d,0x%llx,%d,%d,%d,%d,%d,%d,%d,%d,-1,-1,%d,%d\n" ,MLX5HWS_DEBUG_RES_TYPE_MATCHER_ATTR ,HWS_PTR_TO_ID (matcher), attr->priority, attr->mode, - attr->table.sz_row_log, - attr->table.sz_col_log, + attr->size[MLX5HWS_MATCHER_SIZE_TYPE_RX ].table.sz_row_log, + attr->size[MLX5HWS_MATCHER_SIZE_TYPE_RX ].table.sz_col_log, attr->optimize_using_rule_idx, attr->optimize_flow_src, attr->insert_mode, - attr->distribute_mode); + attr->distribute_mode, + attr->size[MLX5HWS_MATCHER_SIZE_TYPE_TX ].table.sz_row_log, + attr->size[MLX5HWS_MATCHER_SIZE_TYPE_TX ].table.sz_col_log);return 0 ; }
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/matcher.c +75 -32
202 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws.h +18 -10
@@ -93,6 +93,23 @@ enum mlx5hws_matcher_distribute_mode { MLX5HWS_MATCHER_DISTRIBUTE_BY_LINEAR = 0x1, }; +enum mlx5hws_matcher_size_type { +MLX5HWS_MATCHER_SIZE_TYPE_RX , +MLX5HWS_MATCHER_SIZE_TYPE_TX , +MLX5HWS_MATCHER_SIZE_TYPE_MAX , +}; + +union mlx5hws_matcher_size { +struct { +u8 sz_row_log; +u8 sz_col_log; + } table; + +struct { +u8 num_log; + } rule; +}; +struct mlx5hws_matcher_attr {/* Processing priority inside table */ u32 priority;
@@ -107,16 +124,7 @@ struct mlx5hws_matcher_attr {enum mlx5hws_matcher_distribute_mode distribute_mode;/* Define whether the created matcher supports resizing into a bigger matcher */ bool resizable; -union { -struct { -u8 sz_row_log; -u8 sz_col_log; - } table; - -struct { -u8 num_log; - } rule; - }; +union mlx5hws_matcher_size size[MLX5HWS_MATCHER_SIZE_TYPE_MAX ];/* Optional AT attach configuration - Max number of additional AT */ u8 max_num_of_at_attach;/* Optional end FT (miss FT ID) for match RTC (for isolated matcher) */
Signature · 1 line
2.34.1
Mark Bloch
maintainerPATCH v3 7/10 net/mlx5: HWS, Track matcher sizes individually
In reply to Mark BlochFrom: Vlad Dogaru <[email protected]>
Track and grow matcher sizes individually for RX and TX RTCs. This allows RX-only or TX-only use cases to effectively halve the device resources they use.
For testing we used a simple module that inserts 1M RX-only rules and measured the number of pages the device requests, and memory usage as reported by free -h.
Pages Memory
Before this patch: 300k 1.5GiB
After this patch: 160k 900MiB
--- .../mellanox/mlx5/core/steering/hws/bwc.c | 217 +++++++++++++----- .../mellanox/mlx5/core/steering/hws/bwc.h | 15 +- 2 files changed, 171 insertions(+), 61 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/bwc.c +159 -58
@@ -93,12 +93,11 @@ int mlx5hws_bwc_matcher_create_simple(struct mlx5hws_bwc_matcher *bwc_matcher,hws_bwc_matcher_init_attr (bwc_matcher, priority, -MLX5HWS_BWC_MATCHER_INIT_SIZE_LOG , -MLX5HWS_BWC_MATCHER_INIT_SIZE_LOG , + bwc_matcher->rx_size.size_log, + bwc_matcher->tx_size.size_log, &attr); bwc_matcher->priority = priority; - bwc_matcher->size_log =MLX5HWS_BWC_MATCHER_INIT_SIZE_LOG ; bwc_matcher->size_of_at_array =MLX5HWS_BWC_MATCHER_ATTACH_AT_NUM ; bwc_matcher->at = kcalloc(bwc_matcher->size_of_at_array,
@@ -150,6 +149,20 @@ int mlx5hws_bwc_matcher_create_simple(struct mlx5hws_bwc_matcher *bwc_matcher,return -EINVAL ; } +static void +hws_bwc_matcher_init_size_rxtx (struct mlx5hws_bwc_matcher_size *size) +{ + size->size_log =MLX5HWS_BWC_MATCHER_INIT_SIZE_LOG ; +atomic_set (&size->num_of_rules,0 ); +atomic_set (&size->rehash_required, false); +} + +static void hws_bwc_matcher_init_size (struct mlx5hws_bwc_matcher *bwc_matcher) +{ +hws_bwc_matcher_init_size_rxtx (&bwc_matcher->rx_size); +hws_bwc_matcher_init_size_rxtx (&bwc_matcher->tx_size); +} +struct mlx5hws_bwc_matcher * mlx5hws_bwc_matcher_create(struct mlx5hws_table *table, u32 priority,
@@ -170,8 +183,7 @@ mlx5hws_bwc_matcher_create(struct mlx5hws_table *table,if (!bwc_matcher)return NULL ; -atomic_set (&bwc_matcher->num_of_rules,0 ); -atomic_set (&bwc_matcher->rehash_required, false); +hws_bwc_matcher_init_size (bwc_matcher);/* Check if the required match params can be all matched * in single STE, otherwise complex matcher is needed.
@@ -221,12 +233,13 @@ int mlx5hws_bwc_matcher_destroy_simple(struct mlx5hws_bwc_matcher *bwc_matcher)int mlx5hws_bwc_matcher_destroy (struct mlx5hws_bwc_matcher *bwc_matcher) { -u32 num_of_rules =atomic_read (&bwc_matcher->num_of_rules); +u32 rx_rules =atomic_read (&bwc_matcher->rx_size.num_of_rules); +u32 tx_rules =atomic_read (&bwc_matcher->tx_size.num_of_rules); -if (num_of_rules) +if (rx_rules || tx_rules)mlx5hws_err (bwc_matcher->matcher->tbl->ctx, -"BWC matcher destroy: matcher still has %d rules\n" , - num_of_rules); +"BWC matcher destroy: matcher still has %u RX and %u TX rules\n" , + rx_rules, tx_rules);if (bwc_matcher->complex)mlx5hws_bwc_matcher_destroy_complex (bwc_matcher);
@@ -386,6 +399,16 @@ hws_bwc_rule_destroy_hws_sync(struct mlx5hws_bwc_rule *bwc_rule,return 0 ; } +static void hws_bwc_rule_cnt_dec (struct mlx5hws_bwc_rule *bwc_rule) +{ +struct mlx5hws_bwc_matcher *bwc_matcher = bwc_rule->bwc_matcher; + +if (!bwc_rule->skip_rx) +atomic_dec (&bwc_matcher->rx_size.num_of_rules); +if (!bwc_rule->skip_tx) +atomic_dec (&bwc_matcher->tx_size.num_of_rules); +} +int mlx5hws_bwc_rule_destroy_simple (struct mlx5hws_bwc_rule *bwc_rule) {struct mlx5hws_bwc_matcher *bwc_matcher = bwc_rule->bwc_matcher;
@@ -402,7 +425,7 @@ int mlx5hws_bwc_rule_destroy_simple(struct mlx5hws_bwc_rule *bwc_rule)mutex_lock (queue_lock); ret =hws_bwc_rule_destroy_hws_sync (bwc_rule, &attr); -atomic_dec (&bwc_matcher->num_of_rules); +hws_bwc_rule_cnt_dec (bwc_rule);hws_bwc_rule_list_remove (bwc_rule);mutex_unlock (queue_lock);
@@ -489,25 +512,27 @@ hws_bwc_rule_update_sync(struct mlx5hws_bwc_rule *bwc_rule, }static bool -hws_bwc_matcher_size_maxed_out (struct mlx5hws_bwc_matcher *bwc_matcher) +hws_bwc_matcher_size_maxed_out (struct mlx5hws_bwc_matcher *bwc_matcher, +struct mlx5hws_bwc_matcher_size *size) {struct mlx5hws_cmd_query_caps *caps = bwc_matcher->matcher->tbl->ctx->caps;/* check the match RTC size */ -return (bwc_matcher->size_log +MLX5HWS_MATCHER_ASSURED_MAIN_TBL_DEPTH + +return (size->size_log +MLX5HWS_MATCHER_ASSURED_MAIN_TBL_DEPTH +MLX5HWS_BWC_MATCHER_SIZE_LOG_STEP ) > (caps->ste_alloc_log_max -1 ); }static bool hws_bwc_matcher_rehash_size_needed (struct mlx5hws_bwc_matcher *bwc_matcher, +struct mlx5hws_bwc_matcher_size *size,u32 num_of_rules) { -if (unlikely (hws_bwc_matcher_size_maxed_out (bwc_matcher))) +if (unlikely (hws_bwc_matcher_size_maxed_out (bwc_matcher, size)))return false;if (unlikely ((num_of_rules *100 /MLX5HWS_BWC_MATCHER_REHASH_PERCENT_TH ) >= - (1UL << bwc_matcher->size_log))) + (1UL << size->size_log)))return true;return false;
@@ -564,20 +589,21 @@ hws_bwc_matcher_extend_at(struct mlx5hws_bwc_matcher *bwc_matcher, }static int -hws_bwc_matcher_extend_size (struct mlx5hws_bwc_matcher *bwc_matcher) +hws_bwc_matcher_extend_size (struct mlx5hws_bwc_matcher *bwc_matcher, +struct mlx5hws_bwc_matcher_size *size) {struct mlx5hws_context *ctx = bwc_matcher->matcher->tbl->ctx;struct mlx5hws_cmd_query_caps *caps = ctx->caps; -if (unlikely (hws_bwc_matcher_size_maxed_out (bwc_matcher))) { +if (unlikely (hws_bwc_matcher_size_maxed_out (bwc_matcher, size))) {mlx5hws_err (ctx,"Can't resize matcher: depth exceeds limit %d\n" , caps->rtc_log_depth_max);return -ENOMEM ; } - bwc_matcher->size_log = -min (bwc_matcher->size_log +MLX5HWS_BWC_MATCHER_SIZE_LOG_STEP , - caps->ste_alloc_log_max -MLX5HWS_MATCHER_ASSURED_MAIN_TBL_DEPTH ); + size->size_log =min (size->size_log +MLX5HWS_BWC_MATCHER_SIZE_LOG_STEP , + caps->ste_alloc_log_max - +MLX5HWS_MATCHER_ASSURED_MAIN_TBL_DEPTH );return 0 ; }
@@ -697,8 +723,8 @@ static int hws_bwc_matcher_move(struct mlx5hws_bwc_matcher *bwc_matcher)hws_bwc_matcher_init_attr (bwc_matcher, bwc_matcher->priority, - bwc_matcher->size_log, - bwc_matcher->size_log, + bwc_matcher->rx_size.size_log, + bwc_matcher->tx_size.size_log, &matcher_attr); old_matcher = bwc_matcher->matcher;
@@ -736,21 +762,39 @@ static int hws_bwc_matcher_move(struct mlx5hws_bwc_matcher *bwc_matcher)static int hws_bwc_matcher_rehash_size (struct mlx5hws_bwc_matcher *bwc_matcher) { +bool need_rx_rehash, need_tx_rehash;int ret; -/* If the current matcher size is already at its max size, we can't -* do the rehash. Skip it and try adding the rule again - perhaps -* there was some change. + need_rx_rehash =atomic_read (&bwc_matcher->rx_size.rehash_required); + need_tx_rehash =atomic_read (&bwc_matcher->tx_size.rehash_required); + +/* It is possible that another rule has already performed rehash. +* Need to check again if we really need rehash. */ -if (hws_bwc_matcher_size_maxed_out (bwc_matcher)) +if (!need_rx_rehash && !need_tx_rehash)return 0 ; -/* It is possible that other rule has already performed rehash. -* Need to check again if we really need rehash. +/* If the current matcher RX/TX size is already at its max size, +* it can't be rehashed. */ -if (!atomic_read (&bwc_matcher->rehash_required) && - !hws_bwc_matcher_rehash_size_needed (bwc_matcher, -atomic_read (&bwc_matcher->num_of_rules))) +if (need_rx_rehash && +hws_bwc_matcher_size_maxed_out (bwc_matcher, + &bwc_matcher->rx_size)) { +atomic_set (&bwc_matcher->rx_size.rehash_required, false); + need_rx_rehash = false; + } +if (need_tx_rehash && +hws_bwc_matcher_size_maxed_out (bwc_matcher, + &bwc_matcher->tx_size)) { +atomic_set (&bwc_matcher->tx_size.rehash_required, false); + need_tx_rehash = false; + } + +/* If both RX and TX rehash flags are now off, it means that whatever +* we wanted to rehash is now at its max size - no rehash can be done. +* Return and try adding the rule again - perhaps there was some change. +*/ +if (!need_rx_rehash && !need_tx_rehash)return 0 ;/* Now we're done all the checking - do the rehash:
@@ -759,12 +803,22 @@ hws_bwc_matcher_rehash_size(struct mlx5hws_bwc_matcher *bwc_matcher)* - move all the rules to the new matcher * - destroy the old matcher */ +atomic_set (&bwc_matcher->rx_size.rehash_required, false); +atomic_set (&bwc_matcher->tx_size.rehash_required, false); -atomic_set (&bwc_matcher->rehash_required, false); +if (need_rx_rehash) { + ret =hws_bwc_matcher_extend_size (bwc_matcher, + &bwc_matcher->rx_size); +if (ret) +return ret; + } - ret =hws_bwc_matcher_extend_size (bwc_matcher); -if (ret) -return ret; +if (need_tx_rehash) { + ret =hws_bwc_matcher_extend_size (bwc_matcher, + &bwc_matcher->tx_size); +if (ret) +return ret; + }return hws_bwc_matcher_move (bwc_matcher); }
@@ -816,6 +870,62 @@ static int hws_bwc_rule_get_at_idx(struct mlx5hws_bwc_rule *bwc_rule,return at_idx; } +static void hws_bwc_rule_cnt_inc_rxtx (struct mlx5hws_bwc_rule *bwc_rule, +struct mlx5hws_bwc_matcher_size *size) +{ +u32 num_of_rules =atomic_inc_return (&size->num_of_rules); + +if (unlikely (hws_bwc_matcher_rehash_size_needed (bwc_rule->bwc_matcher, + size, num_of_rules))) +atomic_set (&size->rehash_required, true); +} + +static void hws_bwc_rule_cnt_inc (struct mlx5hws_bwc_rule *bwc_rule) +{ +struct mlx5hws_bwc_matcher *bwc_matcher = bwc_rule->bwc_matcher; + +if (!bwc_rule->skip_rx) +hws_bwc_rule_cnt_inc_rxtx (bwc_rule, &bwc_matcher->rx_size); +if (!bwc_rule->skip_tx) +hws_bwc_rule_cnt_inc_rxtx (bwc_rule, &bwc_matcher->tx_size); +} + +static int hws_bwc_rule_cnt_inc_with_rehash (struct mlx5hws_bwc_rule *bwc_rule, +u16 bwc_queue_idx) +{ +struct mlx5hws_bwc_matcher *bwc_matcher = bwc_rule->bwc_matcher; +struct mlx5hws_context *ctx = bwc_matcher->matcher->tbl->ctx; +struct mutex *queue_lock;/* Protect the queue */ +int ret; + +hws_bwc_rule_cnt_inc (bwc_rule); + +if (!atomic_read (&bwc_matcher->rx_size.rehash_required) && + !atomic_read (&bwc_matcher->tx_size.rehash_required)) +return 0 ; + + queue_lock =hws_bwc_get_queue_lock (ctx, bwc_queue_idx); +mutex_unlock (queue_lock); + +hws_bwc_lock_all_queues (ctx); + ret =hws_bwc_matcher_rehash_size (bwc_matcher); +hws_bwc_unlock_all_queues (ctx); + +mutex_lock (queue_lock); + +if (likely (!ret)) +return 0 ; + +/* Failed to rehash. Print a diagnostic and rollback the counters. */ +mlx5hws_err (ctx, +"BWC rule insertion: rehash to sizes [%d, %d] failed (%d)\n" , + bwc_matcher->rx_size.size_log, + bwc_matcher->tx_size.size_log, ret); +hws_bwc_rule_cnt_dec (bwc_rule); + +return ret; +} + int mlx5hws_bwc_rule_create_simple(struct mlx5hws_bwc_rule *bwc_rule, u32 *match_param, struct mlx5hws_rule_action rule_actions[],
@@ -826,7 +936,6 @@ int mlx5hws_bwc_rule_create_simple(struct mlx5hws_bwc_rule *bwc_rule,struct mlx5hws_context *ctx = bwc_matcher->matcher->tbl->ctx;struct mlx5hws_rule_attr rule_attr;struct mutex *queue_lock;/* Protect the queue */ -u32 num_of_rules;int ret =0 ;int at_idx;
@@ -844,26 +953,10 @@ int mlx5hws_bwc_rule_create_simple(struct mlx5hws_bwc_rule *bwc_rule,return -EINVAL ; } -/* check if number of rules require rehash */ - num_of_rules =atomic_inc_return (&bwc_matcher->num_of_rules); - -if (unlikely (hws_bwc_matcher_rehash_size_needed (bwc_matcher, num_of_rules))) { + ret =hws_bwc_rule_cnt_inc_with_rehash (bwc_rule, bwc_queue_idx); +if (unlikely (ret)) {mutex_unlock (queue_lock); - -hws_bwc_lock_all_queues (ctx); - ret =hws_bwc_matcher_rehash_size (bwc_matcher); -hws_bwc_unlock_all_queues (ctx); - -if (ret) { -mlx5hws_err (ctx,"BWC rule insertion: rehash size [%d -> %d] failed (%d)\n" , - bwc_matcher->size_log -MLX5HWS_BWC_MATCHER_SIZE_LOG_STEP , - bwc_matcher->size_log, - ret); -atomic_dec (&bwc_matcher->num_of_rules); -return ret; - } - -mutex_lock (queue_lock); +return ret; } ret = hws_bwc_rule_create_sync(bwc_rule,
@@ -881,8 +974,11 @@ int mlx5hws_bwc_rule_create_simple(struct mlx5hws_bwc_rule *bwc_rule,* It could be because there was collision, or some other problem. * Try rehash by size and insert rule again - last chance. */ +if (!bwc_rule->skip_rx) +atomic_set (&bwc_matcher->rx_size.rehash_required, true); +if (!bwc_rule->skip_tx) +atomic_set (&bwc_matcher->tx_size.rehash_required, true); -atomic_set (&bwc_matcher->rehash_required, true);mutex_unlock (queue_lock);hws_bwc_lock_all_queues (ctx);
@@ -891,7 +987,7 @@ int mlx5hws_bwc_rule_create_simple(struct mlx5hws_bwc_rule *bwc_rule,if (ret) {mlx5hws_err (ctx,"BWC rule insertion: rehash failed (%d)\n" , ret); -atomic_dec (&bwc_matcher->num_of_rules); +hws_bwc_rule_cnt_dec (bwc_rule);return ret; }
@@ -907,7 +1003,7 @@ int mlx5hws_bwc_rule_create_simple(struct mlx5hws_bwc_rule *bwc_rule,if (unlikely (ret)) {mutex_unlock (queue_lock);mlx5hws_err (ctx,"BWC rule insertion failed (%d)\n" , ret); -atomic_dec (&bwc_matcher->num_of_rules); +hws_bwc_rule_cnt_dec (bwc_rule);return ret; }
@@ -937,6 +1033,10 @@ mlx5hws_bwc_rule_create(struct mlx5hws_bwc_matcher *bwc_matcher,if (unlikely (!bwc_rule))return NULL ; + bwc_rule->flow_source = flow_source; +mlx5hws_rule_skip (bwc_matcher->matcher, flow_source, + &bwc_rule->skip_rx, &bwc_rule->skip_tx); + bwc_queue_idx =hws_bwc_gen_queue_idx (ctx);if (bwc_matcher->complex)
@@ -972,7 +1072,8 @@ hws_bwc_rule_action_update(struct mlx5hws_bwc_rule *bwc_rule, idx = bwc_rule->bwc_queue_idx; -mlx5hws_bwc_rule_fill_attr (bwc_matcher, idx,0 , &rule_attr); +mlx5hws_bwc_rule_fill_attr (bwc_matcher, idx, bwc_rule->flow_source, + &rule_attr); queue_lock =hws_bwc_get_queue_lock (ctx, idx);mutex_lock (queue_lock);
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/bwc.h +12 -3
@@ -19,6 +19,13 @@#define MLX5HWS_BWC_POLLING_TIMEOUT 60struct mlx5hws_bwc_matcher_complex_data ; + +struct mlx5hws_bwc_matcher_size { +u8 size_log; +atomic_t num_of_rules; +atomic_t rehash_required; +}; +struct mlx5hws_bwc_matcher {struct mlx5hws_matcher *matcher;struct mlx5hws_match_template *mt;
@@ -27,10 +34,9 @@ struct mlx5hws_bwc_matcher {struct mlx5hws_bwc_matcher *complex_first_bwc_matcher;u8 num_of_at;u8 size_of_at_array; -u8 size_log;u32 priority; -atomic_t num_of_rules; -atomic_t rehash_required; +struct mlx5hws_bwc_matcher_size rx_size; +struct mlx5hws_bwc_matcher_size tx_size;struct list_head *rules; };
@@ -39,7 +45,10 @@ struct mlx5hws_bwc_rule {struct mlx5hws_rule *rule;struct mlx5hws_bwc_rule *isolated_bwc_rule;struct mlx5hws_bwc_complex_rule_hash_node *complex_hash_node; +u32 flow_source;u16 bwc_queue_idx; +bool skip_rx; +bool skip_tx;struct list_head list_node; };
Signature · 1 line
2.34.1
Re: [PATCH net-next v3 07/10] net/mlx5: HWS, Track matcher sizes individually
Quoted 10 lines · Mark Bloch
From: Vlad Dogaru <[email protected]>
Track and grow matcher sizes individually for RX and TX RTCs. This allows RX-only or TX-only use cases to effectively halve the device resources they use.
For testing we used a simple module that inserts 1M RX-only rules and measured the number of pages the device requests, and memory usage as reported by
free -h.
Quoted 7 lines
Pages Memory
Before this patch: 300k 1.5GiB
After this patch: 160k 900MiBSigned-off-by Vlad Dogaru <[email protected]>Reviewed-by Yevgeny Kliteynik <[email protected]>Signed-off-by Mark Bloch <[email protected]>
Mark Bloch
maintainerPATCH v3 8/10 net/mlx5: HWS, Rearrange to prevent forward declaration
In reply to Mark BlochFrom: Yevgeny Kliteynik <[email protected]>
As a preparation for the following patch that will add support for shrinking empty matchers, rearrange the code to prevent forward declaration of functions.
--- .../mellanox/mlx5/core/steering/hws/bwc.c | 247 +++++++++--------- 1 file changed, 124 insertions(+), 123 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/bwc.c +124 -123
@@ -71,6 +71,130 @@ static void hws_bwc_matcher_init_attr(struct mlx5hws_bwc_matcher *bwc_matcher, first_matcher ? first_matcher->matcher->end_ft_id :0 ; } +static int +hws_bwc_matcher_move_all_simple (struct mlx5hws_bwc_matcher *bwc_matcher) +{ +bool move_error = false, poll_error = false, drain_error = false; +struct mlx5hws_context *ctx = bwc_matcher->matcher->tbl->ctx; +struct mlx5hws_matcher *matcher = bwc_matcher->matcher; +u16 bwc_queues =mlx5hws_bwc_queues (ctx); +struct mlx5hws_rule_attr rule_attr; +struct mlx5hws_bwc_rule *bwc_rule; +struct mlx5hws_send_engine *queue; +struct list_head *rules_list; +u32 pending_rules; +int i, ret =0 ; + +mlx5hws_bwc_rule_fill_attr (bwc_matcher,0 ,0 , &rule_attr); + +for (i =0 ; i < bwc_queues; i++) { +if (list_empty (&bwc_matcher->rules[i])) +continue ; + + pending_rules =0 ; + rule_attr.queue_id =mlx5hws_bwc_get_queue_id (ctx, i); + rules_list = &bwc_matcher->rules[i]; + +list_for_each_entry (bwc_rule, rules_list, list_node) { + ret =mlx5hws_matcher_resize_rule_move (matcher, + bwc_rule->rule, + &rule_attr); +if (unlikely (ret && !move_error)) { +mlx5hws_err (ctx, +"Moving BWC rule: move failed (%d), attempting to move rest of the rules\n" , + ret); + move_error = true; + } + + pending_rules++; + ret =mlx5hws_bwc_queue_poll (ctx, + rule_attr.queue_id, + &pending_rules, + false); +if (unlikely (ret && !poll_error)) { +mlx5hws_err (ctx, +"Moving BWC rule: poll failed (%d), attempting to move rest of the rules\n" , + ret); + poll_error = true; + } + } + +if (pending_rules) { + queue = &ctx->send_queue[rule_attr.queue_id]; +mlx5hws_send_engine_flush_queue (queue); + ret =mlx5hws_bwc_queue_poll (ctx, + rule_attr.queue_id, + &pending_rules, + true); +if (unlikely (ret && !drain_error)) { +mlx5hws_err (ctx, +"Moving BWC rule: drain failed (%d), attempting to move rest of the rules\n" , + ret); + drain_error = true; + } + } + } + +if (move_error || poll_error || drain_error) + ret = -EINVAL ; + +return ret; +} + +static int hws_bwc_matcher_move_all (struct mlx5hws_bwc_matcher *bwc_matcher) +{ +if (!bwc_matcher->complex) +return hws_bwc_matcher_move_all_simple (bwc_matcher); + +return mlx5hws_bwc_matcher_move_all_complex (bwc_matcher); +} + +static int hws_bwc_matcher_move (struct mlx5hws_bwc_matcher *bwc_matcher) +{ +struct mlx5hws_context *ctx = bwc_matcher->matcher->tbl->ctx; +struct mlx5hws_matcher_attr matcher_attr = {0 }; +struct mlx5hws_matcher *old_matcher; +struct mlx5hws_matcher *new_matcher; +int ret; + +hws_bwc_matcher_init_attr (bwc_matcher, + bwc_matcher->priority, + bwc_matcher->rx_size.size_log, + bwc_matcher->tx_size.size_log, + &matcher_attr); + + old_matcher = bwc_matcher->matcher; + new_matcher =mlx5hws_matcher_create (old_matcher->tbl, + &bwc_matcher->mt,1 , + bwc_matcher->at, + bwc_matcher->num_of_at, + &matcher_attr); +if (!new_matcher) { +mlx5hws_err (ctx,"Rehash error: matcher creation failed\n" ); +return -ENOMEM ; + } + + ret =mlx5hws_matcher_resize_set_target (old_matcher, new_matcher); +if (ret) { +mlx5hws_err (ctx,"Rehash error: failed setting resize target\n" ); +return ret; + } + + ret =hws_bwc_matcher_move_all (bwc_matcher); +if (ret) +mlx5hws_err (ctx,"Rehash error: moving rules failed, attempting to remove the old matcher\n" ); + +/* Error during rehash can't be rolled back. +* The best option here is to allow the rehash to complete and remove +* the old matcher - can't leave the matcher in the 'in_resize' state. +*/ + + bwc_matcher->matcher = new_matcher; +mlx5hws_matcher_destroy (old_matcher); + +return ret; +} + int mlx5hws_bwc_matcher_create_simple(struct mlx5hws_bwc_matcher *bwc_matcher, struct mlx5hws_table *table, u32 priority,
@@ -636,129 +760,6 @@ hws_bwc_matcher_find_at(struct mlx5hws_bwc_matcher *bwc_matcher,return -1 ; } -static int hws_bwc_matcher_move_all_simple (struct mlx5hws_bwc_matcher *bwc_matcher) -{ -bool move_error = false, poll_error = false, drain_error = false; -struct mlx5hws_context *ctx = bwc_matcher->matcher->tbl->ctx; -struct mlx5hws_matcher *matcher = bwc_matcher->matcher; -u16 bwc_queues =mlx5hws_bwc_queues (ctx); -struct mlx5hws_rule_attr rule_attr; -struct mlx5hws_bwc_rule *bwc_rule; -struct mlx5hws_send_engine *queue; -struct list_head *rules_list; -u32 pending_rules; -int i, ret =0 ; - -mlx5hws_bwc_rule_fill_attr (bwc_matcher,0 ,0 , &rule_attr); - -for (i =0 ; i < bwc_queues; i++) { -if (list_empty (&bwc_matcher->rules[i])) -continue ; - - pending_rules =0 ; - rule_attr.queue_id =mlx5hws_bwc_get_queue_id (ctx, i); - rules_list = &bwc_matcher->rules[i]; - -list_for_each_entry (bwc_rule, rules_list, list_node) { - ret =mlx5hws_matcher_resize_rule_move (matcher, - bwc_rule->rule, - &rule_attr); -if (unlikely (ret && !move_error)) { -mlx5hws_err (ctx, -"Moving BWC rule: move failed (%d), attempting to move rest of the rules\n" , - ret); - move_error = true; - } - - pending_rules++; - ret =mlx5hws_bwc_queue_poll (ctx, - rule_attr.queue_id, - &pending_rules, - false); -if (unlikely (ret && !poll_error)) { -mlx5hws_err (ctx, -"Moving BWC rule: poll failed (%d), attempting to move rest of the rules\n" , - ret); - poll_error = true; - } - } - -if (pending_rules) { - queue = &ctx->send_queue[rule_attr.queue_id]; -mlx5hws_send_engine_flush_queue (queue); - ret =mlx5hws_bwc_queue_poll (ctx, - rule_attr.queue_id, - &pending_rules, - true); -if (unlikely (ret && !drain_error)) { -mlx5hws_err (ctx, -"Moving BWC rule: drain failed (%d), attempting to move rest of the rules\n" , - ret); - drain_error = true; - } - } - } - -if (move_error || poll_error || drain_error) - ret = -EINVAL ; - -return ret; -} - -static int hws_bwc_matcher_move_all (struct mlx5hws_bwc_matcher *bwc_matcher) -{ -if (!bwc_matcher->complex) -return hws_bwc_matcher_move_all_simple (bwc_matcher); - -return mlx5hws_bwc_matcher_move_all_complex (bwc_matcher); -} - -static int hws_bwc_matcher_move (struct mlx5hws_bwc_matcher *bwc_matcher) -{ -struct mlx5hws_context *ctx = bwc_matcher->matcher->tbl->ctx; -struct mlx5hws_matcher_attr matcher_attr = {0 }; -struct mlx5hws_matcher *old_matcher; -struct mlx5hws_matcher *new_matcher; -int ret; - -hws_bwc_matcher_init_attr (bwc_matcher, - bwc_matcher->priority, - bwc_matcher->rx_size.size_log, - bwc_matcher->tx_size.size_log, - &matcher_attr); - - old_matcher = bwc_matcher->matcher; - new_matcher =mlx5hws_matcher_create (old_matcher->tbl, - &bwc_matcher->mt,1 , - bwc_matcher->at, - bwc_matcher->num_of_at, - &matcher_attr); -if (!new_matcher) { -mlx5hws_err (ctx,"Rehash error: matcher creation failed\n" ); -return -ENOMEM ; - } - - ret =mlx5hws_matcher_resize_set_target (old_matcher, new_matcher); -if (ret) { -mlx5hws_err (ctx,"Rehash error: failed setting resize target\n" ); -return ret; - } - - ret =hws_bwc_matcher_move_all (bwc_matcher); -if (ret) -mlx5hws_err (ctx,"Rehash error: moving rules failed, attempting to remove the old matcher\n" ); - -/* Error during rehash can't be rolled back. -* The best option here is to allow the rehash to complete and remove -* the old matcher - can't leave the matcher in the 'in_resize' state. -*/ - - bwc_matcher->matcher = new_matcher; -mlx5hws_matcher_destroy (old_matcher); - -return ret; -} -static int hws_bwc_matcher_rehash_size (struct mlx5hws_bwc_matcher *bwc_matcher) {
Signature · 1 line
2.34.1
Re: [PATCH net-next v3 08/10] net/mlx5: HWS, Rearrange to prevent forward declaration
Quoted 8 lines · Mark Bloch
From: Yevgeny Kliteynik <[email protected]>
As a preparation for the following patch that will add support for shrinking empty matchers, rearrange the code to prevent forward declaration of functions.
Signed-off-by Yevgeny Kliteynik <[email protected]>Signed-off-by Mark Bloch <[email protected]>
Mark Bloch
maintainerPATCH v3 9/10 net/mlx5: HWS, Shrink empty matchers
In reply to Mark BlochFrom: Yevgeny Kliteynik <[email protected]>
Matcher size is dynamic: it starts at initial size, and then it grows through rehash as more and more rules are added to this matcher. When rules are deleted, matcher's size is not decreased. Rehash approach is greedy. The idea is: if the matcher got to a certain size at some point, chances are - it will get to this size again, so it is better to avoid costly rehash operations whenever possible.
However, when all the rules of the matcher are deleted, this should be viewed as special case. If the matcher actually got to the point where it has zero rules, it might be an indication that some usecase from the past is no longer happening. This is where some ICM can be freed.
This patch handles this case: when a number of rules in a matcher goes down to zero, the matcher's tables are shrunk to the initial size.
--- .../mellanox/mlx5/core/steering/hws/bwc.c | 66 ++++++++++++++++++- 1 file changed, 65 insertions(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/bwc.c +65 -1
@@ -533,6 +533,70 @@ static void hws_bwc_rule_cnt_dec(struct mlx5hws_bwc_rule *bwc_rule)atomic_dec (&bwc_matcher->tx_size.num_of_rules); } +static int +hws_bwc_matcher_rehash_shrink (struct mlx5hws_bwc_matcher *bwc_matcher) +{ +struct mlx5hws_bwc_matcher_size *rx_size = &bwc_matcher->rx_size; +struct mlx5hws_bwc_matcher_size *tx_size = &bwc_matcher->tx_size; + +/* It is possible that another thread has added a rule. +* Need to check again if we really need rehash/shrink. +*/ +if (atomic_read (&rx_size->num_of_rules) || +atomic_read (&tx_size->num_of_rules)) +return 0 ; + +/* If the current matcher RX/TX size is already at its initial size. */ +if (rx_size->size_log ==MLX5HWS_BWC_MATCHER_INIT_SIZE_LOG && + tx_size->size_log ==MLX5HWS_BWC_MATCHER_INIT_SIZE_LOG ) +return 0 ; + +/* Now we've done all the checking - do the shrinking: +* - reset match RTC size to the initial size +* - create new matcher +* - move the rules, which will not do anything as the matcher is empty +* - destroy the old matcher +*/ + + rx_size->size_log =MLX5HWS_BWC_MATCHER_INIT_SIZE_LOG ; + tx_size->size_log =MLX5HWS_BWC_MATCHER_INIT_SIZE_LOG ; + +return hws_bwc_matcher_move (bwc_matcher); +} + +static int hws_bwc_rule_cnt_dec_with_shrink (struct mlx5hws_bwc_rule *bwc_rule, +u16 bwc_queue_idx) +{ +struct mlx5hws_bwc_matcher *bwc_matcher = bwc_rule->bwc_matcher; +struct mlx5hws_context *ctx = bwc_matcher->matcher->tbl->ctx; +struct mutex *queue_lock;/* Protect the queue */ +int ret; + +hws_bwc_rule_cnt_dec (bwc_rule); + +if (atomic_read (&bwc_matcher->rx_size.num_of_rules) || +atomic_read (&bwc_matcher->tx_size.num_of_rules)) +return 0 ; + +/* Matcher has no more rules - shrink it to save ICM. */ + + queue_lock =hws_bwc_get_queue_lock (ctx, bwc_queue_idx); +mutex_unlock (queue_lock); + +hws_bwc_lock_all_queues (ctx); + ret =hws_bwc_matcher_rehash_shrink (bwc_matcher); +hws_bwc_unlock_all_queues (ctx); + +mutex_lock (queue_lock); + +if (unlikely (ret)) +mlx5hws_err (ctx, +"BWC rule deletion: shrinking empty matcher failed (%d)\n" , + ret); + +return ret; +} +int mlx5hws_bwc_rule_destroy_simple (struct mlx5hws_bwc_rule *bwc_rule) {struct mlx5hws_bwc_matcher *bwc_matcher = bwc_rule->bwc_matcher;
@@ -549,8 +613,8 @@ int mlx5hws_bwc_rule_destroy_simple(struct mlx5hws_bwc_rule *bwc_rule)mutex_lock (queue_lock); ret =hws_bwc_rule_destroy_hws_sync (bwc_rule, &attr); -hws_bwc_rule_cnt_dec (bwc_rule);hws_bwc_rule_list_remove (bwc_rule); +hws_bwc_rule_cnt_dec_with_shrink (bwc_rule, idx);mutex_unlock (queue_lock);
Signature · 1 line
2.34.1
Re: [PATCH net-next v3 09/10] net/mlx5: HWS, Shrink empty matchers
Quoted 15 lines · Mark Bloch
From: Yevgeny Kliteynik <[email protected]>
Matcher size is dynamic: it starts at initial size, and then it grows through rehash as more and more rules are added to this matcher. When rules are deleted, matcher's size is not decreased. Rehash approach is greedy. The idea is: if the matcher got to a certain size at some point, chances are - it will get to this size again, so it is better to avoid costly rehash operations whenever possible.
However, when all the rules of the matcher are deleted, this should be viewed as special case. If the matcher actually got to the point where it has zero rules, it might be an indication that some usecase from the past is no longer happening. This is where some ICM can be freed.
Quoted 7 lines
This patch handles this case: when a number of rules in a matcher goes down to zero, the matcher's tables are shrunk to the initial size.
Signed-off-by Yevgeny Kliteynik <[email protected]>Reviewed-by Vlad Dogaru <[email protected]>Signed-off-by Mark Bloch <[email protected]>
Mark Bloch
maintainerPATCH v3 10/10 net/mlx5: Add HWS as secondary steering mode
In reply to Mark BlochFrom: Moshe Shemesh <[email protected]>
Add HW Steering (HWS) as a secondary option for device steering mode. If the device does not support SW Steering (SWS), HW Steering will be used as the default, provided it is supported. FW Steering will now be selected as the default only if both HWS and SWS are unavailable.
--- drivers/net/ethernet/mellanox/mlx5/core/fs_core.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/fs_core.c +2 -0
@@ -3919,6 +3919,8 @@ int mlx5_fs_core_alloc(struct mlx5_core_dev *dev)if (mlx5_fs_dr_is_supported (dev)) steering->mode =MLX5_FLOW_STEERING_MODE_SMFS ; +else if (mlx5_fs_hws_is_supported (dev)) + steering->mode =MLX5_FLOW_STEERING_MODE_HMFS ;else steering->mode =MLX5_FLOW_STEERING_MODE_DMFS ;
Signature · 1 line
2.34.1
Re: [PATCH net-next v3 10/10] net/mlx5: Add HWS as secondary steering mode
Quoted 10 lines · Mark Bloch
From: Moshe Shemesh <[email protected]>
Add HW Steering (HWS) as a secondary option for device steering mode. If the device does not support SW Steering (SWS), HW Steering will be used as the default, provided it is supported. FW Steering will now be selected as the default only if both HWS and SWS are unavailable.
Signed-off-by Moshe Shemesh <[email protected]>Reviewed-by Yevgeny Kliteynik <[email protected]>Signed-off-by Mark Bloch <[email protected]>
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Quoted 10 lines · On Thu, 3 Jul 2025 21:54:21 +0300 you wrote:
This series optimizes ICM usage for unidirectional rules and empty matchers and with the last patch we make hardware steering the default FDB steering provider for NICs that don't support software steering.
Hardware steering (HWS) uses a type of rule table container (RTC) that is unidirectional, so matchers consist of two RTCs to accommodate bidirectional rules.
[...]
Here is the summary with links:
- [net-next,v3,01/10] net/mlx5: HWS, remove unused create_dest_array parameter
https://git.kernel.org/netdev/net-next/c/60afb51c8941 - [net-next,v3,02/10] net/mlx5: HWS, remove incorrect comment https://git.kernel.org/netdev/net-next/c/26b06579d50d - [net-next,v3,03/10] net/mlx5: HWS, Export rule skip logic https://git.kernel.org/netdev/net-next/c/d8e7ab591b50 - [net-next,v3,04/10] net/mlx5: HWS, Refactor rule skip logic https://git.kernel.org/netdev/net-next/c/3dcac700d20b - [net-next,v3,05/10] net/mlx5: HWS, Create STEs directly from matcher https://git.kernel.org/netdev/net-next/c/59807d071724 - [net-next,v3,06/10] net/mlx5: HWS, Decouple matcher RX and TX sizes https://git.kernel.org/netdev/net-next/c/c8332ce09691 - [net-next,v3,07/10] net/mlx5: HWS, Track matcher sizes individually https://git.kernel.org/netdev/net-next/c/6b44fffdc7b7 - [net-next,v3,08/10] net/mlx5: HWS, Rearrange to prevent forward declaration https://git.kernel.org/netdev/net-next/c/29063103f864 - [net-next,v3,09/10] net/mlx5: HWS, Shrink empty matchers https://git.kernel.org/netdev/net-next/c/96e4c4a1a5bc - [net-next,v3,10/10] net/mlx5: Add HWS as secondary steering mode https://git.kernel.org/netdev/net-next/c/a9aec713d0d9
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html