net/mlx5: ConnectX-8 SW Steering + Rate management on traffic classes
Subsystems (7)
Tariq Toukan
maintainerv4 cover net/mlx5: ConnectX-8 SW Steering + Rate management on traffic classes
Hi,
This patchset starts with 3 patches that modify the IFC, targeted to mlx5-next in order to be taken to rdma-next branch side sooner than in the next merge window.
This patchset consists of two features:
1. In patches 4-5, Itamar adds SW Steering support for ConnectX-8.
2. Followed by patches by Carolina that add rate management support on traffic classes in devlink and mlx5, more details below [1].
Series generated against:
commit e8e7be7d212d ("mctp i2c: drop check because i2c_unregister_device() is NULL safe")
Regards,
Tariq
V4:
- Renamed the nested attribute for traffic class bandwidth to DEVLINK_ATTR_RATE_TC_BWS.
- Changed the order of the attributes in devlink.h.
- Refactored the initialization tc-bw array in devlink_nl_rate_tc_bw_set().
- Added extack messages to provide clear feedback on issues with tc-bw arguments.
- Updated rate-tc-bws to support a multi-attr set, where each attribute includes an index and the corresponding bandwidth for that traffic class.
- Handled the issue where the user could provide DEVLINK_ATTR_RATE_TC_BWS with duplicate indices.
- Provided ynl exmaples in devlink patch commit message.
- Take IFC patches to beginning of the series, targeted for mlx5-next.
V3:
- Dropped rate-tc-index, using tc-bw array index instead.
- Renamed rate-bw to rate-tc-bw.
- Documneted what the rate-tc-bw represents and added a range check for validation.
- Intorduced devlink_nl_rate_tc_bw_set() to parse and set the TC bandwidth values.
- Updated the user API in the commit message of patch 1/6 to ensure bandwidths sum equals 100.
- Fixed missing filling of rate-parent in devlink_nl_rate_fill().
V2:
- Included <linux/dcbnl.h> in devlink.h to resolve missing IEEE_8021QAZ_MAX_TCS definition.
- Refactored the rate-tc-bw attribute structure to use a separate rate-tc-index.
- Updated patch 2/6 title.
[1]
This patch series extends the devlink-rate API to support traffic class (TC) bandwidth management, enabling more granular control over traffic shaping and rate limiting across multiple TCs. The API now allows users to specify bandwidth proportions for different traffic classes in a single command. This is particularly useful for managing Enhanced Transmission Selection (ETS) for groups of Virtual Functions (VFs), allowing precise bandwidth allocation across traffic classes.
Additionally the series refines the QoS handling in net/mlx5 to support TC arbitration and bandwidth management on vports and rate nodes.
Extend devlink-rate API to support rate management on TCs:
- devlink: Extend the devlink rate API to support traffic class bandwidth management
Introduce a no-op implementation:
- net/mlx5: Add no-op implementation for setting tc-bw on rate objects
Add support for enabling and disabling TC QoS on vports and nodes:
- net/mlx5: Add support for setting tc-bw on nodes
- net/mlx5: Add traffic class scheduling support for vport QoS
Support for setting tc-bw on rate objects:
- net/mlx5: Manage TC arbiter nodes and implement full support for tc-bw
Carolina Jubran (6):
net/mlx5: Add support for new scheduling elements
devlink: Extend devlink rate API with traffic classes bandwidth
management
net/mlx5: Add no-op implementation for setting tc-bw on rate objects net/mlx5: Add support for setting tc-bw on nodes
net/mlx5: Add traffic class scheduling support for vport QoS net/mlx5: Manage TC arbiter nodes and implement full support for tc-bw
Cosmin Ratiu (2):
net/mlx5: ifc: Reorganize mlx5_ifc_flow_table_context_bits net/mlx5: qos: Add ifc support for cross-esw scheduling
Itamar Gozlan (2):
net/mlx5: DR, Expand SWS STE callbacks and consolidate common structs net/mlx5: DR, Add support for ConnectX-8 steering
Yevgeny Kliteynik (1):
net/mlx5: Add ConnectX-8 device to ifc
Documentation/netlink/specs/devlink.yaml | 28 +- .../net/ethernet/mellanox/mlx5/core/Makefile | 1 + .../net/ethernet/mellanox/mlx5/core/devlink.c | 2 + .../net/ethernet/mellanox/mlx5/core/esw/qos.c | 795 +++++++++++++++++- .../net/ethernet/mellanox/mlx5/core/esw/qos.h | 4 + .../net/ethernet/mellanox/mlx5/core/eswitch.h | 13 +- drivers/net/ethernet/mellanox/mlx5/core/rl.c | 4 + .../mlx5/core/steering/sws/dr_domain.c | 2 +- .../mellanox/mlx5/core/steering/sws/dr_ste.c | 6 +- .../mellanox/mlx5/core/steering/sws/dr_ste.h | 19 +- .../mlx5/core/steering/sws/dr_ste_v0.c | 6 +- .../mlx5/core/steering/sws/dr_ste_v1.c | 207 +---- .../mlx5/core/steering/sws/dr_ste_v1.h | 147 +++- .../mlx5/core/steering/sws/dr_ste_v2.c | 169 +--- .../mlx5/core/steering/sws/dr_ste_v2.h | 168 ++++ .../mlx5/core/steering/sws/dr_ste_v3.c | 221 +++++ .../mlx5/core/steering/sws/mlx5_ifc_dr.h | 40 + .../mellanox/mlx5/core/steering/sws/mlx5dr.h | 2 +- include/linux/mlx5/mlx5_ifc.h | 56 +- include/net/devlink.h | 7 + include/uapi/linux/devlink.h | 4 + net/devlink/netlink_gen.c | 15 +- net/devlink/netlink_gen.h | 1 + net/devlink/rate.c | 124 +++ 24 files changed, 1645 insertions(+), 396 deletions(-) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/steering/sws/dr_ste_v2.h create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/steering/sws/dr_ste_v3.c
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v4 1/11 net/mlx5: ifc: Reorganize mlx5_ifc_flow_table_context_bits
From: Cosmin Ratiu <[email protected]>
The nested union at the end is not in the same style as the rest of the code, so un-nest it to make the style uniformly applied again.
--- include/linux/mlx5/mlx5_ifc.h | 30 +++++++++++++++++------------- 1 file changed, 17 insertions(+), 13 deletions(-)
include/linux/mlx5/mlx5_ifc.h +17 -13
@@ -6324,6 +6324,20 @@ struct mlx5_ifc_modify_other_hca_cap_in_bits {struct mlx5_ifc_other_hca_cap_bits other_capability; }; +struct mlx5_ifc_sw_owner_icm_root_params_bits { +u8 sw_owner_icm_root_1[0x40 ]; + +u8 sw_owner_icm_root_0[0x40 ]; +}; + +struct mlx5_ifc_rtc_params_bits { +u8 rtc_id_0[0x20 ]; + +u8 rtc_id_1[0x20 ]; + +u8 reserved_at_40[0x40 ]; +}; +struct mlx5_ifc_flow_table_context_bits {u8 reformat_en[0x1 ];u8 decap_en[0x1 ];
@@ -6342,20 +6356,10 @@ struct mlx5_ifc_flow_table_context_bits {u8 lag_master_next_table_id[0x18 ];u8 reserved_at_60[0x60 ]; -union { -struct { -u8 sw_owner_icm_root_1[0x40 ]; - -u8 sw_owner_icm_root_0[0x40 ]; - } sws; -struct { -u8 rtc_id_0[0x20 ]; -u8 rtc_id_1[0x20 ]; - -u8 reserved_at_100[0x40 ]; - - } hws; +union { +struct mlx5_ifc_sw_owner_icm_root_params_bits sws; +struct mlx5_ifc_rtc_params_bits hws; }; };
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v4 2/11 net/mlx5: Add ConnectX-8 device to ifc
In reply to Tariq ToukanFrom: Yevgeny Kliteynik <[email protected]>
In preparation for ConnectX-8 SWS support, add enum for the new device type.
--- include/linux/mlx5/mlx5_ifc.h | 1 + 1 file changed, 1 insertion(+)
include/linux/mlx5/mlx5_ifc.h +1 -0
@@ -1590,6 +1590,7 @@ enum {MLX5_STEERING_FORMAT_CONNECTX_5 =0 ,MLX5_STEERING_FORMAT_CONNECTX_6DX =1 ,MLX5_STEERING_FORMAT_CONNECTX_7 =2 , + MLX5_STEERING_FORMAT_CONNECTX_8 = 3, }; struct mlx5_ifc_cmd_hca_cap_bits {
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v4 3/11 net/mlx5: Add support for new scheduling elements
In reply to Tariq ToukanFrom: Carolina Jubran <[email protected]>
Introduce new scheduling elements in the E-Switch QoS hierarchy to enhance traffic management capabilities. This patch adds support for:
- Rate Limit scheduling elements: Enables bandwidth limitation across multiple nodes without a shared ancestor, providing a mechanism for more granular control of bandwidth allocation.
- Traffic Class Transmit Scheduling Arbiter (TSAR): Introduces the infrastructure for creating Traffic Class TSARs, allowing hierarchical arbitration based on traffic classes.
- Traffic Class Arbiter TSAR: Adds support for a TSAR capable of managing arbitration between multiple traffic classes, enabling improved bandwidth prioritization and traffic management.
No functional changes are introduced in this patch.
--- drivers/net/ethernet/mellanox/mlx5/core/rl.c | 4 ++++ include/linux/mlx5/mlx5_ifc.h | 14 +++++++++++--- 2 files changed, 15 insertions(+), 3 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/rl.c +4 -0
@@ -56,6 +56,8 @@ bool mlx5_qos_tsar_type_supported(struct mlx5_core_dev *dev, int type, u8 hierarreturn cap &TSAR_TYPE_CAP_MASK_ROUND_ROBIN ;case TSAR_ELEMENT_TSAR_TYPE_ETS :return cap &TSAR_TYPE_CAP_MASK_ETS ; +case TSAR_ELEMENT_TSAR_TYPE_TC_ARB : +return cap &TSAR_TYPE_CAP_MASK_TC_ARB ; }return false;
@@ -87,6 +89,8 @@ bool mlx5_qos_element_type_supported(struct mlx5_core_dev *dev, int type, u8 hiereturn cap &ELEMENT_TYPE_CAP_MASK_PARA_VPORT_TC ;case SCHEDULING_CONTEXT_ELEMENT_TYPE_QUEUE_GROUP :return cap &ELEMENT_TYPE_CAP_MASK_QUEUE_GROUP ; +case SCHEDULING_CONTEXT_ELEMENT_TYPE_RATE_LIMIT : +return cap &ELEMENT_TYPE_CAP_MASK_RATE_LIMIT ; }return false;
include/linux/mlx5/mlx5_ifc.h +11 -3
@@ -1103,7 +1103,8 @@ struct mlx5_ifc_qos_cap_bits {u8 packet_pacing_min_rate[0x20 ]; -u8 reserved_at_80[0x10 ]; +u8 reserved_at_80[0xb ]; +u8 log_esw_max_rate_limit[0x5 ];u8 packet_pacing_rate_table_size[0x10 ];u8 esw_element_type[0x10 ];
@@ -4104,6 +4105,7 @@ enum {SCHEDULING_CONTEXT_ELEMENT_TYPE_VPORT_TC =0x2 ,SCHEDULING_CONTEXT_ELEMENT_TYPE_PARA_VPORT_TC =0x3 ,SCHEDULING_CONTEXT_ELEMENT_TYPE_QUEUE_GROUP =0x4 , + SCHEDULING_CONTEXT_ELEMENT_TYPE_RATE_LIMIT = 0x5, }; enum {
@@ -4112,22 +4114,26 @@ enum {ELEMENT_TYPE_CAP_MASK_VPORT_TC =1 <<2 ,ELEMENT_TYPE_CAP_MASK_PARA_VPORT_TC =1 <<3 ,ELEMENT_TYPE_CAP_MASK_QUEUE_GROUP =1 <<4 , + ELEMENT_TYPE_CAP_MASK_RATE_LIMIT = 1 << 5, };enum {TSAR_ELEMENT_TSAR_TYPE_DWRR =0x0 ,TSAR_ELEMENT_TSAR_TYPE_ROUND_ROBIN =0x1 ,TSAR_ELEMENT_TSAR_TYPE_ETS =0x2 , +TSAR_ELEMENT_TSAR_TYPE_TC_ARB =0x3 , };enum {TSAR_TYPE_CAP_MASK_DWRR =1 <<0 ,TSAR_TYPE_CAP_MASK_ROUND_ROBIN =1 <<1 ,TSAR_TYPE_CAP_MASK_ETS =1 <<2 , +TSAR_TYPE_CAP_MASK_TC_ARB =1 <<3 , };struct mlx5_ifc_tsar_element_bits { -u8 reserved_at_0[0x8 ]; +u8 traffic_class[0x4 ]; +u8 reserved_at_4[0x4 ];u8 tsar_type[0x8 ];u8 reserved_at_10[0x10 ]; };
@@ -4164,7 +4170,9 @@ struct mlx5_ifc_scheduling_context_bits {u8 max_average_bw[0x20 ]; -u8 reserved_at_e0[0x120 ]; +u8 max_bw_obj_id[0x20 ]; + +u8 reserved_at_100[0x100 ]; };struct mlx5_ifc_rqtc_bits {
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v4 4/11 net/mlx5: qos: Add ifc support for cross-esw scheduling
In reply to Tariq ToukanFrom: Cosmin Ratiu <[email protected]>
This adds the capability bit and the vport element fields related to cross-esw scheduling.
--- include/linux/mlx5/mlx5_ifc.h | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)
include/linux/mlx5/mlx5_ifc.h +8 -3
@@ -1095,7 +1095,9 @@ struct mlx5_ifc_qos_cap_bits {u8 log_esw_max_sched_depth[0x4 ];u8 reserved_at_10[0x10 ]; -u8 reserved_at_20[0xb ]; +u8 reserved_at_20[0x9 ]; +u8 esw_cross_esw_sched[0x1 ]; +u8 reserved_at_2a[0x1 ];u8 log_max_qos_nic_queue_group[0x5 ];u8 reserved_at_30[0x10 ];
@@ -4139,13 +4141,16 @@ struct mlx5_ifc_tsar_element_bits { };struct mlx5_ifc_vport_element_bits { -u8 reserved_at_0[0x10 ]; +u8 reserved_at_0[0x4 ]; +u8 eswitch_owner_vhca_id_valid[0x1 ]; +u8 eswitch_owner_vhca_id[0xb ];u8 vport_number[0x10 ]; };struct mlx5_ifc_vport_tc_element_bits {u8 traffic_class[0x4 ]; -u8 reserved_at_4[0xc ]; +u8 eswitch_owner_vhca_id_valid[0x1 ]; +u8 eswitch_owner_vhca_id[0xb ];u8 vport_number[0x10 ]; };
Signature · 1 line
2.44.0
Re: [PATCH net-next V4 04/11] net/mlx5: qos: Add ifc support for cross-esw scheduling
This IFC patch is targeted to mlx5-next.
Sorry for the confusion.
Quoted 45 lines · Tariq Toukan
From: Cosmin Ratiu <[email protected]>
This adds the capability bit and the vport element fields related to cross-esw scheduling.
Signed-off-by Cosmin Ratiu <[email protected]>Signed-off-by Tariq Toukan <[email protected]>--- include/linux/mlx5/mlx5_ifc.h | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)include/linux/mlx5/mlx5_ifc.h +8
-3@@ -1095,7 +1095,9 @@ struct mlx5_ifc_qos_cap_bits {u8 log_esw_max_sched_depth[0x4 ];u8 reserved_at_10[0x10 ]; -u8 reserved_at_20[0xb ]; +u8 reserved_at_20[0x9 ]; +u8 esw_cross_esw_sched[0x1 ]; +u8 reserved_at_2a[0x1 ];u8 log_max_qos_nic_queue_group[0x5 ];u8 reserved_at_30[0x10 ];@@ -4139,13 +4141,16 @@ struct mlx5_ifc_tsar_element_bits { };struct mlx5_ifc_vport_element_bits { -u8 reserved_at_0[0x10 ]; +u8 reserved_at_0[0x4 ]; +u8 eswitch_owner_vhca_id_valid[0x1 ]; +u8 eswitch_owner_vhca_id[0xb ];u8 vport_number[0x10 ]; };struct mlx5_ifc_vport_tc_element_bits {u8 traffic_class[0x4 ]; -u8 reserved_at_4[0xc ]; +u8 eswitch_owner_vhca_id_valid[0x1 ]; +u8 eswitch_owner_vhca_id[0xb ];u8 vport_number[0x10 ]; };
Tariq Toukan
maintainerPATCH v4 5/11 net/mlx5: DR, Expand SWS STE callbacks and consolidate common structs
In reply to Tariq ToukanFrom: Itamar Gozlan <[email protected]>
Expand SWS STE callbacks to support ConnectX-8 hardware. Move common enums and structures to a shared header file.
--- .../mellanox/mlx5/core/steering/sws/dr_ste.c | 4 +- .../mellanox/mlx5/core/steering/sws/dr_ste.h | 18 +- .../mlx5/core/steering/sws/dr_ste_v0.c | 6 +- .../mlx5/core/steering/sws/dr_ste_v1.c | 207 ++++-------------- .../mlx5/core/steering/sws/dr_ste_v1.h | 147 ++++++++++++- .../mlx5/core/steering/sws/dr_ste_v2.c | 169 +------------- .../mlx5/core/steering/sws/dr_ste_v2.h | 168 ++++++++++++++ 7 files changed, 377 insertions(+), 342 deletions(-) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/steering/sws/dr_ste_v2.h
drivers/net/ethernet/mellanox/mlx5/core/steering/sws/dr_ste.c +2 -2
@@ -555,7 +555,7 @@ void mlx5dr_ste_set_actions_tx(struct mlx5dr_ste_ctx *ste_ctx,struct mlx5dr_ste_actions_attr *attr, u32 *added_stes) { - ste_ctx->set_actions_tx(dmn, action_type_set, ste_ctx->actions_caps, + ste_ctx->set_actions_tx(ste_ctx, dmn, action_type_set, ste_ctx->actions_caps,hw_ste_arr ,attr ,added_stes ); }
@@ -566,7 +566,7 @@ void mlx5dr_ste_set_actions_rx(struct mlx5dr_ste_ctx *ste_ctx,struct mlx5dr_ste_actions_attr *attr, u32 *added_stes) { - ste_ctx->set_actions_rx(dmn, action_type_set, ste_ctx->actions_caps, + ste_ctx->set_actions_rx(ste_ctx, dmn, action_type_set, ste_ctx->actions_caps,hw_ste_arr ,attr ,added_stes ); }
drivers/net/ethernet/mellanox/mlx5/core/steering/sws/dr_ste.h +15 -3
@@ -160,13 +160,15 @@ struct mlx5dr_ste_ctx {/* Actions */ u32 actions_caps; -void (*set_actions_rx)(struct mlx5dr_domain *dmn, +void (*set_actions_rx)(struct mlx5dr_ste_ctx *ste_ctx, +struct mlx5dr_domain *dmn,u8 *action_type_set,u32 actions_caps,u8 *hw_ste_arr,struct mlx5dr_ste_actions_attr *attr,u32 *added_stes); - void (*set_actions_tx)(struct mlx5dr_domain *dmn, + void (*set_actions_tx)(struct mlx5dr_ste_ctx *ste_ctx, + struct mlx5dr_domain *dmn, u8 *action_type_set, u32 actions_caps, u8 *hw_ste_arr,
@@ -197,7 +199,17 @@ struct mlx5dr_ste_ctx { u16 *used_hw_action_num);int (*alloc_modify_hdr_chunk)(struct mlx5dr_action *action);void (*dealloc_modify_hdr_chunk)(struct mlx5dr_action *action); - +/* Actions bit set */ +void (*set_encap)(u8 *hw_ste_p,u8 *d_action, +u32 reformat_id,int size); +void (*set_push_vlan)(u8 *ste,u8 *d_action, +u32 vlan_hdr); +void (*set_pop_vlan)(u8 *hw_ste_p,u8 *s_action, +u8 vlans_num); +void (*set_rx_decap)(u8 *hw_ste_p,u8 *s_action); +void (*set_encap_l3)(u8 *hw_ste_p,u8 *frst_s_action, +u8 *scnd_d_action,u32 reformat_id, +int size);/* Send */ void (*prepare_for_postsend)(u8 *hw_ste_p,u32 ste_size); };
drivers/net/ethernet/mellanox/mlx5/core/steering/sws/dr_ste_v0.c +4 -2
@@ -406,7 +406,8 @@ static void dr_ste_v0_arr_init_next(u8 **last_ste, } static void -dr_ste_v0_set_actions_tx(struct mlx5dr_domain *dmn, +dr_ste_v0_set_actions_tx(struct mlx5dr_ste_ctx *ste_ctx, + struct mlx5dr_domain *dmn, u8 *action_type_set, u32 actions_caps, u8 *last_ste,
@@ -476,7 +477,8 @@ dr_ste_v0_set_actions_tx(struct mlx5dr_domain *dmn, } static void -dr_ste_v0_set_actions_rx(struct mlx5dr_domain *dmn, +dr_ste_v0_set_actions_rx(struct mlx5dr_ste_ctx *ste_ctx, + struct mlx5dr_domain *dmn, u8 *action_type_set, u32 actions_caps, u8 *last_ste,
drivers/net/ethernet/mellanox/mlx5/core/steering/sws/dr_ste_v1.c +38 -169
325 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/steering/sws/dr_ste_v1.h +143 -4
160 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/steering/sws/dr_ste_v2.c +7 -162
181 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/steering/sws/dr_ste_v2.h new file +168 -0
168 lines not on this page · raw message
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v4 6/11 net/mlx5: DR, Add support for ConnectX-8 steering
In reply to Tariq ToukanFrom: Itamar Gozlan <[email protected]>
Add support for a new steering format version that is implemented by ConnectX-8.
Except for several differences, the STEv3 is identical to STEv2, so for most callbacks STEv3 context struct will call STEv2 functions.
--- .../net/ethernet/mellanox/mlx5/core/Makefile | 1 + .../mlx5/core/steering/sws/dr_domain.c | 2 +- .../mellanox/mlx5/core/steering/sws/dr_ste.c | 2 + .../mellanox/mlx5/core/steering/sws/dr_ste.h | 1 + .../mlx5/core/steering/sws/dr_ste_v3.c | 221 ++++++++++++++++++ .../mlx5/core/steering/sws/mlx5_ifc_dr.h | 40 ++++ .../mellanox/mlx5/core/steering/sws/mlx5dr.h | 2 +- 7 files changed, 267 insertions(+), 2 deletions(-) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/steering/sws/dr_ste_v3.c
drivers/net/ethernet/mellanox/mlx5/core/Makefile +1 -0
@@ -123,6 +123,7 @@ mlx5_core-$(CONFIG_MLX5_SW_STEERING) += steering/sws/dr_domain.o \ steering/sws/dr_ste_v0.o \ steering/sws/dr_ste_v1.o \ steering/sws/dr_ste_v2.o \ + steering/sws/dr_ste_v3.o \ steering/sws/dr_cmd.o \ steering/sws/dr_fw.o \ steering/sws/dr_action.o \
drivers/net/ethernet/mellanox/mlx5/core/steering/sws/dr_domain.c +1 -1
@@ -8,7 +8,7 @@#define DR_DOMAIN_SW_STEERING_SUPPORTED (dmn, dmn_type) \ ((dmn)->info.caps.dmn_type##_sw_owner || \ ((dmn)->info.caps.dmn_type##_sw_owner_v2 && \ - (dmn)->info.caps.sw_format_ver <= MLX5_STEERING_FORMAT_CONNECTX_7)) + (dmn)->info.caps.sw_format_ver <= MLX5_STEERING_FORMAT_CONNECTX_8))bool mlx5dr_domain_is_support_ptrn_arg (struct mlx5dr_domain *dmn) {
drivers/net/ethernet/mellanox/mlx5/core/steering/sws/dr_ste.c +2 -0
@@ -1458,6 +1458,8 @@ struct mlx5dr_ste_ctx *mlx5dr_ste_get_ctx(u8 version)return mlx5dr_ste_get_ctx_v1 (); else if (version == MLX5_STEERING_FORMAT_CONNECTX_7) return mlx5dr_ste_get_ctx_v2(); + else if (version == MLX5_STEERING_FORMAT_CONNECTX_8) + return mlx5dr_ste_get_ctx_v3();return NULL ; }
drivers/net/ethernet/mellanox/mlx5/core/steering/sws/dr_ste.h +1 -0
@@ -217,5 +217,6 @@ struct mlx5dr_ste_ctx {struct mlx5dr_ste_ctx *mlx5dr_ste_get_ctx_v0 (void );struct mlx5dr_ste_ctx *mlx5dr_ste_get_ctx_v1 (void );struct mlx5dr_ste_ctx *mlx5dr_ste_get_ctx_v2 (void ); +struct mlx5dr_ste_ctx *mlx5dr_ste_get_ctx_v3 (void );#endif /* _DR_STE_ */
drivers/net/ethernet/mellanox/mlx5/core/steering/sws/dr_ste_v3.c new file +221 -0
221 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/steering/sws/mlx5_ifc_dr.h +40 -0
@@ -600,4 +600,44 @@ struct mlx5_ifc_ste_double_action_aso_v1_bits { }; }; +struct mlx5_ifc_ste_single_action_remove_header_v3_bits { +u8 action_id[0x8 ]; +u8 start_anchor[0x7 ]; +u8 end_anchor[0x7 ]; +u8 reserved_at_16[0x1 ]; +u8 outer_l4_remove[0x1 ]; +u8 reserved_at_18[0x4 ]; +u8 decap[0x1 ]; +u8 vni_to_cqe[0x1 ]; +u8 qos_profile[0x2 ]; +}; + +struct mlx5_ifc_ste_single_action_remove_header_size_v3_bits { +u8 action_id[0x8 ]; +u8 start_anchor[0x7 ]; +u8 start_offset[0x8 ]; +u8 outer_l4_remove[0x1 ]; +u8 reserved_at_18[0x2 ]; +u8 remove_size[0x6 ]; +}; + +struct mlx5_ifc_ste_double_action_insert_with_inline_v3_bits { +u8 action_id[0x8 ]; +u8 start_anchor[0x7 ]; +u8 start_offset[0x8 ]; +u8 reserved_at_17[0x9 ]; + +u8 inline_data[0x20 ]; +}; + +struct mlx5_ifc_ste_double_action_insert_with_ptr_v3_bits { +u8 action_id[0x8 ]; +u8 start_anchor[0x7 ]; +u8 start_offset[0x8 ]; +u8 size[0x6 ]; +u8 attributes[0x3 ]; + +u8 pointer[0x20 ]; +}; +#endif /* MLX5_IFC_DR_H */
drivers/net/ethernet/mellanox/mlx5/core/steering/sws/mlx5dr.h +1 -1
@@ -160,7 +160,7 @@ mlx5dr_is_supported(struct mlx5_core_dev *dev) (MLX5_CAP_ESW_FLOWTABLE_FDB (dev, sw_owner) || (MLX5_CAP_ESW_FLOWTABLE_FDB (dev, sw_owner_v2) && (MLX5_CAP_GEN (dev, steering_format_version) <= -MLX5_STEERING_FORMAT_CONNECTX_7 ))); +MLX5_STEERING_FORMAT_CONNECTX_8 ))); }/* buddy functions & structure */
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v4 7/11 devlink: Extend devlink rate API with traffic classes bandwidth management
In reply to Tariq ToukanFrom: Carolina Jubran <[email protected]>
Introduce support for specifying bandwidth proportions between traffic classes (TC) in the devlink-rate API. This new option allows users to allocate bandwidth across multiple traffic classes in a single command.
This feature provides a more granular control over traffic management, especially for scenarios requiring Enhanced Transmission Selection.
Users can now define a specific bandwidth share for each traffic class, such as allocating 20% for TC0 (TCP/UDP) and 80% for TC5 (RoCE).
Example:
DEV=pci/0000:08:00.0
$ devlink port function rate add $DEV/vfs_group tx_share 10Gbit \ tx_max 50Gbit tc-bw 0:20 1:0 2:0 3:0 4:0 5:80 6:0 7:0
$ devlink port function rate set $DEV/vfs_group \ tc-bw 0:20 1:0 2:0 3:0 4:0 5:20 6:60 7:0
Example usage with ynl:
./tools/net/ynl/cli.py --spec Documentation/netlink/specs/devlink.yaml \
--do rate-set --json '{
"bus-name": "pci",
"dev-name": "0000:08:00.0",
"port-index": 1,
"rate-tc-bws": [
{"rate-tc-index": 0, "rate-tc-bw": 50},
{"rate-tc-index": 1, "rate-tc-bw": 50},
{"rate-tc-index": 2, "rate-tc-bw": 0},
{"rate-tc-index": 3, "rate-tc-bw": 0},
{"rate-tc-index": 4, "rate-tc-bw": 0},
{"rate-tc-index": 5, "rate-tc-bw": 0},
{"rate-tc-index": 6, "rate-tc-bw": 0},
{"rate-tc-index": 7, "rate-tc-bw": 0} ]
}'
./tools/net/ynl/cli.py --spec Documentation/netlink/specs/devlink.yaml \
--do rate-get --json '{
"bus-name": "pci",
"dev-name": "0000:08:00.0",
"port-index": 1
}'
output for rate-get:
{'bus-name': 'pci',
'dev-name': '0000:08:00.0',
'port-index': 1,
'rate-tc-bws': [{'rate-tc-bw': 50, 'rate-tc-index': 0},
{'rate-tc-bw': 50, 'rate-tc-index': 1},
{'rate-tc-bw': 0, 'rate-tc-index': 2},
{'rate-tc-bw': 0, 'rate-tc-index': 3},
{'rate-tc-bw': 0, 'rate-tc-index': 4},
{'rate-tc-bw': 0, 'rate-tc-index': 5},
{'rate-tc-bw': 0, 'rate-tc-index': 6},
{'rate-tc-bw': 0, 'rate-tc-index': 7}], 'rate-tx-max': 0,
'rate-tx-priority': 0,
'rate-tx-share': 0,
'rate-tx-weight': 0,
'rate-type': 'leaf'}
--- Documentation/netlink/specs/devlink.yaml | 28 ++++- include/net/devlink.h | 7 ++ include/uapi/linux/devlink.h | 4 + net/devlink/netlink_gen.c | 15 ++- net/devlink/netlink_gen.h | 1 + net/devlink/rate.c | 124 +++++++++++++++++++++++ 6 files changed, 174 insertions(+), 5 deletions(-)
Documentation/netlink/specs/devlink.yaml +27 -1
@@ -820,7 +820,23 @@ attribute-sets: - name: region-direct type: flag - + - + name: rate-tc-bws + type: nest + multi-attr: true + nested-attributes: dl-rate-tc-bws + - + name: rate-tc-index + type: u8 + - + name: rate-tc-bw + type: u32 + doc: | + Specifies the bandwidth allocation for the Traffic Class as a + percentage. + checks: + min: 0 + max: 100 - name: dl-dev-stats subset-of: devlink
@@ -1225,6 +1241,14 @@ attribute-sets: - name: flash type: flag + - + name: dl-rate-tc-bws + subset-of: devlink + attributes: + - + name: rate-tc-index + - + name: rate-tc-bw operations: enum-model: directional
@@ -2149,6 +2173,7 @@ operations: - rate-tx-priority - rate-tx-weight - rate-parent-node-name + - rate-tc-bws - name: rate-new
@@ -2169,6 +2194,7 @@ operations: - rate-tx-priority - rate-tx-weight - rate-parent-node-name + - rate-tc-bws - name: rate-del
include/net/devlink.h +7 -0
@@ -20,6 +20,7 @@#include <uapi/linux/devlink.h> #include <linux/xarray.h> #include <linux/firmware.h> +#include <linux/dcbnl.h> struct devlink ;struct devlink_linecard ;
@@ -117,6 +118,8 @@ struct devlink_rate {u32 tx_priority;u32 tx_weight; + +u32 tc_bw[IEEE_8021QAZ_MAX_TCS ]; };struct devlink_port {
@@ -1469,6 +1472,8 @@ struct devlink_ops { u32 tx_priority, struct netlink_ext_ack *extack);int (*rate_leaf_tx_weight_set)(struct devlink_rate *devlink_rate,void *priv,u32 tx_weight,struct netlink_ext_ack *extack); +int (*rate_leaf_tc_bw_set)(struct devlink_rate *devlink_rate,void *priv, +u32 *tc_bw,struct netlink_ext_ack *extack);int (*rate_node_tx_share_set)(struct devlink_rate *devlink_rate,void *priv,u64 tx_share,struct netlink_ext_ack *extack); int (*rate_node_tx_max_set)(struct devlink_rate *devlink_rate, void *priv,
@@ -1477,6 +1482,8 @@ struct devlink_ops { u32 tx_priority, struct netlink_ext_ack *extack);int (*rate_node_tx_weight_set)(struct devlink_rate *devlink_rate,void *priv,u32 tx_weight,struct netlink_ext_ack *extack); +int (*rate_node_tc_bw_set)(struct devlink_rate *devlink_rate,void *priv, +u32 *tc_bw,struct netlink_ext_ack *extack);int (*rate_node_new)(struct devlink_rate *rate_node,void **priv,struct netlink_ext_ack *extack); int (*rate_node_del)(struct devlink_rate *rate_node, void *priv,
include/uapi/linux/devlink.h +4 -0
@@ -614,6 +614,10 @@ enum devlink_attr {DEVLINK_ATTR_REGION_DIRECT ,/* flag */ +DEVLINK_ATTR_RATE_TC_BWS ,/* nested */ +DEVLINK_ATTR_RATE_TC_INDEX ,/* u8 */ +DEVLINK_ATTR_RATE_TC_BW ,/* u32 */ +/* Add new attributes above here, update the spec in * Documentation/netlink/specs/devlink.yaml and re-generate * net/devlink/netlink_gen.c.
net/devlink/netlink_gen.c +11 -4
@@ -18,6 +18,11 @@ const struct nla_policy devlink_dl_port_function_nl_policy[DEVLINK_PORT_FN_ATTR_ [DEVLINK_PORT_FN_ATTR_CAPS] = NLA_POLICY_BITFIELD32(15), }; +const struct nla_policy devlink_dl_rate_tc_bws_nl_policy[DEVLINK_ATTR_RATE_TC_BW +1 ] = { + [DEVLINK_ATTR_RATE_TC_INDEX ] = { .type =NLA_U8 , }, + [DEVLINK_ATTR_RATE_TC_BW ] =NLA_POLICY_RANGE (NLA_U32 ,0 ,100 ), +}; +const struct nla_policy devlink_dl_selftest_id_nl_policy[DEVLINK_ATTR_SELFTEST_ID_FLASH +1 ] = { [DEVLINK_ATTR_SELFTEST_ID_FLASH ] = { .type =NLA_FLAG , }, };
@@ -496,7 +501,7 @@ static const struct nla_policy devlink_rate_get_dump_nl_policy[DEVLINK_ATTR_DEV_ };/* DEVLINK_CMD_RATE_SET - do */ -static const struct nla_policy devlink_rate_set_nl_policy[DEVLINK_ATTR_RATE_TX_WEIGHT +1 ] = { +static const struct nla_policy devlink_rate_set_nl_policy[DEVLINK_ATTR_RATE_TC_BWS +1 ] = { [DEVLINK_ATTR_BUS_NAME ] = { .type =NLA_NUL_STRING , }, [DEVLINK_ATTR_DEV_NAME ] = { .type =NLA_NUL_STRING , }, [DEVLINK_ATTR_RATE_NODE_NAME ] = { .type =NLA_NUL_STRING , },
@@ -505,10 +510,11 @@ static const struct nla_policy devlink_rate_set_nl_policy[DEVLINK_ATTR_RATE_TX_W [DEVLINK_ATTR_RATE_TX_PRIORITY] = { .type = NLA_U32, }, [DEVLINK_ATTR_RATE_TX_WEIGHT] = { .type = NLA_U32, }, [DEVLINK_ATTR_RATE_PARENT_NODE_NAME] = { .type = NLA_NUL_STRING, }, + [DEVLINK_ATTR_RATE_TC_BWS] = NLA_POLICY_NESTED(devlink_dl_rate_tc_bws_nl_policy), };/* DEVLINK_CMD_RATE_NEW - do */ -static const struct nla_policy devlink_rate_new_nl_policy[DEVLINK_ATTR_RATE_TX_WEIGHT +1 ] = { +static const struct nla_policy devlink_rate_new_nl_policy[DEVLINK_ATTR_RATE_TC_BWS +1 ] = { [DEVLINK_ATTR_BUS_NAME ] = { .type =NLA_NUL_STRING , }, [DEVLINK_ATTR_DEV_NAME ] = { .type =NLA_NUL_STRING , }, [DEVLINK_ATTR_RATE_NODE_NAME ] = { .type =NLA_NUL_STRING , },
@@ -517,6 +523,7 @@ static const struct nla_policy devlink_rate_new_nl_policy[DEVLINK_ATTR_RATE_TX_W [DEVLINK_ATTR_RATE_TX_PRIORITY] = { .type = NLA_U32, }, [DEVLINK_ATTR_RATE_TX_WEIGHT] = { .type = NLA_U32, }, [DEVLINK_ATTR_RATE_PARENT_NODE_NAME] = { .type = NLA_NUL_STRING, }, + [DEVLINK_ATTR_RATE_TC_BWS] = NLA_POLICY_NESTED(devlink_dl_rate_tc_bws_nl_policy), };/* DEVLINK_CMD_RATE_DEL - do */
@@ -1164,7 +1171,7 @@ const struct genl_split_ops devlink_nl_ops[74] = { .doit = devlink_nl_rate_set_doit, .post_doit = devlink_nl_post_doit, .policy = devlink_rate_set_nl_policy, - .maxattr = DEVLINK_ATTR_RATE_TX_WEIGHT, + .maxattr = DEVLINK_ATTR_RATE_TC_BWS, .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, }, {
@@ -1174,7 +1181,7 @@ const struct genl_split_ops devlink_nl_ops[74] = { .doit = devlink_nl_rate_new_doit, .post_doit = devlink_nl_post_doit, .policy = devlink_rate_new_nl_policy, - .maxattr = DEVLINK_ATTR_RATE_TX_WEIGHT, + .maxattr = DEVLINK_ATTR_RATE_TC_BWS, .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, }, {
net/devlink/netlink_gen.h +1 -0
@@ -13,6 +13,7 @@/* Common nested types */ extern const struct nla_policy devlink_dl_port_function_nl_policy[DEVLINK_PORT_FN_ATTR_CAPS +1 ]; +extern const struct nla_policy devlink_dl_rate_tc_bws_nl_policy[DEVLINK_ATTR_RATE_TC_BW +1 ];extern const struct nla_policy devlink_dl_selftest_id_nl_policy[DEVLINK_ATTR_SELFTEST_ID_FLASH +1 ];/* Ops table for devlink */
net/devlink/rate.c +124 -0
160 lines not on this page · raw message
Signature · 1 line
2.44.0
kernel test robot
build botRe: [PATCH net-next V4 07/11] devlink: Extend devlink rate API with traffic classes bandwidth management
kernel test robot noticed the following build warnings:
Hi Tariq,
kernel test robot noticed the following build warnings:
[auto build test WARNING on net-next/main]
url: https://github.com/intel-lab-lkp/linux/commits/Tariq-Toukan/net-mlx5-ifc-Reorganize-mlx5_ifc_flow_table_context_bits/20241204-124235 base: net-next/main patch link: https://lore.kernel.org/r/20241203202924.228440-8-tariqt%40nvidia.com patch subject: [PATCH net-next V4 07/11] devlink: Extend devlink rate API with traffic classes bandwidth management config: x86_64-buildonly-randconfig-001-20241205 (https://download.01.org/0day-ci/archive/20241205/[email protected]/config) compiler: clang version 19.1.3 (https://github.com/llvm/llvm-project ab51eccf88f5321e7c60591c5546b254b6afab99) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20241205/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
In file included from net/devlink/rate.c:7:
In file included from net/devlink/devl_internal.h:7:
In file included from include/linux/etherdevice.h:20:
In file included from include/linux/if_ether.h:19:
In file included from include/linux/skbuff.h:17:
In file included from include/linux/bvec.h:10:
In file included from include/linux/highmem.h:8:
In file included from include/linux/cacheflush.h:5:
In file included from arch/x86/include/asm/cacheflush.h:5:
In file included from include/linux/mm.h:2223:
include/linux/vmstat.h:504:43: warning: arithmetic between different enumeration types ('enum zone_stat_item' and 'enum numa_stat_item') [-Wenum-enum-conversion]
504 | return vmstat_text[NR_VM_ZONE_STAT_ITEMS +
| ~~~~~~~~~~~~~~~~~~~~~ ^
505 | item];
| ~~~~
include/linux/vmstat.h:511:43: warning: arithmetic between different enumeration types ('enum zone_stat_item' and 'enum numa_stat_item') [-Wenum-enum-conversion]
511 | return vmstat_text[NR_VM_ZONE_STAT_ITEMS +
| ~~~~~~~~~~~~~~~~~~~~~ ^
512 | NR_VM_NUMA_EVENT_ITEMS +
| ~~~~~~~~~~~~~~~~~~~~~~
include/linux/vmstat.h:518:36: warning: arithmetic between different enumeration types ('enum node_stat_item' and 'enum lru_list') [-Wenum-enum-conversion]
518 | return node_stat_name(NR_LRU_BASE + lru) + 3; // skip "nr_"
| ~~~~~~~~~~~ ^ ~~~
include/linux/vmstat.h:524:43: warning: arithmetic between different enumeration types ('enum zone_stat_item' and 'enum numa_stat_item') [-Wenum-enum-conversion]
524 | return vmstat_text[NR_VM_ZONE_STAT_ITEMS +
| ~~~~~~~~~~~~~~~~~~~~~ ^
525 | NR_VM_NUMA_EVENT_ITEMS +
| ~~~~~~~~~~~~~~~~~~~~~~
In file included from net/devlink/rate.c:7:
net/devlink/devl_internal.h:29:19: warning: arithmetic between different enumeration types ('enum devlink_reload_limit' and 'enum devlink_reload_action') [-Wenum-enum-conversion]
29 | u32 reload_stats[DEVLINK_RELOAD_STATS_ARRAY_SIZE];
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
net/devlink/devl_internal.h:26:30: note: expanded from macro 'DEVLINK_RELOAD_STATS_ARRAY_SIZE'
26 | (__DEVLINK_RELOAD_LIMIT_MAX * __DEVLINK_RELOAD_ACTION_MAX)
| ~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~
net/devlink/devl_internal.h:30:26: warning: arithmetic between different enumeration types ('enum devlink_reload_limit' and 'enum devlink_reload_action') [-Wenum-enum-conversion]
30 | u32 remote_reload_stats[DEVLINK_RELOAD_STATS_ARRAY_SIZE];
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
net/devlink/devl_internal.h:26:30: note: expanded from macro 'DEVLINK_RELOAD_STATS_ARRAY_SIZE'
26 | (__DEVLINK_RELOAD_LIMIT_MAX * __DEVLINK_RELOAD_ACTION_MAX)
| ~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> net/devlink/rate.c:403:4: warning: 'snprintf' will always be truncated; specified size is 80, but format string expands to at least 83 [-Wformat-truncation]
403 | NL_SET_ERR_MSG_FMT(info->extack,
| ^
include/linux/netlink.h:116:6: note: expanded from macro 'NL_SET_ERR_MSG_FMT'
116 | if (snprintf(__extack->_msg_buf, NETLINK_MAX_FMTMSG_LEN, \
| ^7 warnings generated.
vim +/snprintf +403 net/devlink/rate.c
381
382 static int devlink_nl_rate_tc_bw_set(struct devlink_rate *devlink_rate,
383 struct genl_info *info)
384 {
385 DECLARE_BITMAP(bitmap, IEEE_8021QAZ_MAX_TCS) = {};
386 struct devlink *devlink = devlink_rate->devlink;
387 const struct devlink_ops *ops = devlink->ops;
388 u32 tc_bw[IEEE_8021QAZ_MAX_TCS] = {};
389 int rem, err = -EOPNOTSUPP, i;
390 struct nlattr *attr;
391
392 nla_for_each_attr(attr, genlmsg_data(info->genlhdr),
393 genlmsg_len(info->genlhdr), rem) {
394 if (nla_type(attr) == DEVLINK_ATTR_RATE_TC_BWS) {
395 err = devlink_nl_rate_tc_bw_parse(attr, tc_bw, bitmap, info->extack);
396 if (err)
397 return err;
398 }
399 }
400
401 for (i = 0; i < IEEE_8021QAZ_MAX_TCS; i++) {
402 if (!test_bit(i, bitmap)) {
> 403 NL_SET_ERR_MSG_FMT(info->extack,
404 "Incomplete traffic class bandwidth values, all %u traffic classes must be specified",
405 IEEE_8021QAZ_MAX_TCS);
406 return -EINVAL;
407 }
408 }
409
410 if (devlink_rate_is_leaf(devlink_rate))
411 err = ops->rate_leaf_tc_bw_set(devlink_rate, devlink_rate->priv, tc_bw,
412 info->extack);
413 else if (devlink_rate_is_node(devlink_rate))
414 err = ops->rate_node_tc_bw_set(devlink_rate, devlink_rate->priv, tc_bw,
415 info->extack);
416
417 if (err)
418 return err;
419
420 memcpy(devlink_rate->tc_bw, tc_bw, sizeof(tc_bw));
421
422 return 0;
423 }
424
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [PATCH net-next V4 07/11] devlink: Extend devlink rate API with traffic classes bandwidth management
In reply to Tariq Toukankernel test robot noticed the following build warnings:
Hi Tariq,
kernel test robot noticed the following build warnings:
[auto build test WARNING on net-next/main]
url: https://github.com/intel-lab-lkp/linux/commits/Tariq-Toukan/net-mlx5-ifc-Reorganize-mlx5_ifc_flow_table_context_bits/20241204-124235 base: net-next/main patch link: https://lore.kernel.org/r/20241203202924.228440-8-tariqt%40nvidia.com patch subject: [PATCH net-next V4 07/11] devlink: Extend devlink rate API with traffic classes bandwidth management config: arm-randconfig-001 (https://download.01.org/0day-ci/archive/20241205/[email protected]/config) compiler: arm-linux-gnueabi-gcc (GCC) 14.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20241205/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
In file included from include/uapi/linux/neighbour.h:6,
from include/linux/netdevice.h:44, from include/linux/etherdevice.h:21, from net/devlink/devl_internal.h:7, from net/devlink/rate.c:7:
net/devlink/rate.c: In function 'devlink_nl_rate_tc_bw_set':
>> include/linux/netlink.h:116:13: warning: ' traffic classes must be spe...' directive output truncated writing 34 bytes into a region of size 32 [-Wformat-truncation=]
116 | if (snprintf(__extack->_msg_buf, NETLINK_MAX_FMTMSG_LEN, \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
117 | "%s" fmt "%s", "", ##args, "") >= \
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
net/devlink/rate.c:403:25: note: in expansion of macro 'NL_SET_ERR_MSG_FMT'
403 | NL_SET_ERR_MSG_FMT(info->extack,
| ^~~~~~~~~~~~~~~~~~
include/linux/netlink.h:116:13: note: 'snprintf' output 83 bytes into a destination of size 80
116 | if (snprintf(__extack->_msg_buf, NETLINK_MAX_FMTMSG_LEN, \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
117 | "%s" fmt "%s", "", ##args, "") >= \
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
net/devlink/rate.c:403:25: note: in expansion of macro 'NL_SET_ERR_MSG_FMT'
403 | NL_SET_ERR_MSG_FMT(info->extack,
| ^~~~~~~~~~~~~~~~~~vim +116 include/linux/netlink.h
2d4bc93368f5a0d Johannes Berg 2017-04-12 107
51c352bdbcd23d7 Edward Cree 2022-10-18 108 /* We splice fmt with %s at each end even in the snprintf so that both calls
51c352bdbcd23d7 Edward Cree 2022-10-18 109 * can use the same string constant, avoiding its duplication in .ro
51c352bdbcd23d7 Edward Cree 2022-10-18 110 */
51c352bdbcd23d7 Edward Cree 2022-10-18 111 #define NL_SET_ERR_MSG_FMT(extack, fmt, args...) do { \
51c352bdbcd23d7 Edward Cree 2022-10-18 112 struct netlink_ext_ack *__extack = (extack); \
51c352bdbcd23d7 Edward Cree 2022-10-18 113 \
51c352bdbcd23d7 Edward Cree 2022-10-18 114 if (!__extack) \
51c352bdbcd23d7 Edward Cree 2022-10-18 115 break; \
51c352bdbcd23d7 Edward Cree 2022-10-18 @116 if (snprintf(__extack->_msg_buf, NETLINK_MAX_FMTMSG_LEN, \
51c352bdbcd23d7 Edward Cree 2022-10-18 117 "%s" fmt "%s", "", ##args, "") >= \
51c352bdbcd23d7 Edward Cree 2022-10-18 118 NETLINK_MAX_FMTMSG_LEN) \
51c352bdbcd23d7 Edward Cree 2022-10-18 119 net_warn_ratelimited("%s" fmt "%s", "truncated extack: ", \
51c352bdbcd23d7 Edward Cree 2022-10-18 120 ##args, "\n"); \
51c352bdbcd23d7 Edward Cree 2022-10-18 121 \
51c352bdbcd23d7 Edward Cree 2022-10-18 122 do_trace_netlink_extack(__extack->_msg_buf); \
51c352bdbcd23d7 Edward Cree 2022-10-18 123 \
51c352bdbcd23d7 Edward Cree 2022-10-18 124 __extack->_msg = __extack->_msg_buf; \
51c352bdbcd23d7 Edward Cree 2022-10-18 125 } while (0)
51c352bdbcd23d7 Edward Cree 2022-10-18 126 Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Tariq Toukan
maintainerPATCH v4 8/11 net/mlx5: Add no-op implementation for setting tc-bw on rate objects
In reply to Tariq ToukanFrom: Carolina Jubran <[email protected]>
Introduce mlx5_esw_devlink_rate_node_tc_bw_set() and mlx5_esw_devlink_rate_leaf_tc_bw_set() with no-op logic.
Future patches will add support for setting traffic class bandwidth on rate objects.
--- drivers/net/ethernet/mellanox/mlx5/core/devlink.c | 2 ++ drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c | 14 ++++++++++++++ drivers/net/ethernet/mellanox/mlx5/core/esw/qos.h | 4 ++++ 3 files changed, 20 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/devlink.c +2 -0
@@ -320,6 +320,8 @@ static const struct devlink_ops mlx5_devlink_ops = { .eswitch_encap_mode_get = mlx5_devlink_eswitch_encap_mode_get, .rate_leaf_tx_share_set = mlx5_esw_devlink_rate_leaf_tx_share_set, .rate_leaf_tx_max_set = mlx5_esw_devlink_rate_leaf_tx_max_set, + .rate_leaf_tc_bw_set = mlx5_esw_devlink_rate_leaf_tc_bw_set, + .rate_node_tc_bw_set = mlx5_esw_devlink_rate_node_tc_bw_set, .rate_node_tx_share_set = mlx5_esw_devlink_rate_node_tx_share_set, .rate_node_tx_max_set = mlx5_esw_devlink_rate_node_tx_max_set, .rate_node_new = mlx5_esw_devlink_rate_node_new,
drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c +14 -0
@@ -882,6 +882,20 @@ int mlx5_esw_devlink_rate_leaf_tx_max_set(struct devlink_rate *rate_leaf, void *return err; } +int mlx5_esw_devlink_rate_leaf_tc_bw_set (struct devlink_rate *rate_leaf,void *priv, +u32 *tc_bw,struct netlink_ext_ack *extack) +{ +NL_SET_ERR_MSG_MOD (extack,"TC bandwidth shares are not supported on leafs" ); +return -EOPNOTSUPP ; +} + +int mlx5_esw_devlink_rate_node_tc_bw_set (struct devlink_rate *rate_node,void *priv, +u32 *tc_bw,struct netlink_ext_ack *extack) +{ +NL_SET_ERR_MSG_MOD (extack,"TC bandwidth shares are not supported on nodes" ); +return -EOPNOTSUPP ; +} +int mlx5_esw_devlink_rate_node_tx_share_set (struct devlink_rate *rate_node,void *priv,u64 tx_share,struct netlink_ext_ack *extack) {
drivers/net/ethernet/mellanox/mlx5/core/esw/qos.h +4 -0
@@ -21,6 +21,10 @@ int mlx5_esw_devlink_rate_leaf_tx_share_set(struct devlink_rate *rate_leaf, void u64 tx_share, struct netlink_ext_ack *extack);int mlx5_esw_devlink_rate_leaf_tx_max_set (struct devlink_rate *rate_leaf,void *priv,u64 tx_max,struct netlink_ext_ack *extack); +int mlx5_esw_devlink_rate_leaf_tc_bw_set (struct devlink_rate *rate_node,void *priv, +u32 *tc_bw,struct netlink_ext_ack *extack); +int mlx5_esw_devlink_rate_node_tc_bw_set (struct devlink_rate *rate_node,void *priv, +u32 *tc_bw,struct netlink_ext_ack *extack);int mlx5_esw_devlink_rate_node_tx_share_set (struct devlink_rate *rate_node,void *priv,u64 tx_share,struct netlink_ext_ack *extack); int mlx5_esw_devlink_rate_node_tx_max_set(struct devlink_rate *rate_node, void *priv,
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v4 9/11 net/mlx5: Add support for setting tc-bw on nodes
In reply to Tariq ToukanFrom: Carolina Jubran <[email protected]>
Introduce support for enabling and disabling Traffic Class (TC) arbitration for existing devlink rate nodes. This patch adds support for a new scheduling node type, SCHED_NODE_TYPE_TC_ARBITER_TSAR.
Key changes include:
- New helper functions for transitioning existing rate nodes to TC arbiter nodes and vice versa. These functions handle the allocation of TC arbiter nodes, copying of child nodes, and restoring vport QoS settings when TC arbitration is disabled.
- Implementation of mlx5_esw_devlink_rate_node_tc_bw_set() to manage tc-bw configuration on nodes.
- Introduced stubs for esw_qos_tc_arbiter_scheduling_setup() and esw_qos_tc_arbiter_scheduling_teardown(), which will be extended in future patches to provide full support for tc-bw on devlink rate objects.
- Validation functions for tc-bw settings, allowing graceful handling of unsupported traffic class bandwidth configurations.
- Updated __esw_qos_alloc_node() to insert the new node into the parent’s children list only if the parent is not NULL. For the root TSAR, the new node is inserted directly after the allocation call.
This patch lays the groundwork for future support for configuring tc-bw on devlink rate nodes. Although the infrastructure is in place, full support for tc-bw is not yet implemented; attempts to set tc-bw on nodes will return -EOPNOTSUPP.
No functional changes are introduced at this stage.
--- .../net/ethernet/mellanox/mlx5/core/esw/qos.c | 260 +++++++++++++++++- 1 file changed, 246 insertions(+), 14 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c +246 -14
@@ -64,11 +64,13 @@ static void esw_qos_domain_release(struct mlx5_eswitch *esw)enum sched_node_type {SCHED_NODE_TYPE_VPORTS_TSAR ,SCHED_NODE_TYPE_VPORT , +SCHED_NODE_TYPE_TC_ARBITER_TSAR , };static const char *const sched_node_type_str[] = { [SCHED_NODE_TYPE_VPORTS_TSAR ] ="vports TSAR" , [SCHED_NODE_TYPE_VPORT ] ="vport" , + [SCHED_NODE_TYPE_TC_ARBITER_TSAR ] ="TC Arbiter TSAR" , };struct mlx5_esw_sched_node {
@@ -92,6 +94,13 @@ struct mlx5_esw_sched_node {struct mlx5_vport *vport; }; +static int esw_qos_num_tcs (struct mlx5_core_dev *dev) +{ +int num_tcs =mlx5_max_tc (dev) +1 ; + +return num_tcs <IEEE_8021QAZ_MAX_TCS ? num_tcs :IEEE_8021QAZ_MAX_TCS ; +} +static void esw_qos_node_set_parent (struct mlx5_esw_sched_node *node,struct mlx5_esw_sched_node *parent) {
@@ -101,6 +110,15 @@ esw_qos_node_set_parent(struct mlx5_esw_sched_node *node, struct mlx5_esw_sched_ node->esw = parent->esw; } +static void +esw_qos_nodes_set_parent (struct list_head *nodes,struct mlx5_esw_sched_node *parent) +{ +struct mlx5_esw_sched_node *node, *tmp; + +list_for_each_entry_safe (node, tmp, nodes, entry) +esw_qos_node_set_parent (node, parent); +} +void mlx5_esw_qos_vport_qos_free (struct mlx5_vport *vport) {kfree (vport->qos.sched_node);
@@ -126,16 +144,23 @@ mlx5_esw_qos_vport_get_parent(const struct mlx5_vport *vport)static void esw_qos_sched_elem_warn (struct mlx5_esw_sched_node *node,int err,const char *op) { -if (node->vport) { +switch (node->type) { +case SCHED_NODE_TYPE_VPORT :esw_warn (node->esw->dev,"E-Switch %s %s scheduling element failed (vport=%d,err=%d)\n" , op, sched_node_type_str[node->type], node->vport->vport, err); -return ; +break ; +case SCHED_NODE_TYPE_TC_ARBITER_TSAR : +case SCHED_NODE_TYPE_VPORTS_TSAR : +esw_warn (node->esw->dev, +"E-Switch %s %s scheduling element failed (err=%d)\n" , + op, sched_node_type_str[node->type], err); +break ; +default : +esw_warn (node->esw->dev, +"E-Switch %s scheduling element failed (err=%d)\n" , op, err); +break ; } - -esw_warn (node->esw->dev, -"E-Switch %s %s scheduling element failed (err=%d)\n" , - op, sched_node_type_str[node->type], err); } static int esw_qos_node_create_sched_element(struct mlx5_esw_sched_node *node, void *ctx,
@@ -358,7 +383,6 @@ static struct mlx5_esw_sched_node * __esw_qos_alloc_node(struct mlx5_eswitch *esw, u32 tsar_ix, enum sched_node_type type, struct mlx5_esw_sched_node *parent) { - struct list_head *parent_children; struct mlx5_esw_sched_node *node; node =kzalloc (sizeof (*node),GFP_KERNEL );
@@ -370,8 +394,10 @@ __esw_qos_alloc_node(struct mlx5_eswitch *esw, u32 tsar_ix, enum sched_node_type node->type = type; node->parent = parent;INIT_LIST_HEAD (&node->children); - parent_children = parent ? &parent->children : &esw->qos.domain->nodes; -list_add_tail (&node->entry, parent_children); +if (parent) +list_add_tail (&node->entry, &parent->children); +else +INIT_LIST_HEAD (&node->entry);return node; }
@@ -409,6 +435,7 @@ __esw_qos_create_vports_sched_node(struct mlx5_eswitch *esw, struct mlx5_esw_sch goto err_alloc_node; } +list_add_tail (&node->entry, &esw->qos.domain->nodes);esw_qos_normalize_min_rate (esw,NULL , extack);trace_mlx5_esw_node_qos_create (esw->dev, node, node->ix);
@@ -475,11 +502,11 @@ static int esw_qos_create(struct mlx5_eswitch *esw, struct netlink_ext_ack *exta/* The eswitch doesn't support scheduling nodes. * Create a software-only node0 using the root TSAR to attach vport QoS to. */ -if (!__esw_qos_alloc_node (esw, - esw->qos.root_tsar_ix, -SCHED_NODE_TYPE_VPORTS_TSAR , +if (!__esw_qos_alloc_node (esw, esw->qos.root_tsar_ix,SCHED_NODE_TYPE_VPORTS_TSAR ,NULL )) esw->qos.node0 =ERR_PTR (-ENOMEM ); +else +list_add_tail (&esw->qos.node0->entry, &esw->qos.domain->nodes); }if (IS_ERR (esw->qos.node0)) { err =PTR_ERR (esw->qos.node0);
@@ -537,6 +564,17 @@ static void esw_qos_put(struct mlx5_eswitch *esw)esw_qos_destroy (esw); } +static void esw_qos_tc_arbiter_scheduling_teardown (struct mlx5_esw_sched_node *node, +struct netlink_ext_ack *extack) +{} + +static int esw_qos_tc_arbiter_scheduling_setup (struct mlx5_esw_sched_node *node, +struct netlink_ext_ack *extack) +{ +NL_SET_ERR_MSG_MOD (extack,"TC arbiter elements are not supported." ); +return -EOPNOTSUPP ; +} +static void esw_qos_vport_disable (struct mlx5_vport *vport,struct netlink_ext_ack *extack) {struct mlx5_esw_sched_node *vport_node = vport->qos.sched_node;
@@ -699,6 +737,157 @@ static int esw_qos_vport_update_parent(struct mlx5_vport *vport, struct mlx5_eswreturn err; } +static void esw_qos_switch_vport_tcs_to_vport (struct mlx5_esw_sched_node *tc_arbiter_node, +struct mlx5_esw_sched_node *node, +struct netlink_ext_ack *extack) +{ +struct mlx5_esw_sched_node *vports_tc_node, *vport_tc_node, *tmp; + + vports_tc_node = list_first_entry(&tc_arbiter_node->children, struct mlx5_esw_sched_node, + entry); + +list_for_each_entry_safe (vport_tc_node, tmp, &vports_tc_node->children, entry) +esw_qos_vport_update_parent (vport_tc_node->vport, node, extack); +} + +static int esw_qos_switch_tc_arbiter_node_to_vports (struct mlx5_esw_sched_node *tc_arbiter_node, +struct mlx5_esw_sched_node *node, +struct netlink_ext_ack *extack) +{ +u32 parent_tsar_ix = node->parent ? node->parent->ix : node->esw->qos.root_tsar_ix; +int err; + + err =esw_qos_create_node_sched_elem (node->esw->dev, parent_tsar_ix, &node->ix); +if (err) { +NL_SET_ERR_MSG_MOD (extack, +"Failed to create scheduling element for vports node when disabliing vports TC QoS" ); +return err; + } + + node->type =SCHED_NODE_TYPE_VPORTS_TSAR ; + +/* Disable TC QoS for vports in the arbiter node. */ +esw_qos_switch_vport_tcs_to_vport (tc_arbiter_node, node, extack); + +return 0 ; +} + +static int esw_qos_switch_vports_node_to_tc_arbiter (struct mlx5_esw_sched_node *node, +struct mlx5_esw_sched_node *tc_arbiter_node, +struct netlink_ext_ack *extack) +{ +struct mlx5_esw_sched_node *vport_node, *tmp; +struct mlx5_vport *vport; +int err; + +/* Enable TC QoS for each vport in the node. */ +list_for_each_entry_safe (vport_node, tmp, &node->children, entry) { + vport = vport_node->vport; + err =esw_qos_vport_update_parent (vport, tc_arbiter_node, extack); +if (err) + goto err_out; + } + +/* Destroy the current vports node TSAR. */ + err =mlx5_destroy_scheduling_element_cmd (node->esw->dev,SCHEDULING_HIERARCHY_E_SWITCH , + node->ix); +if (err) + goto err_out; + +return 0 ; +err_out: +/* Restore vports back into the node if an error occurs. */ +esw_qos_switch_vport_tcs_to_vport (tc_arbiter_node, node,NULL ); + +return err; +} + +static struct mlx5_esw_sched_node *esw_qos_move_node (struct mlx5_esw_sched_node *curr_node) +{ +struct mlx5_esw_sched_node *new_node; + + new_node =__esw_qos_alloc_node (curr_node->esw, curr_node->ix, curr_node->type,NULL ); +if (!IS_ERR (new_node)) +esw_qos_nodes_set_parent (&curr_node->children, new_node); + +return new_node; +} + +static int esw_qos_node_disable_tc_arbitration (struct mlx5_esw_sched_node *node, +struct netlink_ext_ack *extack) +{ +struct mlx5_esw_sched_node *curr_node; +int err; + +if (node->type !=SCHED_NODE_TYPE_TC_ARBITER_TSAR ) +return 0 ; + +/* Allocate a new rate node to hold the current state, which will allow +* for restoring the vports back to this node after disabling TC arbitration. +*/ + curr_node =esw_qos_move_node (node); +if (IS_ERR (curr_node)) { +NL_SET_ERR_MSG_MOD (extack,"Failed setting up vports node" ); +return PTR_ERR (curr_node); + } + +/* Disable TC QoS for all vports, and assign them back to the node. */ + err =esw_qos_switch_tc_arbiter_node_to_vports (curr_node, node, extack); +if (err) + goto err_out; + +/* Clean up the TC arbiter node after disabling TC QoS for vports. */ +esw_qos_tc_arbiter_scheduling_teardown (curr_node, extack); + goto out; +err_out: +esw_qos_nodes_set_parent (&curr_node->children, node); +out: +__esw_qos_free_node (curr_node); +return err; +} + +static int esw_qos_node_enable_tc_arbitration (struct mlx5_esw_sched_node *node, +struct netlink_ext_ack *extack) +{ +struct mlx5_esw_sched_node *curr_node; +int err; + +if (node->type ==SCHED_NODE_TYPE_TC_ARBITER_TSAR ) +return 0 ; + +/* Allocate a new node that will store the information of the current node. +* This will be used later to restore the node if necessary. +*/ + curr_node =esw_qos_move_node (node); +if (IS_ERR (curr_node)) { +NL_SET_ERR_MSG_MOD (extack,"Failed setting up node TC QoS" ); +return PTR_ERR (curr_node); + } + +/* Initialize the TC arbiter node for QoS management. +* This step prepares the node for handling Traffic Class arbitration. +*/ + err =esw_qos_tc_arbiter_scheduling_setup (node, extack); +if (err) + goto err_setup; + +/* Enable TC QoS for each vport within the current node. */ + err =esw_qos_switch_vports_node_to_tc_arbiter (curr_node, node, extack); +if (err) + goto err_switch_vports; + goto out; + +err_switch_vports: +esw_qos_tc_arbiter_scheduling_teardown (node,NULL ); + node->ix = curr_node->ix; + node->type = curr_node->type; +err_setup: +esw_qos_nodes_set_parent (&curr_node->children, node); +out: +__esw_qos_free_node (curr_node); +return err; +} +static u32 mlx5_esw_qos_lag_link_speed_get_locked (struct mlx5_core_dev *mdev) {struct ethtool_link_ksettings lksettings;
@@ -824,6 +1013,30 @@ static int esw_qos_devlink_rate_to_mbps(struct mlx5_core_dev *mdev, const char *return 0 ; } +static bool esw_qos_validate_unsupported_tc_bw (struct mlx5_eswitch *esw,u32 *tc_bw) +{ +int i, num_tcs =esw_qos_num_tcs (esw->dev); + +for (i = num_tcs; i <IEEE_8021QAZ_MAX_TCS ; i++) { +if (tc_bw[i]) +return false; + } + +return true; +} + +static bool esw_qos_tc_bw_disabled (u32 *tc_bw) +{ +int i; + +for (i =0 ; i <IEEE_8021QAZ_MAX_TCS ; i++) { +if (tc_bw[i]) +return false; + } + +return true; +} +int mlx5_esw_qos_init (struct mlx5_eswitch *esw) {if (esw->qos.domain)
@@ -892,8 +1105,27 @@ int mlx5_esw_devlink_rate_leaf_tc_bw_set(struct devlink_rate *rate_leaf, void *pint mlx5_esw_devlink_rate_node_tc_bw_set (struct devlink_rate *rate_node,void *priv,u32 *tc_bw,struct netlink_ext_ack *extack) { -NL_SET_ERR_MSG_MOD (extack,"TC bandwidth shares are not supported on nodes" ); -return -EOPNOTSUPP ; +struct mlx5_esw_sched_node *node = priv; +struct mlx5_eswitch *esw = node->esw; +bool disable; +int err; + +if (!esw_qos_validate_unsupported_tc_bw (esw, tc_bw)) { +NL_SET_ERR_MSG_MOD (extack,"E-Switch traffic classes number is not supported" ); +return -EOPNOTSUPP ; + } + + disable =esw_qos_tc_bw_disabled (tc_bw); +esw_qos_lock (esw); +if (disable) { + err =esw_qos_node_disable_tc_arbitration (node, extack); + goto unlock; + } + + err =esw_qos_node_enable_tc_arbitration (node, extack); +unlock: +esw_qos_unlock (esw); +return err; } int mlx5_esw_devlink_rate_node_tx_share_set(struct devlink_rate *rate_node, void *priv,
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v4 10/11 net/mlx5: Add traffic class scheduling support for vport QoS
In reply to Tariq ToukanFrom: Carolina Jubran <[email protected]>
Introduce support for traffic class (TC) scheduling on vports by allowing the vport to own multiple TC scheduling nodes. This patch enables more granular control of QoS by defining three distinct QoS states for vports, each providing unique scheduling behavior:
1. Regular QoS: The sched_node represents the vport directly, handling QoS as a single scheduling entity.
2. TC QoS on the vport: The sched_node acts as a TC arbiter, enabling TC scheduling directly on the vport.
3. TC QoS on the parent node: The sched_node functions as a rate limiter, with TC arbitration enabled at the parent level, associating multiple scheduling nodes with each vport.
Key changes include:
- Added support for new scheduling elements, vport traffic class and rate limiter.
- New helper functions for creating, destroying, and restoring vport TC scheduling nodes, handling transitions between regular QoS and TC arbitration states.
- Updated esw_qos_vport_enable() and esw_qos_vport_disable() to support both regular QoS and TC arbitration states, ensuring consistent transitions between scheduling modes.
- Introduced a sched_nodes array under vport->qos to store multiple TC scheduling nodes per vport, enabling finer control over per-TC QoS.
- Enhanced esw_qos_vport_update_parent() to handle transitions between the three QoS states based on the current and new parent node types.
This patch lays the groundwork for future support for configuring tc-bw on vports. Although the infrastructure is in place, full support for tc-bw is not yet implemented; attempts to set tc-bw on vports will return -EOPNOTSUPP.
No functional changes are introduced at this stage.
--- .../net/ethernet/mellanox/mlx5/core/esw/qos.c | 360 +++++++++++++++++- .../net/ethernet/mellanox/mlx5/core/eswitch.h | 13 +- 2 files changed, 352 insertions(+), 21 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c +341 -19
@@ -65,12 +65,16 @@ enum sched_node_type {SCHED_NODE_TYPE_VPORTS_TSAR ,SCHED_NODE_TYPE_VPORT ,SCHED_NODE_TYPE_TC_ARBITER_TSAR , +SCHED_NODE_TYPE_RATE_LIMITER , + SCHED_NODE_TYPE_VPORT_TC, };static const char *const sched_node_type_str[] = { [SCHED_NODE_TYPE_VPORTS_TSAR ] ="vports TSAR" , [SCHED_NODE_TYPE_VPORT ] ="vport" , [SCHED_NODE_TYPE_TC_ARBITER_TSAR ] ="TC Arbiter TSAR" , + [SCHED_NODE_TYPE_RATE_LIMITER ] ="Rate Limiter" , + [SCHED_NODE_TYPE_VPORT_TC ] ="vport TC" , }; struct mlx5_esw_sched_node {
@@ -92,6 +96,8 @@ struct mlx5_esw_sched_node {struct list_head children;/* Valid only if this node is associated with a vport. */ struct mlx5_vport *vport; +/* Valid only when this node represents a traffic class. */ +u8 tc; };static int esw_qos_num_tcs (struct mlx5_core_dev *dev)
@@ -121,6 +127,14 @@ esw_qos_nodes_set_parent(struct list_head *nodes, struct mlx5_esw_sched_node *pavoid mlx5_esw_qos_vport_qos_free (struct mlx5_vport *vport) { +if (vport->qos.sched_nodes) { +int i, num_tcs =esw_qos_num_tcs (vport->qos.sched_node->esw->dev); + +for (i =0 ; i < num_tcs; i++) +kfree (vport->qos.sched_nodes[i]); +kfree (vport->qos.sched_nodes); + } +kfree (vport->qos.sched_node);memset (&vport->qos,0 ,sizeof (vport->qos)); }
@@ -145,11 +159,17 @@ mlx5_esw_qos_vport_get_parent(const struct mlx5_vport *vport)static void esw_qos_sched_elem_warn (struct mlx5_esw_sched_node *node,int err,const char *op) {switch (node->type) { +case SCHED_NODE_TYPE_VPORT_TC : +esw_warn (node->esw->dev, +"E-Switch %s %s scheduling element failed (vport=%d,tc=%d,err=%d)\n" , + op, sched_node_type_str[node->type], node->vport->vport, node->tc, err); +break ;case SCHED_NODE_TYPE_VPORT :esw_warn (node->esw->dev,"E-Switch %s %s scheduling element failed (vport=%d,err=%d)\n" , op, sched_node_type_str[node->type], node->vport->vport, err);break ; +case SCHED_NODE_TYPE_RATE_LIMITER :case SCHED_NODE_TYPE_TC_ARBITER_TSAR :case SCHED_NODE_TYPE_VPORTS_TSAR : esw_warn(node->esw->dev,
@@ -243,6 +263,23 @@ static int esw_qos_sched_elem_config(struct mlx5_esw_sched_node *node, u32 max_rreturn 0 ; } +static int esw_qos_create_rate_limit_element (struct mlx5_esw_sched_node *node, +struct netlink_ext_ack *extack) +{ +u32 sched_ctx[MLX5_ST_SZ_DW (scheduling_context)] = {}; + +if (!mlx5_qos_element_type_supported (node->esw->dev, +SCHEDULING_CONTEXT_ELEMENT_TYPE_RATE_LIMIT , +SCHEDULING_HIERARCHY_E_SWITCH )) +return -EOPNOTSUPP ; + +MLX5_SET (scheduling_context, sched_ctx, max_average_bw, node->max_rate); +MLX5_SET (scheduling_context, sched_ctx, element_type, +SCHEDULING_CONTEXT_ELEMENT_TYPE_RATE_LIMIT ); + +return esw_qos_node_create_sched_element (node, sched_ctx, extack); +} +static u32 esw_qos_calculate_min_rate_divider (struct mlx5_eswitch *esw,struct mlx5_esw_sched_node *parent) {
@@ -379,6 +416,31 @@ static int esw_qos_vport_create_sched_element(struct mlx5_esw_sched_node *vport_return esw_qos_node_create_sched_element (vport_node, sched_ctx, extack); } +static int esw_qos_vport_tc_create_sched_element (struct mlx5_esw_sched_node *vport_tc_node, +u32 rate_limit_elem_ix, +struct netlink_ext_ack *extack) +{ +u32 sched_ctx[MLX5_ST_SZ_DW (scheduling_context)] = {}; +struct mlx5_core_dev *dev = vport_tc_node->esw->dev; +void *attr; + +if (!mlx5_qos_element_type_supported (dev, +SCHEDULING_CONTEXT_ELEMENT_TYPE_VPORT_TC , +SCHEDULING_HIERARCHY_E_SWITCH )) +return -EOPNOTSUPP ; + +MLX5_SET (scheduling_context, sched_ctx, element_type, +SCHEDULING_CONTEXT_ELEMENT_TYPE_VPORT_TC ); + attr =MLX5_ADDR_OF (scheduling_context, sched_ctx, element_attributes); +MLX5_SET (vport_tc_element, attr, vport_number, vport_tc_node->vport->vport); +MLX5_SET (vport_tc_element, attr, traffic_class, vport_tc_node->tc); +MLX5_SET (scheduling_context, sched_ctx, max_bw_obj_id, rate_limit_elem_ix); +MLX5_SET (scheduling_context, sched_ctx, parent_element_id, vport_tc_node->parent->ix); +MLX5_SET (scheduling_context, sched_ctx, bw_share, vport_tc_node->bw_share); + +return esw_qos_node_create_sched_element (vport_tc_node, sched_ctx, extack); +} +static struct mlx5_esw_sched_node *__esw_qos_alloc_node (struct mlx5_eswitch *esw,u32 tsar_ix,enum sched_node_type type,struct mlx5_esw_sched_node *parent)
@@ -575,12 +637,169 @@ static int esw_qos_tc_arbiter_scheduling_setup(struct mlx5_esw_sched_node *node,return -EOPNOTSUPP ; } +static int esw_qos_create_vport_tc_sched_node (struct mlx5_vport *vport, +u32 rate_limit_elem_ix, +struct mlx5_esw_sched_node *vports_tc_node, +struct netlink_ext_ack *extack) +{ +struct mlx5_esw_sched_node *vport_node = vport->qos.sched_node; +struct mlx5_esw_sched_node *vport_tc_node; +u8 tc = vports_tc_node->tc; +int err; + + vport_tc_node =__esw_qos_alloc_node (vport_node->esw,0 ,SCHED_NODE_TYPE_VPORT_TC , + vports_tc_node); +if (!vport_tc_node) +return -ENOMEM ; + + vport_tc_node->min_rate = vport_node->min_rate; + vport_tc_node->tc = tc; + vport_tc_node->vport = vport; + err =esw_qos_vport_tc_create_sched_element (vport_tc_node, rate_limit_elem_ix, extack); +if (err) + goto err_out; + + vport->qos.sched_nodes[tc] = vport_tc_node; + +return 0 ; +err_out: +__esw_qos_free_node (vport_tc_node); +return err; +} + +static void esw_qos_destroy_vport_tc_sched_elements (struct mlx5_vport *vport, +struct netlink_ext_ack *extack) +{ +int i, num_tcs =esw_qos_num_tcs (vport->qos.sched_node->esw->dev); + +for (i =0 ; i < num_tcs; i++) { +if (vport->qos.sched_nodes[i]) +__esw_qos_destroy_node (vport->qos.sched_nodes[i], extack); + } + +kfree (vport->qos.sched_nodes); + vport->qos.sched_nodes =NULL ; +} + +static int esw_qos_create_vport_tc_sched_elements (struct mlx5_vport *vport, +enum sched_node_type type, +struct netlink_ext_ack *extack) +{ +struct mlx5_esw_sched_node *vport_node = vport->qos.sched_node; +struct mlx5_esw_sched_node *tc_arbiter_node, *vports_tc_node; +int err, num_tcs =esw_qos_num_tcs (vport_node->esw->dev); +u32 rate_limit_elem_ix; + + vport->qos.sched_nodes =kcalloc (num_tcs,sizeof (struct mlx5_esw_sched_node *),GFP_KERNEL ); +if (!vport->qos.sched_nodes) { +NL_SET_ERR_MSG_MOD (extack,"Allocating the vport TC scheduling elements failed." ); +return -ENOMEM ; + } + + rate_limit_elem_ix = type ==SCHED_NODE_TYPE_RATE_LIMITER ? vport_node->ix :0 ; + tc_arbiter_node = type ==SCHED_NODE_TYPE_RATE_LIMITER ? vport_node->parent : vport_node; +list_for_each_entry (vports_tc_node, &tc_arbiter_node->children, entry) { + err =esw_qos_create_vport_tc_sched_node (vport, rate_limit_elem_ix, vports_tc_node, + extack); +if (err) + goto err_create_vport_tc; + } + +return 0 ; + +err_create_vport_tc: +esw_qos_destroy_vport_tc_sched_elements (vport,NULL ); + +return err; +} + +static int esw_qos_vport_tc_enable (struct mlx5_vport *vport,enum sched_node_type type, +struct netlink_ext_ack *extack) +{ +struct mlx5_esw_sched_node *vport_node = vport->qos.sched_node; +int err; + +if (type ==SCHED_NODE_TYPE_TC_ARBITER_TSAR && +MLX5_CAP_QOS (vport_node->esw->dev, log_esw_max_sched_depth) <2 ) { +NL_SET_ERR_MSG_MOD (extack,"Setting up TC Arbiter for a vport is not supported." ); +return -EOPNOTSUPP ; + } + +esw_assert_qos_lock_held (vport->dev->priv.eswitch); + +if (type ==SCHED_NODE_TYPE_RATE_LIMITER ) + err =esw_qos_create_rate_limit_element (vport_node, extack); +else + err =esw_qos_tc_arbiter_scheduling_setup (vport_node, extack); +if (err) +return err; + +/* Rate limiters impact multiple nodes not directly connected to them +* and are not direct members of the QoS hierarchy. +* Unlink it from the parent to reflect that. +*/ +if (type ==SCHED_NODE_TYPE_RATE_LIMITER ) +list_del_init (&vport_node->entry); + + err =esw_qos_create_vport_tc_sched_elements (vport, type, extack); +if (err) + goto err_sched_nodes; + +return 0 ; + +err_sched_nodes: +if (type ==SCHED_NODE_TYPE_RATE_LIMITER ) { +esw_qos_node_destroy_sched_element (vport_node,NULL ); +list_add_tail (&vport_node->entry, &vport_node->parent->children); + }else { +esw_qos_tc_arbiter_scheduling_teardown (vport_node,NULL ); + } +return err; +} + +static void esw_qos_vport_tc_disable (struct mlx5_vport *vport,struct netlink_ext_ack *extack) +{ +struct mlx5_esw_sched_node *vport_node = vport->qos.sched_node; +enum sched_node_type curr_type = vport_node->type; + +esw_qos_destroy_vport_tc_sched_elements (vport, extack); + +if (curr_type ==SCHED_NODE_TYPE_RATE_LIMITER ) +esw_qos_node_destroy_sched_element (vport_node, extack); +else +esw_qos_tc_arbiter_scheduling_teardown (vport_node, extack); +} + +static int esw_qos_set_vport_tcs_min_rate (struct mlx5_vport *vport,u32 min_rate, +struct netlink_ext_ack *extack) +{ +struct mlx5_esw_sched_node *vport_node = vport->qos.sched_node; +int err, i, num_tcs =esw_qos_num_tcs (vport_node->esw->dev); + +for (i =0 ; i < num_tcs; i++) { + err =esw_qos_set_node_min_rate (vport->qos.sched_nodes[i], min_rate, extack); +if (err) + goto err_out; + } + vport_node->min_rate = min_rate; + +return 0 ; +err_out: +for (--i; i >=0 ; i--) +esw_qos_set_node_min_rate (vport->qos.sched_nodes[i], vport_node->min_rate, extack); +return err; +} +static void esw_qos_vport_disable (struct mlx5_vport *vport,struct netlink_ext_ack *extack) {struct mlx5_esw_sched_node *vport_node = vport->qos.sched_node;struct mlx5_esw_sched_node *parent = vport_node->parent; +enum sched_node_type curr_type = vport_node->type; -esw_qos_node_destroy_sched_element (vport_node, extack); +if (curr_type ==SCHED_NODE_TYPE_VPORT ) +esw_qos_node_destroy_sched_element (vport_node, extack); +else +esw_qos_vport_tc_disable (vport, extack); vport_node->bw_share =0 ;list_del_init (&vport_node->entry);
@@ -589,7 +808,8 @@ static void esw_qos_vport_disable(struct mlx5_vport *vport, struct netlink_ext_atrace_mlx5_esw_vport_qos_destroy (vport_node->esw->dev, vport); } -static int esw_qos_vport_enable (struct mlx5_vport *vport,struct mlx5_esw_sched_node *parent, +static int esw_qos_vport_enable (struct mlx5_vport *vport,enum sched_node_type type, +struct mlx5_esw_sched_node *parent,struct netlink_ext_ack *extack) {int err;
@@ -597,10 +817,14 @@ static int esw_qos_vport_enable(struct mlx5_vport *vport, struct mlx5_esw_sched_esw_assert_qos_lock_held (vport->dev->priv.eswitch);esw_qos_node_set_parent (vport->qos.sched_node, parent); - err =esw_qos_vport_create_sched_element (vport->qos.sched_node, extack); +if (type ==SCHED_NODE_TYPE_VPORT ) + err =esw_qos_vport_create_sched_element (vport->qos.sched_node, extack); +else + err =esw_qos_vport_tc_enable (vport, type, extack);if (err)return err; + vport->qos.sched_node->type = type;esw_qos_normalize_min_rate (parent->esw, parent, extack);return 0 ;
@@ -628,7 +852,7 @@ static int mlx5_esw_qos_vport_enable(struct mlx5_vport *vport, enum sched_node_t sched_node->min_rate = min_rate; sched_node->vport = vport; vport->qos.sched_node = sched_node; - err =esw_qos_vport_enable (vport, parent, extack); + err =esw_qos_vport_enable (vport, type, parent, extack);if (err)esw_qos_put (esw);
@@ -680,6 +904,8 @@ static int mlx5_esw_qos_set_vport_min_rate(struct mlx5_vport *vport, u32 min_ratif (!vport_node)return mlx5_esw_qos_vport_enable (vport,SCHED_NODE_TYPE_VPORT ,NULL ,0 , min_rate, extack); +else if (vport_node->type ==SCHED_NODE_TYPE_RATE_LIMITER ) +return esw_qos_set_vport_tcs_min_rate (vport, min_rate, extack);else return esw_qos_set_node_min_rate (vport_node, min_rate, extack); }
@@ -712,12 +938,59 @@ bool mlx5_esw_qos_get_vport_rate(struct mlx5_vport *vport, u32 *max_rate, u32 *mreturn enabled; } +static int esw_qos_vport_tc_check_type (enum sched_node_type curr_type, +enum sched_node_type new_type, +struct netlink_ext_ack *extack) +{ +if (curr_type ==SCHED_NODE_TYPE_TC_ARBITER_TSAR && + new_type ==SCHED_NODE_TYPE_RATE_LIMITER ) { +NL_SET_ERR_MSG_MOD (extack, +"Cannot switch from vport-level TC arbitration to node-level TC arbitration" ); +return -EOPNOTSUPP ; + } + +if (curr_type ==SCHED_NODE_TYPE_RATE_LIMITER && + new_type ==SCHED_NODE_TYPE_TC_ARBITER_TSAR ) { +NL_SET_ERR_MSG_MOD (extack, +"Cannot switch from node-level TC arbitration to vport-level TC arbitration" ); +return -EOPNOTSUPP ; + } + +return 0 ; +} + +static int esw_qos_vport_update (struct mlx5_vport *vport,enum sched_node_type type, +struct mlx5_esw_sched_node *parent, +struct netlink_ext_ack *extack) +{ +struct mlx5_esw_sched_node *curr_parent = vport->qos.sched_node->parent; +enum sched_node_type curr_type = vport->qos.sched_node->type; +int err; + +esw_assert_qos_lock_held (vport->dev->priv.eswitch); + parent = parent ?: curr_parent; +if (curr_type == type && curr_parent == parent) +return 0 ; + + err =esw_qos_vport_tc_check_type (curr_type, type, extack); +if (err) +return err; + +esw_qos_vport_disable (vport, extack); + + err =esw_qos_vport_enable (vport, type, parent, extack); +if (err) +esw_qos_vport_enable (vport, curr_type, curr_parent,NULL ); + +return err; +} +static int esw_qos_vport_update_parent (struct mlx5_vport *vport,struct mlx5_esw_sched_node *parent,struct netlink_ext_ack *extack) {struct mlx5_eswitch *esw = vport->dev->priv.eswitch;struct mlx5_esw_sched_node *curr_parent; -int err; +enum sched_node_type type;esw_assert_qos_lock_held (esw); curr_parent = vport->qos.sched_node->parent;
@@ -725,16 +998,17 @@ static int esw_qos_vport_update_parent(struct mlx5_vport *vport, struct mlx5_eswif (curr_parent == parent)return 0 ; -esw_qos_vport_disable (vport, extack); - - err =esw_qos_vport_enable (vport, parent, extack); -if (err) { -if (esw_qos_vport_enable (vport, curr_parent,NULL )) -esw_warn (parent->esw->dev,"vport restore QoS failed (vport=%d)\n" , - vport->vport); - } +/* Set vport QoS type based on parent node type if different from default QoS; +* otherwise, use the vport's current QoS type. +*/ +if (parent->type ==SCHED_NODE_TYPE_TC_ARBITER_TSAR ) + type =SCHED_NODE_TYPE_RATE_LIMITER ; +else if (curr_parent->type ==SCHED_NODE_TYPE_TC_ARBITER_TSAR ) + type =SCHED_NODE_TYPE_VPORT ; +else + type = vport->qos.sched_node->type; -return err; +return esw_qos_vport_update (vport, type, parent, extack); } static void esw_qos_switch_vport_tcs_to_vport(struct mlx5_esw_sched_node *tc_arbiter_node,
@@ -1025,6 +1299,14 @@ static bool esw_qos_validate_unsupported_tc_bw(struct mlx5_eswitch *esw, u32 *tcreturn true; } +static bool esw_qos_vport_validate_unsupported_tc_bw (struct mlx5_vport *vport,u32 *tc_bw) +{ +struct mlx5_eswitch *esw = vport->qos.sched_node ? + vport->qos.sched_node->parent->esw : vport->dev->priv.eswitch; + +return esw_qos_validate_unsupported_tc_bw (esw, tc_bw); +} +static bool esw_qos_tc_bw_disabled (u32 *tc_bw) {int i;
@@ -1098,8 +1380,44 @@ int mlx5_esw_devlink_rate_leaf_tx_max_set(struct devlink_rate *rate_leaf, void *int mlx5_esw_devlink_rate_leaf_tc_bw_set (struct devlink_rate *rate_leaf,void *priv,u32 *tc_bw,struct netlink_ext_ack *extack) { -NL_SET_ERR_MSG_MOD (extack,"TC bandwidth shares are not supported on leafs" ); -return -EOPNOTSUPP ; +struct mlx5_esw_sched_node *vport_node; +struct mlx5_vport *vport = priv; +struct mlx5_eswitch *esw; +bool disable; +int err =0 ; + + esw = vport->dev->priv.eswitch; +if (!mlx5_esw_allowed (esw)) +return -EPERM ; + + disable =esw_qos_tc_bw_disabled (tc_bw); +esw_qos_lock (esw); + +if (!esw_qos_vport_validate_unsupported_tc_bw (vport, tc_bw)) { +NL_SET_ERR_MSG_MOD (extack,"E-Switch traffic classes number is not supported" ); + err = -EOPNOTSUPP ; + goto unlock; + } + + vport_node = vport->qos.sched_node; +if (disable && !vport_node) + goto unlock; + +if (disable && vport_node->type ==SCHED_NODE_TYPE_TC_ARBITER_TSAR ) { + err =esw_qos_vport_update (vport,SCHED_NODE_TYPE_VPORT ,NULL , extack); + goto unlock; + } + +if (!vport_node) { + err =mlx5_esw_qos_vport_enable (vport,SCHED_NODE_TYPE_TC_ARBITER_TSAR ,NULL ,0 ,0 , + extack); + vport_node = vport->qos.sched_node; + }else { + err =esw_qos_vport_update (vport,SCHED_NODE_TYPE_TC_ARBITER_TSAR ,NULL , extack); + } +unlock: +esw_qos_unlock (esw); +return err; } int mlx5_esw_devlink_rate_node_tc_bw_set(struct devlink_rate *rate_node, void *priv,
@@ -1218,10 +1536,14 @@ int mlx5_esw_qos_vport_update_parent(struct mlx5_vport *vport, struct mlx5_esw_s }esw_qos_lock (esw); -if (!vport->qos.sched_node && parent) - err =mlx5_esw_qos_vport_enable (vport,SCHED_NODE_TYPE_VPORT , parent,0 ,0 , extack); -else if (vport->qos.sched_node) +if (!vport->qos.sched_node && parent) { +enum sched_node_type type = parent->type ==SCHED_NODE_TYPE_TC_ARBITER_TSAR ? +SCHED_NODE_TYPE_RATE_LIMITER :SCHED_NODE_TYPE_VPORT ; + + err =mlx5_esw_qos_vport_enable (vport, type, parent,0 ,0 , extack); + }else if (vport->qos.sched_node) { err =esw_qos_vport_update_parent (vport, parent, extack); + }esw_qos_unlock (esw);return err; }
drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +11 -2
@@ -212,10 +212,19 @@ struct mlx5_vport {struct mlx5_vport_info info; -/* Protected with the E-Switch qos domain lock. */ +/* Protected with the E-Switch qos domain lock. The Vport QoS can +* either be disabled (sched_node is NULL) or in one of three states: +* 1. Regular QoS (sched_node is a vport node). +* 2. TC QoS enabled on the vport (sched_node is a TC arbiter). +* 3. TC QoS enabled on the vport's parent node +* (sched_node is a rate limit node). +* When TC is enabled in either mode, the vport owns vport TC scheduling nodes. +*/ struct { -/* Vport scheduling element node. */ +/* Vport scheduling node. */ struct mlx5_esw_sched_node *sched_node; +/* Array of vport traffic class scheduling nodes. */ +struct mlx5_esw_sched_node **sched_nodes; } qos;u16 vport;
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v4 11/11 net/mlx5: Manage TC arbiter nodes and implement full support for tc-bw
In reply to Tariq ToukanFrom: Carolina Jubran <[email protected]>
Introduce support for managing Traffic Class (TC) arbiter nodes and associated vports TC nodes within the E-Switch QoS hierarchy. This patch adds support for the new scheduling node type, SCHED_NODE_TYPE_VPORTS_TC_TSAR, and implements full support for setting tc-bw on both vports and nodes.
Key changes include:
- Introduced the new scheduling node type, SCHED_NODE_TYPE_VPORTS_TC_TSAR, for managing vports within the TC arbiter node.
- New helper functions for creating and destroying vports TC nodes under the TC arbiter.
- Updated the minimum rate normalization function to skip nodes of type SCHED_NODE_TYPE_VPORTS_TC_TSAR. Vports TC TSARs have bandwidth shares configured on them but not minimum rates, so their min_rate cannot be normalized.
- Implementation of esw_qos_tc_arbiter_scheduling_setup() and esw_qos_tc_arbiter_scheduling_teardown() for initializing and cleaning up TC arbiter scheduling elements. These functions now fully support tc-bw configuration on TC arbiter nodes.
- Added esw_qos_tc_arbiter_get_bw_shares() and esw_qos_set_tc_arbiter_bw_shares() to handle the settings of bandwidth shares for vports traffic class TSARs.
- Refactored mlx5_esw_devlink_rate_node_tc_bw_set() and mlx5_esw_devlink_rate_leaf_tc_bw_set() to fully support configuring tc-bw on devlink rate nodes and vports, respectively.
--- .../net/ethernet/mellanox/mlx5/core/esw/qos.c | 185 +++++++++++++++++- 1 file changed, 180 insertions(+), 5 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c +180 -5
@@ -67,6 +67,7 @@ enum sched_node_type {SCHED_NODE_TYPE_TC_ARBITER_TSAR ,SCHED_NODE_TYPE_RATE_LIMITER ,SCHED_NODE_TYPE_VPORT_TC , + SCHED_NODE_TYPE_VPORTS_TC_TSAR, }; static const char * const sched_node_type_str[] = {
@@ -75,6 +76,7 @@ static const char * const sched_node_type_str[] = { [SCHED_NODE_TYPE_TC_ARBITER_TSAR] = "TC Arbiter TSAR", [SCHED_NODE_TYPE_RATE_LIMITER] = "Rate Limiter", [SCHED_NODE_TYPE_VPORT_TC] = "vport TC", + [SCHED_NODE_TYPE_VPORTS_TC_TSAR] = "vports TC TSAR", };struct mlx5_esw_sched_node {
@@ -159,6 +161,11 @@ mlx5_esw_qos_vport_get_parent(const struct mlx5_vport *vport)static void esw_qos_sched_elem_warn (struct mlx5_esw_sched_node *node,int err,const char *op) {switch (node->type) { +case SCHED_NODE_TYPE_VPORTS_TC_TSAR : +esw_warn (node->esw->dev, +"E-Switch %s %s scheduling element failed (tc=%d,err=%d)\n" , + op, sched_node_type_str[node->type], node->tc, err); +break ;case SCHED_NODE_TYPE_VPORT_TC : esw_warn(node->esw->dev, "E-Switch %s %s scheduling element failed (vport=%d,tc=%d,err=%d)\n",
@@ -344,7 +351,11 @@ static void esw_qos_normalize_min_rate(struct mlx5_eswitch *esw,if (node->esw != esw || node->ix == esw->qos.root_tsar_ix)continue ; -esw_qos_update_sched_node_bw_share (node, divider, extack); +/* Vports TC TSARs don't have a minimum rate configured, +* so there's no need to update the bw_share on them. +*/ +if (node->type !=SCHED_NODE_TYPE_VPORTS_TC_TSAR ) +esw_qos_update_sched_node_bw_share (node, divider, extack);if (list_empty (&node->children))continue ;
@@ -476,6 +487,129 @@ static void esw_qos_destroy_node(struct mlx5_esw_sched_node *node, struct netlin__esw_qos_free_node (node); } +static int esw_qos_create_vports_tc_node (struct mlx5_esw_sched_node *parent,u8 tc, +struct netlink_ext_ack *extack) +{ +u32 tsar_ctx[MLX5_ST_SZ_DW (scheduling_context)] = {}; +struct mlx5_core_dev *dev = parent->esw->dev; +struct mlx5_esw_sched_node *vports_tc_node; +void *attr; +int err; + +if (!mlx5_qos_element_type_supported (dev, +SCHEDULING_CONTEXT_ELEMENT_TYPE_TSAR , +SCHEDULING_HIERARCHY_E_SWITCH ) || + !mlx5_qos_tsar_type_supported (dev, +TSAR_ELEMENT_TSAR_TYPE_DWRR , +SCHEDULING_HIERARCHY_E_SWITCH )) +return -EOPNOTSUPP ; + + vports_tc_node =__esw_qos_alloc_node (parent->esw,0 ,SCHED_NODE_TYPE_VPORTS_TC_TSAR , + parent); +if (!vports_tc_node) { +NL_SET_ERR_MSG_MOD (extack,"E-Switch alloc node failed" ); +esw_warn (dev,"Failed to alloc vports TC node (tc=%d)\n" , tc); +return -ENOMEM ; + } + + attr =MLX5_ADDR_OF (scheduling_context, tsar_ctx, element_attributes); +MLX5_SET (tsar_element, attr, tsar_type,TSAR_ELEMENT_TSAR_TYPE_DWRR ); +MLX5_SET (tsar_element, attr, traffic_class, tc); +MLX5_SET (scheduling_context, tsar_ctx, parent_element_id, parent->ix); +MLX5_SET (scheduling_context, tsar_ctx, element_type,SCHEDULING_CONTEXT_ELEMENT_TYPE_TSAR ); + + err =esw_qos_node_create_sched_element (vports_tc_node, tsar_ctx, extack); +if (err) + goto err_create_sched_element; + + vports_tc_node->tc = tc; + +return 0 ; + +err_create_sched_element: +__esw_qos_free_node (vports_tc_node); +return err; +} + +static void +esw_qos_tc_arbiter_get_bw_shares (struct mlx5_esw_sched_node *tc_arbiter_node,u32 *tc_bw) +{ +struct mlx5_esw_sched_node *vports_tc_node; + +list_for_each_entry (vports_tc_node, &tc_arbiter_node->children, entry) + tc_bw[vports_tc_node->tc] = vports_tc_node->bw_share; +} + +static void esw_qos_set_tc_arbiter_bw_shares (struct mlx5_esw_sched_node *tc_arbiter_node, +u32 *tc_bw,struct netlink_ext_ack *extack) +{ +struct mlx5_esw_sched_node *vports_tc_node; + +list_for_each_entry (vports_tc_node, &tc_arbiter_node->children, entry) { +u32 bw_share; +u8 tc; + + tc = vports_tc_node->tc; + bw_share = tc_bw[tc] ?:MLX5_MIN_BW_SHARE ; +esw_qos_sched_elem_config (vports_tc_node,0 , bw_share, extack); + } +} + +static void esw_qos_destroy_vports_tc_nodes (struct mlx5_esw_sched_node *tc_arbiter_node, +struct netlink_ext_ack *extack) +{ +struct mlx5_esw_sched_node *vports_tc_node, *tmp; + +list_for_each_entry_safe (vports_tc_node, tmp, &tc_arbiter_node->children, entry) +esw_qos_destroy_node (vports_tc_node, extack); +} + +static int esw_qos_create_vports_tc_nodes (struct mlx5_esw_sched_node *tc_arbiter_node, +struct netlink_ext_ack *extack) +{ +struct mlx5_eswitch *esw = tc_arbiter_node->esw; +int err, i, num_tcs =esw_qos_num_tcs (esw->dev); + +for (i =0 ; i < num_tcs; i++) { + err =esw_qos_create_vports_tc_node (tc_arbiter_node, i, extack); +if (err) + goto err_tc_node_create; + } + +return 0 ; + +err_tc_node_create: +esw_qos_destroy_vports_tc_nodes (tc_arbiter_node,NULL ); +return err; +} + +static int esw_qos_create_tc_arbiter_sched_elem (struct mlx5_esw_sched_node *tc_arbiter_node, +struct netlink_ext_ack *extack) +{ +u32 tsar_ctx[MLX5_ST_SZ_DW (scheduling_context)] = {}; +u32 tsar_parent_ix; +void *attr; + +if (!mlx5_qos_tsar_type_supported (tc_arbiter_node->esw->dev, +TSAR_ELEMENT_TSAR_TYPE_TC_ARB , +SCHEDULING_HIERARCHY_E_SWITCH )) { +NL_SET_ERR_MSG_MOD (extack, +"E-Switch TC Arbiter scheduling element is not supported" ); +return -EOPNOTSUPP ; + } + + attr =MLX5_ADDR_OF (scheduling_context, tsar_ctx, element_attributes); +MLX5_SET (tsar_element, attr, tsar_type,TSAR_ELEMENT_TSAR_TYPE_TC_ARB ); + tsar_parent_ix = tc_arbiter_node->parent ? tc_arbiter_node->parent->ix : + tc_arbiter_node->esw->qos.root_tsar_ix; +MLX5_SET (scheduling_context, tsar_ctx, parent_element_id, tsar_parent_ix); +MLX5_SET (scheduling_context, tsar_ctx, element_type,SCHEDULING_CONTEXT_ELEMENT_TYPE_TSAR ); +MLX5_SET (scheduling_context, tsar_ctx, max_average_bw, tc_arbiter_node->max_rate); +MLX5_SET (scheduling_context, tsar_ctx, bw_share, tc_arbiter_node->bw_share); + +return esw_qos_node_create_sched_element (tc_arbiter_node, tsar_ctx, extack); +} +static struct mlx5_esw_sched_node *__esw_qos_create_vports_sched_node (struct mlx5_eswitch *esw,struct mlx5_esw_sched_node *parent,struct netlink_ext_ack *extack)
@@ -539,6 +673,9 @@ static void __esw_qos_destroy_node(struct mlx5_esw_sched_node *node, struct netl {struct mlx5_eswitch *esw = node->esw; +if (node->type ==SCHED_NODE_TYPE_TC_ARBITER_TSAR ) +esw_qos_destroy_vports_tc_nodes (node, extack); +trace_mlx5_esw_node_qos_destroy (esw->dev, node, node->ix);esw_qos_destroy_node (node, extack);esw_qos_normalize_min_rate (esw,NULL , extack);
@@ -628,13 +765,38 @@ static void esw_qos_put(struct mlx5_eswitch *esw)static void esw_qos_tc_arbiter_scheduling_teardown (struct mlx5_esw_sched_node *node,struct netlink_ext_ack *extack) -{} +{ +/* Clean up all Vports TC nodes within the TC arbiter node. */ +esw_qos_destroy_vports_tc_nodes (node, extack); +/* Destroy the scheduling element for the TC arbiter node itself. */ +esw_qos_node_destroy_sched_element (node, extack); +}static int esw_qos_tc_arbiter_scheduling_setup (struct mlx5_esw_sched_node *node,struct netlink_ext_ack *extack) { -NL_SET_ERR_MSG_MOD (extack,"TC arbiter elements are not supported." ); -return -EOPNOTSUPP ; +u32 curr_ix = node->ix; +int err; + + err =esw_qos_create_tc_arbiter_sched_elem (node, extack); +if (err) +return err; +/* Initialize the vports TC nodes within created TC arbiter TSAR. */ + err =esw_qos_create_vports_tc_nodes (node, extack); +if (err) + goto err_vports_tc_nodes; + + node->type =SCHED_NODE_TYPE_TC_ARBITER_TSAR ; + +return 0 ; + +err_vports_tc_nodes: +/* If initialization fails, clean up the scheduling element +* for the TC arbiter node. +*/ +esw_qos_node_destroy_sched_element (node,NULL ); + node->ix = curr_ix; +return err; } static int esw_qos_create_vport_tc_sched_node(struct mlx5_vport *vport,
@@ -965,6 +1127,7 @@ static int esw_qos_vport_update(struct mlx5_vport *vport, enum sched_node_type t {struct mlx5_esw_sched_node *curr_parent = vport->qos.sched_node->parent;enum sched_node_type curr_type = vport->qos.sched_node->type; +u32 curr_tc_bw[IEEE_8021QAZ_MAX_TCS ] = {0 };int err;esw_assert_qos_lock_held (vport->dev->priv.eswitch);
@@ -976,11 +1139,19 @@ static int esw_qos_vport_update(struct mlx5_vport *vport, enum sched_node_type tif (err)return err; +if (curr_type ==SCHED_NODE_TYPE_TC_ARBITER_TSAR && curr_type == type) +esw_qos_tc_arbiter_get_bw_shares (vport->qos.sched_node, curr_tc_bw); +esw_qos_vport_disable (vport, extack); err =esw_qos_vport_enable (vport, type, parent, extack); -if (err) +if (err) {esw_qos_vport_enable (vport, curr_type, curr_parent,NULL ); + extack =NULL ; + } + +if (curr_type ==SCHED_NODE_TYPE_TC_ARBITER_TSAR && curr_type == type) +esw_qos_set_tc_arbiter_bw_shares (vport->qos.sched_node, curr_tc_bw, extack);return err; }
@@ -1415,6 +1586,8 @@ int mlx5_esw_devlink_rate_leaf_tc_bw_set(struct devlink_rate *rate_leaf, void *p }else { err =esw_qos_vport_update (vport,SCHED_NODE_TYPE_TC_ARBITER_TSAR ,NULL , extack); } +if (!err) +esw_qos_set_tc_arbiter_bw_shares (vport_node, tc_bw, extack); unlock:esw_qos_unlock (esw);return err;
@@ -1441,6 +1614,8 @@ int mlx5_esw_devlink_rate_node_tc_bw_set(struct devlink_rate *rate_node, void *p } err =esw_qos_node_enable_tc_arbitration (node, extack); +if (!err) +esw_qos_set_tc_arbiter_bw_shares (node, tc_bw, extack); unlock:esw_qos_unlock (esw);return err;
Signature · 1 line
2.44.0
Quoted 4 lines · Tariq Toukan
Hi,
This patchset starts with 3 patches that modify the IFC, targeted to mlx5-next
First 4 patches to mlx5-next, not 3.
Sorry for the confusion.
Quoted 3 lines
in order to be taken to rdma-next branch side sooner than in the next merge window.
Quoted 6 lines · Tariq Toukan
Hi,
This patchset starts with 3 patches that modify the IFC, targeted to mlx5-next in order to be taken to rdma-next branch side sooner than in the next merge window.
Quoted 7 lines
This patchset consists of two features:
1. In patches 4-5, Itamar adds SW Steering support for ConnectX-8.
2. Followed by patches by Carolina that add rate management support on traffic classes in devlink and mlx5, more details below [1].Series generated against:
commite8e7be7d212d("mctp i2c: drop check becausei2c_unregister_device()is NULL safe")
From the link https://people.kernel.org/monsieuricon/all-patches-must-include-base-commit-info,
If we use --base=auto or the commit id (in this patch, the commit id should be e8e7be7d212d), then we will notice that the commits will have the base-commit: tailer at the very bottom.
This seems somewhat professional compared to the above. ^_^
Best Regards,
Zhu Yanjun
Quoted 115 lines
Regards,
TariqV4:
- Renamed the nested attribute for traffic class bandwidth toDEVLINK_ATTR_RATE_TC_BWS.
- Changed the order of the attributes indevlink.h.
- Refactored the initialization tc-bw array indevlink_nl_rate_tc_bw_set().
- Added extack messages to provide clear feedback on issues with tc-bw arguments.
- Updatedrate-tc-bwsto support a multi-attr set, where each attribute includes an index and the corresponding bandwidth for that traffic class.
- Handled the issue where the user could provideDEVLINK_ATTR_RATE_TC_BWSwith duplicate indices.
- Provided ynl exmaples in devlink patch commit message.
- Take IFC patches to beginning of the series, targeted for mlx5-next.V3:
- Dropped rate-tc-index, using tc-bw array index instead.
- Renamed rate-bw to rate-tc-bw.
- Documneted what the rate-tc-bw represents and added a range check for validation.
- Intorduceddevlink_nl_rate_tc_bw_set()to parse and set the TC bandwidth values.
- Updated the user API in the commit message of patch 1/6 to ensure bandwidths sum equals 100.
- Fixed missing filling of rate-parent indevlink_nl_rate_fill().V2:
- Included <linux/dcbnl.h> indevlink.hto resolve missingIEEE_8021QAZ_MAX_TCSdefinition.
- Refactored the rate-tc-bw attribute structure to use a separate rate-tc-index.
- Updated patch 2/6 title.[1]
This patch series extends the devlink-rate API to support traffic class (TC) bandwidth management, enabling more granular control over traffic shaping and rate limiting across multiple TCs. The API now allows users to specify bandwidth proportions for different traffic classes in a single command. This is particularly useful for managing Enhanced Transmission Selection (ETS) for groups of Virtual Functions (VFs), allowing precise bandwidth allocation across traffic classes.Additionally the series refines the QoS handling in net/mlx5 to support TC arbitration and bandwidth management on vports and rate nodes.
Extend devlink-rate API to support rate management on TCs:
- devlink: Extend the devlink rate API to support traffic class bandwidth managementIntroduce a no-op implementation:
- net/mlx5: Add no-op implementation for setting tc-bw on rate objectsAdd support for enabling and disabling TC QoS on vports and nodes:
- net/mlx5: Add support for setting tc-bw on nodes
- net/mlx5: Add traffic class scheduling support for vport QoSSupport for setting tc-bw on rate objects:
- net/mlx5: Manage TC arbiter nodes and implement full support for tc-bwCarolina Jubran (6):
net/mlx5: Add support for new scheduling elements
devlink: Extend devlink rate API with traffic classes bandwidth
management
net/mlx5: Add no-op implementation for setting tc-bw on rate objects net/mlx5: Add support for setting tc-bw on nodes
net/mlx5: Add traffic class scheduling support for vport QoS net/mlx5: Manage TC arbiter nodes and implement full support for tc-bwCosmin Ratiu (2):
net/mlx5: ifc: Reorganizemlx5_ifc_flow_table_context_bitsnet/mlx5: qos: Add ifc support for cross-esw schedulingItamar Gozlan (2):
net/mlx5: DR, Expand SWS STE callbacks and consolidate common structs net/mlx5: DR, Add support for ConnectX-8 steeringYevgeny Kliteynik (1):
net/mlx5: Add ConnectX-8 device to ifcDocumentation/netlink/specs/devlink.yaml | 28 +- .../net/ethernet/mellanox/mlx5/core/Makefile | 1 + .../net/ethernet/mellanox/mlx5/core/devlink.c | 2 + .../net/ethernet/mellanox/mlx5/core/esw/qos.c | 795 +++++++++++++++++- .../net/ethernet/mellanox/mlx5/core/esw/qos.h | 4 + .../net/ethernet/mellanox/mlx5/core/eswitch.h | 13 +- drivers/net/ethernet/mellanox/mlx5/core/rl.c | 4 + .../mlx5/core/steering/sws/dr_domain.c | 2 +- .../mellanox/mlx5/core/steering/sws/dr_ste.c | 6 +- .../mellanox/mlx5/core/steering/sws/dr_ste.h | 19 +- .../mlx5/core/steering/sws/dr_ste_v0.c | 6 +- .../mlx5/core/steering/sws/dr_ste_v1.c | 207 +---- .../mlx5/core/steering/sws/dr_ste_v1.h | 147 +++- .../mlx5/core/steering/sws/dr_ste_v2.c | 169 +--- .../mlx5/core/steering/sws/dr_ste_v2.h | 168 ++++ .../mlx5/core/steering/sws/dr_ste_v3.c | 221 +++++ .../mlx5/core/steering/sws/mlx5_ifc_dr.h | 40 + .../mellanox/mlx5/core/steering/sws/mlx5dr.h | 2 +- include/linux/mlx5/mlx5_ifc.h | 56 +- include/net/devlink.h | 7 + include/uapi/linux/devlink.h | 4 + net/devlink/netlink_gen.c | 15 +- net/devlink/netlink_gen.h | 1 + net/devlink/rate.c | 124 +++ 24 files changed, 1645 insertions(+), 396 deletions(-)create mode 100644
drivers/net/ethernet/mellanox/mlx5/core/steering/sws/dr_ste_v2.hcreate mode 100644drivers/net/ethernet/mellanox/mlx5/core/steering/sws/dr_ste_v3.c
Quoted 18 lines · Zhu Yanjun
Quoted 6 lines · Tariq Toukan
Hi,
This patchset starts with 3 patches that modify the IFC, targeted to mlx5-next in order to be taken to rdma-next branch side sooner than in the next merge window.
Quoted 7 lines
This patchset consists of two features:
1. In patches 4-5, Itamar adds SW Steering support for ConnectX-8.
2. Followed by patches by Carolina that add rate management support on traffic classes in devlink and mlx5, more details below [1].Series generated against:
commite8e7be7d212d("mctp i2c: drop check becausei2c_unregister_device()is NULL safe")From the link https://people.kernel.org/monsieuricon/all-patches-must-
include-base-commit-info,
Quoted 7 lines
If we use --base=auto or the commit id (in this patch, the commit id should be
e8e7be7d212d), then we will notice that the commits will have the base-commit: tailer at the very bottom.This seems somewhat professional compared to the above. ^_^
Best Regards,
Zhu Yanjun
Thanks for your comment!
Noted. I'll use it in my next submissions.