Support resource sharing among ibv_devices
cover Support resource sharing among ibv_devices
Add the option to use resources (Tables, Matchers, Actions, etc.) from one gvmi (AKA ibv_cntext) to other gvmi's.
When specific gvmi allows other to use its resources, steering objects Will created and used on that gvmi.
It is done by aliases objects that map between local resources to shared/remote resourses.
That in order to allow sharing between few ports.
Erez Shitrit (6):
mailmap: add new contributors to the list net/mlx5/hws: add capabilities fields for vhca access net/mlx5/hws: add vhca identifier ID to the caps net/mlx5/hws: support shared ibv-context with local one net/mlx5/hws: support actions while shared resources is used net/mlx5/hws: add debug details for cross gvmi
Yevgeny Kliteynik (5):
net/mlx5/hws: remove wrong PRM capability macros net/mlx5/hws: add PRM definitions for cross-vhca capabilities net/mlx5/hws: read cross-vhca capabilities net/mlx5/hws: added allow-other-vhca-access command net/mlx5/hws: added command to create alias objects
.mailmap | 1 + drivers/common/mlx5/mlx5_prm.h | 83 ++++++++-- drivers/net/mlx5/hws/mlx5dr.h | 7 +- drivers/net/mlx5/hws/mlx5dr_action.c | 69 +++++++- drivers/net/mlx5/hws/mlx5dr_action.h | 3 + drivers/net/mlx5/hws/mlx5dr_cmd.c | 118 ++++++++++++- drivers/net/mlx5/hws/mlx5dr_cmd.h | 25 +++ drivers/net/mlx5/hws/mlx5dr_context.c | 34 +++- drivers/net/mlx5/hws/mlx5dr_context.h | 22 +++ drivers/net/mlx5/hws/mlx5dr_debug.c | 24 ++- drivers/net/mlx5/hws/mlx5dr_internal.h | 1 + drivers/net/mlx5/hws/mlx5dr_matcher.c | 218 +++++++++++++++++++++++-- drivers/net/mlx5/hws/mlx5dr_matcher.h | 9 + drivers/net/mlx5/hws/mlx5dr_table.c | 191 +++++++++++++++++++++- drivers/net/mlx5/hws/mlx5dr_table.h | 4 +- drivers/net/mlx5/mlx5_devx.c | 2 +- 16 files changed, 750 insertions(+), 61 deletions(-)
Signature · 1 line
2.18.2
PATCH 1/11 mailmap: add new contributors to the list
Added Yevgeny Kliteynik to the list
--- .mailmap | 1 + 1 file changed, 1 insertion(+)
.mailmap +1 -0
@@ -1495,6 +1495,7 @@ Yash Sharma <[email protected]> Yasufumi Ogawa <[email protected]> <[email protected]> Yelena Krivosheev <[email protected]> Yerden Zhumabekov <[email protected]> <[email protected]> +Yevgeny Kliteynik <[email protected]> Yicai Lu <[email protected]> Yiding Zhou <[email protected]> Yi Li <[email protected]>
Signature · 1 line
2.18.2
PATCH 2/11 net/mlx5/hws: add capabilities fields for vhca access
In reply to Erez ShitritThe new fields define the ability to access from one vhca to other one.
--- drivers/common/mlx5/mlx5_prm.h | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/common/mlx5/mlx5_prm.h +4 -2
@@ -2115,8 +2115,10 @@ struct mlx5_ifc_cmd_hca_cap_2_bits {u8 log_conn_track_max_alloc[0x5 ];u8 reserved_at_d8[0x3 ];u8 log_max_conn_track_offload[0x5 ]; -u8 reserved_at_e0[0x20 ];/* End of DW7. */ -u8 reserved_at_100[0x60 ]; +u8 cross_vhca_object_to_object_supported[0x20 ];/* End of DW7. */ +u8 allowed_object_for_other_vhca_access_high[0x20 ]; +u8 allowed_object_for_other_vhca_access[0x20 ]; +u8 reserved_at_140[0x20 ];u8 reserved_at_160[0x3 ];u8 hairpin_sq_wqe_bb_size[0x5 ];u8 hairpin_sq_wq_in_host_mem[0x1 ];
Signature · 1 line
2.18.2
RE: [PATCH V1 02/11] net/mlx5/hws: add capabilities fields for vhca access
Quoted 11 lines
-----Original Message-----
From: Erez Shitrit <[email protected]>
Sent: Monday, December 19, 2022 10:24 AM
To: Erez Shitrit <[email protected]>; Alex Vesker <[email protected]>; Slava Ovsiienko <[email protected]>; NBU-Contact-Thomas Monjalon (EXTERNAL) <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]
Subject: [PATCH V1 02/11]net/mlx5/hws: add capabilities fields for vhca access
Quoted 4 lines
The new fields define the ability to access from one vhca to other one.
Signed-off-by Erez Shitrit <[email protected]>Reviewed-by Alex Vesker <[email protected]>
v2 cover Support resource sharing among ibv_devices
In reply to Erez ShitritAdd the option to use resources (Tables, Matchers, Actions, etc.) from one gvmi (AKA ibv_cntext) to other gvmi's.
When specific gvmi allows other to use its resources, steering objects Will created and used on that gvmi.
It is done by aliases objects that map between local resources to shared/remote resources.
That in order to allow sharing between few ports.
---
Changes from V1:
- remove mailing list patch from that series, it will be sent separately
- remove the duplicated set that sent by mistake
---
Erez Shitrit (5):
net/mlx5/hws: add capabilities fields for vhca access net/mlx5/hws: add vhca identifier ID to the caps net/mlx5/hws: support shared ibv-context with local one net/mlx5/hws: support actions while shared resources is used net/mlx5/hws: add debug details for cross gvmi
Yevgeny Kliteynik (5):
net/mlx5/hws: remove wrong PRM capability macros net/mlx5/hws: add PRM definitions for cross-vhca capabilities net/mlx5/hws: read cross-vhca capabilities net/mlx5/hws: added allow-other-vhca-access command net/mlx5/hws: added command to create alias objects
drivers/common/mlx5/mlx5_prm.h | 83 ++++++++-- drivers/net/mlx5/hws/mlx5dr.h | 7 +- drivers/net/mlx5/hws/mlx5dr_action.c | 69 +++++++- drivers/net/mlx5/hws/mlx5dr_action.h | 3 + drivers/net/mlx5/hws/mlx5dr_cmd.c | 118 ++++++++++++- drivers/net/mlx5/hws/mlx5dr_cmd.h | 25 +++ drivers/net/mlx5/hws/mlx5dr_context.c | 34 +++- drivers/net/mlx5/hws/mlx5dr_context.h | 22 +++ drivers/net/mlx5/hws/mlx5dr_debug.c | 24 ++- drivers/net/mlx5/hws/mlx5dr_internal.h | 1 + drivers/net/mlx5/hws/mlx5dr_matcher.c | 218 +++++++++++++++++++++++-- drivers/net/mlx5/hws/mlx5dr_matcher.h | 9 + drivers/net/mlx5/hws/mlx5dr_table.c | 191 +++++++++++++++++++++- drivers/net/mlx5/hws/mlx5dr_table.h | 4 +- drivers/net/mlx5/mlx5_devx.c | 2 +- 15 files changed, 749 insertions(+), 61 deletions(-)
Signature · 1 line
2.18.2
PATCH v2 1/10 net/mlx5/hws: add capabilities fields for vhca access
The new fields define the ability to access from one vhca to other one.
--- drivers/common/mlx5/mlx5_prm.h | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/common/mlx5/mlx5_prm.h +4 -2
@@ -2115,8 +2115,10 @@ struct mlx5_ifc_cmd_hca_cap_2_bits {u8 log_conn_track_max_alloc[0x5 ];u8 reserved_at_d8[0x3 ];u8 log_max_conn_track_offload[0x5 ]; -u8 reserved_at_e0[0x20 ];/* End of DW7. */ -u8 reserved_at_100[0x60 ]; +u8 cross_vhca_object_to_object_supported[0x20 ];/* End of DW7. */ +u8 allowed_object_for_other_vhca_access_high[0x20 ]; +u8 allowed_object_for_other_vhca_access[0x20 ]; +u8 reserved_at_140[0x20 ];u8 reserved_at_160[0x3 ];u8 hairpin_sq_wqe_bb_size[0x5 ];u8 hairpin_sq_wq_in_host_mem[0x1 ];
Signature · 1 line
2.18.2
PATCH v2 2/10 net/mlx5/hws: remove wrong PRM capability macros
In reply to Erez ShitritFrom: Yevgeny Kliteynik <[email protected]>
Removing macros for wrong capability checks.
--- drivers/common/mlx5/mlx5_prm.h | 6 ------ 1 file changed, 6 deletions(-)
drivers/common/mlx5/mlx5_prm.h +0 -6
@@ -1332,12 +1332,6 @@ enum { (1ULL << MLX5_GENERAL_OBJ_TYPE_GENEVE_TLV_OPT)#define MLX5_GENERAL_OBJ_TYPES_CAP_CONN_TRACK_OFFLOAD \ (1ULL << MLX5_GENERAL_OBJ_TYPE_CONN_TRACK_OFFLOAD) -#define MLX5_GENERAL_OBJ_TYPES_CAP_RTC \ - (1ULL << MLX5_GENERAL_OBJ_TYPE_RTC) -#define MLX5_GENERAL_OBJ_TYPES_CAP_STC \ - (1ULL << MLX5_GENERAL_OBJ_TYPE_STC) -#define MLX5_GENERAL_OBJ_TYPES_CAP_STE \ - (1ULL << MLX5_GENERAL_OBJ_TYPE_STE)#define MLX5_GENERAL_OBJ_TYPES_CAP_DEFINER \ (1ULL << MLX5_GENERAL_OBJ_TYPE_DEFINER)#define MLX5_GENERAL_OBJ_TYPES_CAP_DEK \
Signature · 1 line
2.18.2
PATCH v2 3/10 net/mlx5/hws: add PRM definitions for cross-vhca capabilities
In reply to Erez ShitritFrom: Yevgeny Kliteynik <[email protected]>
Each new cap was defined.
--- drivers/common/mlx5/mlx5_prm.h | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-)
drivers/common/mlx5/mlx5_prm.h +18 -1
@@ -1930,7 +1930,9 @@ struct mlx5_ifc_flow_table_prop_layout_bits {u8 metadata_reg_a_width[0x8 ];u8 reserved_at_60[0xa ];u8 reparse[0x1 ]; -u8 reserved_at_6b[0xd ]; +u8 reserved_at_6b[0x1 ]; +u8 cross_vhca_object[0x1 ]; +u8 reserved_at_6d[0xb ];u8 log_max_ft_num[0x8 ];u8 reserved_at_80[0x10 ];u8 log_max_flow_counter[0x8 ];
@@ -2084,6 +2086,19 @@ struct mlx5_ifc_flow_table_esw_cap_bits {u8 reserved_at_C00[0x7400 ]; }; +enum mlx5_ifc_cross_vhca_object_to_object_supported_types { +MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_STC_TO_TIR =1 <<10 , +MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_STC_TO_FT =1 <<11 , +MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_FT_TO_FT =1 <<12 , +MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_FT_TO_RTC =1 <<13 , +}; + +enum mlx5_ifc_cross_vhca_allowed_objects_types { +MLX5_CROSS_VHCA_ALLOWED_OBJS_TIR =1 <<0x8 , +MLX5_CROSS_VHCA_ALLOWED_OBJS_FT =1 <<0x9 , +MLX5_CROSS_VHCA_ALLOWED_OBJS_RTC =1 <<0xa , +}; +/* * HCA Capabilities 2 */
@@ -3108,6 +3123,8 @@ enum {MLX5_GENERAL_OBJ_TYPE_RTC =0x0041 ,MLX5_GENERAL_OBJ_TYPE_STE =0x0042 ,MLX5_GENERAL_OBJ_TYPE_MODIFY_HEADER_PATTERN =0x0043 , +MLX5_GENERAL_OBJ_TYPE_FT_ALIAS =0xff15 , + MLX5_GENERAL_OBJ_TYPE_TIR_ALIAS = 0xff16, }; struct mlx5_ifc_general_obj_in_cmd_hdr_bits {
Signature · 1 line
2.18.2
PATCH v2 4/10 net/mlx5/hws: read cross-vhca capabilities
In reply to Erez ShitritFrom: Yevgeny Kliteynik <[email protected]>
And keep them for future processing.
--- drivers/net/mlx5/hws/mlx5dr_cmd.c | 24 ++++++++++++++++++++++++ drivers/net/mlx5/hws/mlx5dr_cmd.h | 1 + 2 files changed, 25 insertions(+)
drivers/net/mlx5/hws/mlx5dr_cmd.c +24 -0
@@ -723,6 +723,7 @@ int mlx5dr_cmd_query_caps(struct ibv_context *ctx,uint32_t in[MLX5_ST_SZ_DW (query_hca_cap_in)] = {0 };const struct flow_hw_port_info *port_info;struct ibv_device_attr_ex attr_ex; +u32 res;int ret;MLX5_SET (query_hca_cap_in, in, opcode,MLX5_CMD_OP_QUERY_HCA_CAP );
@@ -798,6 +799,23 @@ int mlx5dr_cmd_query_caps(struct ibv_context *ctx, capability.cmd_hca_cap_2. format_select_dw_gtpu_first_ext_dw_0); +/* check cross-VHCA support in cap2 */ + res = +MLX5_GET (query_hca_cap_out, out, + capability.cmd_hca_cap_2.cross_vhca_object_to_object_supported); + + caps->cross_vhca_resources = (res &MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_STC_TO_TIR ) && + (res &MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_STC_TO_FT ) && + (res &MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_FT_TO_RTC ); + + res = +MLX5_GET (query_hca_cap_out, out, + capability.cmd_hca_cap_2.allowed_object_for_other_vhca_access); + + caps->cross_vhca_resources &= (res &MLX5_CROSS_VHCA_ALLOWED_OBJS_TIR ) && + (res &MLX5_CROSS_VHCA_ALLOWED_OBJS_FT ) && + (res &MLX5_CROSS_VHCA_ALLOWED_OBJS_RTC ); +MLX5_SET (query_hca_cap_in, in, op_mod,MLX5_GET_HCA_CAP_OP_MOD_NIC_FLOW_TABLE |MLX5_HCA_CAP_OPMOD_GET_CUR );
@@ -817,6 +835,12 @@ int mlx5dr_cmd_query_caps(struct ibv_context *ctx, capability.flow_table_nic_cap. flow_table_properties_nic_receive.reparse); +/* check cross-VHCA support in flow table properties */ + res = +MLX5_GET (query_hca_cap_out, out, + capability.flow_table_nic_cap.flow_table_properties_nic_receive.cross_vhca_object); + caps->cross_vhca_resources &= res; +if (caps->wqe_based_update) { MLX5_SET(query_hca_cap_in, in, op_mod, MLX5_GET_HCA_CAP_OP_MOD_WQE_BASED_FLOW_TABLE |
drivers/net/mlx5/hws/mlx5dr_cmd.h +1 -0
@@ -163,6 +163,7 @@ struct mlx5dr_cmd_query_caps {uint8_t sq_ts_format;uint64_t definer_format_sup;uint32_t trivial_match_definer; +bool cross_vhca_resources;char fw_ver[64 ]; };
Signature · 1 line
2.18.2
PATCH v2 5/10 net/mlx5/hws: added allow-other-vhca-access command
In reply to Erez ShitritFrom: Yevgeny Kliteynik <[email protected]>
Added FW command to allow creation of alias objects.
--- drivers/common/mlx5/mlx5_prm.h | 23 +++++++++++++++++++++++ drivers/net/mlx5/hws/mlx5dr_cmd.c | 28 ++++++++++++++++++++++++++++ drivers/net/mlx5/hws/mlx5dr_cmd.h | 9 +++++++++ 3 files changed, 60 insertions(+)
drivers/common/mlx5/mlx5_prm.h +23 -0
@@ -1133,6 +1133,7 @@ enum {MLX5_CMD_SET_REGEX_REGISTERS =0xb06 ,MLX5_CMD_QUERY_REGEX_REGISTERS =0xb07 ,MLX5_CMD_OP_ACCESS_REGISTER_USER =0xb0c , + MLX5_CMD_OP_ALLOW_OTHER_VHCA_ACCESS = 0xb16, }; enum {
@@ -3150,6 +3151,28 @@ struct mlx5_ifc_general_obj_out_cmd_hdr_bits {u8 reserved_at_60[0x20 ]; }; +struct mlx5_ifc_allow_other_vhca_access_in_bits { +u8 opcode[0x10 ]; +u8 uid[0x10 ]; +u8 reserved_at_20[0x10 ]; +u8 op_mod[0x10 ]; +u8 reserved_at_40[0x50 ]; +u8 object_type_to_be_accessed[0x10 ]; +u8 object_id_to_be_accessed[0x20 ]; +u8 reserved_at_c0[0x40 ]; +union { +u8 access_key_raw[0x100 ]; +u8 access_key[8 ][0x20 ]; + }; +}; + +struct mlx5_ifc_allow_other_vhca_access_out_bits { +u8 status[0x8 ]; +u8 reserved_at_8[0x18 ]; +u8 syndrome[0x20 ]; +u8 reserved_at_40[0x40 ]; +}; +struct mlx5_ifc_virtio_q_counters_bits {u8 modify_field_select[0x40 ];u8 reserved_at_40[0x40 ];
drivers/net/mlx5/hws/mlx5dr_cmd.c +28 -0
@@ -716,6 +716,34 @@ int mlx5dr_cmd_sq_modify_rdy(struct mlx5dr_devx_obj *devx_obj)return ret; } +int mlx5dr_cmd_allow_other_vhca_access (struct ibv_context *ctx, +struct mlx5dr_cmd_allow_other_vhca_access_attr *attr) +{ +uint32_t out[MLX5_ST_SZ_DW (allow_other_vhca_access_out)] = {0 }; +uint32_t in[MLX5_ST_SZ_DW (allow_other_vhca_access_in)] = {0 }; +void *key; +int ret; + +MLX5_SET (allow_other_vhca_access_in, + in, opcode,MLX5_CMD_OP_ALLOW_OTHER_VHCA_ACCESS ); +MLX5_SET (allow_other_vhca_access_in, + in, object_type_to_be_accessed, attr->obj_type); +MLX5_SET (allow_other_vhca_access_in, + in, object_id_to_be_accessed, attr->obj_id); + + key =MLX5_ADDR_OF (allow_other_vhca_access_in, in, access_key); +memcpy (key, attr->access_key,sizeof (attr->access_key)); + + ret = mlx5_glue->devx_general_cmd (ctx, in,sizeof (in), out,sizeof (out)); +if (ret) { +DR_LOG (ERR ,"Failed to execute ALLOW_OTHER_VHCA_ACCESS command" ); + rte_errno = errno; +return rte_errno; + } + +return 0 ; +} +int mlx5dr_cmd_query_caps (struct ibv_context *ctx,struct mlx5dr_cmd_query_caps *caps) {
drivers/net/mlx5/hws/mlx5dr_cmd.h +9 -0
@@ -124,6 +124,12 @@ struct mlx5dr_cmd_sq_create_attr {uint32_t ts_format; }; +struct mlx5dr_cmd_allow_other_vhca_access_attr { +uint16_t obj_type; +uint32_t obj_id; +uint8_t access_key[32 ]; +}; +struct mlx5dr_cmd_query_ft_caps {uint8_t max_level;uint8_t reparse;
@@ -230,4 +236,7 @@ void mlx5dr_cmd_set_attr_connect_miss_tbl(struct mlx5dr_context *ctx,uint32_t fw_ft_type, enum mlx5dr_table_type type, struct mlx5dr_cmd_ft_modify_attr *ft_attr); + +int mlx5dr_cmd_allow_other_vhca_access (struct ibv_context *ctx, +struct mlx5dr_cmd_allow_other_vhca_access_attr *attr);#endif /* MLX5DR_CMD_H_ */
Signature · 1 line
2.18.2
PATCH v2 6/10 net/mlx5/hws: added command to create alias objects
In reply to Erez ShitritFrom: Yevgeny Kliteynik <[email protected]>
Added support for a command that can create alias objects - objects that are accessed across VHCA.
--- drivers/common/mlx5/mlx5_prm.h | 23 ++++++++++++++-- drivers/net/mlx5/hws/mlx5dr_cmd.c | 44 +++++++++++++++++++++++++++++++ drivers/net/mlx5/hws/mlx5dr_cmd.h | 11 ++++++++ 3 files changed, 76 insertions(+), 2 deletions(-)
drivers/common/mlx5/mlx5_prm.h +21 -2
@@ -3135,9 +3135,10 @@ struct mlx5_ifc_general_obj_in_cmd_hdr_bits {u8 obj_id[0x20 ];union {struct { -u8 reserved_at_60[0x3 ]; +u8 alias_object[0x1 ]; +u8 reserved_at_61[0x2 ];u8 log_obj_range[0x5 ]; -u8 reserved_at_58[0x18 ]; +u8 reserved_at_68[0x18 ]; };u8 obj_offset[0x20 ]; };
@@ -3234,6 +3235,19 @@ struct mlx5_ifc_rtc_bits {u8 reserved_at_180[0x280 ]; }; +struct mlx5_ifc_alias_context_bits { +u8 vhca_id_to_be_accessed[0x10 ]; +u8 reserved_at_10[0xd ]; +u8 status[0x3 ]; +u8 object_id_to_be_accessed[0x20 ]; +u8 reserved_at_40[0x40 ]; +union { +u8 access_key_raw[0x100 ]; +u8 access_key[8 ][0x20 ]; + }; +u8 metadata[0x80 ]; +}; +enum mlx5_ifc_stc_action_type {MLX5_IFC_STC_ACTION_TYPE_NOP =0x00 ,MLX5_IFC_STC_ACTION_TYPE_COPY =0x05 ,
@@ -3478,6 +3492,11 @@ struct mlx5_ifc_create_header_modify_pattern_in_bits {struct mlx5_ifc_header_modify_pattern_in_bits pattern; }; +struct mlx5_ifc_create_alias_obj_in_bits { +struct mlx5_ifc_general_obj_in_cmd_hdr_bits hdr; +struct mlx5_ifc_alias_context_bits alias_ctx; +}; +enum { MLX5_CRYPTO_KEY_SIZE_128b =0x0 , MLX5_CRYPTO_KEY_SIZE_256b =0x1 ,
drivers/net/mlx5/hws/mlx5dr_cmd.c +44 -0
50 lines not on this page · raw message
drivers/net/mlx5/hws/mlx5dr_cmd.h +11 -0
@@ -46,6 +46,13 @@ struct mlx5dr_cmd_rtc_create_attr {bool is_jumbo; }; +struct mlx5dr_cmd_alias_obj_create_attr { +uint32_t obj_id; +uint16_t vhca_id; +uint16_t obj_type; +uint8_t access_key[32 ]; +}; +struct mlx5dr_cmd_stc_create_attr {uint8_t log_obj_range;uint8_t table_type;
@@ -217,6 +224,10 @@ mlx5dr_cmd_header_modify_pattern_create(struct ibv_context *ctx,uint32_t pattern_length, uint8_t *actions); +struct mlx5dr_devx_obj * +mlx5dr_cmd_alias_obj_create (struct ibv_context *ctx, +struct mlx5dr_cmd_alias_obj_create_attr *alias_attr); +int mlx5dr_cmd_sq_modify_rdy (struct mlx5dr_devx_obj *devx_obj); int mlx5dr_cmd_query_ib_port(struct ibv_context *ctx,
Signature · 1 line
2.18.2
PATCH v2 7/10 net/mlx5/hws: add vhca identifier ID to the caps
In reply to Erez ShitritAnd read it in the query_cap function
--- drivers/net/mlx5/hws/mlx5dr_cmd.c | 3 +++ drivers/net/mlx5/hws/mlx5dr_cmd.h | 1 + 2 files changed, 4 insertions(+)
drivers/net/mlx5/hws/mlx5dr_cmd.c +3 -0
@@ -837,6 +837,9 @@ int mlx5dr_cmd_query_caps(struct ibv_context *ctx,MLX5_GET64 (query_hca_cap_out, out, capability.cmd_hca_cap.match_definer_format_supported); + caps->vhca_id =MLX5_GET (query_hca_cap_out, out, + capability.cmd_hca_cap.vhca_id); + caps->sq_ts_format =MLX5_GET (query_hca_cap_out, out, capability.cmd_hca_cap.sq_ts_format);
drivers/net/mlx5/hws/mlx5dr_cmd.h +1 -0
@@ -176,6 +176,7 @@ struct mlx5dr_cmd_query_caps {uint8_t sq_ts_format;uint64_t definer_format_sup;uint32_t trivial_match_definer; +uint32_t vhca_id;bool cross_vhca_resources;char fw_ver[64 ]; };
Signature · 1 line
2.18.2
PATCH v2 8/10 net/mlx5/hws: support shared ibv-context with local one
In reply to Erez ShitritThe idea is to have a shared ibv_context that all the resources allocated on it (FT + TIR are exceptions)
When ever a resource created locally an alias object to that resource allocated and it used in the other context.
The connections between the resources are done according to each type of the resource, to the original resource or to its alias resource.
--- drivers/common/mlx5/mlx5_prm.h | 6 + drivers/net/mlx5/hws/mlx5dr.h | 2 + drivers/net/mlx5/hws/mlx5dr_action.c | 9 +- drivers/net/mlx5/hws/mlx5dr_cmd.c | 19 ++- drivers/net/mlx5/hws/mlx5dr_cmd.h | 7 +- drivers/net/mlx5/hws/mlx5dr_context.c | 34 +++- drivers/net/mlx5/hws/mlx5dr_context.h | 22 +++ drivers/net/mlx5/hws/mlx5dr_debug.c | 2 +- drivers/net/mlx5/hws/mlx5dr_internal.h | 1 + drivers/net/mlx5/hws/mlx5dr_matcher.c | 218 +++++++++++++++++++++++-- drivers/net/mlx5/hws/mlx5dr_matcher.h | 9 + drivers/net/mlx5/hws/mlx5dr_table.c | 191 +++++++++++++++++++++- drivers/net/mlx5/hws/mlx5dr_table.h | 4 +- 13 files changed, 484 insertions(+), 40 deletions(-)
drivers/common/mlx5/mlx5_prm.h +6 -0
@@ -291,6 +291,12 @@#define MAX_ACTIONS_DATA_IN_HEADER_MODIFY 512 +/* Alias FT id passed to the ALLOW_OTHER_VHCA_ACCESS & CREATE_GENERAL_OBJECT +* commands should have the following format: +* {table_type: 8bits, table_id: 24bits}. +*/ +#define FT_ID_FT_TYPE_OFFSET 24 +/* Completion mode. */ enum mlx5_completion_mode {MLX5_COMP_ONLY_ERR =0x0 ,
drivers/net/mlx5/hws/mlx5dr.h +2 -0
@@ -96,6 +96,8 @@ struct mlx5dr_context_attr {size_t initial_log_ste_memory;/* Currently not in use */ /* Optional PD used for allocating res ources */ struct ibv_pd *pd; +/* Optional other ctx for resources allocation, all objects will be created on it */ +struct ibv_context *shared_ibv_ctx; };struct mlx5dr_table_attr {
drivers/net/mlx5/hws/mlx5dr_action.c +7 -2
@@ -1147,6 +1147,7 @@ mlx5dr_action_create_reformat_root(struct mlx5dr_action *action, {enum mlx5dv_flow_table_type ft_type =0 ;/*fix compilation warn*/ uint32_t verb_reformat_type =0 ; +struct ibv_context *ibv_ctx;int ret;/* Convert action to FT type and verbs reformat type */
@@ -1157,8 +1158,9 @@ mlx5dr_action_create_reformat_root(struct mlx5dr_action *action,mlx5dr_action_conv_reformat_to_verbs (action->type, &verb_reformat_type);/* Create the reformat type for root table */ + ibv_ctx =mlx5dr_context_get_local_ibv (action->ctx); action->flow_action = - mlx5_glue->dv_create_flow_action_packet_reformat_root(action->ctx->ibv_ctx, + mlx5_glue->dv_create_flow_action_packet_reformat_root(ibv_ctx, data_sz, data, verb_reformat_type,
@@ -1496,14 +1498,17 @@ mlx5dr_action_create_modify_header_root(struct mlx5dr_action *action, __be64 *actions) {enum mlx5dv_flow_table_type ft_type =0 ; +struct ibv_context *local_ibv_ctx;int ret; ret =mlx5dr_action_conv_flags_to_ft_type (action->flags, &ft_type);if (ret)return rte_errno; + local_ibv_ctx =mlx5dr_context_get_local_ibv (action->ctx); + action->flow_action = - mlx5_glue->dv_create_flow_action_modify_header_root (action->ctx->ibv_ctx, + mlx5_glue->dv_create_flow_action_modify_header_root (local_ibv_ctx, actions_sz, (uint64_t *)actions, ft_type);
drivers/net/mlx5/hws/mlx5dr_cmd.c +12 -7
@@ -217,18 +217,23 @@ void mlx5dr_cmd_set_attr_connect_miss_tbl(struct mlx5dr_context *ctx, {struct mlx5dr_devx_obj *default_miss_tbl; -if (type !=MLX5DR_TABLE_TYPE_FDB ) +if (type !=MLX5DR_TABLE_TYPE_FDB && !mlx5dr_context_shared_gvmi_used (ctx))return ; - default_miss_tbl = ctx->common_res[type].default_miss->ft; -if (!default_miss_tbl) { -assert (false); -return ; - } ft_attr->modify_fs =MLX5_IFC_MODIFY_FLOW_TABLE_MISS_ACTION ; ft_attr->type = fw_ft_type; ft_attr->table_miss_action =MLX5_IFC_MODIFY_FLOW_TABLE_MISS_ACTION_GOTO_TBL ; - ft_attr->table_miss_id = default_miss_tbl->id; + +if (type ==MLX5DR_TABLE_TYPE_FDB ) { + default_miss_tbl = ctx->common_res[type].default_miss->ft; +if (!default_miss_tbl) { +assert (false); +return ; + } + ft_attr->table_miss_id = default_miss_tbl->id; + }else { + ft_attr->table_miss_id = ctx->gvmi_res[type].aliased_end_ft->id; + } } struct mlx5dr_devx_obj *
drivers/net/mlx5/hws/mlx5dr_cmd.h +5 -2
@@ -11,6 +11,8 @@ struct mlx5dr_cmd_ft_create_attr {bool rtc_valid; }; +#define ACCESS_KEY_LEN 32 +struct mlx5dr_cmd_ft_modify_attr {uint8_t type;uint32_t rtc_id_0;
@@ -50,7 +52,7 @@ struct mlx5dr_cmd_alias_obj_create_attr {uint32_t obj_id;uint16_t vhca_id;uint16_t obj_type; -uint8_t access_key[32 ]; +uint8_t access_key[ACCESS_KEY_LEN ]; };struct mlx5dr_cmd_stc_create_attr {
@@ -134,7 +136,7 @@ struct mlx5dr_cmd_sq_create_attr {struct mlx5dr_cmd_allow_other_vhca_access_attr {uint16_t obj_type;uint32_t obj_id; -uint8_t access_key[32 ]; +uint8_t access_key[ACCESS_KEY_LEN ]; };struct mlx5dr_cmd_query_ft_caps {
@@ -178,6 +180,7 @@ struct mlx5dr_cmd_query_caps {uint32_t trivial_match_definer;uint32_t vhca_id;bool cross_vhca_resources; +uint32_t shared_vhca_id;char fw_ver[64 ]; };
drivers/net/mlx5/hws/mlx5dr_context.c +33 -1
@@ -178,6 +178,36 @@ static void mlx5dr_context_uninit_hws(struct mlx5dr_context *ctx)mlx5dr_context_uninit_pd (ctx); } +static int mlx5dr_context_init_shared_ctx (struct mlx5dr_context *ctx, +struct ibv_context *ibv_ctx, +struct mlx5dr_context_attr *attr) +{ +struct mlx5dr_cmd_query_caps shared_caps = {0 }; +int ret; + +if (!attr->shared_ibv_ctx) { + ctx->ibv_ctx = ibv_ctx; + }else { + ctx->ibv_ctx = attr->shared_ibv_ctx; + ctx->local_ibv_ctx = ibv_ctx; + ret =mlx5dr_cmd_query_caps (attr->shared_ibv_ctx, &shared_caps); +if (ret || !shared_caps.cross_vhca_resources) { +DR_LOG (INFO ,"No cross_vhca_resources cap for shared ibv" ); + rte_errno =ENOTSUP ; +return rte_errno; + } + ctx->caps->shared_vhca_id = shared_caps.vhca_id; + } + +if (ctx->local_ibv_ctx && !ctx->caps->cross_vhca_resources) { +DR_LOG (INFO ,"No cross_vhca_resources cap for local ibv" ); + rte_errno =ENOTSUP ; +return rte_errno; + } + +return 0 ; +} +struct mlx5dr_context *mlx5dr_context_open (struct ibv_context *ibv_ctx,struct mlx5dr_context_attr *attr) {
@@ -190,7 +220,6 @@ struct mlx5dr_context *mlx5dr_context_open(struct ibv_context *ibv_ctx,return NULL ; } - ctx->ibv_ctx = ibv_ctx;pthread_spin_init (&ctx->ctrl_lock,PTHREAD_PROCESS_PRIVATE ); ctx->caps =simple_calloc (1 ,sizeof (*ctx->caps));
@@ -201,6 +230,9 @@ struct mlx5dr_context *mlx5dr_context_open(struct ibv_context *ibv_ctx,if (ret) goto free_caps; +if (mlx5dr_context_init_shared_ctx (ctx, ibv_ctx, attr)) + goto free_caps; + ret =mlx5dr_context_init_hws (ctx, attr);if (ret) goto free_caps;
drivers/net/mlx5/hws/mlx5dr_context.h +22 -0
@@ -22,13 +22,22 @@ struct mlx5dr_context_common_res {struct mlx5dr_cmd_forward_tbl *default_miss; }; +struct mlx5dr_context_shared_gvmi_res { +struct mlx5dr_devx_obj *end_ft; +struct mlx5dr_devx_obj *aliased_end_ft; +uint32_t refcount; +}; +struct mlx5dr_context {struct ibv_context *ibv_ctx; +/* When local_ibv_ctx is not NULL means we are using shared_ibv for resources */ +struct ibv_context *local_ibv_ctx;struct mlx5dr_cmd_query_caps *caps;struct ibv_pd *pd;uint32_t pd_num;struct mlx5dr_pool *stc_pool[MLX5DR_TABLE_TYPE_MAX ];struct mlx5dr_context_common_res common_res[MLX5DR_TABLE_TYPE_MAX ]; +struct mlx5dr_context_shared_gvmi_res gvmi_res[MLX5DR_TABLE_TYPE_MAX ];struct mlx5dr_pattern_cache *pattern_cache;pthread_spinlock_t ctrl_lock;enum mlx5dr_context_flags flags;
@@ -37,4 +46,17 @@ struct mlx5dr_context { LIST_HEAD(table_head, mlx5dr_table) head; }; +static inline bool mlx5dr_context_shared_gvmi_used (struct mlx5dr_context *ctx) +{ +return ctx->local_ibv_ctx ? true : false; +} + +static inline struct ibv_context * +mlx5dr_context_get_local_ibv (struct mlx5dr_context *ctx) +{ +if (mlx5dr_context_shared_gvmi_used (ctx)) +return ctx->local_ibv_ctx; + +return ctx->ibv_ctx; +}#endif /* MLX5DR_CONTEXT_H_ */
drivers/net/mlx5/hws/mlx5dr_debug.c +1 -1
@@ -405,7 +405,7 @@ static int mlx5dr_debug_dump_context_info(FILE *f, struct mlx5dr_context *ctx)MLX5DR_DEBUG_RES_TYPE_CONTEXT , (uint64_t )(uintptr_t )ctx, ctx->flags &MLX5DR_CONTEXT_FLAG_HWS_SUPPORT , - mlx5_glue->get_device_name (ctx->ibv_ctx->device), + mlx5_glue->get_device_name (mlx5dr_context_get_local_ibv (ctx)->device), DEBUG_VERSION);if (ret <0 ) { rte_errno =EINVAL ;
drivers/net/mlx5/hws/mlx5dr_internal.h +1 -0
@@ -18,6 +18,7 @@#endif #include <rte_flow.h> #include <rte_gtp.h> +#include <rte_random.h> #include "mlx5_prm.h" #include "mlx5_glue.h"
drivers/net/mlx5/hws/mlx5dr_matcher.c +201 -17
289 lines not on this page · raw message
drivers/net/mlx5/hws/mlx5dr_matcher.h +9 -0
@@ -42,6 +42,8 @@ struct mlx5dr_matcher_match_ste {struct mlx5dr_devx_obj *rtc_0;struct mlx5dr_devx_obj *rtc_1;struct mlx5dr_pool *pool; +/* Currently not support FDB aliased */ +struct mlx5dr_devx_obj *aliased_rtc_0; };struct mlx5dr_matcher_action_ste {
@@ -73,4 +75,11 @@ int mlx5dr_matcher_conv_items_to_prm(uint64_t *match_buf,uint8_t *match_criteria, bool is_value); +int mlx5dr_matcher_create_aliased_obj (struct mlx5dr_context *ctx, +struct ibv_context *ibv_owner, +struct ibv_context *ibv_allowed, +uint16_t vhca_id_to_be_accessed, +uint32_t aliased_object_id, +uint16_t object_type, +struct mlx5dr_devx_obj **obj);#endif /* MLX5DR_MATCHER_H_ */
drivers/net/mlx5/hws/mlx5dr_table.c +182 -9
243 lines not on this page · raw message
drivers/net/mlx5/hws/mlx5dr_table.h +3 -1
@@ -10,6 +10,7 @@struct mlx5dr_table {struct mlx5dr_context *ctx;struct mlx5dr_devx_obj *ft; +struct mlx5dr_devx_obj *local_ft;enum mlx5dr_table_type type;uint32_t fw_ft_type;uint32_t level;
@@ -37,7 +38,8 @@ static inline bool mlx5dr_table_is_root(struct mlx5dr_table *tbl)return (tbl->level ==MLX5DR_ROOT_LEVEL ); } -struct mlx5dr_devx_obj *mlx5dr_table_create_default_ft (struct mlx5dr_table *tbl); +struct mlx5dr_devx_obj *mlx5dr_table_create_default_ft (struct ibv_context *ibv, +struct mlx5dr_table *tbl);void mlx5dr_table_destroy_default_ft (struct mlx5dr_table *tbl,struct mlx5dr_devx_obj *ft_obj);
Signature · 1 line
2.18.2
PATCH v2 9/10 net/mlx5/hws: support actions while shared resources is used
In reply to Erez ShitritTIR/FT actions are different in the context of shared ibv resource, it created on the local ibv_context and aliased to the shared ibv_context. Other actions should be created on the shared ibv resource, new flag was added to indicates where this object came from.
--- drivers/net/mlx5/hws/mlx5dr.h | 5 ++- drivers/net/mlx5/hws/mlx5dr_action.c | 60 ++++++++++++++++++++++++++-- drivers/net/mlx5/hws/mlx5dr_action.h | 3 ++ drivers/net/mlx5/mlx5_devx.c | 2 +- 4 files changed, 64 insertions(+), 6 deletions(-)
drivers/net/mlx5/hws/mlx5dr.h +4 -1
@@ -410,12 +410,15 @@ mlx5dr_action_create_dest_vport(struct mlx5dr_context *ctx,* Direct rule TIR devx object. * @param[in] flags * Action creation flags. (enum mlx5dr_action_flags) +* @param[in] is_local +* indicates where the tir object was created, local gvmi or other gvmi * @return pointer to mlx5dr_action on success NULL otherwise. */ struct mlx5dr_action *mlx5dr_action_create_dest_tir (struct mlx5dr_context *ctx,struct mlx5dr_devx_obj *obj, -uint32_t flags); +uint32_t flags, +bool is_local);/* Create direct rule TAG action. *
drivers/net/mlx5/hws/mlx5dr_action.c +56 -4
98 lines not on this page · raw message
drivers/net/mlx5/hws/mlx5dr_action.h +3 -0
@@ -138,6 +138,9 @@ struct mlx5dr_action {uint16_t vport_num;uint16_t esw_owner_vhca_id; } vport; +struct { +struct mlx5dr_devx_obj *devx_obj; + } alias; }; };
drivers/net/mlx5/mlx5_devx.c +1 -1
@@ -913,7 +913,7 @@ mlx5_devx_hrxq_new(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq,if (hrxq->hws_flags) { hrxq->action =mlx5dr_action_create_dest_tir (priv->dr_ctx, - (struct mlx5dr_devx_obj *)hrxq->tir, hrxq->hws_flags); + (struct mlx5dr_devx_obj *)hrxq->tir, hrxq->hws_flags, true);if (!hrxq->action) goto error;return 0 ;
Signature · 1 line
2.18.2
PATCH v2 10/10 net/mlx5/hws: add debug details for cross gvmi
In reply to Erez ShitritFor context, table and matcher.
--- drivers/net/mlx5/hws/mlx5dr_debug.c | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-)
drivers/net/mlx5/hws/mlx5dr_debug.c +16 -6
@@ -177,6 +177,7 @@ mlx5dr_debug_dump_matcher_attr(FILE *f, struct mlx5dr_matcher *matcher)static int mlx5dr_debug_dump_matcher (FILE *f,struct mlx5dr_matcher *matcher) { +bool is_shared =mlx5dr_context_shared_gvmi_used (matcher->tbl->ctx);bool is_root = matcher->tbl->level ==MLX5DR_ROOT_LEVEL ;enum mlx5dr_table_type tbl_type = matcher->tbl->type;struct mlx5dr_devx_obj *ste_0, *ste_1 =NULL ;
@@ -205,11 +206,13 @@ static int mlx5dr_debug_dump_matcher(FILE *f, struct mlx5dr_matcher *matcher) ste_1 =NULL ; } - ret =fprintf (f,",%d,%d,%d,%d" , + ret =fprintf (f,",%d,%d,%d,%d,%d" , matcher->match_ste.rtc_0 ? matcher->match_ste.rtc_0->id :0 , ste_0 ? (int )ste_0->id :-1 , matcher->match_ste.rtc_1 ? matcher->match_ste.rtc_1->id :0 , - ste_1 ? (int )ste_1->id :-1 ); + ste_1 ? (int )ste_1->id :-1 , + is_shared && !is_root ? + matcher->match_ste.aliased_rtc_0->id :0 );if (ret <0 ) goto out_err;
@@ -253,18 +256,20 @@ static int mlx5dr_debug_dump_matcher(FILE *f, struct mlx5dr_matcher *matcher)static int mlx5dr_debug_dump_table (FILE *f,struct mlx5dr_table *tbl) { +bool is_shared =mlx5dr_context_shared_gvmi_used (tbl->ctx);bool is_root = tbl->level ==MLX5DR_ROOT_LEVEL ;struct mlx5dr_matcher *matcher;int ret; - ret =fprintf (f,"%d,0x%" PRIx64",0x%" PRIx64",%d,%d,%d,%d\n" , + ret =fprintf (f,"%d,0x%" PRIx64",0x%" PRIx64",%d,%d,%d,%d,%d\n" ,MLX5DR_DEBUG_RES_TYPE_TABLE , (uint64_t )(uintptr_t )tbl, (uint64_t )(uintptr_t )tbl->ctx, is_root ?0 : tbl->ft->id, tbl->type, is_root ?0 : tbl->fw_ft_type, - tbl->level); + tbl->level, + is_shared && !is_root ? tbl->local_ft->id :0 );if (ret <0 ) { rte_errno =EINVAL ;return rte_errno;
@@ -383,12 +388,17 @@ static int mlx5dr_debug_dump_context_attr(FILE *f, struct mlx5dr_context *ctx) {int ret; - ret =fprintf (f,"%u,0x%" PRIx64",%d,%zu,%d\n" , + ret =fprintf (f,"%u,0x%" PRIx64",%d,%zu,%d,%s,%d,%d\n" ,MLX5DR_DEBUG_RES_TYPE_CONTEXT_ATTR , (uint64_t )(uintptr_t )ctx, ctx->pd_num, ctx->queues, - ctx->send_queue->num_entries); + ctx->send_queue->num_entries, +mlx5dr_context_shared_gvmi_used (ctx) ? + mlx5_glue->get_device_name (ctx->ibv_ctx->device) :"None" , + ctx->caps->vhca_id, +mlx5dr_context_shared_gvmi_used (ctx) ? + ctx->caps->shared_vhca_id :0xffff );if (ret <0 ) { rte_errno =EINVAL ;return rte_errno;
Signature · 1 line
2.18.2
Hi,
Quoted 57 lines
-----Original Message-----
From: Erez Shitrit <[email protected]>
Sent: Thursday, January 5, 2023 3:03 PM
To: [email protected]
Cc: Alex Vesker <[email protected]>; Raslan Darawsheh <[email protected]>
Subject: [PATCH v2 00/10] Support resource sharing amongibv_devicesAdd the option to use resources (Tables, Matchers, Actions, etc.) from one gvmi (AKA
ibv_cntext) to other gvmi's.When specific gvmi allows other to use its resources, steering objects Will created and used on that gvmi.
It is done by aliases objects that map between local resources to shared/remote resources.
That in order to allow sharing between few ports.
---
Changes from V1:
- remove mailing list patch from that series, it will be sent separately
- remove the duplicated set that sent by mistake
---Erez Shitrit (5):
net/mlx5/hws: add capabilities fields for vhca accessnet/mlx5/hws: add vhca identifier ID to the capsnet/mlx5/hws: support shared ibv-context with local onenet/mlx5/hws: support actions while shared resources is usednet/mlx5/hws: add debug details for cross gvmiYevgeny Kliteynik (5):
net/mlx5/hws: remove wrong PRM capability macrosnet/mlx5/hws: add PRM definitions for cross-vhca capabilitiesnet/mlx5/hws: read cross-vhca capabilitiesnet/mlx5/hws: added allow-other-vhca-access commandnet/mlx5/hws: added command to create alias objectsdrivers/common/mlx5/mlx5_prm.h | 83 ++++++++-- drivers/net/mlx5/hws/mlx5dr.h | 7 +- drivers/net/mlx5/hws/mlx5dr_action.c | 69 +++++++- drivers/net/mlx5/hws/mlx5dr_action.h | 3 + drivers/net/mlx5/hws/mlx5dr_cmd.c | 118 ++++++++++++- drivers/net/mlx5/hws/mlx5dr_cmd.h | 25 +++ drivers/net/mlx5/hws/mlx5dr_context.c | 34 +++- drivers/net/mlx5/hws/mlx5dr_context.h | 22 +++ drivers/net/mlx5/hws/mlx5dr_debug.c | 24 ++- drivers/net/mlx5/hws/mlx5dr_internal.h | 1 + drivers/net/mlx5/hws/mlx5dr_matcher.c | 218 +++++++++++++++++++++++-- drivers/net/mlx5/hws/mlx5dr_matcher.h | 9 + drivers/net/mlx5/hws/mlx5dr_table.c | 191 +++++++++++++++++++++- drivers/net/mlx5/hws/mlx5dr_table.h | 4 +- drivers/net/mlx5/mlx5_devx.c | 2 +- 15 files changed, 749 insertions(+), 61 deletions(-)
Quoted 2 lines
--
2.18.2
Series applied to next-net-mlx,
Kindest regards,
Raslan Darawsheh
PATCH 3/11 net/mlx5/hws: remove wrong PRM capability macros
In reply to Erez ShitritFrom: Yevgeny Kliteynik <[email protected]>
Removing macros for wrong capability checks.
--- drivers/common/mlx5/mlx5_prm.h | 6 ------ 1 file changed, 6 deletions(-)
drivers/common/mlx5/mlx5_prm.h +0 -6
@@ -1332,12 +1332,6 @@ enum { (1ULL << MLX5_GENERAL_OBJ_TYPE_GENEVE_TLV_OPT)#define MLX5_GENERAL_OBJ_TYPES_CAP_CONN_TRACK_OFFLOAD \ (1ULL << MLX5_GENERAL_OBJ_TYPE_CONN_TRACK_OFFLOAD) -#define MLX5_GENERAL_OBJ_TYPES_CAP_RTC \ - (1ULL << MLX5_GENERAL_OBJ_TYPE_RTC) -#define MLX5_GENERAL_OBJ_TYPES_CAP_STC \ - (1ULL << MLX5_GENERAL_OBJ_TYPE_STC) -#define MLX5_GENERAL_OBJ_TYPES_CAP_STE \ - (1ULL << MLX5_GENERAL_OBJ_TYPE_STE)#define MLX5_GENERAL_OBJ_TYPES_CAP_DEFINER \ (1ULL << MLX5_GENERAL_OBJ_TYPE_DEFINER)#define MLX5_GENERAL_OBJ_TYPES_CAP_DEK \
Signature · 1 line
2.18.2
RE: [PATCH V1 03/11] net/mlx5/hws: remove wrong PRM capability macros
Quoted 10 lines
-----Original Message-----
From: Erez Shitrit <[email protected]>
Sent: Monday, December 19, 2022 10:24 AM
To: Erez Shitrit <[email protected]>; Alex Vesker <[email protected]>; Slava Ovsiienko <[email protected]>; NBU-Contact-Thomas Monjalon (EXTERNAL) <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]; Yevgeny Kliteynik <[email protected]>
Subject: [PATCH V1 03/11]net/mlx5/hws: remove wrong PRM capability macros
Quoted 7 lines
From: Yevgeny Kliteynik <[email protected]>
Removing macros for wrong capability checks.
Signed-off-by Yevgeny Kliteynik <[email protected]>Reviewed-by Alex Vesker <[email protected]>
PATCH 4/11 net/mlx5/hws: add PRM definitions for cross-vhca capabilities
In reply to Erez ShitritFrom: Yevgeny Kliteynik <[email protected]>
Each new cap was defined.
--- drivers/common/mlx5/mlx5_prm.h | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-)
drivers/common/mlx5/mlx5_prm.h +18 -1
@@ -1930,7 +1930,9 @@ struct mlx5_ifc_flow_table_prop_layout_bits {u8 metadata_reg_a_width[0x8 ];u8 reserved_at_60[0xa ];u8 reparse[0x1 ]; -u8 reserved_at_6b[0xd ]; +u8 reserved_at_6b[0x1 ]; +u8 cross_vhca_object[0x1 ]; +u8 reserved_at_6d[0xb ];u8 log_max_ft_num[0x8 ];u8 reserved_at_80[0x10 ];u8 log_max_flow_counter[0x8 ];
@@ -2084,6 +2086,19 @@ struct mlx5_ifc_flow_table_esw_cap_bits {u8 reserved_at_C00[0x7400 ]; }; +enum mlx5_ifc_cross_vhca_object_to_object_supported_types { +MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_STC_TO_TIR =1 <<10 , +MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_STC_TO_FT =1 <<11 , +MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_FT_TO_FT =1 <<12 , +MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_FT_TO_RTC =1 <<13 , +}; + +enum mlx5_ifc_cross_vhca_allowed_objects_types { +MLX5_CROSS_VHCA_ALLOWED_OBJS_TIR =1 <<0x8 , +MLX5_CROSS_VHCA_ALLOWED_OBJS_FT =1 <<0x9 , +MLX5_CROSS_VHCA_ALLOWED_OBJS_RTC =1 <<0xa , +}; +/* * HCA Capabilities 2 */
@@ -3108,6 +3123,8 @@ enum {MLX5_GENERAL_OBJ_TYPE_RTC =0x0041 ,MLX5_GENERAL_OBJ_TYPE_STE =0x0042 ,MLX5_GENERAL_OBJ_TYPE_MODIFY_HEADER_PATTERN =0x0043 , +MLX5_GENERAL_OBJ_TYPE_FT_ALIAS =0xff15 , + MLX5_GENERAL_OBJ_TYPE_TIR_ALIAS = 0xff16, }; struct mlx5_ifc_general_obj_in_cmd_hdr_bits {
Signature · 1 line
2.18.2
RE: [PATCH V1 04/11] net/mlx5/hws: add PRM definitions for cross-vhca capabilities
Quoted 10 lines
-----Original Message-----
From: Erez Shitrit <[email protected]>
Sent: Monday, December 19, 2022 10:24 AM
To: Erez Shitrit <[email protected]>; Alex Vesker <[email protected]>; Slava Ovsiienko <[email protected]>; NBU-Contact-Thomas Monjalon (EXTERNAL) <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]; Yevgeny Kliteynik <[email protected]>
Subject: [PATCH V1 04/11]net/mlx5/hws: add PRM definitions for cross-vhca capabilities
Quoted 7 lines
From: Yevgeny Kliteynik <[email protected]>
Each new cap was defined.
Signed-off-by Yevgeny Kliteynik <[email protected]>Reviewed-by Alex Vesker <[email protected]>
PATCH 5/11 net/mlx5/hws: read cross-vhca capabilities
In reply to Erez ShitritFrom: Yevgeny Kliteynik <[email protected]>
And keep them for future processing.
--- drivers/net/mlx5/hws/mlx5dr_cmd.c | 24 ++++++++++++++++++++++++ drivers/net/mlx5/hws/mlx5dr_cmd.h | 1 + 2 files changed, 25 insertions(+)
drivers/net/mlx5/hws/mlx5dr_cmd.c +24 -0
@@ -723,6 +723,7 @@ int mlx5dr_cmd_query_caps(struct ibv_context *ctx,uint32_t in[MLX5_ST_SZ_DW (query_hca_cap_in)] = {0 };const struct flow_hw_port_info *port_info;struct ibv_device_attr_ex attr_ex; +u32 res;int ret;MLX5_SET (query_hca_cap_in, in, opcode,MLX5_CMD_OP_QUERY_HCA_CAP );
@@ -798,6 +799,23 @@ int mlx5dr_cmd_query_caps(struct ibv_context *ctx, capability.cmd_hca_cap_2. format_select_dw_gtpu_first_ext_dw_0); +/* check cross-VHCA support in cap2 */ + res = +MLX5_GET (query_hca_cap_out, out, + capability.cmd_hca_cap_2.cross_vhca_object_to_object_supported); + + caps->cross_vhca_resources = (res &MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_STC_TO_TIR ) && + (res &MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_STC_TO_FT ) && + (res &MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_FT_TO_RTC ); + + res = +MLX5_GET (query_hca_cap_out, out, + capability.cmd_hca_cap_2.allowed_object_for_other_vhca_access); + + caps->cross_vhca_resources &= (res &MLX5_CROSS_VHCA_ALLOWED_OBJS_TIR ) && + (res &MLX5_CROSS_VHCA_ALLOWED_OBJS_FT ) && + (res &MLX5_CROSS_VHCA_ALLOWED_OBJS_RTC ); +MLX5_SET (query_hca_cap_in, in, op_mod,MLX5_GET_HCA_CAP_OP_MOD_NIC_FLOW_TABLE |MLX5_HCA_CAP_OPMOD_GET_CUR );
@@ -817,6 +835,12 @@ int mlx5dr_cmd_query_caps(struct ibv_context *ctx, capability.flow_table_nic_cap. flow_table_properties_nic_receive.reparse); +/* check cross-VHCA support in flow table properties */ + res = +MLX5_GET (query_hca_cap_out, out, + capability.flow_table_nic_cap.flow_table_properties_nic_receive.cross_vhca_object); + caps->cross_vhca_resources &= res; +if (caps->wqe_based_update) { MLX5_SET(query_hca_cap_in, in, op_mod, MLX5_GET_HCA_CAP_OP_MOD_WQE_BASED_FLOW_TABLE |
drivers/net/mlx5/hws/mlx5dr_cmd.h +1 -0
@@ -163,6 +163,7 @@ struct mlx5dr_cmd_query_caps {uint8_t sq_ts_format;uint64_t definer_format_sup;uint32_t trivial_match_definer; +bool cross_vhca_resources;char fw_ver[64 ]; };
Signature · 1 line
2.18.2
RE: [PATCH V1 05/11] net/mlx5/hws: read cross-vhca capabilities
Quoted 9 lines
-----Original Message-----
From: Erez Shitrit <[email protected]>
Sent: Monday, December 19, 2022 10:24 AM
To: Erez Shitrit <[email protected]>; Alex Vesker <[email protected]>; Slava Ovsiienko <[email protected]>; NBU-Contact-Thomas Monjalon (EXTERNAL) <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]; Yevgeny Kliteynik <[email protected]>
Subject: [PATCH V1 05/11]net/mlx5/hws: read cross-vhca capabilities
Quoted 7 lines
From: Yevgeny Kliteynik <[email protected]>
And keep them for future processing.
Signed-off-by Yevgeny Kliteynik <[email protected]>Reviewed-by Alex Vesker <[email protected]>
PATCH 6/11 net/mlx5/hws: added allow-other-vhca-access command
In reply to Erez ShitritFrom: Yevgeny Kliteynik <[email protected]>
Added FW command to allow creation of alias objects.
--- drivers/common/mlx5/mlx5_prm.h | 23 +++++++++++++++++++++++ drivers/net/mlx5/hws/mlx5dr_cmd.c | 28 ++++++++++++++++++++++++++++ drivers/net/mlx5/hws/mlx5dr_cmd.h | 9 +++++++++ 3 files changed, 60 insertions(+)
drivers/common/mlx5/mlx5_prm.h +23 -0
@@ -1133,6 +1133,7 @@ enum {MLX5_CMD_SET_REGEX_REGISTERS =0xb06 ,MLX5_CMD_QUERY_REGEX_REGISTERS =0xb07 ,MLX5_CMD_OP_ACCESS_REGISTER_USER =0xb0c , + MLX5_CMD_OP_ALLOW_OTHER_VHCA_ACCESS = 0xb16, }; enum {
@@ -3150,6 +3151,28 @@ struct mlx5_ifc_general_obj_out_cmd_hdr_bits {u8 reserved_at_60[0x20 ]; }; +struct mlx5_ifc_allow_other_vhca_access_in_bits { +u8 opcode[0x10 ]; +u8 uid[0x10 ]; +u8 reserved_at_20[0x10 ]; +u8 op_mod[0x10 ]; +u8 reserved_at_40[0x50 ]; +u8 object_type_to_be_accessed[0x10 ]; +u8 object_id_to_be_accessed[0x20 ]; +u8 reserved_at_c0[0x40 ]; +union { +u8 access_key_raw[0x100 ]; +u8 access_key[8 ][0x20 ]; + }; +}; + +struct mlx5_ifc_allow_other_vhca_access_out_bits { +u8 status[0x8 ]; +u8 reserved_at_8[0x18 ]; +u8 syndrome[0x20 ]; +u8 reserved_at_40[0x40 ]; +}; +struct mlx5_ifc_virtio_q_counters_bits {u8 modify_field_select[0x40 ];u8 reserved_at_40[0x40 ];
drivers/net/mlx5/hws/mlx5dr_cmd.c +28 -0
@@ -716,6 +716,34 @@ int mlx5dr_cmd_sq_modify_rdy(struct mlx5dr_devx_obj *devx_obj)return ret; } +int mlx5dr_cmd_allow_other_vhca_access (struct ibv_context *ctx, +struct mlx5dr_cmd_allow_other_vhca_access_attr *attr) +{ +uint32_t out[MLX5_ST_SZ_DW (allow_other_vhca_access_out)] = {0 }; +uint32_t in[MLX5_ST_SZ_DW (allow_other_vhca_access_in)] = {0 }; +void *key; +int ret; + +MLX5_SET (allow_other_vhca_access_in, + in, opcode,MLX5_CMD_OP_ALLOW_OTHER_VHCA_ACCESS ); +MLX5_SET (allow_other_vhca_access_in, + in, object_type_to_be_accessed, attr->obj_type); +MLX5_SET (allow_other_vhca_access_in, + in, object_id_to_be_accessed, attr->obj_id); + + key =MLX5_ADDR_OF (allow_other_vhca_access_in, in, access_key); +memcpy (key, attr->access_key,sizeof (attr->access_key)); + + ret = mlx5_glue->devx_general_cmd (ctx, in,sizeof (in), out,sizeof (out)); +if (ret) { +DR_LOG (ERR ,"Failed to execute ALLOW_OTHER_VHCA_ACCESS command" ); + rte_errno = errno; +return rte_errno; + } + +return 0 ; +} +int mlx5dr_cmd_query_caps (struct ibv_context *ctx,struct mlx5dr_cmd_query_caps *caps) {
drivers/net/mlx5/hws/mlx5dr_cmd.h +9 -0
@@ -124,6 +124,12 @@ struct mlx5dr_cmd_sq_create_attr {uint32_t ts_format; }; +struct mlx5dr_cmd_allow_other_vhca_access_attr { +uint16_t obj_type; +uint32_t obj_id; +uint8_t access_key[32 ]; +}; +struct mlx5dr_cmd_query_ft_caps {uint8_t max_level;uint8_t reparse;
@@ -230,4 +236,7 @@ void mlx5dr_cmd_set_attr_connect_miss_tbl(struct mlx5dr_context *ctx,uint32_t fw_ft_type, enum mlx5dr_table_type type, struct mlx5dr_cmd_ft_modify_attr *ft_attr); + +int mlx5dr_cmd_allow_other_vhca_access (struct ibv_context *ctx, +struct mlx5dr_cmd_allow_other_vhca_access_attr *attr);#endif /* MLX5DR_CMD_H_ */
Signature · 1 line
2.18.2
RE: [PATCH V1 06/11] net/mlx5/hws: added allow-other-vhca-access command
Quoted 10 lines
-----Original Message-----
From: Erez Shitrit <[email protected]>
Sent: Monday, December 19, 2022 10:24 AM
To: Erez Shitrit <[email protected]>; Alex Vesker <[email protected]>; Slava Ovsiienko <[email protected]>; NBU-Contact-Thomas Monjalon (EXTERNAL) <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]; Yevgeny Kliteynik <[email protected]>
Subject: [PATCH V1 06/11]net/mlx5/hws: added allow-other-vhca-access command
Quoted 7 lines
From: Yevgeny Kliteynik <[email protected]>
Added FW command to allow creation of alias objects.
Signed-off-by Yevgeny Kliteynik <[email protected]>Reviewed-by Alex Vesker <[email protected]>
PATCH 7/11 net/mlx5/hws: added command to create alias objects
In reply to Erez ShitritFrom: Yevgeny Kliteynik <[email protected]>
Added support for a command that can create alias objects - objects that are accessed across VHCA.
--- drivers/common/mlx5/mlx5_prm.h | 23 ++++++++++++++-- drivers/net/mlx5/hws/mlx5dr_cmd.c | 44 +++++++++++++++++++++++++++++++ drivers/net/mlx5/hws/mlx5dr_cmd.h | 11 ++++++++ 3 files changed, 76 insertions(+), 2 deletions(-)
drivers/common/mlx5/mlx5_prm.h +21 -2
@@ -3135,9 +3135,10 @@ struct mlx5_ifc_general_obj_in_cmd_hdr_bits {u8 obj_id[0x20 ];union {struct { -u8 reserved_at_60[0x3 ]; +u8 alias_object[0x1 ]; +u8 reserved_at_61[0x2 ];u8 log_obj_range[0x5 ]; -u8 reserved_at_58[0x18 ]; +u8 reserved_at_68[0x18 ]; };u8 obj_offset[0x20 ]; };
@@ -3234,6 +3235,19 @@ struct mlx5_ifc_rtc_bits {u8 reserved_at_180[0x280 ]; }; +struct mlx5_ifc_alias_context_bits { +u8 vhca_id_to_be_accessed[0x10 ]; +u8 reserved_at_10[0xd ]; +u8 status[0x3 ]; +u8 object_id_to_be_accessed[0x20 ]; +u8 reserved_at_40[0x40 ]; +union { +u8 access_key_raw[0x100 ]; +u8 access_key[8 ][0x20 ]; + }; +u8 metadata[0x80 ]; +}; +enum mlx5_ifc_stc_action_type {MLX5_IFC_STC_ACTION_TYPE_NOP =0x00 ,MLX5_IFC_STC_ACTION_TYPE_COPY =0x05 ,
@@ -3478,6 +3492,11 @@ struct mlx5_ifc_create_header_modify_pattern_in_bits {struct mlx5_ifc_header_modify_pattern_in_bits pattern; }; +struct mlx5_ifc_create_alias_obj_in_bits { +struct mlx5_ifc_general_obj_in_cmd_hdr_bits hdr; +struct mlx5_ifc_alias_context_bits alias_ctx; +}; +enum { MLX5_CRYPTO_KEY_SIZE_128b =0x0 , MLX5_CRYPTO_KEY_SIZE_256b =0x1 ,
drivers/net/mlx5/hws/mlx5dr_cmd.c +44 -0
50 lines not on this page · raw message
drivers/net/mlx5/hws/mlx5dr_cmd.h +11 -0
@@ -46,6 +46,13 @@ struct mlx5dr_cmd_rtc_create_attr {bool is_jumbo; }; +struct mlx5dr_cmd_alias_obj_create_attr { +uint32_t obj_id; +uint16_t vhca_id; +uint16_t obj_type; +uint8_t access_key[32 ]; +}; +struct mlx5dr_cmd_stc_create_attr {uint8_t log_obj_range;uint8_t table_type;
@@ -217,6 +224,10 @@ mlx5dr_cmd_header_modify_pattern_create(struct ibv_context *ctx,uint32_t pattern_length, uint8_t *actions); +struct mlx5dr_devx_obj * +mlx5dr_cmd_alias_obj_create (struct ibv_context *ctx, +struct mlx5dr_cmd_alias_obj_create_attr *alias_attr); +int mlx5dr_cmd_sq_modify_rdy (struct mlx5dr_devx_obj *devx_obj); int mlx5dr_cmd_query_ib_port(struct ibv_context *ctx,
Signature · 1 line
2.18.2
RE: [PATCH V1 07/11] net/mlx5/hws: added command to create alias objects
Quoted 10 lines
-----Original Message-----
From: Erez Shitrit <[email protected]>
Sent: Monday, December 19, 2022 10:24 AM
To: Erez Shitrit <[email protected]>; Alex Vesker <[email protected]>; Slava Ovsiienko <[email protected]>; NBU-Contact-Thomas Monjalon (EXTERNAL) <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]; Yevgeny Kliteynik <[email protected]>
Subject: [PATCH V1 07/11]net/mlx5/hws: added command to create alias objects
Quoted 8 lines
From: Yevgeny Kliteynik <[email protected]>
Added support for a command that can create alias objects - objects that are accessed across VHCA.
Signed-off-by Yevgeny Kliteynik <[email protected]>Reviewed-by Alex Vesker <[email protected]>
PATCH 8/11 net/mlx5/hws: add vhca identifier ID to the caps
In reply to Erez ShitritAnd read it in the query_cap function
--- drivers/net/mlx5/hws/mlx5dr_cmd.c | 3 +++ drivers/net/mlx5/hws/mlx5dr_cmd.h | 1 + 2 files changed, 4 insertions(+)
drivers/net/mlx5/hws/mlx5dr_cmd.c +3 -0
@@ -837,6 +837,9 @@ int mlx5dr_cmd_query_caps(struct ibv_context *ctx,MLX5_GET64 (query_hca_cap_out, out, capability.cmd_hca_cap.match_definer_format_supported); + caps->vhca_id =MLX5_GET (query_hca_cap_out, out, + capability.cmd_hca_cap.vhca_id); + caps->sq_ts_format =MLX5_GET (query_hca_cap_out, out, capability.cmd_hca_cap.sq_ts_format);
drivers/net/mlx5/hws/mlx5dr_cmd.h +1 -0
@@ -176,6 +176,7 @@ struct mlx5dr_cmd_query_caps {uint8_t sq_ts_format;uint64_t definer_format_sup;uint32_t trivial_match_definer; +uint32_t vhca_id;bool cross_vhca_resources;char fw_ver[64 ]; };
Signature · 1 line
2.18.2
RE: [PATCH V1 08/11] net/mlx5/hws: add vhca identifier ID to the caps
Quoted 10 lines
-----Original Message-----
From: Erez Shitrit <[email protected]>
Sent: Monday, December 19, 2022 10:24 AM
To: Erez Shitrit <[email protected]>; Alex Vesker <[email protected]>; Slava Ovsiienko <[email protected]>; NBU-Contact-Thomas Monjalon (EXTERNAL) <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]
Subject: [PATCH V1 08/11]net/mlx5/hws: add vhca identifier ID to the caps
Quoted 4 lines
And read it in the
query_capfunctionSigned-off-by Erez Shitrit <[email protected]>Reviewed-by Alex Vesker <[email protected]>
PATCH 9/11 net/mlx5/hws: support shared ibv-context with local one
In reply to Erez ShitritThe idea is to have a shared ibv_context that all the resources allocated on it (FT + TIR are exceptions)
When ever a resource created locally an alias object to that resource allocated and it used in the other context.
The connections between the resources are done according to each type of the resource, to the original resource or to its alias resource.
--- drivers/common/mlx5/mlx5_prm.h | 6 + drivers/net/mlx5/hws/mlx5dr.h | 2 + drivers/net/mlx5/hws/mlx5dr_action.c | 9 +- drivers/net/mlx5/hws/mlx5dr_cmd.c | 19 ++- drivers/net/mlx5/hws/mlx5dr_cmd.h | 7 +- drivers/net/mlx5/hws/mlx5dr_context.c | 34 +++- drivers/net/mlx5/hws/mlx5dr_context.h | 22 +++ drivers/net/mlx5/hws/mlx5dr_debug.c | 2 +- drivers/net/mlx5/hws/mlx5dr_internal.h | 1 + drivers/net/mlx5/hws/mlx5dr_matcher.c | 218 +++++++++++++++++++++++-- drivers/net/mlx5/hws/mlx5dr_matcher.h | 9 + drivers/net/mlx5/hws/mlx5dr_table.c | 191 +++++++++++++++++++++- drivers/net/mlx5/hws/mlx5dr_table.h | 4 +- 13 files changed, 484 insertions(+), 40 deletions(-)
drivers/common/mlx5/mlx5_prm.h +6 -0
@@ -291,6 +291,12 @@#define MAX_ACTIONS_DATA_IN_HEADER_MODIFY 512 +/* Alias FT id passed to the ALLOW_OTHER_VHCA_ACCESS & CREATE_GENERAL_OBJECT +* commands should have the following format: +* {table_type: 8bits, table_id: 24bits}. +*/ +#define FT_ID_FT_TYPE_OFFSET 24 +/* Completion mode. */ enum mlx5_completion_mode {MLX5_COMP_ONLY_ERR =0x0 ,
drivers/net/mlx5/hws/mlx5dr.h +2 -0
@@ -96,6 +96,8 @@ struct mlx5dr_context_attr {size_t initial_log_ste_memory;/* Currently not in use */ /* Optional PD used for allocating res ources */ struct ibv_pd *pd; +/* Optional other ctx for resources allocation, all objects will be created on it */ +struct ibv_context *shared_ibv_ctx; };struct mlx5dr_table_attr {
drivers/net/mlx5/hws/mlx5dr_action.c +7 -2
@@ -1147,6 +1147,7 @@ mlx5dr_action_create_reformat_root(struct mlx5dr_action *action, {enum mlx5dv_flow_table_type ft_type =0 ;/*fix compilation warn*/ uint32_t verb_reformat_type =0 ; +struct ibv_context *ibv_ctx;int ret;/* Convert action to FT type and verbs reformat type */
@@ -1157,8 +1158,9 @@ mlx5dr_action_create_reformat_root(struct mlx5dr_action *action,mlx5dr_action_conv_reformat_to_verbs (action->type, &verb_reformat_type);/* Create the reformat type for root table */ + ibv_ctx =mlx5dr_context_get_local_ibv (action->ctx); action->flow_action = - mlx5_glue->dv_create_flow_action_packet_reformat_root(action->ctx->ibv_ctx, + mlx5_glue->dv_create_flow_action_packet_reformat_root(ibv_ctx, data_sz, data, verb_reformat_type,
@@ -1496,14 +1498,17 @@ mlx5dr_action_create_modify_header_root(struct mlx5dr_action *action, __be64 *actions) {enum mlx5dv_flow_table_type ft_type =0 ; +struct ibv_context *local_ibv_ctx;int ret; ret =mlx5dr_action_conv_flags_to_ft_type (action->flags, &ft_type);if (ret)return rte_errno; + local_ibv_ctx =mlx5dr_context_get_local_ibv (action->ctx); + action->flow_action = - mlx5_glue->dv_create_flow_action_modify_header_root (action->ctx->ibv_ctx, + mlx5_glue->dv_create_flow_action_modify_header_root (local_ibv_ctx, actions_sz, (uint64_t *)actions, ft_type);
drivers/net/mlx5/hws/mlx5dr_cmd.c +12 -7
@@ -217,18 +217,23 @@ void mlx5dr_cmd_set_attr_connect_miss_tbl(struct mlx5dr_context *ctx, {struct mlx5dr_devx_obj *default_miss_tbl; -if (type !=MLX5DR_TABLE_TYPE_FDB ) +if (type !=MLX5DR_TABLE_TYPE_FDB && !mlx5dr_context_shared_gvmi_used (ctx))return ; - default_miss_tbl = ctx->common_res[type].default_miss->ft; -if (!default_miss_tbl) { -assert (false); -return ; - } ft_attr->modify_fs =MLX5_IFC_MODIFY_FLOW_TABLE_MISS_ACTION ; ft_attr->type = fw_ft_type; ft_attr->table_miss_action =MLX5_IFC_MODIFY_FLOW_TABLE_MISS_ACTION_GOTO_TBL ; - ft_attr->table_miss_id = default_miss_tbl->id; + +if (type ==MLX5DR_TABLE_TYPE_FDB ) { + default_miss_tbl = ctx->common_res[type].default_miss->ft; +if (!default_miss_tbl) { +assert (false); +return ; + } + ft_attr->table_miss_id = default_miss_tbl->id; + }else { + ft_attr->table_miss_id = ctx->gvmi_res[type].aliased_end_ft->id; + } } struct mlx5dr_devx_obj *
drivers/net/mlx5/hws/mlx5dr_cmd.h +5 -2
@@ -11,6 +11,8 @@ struct mlx5dr_cmd_ft_create_attr {bool rtc_valid; }; +#define ACCESS_KEY_LEN 32 +struct mlx5dr_cmd_ft_modify_attr {uint8_t type;uint32_t rtc_id_0;
@@ -50,7 +52,7 @@ struct mlx5dr_cmd_alias_obj_create_attr {uint32_t obj_id;uint16_t vhca_id;uint16_t obj_type; -uint8_t access_key[32 ]; +uint8_t access_key[ACCESS_KEY_LEN ]; };struct mlx5dr_cmd_stc_create_attr {
@@ -134,7 +136,7 @@ struct mlx5dr_cmd_sq_create_attr {struct mlx5dr_cmd_allow_other_vhca_access_attr {uint16_t obj_type;uint32_t obj_id; -uint8_t access_key[32 ]; +uint8_t access_key[ACCESS_KEY_LEN ]; };struct mlx5dr_cmd_query_ft_caps {
@@ -178,6 +180,7 @@ struct mlx5dr_cmd_query_caps {uint32_t trivial_match_definer;uint32_t vhca_id;bool cross_vhca_resources; +uint32_t shared_vhca_id;char fw_ver[64 ]; };
drivers/net/mlx5/hws/mlx5dr_context.c +33 -1
@@ -178,6 +178,36 @@ static void mlx5dr_context_uninit_hws(struct mlx5dr_context *ctx)mlx5dr_context_uninit_pd (ctx); } +static int mlx5dr_context_init_shared_ctx (struct mlx5dr_context *ctx, +struct ibv_context *ibv_ctx, +struct mlx5dr_context_attr *attr) +{ +struct mlx5dr_cmd_query_caps shared_caps = {0 }; +int ret; + +if (!attr->shared_ibv_ctx) { + ctx->ibv_ctx = ibv_ctx; + }else { + ctx->ibv_ctx = attr->shared_ibv_ctx; + ctx->local_ibv_ctx = ibv_ctx; + ret =mlx5dr_cmd_query_caps (attr->shared_ibv_ctx, &shared_caps); +if (ret || !shared_caps.cross_vhca_resources) { +DR_LOG (INFO ,"No cross_vhca_resources cap for shared ibv" ); + rte_errno =ENOTSUP ; +return rte_errno; + } + ctx->caps->shared_vhca_id = shared_caps.vhca_id; + } + +if (ctx->local_ibv_ctx && !ctx->caps->cross_vhca_resources) { +DR_LOG (INFO ,"No cross_vhca_resources cap for local ibv" ); + rte_errno =ENOTSUP ; +return rte_errno; + } + +return 0 ; +} +struct mlx5dr_context *mlx5dr_context_open (struct ibv_context *ibv_ctx,struct mlx5dr_context_attr *attr) {
@@ -190,7 +220,6 @@ struct mlx5dr_context *mlx5dr_context_open(struct ibv_context *ibv_ctx,return NULL ; } - ctx->ibv_ctx = ibv_ctx;pthread_spin_init (&ctx->ctrl_lock,PTHREAD_PROCESS_PRIVATE ); ctx->caps =simple_calloc (1 ,sizeof (*ctx->caps));
@@ -201,6 +230,9 @@ struct mlx5dr_context *mlx5dr_context_open(struct ibv_context *ibv_ctx,if (ret) goto free_caps; +if (mlx5dr_context_init_shared_ctx (ctx, ibv_ctx, attr)) + goto free_caps; + ret =mlx5dr_context_init_hws (ctx, attr);if (ret) goto free_caps;
drivers/net/mlx5/hws/mlx5dr_context.h +22 -0
@@ -22,13 +22,22 @@ struct mlx5dr_context_common_res {struct mlx5dr_cmd_forward_tbl *default_miss; }; +struct mlx5dr_context_shared_gvmi_res { +struct mlx5dr_devx_obj *end_ft; +struct mlx5dr_devx_obj *aliased_end_ft; +uint32_t refcount; +}; +struct mlx5dr_context {struct ibv_context *ibv_ctx; +/* When local_ibv_ctx is not NULL means we are using shared_ibv for resources */ +struct ibv_context *local_ibv_ctx;struct mlx5dr_cmd_query_caps *caps;struct ibv_pd *pd;uint32_t pd_num;struct mlx5dr_pool *stc_pool[MLX5DR_TABLE_TYPE_MAX ];struct mlx5dr_context_common_res common_res[MLX5DR_TABLE_TYPE_MAX ]; +struct mlx5dr_context_shared_gvmi_res gvmi_res[MLX5DR_TABLE_TYPE_MAX ];struct mlx5dr_pattern_cache *pattern_cache;pthread_spinlock_t ctrl_lock;enum mlx5dr_context_flags flags;
@@ -37,4 +46,17 @@ struct mlx5dr_context { LIST_HEAD(table_head, mlx5dr_table) head; }; +static inline bool mlx5dr_context_shared_gvmi_used (struct mlx5dr_context *ctx) +{ +return ctx->local_ibv_ctx ? true : false; +} + +static inline struct ibv_context * +mlx5dr_context_get_local_ibv (struct mlx5dr_context *ctx) +{ +if (mlx5dr_context_shared_gvmi_used (ctx)) +return ctx->local_ibv_ctx; + +return ctx->ibv_ctx; +}#endif /* MLX5DR_CONTEXT_H_ */
drivers/net/mlx5/hws/mlx5dr_debug.c +1 -1
@@ -405,7 +405,7 @@ static int mlx5dr_debug_dump_context_info(FILE *f, struct mlx5dr_context *ctx)MLX5DR_DEBUG_RES_TYPE_CONTEXT , (uint64_t )(uintptr_t )ctx, ctx->flags &MLX5DR_CONTEXT_FLAG_HWS_SUPPORT , - mlx5_glue->get_device_name (ctx->ibv_ctx->device), + mlx5_glue->get_device_name (mlx5dr_context_get_local_ibv (ctx)->device), DEBUG_VERSION);if (ret <0 ) { rte_errno =EINVAL ;
drivers/net/mlx5/hws/mlx5dr_internal.h +1 -0
@@ -18,6 +18,7 @@#endif #include <rte_flow.h> #include <rte_gtp.h> +#include <rte_random.h> #include "mlx5_prm.h" #include "mlx5_glue.h"
drivers/net/mlx5/hws/mlx5dr_matcher.c +201 -17
289 lines not on this page · raw message
drivers/net/mlx5/hws/mlx5dr_matcher.h +9 -0
@@ -42,6 +42,8 @@ struct mlx5dr_matcher_match_ste {struct mlx5dr_devx_obj *rtc_0;struct mlx5dr_devx_obj *rtc_1;struct mlx5dr_pool *pool; +/* Currently not support FDB aliased */ +struct mlx5dr_devx_obj *aliased_rtc_0; };struct mlx5dr_matcher_action_ste {
@@ -73,4 +75,11 @@ int mlx5dr_matcher_conv_items_to_prm(uint64_t *match_buf,uint8_t *match_criteria, bool is_value); +int mlx5dr_matcher_create_aliased_obj (struct mlx5dr_context *ctx, +struct ibv_context *ibv_owner, +struct ibv_context *ibv_allowed, +uint16_t vhca_id_to_be_accessed, +uint32_t aliased_object_id, +uint16_t object_type, +struct mlx5dr_devx_obj **obj);#endif /* MLX5DR_MATCHER_H_ */
drivers/net/mlx5/hws/mlx5dr_table.c +182 -9
243 lines not on this page · raw message
drivers/net/mlx5/hws/mlx5dr_table.h +3 -1
@@ -10,6 +10,7 @@struct mlx5dr_table {struct mlx5dr_context *ctx;struct mlx5dr_devx_obj *ft; +struct mlx5dr_devx_obj *local_ft;enum mlx5dr_table_type type;uint32_t fw_ft_type;uint32_t level;
@@ -37,7 +38,8 @@ static inline bool mlx5dr_table_is_root(struct mlx5dr_table *tbl)return (tbl->level ==MLX5DR_ROOT_LEVEL ); } -struct mlx5dr_devx_obj *mlx5dr_table_create_default_ft (struct mlx5dr_table *tbl); +struct mlx5dr_devx_obj *mlx5dr_table_create_default_ft (struct ibv_context *ibv, +struct mlx5dr_table *tbl);void mlx5dr_table_destroy_default_ft (struct mlx5dr_table *tbl,struct mlx5dr_devx_obj *ft_obj);
Signature · 1 line
2.18.2
RE: [PATCH V1 09/11] net/mlx5/hws: support shared ibv-context with local one
Quoted 11 lines
-----Original Message-----
From: Erez Shitrit <[email protected]>
Sent: Monday, December 19, 2022 10:24 AM
To: Erez Shitrit <[email protected]>; Alex Vesker <[email protected]>; Slava Ovsiienko <[email protected]>; NBU-Contact-Thomas Monjalon (EXTERNAL) <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]
Subject: [PATCH V1 09/11]net/mlx5/hws: support shared ibv-context with local one
Quoted 8 lines
The idea is to have a shared
ibv_contextthat all the resources allocated on it (FT + TIR are exceptions) When ever a resource created locally an alias object to that resource allocated and it used in the other context.
The connections between the resources are done according to each type of the resource, to the original resource or to its alias resource.Signed-off-by Erez Shitrit <[email protected]>Reviewed-by Alex Vesker <[email protected]>
PATCH 10/11 net/mlx5/hws: support actions while shared resources is used
In reply to Erez ShitritTIR/FT actions are different in the context of shared ibv resource, it created on the local ibv_context and aliased to the shared ibv_context. Other actions should be created on the shared ibv resource, new flag was added to indicates where this object came from.
--- drivers/net/mlx5/hws/mlx5dr.h | 5 ++- drivers/net/mlx5/hws/mlx5dr_action.c | 60 ++++++++++++++++++++++++++-- drivers/net/mlx5/hws/mlx5dr_action.h | 3 ++ drivers/net/mlx5/mlx5_devx.c | 2 +- 4 files changed, 64 insertions(+), 6 deletions(-)
drivers/net/mlx5/hws/mlx5dr.h +4 -1
@@ -410,12 +410,15 @@ mlx5dr_action_create_dest_vport(struct mlx5dr_context *ctx,* Direct rule TIR devx object. * @param[in] flags * Action creation flags. (enum mlx5dr_action_flags) +* @param[in] is_local +* indicates where the tir object was created, local gvmi or other gvmi * @return pointer to mlx5dr_action on success NULL otherwise. */ struct mlx5dr_action *mlx5dr_action_create_dest_tir (struct mlx5dr_context *ctx,struct mlx5dr_devx_obj *obj, -uint32_t flags); +uint32_t flags, +bool is_local);/* Create direct rule TAG action. *
drivers/net/mlx5/hws/mlx5dr_action.c +56 -4
98 lines not on this page · raw message
drivers/net/mlx5/hws/mlx5dr_action.h +3 -0
@@ -138,6 +138,9 @@ struct mlx5dr_action {uint16_t vport_num;uint16_t esw_owner_vhca_id; } vport; +struct { +struct mlx5dr_devx_obj *devx_obj; + } alias; }; };
drivers/net/mlx5/mlx5_devx.c +1 -1
@@ -913,7 +913,7 @@ mlx5_devx_hrxq_new(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq,if (hrxq->hws_flags) { hrxq->action =mlx5dr_action_create_dest_tir (priv->dr_ctx, - (struct mlx5dr_devx_obj *)hrxq->tir, hrxq->hws_flags); + (struct mlx5dr_devx_obj *)hrxq->tir, hrxq->hws_flags, true);if (!hrxq->action) goto error;return 0 ;
Signature · 1 line
2.18.2
RE: [PATCH V1 10/11] net/mlx5/hws: support actions while shared resources is used
Quoted 11 lines
-----Original Message-----
From: Erez Shitrit <[email protected]>
Sent: Monday, December 19, 2022 10:24 AM
To: Erez Shitrit <[email protected]>; Alex Vesker <[email protected]>; Slava Ovsiienko <[email protected]>; NBU-Contact-Thomas Monjalon (EXTERNAL) <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]
Subject: [PATCH V1 10/11]net/mlx5/hws: support actions while shared resources is used
Quoted 7 lines
TIR/FT actions are different in the context of shared ibv resource, it created on the local
ibv_contextand aliased to the sharedibv_context. Other actions should be created on the shared ibv resource, new flag was added to indicates where this object came from.Signed-off-by Erez Shitrit <[email protected]>Reviewed-by Alex Vesker <[email protected]>
PATCH 11/11 net/mlx5/hws: add debug details for cross gvmi
In reply to Erez ShitritFor context, table and matcher.
--- drivers/net/mlx5/hws/mlx5dr_debug.c | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-)
drivers/net/mlx5/hws/mlx5dr_debug.c +16 -6
@@ -177,6 +177,7 @@ mlx5dr_debug_dump_matcher_attr(FILE *f, struct mlx5dr_matcher *matcher)static int mlx5dr_debug_dump_matcher (FILE *f,struct mlx5dr_matcher *matcher) { +bool is_shared =mlx5dr_context_shared_gvmi_used (matcher->tbl->ctx);bool is_root = matcher->tbl->level ==MLX5DR_ROOT_LEVEL ;enum mlx5dr_table_type tbl_type = matcher->tbl->type;struct mlx5dr_devx_obj *ste_0, *ste_1 =NULL ;
@@ -205,11 +206,13 @@ static int mlx5dr_debug_dump_matcher(FILE *f, struct mlx5dr_matcher *matcher) ste_1 =NULL ; } - ret =fprintf (f,",%d,%d,%d,%d" , + ret =fprintf (f,",%d,%d,%d,%d,%d" , matcher->match_ste.rtc_0 ? matcher->match_ste.rtc_0->id :0 , ste_0 ? (int )ste_0->id :-1 , matcher->match_ste.rtc_1 ? matcher->match_ste.rtc_1->id :0 , - ste_1 ? (int )ste_1->id :-1 ); + ste_1 ? (int )ste_1->id :-1 , + is_shared && !is_root ? + matcher->match_ste.aliased_rtc_0->id :0 );if (ret <0 ) goto out_err;
@@ -253,18 +256,20 @@ static int mlx5dr_debug_dump_matcher(FILE *f, struct mlx5dr_matcher *matcher)static int mlx5dr_debug_dump_table (FILE *f,struct mlx5dr_table *tbl) { +bool is_shared =mlx5dr_context_shared_gvmi_used (tbl->ctx);bool is_root = tbl->level ==MLX5DR_ROOT_LEVEL ;struct mlx5dr_matcher *matcher;int ret; - ret =fprintf (f,"%d,0x%" PRIx64",0x%" PRIx64",%d,%d,%d,%d\n" , + ret =fprintf (f,"%d,0x%" PRIx64",0x%" PRIx64",%d,%d,%d,%d,%d\n" ,MLX5DR_DEBUG_RES_TYPE_TABLE , (uint64_t )(uintptr_t )tbl, (uint64_t )(uintptr_t )tbl->ctx, is_root ?0 : tbl->ft->id, tbl->type, is_root ?0 : tbl->fw_ft_type, - tbl->level); + tbl->level, + is_shared && !is_root ? tbl->local_ft->id :0 );if (ret <0 ) { rte_errno =EINVAL ;return rte_errno;
@@ -383,12 +388,17 @@ static int mlx5dr_debug_dump_context_attr(FILE *f, struct mlx5dr_context *ctx) {int ret; - ret =fprintf (f,"%u,0x%" PRIx64",%d,%zu,%d\n" , + ret =fprintf (f,"%u,0x%" PRIx64",%d,%zu,%d,%s,%d,%d\n" ,MLX5DR_DEBUG_RES_TYPE_CONTEXT_ATTR , (uint64_t )(uintptr_t )ctx, ctx->pd_num, ctx->queues, - ctx->send_queue->num_entries); + ctx->send_queue->num_entries, +mlx5dr_context_shared_gvmi_used (ctx) ? + mlx5_glue->get_device_name (ctx->ibv_ctx->device) :"None" , + ctx->caps->vhca_id, +mlx5dr_context_shared_gvmi_used (ctx) ? + ctx->caps->shared_vhca_id :0xffff );if (ret <0 ) { rte_errno =EINVAL ;return rte_errno;
Signature · 1 line
2.18.2
RE: [PATCH V1 11/11] net/mlx5/hws: add debug details for cross gvmi
Quoted 10 lines
-----Original Message-----
From: Erez Shitrit <[email protected]>
Sent: Monday, December 19, 2022 10:24 AM
To: Erez Shitrit <[email protected]>; Alex Vesker <[email protected]>; Slava Ovsiienko <[email protected]>; NBU-Contact-Thomas Monjalon (EXTERNAL) <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]; Hamdan Igbaria <[email protected]>
Subject: [PATCH V1 11/11]net/mlx5/hws: add debug details for cross gvmi
Quoted 5 lines
For context, table and matcher.
Signed-off-by Erez Shitrit <[email protected]>Signed-off-by Hamdan Igbaria <[email protected]>Reviewed-by Alex Vesker <[email protected]>