Skip to contentSkip to conversation

net/mlx5: HWS misc enhancements

PATCH Superseded Tariq Toukan 2 participants 8 messages Aug 10 – Aug 12

Subsystems (4)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Tariq Toukan

author maintainer
#0 lore raw

v2 cover net/mlx5: HWS misc enhancements

Hi,

This series by Yevgeny brings several misc improvements to the mlx5 HWS driver.

Regards,
Tariq

V2:
- Rebased.

V1:
https://lore.kernel.org/all/[email protected]/

Yevgeny Kliteynik (5):
net/mlx5: HWS, Print more details for bad completion net/mlx5: HWS, Log syndrome on STC modify failure net/mlx5: HWS, Set the num of queues only when alloc succeeded net/mlx5: HWS, Remove redundant MLX5_SET in RTC creation net/mlx5: HWS, Remove redundant FW command when reading caps

 .../mellanox/mlx5/core/steering/hws/cmd.c     | 15 ++--
 .../mellanox/mlx5/core/steering/hws/send.c    | 72 +++++++++++++------
 2 files changed, 56 insertions(+), 31 deletions(-)

base-commit: 001b5d347d8ba39b2dccaefcc57967b18caec8fe

Signature · 1 line
2.44.0
author maintainer
#1 lore raw

PATCH v2 1/5 net/mlx5: HWS, Print more details for bad completion

From: Yevgeny Kliteynik <[email protected]>

When polling for completion returned completion with error, parse some more details: QP number and WQE count. Also, extract all the long value-to-string if conditions to a short value-to-string functions: do it for rule resize state, rule status, and syndrome.

---
 .../mellanox/mlx5/core/steering/hws/send.c    | 68 +++++++++++++------
 1 file changed, 49 insertions(+), 19 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/send.c +49 -19
index aed009aec4fe..1c9206f74aaa 100644
@@ -344,6 +344,36 @@ hws_send_engine_update_rule_resize(struct mlx5hws_send_engine *queue,
 	}
 }
 
+static const char *hws_rule_status_to_string(enum mlx5hws_rule_status status)
+{
+	switch (status) {
+	case MLX5HWS_RULE_STATUS_CREATING: return "CREATING";
+	case MLX5HWS_RULE_STATUS_UPDATING: return "UPDATING";
+	case MLX5HWS_RULE_STATUS_DELETING: return "DELETING";
+	case MLX5HWS_RULE_STATUS_FAILING: return "FAILING";
+	default: return "NA";
+	}
+}
+
+static const char *hws_rule_resize_state_to_string(u8 state)
+{
+	switch (state) {
+	case MLX5HWS_RULE_RESIZE_STATE_IDLE: return "IDLE";
+	case MLX5HWS_RULE_RESIZE_STATE_WRITING: return "WRITING";
+	case MLX5HWS_RULE_RESIZE_STATE_DELETING: return "DELETING";
+	default: return "UNKNOWN";
+	}
+}
+
+static const char *hws_gta_syndrome_to_string(u8 syndrome)
+{
+	switch (syndrome) {
+	case 1: return "SET_FLOW_FAIL";
+	case 2: return "DISABLE_FLOW_FAIL";
+	default: return "UNKNOWN";
+	}
+}
+
 static void hws_send_engine_dump_error_cqe(struct mlx5hws_send_engine *queue,
 					   struct mlx5hws_send_ring_priv *priv,
 					   struct mlx5_cqe64 *cqe)
@@ -352,6 +382,7 @@ static void hws_send_engine_dump_error_cqe(struct mlx5hws_send_engine *queue,
 	struct mlx5hws_context *ctx = priv->rule->matcher->tbl->ctx;
 	u32 opcode = cqe ? get_cqe_opcode(cqe) : 0;
 	struct mlx5hws_rule *rule = priv->rule;
+	u8 syndrome;
 
 	/* If something bad happens and lots of rules are failing, we don't
 	 * want to pollute dmesg. Print only the first bad cqe per engine,
@@ -364,26 +395,17 @@ static void hws_send_engine_dump_error_cqe(struct mlx5hws_send_engine *queue,
 
 	if (mlx5hws_rule_move_in_progress(rule))
 		mlx5hws_err(ctx,
-			    "--- rule 0x%08llx: error completion moving rule: phase %s, wqes left %d\n",
+			    "--- rule 0x%08llx: error completion moving rule: phase %s (%d), wqes left %d\n",
 			    HWS_PTR_TO_ID(rule),
-			    rule->resize_info->state ==
-			    MLX5HWS_RULE_RESIZE_STATE_WRITING ? "WRITING" :
-			    rule->resize_info->state ==
-			    MLX5HWS_RULE_RESIZE_STATE_DELETING ? "DELETING" :
-			    "UNKNOWN",
+			    hws_rule_resize_state_to_string
+				(rule->resize_info->state),
+			    rule->resize_info->state,
 			    rule->pending_wqes);
 	else
 		mlx5hws_err(ctx,
 			    "--- rule 0x%08llx: error completion %s (%d), wqes left %d\n",
 			    HWS_PTR_TO_ID(rule),
-			    rule->status ==
-			    MLX5HWS_RULE_STATUS_CREATING ? "CREATING" :
-			    rule->status ==
-			    MLX5HWS_RULE_STATUS_DELETING ? "DELETING" :
-			    rule->status ==
-			    MLX5HWS_RULE_STATUS_FAILING ? "FAILING" :
-			    rule->status ==
-			    MLX5HWS_RULE_STATUS_UPDATING ? "UPDATING" : "NA",
+			    hws_rule_status_to_string(rule->status),
 			    rule->status,
 			    rule->pending_wqes);
 
@@ -423,6 +445,15 @@ static void hws_send_engine_dump_error_cqe(struct mlx5hws_send_engine *queue,
 			    "    rule 0x%08llx:  |--- syndrome = 0x%x\n",
 			    HWS_PTR_TO_ID(rule),
 			    err_cqe->syndrome);
+		mlx5hws_err(ctx,
+			    "    rule 0x%08llx:  |--- QPN = 0x%x\n",
+			    HWS_PTR_TO_ID(rule),
+			    be32_to_cpu(err_cqe->s_wqe_opcode_qpn) &
+					0xffffff);
+		mlx5hws_err(ctx,
+			    "    rule 0x%08llx:  |--- WQE_CNT = 0x%04x\n",
+			    HWS_PTR_TO_ID(rule),
+			    (u32)be16_to_cpu(err_cqe->wqe_counter));
 	}
 
 	mlx5hws_err(ctx,
@@ -433,13 +464,12 @@ static void hws_send_engine_dump_error_cqe(struct mlx5hws_send_engine *queue,
 		    HWS_PTR_TO_ID(rule),
 		    (be32_to_cpu(cqe->byte_cnt) & 0x80000000) ?
 		    "FAILURE" : "SUCCESS");
+	/* syndrome is in the lower 2 bits of byte_cnt */
+	syndrome = be32_to_cpu(cqe->byte_cnt) & 3;
 	mlx5hws_err(ctx,
-		    "    rule 0x%08llx:  |------- SYNDROME = %s\n",
+		    "    rule 0x%08llx:  |------- SYNDROME = %s (%u)\n",
 		    HWS_PTR_TO_ID(rule),
-		    ((be32_to_cpu(cqe->byte_cnt) & 0x00000003) == 1) ?
-		    "SET_FLOW_FAIL" :
-		    ((be32_to_cpu(cqe->byte_cnt) & 0x00000003) == 2) ?
-		    "DISABLE_FLOW_FAIL" : "UNKNOWN");
+		    hws_gta_syndrome_to_string(syndrome), syndrome);
 	mlx5hws_err(ctx,
 		    "    rule 0x%08llx: cqe->sop_drop_qpn  = 0x%08x\n",
 		    HWS_PTR_TO_ID(rule), be32_to_cpu(cqe->sop_drop_qpn));
Signature · 1 line
2.44.0
reviewer
#2 lore raw

Re: [PATCH net-next V2 1/5] net/mlx5: HWS, Print more details for bad completion

Quoted 11 lines · Tariq Toukan

From: Yevgeny Kliteynik <[email protected]>

When polling for completion returned completion with error, parse some more details: QP number and WQE count. Also, extract all the long value-to-string if conditions to a short value-to-string functions: do it for rule resize state, rule status, and syndrome.

Overall this series looks good to me.
But the AI-generated review of this patch, pasted below, does seem relevant.

...

drivers/net/ethernet/mellanox/mlx5/core/steering/hws/send.c static void hws_send_engine_dump_error_cqe(struct mlx5hws_send_engine *queue, · line 452

Quoted 9 lines
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/send.c +5 -0
@@ -423,6 +445,15 @@ static void hws_send_engine_dump_error_cqe(struct mlx5hws_send_engine *queue,
 			    "    rule 0x%08llx:  |--- syndrome = 0x%x\n",
 			    HWS_PTR_TO_ID(rule),
 			    err_cqe->syndrome);
+		mlx5hws_err(ctx,
+			    "    rule 0x%08llx:  |--- QPN = 0x%x\n",
+			    HWS_PTR_TO_ID(rule),
+			    be32_to_cpu(err_cqe->s_wqe_opcode_qpn) &
+					0xffffff);

AI-generated review text:

Is this new QPN line printing something that the same function already dumps a few lines later?

At the tail of hws_send_engine_dump_error_cqe() there is already:

	mlx5hws_err(ctx,
		    "    rule 0x%08llx:  |------------ qpn = 0x%06x\n",
		    HWS_PTR_TO_ID(rule),
		    be32_to_cpu(cqe->sop_drop_qpn) & 0xffffff);

err_cqe is just a cast of the same cqe pointer, and both fields land on the same four bytes of the 64 byte CQE (include/linux/mlx5/device.h):

	struct mlx5_err_cqe {
		u8	rsvd0[32];	/* 0..31  */
		__be32	srqn;		/* 32..35 */
		u8	rsvd1[18];	/* 36..53 */
		u8	vendor_err_synd;/* 54     */
		u8	syndrome;	/* 55     */
		__be32	s_wqe_opcode_qpn;/* 56..59 */
		...
	};
	struct mlx5_cqe64 {
		...
		__be32	byte_cnt;	/* 44..47 */
		__be32	timestamp_h;	/* 48..51 */
		__be32	timestamp_l;	/* 52..55 */
		__be32	sop_drop_qpn;	/* 56..59 */
		...
	};

Same offset, same 0xffffff mask, so a single dump now reports the QP number twice, once as "QPN = 0x%x" and once as "qpn = 0x%06x", with different labels and widths.

Would it make sense to drop the new QPN line, or at least align its label and width with the existing one, given the one-shot queue->error_cqe_printed guard is there to keep this dump compact?

Source: https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20260810092630.3137666-1-tariqt%40nvidia.com

Quoted 7 lines
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/send.c +4 -0
+		mlx5hws_err(ctx,
+			    "    rule 0x%08llx:  |--- WQE_CNT = 0x%04x\n",
+			    HWS_PTR_TO_ID(rule),
+			    (u32)be16_to_cpu(err_cqe->wqe_counter));
 	}
 
 	mlx5hws_err(ctx,

...

author maintainer
#3 lore raw

Re: [PATCH net-next V2 1/5] net/mlx5: HWS, Print more details for bad completion

Quoted 9 lines · Simon Horman
Quoted 8 lines · Tariq Toukan

From: Yevgeny Kliteynik <[email protected]>

When polling for completion returned completion with error, parse some more details: QP number and WQE count. Also, extract all the long value-to-string if conditions to a short value-to-string functions: do it for rule resize state, rule status, and syndrome.

Quoted 7 lines
Quoted 3 lines

Overall this series looks good to me.
But the AI-generated review of this patch, pasted below, does seem relevant.

Thanks.
Yeah it seems relevant indeed.
We'll handle.

Quoted 65 lines

...

drivers/net/ethernet/mellanox/mlx5/core/steering/hws/send.c static void hws_send_engine_dump_error_cqe(struct mlx5hws_send_engine *queue, · line 452

Quoted 9 lines
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/send.c +5 -0
@@ -423,6 +445,15 @@ static void hws_send_engine_dump_error_cqe(struct mlx5hws_send_engine *queue,
  			    "    rule 0x%08llx:  |--- syndrome = 0x%x\n",
  			    HWS_PTR_TO_ID(rule),
  			    err_cqe->syndrome);
+		mlx5hws_err(ctx,
+			    "    rule 0x%08llx:  |--- QPN = 0x%x\n",
+			    HWS_PTR_TO_ID(rule),
+			    be32_to_cpu(err_cqe->s_wqe_opcode_qpn) &
+					0xffffff);

AI-generated review text:

   Is this new QPN line printing something that the same function already
   dumps a few lines later?

   At the tail of hws_send_engine_dump_error_cqe() there is already:

	mlx5hws_err(ctx,
		    "    rule 0x%08llx:  |------------ qpn = 0x%06x\n",
		    HWS_PTR_TO_ID(rule),
		    be32_to_cpu(cqe->sop_drop_qpn) & 0xffffff);

   err_cqe is just a cast of the same cqe pointer, and both fields land on
   the same four bytes of the 64 byte CQE (include/linux/mlx5/device.h):

	struct mlx5_err_cqe {
		u8	rsvd0[32];	/* 0..31  */
		__be32	srqn;		/* 32..35 */
		u8	rsvd1[18];	/* 36..53 */
		u8	vendor_err_synd;/* 54     */
		u8	syndrome;	/* 55     */
		__be32	s_wqe_opcode_qpn;/* 56..59 */
		...
	};
	struct mlx5_cqe64 {
		...
		__be32	byte_cnt;	/* 44..47 */
		__be32	timestamp_h;	/* 48..51 */
		__be32	timestamp_l;	/* 52..55 */
		__be32	sop_drop_qpn;	/* 56..59 */
		...
	};

   Same offset, same 0xffffff mask, so a single dump now reports the QP
   number twice, once as "QPN = 0x%x" and once as "qpn = 0x%06x", with
   different labels and widths.

   Would it make sense to drop the new QPN line, or at least align its label
   and width with the existing one, given the one-shot
   queue->error_cqe_printed guard is there to keep this dump compact?

Source: https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20260810092630.3137666-1-tariqt%40nvidia.com

Quoted 7 lines
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/send.c +4 -0
+		mlx5hws_err(ctx,
+			    "    rule 0x%08llx:  |--- WQE_CNT = 0x%04x\n",
+			    HWS_PTR_TO_ID(rule),
+			    (u32)be16_to_cpu(err_cqe->wqe_counter));
  	}
 
  	mlx5hws_err(ctx,

...

author maintainer
#4 lore raw

PATCH v2 2/5 net/mlx5: HWS, Log syndrome on STC modify failure

In reply to Tariq Toukan

From: Yevgeny Kliteynik <[email protected]>

When mlx5_cmd_exec fails for STC modify, include the command syndrome from the output buffer in the error message to aid debugging.

---
 .../net/ethernet/mellanox/mlx5/core/steering/hws/cmd.c    | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/cmd.c +5 -3
index 8fae90101653..2cdc03421433 100644
@@ -611,9 +611,11 @@ int mlx5hws_cmd_stc_modify(struct mlx5_core_dev *mdev,
 
 	ret = mlx5_cmd_exec(mdev, in, sizeof(in), out, sizeof(out));
 	if (ret)
-		mlx5_core_err(mdev, "Failed to modify STC FW action_type %d\n",
-			      stc_attr->action_type);
-
+		mlx5_core_err(mdev,
+			      "Failed to modify STC action_type %d, err %d, syndrome 0x%x\n",
+			      stc_attr->action_type, ret,
+			      MLX5_GET(general_obj_out_cmd_hdr,
+				       out, syndrome));
 	return ret;
 }
 
Signature · 1 line
2.44.0
author maintainer
#5 lore raw

PATCH v2 3/5 net/mlx5: HWS, Set the num of queues only when alloc succeeded

In reply to Tariq Toukan

From: Yevgeny Kliteynik <[email protected]>

When initializing send queues, set the number of queues only when allocations are over.

---
 drivers/net/ethernet/mellanox/mlx5/core/steering/hws/send.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/send.c +2 -2
index 1c9206f74aaa..f0c3a2bda0c7 100644
@@ -1121,8 +1121,6 @@ static int hws_bwc_send_queues_init(struct mlx5hws_context *ctx)
 	if (!mlx5hws_context_bwc_supported(ctx))
 		return 0;
 
-	ctx->queues += bwc_queues;
-
 	ctx->bwc_send_queue_locks = kzalloc_objs(*ctx->bwc_send_queue_locks,
 						 bwc_queues);
 
@@ -1134,6 +1132,8 @@ static int hws_bwc_send_queues_init(struct mlx5hws_context *ctx)
 	if (!ctx->bwc_lock_class_keys)
 		goto err_lock_class_keys;
 
+	ctx->queues += bwc_queues;
+
 	for (i = 0; i < bwc_queues; i++) {
 		mutex_init(&ctx->bwc_send_queue_locks[i]);
 		lockdep_register_key(ctx->bwc_lock_class_keys + i);
Signature · 1 line
2.44.0
author maintainer
#6 lore raw

PATCH v2 4/5 net/mlx5: HWS, Remove redundant MLX5_SET in RTC creation

In reply to Tariq Toukan

From: Yevgeny Kliteynik <[email protected]>

Remove duplicated setting of field in mlx5hws_cmd_rtc_create().

---
 drivers/net/ethernet/mellanox/mlx5/core/steering/hws/cmd.c | 1 -
 1 file changed, 1 deletion(-)
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/cmd.c +0 -1
index 2cdc03421433..3782d793550d 100644
@@ -398,7 +398,6 @@ int mlx5hws_cmd_rtc_create(struct mlx5_core_dev *mdev,
 	MLX5_SET(rtc, attr, update_method, rtc_attr->fw_gen_wqe);
 	MLX5_SET(rtc, attr, update_index_mode, rtc_attr->update_index_mode);
 	MLX5_SET(rtc, attr, access_index_mode, rtc_attr->access_index_mode);
-	MLX5_SET(rtc, attr, num_hash_definer, rtc_attr->num_hash_definer);
 	MLX5_SET(rtc, attr, log_depth, rtc_attr->log_depth);
 	MLX5_SET(rtc, attr, log_hash_size, rtc_attr->log_size);
 	MLX5_SET(rtc, attr, table_type, rtc_attr->table_type);
Signature · 1 line
2.44.0
author maintainer
#7 lore raw

PATCH v2 5/5 net/mlx5: HWS, Remove redundant FW command when reading caps

In reply to Tariq Toukan

From: Yevgeny Kliteynik <[email protected]>

Remove redundant FW query that isn't really in use.

---
 drivers/net/ethernet/mellanox/mlx5/core/steering/hws/cmd.c | 6 ------
 1 file changed, 6 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/cmd.c +0 -6
index 3782d793550d..775dc5c2d7a6 100644
@@ -1182,12 +1182,6 @@ int mlx5hws_cmd_query_caps(struct mlx5_core_dev *mdev,
 						capability.e_switch_cap.merged_eswitch);
 	}
 
-	ret = mlx5_cmd_exec(mdev, in, sizeof(in), out, out_size);
-	if (ret) {
-		mlx5_core_err(mdev, "Failed to query device attributes\n");
-		goto out;
-	}
-
 	snprintf(caps->fw_ver, sizeof(caps->fw_ver), "%d.%d.%d",
 		 fw_rev_maj(mdev), fw_rev_min(mdev), fw_rev_sub(mdev));
 
Signature · 1 line
2.44.0