Skip to contentSkip to conversation

net/mlx5: mitigate the Tx queue parameter adjustment

PATCH Superseded Viacheslav Ovsiienko 4 participants 6 messages Apr 23, 2025 – May 12, 2025 Network drivers

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 net/mlx5: mitigate the Tx queue parameter adjustment

he DPDK API rte_eth_tx_queue_setup() has a parameter nb_tx_desc specifying the desired queue capacity, measured in packets.

The ConnectX NIC series has a hardware-imposed queue size limit of 32K WQEs (packet hardware descriptors). Typically, one packet requires one WQE to be sent.

There is a special offload option, data-inlining, to improve performance for small packets. Also, NICs in some configurations require a minimum amount of inline data for the steering engine to operate correctly.

In the case of inline data, more than one WQEs might be required to send a single packet. The mlx5 PMD takes this into account and adjusts the number of queue WQEs accordingly.

If the requested queue capacity can't be satisfied due to the hardware queue size limit, the mlx5 PMD rejected the queue creation, causing unresolvable application failure.

The patch provides the following:

- fixes the calculation of the number of required WQEs to send a single packet with inline data, making it more precise and extending the painless operating range.

- If the requested queue capacity can't be satisfied due to WQE number adjustment for inline data, it no longer causes a severe error. Instead, a warning message is emitted, and the queue is created with the maximum available size, with a reported success.

Please note that the inline data size depends on many options (NIC configuration, queue offload flags, packet offload flags,
packet size, etc.), so the actual queue capacity might not be impacted at all.

---
 drivers/net/mlx5/mlx5_txq.c | 79 +++++++++++++------------------------
 1 file changed, 27 insertions(+), 52 deletions(-)
drivers/net/mlx5/mlx5_txq.c +27 -52
index 3e93517323..6122a79fdf 100644
@@ -731,7 +731,7 @@ txq_calc_inline_max(struct mlx5_txq_ctrl *txq_ctrl)
 	if (!wqe_size)
 		return 0;
 	/*
-	 * This calculation is derived from tthe source of
+	 * This calculation is derived from the source of
 	 * mlx5_calc_send_wqe() in rdma_core library.
 	 */
 	wqe_size = wqe_size * MLX5_WQE_SIZE -
@@ -739,7 +739,7 @@ txq_calc_inline_max(struct mlx5_txq_ctrl *txq_ctrl)
 		   MLX5_WQE_ESEG_SIZE -
 		   MLX5_WSEG_SIZE -
 		   MLX5_WSEG_SIZE +
-		   MLX5_DSEG_MIN_INLINE_SIZE;
+		   MLX5_ESEG_MIN_INLINE_SIZE;
 	return wqe_size;
 }
 
@@ -964,15 +964,13 @@ txq_set_params(struct mlx5_txq_ctrl *txq_ctrl)
  *
  * @param txq_ctrl
  *   Pointer to Tx queue control structure.
- *
- * @return
- *   Zero on success, otherwise the parameters can not be adjusted.
  */
-static int
+static void
 txq_adjust_params(struct mlx5_txq_ctrl *txq_ctrl)
 {
 	struct mlx5_priv *priv = txq_ctrl->priv;
 	struct mlx5_port_config *config = &priv->config;
+	const unsigned int desc = 1 << txq_ctrl->txq.elts_n;
 	unsigned int max_inline;
 
 	max_inline = txq_calc_inline_max(txq_ctrl);
@@ -981,82 +979,60 @@ txq_adjust_params(struct mlx5_txq_ctrl *txq_ctrl)
 		 * Inline data feature is not engaged at all.
 		 * There is nothing to adjust.
 		 */
-		return 0;
+		return;
 	}
 	if (txq_ctrl->max_inline_data <= max_inline) {
 		/*
 		 * The requested inline data length does not
 		 * exceed queue capabilities.
 		 */
-		return 0;
+		return;
 	}
 	if (txq_ctrl->txq.inlen_mode > max_inline) {
-		DRV_LOG(ERR,
-			"minimal data inline requirements (%u) are not"
-			" satisfied (%u) on port %u, try the smaller"
-			" Tx queue size (%d)",
-			txq_ctrl->txq.inlen_mode, max_inline,
-			priv->dev_data->port_id, priv->sh->dev_cap.max_qp_wr);
-		goto error;
+		DRV_LOG(WARNING,
+			"minimal data inline requirements (%u) are not satisfied (%u) on port %u,"
+			" the Tx queue capacity (%d) may not be guaranteed",
+			txq_ctrl->txq.inlen_mode, max_inline, priv->dev_data->port_id, desc);
 	}
 	if (txq_ctrl->txq.inlen_send > max_inline &&
 	    config->txq_inline_max != MLX5_ARG_UNSET &&
 	    config->txq_inline_max > (int)max_inline) {
-		DRV_LOG(ERR,
-			"txq_inline_max requirements (%u) are not"
-			" satisfied (%u) on port %u, try the smaller"
-			" Tx queue size (%d)",
-			txq_ctrl->txq.inlen_send, max_inline,
-			priv->dev_data->port_id, priv->sh->dev_cap.max_qp_wr);
-		goto error;
+		DRV_LOG(WARNING,
+			"txq_inline_max requirements (%u) are not satisfied (%u) on port %u,"
+			" the Tx queue capacity (%d) may not be guaranteed",
+			txq_ctrl->txq.inlen_send, max_inline, priv->dev_data->port_id, desc);
 	}
 	if (txq_ctrl->txq.inlen_empw > max_inline &&
 	    config->txq_inline_mpw != MLX5_ARG_UNSET &&
 	    config->txq_inline_mpw > (int)max_inline) {
-		DRV_LOG(ERR,
-			"txq_inline_mpw requirements (%u) are not"
-			" satisfied (%u) on port %u, try the smaller"
-			" Tx queue size (%d)",
-			txq_ctrl->txq.inlen_empw, max_inline,
-			priv->dev_data->port_id, priv->sh->dev_cap.max_qp_wr);
-		goto error;
+		DRV_LOG(WARNING,
+			"txq_inline_mpw requirements (%u) are not satisfied (%u) on port %u,"
+			" the Tx queue capacity (%d) may not be guaranteed",
+			txq_ctrl->txq.inlen_empw, max_inline, priv->dev_data->port_id, desc);
 	}
 	if (txq_ctrl->txq.tso_en && max_inline < MLX5_MAX_TSO_HEADER) {
-		DRV_LOG(ERR,
-			"tso header inline requirements (%u) are not"
-			" satisfied (%u) on port %u, try the smaller"
-			" Tx queue size (%d)",
-			MLX5_MAX_TSO_HEADER, max_inline,
-			priv->dev_data->port_id, priv->sh->dev_cap.max_qp_wr);
-		goto error;
+		DRV_LOG(WARNING,
+			"tso header inline requirements (%u) are not satisfied (%u) on port %u,"
+			" the Tx queue capacity (%d) may not be guaranteed",
+			MLX5_MAX_TSO_HEADER, max_inline, priv->dev_data->port_id, desc);
 	}
 	if (txq_ctrl->txq.inlen_send > max_inline) {
 		DRV_LOG(WARNING,
-			"adjust txq_inline_max (%u->%u)"
-			" due to large Tx queue on port %u",
-			txq_ctrl->txq.inlen_send, max_inline,
-			priv->dev_data->port_id);
+			"adjust txq_inline_max (%u->%u) due to large Tx queue on port %u",
+			txq_ctrl->txq.inlen_send, max_inline, priv->dev_data->port_id);
 		txq_ctrl->txq.inlen_send = max_inline;
 	}
 	if (txq_ctrl->txq.inlen_empw > max_inline) {
 		DRV_LOG(WARNING,
-			"adjust txq_inline_mpw (%u->%u)"
-			"due to large Tx queue on port %u",
-			txq_ctrl->txq.inlen_empw, max_inline,
-			priv->dev_data->port_id);
+			"adjust txq_inline_mpw (%u->%u) due to large Tx queue on port %u",
+			txq_ctrl->txq.inlen_empw, max_inline, priv->dev_data->port_id);
 		txq_ctrl->txq.inlen_empw = max_inline;
 	}
 	txq_ctrl->max_inline_data = RTE_MAX(txq_ctrl->txq.inlen_send,
 					    txq_ctrl->txq.inlen_empw);
-	MLX5_ASSERT(txq_ctrl->max_inline_data <= max_inline);
-	MLX5_ASSERT(txq_ctrl->txq.inlen_mode <= max_inline);
 	MLX5_ASSERT(txq_ctrl->txq.inlen_mode <= txq_ctrl->txq.inlen_send);
 	MLX5_ASSERT(txq_ctrl->txq.inlen_mode <= txq_ctrl->txq.inlen_empw ||
 		    !txq_ctrl->txq.inlen_empw);
-	return 0;
-error:
-	rte_errno = ENOMEM;
-	return -ENOMEM;
 }
 
 /**
@@ -1105,8 +1081,7 @@ mlx5_txq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,
 	tmpl->txq.port_id = dev->data->port_id;
 	tmpl->txq.idx = idx;
 	txq_set_params(tmpl);
-	if (txq_adjust_params(tmpl))
-		goto error;
+	txq_adjust_params(tmpl);
 	if (txq_calc_wqebb_cnt(tmpl) >
 	    priv->sh->dev_cap.max_qp_wr) {
 		DRV_LOG(ERR,
Signature · 1 line
2.34.1
#1 lore raw

drivers/net/mlx5/mlx5_txq.c txq_adjust_params(struct mlx5_txq_ctrl *txq_ctrl) · line 1017

Quoted 4 lines · On Wed, 23 Apr 2025 15:28:07 +0300 Viacheslav Ovsiienko…
drivers/net/mlx5/mlx5_txq.c +4 -0
+		DRV_LOG(WARNING,
+			"tso header inline requirements (%u) are not satisfied (%u) on port %u,"
+			" the Tx queue capacity (%d) may not be guaranteed",
+			MLX5_MAX_TSO_HEADER, max_inline, priv->dev_data->port_id, desc);

If your messages are so long they can't fit on a single line, then they are too wordy. Please be more terse.

#2 lore raw

PATCH v2 1/1 net/mlx5: mitigate the Tx queue parameter adjustment

he DPDK API rte_eth_tx_queue_setup() has a parameter nb_tx_desc specifying the desired queue capacity, measured in packets.

The ConnectX NIC series has a hardware-imposed queue size limit of 32K WQEs (packet hardware descriptors). Typically, one packet requires one WQE to be sent.

There is a special offload option, data-inlining, to improve performance for small packets. Also, NICs in some configurations require a minimum amount of inline data for the steering engine to operate correctly.

In the case of inline data, more than one WQEs might be required to send a single packet. The mlx5 PMD takes this into account and adjusts the number of queue WQEs accordingly.

If the requested queue capacity can't be satisfied due to the hardware queue size limit, the mlx5 PMD rejected the queue creation, causing unresolvable application failure.

The patch provides the following:

- fixes the calculation of the number of required WQEs to send a single packet with inline data, making it more precise and extending the painless operating range.

- If the requested queue capacity can't be satisfied due to WQE number adjustment for inline data, it no longer causes a severe error. Instead, a warning message is emitted, and the queue is created with the maximum available size, with a reported success.

Please note that the inline data size depends on many options (NIC configuration, queue offload flags, packet offload flags,
packet size, etc.), so the actual queue capacity might not be impacted at all.

---
v2: diagnostics messages made less wordy

---
 drivers/net/mlx5/mlx5_txq.c | 74 +++++++++++--------------------------
 1 file changed, 22 insertions(+), 52 deletions(-)
drivers/net/mlx5/mlx5_txq.c +22 -52
index 3e93517323..eebf3c2534 100644
@@ -731,7 +731,7 @@ txq_calc_inline_max(struct mlx5_txq_ctrl *txq_ctrl)
 	if (!wqe_size)
 		return 0;
 	/*
-	 * This calculation is derived from tthe source of
+	 * This calculation is derived from the source of
 	 * mlx5_calc_send_wqe() in rdma_core library.
 	 */
 	wqe_size = wqe_size * MLX5_WQE_SIZE -
@@ -739,7 +739,7 @@ txq_calc_inline_max(struct mlx5_txq_ctrl *txq_ctrl)
 		   MLX5_WQE_ESEG_SIZE -
 		   MLX5_WSEG_SIZE -
 		   MLX5_WSEG_SIZE +
-		   MLX5_DSEG_MIN_INLINE_SIZE;
+		   MLX5_ESEG_MIN_INLINE_SIZE;
 	return wqe_size;
 }
 
@@ -964,11 +964,8 @@ txq_set_params(struct mlx5_txq_ctrl *txq_ctrl)
  *
  * @param txq_ctrl
  *   Pointer to Tx queue control structure.
- *
- * @return
- *   Zero on success, otherwise the parameters can not be adjusted.
  */
-static int
+static void
 txq_adjust_params(struct mlx5_txq_ctrl *txq_ctrl)
 {
 	struct mlx5_priv *priv = txq_ctrl->priv;
@@ -981,82 +978,56 @@ txq_adjust_params(struct mlx5_txq_ctrl *txq_ctrl)
 		 * Inline data feature is not engaged at all.
 		 * There is nothing to adjust.
 		 */
-		return 0;
+		return;
 	}
 	if (txq_ctrl->max_inline_data <= max_inline) {
 		/*
 		 * The requested inline data length does not
 		 * exceed queue capabilities.
 		 */
-		return 0;
+		return;
 	}
 	if (txq_ctrl->txq.inlen_mode > max_inline) {
-		DRV_LOG(ERR,
-			"minimal data inline requirements (%u) are not"
-			" satisfied (%u) on port %u, try the smaller"
-			" Tx queue size (%d)",
-			txq_ctrl->txq.inlen_mode, max_inline,
-			priv->dev_data->port_id, priv->sh->dev_cap.max_qp_wr);
-		goto error;
+		DRV_LOG(WARNING,
+			"minimal data inline requirements (%u) are not satisfied (%u) on port %u",
+			txq_ctrl->txq.inlen_mode, max_inline, priv->dev_data->port_id);
 	}
 	if (txq_ctrl->txq.inlen_send > max_inline &&
 	    config->txq_inline_max != MLX5_ARG_UNSET &&
 	    config->txq_inline_max > (int)max_inline) {
-		DRV_LOG(ERR,
-			"txq_inline_max requirements (%u) are not"
-			" satisfied (%u) on port %u, try the smaller"
-			" Tx queue size (%d)",
-			txq_ctrl->txq.inlen_send, max_inline,
-			priv->dev_data->port_id, priv->sh->dev_cap.max_qp_wr);
-		goto error;
+		DRV_LOG(WARNING,
+			"txq_inline_max requirements (%u) are not satisfied (%u) on port %u",
+			txq_ctrl->txq.inlen_send, max_inline, priv->dev_data->port_id);
 	}
 	if (txq_ctrl->txq.inlen_empw > max_inline &&
 	    config->txq_inline_mpw != MLX5_ARG_UNSET &&
 	    config->txq_inline_mpw > (int)max_inline) {
-		DRV_LOG(ERR,
-			"txq_inline_mpw requirements (%u) are not"
-			" satisfied (%u) on port %u, try the smaller"
-			" Tx queue size (%d)",
-			txq_ctrl->txq.inlen_empw, max_inline,
-			priv->dev_data->port_id, priv->sh->dev_cap.max_qp_wr);
-		goto error;
+		DRV_LOG(WARNING,
+			"txq_inline_mpw requirements (%u) are not satisfied (%u) on port %u",
+			txq_ctrl->txq.inlen_empw, max_inline, priv->dev_data->port_id);
 	}
 	if (txq_ctrl->txq.tso_en && max_inline < MLX5_MAX_TSO_HEADER) {
-		DRV_LOG(ERR,
-			"tso header inline requirements (%u) are not"
-			" satisfied (%u) on port %u, try the smaller"
-			" Tx queue size (%d)",
-			MLX5_MAX_TSO_HEADER, max_inline,
-			priv->dev_data->port_id, priv->sh->dev_cap.max_qp_wr);
-		goto error;
+		DRV_LOG(WARNING,
+			"tso header inline requirements (%u) are not satisfied (%u) on port %u",
+			MLX5_MAX_TSO_HEADER, max_inline, priv->dev_data->port_id);
 	}
 	if (txq_ctrl->txq.inlen_send > max_inline) {
 		DRV_LOG(WARNING,
-			"adjust txq_inline_max (%u->%u)"
-			" due to large Tx queue on port %u",
-			txq_ctrl->txq.inlen_send, max_inline,
-			priv->dev_data->port_id);
+			"adjust txq_inline_max (%u->%u) due to large Tx queue on port %u",
+			txq_ctrl->txq.inlen_send, max_inline, priv->dev_data->port_id);
 		txq_ctrl->txq.inlen_send = max_inline;
 	}
 	if (txq_ctrl->txq.inlen_empw > max_inline) {
 		DRV_LOG(WARNING,
-			"adjust txq_inline_mpw (%u->%u)"
-			"due to large Tx queue on port %u",
-			txq_ctrl->txq.inlen_empw, max_inline,
-			priv->dev_data->port_id);
+			"adjust txq_inline_mpw (%u->%u) due to large Tx queue on port %u",
+			txq_ctrl->txq.inlen_empw, max_inline, priv->dev_data->port_id);
 		txq_ctrl->txq.inlen_empw = max_inline;
 	}
 	txq_ctrl->max_inline_data = RTE_MAX(txq_ctrl->txq.inlen_send,
 					    txq_ctrl->txq.inlen_empw);
-	MLX5_ASSERT(txq_ctrl->max_inline_data <= max_inline);
-	MLX5_ASSERT(txq_ctrl->txq.inlen_mode <= max_inline);
 	MLX5_ASSERT(txq_ctrl->txq.inlen_mode <= txq_ctrl->txq.inlen_send);
 	MLX5_ASSERT(txq_ctrl->txq.inlen_mode <= txq_ctrl->txq.inlen_empw ||
 		    !txq_ctrl->txq.inlen_empw);
-	return 0;
-error:
-	rte_errno = ENOMEM;
-	return -ENOMEM;
 }
 
 /**
@@ -1105,8 +1076,7 @@ mlx5_txq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,
 	tmpl->txq.port_id = dev->data->port_id;
 	tmpl->txq.idx = idx;
 	txq_set_params(tmpl);
-	if (txq_adjust_params(tmpl))
-		goto error;
+	txq_adjust_params(tmpl);
 	if (txq_calc_wqebb_cnt(tmpl) >
 	    priv->sh->dev_cap.max_qp_wr) {
 		DRV_LOG(ERR,
Signature · 1 line
2.34.1
#3 lore raw

Hello all,

I tested v1 patch on CX-4 card and I can confirm my application boots now! For traceability I am adding the original discussion thread:
https://mails.dpdk.org/archives/users/2025-April/008242.html

Probably a some other problem, but it still outputs these logs:

Config: dpdk: 0000:b3:00.0: setting up TX queue 0: tx_desc: 32768 tx: offloads: 0x10000 hthresh: 0 pthresh: 0 wthresh: 0 tx_free_thresh: 0 tx_rs_thresh: 0 txq_deferred_start: 0 [DeviceConfigureQueues:runmode-dpdk.c:1487]
mlx5_net: adjust txq_inline_max (290->18) due to large Tx queue on port 1
Config: dpdk: 0000:b3:00.0: setting up TX queue 1: tx_desc: 32768 tx: offloads: 0x10000 hthresh: 0 pthresh: 0 wthresh: 0 tx_free_thresh: 0 tx_rs_thresh: 0 txq_deferred_start: 0 [DeviceConfigureQueues:runmode-dpdk.c:1487]
mlx5_net: adjust txq_inline_max (290->18) due to large Tx queue on port 1
Config: dpdk: 0000:b3:00.0: setting up TX queue 2: tx_desc: 32768 tx: offloads: 0x10000 hthresh: 0 pthresh: 0 wthresh: 0 tx_free_thresh: 0 tx_rs_thresh: 0 txq_deferred_start: 0 [DeviceConfigureQueues:runmode-dpdk.c:1487]
mlx5_net: adjust txq_inline_max (290->18) due to large Tx queue on port 1

Is there any way how I can avoid these logs?

Thank you.

Lukas

Quoted 191 lines · Viacheslav Ovsiienko

he DPDK API rte_eth_tx_queue_setup() has a parameter nb_tx_desc specifying the desired queue capacity, measured in packets.

The ConnectX NIC series has a hardware-imposed queue size limit of 32K WQEs (packet hardware descriptors). Typically, one packet requires one WQE to be sent.

There is a special offload option, data-inlining, to improve performance for small packets. Also, NICs in some configurations require a minimum amount of inline data for the steering engine to operate correctly.

In the case of inline data, more than one WQEs might be required to send a single packet. The mlx5 PMD takes this into account and adjusts the number of queue WQEs accordingly.

If the requested queue capacity can't be satisfied due to the hardware queue size limit, the mlx5 PMD rejected the queue creation, causing unresolvable application failure.

The patch provides the following:

- fixes the calculation of the number of required WQEs to send a single packet with inline data, making it more precise and extending the painless operating range.

- If the requested queue capacity can't be satisfied due to WQE number adjustment for inline data, it no longer causes a severe error. Instead, a warning message is emitted, and the queue is created with the maximum available size, with a reported success.

Please note that the inline data size depends on many options (NIC configuration, queue offload flags, packet offload flags,
packet size, etc.), so the actual queue capacity might not be impacted at all.

---
v2: diagnostics messages made less wordy

---
  drivers/net/mlx5/mlx5_txq.c | 74 +++++++++++--------------------------
  1 file changed, 22 insertions(+), 52 deletions(-)
drivers/net/mlx5/mlx5_txq.c +22 -52
index 3e93517323..eebf3c2534 100644
@@ -731,7 +731,7 @@ txq_calc_inline_max(struct mlx5_txq_ctrl *txq_ctrl)
  	if (!wqe_size)
  		return 0;
  	/*
-	 * This calculation is derived from tthe source of
+	 * This calculation is derived from the source of
  	 * mlx5_calc_send_wqe() in rdma_core library.
  	 */
  	wqe_size = wqe_size * MLX5_WQE_SIZE -
@@ -739,7 +739,7 @@ txq_calc_inline_max(struct mlx5_txq_ctrl *txq_ctrl)
  		   MLX5_WQE_ESEG_SIZE -
  		   MLX5_WSEG_SIZE -
  		   MLX5_WSEG_SIZE +
-		   MLX5_DSEG_MIN_INLINE_SIZE;
+		   MLX5_ESEG_MIN_INLINE_SIZE;
  	return wqe_size;
  }
 
@@ -964,11 +964,8 @@ txq_set_params(struct mlx5_txq_ctrl *txq_ctrl)
   *
   * @param txq_ctrl
   *   Pointer to Tx queue control structure.
- *
- * @return
- *   Zero on success, otherwise the parameters can not be adjusted.
   */
-static int
+static void
  txq_adjust_params(struct mlx5_txq_ctrl *txq_ctrl)
  {
  	struct mlx5_priv *priv = txq_ctrl->priv;
@@ -981,82 +978,56 @@ txq_adjust_params(struct mlx5_txq_ctrl *txq_ctrl)
  		 * Inline data feature is not engaged at all.
  		 * There is nothing to adjust.
  		 */
-		return 0;
+		return;
  	}
  	if (txq_ctrl->max_inline_data <= max_inline) {
  		/*
  		 * The requested inline data length does not
  		 * exceed queue capabilities.
  		 */
-		return 0;
+		return;
  	}
  	if (txq_ctrl->txq.inlen_mode > max_inline) {
-		DRV_LOG(ERR,
-			"minimal data inline requirements (%u) are not"
-			" satisfied (%u) on port %u, try the smaller"
-			" Tx queue size (%d)",
-			txq_ctrl->txq.inlen_mode, max_inline,
-			priv->dev_data->port_id, priv->sh->dev_cap.max_qp_wr);
-		goto error;
+		DRV_LOG(WARNING,
+			"minimal data inline requirements (%u) are not satisfied (%u) on port %u",
+			txq_ctrl->txq.inlen_mode, max_inline, priv->dev_data->port_id);
  	}
  	if (txq_ctrl->txq.inlen_send > max_inline &&
  	    config->txq_inline_max != MLX5_ARG_UNSET &&
  	    config->txq_inline_max > (int)max_inline) {
-		DRV_LOG(ERR,
-			"txq_inline_max requirements (%u) are not"
-			" satisfied (%u) on port %u, try the smaller"
-			" Tx queue size (%d)",
-			txq_ctrl->txq.inlen_send, max_inline,
-			priv->dev_data->port_id, priv->sh->dev_cap.max_qp_wr);
-		goto error;
+		DRV_LOG(WARNING,
+			"txq_inline_max requirements (%u) are not satisfied (%u) on port %u",
+			txq_ctrl->txq.inlen_send, max_inline, priv->dev_data->port_id);
  	}
  	if (txq_ctrl->txq.inlen_empw > max_inline &&
  	    config->txq_inline_mpw != MLX5_ARG_UNSET &&
  	    config->txq_inline_mpw > (int)max_inline) {
-		DRV_LOG(ERR,
-			"txq_inline_mpw requirements (%u) are not"
-			" satisfied (%u) on port %u, try the smaller"
-			" Tx queue size (%d)",
-			txq_ctrl->txq.inlen_empw, max_inline,
-			priv->dev_data->port_id, priv->sh->dev_cap.max_qp_wr);
-		goto error;
+		DRV_LOG(WARNING,
+			"txq_inline_mpw requirements (%u) are not satisfied (%u) on port %u",
+			txq_ctrl->txq.inlen_empw, max_inline, priv->dev_data->port_id);
  	}
  	if (txq_ctrl->txq.tso_en && max_inline < MLX5_MAX_TSO_HEADER) {
-		DRV_LOG(ERR,
-			"tso header inline requirements (%u) are not"
-			" satisfied (%u) on port %u, try the smaller"
-			" Tx queue size (%d)",
-			MLX5_MAX_TSO_HEADER, max_inline,
-			priv->dev_data->port_id, priv->sh->dev_cap.max_qp_wr);
-		goto error;
+		DRV_LOG(WARNING,
+			"tso header inline requirements (%u) are not satisfied (%u) on port %u",
+			MLX5_MAX_TSO_HEADER, max_inline, priv->dev_data->port_id);
  	}
  	if (txq_ctrl->txq.inlen_send > max_inline) {
  		DRV_LOG(WARNING,
-			"adjust txq_inline_max (%u->%u)"
-			" due to large Tx queue on port %u",
-			txq_ctrl->txq.inlen_send, max_inline,
-			priv->dev_data->port_id);
+			"adjust txq_inline_max (%u->%u) due to large Tx queue on port %u",
+			txq_ctrl->txq.inlen_send, max_inline, priv->dev_data->port_id);
  		txq_ctrl->txq.inlen_send = max_inline;
  	}
  	if (txq_ctrl->txq.inlen_empw > max_inline) {
  		DRV_LOG(WARNING,
-			"adjust txq_inline_mpw (%u->%u)"
-			"due to large Tx queue on port %u",
-			txq_ctrl->txq.inlen_empw, max_inline,
-			priv->dev_data->port_id);
+			"adjust txq_inline_mpw (%u->%u) due to large Tx queue on port %u",
+			txq_ctrl->txq.inlen_empw, max_inline, priv->dev_data->port_id);
  		txq_ctrl->txq.inlen_empw = max_inline;
  	}
  	txq_ctrl->max_inline_data = RTE_MAX(txq_ctrl->txq.inlen_send,
  					    txq_ctrl->txq.inlen_empw);
-	MLX5_ASSERT(txq_ctrl->max_inline_data <= max_inline);
-	MLX5_ASSERT(txq_ctrl->txq.inlen_mode <= max_inline);
  	MLX5_ASSERT(txq_ctrl->txq.inlen_mode <= txq_ctrl->txq.inlen_send);
  	MLX5_ASSERT(txq_ctrl->txq.inlen_mode <= txq_ctrl->txq.inlen_empw ||
  		    !txq_ctrl->txq.inlen_empw);
-	return 0;
-error:
-	rte_errno = ENOMEM;
-	return -ENOMEM;
  }
 
  /**
@@ -1105,8 +1076,7 @@ mlx5_txq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,
  	tmpl->txq.port_id = dev->data->port_id;
  	tmpl->txq.idx = idx;
  	txq_set_params(tmpl);
-	if (txq_adjust_params(tmpl))
-		goto error;
+	txq_adjust_params(tmpl);
  	if (txq_calc_wqebb_cnt(tmpl) >
  	    priv->sh->dev_cap.max_qp_wr) {
  		DRV_LOG(ERR,
#4 lore raw

Hi, Lukáš

I think this message is important to notify users that something is going as not expected. Either queue size or inline length should be adjusted by user/developer.
To suppress the message in your specific case (with max queue) the devarg "txq_inline_max=18" can be used. So, the message encourages us to recognize the inline data length is truncated.

With best regards,
Slava

Quoted 250 lines

-----Original Message-----
From: Lukáš Šišmiš <[email protected]>
Sent: Friday, April 25, 2025 2:55 PM
To: Slava Ovsiienko <[email protected]>; [email protected]
Cc: Raslan Darawsheh <[email protected]>; Matan Azrad <[email protected]>; Suanming Mou <[email protected]>; Dariusz Sosnowski <[email protected]>
Subject: Re: [PATCH v2] net/mlx5: mitigate the Tx queue parameter adjustment

Hello all,

I tested v1 patch on CX-4 card and I can confirm my application boots now! For traceability I am adding the original discussion thread:
https://mails.dpdk.org/archives/users/2025-April/008242.html

Probably a some other problem, but it still outputs these logs:

Config: dpdk: 0000:b3:00.0: setting up TX queue 0: tx_desc: 32768 tx:
offloads: 0x10000 hthresh: 0 pthresh: 0 wthresh: 0 tx_free_thresh: 0
tx_rs_thresh: 0 txq_deferred_start: 0 [DeviceConfigureQueues:runmode-dpdk.c:1487]
mlx5_net: adjust txq_inline_max (290->18) due to large Tx queue on port 1
Config: dpdk: 0000:b3:00.0: setting up TX queue 1: tx_desc: 32768 tx:
offloads: 0x10000 hthresh: 0 pthresh: 0 wthresh: 0 tx_free_thresh: 0
tx_rs_thresh: 0 txq_deferred_start: 0 [DeviceConfigureQueues:runmode-dpdk.c:1487]
mlx5_net: adjust txq_inline_max (290->18) due to large Tx queue on port 1
Config: dpdk: 0000:b3:00.0: setting up TX queue 2: tx_desc: 32768 tx:
offloads: 0x10000 hthresh: 0 pthresh: 0 wthresh: 0 tx_free_thresh: 0
tx_rs_thresh: 0 txq_deferred_start: 0 [DeviceConfigureQueues:runmode-dpdk.c:1487]
mlx5_net: adjust txq_inline_max (290->18) due to large Tx queue on port 1

Is there any way how I can avoid these logs?

Thank you.

Lukas

Quoted 96 lines · Viacheslav Ovsiienko

he DPDK API rte_eth_tx_queue_setup() has a parameter nb_tx_desc specifying the desired queue capacity, measured in packets.

The ConnectX NIC series has a hardware-imposed queue size limit of 32K WQEs (packet hardware descriptors). Typically, one packet requires one WQE to be sent.

There is a special offload option, data-inlining, to improve performance for small packets. Also, NICs in some configurations require a minimum amount of inline data for the steering engine to operate correctly.

In the case of inline data, more than one WQEs might be required to send a single packet. The mlx5 PMD takes this into account and adjusts the number of queue WQEs accordingly.

If the requested queue capacity can't be satisfied due to the hardware queue size limit, the mlx5 PMD rejected the queue creation, causing unresolvable application failure.

The patch provides the following:

- fixes the calculation of the number of required WQEs to send a single packet with inline data, making it more precise and extending the painless operating range.

- If the requested queue capacity can't be satisfied due to WQE number adjustment for inline data, it no longer causes a severe error. Instead, a warning message is emitted, and the queue is created with the maximum available size, with a reported success.

Please note that the inline data size depends on many options (NIC configuration, queue offload flags, packet offload flags,
packet size, etc.), so the actual queue capacity might not be impacted at all.

---
v2: diagnostics messages made less wordy

---
  drivers/net/mlx5/mlx5_txq.c | 74 +++++++++++--------------------------
  1 file changed, 22 insertions(+), 52 deletions(-)
drivers/net/mlx5/mlx5_txq.c +5 -8
index 3e93517323..eebf3c2534 100644
@@ -731,7 +731,7 @@ txq_calc_inline_max(struct mlx5_txq_ctrl *txq_ctrl)
  	if (!wqe_size)
  		return 0;
  	/*
-	 * This calculation is derived from tthe source of
+	 * This calculation is derived from the source of
  	 * mlx5_calc_send_wqe() in rdma_core library.
  	 */
  	wqe_size = wqe_size * MLX5_WQE_SIZE -
@@ -739,7 +739,7 @@ txq_calc_inline_max(struct mlx5_txq_ctrl *txq_ctrl)
  		   MLX5_WQE_ESEG_SIZE -
  		   MLX5_WSEG_SIZE -
  		   MLX5_WSEG_SIZE +
-		   MLX5_DSEG_MIN_INLINE_SIZE;
+		   MLX5_ESEG_MIN_INLINE_SIZE;
  	return wqe_size;
  }
 
@@ -964,11 +964,8 @@ txq_set_params(struct mlx5_txq_ctrl *txq_ctrl)
   *
   * @param txq_ctrl
   *   Pointer to Tx queue control structure.
- *
- * @return
- *   Zero on success, otherwise the parameters can not be adjusted.
   */
-static int
+static void
  txq_adjust_params(struct mlx5_txq_ctrl *txq_ctrl)
  {
  	struct mlx5_priv *priv = txq_ctrl->priv;
@@ -981,82 +978,56 @@ txq_adjust_params(struct mlx5_txq_ctrl *txq_ctrl)
  		 * Inline data feature is not engaged at all.
  		 * There is nothing to adjust.
  		 */
-		return 0;
+		return;
  	}
  	if (txq_ctrl->max_inline_data <= max_inline) {
  		/*
  		 * The requested inline data length does not
  		 * exceed queue capabilities.
  		 */
-		return 0;
+		return;
  	}
  	if (txq_ctrl->txq.inlen_mode > max_inline) {

drivers/net/mlx5/mlx5_txq.c txq_adjust_params(struct mlx5_txq_ctrl *txq_ctrl) · line 993

Quoted 10 lines
drivers/net/mlx5/mlx5_txq.c +2 -7
-		DRV_LOG(ERR,
-			"minimal data inline requirements (%u) are not"
-			" satisfied (%u) on port %u, try the smaller"
-			" Tx queue size (%d)",
-			txq_ctrl->txq.inlen_mode, max_inline,
-			priv->dev_data->port_id, priv->sh-
 dev_cap.max_qp_wr);
-		goto error;
+		DRV_LOG(WARNING,
+			"minimal data inline requirements (%u) are not

satisfied (%u) on port %u",

Quoted 6 lines
drivers/net/mlx5/mlx5_txq.c +1 -0
+			txq_ctrl->txq.inlen_mode, max_inline, priv-
 dev_data->port_id);
  	}
  	if (txq_ctrl->txq.inlen_send > max_inline &&
  	    config->txq_inline_max != MLX5_ARG_UNSET &&
  	    config->txq_inline_max > (int)max_inline) {

drivers/net/mlx5/mlx5_txq.c txq_adjust_params(struct mlx5_txq_ctrl *txq_ctrl)

Quoted 10 lines
drivers/net/mlx5/mlx5_txq.c +2 -7
-		DRV_LOG(ERR,
-			"txq_inline_max requirements (%u) are not"
-			" satisfied (%u) on port %u, try the smaller"
-			" Tx queue size (%d)",
-			txq_ctrl->txq.inlen_send, max_inline,
-			priv->dev_data->port_id, priv->sh-
 dev_cap.max_qp_wr);
-		goto error;
+		DRV_LOG(WARNING,
+			"txq_inline_max requirements (%u) are not satisfied

(%u) on port %u",

Quoted 6 lines
drivers/net/mlx5/mlx5_txq.c +1 -0
+			txq_ctrl->txq.inlen_send, max_inline, priv->dev_data-
 port_id);
  	}
  	if (txq_ctrl->txq.inlen_empw > max_inline &&
  	    config->txq_inline_mpw != MLX5_ARG_UNSET &&
  	    config->txq_inline_mpw > (int)max_inline) {

drivers/net/mlx5/mlx5_txq.c txq_adjust_params(struct mlx5_txq_ctrl *txq_ctrl)

Quoted 10 lines
drivers/net/mlx5/mlx5_txq.c +2 -7
-		DRV_LOG(ERR,
-			"txq_inline_mpw requirements (%u) are not"
-			" satisfied (%u) on port %u, try the smaller"
-			" Tx queue size (%d)",
-			txq_ctrl->txq.inlen_empw, max_inline,
-			priv->dev_data->port_id, priv->sh-
 dev_cap.max_qp_wr);
-		goto error;
+		DRV_LOG(WARNING,
+			"txq_inline_mpw requirements (%u) are not satisfied

(%u) on port %u",

Quoted 4 lines
drivers/net/mlx5/mlx5_txq.c +1 -0
+			txq_ctrl->txq.inlen_empw, max_inline, priv-
 dev_data->port_id);
  	}
  	if (txq_ctrl->txq.tso_en && max_inline < MLX5_MAX_TSO_HEADER) {

drivers/net/mlx5/mlx5_txq.c txq_adjust_params(struct mlx5_txq_ctrl *txq_ctrl)

Quoted 10 lines
drivers/net/mlx5/mlx5_txq.c +2 -7
-		DRV_LOG(ERR,
-			"tso header inline requirements (%u) are not"
-			" satisfied (%u) on port %u, try the smaller"
-			" Tx queue size (%d)",
-			MLX5_MAX_TSO_HEADER, max_inline,
-			priv->dev_data->port_id, priv->sh-
 dev_cap.max_qp_wr);
-		goto error;
+		DRV_LOG(WARNING,
+			"tso header inline requirements (%u) are not satisfied

(%u) on port %u",

drivers/net/mlx5/mlx5_txq.c txq_adjust_params(struct mlx5_txq_ctrl *txq_ctrl)

Quoted 10 lines
drivers/net/mlx5/mlx5_txq.c +2 -4
+			MLX5_MAX_TSO_HEADER, max_inline, priv-
 dev_data->port_id);
  	}
  	if (txq_ctrl->txq.inlen_send > max_inline) {
  		DRV_LOG(WARNING,
-			"adjust txq_inline_max (%u->%u)"
-			" due to large Tx queue on port %u",
-			txq_ctrl->txq.inlen_send, max_inline,
-			priv->dev_data->port_id);
+			"adjust txq_inline_max (%u->%u) due to large Tx

queue on port %u",

drivers/net/mlx5/mlx5_txq.c txq_adjust_params(struct mlx5_txq_ctrl *txq_ctrl)

Quoted 11 lines
drivers/net/mlx5/mlx5_txq.c +2 -4
+			txq_ctrl->txq.inlen_send, max_inline, priv->dev_data-
 port_id);
  		txq_ctrl->txq.inlen_send = max_inline;
  	}
  	if (txq_ctrl->txq.inlen_empw > max_inline) {
  		DRV_LOG(WARNING,
-			"adjust txq_inline_mpw (%u->%u)"
-			"due to large Tx queue on port %u",
-			txq_ctrl->txq.inlen_empw, max_inline,
-			priv->dev_data->port_id);
+			"adjust txq_inline_mpw (%u->%u) due to large Tx

queue on port %u",

drivers/net/mlx5/mlx5_txq.c txq_adjust_params(struct mlx5_txq_ctrl *txq_ctrl)

Quoted 10 lines
drivers/net/mlx5/mlx5_txq.c +1 -2
+			txq_ctrl->txq.inlen_empw, max_inline, priv-
 dev_data->port_id);
  		txq_ctrl->txq.inlen_empw = max_inline;
  	}
  	txq_ctrl->max_inline_data = RTE_MAX(txq_ctrl->txq.inlen_send,
  					    txq_ctrl->txq.inlen_empw);
-	MLX5_ASSERT(txq_ctrl->max_inline_data <= max_inline);
-	MLX5_ASSERT(txq_ctrl->txq.inlen_mode <= max_inline);
  	MLX5_ASSERT(txq_ctrl->txq.inlen_mode <= txq_ctrl->txq.inlen_send);
  	MLX5_ASSERT(txq_ctrl->txq.inlen_mode <= txq_ctrl->txq.inlen_empw
||
>   		    !txq_ctrl->txq.inlen_empw);

drivers/net/mlx5/mlx5_txq.c mlx5_txq_new(struct rte_eth_dev *dev, uint16_t

Quoted 8 lines
drivers/net/mlx5/mlx5_txq.c +0 -4
-	return 0;
-error:
-	rte_errno = ENOMEM;
-	return -ENOMEM;
  }
 
  /**
@@ -1105,8 +1076,7 @@ mlx5_txq_new(struct rte_eth_dev *dev, uint16_t

idx, uint16_t desc,

Quoted 9 lines
drivers/net/mlx5/mlx5_txq.c +1 -2
  	tmpl->txq.port_id = dev->data->port_id;
  	tmpl->txq.idx = idx;
  	txq_set_params(tmpl);
-	if (txq_adjust_params(tmpl))
-		goto error;
+	txq_adjust_params(tmpl);
  	if (txq_calc_wqebb_cnt(tmpl) >
  	    priv->sh->dev_cap.max_qp_wr) {
  		DRV_LOG(ERR,
#5 lore raw
In reply to Viacheslav Ovsiienko

Hi,

Quoted 39 lines · Viacheslav Ovsiienko

he DPDK API rte_eth_tx_queue_setup() has a parameter nb_tx_desc specifying the desired queue capacity, measured in packets.

The ConnectX NIC series has a hardware-imposed queue size limit of 32K WQEs (packet hardware descriptors). Typically, one packet requires one WQE to be sent.

There is a special offload option, data-inlining, to improve performance for small packets. Also, NICs in some configurations require a minimum amount of inline data for the steering engine to operate correctly.

In the case of inline data, more than one WQEs might be required to send a single packet. The mlx5 PMD takes this into account and adjusts the number of queue WQEs accordingly.

If the requested queue capacity can't be satisfied due to the hardware queue size limit, the mlx5 PMD rejected the queue creation, causing unresolvable application failure.

The patch provides the following:

- fixes the calculation of the number of required WQEs to send a single packet with inline data, making it more precise and extending the painless operating range.

- If the requested queue capacity can't be satisfied due to WQE number adjustment for inline data, it no longer causes a severe error. Instead, a warning message is emitted, and the queue is created with the maximum available size, with a reported success.

Please note that the inline data size depends on many options (NIC configuration, queue offload flags, packet offload flags,
packet size, etc.), so the actual queue capacity might not be impacted at all.

Patch applied to next-net-mlx,

Signature · 2 lines
Kindest regards
Raslan Darawsheh