Skip to contentSkip to conversation

Direct re-arming of buffers on receive side

PATCH Superseded Feifei Wang 10 participants 145 messages Apr 20, 2022 – Sep 27, 2023 1 message not in the archive Page 1 of 3

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover Direct re-arming of buffers on receive side

Currently, the transmit side frees the buffers into the lcore cache and the receive side allocates buffers from the lcore cache. The transmit side typically frees 32 buffers resulting in 32*8=256B of stores to lcore cache. The receive side allocates 32 buffers and stores them in the receive side software ring, resulting in 32*8=256B of stores and 256B of load from the lcore cache.

This patch proposes a mechanism to avoid freeing to/allocating from the lcore cache. i.e. the receive side will free the buffers from transmit side directly into it's software ring. This will avoid the 256B of loads and stores introduced by the lcore cache. It also frees up the cache lines used by the lcore cache.

However, this solution poses several constraints:

1)The receive queue needs to know which transmit queue it should take the buffers from. The application logic decides which transmit port to use to send out the packets. In many use cases the NIC might have a single port ([1], [2], [3]), in which case a given transmit queue is always mapped to a single receive queue (1:1 Rx queue: Tx queue). This is easy to configure.

If the NIC has 2 ports (there are several references), then we will have 1:2 (RX queue: TX queue) mapping which is still easy to configure. However, if this is generalized to 'N' ports, the configuration can be long. More over the PMD would have to scan a list of transmit queues to pull the buffers from.

2)The other factor that needs to be considered is 'run-to-completion' vs 'pipeline' models. In the run-to-completion model, the receive side and the transmit side are running on the same lcore serially. In the pipeline model. The receive side and transmit side might be running on different lcores in parallel. This requires locking. This is not supported at this point.

3)Tx and Rx buffers must be from the same mempool. And we also must ensure Tx buffer free number is equal to Rx buffer free number:
(txq->tx_rs_thresh == RTE_I40E_RXQ_REARM_THRESH)
Thus, 'tx_next_dd' can be updated correctly in direct-rearm mode. This is due to tx_next_dd is a variable to compute tx sw-ring free location. Its value will be one more round than the position where next time free starts.

Current status in this RFC:
1)An API is added to allow for mapping a TX queue to a RX queue.
Currently it supports 1:1 mapping.
2)The i40e driver is changed to do the direct re-arm of the receive
side.
3)L3fwd application is modified to do the direct rearm mapping automatically without user config. This follows the rules that the thread can map TX queue to a RX queue based on the first received package destination port.

Testing status:
1.The testing results for L3fwd are as follows:
-------------------------------------------------------------------
enabled direct rearm
-------------------------------------------------------------------
Arm:
N1SDP(neon path):
without fast-free mode with fast-free mode

	+14.1%				+7.0%

Ampere Altra(neon path):
without fast-free mode with fast-free mode

	+17.1				+14.0%

X86:
Dell-8268(limit frequency):
sse path:
without fast-free mode with fast-free mode

	+6.96%				+2.02%
avx2 path:
without fast-free mode		with fast-free mode
	+9.04%				+7.75%
avx512 path:
without fast-free mode		with fast-free mode
	+5.43%				+1.57%

-------------------------------------------------------------------
This patch can not affect base performance of normal mode. Furthermore, the reason for that limiting the CPU frequency is that dell-8268 can encounter i40e NIC bottleneck with maximum frequency.

2.The testing results for VPP-L3fwd are as follows:
-------------------------------------------------------------------
Arm:
N1SDP(neon path):
with direct re-arm mode enabled

	+7.0%

-------------------------------------------------------------------
For Ampere Altra and X86,VPP-L3fwd test has not been done.

Reference:
[1] https://store.nvidia.com/en-us/networking/store/product/MCX623105AN-CDAT/NVIDIAMCX623105ANCDATConnectX6DxENAdapterCard100GbECryptoDisabled/
[2] https://www.intel.com/content/www/us/en/products/sku/192561/intel-ethernet-network-adapter-e810cqda1/specifications.html
[3] https://www.broadcom.com/products/ethernet-connectivity/network-adapters/100gb-nic-ocp/n1100g

Feifei Wang (5):
net/i40e: remove redundant Dtype initialization
net/i40e: enable direct rearm mode
ethdev: add API for direct rearm mode
net/i40e: add direct rearm mode internal API
examples/l3fwd: enable direct rearm mode

 drivers/net/i40e/i40e_ethdev.c          |  34 +++
 drivers/net/i40e/i40e_rxtx.c            |   4 -
 drivers/net/i40e/i40e_rxtx.h            |   4 +
 drivers/net/i40e/i40e_rxtx_common_avx.h | 269 ++++++++++++++++++++++++
 drivers/net/i40e/i40e_rxtx_vec_avx2.c   |  14 +-
 drivers/net/i40e/i40e_rxtx_vec_avx512.c | 249 +++++++++++++++++++++-
 drivers/net/i40e/i40e_rxtx_vec_neon.c   | 141 ++++++++++++-
 drivers/net/i40e/i40e_rxtx_vec_sse.c    | 170 ++++++++++++++-
 examples/l3fwd/l3fwd_lpm.c              |  16 +-
 lib/ethdev/ethdev_driver.h              |  15 ++
 lib/ethdev/rte_ethdev.c                 |  14 ++
 lib/ethdev/rte_ethdev.h                 |  31 +++
 lib/ethdev/version.map                  |   1 +
 13 files changed, 949 insertions(+), 13 deletions(-)
Signature · 1 line
2.25.1
#1 lore raw

PATCH 1/5 net/i40e: remove redundant Dtype initialization

The Dtype field is set to 0xf by the NIC to indicate DMA completion, only after the CPU requests to be informed by setting the RS bit. Hence, it is not required to set Dtype to 0xf during initialization.

Not setting the Dtype field to 0xf helps to know that a given descriptor is not sent to the NIC yet after initialization.

---
 drivers/net/i40e/i40e_rxtx.c | 4 ----
 1 file changed, 4 deletions(-)
drivers/net/i40e/i40e_rxtx.c +0 -4
index 25a28ecea2..745734d5e4 100644
@@ -2767,10 +2767,6 @@ i40e_reset_tx_queue(struct i40e_tx_queue *txq)
 
 	prev = (uint16_t)(txq->nb_tx_desc - 1);
 	for (i = 0; i < txq->nb_tx_desc; i++) {
-		volatile struct i40e_tx_desc *txd = &txq->tx_ring[i];
-
-		txd->cmd_type_offset_bsz =
-			rte_cpu_to_le_64(I40E_TX_DESC_DTYPE_DESC_DONE);
 		txe[i].mbuf =  NULL;
 		txe[i].last_id = i;
 		txe[prev].next_id = i;
Signature · 1 line
2.25.1
#2 lore raw

PATCH 2/5 net/i40e: enable direct rearm mode

In reply to Feifei Wang

For i40e driver, enable direct re-arm mode. This patch supports the case of mapping Rx/Tx queues from the same single lcore.

---
 drivers/net/i40e/i40e_rxtx.h            |   4 +
 drivers/net/i40e/i40e_rxtx_common_avx.h | 269 ++++++++++++++++++++++++
 drivers/net/i40e/i40e_rxtx_vec_avx2.c   |  14 +-
 drivers/net/i40e/i40e_rxtx_vec_avx512.c | 249 +++++++++++++++++++++-
 drivers/net/i40e/i40e_rxtx_vec_neon.c   | 141 ++++++++++++-
 drivers/net/i40e/i40e_rxtx_vec_sse.c    | 170 ++++++++++++++-
 6 files changed, 839 insertions(+), 8 deletions(-)
drivers/net/i40e/i40e_rxtx.h +4 -0
index 5e6eecc501..1fdf4305f4 100644
@@ -102,6 +102,8 @@ struct i40e_rx_queue {
 
 	uint16_t rxrearm_nb;	/**< number of remaining to be re-armed */
 	uint16_t rxrearm_start;	/**< the idx we start the re-arming from */
+	uint16_t direct_rxrearm_port; /** device TX port ID for direct re-arm mode */
+	uint16_t direct_rxrearm_queue; /** TX queue index for direct re-arm mode */
 	uint64_t mbuf_initializer; /**< value to init mbufs */
 
 	uint16_t port_id; /**< device port ID */
@@ -121,6 +123,8 @@ struct i40e_rx_queue {
 	uint16_t rx_using_sse; /**<flag indicate the usage of vPMD for rx */
 	uint8_t dcb_tc;         /**< Traffic class of rx queue */
 	uint64_t offloads; /**< Rx offload flags of RTE_ETH_RX_OFFLOAD_* */
+	/**<  0 if direct re-arm mode disabled, 1 when enabled */
+	bool direct_rxrearm_enable;
 	const struct rte_memzone *mz;
 };
 
drivers/net/i40e/i40e_rxtx_common_avx.h +269 -0

275 lines not on this page · raw message

drivers/net/i40e/i40e_rxtx_vec_avx2.c +12 -2
index c73b2a321b..fcb7ba0273 100644
@@ -25,6 +25,12 @@ i40e_rxq_rearm(struct i40e_rx_queue *rxq)
 	return i40e_rxq_rearm_common(rxq, false);
 }
 
+static __rte_always_inline void
+i40e_rxq_direct_rearm(struct i40e_rx_queue *rxq)
+{
+	return i40e_rxq_direct_rearm_common(rxq, false);
+}
+
 #ifndef RTE_LIBRTE_I40E_16BYTE_RX_DESC
 /* Handles 32B descriptor FDIR ID processing:
  * rxdp: receive descriptor ring, required to load 2nd 16B half of each desc
@@ -128,8 +134,12 @@ _recv_raw_pkts_vec_avx2(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts,
 	/* See if we need to rearm the RX queue - gives the prefetch a bit
 	 * of time to act
 	 */
-	if (rxq->rxrearm_nb > RTE_I40E_RXQ_REARM_THRESH)
-		i40e_rxq_rearm(rxq);
+	if (rxq->rxrearm_nb > RTE_I40E_RXQ_REARM_THRESH) {
+		if (rxq->direct_rxrearm_enable)
+			i40e_rxq_direct_rearm(rxq);
+		else
+			i40e_rxq_rearm(rxq);
+	}
 
 	/* Before we start moving massive data around, check to see if
 	 * there is actually a packet available
drivers/net/i40e/i40e_rxtx_vec_avx512.c +247 -2

267 lines not on this page · raw message

drivers/net/i40e/i40e_rxtx_vec_neon.c +139 -2

153 lines not on this page · raw message

drivers/net/i40e/i40e_rxtx_vec_sse.c +168 -2

182 lines not on this page · raw message

Signature · 1 line
2.25.1
#3 lore raw

Re: [PATCH v1 2/5] net/i40e: enable direct rearm mode

Quoted 73 lines

For i40e driver, enable direct re-arm mode. This patch supports the case of mapping Rx/Tx queues from the same single lcore.

---
  drivers/net/i40e/i40e_rxtx.h            |   4 +
  drivers/net/i40e/i40e_rxtx_common_avx.h | 269 ++++++++++++++++++++++++
  drivers/net/i40e/i40e_rxtx_vec_avx2.c   |  14 +-
  drivers/net/i40e/i40e_rxtx_vec_avx512.c | 249 +++++++++++++++++++++-
  drivers/net/i40e/i40e_rxtx_vec_neon.c   | 141 ++++++++++++-
  drivers/net/i40e/i40e_rxtx_vec_sse.c    | 170 ++++++++++++++-
  6 files changed, 839 insertions(+), 8 deletions(-)
drivers/net/i40e/i40e_rxtx.h +4 -0
index 5e6eecc501..1fdf4305f4 100644
@@ -102,6 +102,8 @@ struct i40e_rx_queue {
 
  	uint16_t rxrearm_nb;	/**< number of remaining to be re-armed */
  	uint16_t rxrearm_start;	/**< the idx we start the re-arming from */
+	uint16_t direct_rxrearm_port; /** device TX port ID for direct re-arm mode */
+	uint16_t direct_rxrearm_queue; /** TX queue index for direct re-arm mode */
  	uint64_t mbuf_initializer; /**< value to init mbufs */
 
  	uint16_t port_id; /**< device port ID */
@@ -121,6 +123,8 @@ struct i40e_rx_queue {
  	uint16_t rx_using_sse; /**<flag indicate the usage of vPMD for rx */
  	uint8_t dcb_tc;         /**< Traffic class of rx queue */
  	uint64_t offloads; /**< Rx offload flags of RTE_ETH_RX_OFFLOAD_* */
+	/**<  0 if direct re-arm mode disabled, 1 when enabled */
+	bool direct_rxrearm_enable;
  	const struct rte_memzone *mz;
  };
 
drivers/net/i40e/i40e_rxtx_common_avx.h +26 -0
index cfc1e63173..a742723e07 100644
@@ -209,6 +209,275 @@ i40e_rxq_rearm_common(struct i40e_rx_queue *rxq, __rte_unused bool avx512)
  	/* Update the tail pointer on the NIC */
  	I40E_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id);
  }
+
+static __rte_always_inline void
+i40e_rxq_direct_rearm_common(struct i40e_rx_queue *rxq, __rte_unused bool avx512)
+{
+	struct rte_eth_dev *dev;
+	struct i40e_tx_queue *txq;rivers/net/i40e/i40e_rxtx_common_avx.h
+	volatile union i40e_rx_desc *rxdp;
+	struct i40e_tx_entry *txep;
+	struct i40e_rx_entry *rxep;
+	struct rte_mbuf *m[RTE_I40E_RXQ_REARM_THRESH];
+	uint16_t tx_port_id, tx_queue_id;
+	uint16_t rx_id;
+	uint16_t i, n;
+	uint16_t nb_rearm = 0;
+
+	rxdp = rxq->rx_ring + rxq->rxrearm_start;
+	rxep = &rxq->sw_ring[rxq->rxrearm_start];
+
+	tx_port_id = rxq->direct_rxrearm_port;
+	tx_queue_id = rxq->direct_rxrearm_queue;
+	dev = &rte_eth_devices[tx_port_id];
+	txq = dev->data->tx_queues[tx_queue_id];
+
+	/* check Rx queue is able to take in the whole
+	 * batch of free mbufs from Tx queue
+	 */

drivers/net/i40e/i40e_rxtx_common_avx.h i40e_rxq_rearm_common(struct i40e_rx_queue *rxq, __rte_unused bool avx512) · line 247

Quoted 10 lines
drivers/net/i40e/i40e_rxtx_common_avx.h +10 -0
+	if (rxq->rxrearm_nb > txq->tx_rs_thresh) {
+		/* check DD bits on threshold descriptor */
+		if ((txq->tx_ring[txq->tx_next_dd].cmd_type_offset_bsz &
+				rte_cpu_to_le_64(I40E_TXD_QW1_DTYPE_MASK)) !=
+				rte_cpu_to_le_64(I40E_TX_DESC_DTYPE_DESC_DONE)) {
+			goto mempool_bulk;
+		}
+
+		if (txq->tx_rs_thresh != RTE_I40E_RXQ_REARM_THRESH)
+			goto mempool_bulk;

I think all these checks (is this mode can be enabled) should be done at config phase, not at data-path.

drivers/net/i40e/i40e_rxtx_common_avx.h i40e_rxq_rearm_common(struct i40e_rx_queue *rxq, __rte_unused bool avx512) · line 254

Quoted 7 lines
drivers/net/i40e/i40e_rxtx_common_avx.h +7 -0
+
+		n = txq->tx_rs_thresh;
+
+		/* first buffer to free from S/W ring is at index
+		 * tx_next_dd - (tx_rs_thresh-1)
+		 */
+		txep = &txq->sw_ring[txq->tx_next_dd - (n - 1)];

It really looks bad that RX function acesses and modifies TXQ data directly. Would be much better to hide TXD checking/manipulation into a separate TXQ function (txq_mbuf() or so) that RX path can invoke.

Quoted 10 lines
drivers/net/i40e/i40e_rxtx_common_avx.h +10 -0
+
+		if (txq->offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) {
+			/* directly put mbufs from Tx to Rx,
+			 * and initialize the mbufs in vector
+			 */
+			for (i = 0; i < n; i++)
+				rxep[i].mbuf = txep[i].mbuf;
+		} else {
+			for (i = 0; i < n; i++) {
+				m[i] = rte_pktmbuf_prefree_seg(txep[i].mbuf);

drivers/net/i40e/i40e_rxtx_common_avx.h i40e_rxq_rearm_common(struct i40e_rx_queue *rxq, __rte_unused bool avx512) · line 274

Quoted 10 lines
drivers/net/i40e/i40e_rxtx_common_avx.h +10 -0
+				/* ensure each Tx freed buffer is valid */
+				if (m[i] != NULL)
+					nb_rearm++;
+			}
+
+			if (nb_rearm != n) {
+				txq->nb_tx_free = (uint16_t)(txq->nb_tx_free + txq->tx_rs_thresh);
+				txq->tx_next_dd = (uint16_t)(txq->tx_next_dd + txq->tx_rs_thresh);
+				if (txq->tx_next_dd >= txq->nb_tx_desc)
+					txq->tx_next_dd = (uint16_t)(txq->tx_rs_thresh - 1);

So if nb_rearm != 0 what would happen with mbufs collected in m[]? Are you just dropping/forgetting them?

Quoted 5 lines
drivers/net/i40e/i40e_rxtx_common_avx.h +4 -0
+
+				goto mempool_bulk;
 
+			} else {
+				for (i = 0; i < n; i++)

drivers/net/i40e/i40e_rxtx_common_avx.h i40e_rxq_rearm_common(struct i40e_rx_queue *rxq, __rte_unused bool avx512) · line 288

Quoted 10 lines
drivers/net/i40e/i40e_rxtx_common_avx.h +10 -0
+					rxep[i].mbuf = m[i];
+			}
+		}
+
+		/* update counters for Tx */
+		txq->nb_tx_free = (uint16_t)(txq->nb_tx_free + txq->tx_rs_thresh);
+		txq->tx_next_dd = (uint16_t)(txq->tx_next_dd + txq->tx_rs_thresh);
+		if (txq->tx_next_dd >= txq->nb_tx_desc)
+			txq->tx_next_dd = (uint16_t)(txq->tx_rs_thresh - 1);
+	} else {

853 lines not shown · raw message

Part of this message is not shown here. View the raw message.

#4 lore raw

PATCH 3/5 ethdev: add API for direct rearm mode

In reply to Feifei Wang

Add API for enabling direct rearm mode and for mapping RX and TX queues. Currently, the API supports 1:1(txq : rxq) mapping.

---
 lib/ethdev/ethdev_driver.h | 15 +++++++++++++++
 lib/ethdev/rte_ethdev.c    | 14 ++++++++++++++
 lib/ethdev/rte_ethdev.h    | 31 +++++++++++++++++++++++++++++++
 lib/ethdev/version.map     |  1 +
 4 files changed, 61 insertions(+)
lib/ethdev/ethdev_driver.h +15 -0
index 69d9dc21d8..22022f6da9 100644
@@ -485,6 +485,16 @@ typedef int (*eth_rx_enable_intr_t)(struct rte_eth_dev *dev,
 typedef int (*eth_rx_disable_intr_t)(struct rte_eth_dev *dev,
 				    uint16_t rx_queue_id);
 
+/** @internal Enable direct rearm of a receive queue of an Ethernet device. */
+typedef int (*eth_rx_direct_rearm_enable_t)(struct rte_eth_dev *dev,
+						uint16_t queue_id);
+
+/**< @internal map Rx/Tx queue of direct rearm mode */
+typedef int (*eth_rx_direct_rearm_map_t)(struct rte_eth_dev *dev,
+					uint16_t rx_queue_id,
+					uint16_t tx_port_id,
+					uint16_t tx_queue_id);
+
 /** @internal Release memory resources allocated by given Rx/Tx queue. */
 typedef void (*eth_queue_release_t)(struct rte_eth_dev *dev,
 				    uint16_t queue_id);
@@ -1152,6 +1162,11 @@ struct eth_dev_ops {
 	/** Disable Rx queue interrupt */
 	eth_rx_disable_intr_t      rx_queue_intr_disable;
 
+	/** Enable Rx queue direct rearm mode */
+	eth_rx_direct_rearm_enable_t rx_queue_direct_rearm_enable;
+	/** Map Rx/Tx queue for direct rearm mode */
+	eth_rx_direct_rearm_map_t  rx_queue_direct_rearm_map;
+
 	eth_tx_queue_setup_t       tx_queue_setup;/**< Set up device Tx queue */
 	eth_queue_release_t        tx_queue_release; /**< Release Tx queue */
 	eth_tx_done_cleanup_t      tx_done_cleanup;/**< Free Tx ring mbufs */
lib/ethdev/rte_ethdev.c +14 -0
index 29a3d80466..8e6f0284f4 100644
@@ -2139,6 +2139,20 @@ rte_eth_tx_hairpin_queue_setup(uint16_t port_id, uint16_t tx_queue_id,
 	return eth_err(port_id, ret);
 }
 
+int
+rte_eth_direct_rxrearm_map(uint16_t rx_port_id, uint16_t rx_queue_id,
+		uint16_t tx_port_id, uint16_t tx_queue_id)
+{
+	struct rte_eth_dev *dev;
+
+	dev = &rte_eth_devices[rx_port_id];
+	(*dev->dev_ops->rx_queue_direct_rearm_enable)(dev, rx_queue_id);
+	(*dev->dev_ops->rx_queue_direct_rearm_map)(dev, rx_queue_id,
+			tx_port_id, tx_queue_id);
+
+	return 0;
+}
+
 int
 rte_eth_hairpin_bind(uint16_t tx_port, uint16_t rx_port)
 {
lib/ethdev/rte_ethdev.h +31 -0
index 04cff8ee10..4a431fcbed 100644
@@ -5190,6 +5190,37 @@ __rte_experimental
 int rte_eth_dev_hairpin_capability_get(uint16_t port_id,
 				       struct rte_eth_hairpin_cap *cap);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Enable direct re-arm mode. In this mode the RX queue will be re-armed using
+ * buffers that have completed transmission on the transmit side.
+ *
+ * @note
+ *   It is assumed that the buffers have completed transmission belong to the
+ *   mempool used at the receive side, and have refcnt = 1.
+ *
+ * @param rx_port_id
+ *   Port identifying the receive side.
+ * @param rx_queue_id
+ *   The index of the receive queue identifying the receive side.
+ *   The value must be in the range [0, nb_rx_queue - 1] previously supplied
+ *   to rte_eth_dev_configure().
+ * @param tx_port_id
+ *   Port identifying the transmit side.
+ * @param tx_queue_id
+ *   The index of the transmit queue identifying the transmit side.
+ *   The value must be in the range [0, nb_tx_queue - 1] previously supplied
+ *   to rte_eth_dev_configure().
+ *
+ * @return
+ *   - (0) if successful.
+ */
+__rte_experimental
+int rte_eth_direct_rxrearm_map(uint16_t rx_port_id, uint16_t rx_queue_id,
+			       uint16_t tx_port_id, uint16_t tx_queue_id);
+
 /**
  * @warning
  * @b EXPERIMENTAL: this structure may change without prior notice.
lib/ethdev/version.map +1 -0
index 20391ab29e..68d664498c 100644
@@ -279,6 +279,7 @@ EXPERIMENTAL {
 	rte_flow_async_action_handle_create;
 	rte_flow_async_action_handle_destroy;
 	rte_flow_async_action_handle_update;
+	rte_eth_direct_rxrearm_map;
 };
 
 INTERNAL {
Signature · 1 line
2.25.1
#5 lore raw

RE: [PATCH v1 3/5] ethdev: add API for direct rearm mode

Quoted 42 lines

From: Feifei Wang [mailto:[email protected]]
Sent: Wednesday, 20 April 2022 10.17

Add API for enabling direct rearm mode and for mapping RX and TX queues. Currently, the API supports 1:1(txq : rxq) mapping.

---
 lib/ethdev/ethdev_driver.h | 15 +++++++++++++++
 lib/ethdev/rte_ethdev.c    | 14 ++++++++++++++
 lib/ethdev/rte_ethdev.h    | 31 +++++++++++++++++++++++++++++++
 lib/ethdev/version.map     |  1 +
 4 files changed, 61 insertions(+)
lib/ethdev/ethdev_driver.h +1 -0
index 69d9dc21d8..22022f6da9 100644
@@ -485,6 +485,16 @@ typedef int (*eth_rx_enable_intr_t)(struct
 rte_eth_dev *dev,
 typedef int (*eth_rx_disable_intr_t)(struct rte_eth_dev *dev,
 				    uint16_t rx_queue_id);
 
+/** @internal Enable direct rearm of a receive queue of an Ethernet

device. */

lib/ethdev/ethdev_driver.h +9 -0
+typedef int (*eth_rx_direct_rearm_enable_t)(struct rte_eth_dev *dev,
+						uint16_t queue_id);
+
+/**< @internal map Rx/Tx queue of direct rearm mode */
+typedef int (*eth_rx_direct_rearm_map_t)(struct rte_eth_dev *dev,
+					uint16_t rx_queue_id,
+					uint16_t tx_port_id,
+					uint16_t tx_queue_id);
+
 /** @internal Release memory resources allocated by given Rx/Tx queue.

*/
typedef void (*eth_queue_release_t)(struct rte_eth_dev *dev,

lib/ethdev/ethdev_driver.h +0 -0
 				    uint16_t queue_id);

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1166

Quoted 5 lines
lib/ethdev/ethdev_driver.h +2 -0
@@ -1152,6 +1162,11 @@ struct eth_dev_ops {
 	/** Disable Rx queue interrupt */
 	eth_rx_disable_intr_t      rx_queue_intr_disable;
 
+	/** Enable Rx queue direct rearm mode */
+	eth_rx_direct_rearm_enable_t rx_queue_direct_rearm_enable;

A disable function seems to be missing.

Quoted 18 lines
lib/ethdev/ethdev_driver.h +3 -0
+	/** Map Rx/Tx queue for direct rearm mode */
+	eth_rx_direct_rearm_map_t  rx_queue_direct_rearm_map;
+
 	eth_tx_queue_setup_t       tx_queue_setup;/**< Set up device Tx

queue */
eth_queue_release_t tx_queue_release; /**< Release Tx
queue */
eth_tx_done_cleanup_t tx_done_cleanup;/**< Free Tx ring
mbufs */

lib/ethdev/rte_ethdev.c +0 -0
index 29a3d80466..8e6f0284f4 100644
@@ -2139,6 +2139,20 @@ rte_eth_tx_hairpin_queue_setup(uint16_t port_id,
 uint16_t tx_queue_id,
 	return eth_err(port_id, ret);
 }
 

lib/ethdev/rte_ethdev.c rte_eth_tx_hairpin_queue_setup(uint16_t port_id, · line 2152

Quoted 10 lines
lib/ethdev/rte_ethdev.c +10 -0
+int
+rte_eth_direct_rxrearm_map(uint16_t rx_port_id, uint16_t rx_queue_id,
+		uint16_t tx_port_id, uint16_t tx_queue_id)
+{
+	struct rte_eth_dev *dev;
+
+	dev = &rte_eth_devices[rx_port_id];
+	(*dev->dev_ops->rx_queue_direct_rearm_enable)(dev, rx_queue_id);
+	(*dev->dev_ops->rx_queue_direct_rearm_map)(dev, rx_queue_id,
+			tx_port_id, tx_queue_id);

Here you enable the mapping before you configure it. It could cause the driver to use an uninitialized map, if it processes packets between these two function calls.

Error handling is missing. Not all drivers support this feature, and the parameters should be validated.

Regarding driver support, the driver should also expose a capability flag to the application, similar to the RTE_ETH_DEV_CAPA_RXQ_SHARE or RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE flags. The documentation for this flag could include the description of all the restrictions to using it.

Quoted 50 lines
lib/ethdev/rte_ethdev.c +4 -0
+
+	return 0;
+}
+
 int
 rte_eth_hairpin_bind(uint16_t tx_port, uint16_t rx_port)

{

lib/ethdev/rte_ethdev.h +3 -0
index 04cff8ee10..4a431fcbed 100644
@@ -5190,6 +5190,37 @@ __rte_experimental
 int rte_eth_dev_hairpin_capability_get(uint16_t port_id,
 				       struct rte_eth_hairpin_cap *cap);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior

notice
+ *
+ * Enable direct re-arm mode. In this mode the RX queue will be re- armed using
+ * buffers that have completed transmission on the transmit side.
+ *
+ * @note
+ * It is assumed that the buffers have completed transmission belong
to the
+ * mempool used at the receive side, and have refcnt = 1.
+ *
+ * @param rx_port_id
+ * Port identifying the receive side.
+ * @param rx_queue_id
+ * The index of the receive queue identifying the receive side.
+ * The value must be in the range [0, nb_rx_queue - 1] previously
supplied
+ * to rte_eth_dev_configure().
+ * @param tx_port_id
+ * Port identifying the transmit side.
+ * @param tx_queue_id
+ * The index of the transmit queue identifying the transmit side.
+ * The value must be in the range [0, nb_tx_queue - 1] previously
supplied
+ * to rte_eth_dev_configure().
+ *
+ * @return
+ * - (0) if successful.
+ */

lib/ethdev/rte_ethdev.h +2 -0
+__rte_experimental
+int rte_eth_direct_rxrearm_map(uint16_t rx_port_id, uint16_t

rx_queue_id,

lib/ethdev/rte_ethdev.h __rte_experimental · line 5223

Quoted 2 lines
lib/ethdev/rte_ethdev.h +2 -0
+			       uint16_t tx_port_id, uint16_t tx_queue_id);
+

I agree with the parameters to your proposed API here. Since the relevant use case only needs 1:1 mapping, exposing an API function to take some sort of array with N:M mappings would be premature, and probably not ever come into play anyway.

How do you remove, disable and/or change a mapping?

Quoted 18 lines

/**
* @warning
* @b EXPERIMENTAL: this structure may change without prior notice.

lib/ethdev/version.map +1 -1
index 20391ab29e..68d664498c 100644
@@ -279,6 +279,7 @@ EXPERIMENTAL {
 	rte_flow_async_action_handle_create;
 	rte_flow_async_action_handle_destroy;
 	rte_flow_async_action_handle_update;
+	rte_eth_direct_rxrearm_map;
 };
 
 INTERNAL {
--

2.25.1

#6 lore raw

回复: [PATCH v1 3/5] ethdev: add API for direct rearm mode

Quoted 57 lines

-----邮件原件-----
发件人: Morten Brørup <[email protected]>
发送时间: Wednesday, April 20, 2022 5:59 PM
收件人: Feifei Wang <[email protected]>; [email protected]; Ferruh Yigit <[email protected]>; Andrew Rybchenko <[email protected]>; Ray Kinsella <[email protected]> 抄送: [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; Ruifeng Wang <[email protected]> 主题: RE: [PATCH v1 3/5] ethdev: add API for direct rearm mode

Quoted 45 lines

From: Feifei Wang [mailto:[email protected]]
Sent: Wednesday, 20 April 2022 10.17

Add API for enabling direct rearm mode and for mapping RX and TX queues. Currently, the API supports 1:1(txq : rxq) mapping.

---
 lib/ethdev/ethdev_driver.h | 15 +++++++++++++++
 lib/ethdev/rte_ethdev.c    | 14 ++++++++++++++
 lib/ethdev/rte_ethdev.h    | 31 +++++++++++++++++++++++++++++++
 lib/ethdev/version.map     |  1 +
 4 files changed, 61 insertions(+)
lib/ethdev/ethdev_driver.h +0 -0
index 69d9dc21d8..22022f6da9 100644
@@ -485,6 +485,16 @@ typedef int (*eth_rx_enable_intr_t)(struct

rte_eth_dev *dev, typedef int (*eth_rx_disable_intr_t)(struct rte_eth_dev *dev,

lib/ethdev/ethdev_driver.h +1 -0
 				    uint16_t rx_queue_id);
 
+/** @internal Enable direct rearm of a receive queue of an Ethernet

device. */

lib/ethdev/ethdev_driver.h +9 -0
+typedef int (*eth_rx_direct_rearm_enable_t)(struct rte_eth_dev *dev,
+						uint16_t queue_id);
+
+/**< @internal map Rx/Tx queue of direct rearm mode */ typedef int
+(*eth_rx_direct_rearm_map_t)(struct rte_eth_dev *dev,
+					uint16_t rx_queue_id,
+					uint16_t tx_port_id,
+					uint16_t tx_queue_id);
+
 /** @internal Release memory resources allocated by given Rx/Tx queue.

*/
typedef void (*eth_queue_release_t)(struct rte_eth_dev *dev,

lib/ethdev/ethdev_driver.h +0 -0
 				    uint16_t queue_id);
@@ -1152,6 +1162,11 @@ struct eth_dev_ops {
 	/** Disable Rx queue interrupt */
 	eth_rx_disable_intr_t      rx_queue_intr_disable;
 
Quoted 4 lines

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1166

Quoted 2 lines
lib/ethdev/ethdev_driver.h +2 -0
+	/** Enable Rx queue direct rearm mode */
+	eth_rx_direct_rearm_enable_t rx_queue_direct_rearm_enable;

A disable function seems to be missing.

[Feifei] I will try to use offload bits to enable direct-rearm mode, thus this enable function will be removed and disable function will be unnecessary.

Quoted 27 lines
Quoted 16 lines
lib/ethdev/ethdev_driver.h +3 -0
+	/** Map Rx/Tx queue for direct rearm mode */
+	eth_rx_direct_rearm_map_t  rx_queue_direct_rearm_map;
+
 	eth_tx_queue_setup_t       tx_queue_setup;/**< Set up device Tx

queue */
eth_queue_release_t tx_queue_release; /**< Release Tx
queue */
eth_tx_done_cleanup_t tx_done_cleanup;/**< Free Tx ring
mbufs */

lib/ethdev/rte_ethdev.c index renamed from lib/ethdev/rte_ethdev.c +0 -0

29a3d80466..8e6f0284f4 100644

lib/ethdev/rte_ethdev.c +0 -0
@@ -2139,6 +2139,20 @@ rte_eth_tx_hairpin_queue_setup(uint16_t
 port_id, uint16_t tx_queue_id,

lib/ethdev/rte_ethdev.c rte_eth_tx_hairpin_queue_setup(uint16_t · line 2149

Quoted 10 lines
lib/ethdev/rte_ethdev.c +7 -0
 	return eth_err(port_id, ret);
 }
 
+int
+rte_eth_direct_rxrearm_map(uint16_t rx_port_id, uint16_t rx_queue_id,
+		uint16_t tx_port_id, uint16_t tx_queue_id) {
+	struct rte_eth_dev *dev;
+
+	dev = &rte_eth_devices[rx_port_id];
+	(*dev->dev_ops->rx_queue_direct_rearm_enable)(dev,

rx_queue_id);

Quoted 6 lines

lib/ethdev/rte_ethdev.c rte_eth_tx_hairpin_queue_setup(uint16_t port_id, uint16_t tx_queue_id, · line 2151

Quoted 2 lines
lib/ethdev/rte_ethdev.c +2 -0
+	(*dev->dev_ops->rx_queue_direct_rearm_map)(dev, rx_queue_id,
+			tx_port_id, tx_queue_id);

Here you enable the mapping before you configure it. It could cause the driver to use an uninitialized map, if it processes packets between these two function calls.

[Feifei] I agree with this and will change the code.

Quoted 3 lines

Error handling is missing. Not all drivers support this feature, and the parameters should be validated.

[Feifei] You are right, I think after we use 'rxq->offload' bits, we can use some 'offload bits API' to check if driver can support this.
For the parameters, I will add some check.

Quoted 5 lines

Regarding driver support, the driver should also expose a capability flag to the application, similar to the RTE_ETH_DEV_CAPA_RXQ_SHARE or RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE flags. The documentation for this flag could include the description of all the restrictions to using it.

[Feifei] I will do like this by 'rxq->offload' bits, and add description to the documentation.

Quoted 58 lines
Quoted 50 lines
lib/ethdev/rte_ethdev.c +4 -1
+
+	return 0;
+}
+
 int
 rte_eth_hairpin_bind(uint16_t tx_port, uint16_t rx_port)  { diff
--git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h index

04cff8ee10..4a431fcbed 100644

lib/ethdev/rte_ethdev.h +3 -0
@@ -5190,6 +5190,37 @@ __rte_experimental  int
 rte_eth_dev_hairpin_capability_get(uint16_t port_id,
 				       struct rte_eth_hairpin_cap *cap);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior

notice
+ *
+ * Enable direct re-arm mode. In this mode the RX queue will be re- armed using
+ * buffers that have completed transmission on the transmit side.
+ *
+ * @note
+ * It is assumed that the buffers have completed transmission belong
to the
+ * mempool used at the receive side, and have refcnt = 1.
+ *
+ * @param rx_port_id
+ * Port identifying the receive side.
+ * @param rx_queue_id
+ * The index of the receive queue identifying the receive side.
+ * The value must be in the range [0, nb_rx_queue - 1] previously
supplied
+ * to rte_eth_dev_configure().
+ * @param tx_port_id
+ * Port identifying the transmit side.
+ * @param tx_queue_id
+ * The index of the transmit queue identifying the transmit side.
+ * The value must be in the range [0, nb_tx_queue - 1] previously
supplied
+ * to rte_eth_dev_configure().
+ *
+ * @return
+ * - (0) if successful.
+ */

lib/ethdev/rte_ethdev.h +2 -0
+__rte_experimental
+int rte_eth_direct_rxrearm_map(uint16_t rx_port_id, uint16_t

rx_queue_id,

lib/ethdev/rte_ethdev.h __rte_experimental · line 5223

Quoted 2 lines
lib/ethdev/rte_ethdev.h +2 -0
+			       uint16_t tx_port_id, uint16_t tx_queue_id);
+

I agree with the parameters to your proposed API here. Since the relevant use case only needs 1:1 mapping, exposing an API function to take some sort of array with N:M mappings would be premature, and probably not ever come into play anyway.

Quoted 1 line

How do you remove, disable and/or change a mapping?

[Feifei] It is not recommended that users change the map in the process of sending and receiving packets, which may bring some error risks. If user want to change mapping, he needs to stop the device and call
' rte_eth_direct_rxrearm_map ' API to rewrite the mapping.
Furthermore, for 'rxq->offload', user needs to set it before dev starts. If user want to change it, dev needs to be restarted.

Quoted 19 lines
Quoted 19 lines

/**
* @warning
* @b EXPERIMENTAL: this structure may change without prior notice.

lib/ethdev/version.map index renamed from lib/ethdev/version.map +0 -0

20391ab29e..68d664498c 100644

lib/ethdev/version.map +1 -1
@@ -279,6 +279,7 @@ EXPERIMENTAL {
 	rte_flow_async_action_handle_create;
 	rte_flow_async_action_handle_destroy;
 	rte_flow_async_action_handle_update;
+	rte_eth_direct_rxrearm_map;
 };
 
 INTERNAL {
--

2.25.1

#7 lore raw

Re: [PATCH v1 3/5] ethdev: add API for direct rearm mode

In reply to Feifei Wang
Quoted 52 lines · Feifei Wang

Add API for enabling direct rearm mode and for mapping RX and TX queues. Currently, the API supports 1:1(txq : rxq) mapping.

---
  lib/ethdev/ethdev_driver.h | 15 +++++++++++++++
  lib/ethdev/rte_ethdev.c    | 14 ++++++++++++++
  lib/ethdev/rte_ethdev.h    | 31 +++++++++++++++++++++++++++++++
  lib/ethdev/version.map     |  1 +
  4 files changed, 61 insertions(+)
lib/ethdev/ethdev_driver.h +15 -0
index 69d9dc21d8..22022f6da9 100644
@@ -485,6 +485,16 @@ typedef int (*eth_rx_enable_intr_t)(struct rte_eth_dev *dev,
  typedef int (*eth_rx_disable_intr_t)(struct rte_eth_dev *dev,
  				    uint16_t rx_queue_id);
 
+/** @internal Enable direct rearm of a receive queue of an Ethernet device. */
+typedef int (*eth_rx_direct_rearm_enable_t)(struct rte_eth_dev *dev,
+						uint16_t queue_id);
+
+/**< @internal map Rx/Tx queue of direct rearm mode */
+typedef int (*eth_rx_direct_rearm_map_t)(struct rte_eth_dev *dev,
+					uint16_t rx_queue_id,
+					uint16_t tx_port_id,
+					uint16_t tx_queue_id);
+
  /** @internal Release memory resources allocated by given Rx/Tx queue. */
  typedef void (*eth_queue_release_t)(struct rte_eth_dev *dev,
  				    uint16_t queue_id);
@@ -1152,6 +1162,11 @@ struct eth_dev_ops {
  	/** Disable Rx queue interrupt */
  	eth_rx_disable_intr_t      rx_queue_intr_disable;
 
+	/** Enable Rx queue direct rearm mode */
+	eth_rx_direct_rearm_enable_t rx_queue_direct_rearm_enable;
+	/** Map Rx/Tx queue for direct rearm mode */
+	eth_rx_direct_rearm_map_t  rx_queue_direct_rearm_map;
+
  	eth_tx_queue_setup_t       tx_queue_setup;/**< Set up device Tx queue */
  	eth_queue_release_t        tx_queue_release; /**< Release Tx queue */
  	eth_tx_done_cleanup_t      tx_done_cleanup;/**< Free Tx ring mbufs */
lib/ethdev/rte_ethdev.c +0 -0
index 29a3d80466..8e6f0284f4 100644

lib/ethdev/rte_ethdev.c rte_eth_tx_hairpin_queue_setup(uint16_t port_id, uint16_t tx_queue_id, · line 2148

Quoted 10 lines
lib/ethdev/rte_ethdev.c +7 -0
@@ -2139,6 +2139,20 @@ rte_eth_tx_hairpin_queue_setup(uint16_t port_id, uint16_t tx_queue_id,
  	return eth_err(port_id, ret);
  }
 
+int
+rte_eth_direct_rxrearm_map(uint16_t rx_port_id, uint16_t rx_queue_id,
+		uint16_t tx_port_id, uint16_t tx_queue_id)
+{
+	struct rte_eth_dev *dev;
+
+	dev = &rte_eth_devices[rx_port_id];

I think it is rather control path. So:
We need standard checks that rx_port_id is valid. tx_port_id must be checked as well.
rx_queue_id and tx_queue_id must be checked to be in the rate.

lib/ethdev/rte_ethdev.c rte_eth_tx_hairpin_queue_setup(uint16_t port_id, uint16_t tx_queue_id, · line 2151

Quoted 3 lines
lib/ethdev/rte_ethdev.c +3 -0
+	(*dev->dev_ops->rx_queue_direct_rearm_enable)(dev, rx_queue_id);
+	(*dev->dev_ops->rx_queue_direct_rearm_map)(dev, rx_queue_id,
+			tx_port_id, tx_queue_id);

We must check that function pointers are not NULL as usual. Return values must be checked.
Isn't is safe to setup map and than enable. Otherwise we definitely need disable.
Also, what should happen on Tx port unplug? How to continue if we still have Rx port up and running?

Quoted 15 lines
lib/ethdev/rte_ethdev.c +4 -0
+
+	return 0;
+}
+
  int
  rte_eth_hairpin_bind(uint16_t tx_port, uint16_t rx_port)
  {
lib/ethdev/rte_ethdev.h +0 -0
index 04cff8ee10..4a431fcbed 100644
@@ -5190,6 +5190,37 @@ __rte_experimental
  int rte_eth_dev_hairpin_capability_get(uint16_t port_id,
  				       struct rte_eth_hairpin_cap *cap);
 

lib/ethdev/rte_ethdev.h __rte_experimental · line 5202

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Enable direct re-arm mode. In this mode the RX queue will be re-armed using
+ * buffers that have completed transmission on the transmit side.
+ *
+ * @note
+ *   It is assumed that the buffers have completed transmission belong to the
+ *   mempool used at the receive side, and have refcnt = 1.

I think it is possible to avoid such limitations, but implementation will be less optimized - more checks.

lib/ethdev/rte_ethdev.h __rte_experimental · line 5211

Quoted 9 lines
lib/ethdev/rte_ethdev.h +9 -0
+ *
+ * @param rx_port_id
+ *   Port identifying the receive side.
+ * @param rx_queue_id
+ *   The index of the receive queue identifying the receive side.
+ *   The value must be in the range [0, nb_rx_queue - 1] previously supplied
+ *   to rte_eth_dev_configure().
+ * @param tx_port_id
+ *   Port identifying the transmit side.

I guess there is an assumption that Rx and Tx ports are serviced by the same driver. If so and if it is an API limitation, ethdev layer must check it.

Quoted 27 lines
lib/ethdev/rte_ethdev.h +12 -0
+ * @param tx_queue_id
+ *   The index of the transmit queue identifying the transmit side.
+ *   The value must be in the range [0, nb_tx_queue - 1] previously supplied
+ *   to rte_eth_dev_configure().
+ *
+ * @return
+ *   - (0) if successful.
+ */
+__rte_experimental
+int rte_eth_direct_rxrearm_map(uint16_t rx_port_id, uint16_t rx_queue_id,
+			       uint16_t tx_port_id, uint16_t tx_queue_id);
+
  /**
   * @warning
   * @b EXPERIMENTAL: this structure may change without prior notice.
lib/ethdev/version.map +1 -0
index 20391ab29e..68d664498c 100644
@@ -279,6 +279,7 @@ EXPERIMENTAL {
  	rte_flow_async_action_handle_create;
  	rte_flow_async_action_handle_destroy;
  	rte_flow_async_action_handle_update;
+	rte_eth_direct_rxrearm_map;
  };
 
  INTERNAL {
#8 lore raw

回复: [PATCH v1 3/5] ethdev: add API for direct rearm mode

Quoted 74 lines

-----邮件原件-----
发件人: Andrew Rybchenko <[email protected]> 发送时间: Wednesday, April 20, 2022 6:41 PM
收件人: Feifei Wang <[email protected]>; [email protected]; Ferruh Yigit <[email protected]>; Ray Kinsella <[email protected]> 抄送: [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; Ruifeng Wang <[email protected]> 主题: Re: [PATCH v1 3/5] ethdev: add API for direct rearm mode

lib/ethdev/ethdev_driver.h typedef int (*eth_rx_enable_intr_t)(struct

Quoted 19 lines · Feifei Wang

Add API for enabling direct rearm mode and for mapping RX and TX queues. Currently, the API supports 1:1(txq : rxq) mapping.

---
  lib/ethdev/ethdev_driver.h | 15 +++++++++++++++
  lib/ethdev/rte_ethdev.c    | 14 ++++++++++++++
  lib/ethdev/rte_ethdev.h    | 31 +++++++++++++++++++++++++++++++
  lib/ethdev/version.map     |  1 +
  4 files changed, 61 insertions(+)
lib/ethdev/ethdev_driver.h +0 -0
index 69d9dc21d8..22022f6da9 100644
@@ -485,6 +485,16 @@ typedef int (*eth_rx_enable_intr_t)(struct

rte_eth_dev *dev,

lib/ethdev/ethdev_driver.h typedef int (*eth_rx_enable_intr_t)(struct rte_eth_dev *dev,

Quoted 5 lines
lib/ethdev/ethdev_driver.h +2 -0
  typedef int (*eth_rx_disable_intr_t)(struct rte_eth_dev *dev,
  				    uint16_t rx_queue_id);
 
+/** @internal Enable direct rearm of a receive queue of an Ethernet
+device. */ typedef int (*eth_rx_direct_rearm_enable_t)(struct

rte_eth_dev *dev,

lib/ethdev/ethdev_driver.h typedef int (*eth_rx_enable_intr_t)(struct rte_eth_dev *dev,

Quoted 9 lines
lib/ethdev/ethdev_driver.h +8 -0
+						uint16_t queue_id);
+
+/**< @internal map Rx/Tx queue of direct rearm mode */ typedef int
+(*eth_rx_direct_rearm_map_t)(struct rte_eth_dev *dev,
+					uint16_t rx_queue_id,
+					uint16_t tx_port_id,
+					uint16_t tx_queue_id);
+
  /** @internal Release memory resources allocated by given Rx/Tx queue.

*/

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1170

Quoted 12 lines
lib/ethdev/ethdev_driver.h +5 -0
  typedef void (*eth_queue_release_t)(struct rte_eth_dev *dev,
  				    uint16_t queue_id);
@@ -1152,6 +1162,11 @@ struct eth_dev_ops {
  	/** Disable Rx queue interrupt */
  	eth_rx_disable_intr_t      rx_queue_intr_disable;
 
+	/** Enable Rx queue direct rearm mode */
+	eth_rx_direct_rearm_enable_t rx_queue_direct_rearm_enable;
+	/** Map Rx/Tx queue for direct rearm mode */
+	eth_rx_direct_rearm_map_t  rx_queue_direct_rearm_map;
+
  	eth_tx_queue_setup_t       tx_queue_setup;/**< Set up device Tx

queue */

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1171

Quoted 1 line
lib/ethdev/ethdev_driver.h +0 -0
  	eth_queue_release_t        tx_queue_release; /**< Release Tx queue

*/

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1172

Quoted 1 line
lib/ethdev/ethdev_driver.h +0 -0
  	eth_tx_done_cleanup_t      tx_done_cleanup;/**< Free Tx ring mbufs

*/

lib/ethdev/rte_ethdev.c rte_eth_tx_hairpin_queue_setup(uint16_t

Quoted 5 lines
lib/ethdev/rte_ethdev.c index renamed from lib/ethdev/rte_ethdev.c +0 -0

29a3d80466..8e6f0284f4 100644

lib/ethdev/rte_ethdev.c +0 -0
@@ -2139,6 +2139,20 @@ rte_eth_tx_hairpin_queue_setup(uint16_t

port_id, uint16_t tx_queue_id,

Quoted 3 lines
lib/ethdev/rte_ethdev.c +0 -0
  	return eth_err(port_id, ret);
  }
 
Quoted 11 lines

lib/ethdev/rte_ethdev.c rte_eth_tx_hairpin_queue_setup(uint16_t port_id, uint16_t tx_queue_id, · line 2148

Quoted 6 lines
lib/ethdev/rte_ethdev.c +6 -0
+int
+rte_eth_direct_rxrearm_map(uint16_t rx_port_id, uint16_t rx_queue_id,
+		uint16_t tx_port_id, uint16_t tx_queue_id) {
+	struct rte_eth_dev *dev;
+
+	dev = &rte_eth_devices[rx_port_id];

I think it is rather control path. So:
We need standard checks that rx_port_id is valid. tx_port_id must be checked as well.
rx_queue_id and tx_queue_id must be checked to be in the rate.

[Feifei] You are right, I will add check for these.

Quoted 8 lines

lib/ethdev/rte_ethdev.c rte_eth_tx_hairpin_queue_setup(uint16_t · line 2148

Quoted 2 lines
lib/ethdev/rte_ethdev.c +1 -0
+	(*dev->dev_ops->rx_queue_direct_rearm_enable)(dev,

rx_queue_id);

lib/ethdev/rte_ethdev.c rte_eth_tx_hairpin_queue_setup(uint16_t port_id, uint16_t tx_queue_id, · line 2151

Quoted 2 lines
lib/ethdev/rte_ethdev.c +2 -0
+	(*dev->dev_ops->rx_queue_direct_rearm_map)(dev, rx_queue_id,
+			tx_port_id, tx_queue_id);

We must check that function pointers are not NULL as usual. Return values must be checked.

[Feifei] I agree with this, The check for pointer and return value will be added

Quoted 2 lines

Isn't is safe to setup map and than enable.
Otherwise we definitely need disable.

[Feifei] I will change code that map first and then set 'rxq->offload' to enable direct-rearm mode.

Quoted 2 lines

Also, what should happen on Tx port unplug? How to continue if we still have Rx port up and running?

[Feifei] For direct rearm mode, if Tx port unplug, it means there is no buffer from Tx. And then, Rx will put buffer from mempool as usual for rearm.

Quoted 28 lines
Quoted 17 lines
lib/ethdev/rte_ethdev.c +4 -0
+
+	return 0;
+}
+
  int
  rte_eth_hairpin_bind(uint16_t tx_port, uint16_t rx_port)
  {
lib/ethdev/rte_ethdev.h index renamed from lib/ethdev/rte_ethdev.h +0 -0

04cff8ee10..4a431fcbed 100644

lib/ethdev/rte_ethdev.h +1 -0
@@ -5190,6 +5190,37 @@ __rte_experimental
  int rte_eth_dev_hairpin_capability_get(uint16_t port_id,
  				       struct rte_eth_hairpin_cap *cap);
 
+/**

lib/ethdev/rte_ethdev.h __rte_experimental · line 5203

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior
+notice
+ *
+ * Enable direct re-arm mode. In this mode the RX queue will be
+re-armed using
+ * buffers that have completed transmission on the transmit side.
+ *
+ * @note
+ *   It is assumed that the buffers have completed transmission belong to

the

Quoted 4 lines

lib/ethdev/rte_ethdev.h __rte_experimental · line 5202

Quoted 1 line
lib/ethdev/rte_ethdev.h +1 -0
+ *   mempool used at the receive side, and have refcnt = 1.

I think it is possible to avoid such limitations, but implementation will be less optimized - more checks.

[Feifei] For the first limitation: Rx and Tx buffers should be from the same mempool. If we want to check this, we will add a check for each packet in the data-plane, this will significantly reduce performance. So, it is better to note this for users rather than adding check code.
For the second limitation: refcnt = 1. We have now add code to support 'refcnt = 1' in direct-rearm mode, so this note can be removed in the next version.

Quoted 7 lines

lib/ethdev/rte_ethdev.h __rte_experimental

Quoted 7 lines
lib/ethdev/rte_ethdev.h +6 -0
+ *
+ * @param rx_port_id
+ *   Port identifying the receive side.
+ * @param rx_queue_id
+ *   The index of the receive queue identifying the receive side.
+ *   The value must be in the range [0, nb_rx_queue - 1] previously
Quoted 7 lines

supplied

lib/ethdev/rte_ethdev.h __rte_experimental · line 5211

Quoted 3 lines
lib/ethdev/rte_ethdev.h +3 -0
+ *   to rte_eth_dev_configure().
+ * @param tx_port_id
+ *   Port identifying the transmit side.

I guess there is an assumption that Rx and Tx ports are serviced by the same driver. If so and if it is an API limitation, ethdev layer must check it.

[Feifei] I agree with this. For the check that Rx and Tx port should be the same driver, I will add check for this.

Quoted 30 lines

lib/ethdev/rte_ethdev.h __rte_experimental

Quoted 4 lines
lib/ethdev/rte_ethdev.h +3 -0
+ * @param tx_queue_id
+ *   The index of the transmit queue identifying the transmit side.
+ *   The value must be in the range [0, nb_tx_queue - 1] previously

supplied

lib/ethdev/rte_ethdev.h __rte_experimental

Quoted 7 lines
lib/ethdev/rte_ethdev.h +7 -0
+ *   to rte_eth_dev_configure().
+ *
+ * @return
+ *   - (0) if successful.
+ */
+__rte_experimental
+int rte_eth_direct_rxrearm_map(uint16_t rx_port_id, uint16_t

rx_queue_id,

Quoted 17 lines
lib/ethdev/rte_ethdev.h +2 -0
+			       uint16_t tx_port_id, uint16_t tx_queue_id);
+
  /**
   * @warning
   * @b EXPERIMENTAL: this structure may change without prior notice.
lib/ethdev/version.map index renamed from lib/ethdev/version.map +0 -0

20391ab29e..68d664498c 100644

lib/ethdev/version.map +1 -0
@@ -279,6 +279,7 @@ EXPERIMENTAL {
  	rte_flow_async_action_handle_create;
  	rte_flow_async_action_handle_destroy;
  	rte_flow_async_action_handle_update;
+	rte_eth_direct_rxrearm_map;
  };
 
  INTERNAL {
#9 lore raw

RE: [PATCH v1 3/5] ethdev: add API for direct rearm mode

<snip>

Quoted 89 lines
Quoted 64 lines · Feifei Wang

lib/ethdev/ethdev_driver.h typedef int (*eth_rx_enable_intr_t)(struct

Quoted 19 lines

Add API for enabling direct rearm mode and for mapping RX and TX queues. Currently, the API supports 1:1(txq : rxq) mapping.

---
  lib/ethdev/ethdev_driver.h | 15 +++++++++++++++
  lib/ethdev/rte_ethdev.c    | 14 ++++++++++++++
  lib/ethdev/rte_ethdev.h    | 31 +++++++++++++++++++++++++++++++
  lib/ethdev/version.map     |  1 +
  4 files changed, 61 insertions(+)
lib/ethdev/ethdev_driver.h +0 -0
index 69d9dc21d8..22022f6da9 100644
@@ -485,6 +485,16 @@ typedef int (*eth_rx_enable_intr_t)(struct

rte_eth_dev *dev,

lib/ethdev/ethdev_driver.h typedef int (*eth_rx_enable_intr_t)(struct rte_eth_dev *dev,

Quoted 5 lines
lib/ethdev/ethdev_driver.h +2 -0
  typedef int (*eth_rx_disable_intr_t)(struct rte_eth_dev *dev,
  				    uint16_t rx_queue_id);
 
+/** @internal Enable direct rearm of a receive queue of an Ethernet
+device. */ typedef int (*eth_rx_direct_rearm_enable_t)(struct

rte_eth_dev *dev,

lib/ethdev/ethdev_driver.h typedef int (*eth_rx_enable_intr_t)(struct rte_eth_dev *dev,

Quoted 9 lines
lib/ethdev/ethdev_driver.h +8 -0
+						uint16_t queue_id);
+
+/**< @internal map Rx/Tx queue of direct rearm mode */ typedef int
+(*eth_rx_direct_rearm_map_t)(struct rte_eth_dev *dev,
+					uint16_t rx_queue_id,
+					uint16_t tx_port_id,
+					uint16_t tx_queue_id);
+
  /** @internal Release memory resources allocated by given Rx/Tx queue.

*/

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1170

Quoted 12 lines
lib/ethdev/ethdev_driver.h +5 -0
  typedef void (*eth_queue_release_t)(struct rte_eth_dev *dev,
  				    uint16_t queue_id);
@@ -1152,6 +1162,11 @@ struct eth_dev_ops {
  	/** Disable Rx queue interrupt */
  	eth_rx_disable_intr_t      rx_queue_intr_disable;
 
+	/** Enable Rx queue direct rearm mode */
+	eth_rx_direct_rearm_enable_t rx_queue_direct_rearm_enable;
+	/** Map Rx/Tx queue for direct rearm mode */
+	eth_rx_direct_rearm_map_t  rx_queue_direct_rearm_map;
+
  	eth_tx_queue_setup_t       tx_queue_setup;/**< Set up device Tx

queue */

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1171

Quoted 1 line
lib/ethdev/ethdev_driver.h +0 -0
  	eth_queue_release_t        tx_queue_release; /**< Release Tx queue

*/

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1172

Quoted 1 line
lib/ethdev/ethdev_driver.h +0 -0
  	eth_tx_done_cleanup_t      tx_done_cleanup;/**< Free Tx ring mbufs

*/

lib/ethdev/rte_ethdev.c rte_eth_tx_hairpin_queue_setup(uint16_t

Quoted 5 lines
lib/ethdev/rte_ethdev.c index renamed from lib/ethdev/rte_ethdev.c +0 -0

29a3d80466..8e6f0284f4 100644

lib/ethdev/rte_ethdev.c +0 -0
@@ -2139,6 +2139,20 @@ rte_eth_tx_hairpin_queue_setup(uint16_t

port_id, uint16_t tx_queue_id,

Quoted 3 lines
lib/ethdev/rte_ethdev.c +0 -0
  	return eth_err(port_id, ret);
  }
 
Quoted 11 lines

lib/ethdev/rte_ethdev.c rte_eth_tx_hairpin_queue_setup(uint16_t port_id, uint16_t tx_queue_id, · line 2148

Quoted 6 lines
lib/ethdev/rte_ethdev.c +6 -0
+int
+rte_eth_direct_rxrearm_map(uint16_t rx_port_id, uint16_t rx_queue_id,
+		uint16_t tx_port_id, uint16_t tx_queue_id) {
+	struct rte_eth_dev *dev;
+
+	dev = &rte_eth_devices[rx_port_id];

I think it is rather control path. So:
We need standard checks that rx_port_id is valid. tx_port_id must be checked as well.
rx_queue_id and tx_queue_id must be checked to be in the rate.

[Feifei] You are right, I will add check for these.

Quoted 8 lines

lib/ethdev/rte_ethdev.c rte_eth_tx_hairpin_queue_setup(uint16_t · line 2148

Quoted 2 lines
lib/ethdev/rte_ethdev.c +1 -0
+	(*dev->dev_ops->rx_queue_direct_rearm_enable)(dev,

rx_queue_id);

lib/ethdev/rte_ethdev.c rte_eth_tx_hairpin_queue_setup(uint16_t port_id, uint16_t tx_queue_id, · line 2151

Quoted 2 lines
lib/ethdev/rte_ethdev.c +2 -0
+	(*dev->dev_ops->rx_queue_direct_rearm_map)(dev, rx_queue_id,
+			tx_port_id, tx_queue_id);

We must check that function pointers are not NULL as usual. Return values must be checked.

[Feifei] I agree with this, The check for pointer and return value will be added

Quoted 2 lines

Isn't is safe to setup map and than enable.
Otherwise we definitely need disable.

Quoted 8 lines

[Feifei] I will change code that map first and then set 'rxq->offload' to enable direct-rearm mode.

Quoted 2 lines

Also, what should happen on Tx port unplug? How to continue if we still have Rx port up and running?

[Feifei] For direct rearm mode, if Tx port unplug, it means there is no buffer from Tx.
And then, Rx will put buffer from mempool as usual for rearm.

Andrew, when you say 'TX port unplug', do you mean the 'rte_eth_dev_tx_queue_stop' is called? Is calling 'rte_eth_dev_tx_queue_stop' allowed when the device is running?

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
lib/ethdev/rte_ethdev.c +4 -0
+
+	return 0;
+}
+

<snip>

#10 lore raw

Re: [PATCH v1 3/5] ethdev: add API for direct rearm mode

Quoted 22 lines · Honnappa Nagarahalli
Quoted 22 lines
Quoted 10 lines · Feifei Wang

Review

Quoted 8 lines

Add API for enabling direct rearm mode and for mapping RX and TX queues. Currently, the API supports 1:1(txq : rxq) mapping.

(quoted diff) +1 -0
+	(*dev->dev_ops->rx_queue_direct_rearm_enable)(dev,

rx_queue_id);

Quoted 5 lines

lib/ethdev/rte_ethdev.c rte_eth_tx_hairpin_queue_setup(uint16_t port_id, uint16_t tx_queue_id, · line 2151

Quoted 2 lines
lib/ethdev/rte_ethdev.c +2 -0
+	(*dev->dev_ops->rx_queue_direct_rearm_map)(dev, rx_queue_id,
+			tx_port_id, tx_queue_id);

We must check that function pointers are not NULL as usual. Return values must be checked.

[Feifei] I agree with this, The check for pointer and return value will be added

Quoted 2 lines

Isn't is safe to setup map and than enable.
Otherwise we definitely need disable.

[Feifei] I will change code that map first and then set 'rxq->offload' to enable direct-rearm mode.

Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

Also, what should happen on Tx port unplug? How to continue if we still have Rx port up and running?

[Feifei] For direct rearm mode, if Tx port unplug, it means there is no buffer from Tx.
And then, Rx will put buffer from mempool as usual for rearm.

Andrew, when you say 'TX port unplug', do you mean the 'rte_eth_dev_tx_queue_stop' is called? Is calling 'rte_eth_dev_tx_queue_stop' allowed when the device is running?

I think that deferred start and presence of rte_eth_dev_tx_queue_stop() implies the possibility to stop Tx queue. But, yes, application should care about conditions to have no traffic running to Tx queue.

Anyway, I was talking about hot unplug of the entire device used as Tx port in above config.

#11 lore raw

Re: [PATCH v1 3/5] ethdev: add API for direct rearm mode

In reply to Feifei Wang
Quoted 14 lines · Feifei Wang

Add API for enabling direct rearm mode and for mapping RX and TX queues. Currently, the API supports 1:1(txq : rxq) mapping.

---

+ *
+ * @return
+ * - (0) if successful.
+ */

lib/ethdev/rte_ethdev.h __rte_experimental · line 5222

Quoted 3 lines
lib/ethdev/rte_ethdev.h +3 -0
+__rte_experimental
+int rte_eth_direct_rxrearm_map(uint16_t rx_port_id, uint16_t rx_queue_id,
+                              uint16_t tx_port_id, uint16_t tx_queue_id);

Won't existing rte_eth_hairpin_* APIs work to achieve the same?

#12 lore raw

回复: [PATCH v1 3/5] ethdev: add API for direct rearm mode

Quoted 32 lines

-----邮件原件-----
发件人: Jerin Jacob <[email protected]>
发送时间: Wednesday, April 20, 2022 6:50 PM
收件人: Feifei Wang <[email protected]>
抄送: [email protected]; Ferruh Yigit <[email protected]>; Andrew Rybchenko <[email protected]>; Ray Kinsella <[email protected]>; dpdk-dev <[email protected]>; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; Ruifeng Wang <[email protected]>
主题: Re: [PATCH v1 3/5] ethdev: add API for direct rearm mode

Quoted 14 lines · Feifei Wang

Add API for enabling direct rearm mode and for mapping RX and TX queues. Currently, the API supports 1:1(txq : rxq) mapping.

---

+ *
+ * @return
+ * - (0) if successful.
+ */

Review

Quoted 2 lines
(quoted diff) +2 -0
+__rte_experimental
+int rte_eth_direct_rxrearm_map(uint16_t rx_port_id, uint16_t

rx_queue_id,

Quoted 4 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+                              uint16_t tx_port_id, uint16_t
+tx_queue_id);

Won't existing rte_eth_hairpin_* APIs work to achieve the same?

[Feifei] Thanks for the comment. Look at the hairpin feature which is enabled in MLX5 driver.

I think the most important difference is that hairpin just re-directs the packet from the Rx queue to Tx queue in the same port, and Rx/Tx queue just can record the peer queue id. For direct rearm, it can map Rx queue to the Tx queue which are from different ports. And this needs Rx queue records paired port id and queue id.

Furthermore, hairpin needs to set up new hairpin queue and then it can bind Rx queue to Tx queue. and direct-rearm just can use normal queue to map. This is due to direct rearm needs used buffers and it doesn't care about packet.

#13 lore raw

Re: [PATCH v1 3/5] ethdev: add API for direct rearm mode

In reply to Feifei Wang
Quoted 100 lines · On Wed, 20 Apr 2022 16:16:48 +0800 Feifei Wang <[email protected]> wrote:

Add API for enabling direct rearm mode and for mapping RX and TX queues. Currently, the API supports 1:1(txq : rxq) mapping.

---
 lib/ethdev/ethdev_driver.h | 15 +++++++++++++++
 lib/ethdev/rte_ethdev.c    | 14 ++++++++++++++
 lib/ethdev/rte_ethdev.h    | 31 +++++++++++++++++++++++++++++++
 lib/ethdev/version.map     |  1 +
 4 files changed, 61 insertions(+)
lib/ethdev/ethdev_driver.h +15 -0
index 69d9dc21d8..22022f6da9 100644
@@ -485,6 +485,16 @@ typedef int (*eth_rx_enable_intr_t)(struct rte_eth_dev *dev,
 typedef int (*eth_rx_disable_intr_t)(struct rte_eth_dev *dev,
 				    uint16_t rx_queue_id);
 
+/** @internal Enable direct rearm of a receive queue of an Ethernet device. */
+typedef int (*eth_rx_direct_rearm_enable_t)(struct rte_eth_dev *dev,
+						uint16_t queue_id);
+
+/**< @internal map Rx/Tx queue of direct rearm mode */
+typedef int (*eth_rx_direct_rearm_map_t)(struct rte_eth_dev *dev,
+					uint16_t rx_queue_id,
+					uint16_t tx_port_id,
+					uint16_t tx_queue_id);
+
 /** @internal Release memory resources allocated by given Rx/Tx queue. */
 typedef void (*eth_queue_release_t)(struct rte_eth_dev *dev,
 				    uint16_t queue_id);
@@ -1152,6 +1162,11 @@ struct eth_dev_ops {
 	/** Disable Rx queue interrupt */
 	eth_rx_disable_intr_t      rx_queue_intr_disable;
 
+	/** Enable Rx queue direct rearm mode */
+	eth_rx_direct_rearm_enable_t rx_queue_direct_rearm_enable;
+	/** Map Rx/Tx queue for direct rearm mode */
+	eth_rx_direct_rearm_map_t  rx_queue_direct_rearm_map;
+
 	eth_tx_queue_setup_t       tx_queue_setup;/**< Set up device Tx queue */
 	eth_queue_release_t        tx_queue_release; /**< Release Tx queue */
 	eth_tx_done_cleanup_t      tx_done_cleanup;/**< Free Tx ring mbufs */
lib/ethdev/rte_ethdev.c +14 -0
index 29a3d80466..8e6f0284f4 100644
@@ -2139,6 +2139,20 @@ rte_eth_tx_hairpin_queue_setup(uint16_t port_id, uint16_t tx_queue_id,
 	return eth_err(port_id, ret);
 }
 
+int
+rte_eth_direct_rxrearm_map(uint16_t rx_port_id, uint16_t rx_queue_id,
+		uint16_t tx_port_id, uint16_t tx_queue_id)
+{
+	struct rte_eth_dev *dev;
+
+	dev = &rte_eth_devices[rx_port_id];
+	(*dev->dev_ops->rx_queue_direct_rearm_enable)(dev, rx_queue_id);
+	(*dev->dev_ops->rx_queue_direct_rearm_map)(dev, rx_queue_id,
+			tx_port_id, tx_queue_id);
+
+	return 0;
+}
+
 int
 rte_eth_hairpin_bind(uint16_t tx_port, uint16_t rx_port)
 {
lib/ethdev/rte_ethdev.h +20 -0
index 04cff8ee10..4a431fcbed 100644
@@ -5190,6 +5190,37 @@ __rte_experimental
 int rte_eth_dev_hairpin_capability_get(uint16_t port_id,
 				       struct rte_eth_hairpin_cap *cap);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Enable direct re-arm mode. In this mode the RX queue will be re-armed using
+ * buffers that have completed transmission on the transmit side.
+ *
+ * @note
+ *   It is assumed that the buffers have completed transmission belong to the
+ *   mempool used at the receive side, and have refcnt = 1.
+ *
+ * @param rx_port_id
+ *   Port identifying the receive side.
+ * @param rx_queue_id
+ *   The index of the receive queue identifying the receive side.
+ *   The value must be in the range [0, nb_rx_queue - 1] previously supplied
+ *   to rte_eth_dev_configure().
+ * @param tx_port_id
+ *   Port identifying the transmit side.
+ * @param tx_queue_id

lib/ethdev/rte_ethdev.h __rte_experimental · line 5222

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+ *   The index of the transmit queue identifying the transmit side.
+ *   The value must be in the range [0, nb_tx_queue - 1] previously supplied
+ *   to rte_eth_dev_configure().
+ *
+ * @return
+ *   - (0) if successful.
+ */
+__rte_experimental
+int rte_eth_direct_rxrearm_map(uint16_t rx_port_id, uint16_t rx_queue_id,
+			       uint16_t tx_port_id, uint16_t tx_queue_id);

Just looking at this.

Why is this done via API call and not a flag as part of the receive config? All the other offload and configuration happens via dev config. Doing it this way doesn't follow the existing ethdev model.

#14 lore raw

回复: [PATCH v1 3/5] ethdev: add API for direct rearm mode

Quoted 136 lines

-----邮件原件-----
发件人: Stephen Hemminger <[email protected]>
发送时间: Thursday, April 21, 2022 10:58 PM
收件人: Feifei Wang <[email protected]>
抄送: [email protected]; Ferruh Yigit <[email protected]>; Andrew Rybchenko <[email protected]>; Ray Kinsella <[email protected]>; [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; Ruifeng Wang <[email protected]>
主题: Re: [PATCH v1 3/5] ethdev: add API for direct rearm mode

Quoted 19 lines · On Wed, 20 Apr 2022 16:16:48 +0800 Feifei Wang <[email protected]> wrote:

Add API for enabling direct rearm mode and for mapping RX and TX queues. Currently, the API supports 1:1(txq : rxq) mapping.

---
 lib/ethdev/ethdev_driver.h | 15 +++++++++++++++
 lib/ethdev/rte_ethdev.c    | 14 ++++++++++++++
 lib/ethdev/rte_ethdev.h    | 31 +++++++++++++++++++++++++++++++
 lib/ethdev/version.map     |  1 +
 4 files changed, 61 insertions(+)
lib/ethdev/ethdev_driver.h +0 -0
index 69d9dc21d8..22022f6da9 100644
@@ -485,6 +485,16 @@ typedef int (*eth_rx_enable_intr_t)(struct
Quoted 1 line

rte_eth_dev *dev, typedef int (*eth_rx_disable_intr_t)(struct rte_eth_dev

*dev,

lib/ethdev/ethdev_driver.h typedef int (*eth_rx_enable_intr_t)(struct rte_eth_dev *dev,

Quoted 4 lines
lib/ethdev/ethdev_driver.h +2 -0
 				    uint16_t rx_queue_id);
 
+/** @internal Enable direct rearm of a receive queue of an Ethernet
+device. */ typedef int (*eth_rx_direct_rearm_enable_t)(struct

rte_eth_dev *dev,

Quoted 12 lines
lib/ethdev/ethdev_driver.h +8 -0
+						uint16_t queue_id);
+
+/**< @internal map Rx/Tx queue of direct rearm mode */ typedef int
+(*eth_rx_direct_rearm_map_t)(struct rte_eth_dev *dev,
+					uint16_t rx_queue_id,
+					uint16_t tx_port_id,
+					uint16_t tx_queue_id);
+
 /** @internal Release memory resources allocated by given Rx/Tx

queue. */ typedef void (*eth_queue_release_t)(struct rte_eth_dev *dev,

lib/ethdev/ethdev_driver.h +0 -0
 				    uint16_t queue_id);

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1170

Quoted 9 lines
lib/ethdev/ethdev_driver.h +5 -0
@@ -1152,6 +1162,11 @@ struct eth_dev_ops {
 	/** Disable Rx queue interrupt */
 	eth_rx_disable_intr_t      rx_queue_intr_disable;
 
+	/** Enable Rx queue direct rearm mode */
+	eth_rx_direct_rearm_enable_t rx_queue_direct_rearm_enable;
+	/** Map Rx/Tx queue for direct rearm mode */
+	eth_rx_direct_rearm_map_t  rx_queue_direct_rearm_map;
+
 	eth_tx_queue_setup_t       tx_queue_setup;/**< Set up device Tx

queue */

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1171

Quoted 1 line
lib/ethdev/ethdev_driver.h +0 -0
 	eth_queue_release_t        tx_queue_release; /**< Release Tx queue

*/

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1172

Quoted 1 line
lib/ethdev/ethdev_driver.h +0 -0
 	eth_tx_done_cleanup_t      tx_done_cleanup;/**< Free Tx ring mbufs

*/

lib/ethdev/rte_ethdev.c rte_eth_tx_hairpin_queue_setup(uint16_t

Quoted 5 lines
lib/ethdev/rte_ethdev.c index renamed from lib/ethdev/rte_ethdev.c +0 -0

29a3d80466..8e6f0284f4 100644

lib/ethdev/rte_ethdev.c +0 -0
@@ -2139,6 +2139,20 @@ rte_eth_tx_hairpin_queue_setup(uint16_t

port_id, uint16_t tx_queue_id,

lib/ethdev/rte_ethdev.c rte_eth_tx_hairpin_queue_setup(uint16_t port_id, uint16_t tx_queue_id,

Quoted 10 lines
lib/ethdev/rte_ethdev.c +7 -0
 	return eth_err(port_id, ret);
 }
 
+int
+rte_eth_direct_rxrearm_map(uint16_t rx_port_id, uint16_t rx_queue_id,
+		uint16_t tx_port_id, uint16_t tx_queue_id) {
+	struct rte_eth_dev *dev;
+
+	dev = &rte_eth_devices[rx_port_id];
+	(*dev->dev_ops->rx_queue_direct_rearm_enable)(dev,

rx_queue_id);

Quoted 17 lines
lib/ethdev/rte_ethdev.c +6 -1
+	(*dev->dev_ops->rx_queue_direct_rearm_map)(dev, rx_queue_id,
+			tx_port_id, tx_queue_id);
+
+	return 0;
+}
+
 int
 rte_eth_hairpin_bind(uint16_t tx_port, uint16_t rx_port)  { diff
--git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h index

04cff8ee10..4a431fcbed 100644

lib/ethdev/rte_ethdev.h +1 -0
@@ -5190,6 +5190,37 @@ __rte_experimental  int
 rte_eth_dev_hairpin_capability_get(uint16_t port_id,
 				       struct rte_eth_hairpin_cap *cap);
 
+/**

lib/ethdev/rte_ethdev.h __rte_experimental int · line 5203

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior
+notice
+ *
+ * Enable direct re-arm mode. In this mode the RX queue will be
+re-armed using
+ * buffers that have completed transmission on the transmit side.
+ *
+ * @note
+ *   It is assumed that the buffers have completed transmission belong to

the

lib/ethdev/rte_ethdev.h __rte_experimental

Quoted 7 lines
lib/ethdev/rte_ethdev.h +7 -0
+ *   mempool used at the receive side, and have refcnt = 1.
+ *
+ * @param rx_port_id
+ *   Port identifying the receive side.
+ * @param rx_queue_id
+ *   The index of the receive queue identifying the receive side.
+ *   The value must be in the range [0, nb_rx_queue - 1] previously

supplied

lib/ethdev/rte_ethdev.h __rte_experimental

Quoted 6 lines
lib/ethdev/rte_ethdev.h +6 -0
+ *   to rte_eth_dev_configure().
+ * @param tx_port_id
+ *   Port identifying the transmit side.
+ * @param tx_queue_id
+ *   The index of the transmit queue identifying the transmit side.
+ *   The value must be in the range [0, nb_tx_queue - 1] previously

supplied

lib/ethdev/rte_ethdev.h __rte_experimental

Quoted 7 lines
lib/ethdev/rte_ethdev.h +7 -0
+ *   to rte_eth_dev_configure().
+ *
+ * @return
+ *   - (0) if successful.
+ */
+__rte_experimental
+int rte_eth_direct_rxrearm_map(uint16_t rx_port_id, uint16_t

rx_queue_id,

Quoted 7 lines

lib/ethdev/rte_ethdev.h __rte_experimental · line 5222

Quoted 1 line
lib/ethdev/rte_ethdev.h +1 -0
+			       uint16_t tx_port_id, uint16_t tx_queue_id);

Just looking at this.

Why is this done via API call and not a flag as part of the receive config? All the other offload and configuration happens via dev config. Doing it this way doesn't follow the existing ethdev model.

[Feifei] Agree with this. I will remove direct-rearm enable function and
use "rxq->offload" bit to enable it.
For rte_eth_direct_rxrearm_map, I think it is necessary for users to call it to map Rx/Tx queue.

#15 lore raw

PATCH 4/5 net/i40e: add direct rearm mode internal API

In reply to Feifei Wang

For direct rearm mode, add two internal functions.

One is to enable direct rearm mode in Rx queue.

The other is to map Tx queue with Rx queue to make Rx queue take buffers from the specific Tx queue.

---
 drivers/net/i40e/i40e_ethdev.c | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)
drivers/net/i40e/i40e_ethdev.c +34 -0
index 755786dc10..9e1a523bcc 100644
@@ -369,6 +369,13 @@ static int i40e_dev_rx_queue_intr_enable(struct rte_eth_dev *dev,
 static int i40e_dev_rx_queue_intr_disable(struct rte_eth_dev *dev,
 					  uint16_t queue_id);
 
+static int i40e_dev_rx_queue_direct_rearm_enable(struct rte_eth_dev *dev,
+						uint16_t queue_id);
+static int i40e_dev_rx_queue_direct_rearm_map(struct rte_eth_dev *dev,
+						uint16_t rx_queue_id,
+						uint16_t tx_port_id,
+						uint16_t tx_queue_id);
+
 static int i40e_get_regs(struct rte_eth_dev *dev,
 			 struct rte_dev_reg_info *regs);
 
@@ -477,6 +484,8 @@ static const struct eth_dev_ops i40e_eth_dev_ops = {
 	.rx_queue_setup               = i40e_dev_rx_queue_setup,
 	.rx_queue_intr_enable         = i40e_dev_rx_queue_intr_enable,
 	.rx_queue_intr_disable        = i40e_dev_rx_queue_intr_disable,
+	.rx_queue_direct_rearm_enable = i40e_dev_rx_queue_direct_rearm_enable,
+	.rx_queue_direct_rearm_map    = i40e_dev_rx_queue_direct_rearm_map,
 	.rx_queue_release             = i40e_dev_rx_queue_release,
 	.tx_queue_setup               = i40e_dev_tx_queue_setup,
 	.tx_queue_release             = i40e_dev_tx_queue_release,
@@ -11108,6 +11117,31 @@ i40e_dev_rx_queue_intr_disable(struct rte_eth_dev *dev, uint16_t queue_id)
 	return 0;
 }
 
+static int i40e_dev_rx_queue_direct_rearm_enable(struct rte_eth_dev *dev,
+			uint16_t queue_id)
+{
+	struct i40e_rx_queue *rxq;
+
+	rxq = dev->data->rx_queues[queue_id];
+	rxq->direct_rxrearm_enable = 1;
+
+	return 0;
+}
+
+static int i40e_dev_rx_queue_direct_rearm_map(struct rte_eth_dev *dev,
+				uint16_t rx_queue_id, uint16_t tx_port_id,
+				uint16_t tx_queue_id)
+{
+	struct i40e_rx_queue *rxq;
+
+	rxq = dev->data->rx_queues[rx_queue_id];
+
+	rxq->direct_rxrearm_port = tx_port_id;
+	rxq->direct_rxrearm_queue = tx_queue_id;
+
+	return 0;
+}
+
 /**
  * This function is used to check if the register is valid.
  * Below is the valid registers list for X722 only:
Signature · 1 line
2.25.1
#16 lore raw

Re: [PATCH v1 4/5] net/i40e: add direct rearm mode internal API

20/04/2022 09:16, Feifei Wang пишет:

Quoted 57 lines

For direct rearm mode, add two internal functions.

One is to enable direct rearm mode in Rx queue.

The other is to map Tx queue with Rx queue to make Rx queue take buffers from the specific Tx queue.

---
  drivers/net/i40e/i40e_ethdev.c | 34 ++++++++++++++++++++++++++++++++++
  1 file changed, 34 insertions(+)
drivers/net/i40e/i40e_ethdev.c +20 -0
index 755786dc10..9e1a523bcc 100644
@@ -369,6 +369,13 @@ static int i40e_dev_rx_queue_intr_enable(struct rte_eth_dev *dev,
  static int i40e_dev_rx_queue_intr_disable(struct rte_eth_dev *dev,
  					  uint16_t queue_id);
 
+static int i40e_dev_rx_queue_direct_rearm_enable(struct rte_eth_dev *dev,
+						uint16_t queue_id);
+static int i40e_dev_rx_queue_direct_rearm_map(struct rte_eth_dev *dev,
+						uint16_t rx_queue_id,
+						uint16_t tx_port_id,
+						uint16_t tx_queue_id);
+
  static int i40e_get_regs(struct rte_eth_dev *dev,
  			 struct rte_dev_reg_info *regs);
 
@@ -477,6 +484,8 @@ static const struct eth_dev_ops i40e_eth_dev_ops = {
  	.rx_queue_setup               = i40e_dev_rx_queue_setup,
  	.rx_queue_intr_enable         = i40e_dev_rx_queue_intr_enable,
  	.rx_queue_intr_disable        = i40e_dev_rx_queue_intr_disable,
+	.rx_queue_direct_rearm_enable = i40e_dev_rx_queue_direct_rearm_enable,
+	.rx_queue_direct_rearm_map    = i40e_dev_rx_queue_direct_rearm_map,
  	.rx_queue_release             = i40e_dev_rx_queue_release,
  	.tx_queue_setup               = i40e_dev_tx_queue_setup,
  	.tx_queue_release             = i40e_dev_tx_queue_release,
@@ -11108,6 +11117,31 @@ i40e_dev_rx_queue_intr_disable(struct rte_eth_dev *dev, uint16_t queue_id)
  	return 0;
  }
 
+static int i40e_dev_rx_queue_direct_rearm_enable(struct rte_eth_dev *dev,
+			uint16_t queue_id)
+{
+	struct i40e_rx_queue *rxq;
+
+	rxq = dev->data->rx_queues[queue_id];
+	rxq->direct_rxrearm_enable = 1;
+
+	return 0;
+}
+

drivers/net/i40e/i40e_ethdev.c i40e_dev_rx_queue_intr_disable(struct rte_eth_dev *dev, uint16_t queue_id) · line 11140

Quoted 10 lines
drivers/net/i40e/i40e_ethdev.c +10 -0
+static int i40e_dev_rx_queue_direct_rearm_map(struct rte_eth_dev *dev,
+				uint16_t rx_queue_id, uint16_t tx_port_id,
+				uint16_t tx_queue_id)
+{
+	struct i40e_rx_queue *rxq;
+
+	rxq = dev->data->rx_queues[rx_queue_id];
+
+	rxq->direct_rxrearm_port = tx_port_id;
+	rxq->direct_rxrearm_queue = tx_queue_id;

I don't think this function should not enable that mode blindly. Instead, it needs to check first that all pre-conditions are met (tx/rx threshold values are equal, etc.).

Quoted 7 lines
drivers/net/i40e/i40e_ethdev.c +4 -0
+
+	return 0;
+}
+
  /**
   * This function is used to check if the register is valid.
   * Below is the valid registers list for X722 only:
#17 lore raw

PATCH 5/5 examples/l3fwd: enable direct rearm mode

In reply to Feifei Wang

Enable direct rearm mode. The mapping is decided in the data plane based on the first packet received.

---
 examples/l3fwd/l3fwd_lpm.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)
examples/l3fwd/l3fwd_lpm.c +15 -1
index bec22c44cd..38ffdf4636 100644
@@ -147,7 +147,7 @@ lpm_main_loop(__rte_unused void *dummy)
 	unsigned lcore_id;
 	uint64_t prev_tsc, diff_tsc, cur_tsc;
 	int i, nb_rx;
-	uint16_t portid;
+	uint16_t portid, tx_portid;
 	uint8_t queueid;
 	struct lcore_conf *qconf;
 	const uint64_t drain_tsc = (rte_get_tsc_hz() + US_PER_S - 1) /
@@ -158,6 +158,8 @@ lpm_main_loop(__rte_unused void *dummy)
 
 	const uint16_t n_rx_q = qconf->n_rx_queue;
 	const uint16_t n_tx_p = qconf->n_tx_port;
+	int direct_rearm_map[n_rx_q];
+
 	if (n_rx_q == 0) {
 		RTE_LOG(INFO, L3FWD, "lcore %u has nothing to do\n", lcore_id);
 		return 0;
@@ -169,6 +171,7 @@ lpm_main_loop(__rte_unused void *dummy)
 
 		portid = qconf->rx_queue_list[i].port_id;
 		queueid = qconf->rx_queue_list[i].queue_id;
+		direct_rearm_map[i] = 0;
 		RTE_LOG(INFO, L3FWD,
 			" -- lcoreid=%u portid=%u rxqueueid=%hhu\n",
 			lcore_id, portid, queueid);
@@ -209,6 +212,17 @@ lpm_main_loop(__rte_unused void *dummy)
 			if (nb_rx == 0)
 				continue;
 
+			/* Determine the direct rearm mapping based on the first
+			 * packet received on the rx queue
+			 */
+			if (direct_rearm_map[i] == 0) {
+				tx_portid = lpm_get_dst_port(qconf, pkts_burst[0],
+							portid);
+				rte_eth_direct_rxrearm_map(portid, queueid,
+								tx_portid, queueid);
+				direct_rearm_map[i] = 1;
+			}
+
 #if defined RTE_ARCH_X86 || defined __ARM_NEON \
 			 || defined RTE_ARCH_PPC_64
 			l3fwd_lpm_send_packets(nb_rx, pkts_burst,
Signature · 1 line
2.25.1
#18 lore raw

RE: [PATCH v1 5/5] examples/l3fwd: enable direct rearm mode

Quoted 6 lines

From: Feifei Wang [mailto:[email protected]]
Sent: Wednesday, 20 April 2022 10.17

Enable direct rearm mode. The mapping is decided in the data plane based
on the first packet received.

I usually don't care much about l3fwd, but putting configuration changes in the fast path is just wrong!

Also, l3fwd is often used for benchmarking, and this small piece of code in the fast path will affect benchmark results (although only very little).

Please move it out of the fast path.

#19 lore raw

RE: [PATCH v1 5/5] examples/l3fwd: enable direct rearm mode

<snip>

Quoted 9 lines
Quoted 6 lines

From: Feifei Wang [mailto:[email protected]]
Sent: Wednesday, 20 April 2022 10.17

Enable direct rearm mode. The mapping is decided in the data plane based on the first packet received.

I usually don't care much about l3fwd, but putting configuration changes in the fast path is just wrong!

I would say it depends. In this case the cycles consumed by the API are very less and configuration data is very small and is already in the cache as PMD has accessed the same data structure.

If the configuration needs more cycles than a typical (depending on the application) data plane packet processing needs or brings in enormous amount of data in to the cache, it should not be done on the data plane.

Quoted 3 lines

Also, l3fwd is often used for benchmarking, and this small piece of code in the fast path will affect benchmark results (although only very little).

We do not see any impact on the performance numbers. The reason for putting in the data plane was it covers wider use case in this L3fwd application. If the app were to be simple, the configuration could be done from the control plane. Unfortunately, the performance of L3fwd application matters.

Quoted 2 lines

Please move it out of the fast path.

#20 lore raw

RE: [PATCH v1 5/5] examples/l3fwd: enable direct rearm mode

Quoted 12 lines

From: Honnappa Nagarahalli [mailto:[email protected]]
Sent: Thursday, 21 April 2022 04.35

Quoted 8 lines
Quoted 6 lines

From: Feifei Wang [mailto:[email protected]]
Sent: Wednesday, 20 April 2022 10.17

Enable direct rearm mode. The mapping is decided in the data plane based on the first packet received.

I usually don't care much about l3fwd, but putting configuration

changes in the

Quoted 1 line

fast path is just wrong!

Quoted 9 lines

I would say it depends. In this case the cycles consumed by the API are very less and configuration data is very small and is already in the cache as PMD has accessed the same data structure.

If the configuration needs more cycles than a typical (depending on the application) data plane packet processing needs or brings in enormous amount of data in to the cache, it should not be done on the data plane.

As a matter of principle, configuration changes should be done outside the fast path.

If we allow an exception for this feature, it will set a bad precedent about where to put configuration code.

Quoted 10 lines
Quoted 2 lines

Also, l3fwd is often used for benchmarking, and this small piece of

code in the

Quoted 1 line

fast path will affect benchmark results (although only very little).

We do not see any impact on the performance numbers. The reason for putting in the data plane was it covers wider use case in this L3fwd application. If the app were to be simple, the configuration could be done from the control plane. Unfortunately, the performance of L3fwd application matters.

Let's proceed down that path for the sake of discussion... Then the fast path is missing runtime verification that all preconditions for using remapping are present at any time.

Quoted 2 lines
Quoted 2 lines

Please move it out of the fast path.

BTW, this patch does not call the rte_eth_direct_rxrearm_enable() to enable the feature.

And finally, this feature should be disabled by default, and only enabled by a command line parameter or similar. Otherwise, future l3fwd NIC performance reports will provide misleading performance results, if the feature is utilized. Application developers, when comparing NIC performance results, don't care about the performance for this unique use case; they care about the performance for the generic use case.

#21 lore raw

RE: [PATCH v1 5/5] examples/l3fwd: enable direct rearm mode

(apologies for the late response, this one slipped my mind)

Appreciate if others could weigh their opinions.

<snip>

Quoted 24 lines
Quoted 17 lines

From: Honnappa Nagarahalli [mailto:[email protected]]
Sent: Thursday, 21 April 2022 04.35

Quoted 8 lines
Quoted 6 lines

From: Feifei Wang [mailto:[email protected]]
Sent: Wednesday, 20 April 2022 10.17

Enable direct rearm mode. The mapping is decided in the data plane based on the first packet received.

I usually don't care much about l3fwd, but putting configuration

changes in the

Quoted 1 line

fast path is just wrong!

I would say it depends. In this case the cycles consumed by the API are very less and configuration data is very small and is already in the cache as PMD has accessed the same data structure.

Quoted 6 lines

If the configuration needs more cycles than a typical (depending on the
application) data plane packet processing needs or brings in enormous amount of data in to the cache, it should not be done on the data plane.

Quoted 5 lines

As a matter of principle, configuration changes should be done outside the fast path.

If we allow an exception for this feature, it will set a bad precedent about where to put configuration code.

I think there are other examples though not exactly the same. For ex: the seqlock, we cannot have a scheduled out writer while holding the lock. But, it was mentioned that this can be over come easily by running the writer on an isolated core (which to me breaks some principles).

Quoted 12 lines
Quoted 11 lines
Quoted 3 lines

Also, l3fwd is often used for benchmarking, and this small piece of

code in the

Quoted 1 line

fast path will affect benchmark results (although only very little).

We do not see any impact on the performance numbers. The reason for putting in the data plane was it covers wider use case in this L3fwd application. If the app were to be simple, the configuration could be done from the control plane. Unfortunately, the performance of L3fwd application matters.

Quoted 3 lines

Let's proceed down that path for the sake of discussion... Then the fast path is missing runtime verification that all preconditions for using remapping are present at any time.

Agree, few checks (ensuring that TX and RX buffers are from the same pool, ensuring tx_rs_thresh is same as RX rearm threshold) are missing. We will add these, it is possible to add these checks outside the packet processing loop.

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines

Please move it out of the fast path.

BTW, this patch does not call the rte_eth_direct_rxrearm_enable() to enable the feature.

Quoted 7 lines

And finally, this feature should be disabled by default, and only enabled by a command line parameter or similar. Otherwise, future l3fwd NIC performance reports will provide misleading performance results, if the feature is utilized. Application developers, when comparing NIC performance results, don't care about the performance for this unique use case; they care about the performance for the generic use case.

I think this feature is similar to fast free feature (RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) as you have mentioned in the other thread. It should be handled similar to how fast free feature is handled.

#22 lore raw

RE: [PATCH v1 5/5] examples/l3fwd: enable direct rearm mode

Quoted 39 lines

From: Honnappa Nagarahalli [mailto:[email protected]]
Sent: Wednesday, 11 May 2022 00.02

(apologies for the late response, this one slipped my mind)

Appreciate if others could weigh their opinions.

<snip>

Quoted 8 lines
Quoted 8 lines

From: Honnappa Nagarahalli [mailto:[email protected]]
Sent: Thursday, 21 April 2022 04.35

Quoted 5 lines
Quoted 5 lines

From: Feifei Wang [mailto:[email protected]]
Sent: Wednesday, 20 April 2022 10.17

Enable direct rearm mode. The mapping is decided in the data

plane

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Quoted 1 line

based on the first packet received.

I usually don't care much about l3fwd, but putting configuration

changes in the

Quoted 1 line

fast path is just wrong!

I would say it depends. In this case the cycles consumed by the API are very less and configuration data is very small and is already

in

Quoted 5 lines
Quoted 5 lines

the cache as PMD has accessed the same data structure.

If the configuration needs more cycles than a typical (depending on the
application) data plane packet processing needs or brings in

enormous

Quoted 5 lines
Quoted 3 lines

amount of data in to the cache, it should not be done on the data plane.

As a matter of principle, configuration changes should be done

outside the fast

Quoted 2 lines

path.

Quoted 8 lines
Quoted 1 line

If we allow an exception for this feature, it will set a bad

precedent about

Quoted 1 line

where to put configuration code.

I think there are other examples though not exactly the same. For ex:
the seqlock, we cannot have a scheduled out writer while holding the lock. But, it was mentioned that this can be over come easily by running the writer on an isolated core (which to me breaks some principles).

Referring to a bad example (which breaks some principles) does not change my opinion. ;-)

Quoted 43 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Also, l3fwd is often used for benchmarking, and this small piece

of

Quoted 2 lines
Quoted 2 lines

code in the

Quoted 1 line

fast path will affect benchmark results (although only very

little).

Quoted 2 lines
Quoted 2 lines

We do not see any impact on the performance numbers. The reason for putting in the data plane was it covers wider use case in this

L3fwd

Quoted 1 line
Quoted 1 line

application. If the app were to be simple, the configuration could

be

Quoted 1 line
Quoted 1 line

done from the control plane. Unfortunately, the performance of

L3fwd

Quoted 4 lines
Quoted 2 lines

application matters.

Let's proceed down that path for the sake of discussion... Then the

fast path is

Quoted 1 line

missing runtime verification that all preconditions for using

remapping are

Quoted 1 line

present at any time.

Agree, few checks (ensuring that TX and RX buffers are from the same pool, ensuring tx_rs_thresh is same as RX rearm threshold) are missing. We will add these, it is possible to add these checks outside the packet processing loop.

Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

Please move it out of the fast path.

BTW, this patch does not call the rte_eth_direct_rxrearm_enable() to

enable

Quoted 3 lines

the feature.

And finally, this feature should be disabled by default, and only

enabled by a

Quoted 1 line

command line parameter or similar. Otherwise, future l3fwd NIC

performance

Quoted 1 line

reports will provide misleading performance results, if the feature

is utilized.

Quoted 1 line

Application developers, when comparing NIC performance results, don't

Quoted 8 lines

care

Quoted 3 lines

about the performance for this unique use case; they care about the performance for the generic use case.

I think this feature is similar to fast free feature (RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) as you have mentioned in the other thread. It should be handled similar to how fast free feature is handled.

I agree with this comparison.

Quickly skimming l3fwd/main.c reveals that RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE is used without checking preconditions, and thus might be buggy. E.g. what happens when the NICs support RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE, and l3fwd is run with the "per-port-pool" command line option? Obviously, the "direct rearm" patch should not be punished because of bugs in similar features ("fast free"). But it is not a valid reason to allow similar bugs. You mentioned above that precondition checking will be added, so pardon me for ranting a bit here. ;-)

Furthermore, if using l3fwd for NIC performance reports, I find the results misleading if application specific optimizations are used without mentioning it in the report. This applies to both "fast free" and "direct rearm" optimizations - they only work in specific application scenarios, and thus the l3fwd NIC performance test should be run without these optimization, or at least mention that the report only covers these specific applications. Which is why I prefer that such optimizations must be explicitly enabled through a command line parameter, and not used in testing for official NIC performance reports. Taking one step back, the real problem here is that an *example* application is used for NIC performance testing, and this is the main reason for my performance related objections. I should probably object to using l3fwd for NIC performance testing instead.

I don't feel strongly about l3fwd, so I will not object to the l3fwd patch. Just providing some feedback. :-)

#23 lore raw

Re: [PATCH v1 5/5] examples/l3fwd: enable direct rearm mode

In reply to Feifei Wang

20/04/2022 09:16, Feifei Wang пишет:

Quoted 46 lines

Enable direct rearm mode. The mapping is decided in the data plane based on the first packet received.

---
  examples/l3fwd/l3fwd_lpm.c | 16 +++++++++++++++-
  1 file changed, 15 insertions(+), 1 deletion(-)
examples/l3fwd/l3fwd_lpm.c +5 -1
index bec22c44cd..38ffdf4636 100644
@@ -147,7 +147,7 @@ lpm_main_loop(__rte_unused void *dummy)
  	unsigned lcore_id;
  	uint64_t prev_tsc, diff_tsc, cur_tsc;
  	int i, nb_rx;
-	uint16_t portid;
+	uint16_t portid, tx_portid;
  	uint8_t queueid;
  	struct lcore_conf *qconf;
  	const uint64_t drain_tsc = (rte_get_tsc_hz() + US_PER_S - 1) /
@@ -158,6 +158,8 @@ lpm_main_loop(__rte_unused void *dummy)
 
  	const uint16_t n_rx_q = qconf->n_rx_queue;
  	const uint16_t n_tx_p = qconf->n_tx_port;
+	int direct_rearm_map[n_rx_q];
+
  	if (n_rx_q == 0) {
  		RTE_LOG(INFO, L3FWD, "lcore %u has nothing to do\n", lcore_id);
  		return 0;
@@ -169,6 +171,7 @@ lpm_main_loop(__rte_unused void *dummy)
 
  		portid = qconf->rx_queue_list[i].port_id;
  		queueid = qconf->rx_queue_list[i].queue_id;
+		direct_rearm_map[i] = 0;
  		RTE_LOG(INFO, L3FWD,
  			" -- lcoreid=%u portid=%u rxqueueid=%hhu\n",
  			lcore_id, portid, queueid);
@@ -209,6 +212,17 @@ lpm_main_loop(__rte_unused void *dummy)
  			if (nb_rx == 0)
  				continue;
 
+			/* Determine the direct rearm mapping based on the first

examples/l3fwd/l3fwd_lpm.c lpm_main_loop(__rte_unused void *dummy) · line 225

Quoted 10 lines
examples/l3fwd/l3fwd_lpm.c +10 -0
+			 * packet received on the rx queue
+			 */
+			if (direct_rearm_map[i] == 0) {
+				tx_portid = lpm_get_dst_port(qconf, pkts_burst[0],
+							portid);
+				rte_eth_direct_rxrearm_map(portid, queueid,
+								tx_portid, queueid);
+				direct_rearm_map[i] = 1;
+			}
+

That just doesn't look right to me: why to make decision based on the first packet? What would happen if second and all other packets have to be routed
to different ports?
In fact, this direct-rearm mode seems suitable only for hard-coded
one to one mapped forwarding (examples/l2fwd, testpmd).
For l3fwd it can be used safely only when we have one port in use. Also I think it should be selected at init-time and
it shouldn't be on by default.
To summarize, my opinion:
special cmd-line parameter to enable it.
allowable only when we run l3fwd over one port.

Quoted 3 lines
examples/l3fwd/l3fwd_lpm.c +0 -0
  #if defined RTE_ARCH_X86 || defined __ARM_NEON \
  			 || defined RTE_ARCH_PPC_64
  			l3fwd_lpm_send_packets(nb_rx, pkts_burst,
#24 lore raw

Re: [PATCH v1 5/5] examples/l3fwd: enable direct rearm mode

25/05/2022 01:24, Honnappa Nagarahalli пишет:

Quoted 77 lines

From: Konstantin Ananyev <[email protected]>

20/04/2022 09:16, Feifei Wang пишет:

Quoted 47 lines
Quoted 47 lines

Enable direct rearm mode. The mapping is decided in the data plane based on the first packet received.

---
  examples/l3fwd/l3fwd_lpm.c | 16 +++++++++++++++-

  1 file changed, 15 insertions(+), 1 deletion(-)
examples/l3fwd/l3fwd_lpm.c +1 -1
index bec22c44cd..38ffdf4636 100644
@@ -147,7 +147,7 @@ lpm_main_loop(__rte_unused void *dummy)
       unsigned lcore_id;
       uint64_t prev_tsc, diff_tsc, cur_tsc;
       int i, nb_rx;
-    uint16_t portid;
+    uint16_t portid, tx_portid;
       uint8_t queueid;
       struct lcore_conf *qconf;

      const uint64_t drain_tsc = (rte_get_tsc_hz() + US_PER_S - 1) /

examples/l3fwd/l3fwd_lpm.c +8 -0
@@ -158,6 +158,8 @@ lpm_main_loop(__rte_unused void *dummy)
       const uint16_t n_rx_q = qconf->n_rx_queue;
       const uint16_t n_tx_p = qconf->n_tx_port;
+    int direct_rearm_map[n_rx_q];
+
       if (n_rx_q == 0) {
           RTE_LOG(INFO, L3FWD, "lcore %u has nothing to do\n", lcore_id);
           return 0;
@@ -169,6 +171,7 @@ lpm_main_loop(__rte_unused void *dummy)
           portid = qconf->rx_queue_list[i].port_id;
           queueid = qconf->rx_queue_list[i].queue_id;
+        direct_rearm_map[i] = 0;
           RTE_LOG(INFO, L3FWD,
               " -- lcoreid=%u portid=%u rxqueueid=%hhu\n",
               lcore_id, portid, queueid);
@@ -209,6 +212,17 @@ lpm_main_loop(__rte_unused void *dummy)
               if (nb_rx == 0)
                   continue;
+            /* Determine the direct rearm mapping based on the first
+             * packet received on the rx queue
+             */
+            if (direct_rearm_map[i] == 0) {
+                tx_portid = lpm_get_dst_port(qconf, pkts_burst[0],
Quoted 8 lines

examples/l3fwd/l3fwd_lpm.c lpm_main_loop(__rte_unused void *dummy) · line 225

Quoted 6 lines
examples/l3fwd/l3fwd_lpm.c +6 -0
+                            portid);
+                rte_eth_direct_rxrearm_map(portid, queueid,
+                                tx_portid, queueid);
+                direct_rearm_map[i] = 1;
+            }
+

That just doesn't look right to me: why to make decision based on the first packet?

The TX queue depends on the incoming packet. So, this method covers more scenarios than doing it in the control plane where the outgoing queue is not known.

Quoted 2 lines

What would happen if second and all other packets have to be routed to different ports?

This is an example application and it should be fine to make this assumption. More over, it does not cause any problems if packets change in between. When the packets change back, the feature works again.

Quoted 3 lines

In fact, this direct-rearm mode seems suitable only for hard-coded one to one mapped forwarding (examples/l2fwd, testpmd). For l3fwd it can be used safely only when we have one port in use.

Can you elaborate more on the safety issue when more than one port is used?

Quoted 4 lines

Also I think it should be selected at init-time and it shouldn't be on by default.
To summarize, my opinion:
special cmd-line parameter to enable it.

Quoted 6 lines

Can you please elaborate why a command line parameter is required? Other similar features like RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE are enabled without a command line parameter. IMO, this is how it should ber. Essentially we are trying to measure how different PMDs perform, the ones that have implemented performance improvement features would show better performance (i.e. the PMDs implementing the features should not be penalized by asking for additional user input).

From my perspective, main purpose of l3fwd application is to demonstrate DPDK ability to do packet routing based on input packet contents. Making guesses about packet contents is a change in expected behavior. For some cases it might improve performance, for many others - will most likely cause performance drop.
I think that performance drop as default behavior
(running the same parameters as before) should not be allowed. Plus you did not provided ability to switch off that behavior, if undesired.

About comparison with RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE default enablement - I don't think it is correct. Within l3fwd app we can safely guarantee that all RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE pre-requirements are met:
in each TX queue all mbufs will belong to the same mempool and their refcnt will always equal to one.
Here you are making guesses about contents of input packets, without any guarantee that you guess will always be valid.

BTW, what's wrong with using l2fwd to demonstrate that feature? Seems like a natural choice to me.

Quoted 6 lines
Quoted 6 lines

allowable only when we run l3fwd over one port.

Quoted 3 lines
examples/l3fwd/l3fwd_lpm.c +0 -0
   #if defined RTE_ARCH_X86 || defined __ARM_NEON \

               || defined RTE_ARCH_PPC_64 l3fwd_lpm_send_packets(nb_rx, pkts_burst,

#25 lore raw

RE: [PATCH v1 5/5] examples/l3fwd: enable direct rearm mode

<snip>

Quoted 95 lines

25/05/2022 01:24, Honnappa Nagarahalli пишет:

Quoted 74 lines

From: Konstantin Ananyev <[email protected]>

20/04/2022 09:16, Feifei Wang пишет:

Quoted 50 lines
Quoted 50 lines

Enable direct rearm mode. The mapping is decided in the data plane based on the first packet received.

---
  examples/l3fwd/l3fwd_lpm.c | 16 +++++++++++++++-

  1 file changed, 15 insertions(+), 1 deletion(-)
examples/l3fwd/l3fwd_lpm.c +1 -1
index bec22c44cd..38ffdf4636 100644
@@ -147,7 +147,7 @@ lpm_main_loop(__rte_unused void *dummy)
       unsigned lcore_id;
       uint64_t prev_tsc, diff_tsc, cur_tsc;
       int i, nb_rx;
-    uint16_t portid;
+    uint16_t portid, tx_portid;
       uint8_t queueid;
       struct lcore_conf *qconf;

      const uint64_t drain_tsc = (rte_get_tsc_hz() + US_PER_S - 1) /

examples/l3fwd/l3fwd_lpm.c +3 -0
@@ -158,6 +158,8 @@ lpm_main_loop(__rte_unused void *dummy)
       const uint16_t n_rx_q = qconf->n_rx_queue;
       const uint16_t n_tx_p = qconf->n_tx_port;
+    int direct_rearm_map[n_rx_q];
+
       if (n_rx_q == 0) {
           RTE_LOG(INFO, L3FWD, "lcore %u has nothing to do\n",
 lcore_id);
           return 0;
@@ -169,6 +171,7 @@ lpm_main_loop(__rte_unused void *dummy)
           portid = qconf->rx_queue_list[i].port_id;
           queueid = qconf->rx_queue_list[i].queue_id;
+        direct_rearm_map[i] = 0;
           RTE_LOG(INFO, L3FWD,
               " -- lcoreid=%u portid=%u rxqueueid=%hhu\n",

              lcore_id, portid, queueid); @@ -209,6 +212,17 @@ lpm_main_loop(__rte_unused void *dummy)
              if (nb_rx == 0)
                  continue;
+            /* Determine the direct rearm mapping based on the +first
+             * packet received on the rx queue
+             */
+            if (direct_rearm_map[i] == 0) {
+                tx_portid = lpm_get_dst_port(qconf, pkts_burst[0], +                            portid);

Quoted 8 lines
Quoted 5 lines

+                rte_eth_direct_rxrearm_map(portid, queueid, +                                tx_portid, queueid); +                direct_rearm_map[i] = 1;
+            }
+

That just doesn't look right to me: why to make decision based on the first packet?

The TX queue depends on the incoming packet. So, this method covers more scenarios than doing it in the control plane where the outgoing queue is not known.

Quoted 2 lines

What would happen if second and all other packets have to be routed to different ports?

This is an example application and it should be fine to make this assumption.
More over, it does not cause any problems if packets change in between. When
the packets change back, the feature works again.

Quoted 4 lines
Quoted 3 lines

In fact, this direct-rearm mode seems suitable only for hard-coded one to one mapped forwarding (examples/l2fwd, testpmd). For l3fwd it can be used safely only when we have one port in use.

Can you elaborate more on the safety issue when more than one port is

used?

Quoted 10 lines
Quoted 5 lines

Also I think it should be selected at init-time and it shouldn't be on by default.
To summarize, my opinion:
special cmd-line parameter to enable it.

Can you please elaborate why a command line parameter is required? Other similar features like RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE are enabled without a command line parameter. IMO, this is how it should ber. Essentially we are trying to measure how different PMDs perform, the ones that have implemented performance improvement features

would

Quoted 2 lines

show better performance (i.e. the PMDs implementing the features should not be penalized by asking for additional user input).

Quoted 8 lines

From my perspective, main purpose of l3fwd application is to demonstrate
DPDK ability to do packet routing based on input packet contents. Making guesses about packet contents is a change in expected behavior. For some cases it might improve performance, for many others - will most likely cause performance drop.
I think that performance drop as default behavior (running the same parameters as before) should not be allowed.
Plus you did not provided ability to switch off that behavior, if undesired.

There is no drop in L3fwd performance due to this patch.

Quoted 9 lines

About comparison with RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE default enablement - I don't think it is correct.
Within l3fwd app we can safely guarantee that all RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE pre-requirements are met:
in each TX queue all mbufs will belong to the same mempool and their refcnt will always equal to one.
Here you are making guesses about contents of input packets, without any guarantee that you guess will always be valid.

This is not a guess. The code understands the incoming traffic and configures accordingly. So, it should be correct. Since it is a sample application, we do not expect the traffic to be complex. If it is complex, the performance will be the same as before or better.

Quoted 3 lines

BTW, what's wrong with using l2fwd to demonstrate that feature? Seems like a natural choice to me.

The performance of L3fwd application in DPDK has become a industry standard, hence we need to showcase the performance in L3fwd application.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

allowable only when we run l3fwd over one port.

Quoted 3 lines

  #if defined RTE_ARCH_X86 || defined __ARM_NEON \
               || defined RTE_ARCH_PPC_64 l3fwd_lpm_send_packets(nb_rx, pkts_burst,

#26 lore raw

Re: [PATCH v1 5/5] examples/l3fwd: enable direct rearm mode

Quoted 104 lines · Honnappa Nagarahalli

<snip>

Quoted 95 lines

25/05/2022 01:24, Honnappa Nagarahalli пишет:

Quoted 74 lines

From: Konstantin Ananyev <[email protected]>

20/04/2022 09:16, Feifei Wang пишет:

Quoted 50 lines
Quoted 50 lines

Enable direct rearm mode. The mapping is decided in the data plane based on the first packet received.

---
   examples/l3fwd/l3fwd_lpm.c | 16 +++++++++++++++-
   1 file changed, 15 insertions(+), 1 deletion(-)
examples/l3fwd/l3fwd_lpm.c +4 -1
index bec22c44cd..38ffdf4636 100644
@@ -147,7 +147,7 @@ lpm_main_loop(__rte_unused void *dummy)
       unsigned lcore_id;
       uint64_t prev_tsc, diff_tsc, cur_tsc;
       int i, nb_rx;
-    uint16_t portid;
+    uint16_t portid, tx_portid;
       uint8_t queueid;
       struct lcore_conf *qconf;
       const uint64_t drain_tsc = (rte_get_tsc_hz() + US_PER_S - 1) /
@@ -158,6 +158,8 @@ lpm_main_loop(__rte_unused void *dummy)
       const uint16_t n_rx_q = qconf->n_rx_queue;
       const uint16_t n_tx_p = qconf->n_tx_port;
+    int direct_rearm_map[n_rx_q];
+
       if (n_rx_q == 0) {
           RTE_LOG(INFO, L3FWD, "lcore %u has nothing to do\n",
 lcore_id);
           return 0;
@@ -169,6 +171,7 @@ lpm_main_loop(__rte_unused void *dummy)
           portid = qconf->rx_queue_list[i].port_id;
           queueid = qconf->rx_queue_list[i].queue_id;
+        direct_rearm_map[i] = 0;
           RTE_LOG(INFO, L3FWD,
               " -- lcoreid=%u portid=%u rxqueueid=%hhu\n",
               lcore_id, portid, queueid); @@ -209,6 +212,17 @@
 lpm_main_loop(__rte_unused void *dummy)

              if (nb_rx == 0)
                  continue;
+            /* Determine the direct rearm mapping based on the +first
+             * packet received on the rx queue
+             */
+            if (direct_rearm_map[i] == 0) {
+                tx_portid = lpm_get_dst_port(qconf, pkts_burst[0], +                            portid);

Quoted 8 lines
Quoted 5 lines

+                rte_eth_direct_rxrearm_map(portid, queueid, +                                tx_portid, queueid); +                direct_rearm_map[i] = 1;
+            }
+

That just doesn't look right to me: why to make decision based on the first packet?

The TX queue depends on the incoming packet. So, this method covers more scenarios than doing it in the control plane where the outgoing queue is not known.

Quoted 2 lines

What would happen if second and all other packets have to be routed to different ports?

This is an example application and it should be fine to make this assumption.
More over, it does not cause any problems if packets change in between. When
the packets change back, the feature works again.

Quoted 4 lines
Quoted 3 lines

In fact, this direct-rearm mode seems suitable only for hard-coded one to one mapped forwarding (examples/l2fwd, testpmd). For l3fwd it can be used safely only when we have one port in use.

Can you elaborate more on the safety issue when more than one port is

used?

Quoted 10 lines
Quoted 5 lines

Also I think it should be selected at init-time and it shouldn't be on by default.
To summarize, my opinion:
special cmd-line parameter to enable it.

Can you please elaborate why a command line parameter is required? Other similar features like RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE are enabled without a command line parameter. IMO, this is how it should ber. Essentially we are trying to measure how different PMDs perform, the ones that have implemented performance improvement features

would

Quoted 2 lines

show better performance (i.e. the PMDs implementing the features should not be penalized by asking for additional user input).

Quoted 8 lines

From my perspective, main purpose of l3fwd application is to demonstrate
DPDK ability to do packet routing based on input packet contents. Making guesses about packet contents is a change in expected behavior. For some cases it might improve performance, for many others - will most likely cause performance drop.
I think that performance drop as default behavior (running the same parameters as before) should not be allowed.
Plus you did not provided ability to switch off that behavior, if undesired.

Quoted 1 line

There is no drop in L3fwd performance due to this patch.

It is a questionable statement. Sorry. The patch adds code on data path. So, I'm almost confident that it is possible to find a case when performance slightly drops. May be it is always minor and acceptable.

Quoted 24 lines
Quoted 9 lines

About comparison with RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE default enablement - I don't think it is correct.
Within l3fwd app we can safely guarantee that all RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE pre-requirements are met:
in each TX queue all mbufs will belong to the same mempool and their refcnt will always equal to one.
Here you are making guesses about contents of input packets, without any guarantee that you guess will always be valid.

This is not a guess. The code understands the incoming traffic and configures accordingly. So, it should be correct. Since it is a sample application, we do not expect the traffic to be complex. If it is complex, the performance will be the same as before or better.

Quoted 3 lines

BTW, what's wrong with using l2fwd to demonstrate that feature? Seems like a natural choice to me.

The performance of L3fwd application in DPDK has become a industry standard, hence we need to showcase the performance in L3fwd application.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

allowable only when we run l3fwd over one port.

Quoted 3 lines

  #if defined RTE_ARCH_X86 || defined __ARM_NEON \
               || defined RTE_ARCH_PPC_64 l3fwd_lpm_send_packets(nb_rx, pkts_burst,

#27 lore raw

Re: [PATCH v1 5/5] examples/l3fwd: enable direct rearm mode

In reply to Honnappa Nagarahalli

31/05/2022 18:14, Honnappa Nagarahalli пишет:

Quoted 104 lines

<snip>

Quoted 95 lines

25/05/2022 01:24, Honnappa Nagarahalli пишет:

Quoted 74 lines

From: Konstantin Ananyev <[email protected]>

20/04/2022 09:16, Feifei Wang пишет:

Quoted 50 lines
Quoted 50 lines

Enable direct rearm mode. The mapping is decided in the data plane based on the first packet received.

---
   examples/l3fwd/l3fwd_lpm.c | 16 +++++++++++++++-
   1 file changed, 15 insertions(+), 1 deletion(-)
examples/l3fwd/l3fwd_lpm.c +4 -1
index bec22c44cd..38ffdf4636 100644
@@ -147,7 +147,7 @@ lpm_main_loop(__rte_unused void *dummy)
       unsigned lcore_id;
       uint64_t prev_tsc, diff_tsc, cur_tsc;
       int i, nb_rx;
-    uint16_t portid;
+    uint16_t portid, tx_portid;
       uint8_t queueid;
       struct lcore_conf *qconf;
       const uint64_t drain_tsc = (rte_get_tsc_hz() + US_PER_S - 1) /
@@ -158,6 +158,8 @@ lpm_main_loop(__rte_unused void *dummy)
       const uint16_t n_rx_q = qconf->n_rx_queue;
       const uint16_t n_tx_p = qconf->n_tx_port;
+    int direct_rearm_map[n_rx_q];
+
       if (n_rx_q == 0) {
           RTE_LOG(INFO, L3FWD, "lcore %u has nothing to do\n",
 lcore_id);
           return 0;
@@ -169,6 +171,7 @@ lpm_main_loop(__rte_unused void *dummy)
           portid = qconf->rx_queue_list[i].port_id;
           queueid = qconf->rx_queue_list[i].queue_id;
+        direct_rearm_map[i] = 0;
           RTE_LOG(INFO, L3FWD,
               " -- lcoreid=%u portid=%u rxqueueid=%hhu\n",
               lcore_id, portid, queueid); @@ -209,6 +212,17 @@
 lpm_main_loop(__rte_unused void *dummy)

              if (nb_rx == 0)
                  continue;
+            /* Determine the direct rearm mapping based on the +first
+             * packet received on the rx queue
+             */
+            if (direct_rearm_map[i] == 0) {
+                tx_portid = lpm_get_dst_port(qconf, pkts_burst[0], +                            portid);

Quoted 8 lines
Quoted 5 lines

+                rte_eth_direct_rxrearm_map(portid, queueid, +                                tx_portid, queueid); +                direct_rearm_map[i] = 1;
+            }
+

That just doesn't look right to me: why to make decision based on the first packet?

The TX queue depends on the incoming packet. So, this method covers more scenarios than doing it in the control plane where the outgoing queue is not known.

Quoted 2 lines

What would happen if second and all other packets have to be routed to different ports?

This is an example application and it should be fine to make this assumption.
More over, it does not cause any problems if packets change in between. When
the packets change back, the feature works again.

Quoted 4 lines
Quoted 3 lines

In fact, this direct-rearm mode seems suitable only for hard-coded one to one mapped forwarding (examples/l2fwd, testpmd). For l3fwd it can be used safely only when we have one port in use.

Can you elaborate more on the safety issue when more than one port is

used?

Quoted 10 lines
Quoted 5 lines

Also I think it should be selected at init-time and it shouldn't be on by default.
To summarize, my opinion:
special cmd-line parameter to enable it.

Can you please elaborate why a command line parameter is required? Other similar features like RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE are enabled without a command line parameter. IMO, this is how it should ber. Essentially we are trying to measure how different PMDs perform, the ones that have implemented performance improvement features

would

Quoted 2 lines

show better performance (i.e. the PMDs implementing the features should not be penalized by asking for additional user input).

Quoted 8 lines

From my perspective, main purpose of l3fwd application is to demonstrate
DPDK ability to do packet routing based on input packet contents. Making guesses about packet contents is a change in expected behavior. For some cases it might improve performance, for many others - will most likely cause performance drop.
I think that performance drop as default behavior (running the same parameters as before) should not be allowed.
Plus you did not provided ability to switch off that behavior, if undesired.

Quoted 1 line

There is no drop in L3fwd performance due to this patch.

Hmm..
Are you saying even when your guess is wrong, and you constantly hitting slow-path (check tx_queue first - failure, then allocate from mempool) you didn't observe any performance drop?
There is more work to do, and if workload is cpu-bound,
my guess - it should be noticeable.
Also, from previous experience, quite often even after
tiny changes in rx/tx code-path some slowdown was reported. Usually that happened on some low-end ARM cpus (Marvell, NXP).

Quoted 10 lines
Quoted 9 lines

About comparison with RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE default enablement - I don't think it is correct.
Within l3fwd app we can safely guarantee that all RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE pre-requirements are met:
in each TX queue all mbufs will belong to the same mempool and their refcnt will always equal to one.
Here you are making guesses about contents of input packets, without any guarantee that you guess will always be valid.

This is not a guess. The code understands the incoming traffic and configures accordingly. So, it should be correct.

No, it is not.
Your code makes guess about all incoming traffic
based on just one (first) packet.

Since it is a sample application, we do not expect the traffic to be complex. If it is complex, the performance will be the same as before or better.

I am not talking about anything 'complex' here. Doing dynamic routing based on packet contents is a a basic l3fwd functionality.

Quoted 14 lines
Quoted 4 lines

BTW, what's wrong with using l2fwd to demonstrate that feature? Seems like a natural choice to me.

The performance of L3fwd application in DPDK has become a industry standard, hence we need to showcase the performance in L3fwd application.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

allowable only when we run l3fwd over one port.

Quoted 3 lines

  #if defined RTE_ARCH_X86 || defined __ARM_NEON \
               || defined RTE_ARCH_PPC_64 l3fwd_lpm_send_packets(nb_rx, pkts_burst,

#28 lore raw

RE: [PATCH v1 5/5] examples/l3fwd: enable direct rearm mode

(apologies for being late on replying, catching up after vacation)

<snip>

Quoted 109 lines
Quoted 9 lines
Quoted 9 lines

25/05/2022 01:24, Honnappa Nagarahalli пишет:

Quoted 7 lines

From: Konstantin Ananyev <[email protected]>

20/04/2022 09:16, Feifei Wang пишет:

Quoted 4 lines
Quoted 4 lines

Enable direct rearm mode. The mapping is decided in the data plane based on the first packet received.

Suggested-by Honnappa Nagarahalli

<[email protected]>

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
---
   examples/l3fwd/l3fwd_lpm.c | 16 +++++++++++++++-
   1 file changed, 15 insertions(+), 1 deletion(-)
Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
(quoted diff) +0 -0

b/examples/l3fwd/l3fwd_lpm.c index bec22c44cd..38ffdf4636 100644

examples/l3fwd/l3fwd_lpm.c +0 -0
@@ -147,7 +147,7 @@ lpm_main_loop(__rte_unused void *dummy)
       unsigned lcore_id;
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
examples/l3fwd/l3fwd_lpm.c +1 -1
       uint64_t prev_tsc, diff_tsc, cur_tsc;
       int i, nb_rx;
-    uint16_t portid;
+    uint16_t portid, tx_portid;
       uint8_t queueid;
       struct lcore_conf *qconf;
       const uint64_t drain_tsc = (rte_get_tsc_hz() + US_PER_S -

1) / @@ -158,6 +158,8 @@ lpm_main_loop(__rte_unused void

*dummy)

Quoted 64 lines
Quoted 64 lines
Quoted 47 lines
Quoted 26 lines
Quoted 26 lines

      const uint16_t n_rx_q = qconf->n_rx_queue; const uint16_t n_tx_p = qconf->n_tx_port;
+    int direct_rearm_map[n_rx_q];
+
      if (n_rx_q == 0) {
          RTE_LOG(INFO, L3FWD, "lcore %u has nothing to do\n",
lcore_id);
          return 0;

examples/l3fwd/l3fwd_lpm.c +1 -0
@@ -169,6 +171,7 @@ lpm_main_loop(__rte_unused void *dummy)
           portid = qconf->rx_queue_list[i].port_id;
           queueid = qconf->rx_queue_list[i].queue_id;
+        direct_rearm_map[i] = 0;
           RTE_LOG(INFO, L3FWD,
               " -- lcoreid=%u portid=%u rxqueueid=%hhu\n",
               lcore_id, portid, queueid); @@ -209,6 +212,17 @@
 lpm_main_loop(__rte_unused void *dummy)

              if (nb_rx == 0)
                  continue;
+            /* Determine the direct rearm mapping based on the

examples/l3fwd/l3fwd_lpm.c +7 -0
+first
+             * packet received on the rx queue
+             */
+            if (direct_rearm_map[i] == 0) {
+                tx_portid = lpm_get_dst_port(qconf,
+pkts_burst[0],
+                            portid);
Quoted 8 lines
Quoted 5 lines

+                rte_eth_direct_rxrearm_map(portid, queueid, +                                tx_portid, queueid); +                direct_rearm_map[i] = 1;
+            }
+

That just doesn't look right to me: why to make decision based on the first packet?

The TX queue depends on the incoming packet. So, this method covers more scenarios than doing it in the control plane where the outgoing queue is not known.

Quoted 2 lines

What would happen if second and all other packets have to be routed to different ports?

This is an example application and it should be fine to make this assumption.
More over, it does not cause any problems if packets change in between. When
the packets change back, the feature works again.

Quoted 5 lines
Quoted 3 lines

In fact, this direct-rearm mode seems suitable only for hard-coded one to one mapped forwarding (examples/l2fwd, testpmd). For l3fwd it can be used safely only when we have one port in use.

Can you elaborate more on the safety issue when more than one port is

used?

Quoted 11 lines
Quoted 5 lines

Also I think it should be selected at init-time and it shouldn't be on by default.
To summarize, my opinion:
special cmd-line parameter to enable it.

Can you please elaborate why a command line parameter is required? Other similar features like RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE are enabled without a command line parameter. IMO, this is how it should ber. Essentially we are trying to measure how different PMDs perform, the ones that have implemented performance improvement features

Quoted 6 lines
Quoted 6 lines

would

Quoted 2 lines

show better performance (i.e. the PMDs implementing the features should not be penalized by asking for additional user input).

From my perspective, main purpose of l3fwd application is to
demonstrate DPDK ability to do packet routing based on input packet

contents.

Quoted 6 lines
Quoted 6 lines

Making guesses about packet contents is a change in expected behavior. For some cases it might improve performance, for many others - will most likely cause performance drop.
I think that performance drop as default behavior (running the same parameters as before) should not be allowed.
Plus you did not provided ability to switch off that behavior, if undesired.

Quoted 8 lines
Quoted 1 line

There is no drop in L3fwd performance due to this patch.

Hmm..
Are you saying even when your guess is wrong, and you constantly hitting slow- path (check tx_queue first - failure, then allocate from mempool) you didn't observe any performance drop?
There is more work to do, and if workload is cpu-bound, my guess - it should be noticeable.

Well, you do not have to trust our words. The platform used to test is mentioned in the cover letter. Appreciate if you could test on your platform and provide the results.

Quoted 43 lines

Also, from previous experience, quite often even after tiny changes in rx/tx code-path some slowdown was reported.
Usually that happened on some low-end ARM cpus (Marvell, NXP).

Quoted 10 lines
Quoted 9 lines

About comparison with RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE default enablement - I don't think it is correct.
Within l3fwd app we can safely guarantee that all RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE pre-requirements are met:
in each TX queue all mbufs will belong to the same mempool and their refcnt will always equal to one.
Here you are making guesses about contents of input packets, without any guarantee that you guess will always be valid.

This is not a guess. The code understands the incoming traffic and configures

accordingly. So, it should be correct.

No, it is not.
Your code makes guess about all incoming traffic based on just one (first) packet.

Since it is a sample application, we do not expect the traffic to be complex. If it is complex, the performance will be the same as before or better.

I am not talking about anything 'complex' here.
Doing dynamic routing based on packet contents is a a basic l3fwd functionality.

Quoted 5 lines
Quoted 4 lines

BTW, what's wrong with using l2fwd to demonstrate that feature? Seems like a natural choice to me.

The performance of L3fwd application in DPDK has become a industry

standard, hence we need to showcase the performance in L3fwd application.

Quoted 9 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

allowable only when we run l3fwd over one port.

Quoted 3 lines

  #if defined RTE_ARCH_X86 || defined __ARM_NEON \
               || defined RTE_ARCH_PPC_64 l3fwd_lpm_send_packets(nb_rx, pkts_burst,

#29 lore raw
In reply to Feifei Wang
Quoted 23 lines

Currently, the transmit side frees the buffers into the lcore cache and the receive side allocates buffers from the lcore cache. The transmit side typically frees 32 buffers resulting in 32*8=256B of stores to lcore cache. The receive side allocates 32 buffers and stores them in the receive side software ring, resulting in 32*8=256B of stores and 256B of load from the lcore cache.

This patch proposes a mechanism to avoid freeing to/allocating from the lcore cache. i.e. the receive side will free the buffers from transmit side directly into it's software ring. This will avoid the 256B of loads and stores introduced by the lcore cache. It also frees up the cache lines used by the lcore cache.

However, this solution poses several constraints:

1)The receive queue needs to know which transmit queue it should take the buffers from. The application logic decides which transmit port to use to send out the packets. In many use cases the NIC might have a single port ([1], [2], [3]), in which case a given transmit queue is always mapped to a single receive queue (1:1 Rx queue: Tx queue). This is easy to configure.

Quoted 5 lines

If the NIC has 2 ports (there are several references), then we will have 1:2 (RX queue: TX queue) mapping which is still easy to configure. However, if this is generalized to 'N' ports, the configuration can be long. More over the PMD would have to scan a list of transmit queues to pull the buffers from.

Just to re-iterate some generic concerns about this proposal:
- We effectively link RX and TX queues - when this feature is enabled, user can't stop TX queue without stopping linked RX queue first. Right now user is free to start/stop any queues at his will. If that feature will allow to link queues from different ports, then even ports will become dependent and user will have to pay extra care when managing such ports.
- very limited usage scenario - it will have a positive effect only when we have a fixed forwarding mapping: all (or nearly all) packets from the RX queue are forwarded into the same TX queue.

Wonder did you had a chance to consider mempool-cache ZC API,
similar to one we have for the ring?
It would allow us on TX free path to avoid copying mbufs to
temporary array on the stack.
Instead we can put them straight from TX SW ring to the mempool cache.
That should save extra store/load for mbuf and might help to achieve some performance gain without by-passing mempool. It probably wouldn't be as fast as what you proposing,
but might be fast enough to consider as alternative.
Again, it would be a generic one, so we can avoid all
these implications and limitations.

Quoted 92 lines

2)The other factor that needs to be considered is 'run-to-completion' vs 'pipeline' models. In the run-to-completion model, the receive side and the transmit side are running on the same lcore serially. In the pipeline model. The receive side and transmit side might be running on different lcores in parallel. This requires locking. This is not supported at this point.

3)Tx and Rx buffers must be from the same mempool. And we also must ensure Tx buffer free number is equal to Rx buffer free number:
(txq->tx_rs_thresh == RTE_I40E_RXQ_REARM_THRESH)
Thus, 'tx_next_dd' can be updated correctly in direct-rearm mode. This is due to tx_next_dd is a variable to compute tx sw-ring free location. Its value will be one more round than the position where next time free starts.

Current status in this RFC:
1)An API is added to allow for mapping a TX queue to a RX queue.
Currently it supports 1:1 mapping.
2)The i40e driver is changed to do the direct re-arm of the receive
side.
3)L3fwd application is modified to do the direct rearm mapping automatically without user config. This follows the rules that the thread can map TX queue to a RX queue based on the first received package destination port.

Testing status:
1.The testing results for L3fwd are as follows:
-------------------------------------------------------------------
enabled direct rearm
-------------------------------------------------------------------
Arm:
N1SDP(neon path):
without fast-free mode with fast-free mode

	+14.1%				+7.0%

Ampere Altra(neon path):
without fast-free mode with fast-free mode

	+17.1				+14.0%

X86:
Dell-8268(limit frequency):
sse path:
without fast-free mode with fast-free mode

	+6.96%				+2.02%
avx2 path:
without fast-free mode		with fast-free mode
	+9.04%				+7.75%
avx512 path:
without fast-free mode		with fast-free mode
	+5.43%				+1.57%

-------------------------------------------------------------------
This patch can not affect base performance of normal mode. Furthermore, the reason for that limiting the CPU frequency is that dell-8268 can encounter i40e NIC bottleneck with maximum frequency.

2.The testing results for VPP-L3fwd are as follows:
-------------------------------------------------------------------
Arm:
N1SDP(neon path):
with direct re-arm mode enabled

	+7.0%

-------------------------------------------------------------------
For Ampere Altra and X86,VPP-L3fwd test has not been done.

Reference:
[1] https://store.nvidia.com/en-us/networking/store/product/MCX623105AN-CDAT/NVIDIAMCX623105ANCDATConnectX6DxENAdapterCard100GbECryptoDisabled/
[2] https://www.intel.com/content/www/us/en/products/sku/192561/intel-ethernet-network-adapter-e810cqda1/specifications.html
[3] https://www.broadcom.com/products/ethernet-connectivity/network-adapters/100gb-nic-ocp/n1100g

Feifei Wang (5):
net/i40e: remove redundant Dtype initialization net/i40e: enable direct rearm mode
ethdev: add API for direct rearm mode net/i40e: add direct rearm mode internal API examples/l3fwd: enable direct rearm mode

  drivers/net/i40e/i40e_ethdev.c          |  34 +++
  drivers/net/i40e/i40e_rxtx.c            |   4 -
  drivers/net/i40e/i40e_rxtx.h            |   4 +
  drivers/net/i40e/i40e_rxtx_common_avx.h | 269 ++++++++++++++++++++++++
  drivers/net/i40e/i40e_rxtx_vec_avx2.c   |  14 +-
  drivers/net/i40e/i40e_rxtx_vec_avx512.c | 249 +++++++++++++++++++++-
  drivers/net/i40e/i40e_rxtx_vec_neon.c   | 141 ++++++++++++-
  drivers/net/i40e/i40e_rxtx_vec_sse.c    | 170 ++++++++++++++-
  examples/l3fwd/l3fwd_lpm.c              |  16 +-
  lib/ethdev/ethdev_driver.h              |  15 ++
  lib/ethdev/rte_ethdev.c                 |  14 ++
  lib/ethdev/rte_ethdev.h                 |  31 +++
  lib/ethdev/version.map                  |   1 +
  13 files changed, 949 insertions(+), 13 deletions(-)

A message that is not in the archive (sent off-list or not yet received): [email protected]

#31 lore raw

16/05/2022 07:10, Feifei Wang пишет:

Quoted 37 lines
Quoted 29 lines
Quoted 23 lines

Currently, the transmit side frees the buffers into the lcore cache and the receive side allocates buffers from the lcore cache. The transmit side typically frees 32 buffers resulting in 32*8=256B of stores to lcore cache. The receive side allocates 32 buffers and stores them in the receive side software ring, resulting in 32*8=256B of stores and 256B of load from the lcore cache.

This patch proposes a mechanism to avoid freeing to/allocating from the lcore cache. i.e. the receive side will free the buffers from transmit side directly into it's software ring. This will avoid the 256B of loads and stores introduced by the lcore cache. It also frees up the cache lines used by the lcore cache.

However, this solution poses several constraints:

1)The receive queue needs to know which transmit queue it should take the buffers from. The application logic decides which transmit port to use to send out the packets. In many use cases the NIC might have a single port ([1], [2], [3]), in which case a given transmit queue is always mapped to a single receive queue (1:1 Rx queue: Tx queue). This is easy to configure.

Quoted 5 lines

If the NIC has 2 ports (there are several references), then we will have 1:2 (RX queue: TX queue) mapping which is still easy to configure. However, if this is generalized to 'N' ports, the configuration can be long. More over the PMD would have to scan a list of transmit queues to pull the buffers from.

Quoted 7 lines

Just to re-iterate some generic concerns about this proposal:
  - We effectively link RX and TX queues - when this feature is enabled, user can't stop TX queue without stopping linked RX queue first. Right now user is free to start/stop any queues at his will. If that feature will allow to link queues from different ports, then even ports will become dependent and user will have to pay extra care when managing such ports.

Quoted 4 lines

[Feifei] When direct rearm enabled, there are two path for thread to choose. If there are enough Tx freed buffers, Rx can put buffers from Tx. Otherwise, Rx will put buffers from mempool as usual. Thus, users do not need to pay much attention managing ports.

What I am talking about: right now different port or different queues of
the same port can be treated as independent entities:
in general user is free to start/stop (and even reconfigure in some cases) one entity without need to stop other entity. I.E user can stop and re-configure TX queue while keep receiving packets
from RX queue.
With direct re-arm enabled, I think it wouldn't be possible any more:
before stopping/reconfiguring TX queue user would have make sure that
corresponding RX queue wouldn't be used by datapath.

Quoted 7 lines
Quoted 4 lines

- very limited usage scenario - it will have a positive effect only when we have a fixed forwarding mapping: all (or nearly all) packets from the RX queue are forwarded into the same TX queue.

[Feifei] Although the usage scenario is limited, this usage scenario has a wide range of applications, such as NIC with one port.

yes, there are NICs with one port, but no guarantee there wouldn't be several such NICs within the system.

Quoted 16 lines

Furtrhermore, I think this is a tradeoff between performance and flexibility. Our goal is to achieve best performance, this means we need to give up some flexibility decisively. For example of 'FAST_FREE Mode', it deletes most of the buffer check (refcnt > 1, external buffer, chain buffer), chooses a shorest path, and then achieve significant performance improvement.

Quoted 10 lines

Wonder did you had a chance to consider mempool-cache ZC API,
similar to one we have for the ring?
It would allow us on TX free path to avoid copying mbufs to
temporary array on the stack.
Instead we can put them straight from TX SW ring to the mempool cache.
That should save extra store/load for mbuf and might help to achieve some performance gain without by-passing mempool. It probably wouldn't be as fast as what you proposing,
but might be fast enough to consider as alternative.
Again, it would be a generic one, so we can avoid all
these implications and limitations.

Quoted 3 lines

[Feifei] I think this is a good try. However, the most important thing is that if we can bypass the mempool decisively to pursue the significant performance gains.

I understand the intention, and I personally think this is wrong
and dangerous attitude.
We have mempool abstraction in place for very good reason.
So we need to try to improve mempool performance (and API if necessary) at first place, not to avoid it and break our own rules and recommendations.

Quoted 5 lines

For ZC, there maybe a problem for it in i40e. The reason for that put Tx buffers into temporary is that i40e_tx_entry includes buffer pointer and index. Thus we cannot put Tx SW_ring entry into mempool directly, we need to firstlt extract mbuf pointer. Finally, though we use ZC, we still can't avoid using a temporary stack to extract Tx buffer pointers.

When talking about ZC API for mempool cache I meant something like:
void ** mempool_cache_put_zc_start(struct rte_mempool_cache *mc, uint32_t *nb_elem, uint32_t flags);
void mempool_cache_put_zc_finish(struct rte_mempool_cache *mc, uint32_t nb_elem);
i.e. _start_ will return user a pointer inside mp-cache where to put free elems and max number of slots that can be safely filled. _finish_ will update mc->len.
As an example:

/* expect to free N mbufs */
uint32_t n = N;
void **p = mempool_cache_put_zc_start(mc, &n, ...);

/* free up to n elems */
for (i = 0; i != n; i++) {

/* get next free mbuf from somewhere */
mb = extract_and_prefree_mbuf(...);

/* no more free mbufs for now */
if (mb == NULL)

     break;

p[i] = mb;
}

/* finalize ZC put, with _i_ freed elems */
mempool_cache_put_zc_finish(mc, i);

That way, I think we can overcome the issue with i40e_tx_entry you mentioned above. Plus it might be useful in other similar places.

Another alternative is obviously to split i40e_tx_entry into two structs (one for mbuf, second for its metadata) and have a separate array for each of them. Though with that approach we need to make sure no perf drops will be introduced, plus probably more code changes will be required.

#32 lore raw
Quoted 26 lines

From: Konstantin Ananyev [mailto:[email protected]]
Sent: Tuesday, 24 May 2022 03.26

Quoted 3 lines

Furtrhermore, I think this is a tradeoff between performance and flexibility.
Our goal is to achieve best performance, this means we need to give

up some

Quoted 1 line

flexibility decisively. For example of 'FAST_FREE Mode', it deletes

most

Quoted 1 line

of the buffer check (refcnt > 1, external buffer, chain buffer),

chooses a

Quoted 6 lines

shorest path, and then achieve significant performance improvement.

Quoted 5 lines

Wonder did you had a chance to consider mempool-cache ZC API, similar to one we have for the ring?
It would allow us on TX free path to avoid copying mbufs to temporary array on the stack.
Instead we can put them straight from TX SW ring to the mempool

cache.

Quoted 7 lines
Quoted 6 lines

That should save extra store/load for mbuf and might help to achieve some performance gain without by-passing mempool.
It probably wouldn't be as fast as what you proposing, but might be fast enough to consider as alternative. Again, it would be a generic one, so we can avoid all these implications and limitations.

Quoted 8 lines
Quoted 1 line

[Feifei] I think this is a good try. However, the most important

thing

Quoted 2 lines

is that if we can bypass the mempool decisively to pursue the significant performance gains.

I understand the intention, and I personally think this is wrong and dangerous attitude.
We have mempool abstraction in place for very good reason.

Yes, but the abstraction is being violated grossly elsewhere, and mempool code is copy-pasted elsewhere too.

A good example of the current situation is [1]. The cache multiplier (a definition private to the mempool library) is required for some copy-pasted code, and the solution is to expose the private definition and make it part of the public API.

[1] http://inbox.dpdk.org/dev/DM4PR12MB53893BF4C7861068FE8A943BDFFE9@DM4PR12MB5389.namprd12.prod.outlook.com/

The game of abstraction has already been lost. Performance won. :-(

Since we allow bypassing the mbuf/mempool library for other features, it should be allowed for this feature too.

I would even say: Why are the drivers using the mempool library, and not the mbuf library, when freeing and allocating mbufs? This behavior bypasses all the debug assertions in the mbuf library.

As you can probably see, I'm certainly not happy about the abstraction violations in DPDK. But they have been allowed for similar features, so they should be allowed here too.

Quoted 43 lines

So we need to try to improve mempool performance (and API if necessary) at first place, not to avoid it and break our own rules and recommendations.

Quoted 1 line

For ZC, there maybe a problem for it in i40e. The reason for that put

Tx

Quoted 2 lines

buffers
into temporary is that i40e_tx_entry includes buffer pointer and

index.

Quoted 2 lines

Thus we cannot put Tx SW_ring entry into mempool directly, we need to firstlt extract mbuf pointer. Finally, though we use ZC, we still

can't

Quoted 2 lines

avoid
using a temporary stack to extract Tx buffer pointers.

When talking about ZC API for mempool cache I meant something like:
void ** mempool_cache_put_zc_start(struct rte_mempool_cache *mc, uint32_t *nb_elem, uint32_t flags);
void mempool_cache_put_zc_finish(struct rte_mempool_cache *mc, uint32_t nb_elem);
i.e. _start_ will return user a pointer inside mp-cache where to put free elems and max number of slots that can be safely filled. _finish_ will update mc->len.
As an example:

/* expect to free N mbufs */
uint32_t n = N;
void **p = mempool_cache_put_zc_start(mc, &n, ...);

/* free up to n elems */
for (i = 0; i != n; i++) {

   /* get next free mbuf from somewhere */
   mb = extract_and_prefree_mbuf(...);

   /* no more free mbufs for now */
   if (mb == NULL)
      break;

   p[i] = mb;
}
Quoted 5 lines

/* finalize ZC put, with _i_ freed elems */
mempool_cache_put_zc_finish(mc, i);

That way, I think we can overcome the issue with i40e_tx_entry you mentioned above. Plus it might be useful in other similar places.

Great example. This would fit perfectly into the i40e driver, if it didn't already implement the exact same by accessing the mempool cache structure directly. :-(

BTW: I tried patching the mempool library to fix some asymmetry bugs [2], but couldn't get any ACKs for it. It seems to me that the community is too risk averse to dare to modify such a core library.

[2] http://inbox.dpdk.org/dev/[email protected]/

However, adding your mempool ZC feature is not a modification, but an addition, so it should be able to gather support.

#33 lore raw
In reply to Konstantin Ananyev

<snip>

Quoted 50 lines

[[email protected] appears similar to someone who previously sent you email, but may not be that person. Learn why this could be a risk at https://aka.ms/LearnAboutSenderIdentification.]

16/05/2022 07:10, Feifei Wang пишет:

Quoted 38 lines
Quoted 30 lines
Quoted 23 lines

Currently, the transmit side frees the buffers into the lcore cache and the receive side allocates buffers from the lcore cache. The transmit side typically frees 32 buffers resulting in 32*8=256B of stores to lcore cache. The receive side allocates 32 buffers and stores them in the receive side software ring, resulting in 32*8=256B of stores and 256B of load from the lcore cache.

This patch proposes a mechanism to avoid freeing to/allocating from the lcore cache. i.e. the receive side will free the buffers from transmit side directly into it's software ring. This will avoid the 256B of loads and stores introduced by the lcore cache. It also frees up the cache lines used by the lcore cache.

However, this solution poses several constraints:

1)The receive queue needs to know which transmit queue it should take the buffers from. The application logic decides which transmit port to use to send out the packets. In many use cases the NIC might have a single port ([1], [2], [3]), in which case a given transmit queue is always mapped to a single receive queue (1:1 Rx queue: Tx queue). This is easy to configure.

Quoted 6 lines

If the NIC has 2 ports (there are several references), then we will have
1:2 (RX queue: TX queue) mapping which is still easy to configure. However, if this is generalized to 'N' ports, the configuration can be long. More over the PMD would have to scan a list of transmit queues to pull the buffers from.

Quoted 7 lines

Just to re-iterate some generic concerns about this proposal:
- We effectively link RX and TX queues - when this feature is enabled, user can't stop TX queue without stopping linked RX queue first. Right now user is free to start/stop any queues at his will. If that feature will allow to link queues from different ports, then even ports will become dependent and user will have to pay extra care when managing such ports.

Quoted 5 lines

[Feifei] When direct rearm enabled, there are two path for thread to choose. If there are enough Tx freed buffers, Rx can put buffers from Tx.
Otherwise, Rx will put buffers from mempool as usual. Thus, users do not need to pay much attention managing ports.

Quoted 9 lines

What I am talking about: right now different port or different queues of the same port can be treated as independent entities:
in general user is free to start/stop (and even reconfigure in some cases) one entity without need to stop other entity.
I.E user can stop and re-configure TX queue while keep receiving packets from RX queue.
With direct re-arm enabled, I think it wouldn't be possible any more:
before stopping/reconfiguring TX queue user would have make sure that corresponding RX queue wouldn't be used by datapath.

I am trying to understand the problem better. For the TX queue to be stopped, the user must have blocked the data plane from accessing the TX queue. Like Feifei says, the RX side has the normal packet allocation path still available. Also this sounds like a corner case to me, we can handle this through checks in the queue_stop API.

Quoted 11 lines
Quoted 8 lines
Quoted 5 lines

- very limited usage scenario - it will have a positive effect only when we have a fixed forwarding mapping: all (or nearly all) packets from the RX queue are forwarded into the same TX queue.

[Feifei] Although the usage scenario is limited, this usage scenario has a wide range of applications, such as NIC with one port.

yes, there are NICs with one port, but no guarantee there wouldn't be several such NICs within the system.

What I see in my interactions is, a single NIC/DPU is under utilized for a 2 socket system. Some are adding more sockets to the system to better utilize the DPU. The NIC bandwidth continues to grow significantly. I do not think there will be a multi-DPU per server scenario.

Quoted 24 lines
Quoted 7 lines

Furtrhermore, I think this is a tradeoff between performance and flexibility.
Our goal is to achieve best performance, this means we need to give up some flexibility decisively. For example of 'FAST_FREE Mode', it deletes most of the buffer check (refcnt > 1, external buffer, chain buffer), chooses a shorest path, and then achieve significant performance

improvement.

Quoted 12 lines
Quoted 11 lines

Wonder did you had a chance to consider mempool-cache ZC API, similar to one we have for the ring?
It would allow us on TX free path to avoid copying mbufs to temporary array on the stack.
Instead we can put them straight from TX SW ring to the mempool cache. That should save extra store/load for mbuf and might help to achieve some performance gain without by-passing mempool.
It probably wouldn't be as fast as what you proposing, but might be fast enough to consider as alternative.
Again, it would be a generic one, so we can avoid all these implications and limitations.

Quoted 3 lines

[Feifei] I think this is a good try. However, the most important thing is that if we can bypass the mempool decisively to pursue the significant performance gains.

Quoted 5 lines

I understand the intention, and I personally think this is wrong and dangerous attitude.
We have mempool abstraction in place for very good reason.
So we need to try to improve mempool performance (and API if necessary) at first place, not to avoid it and break our own rules and recommendations.

The abstraction can be thought of at a higher level. i.e. the driver manages the buffer allocation/free and is hidden from the application. The application does not need to be aware of how these changes are implemented.

Quoted 38 lines
Quoted 8 lines

For ZC, there maybe a problem for it in i40e. The reason for that put Tx buffers into temporary is that i40e_tx_entry includes buffer pointer and index.
Thus we cannot put Tx SW_ring entry into mempool directly, we need to firstlt extract mbuf pointer. Finally, though we use ZC, we still can't avoid using a temporary stack to extract Tx buffer pointers.

When talking about ZC API for mempool cache I meant something like:
void ** mempool_cache_put_zc_start(struct rte_mempool_cache *mc, uint32_t *nb_elem, uint32_t flags); void mempool_cache_put_zc_finish(struct rte_mempool_cache *mc, uint32_t nb_elem); i.e. _start_ will return user a pointer inside mp-cache where to put free elems and max number of slots that can be safely filled.
_finish_ will update mc->len.
As an example:

/* expect to free N mbufs */
uint32_t n = N;
void **p = mempool_cache_put_zc_start(mc, &n, ...);

/* free up to n elems */
for (i = 0; i != n; i++) {

   /* get next free mbuf from somewhere */
   mb = extract_and_prefree_mbuf(...);

   /* no more free mbufs for now */
   if (mb == NULL)
      break;

   p[i] = mb;
}

/* finalize ZC put, with _i_ freed elems */ mempool_cache_put_zc_finish(mc, i);

Quoted 8 lines

That way, I think we can overcome the issue with i40e_tx_entry you mentioned above. Plus it might be useful in other similar places.

Another alternative is obviously to split i40e_tx_entry into two structs (one for mbuf, second for its metadata) and have a separate array for each of them.
Though with that approach we need to make sure no perf drops will be introduced, plus probably more code changes will be required.

Commit '5171b4ee6b6" already does this (in a different way), but just for AVX512. Unfortunately, it does not record any performance improvements. We could port this to Arm NEON and look at the performance.

Quoted 4 lines
#34 lore raw

24/05/2022 21:14, Honnappa Nagarahalli пишет:

Quoted 61 lines

<snip>

Quoted 50 lines

[[email protected] appears similar to someone who previously sent you email, but may not be that person. Learn why this could be a risk at https://aka.ms/LearnAboutSenderIdentification.]

16/05/2022 07:10, Feifei Wang пишет:

Quoted 38 lines
Quoted 30 lines
Quoted 23 lines

Currently, the transmit side frees the buffers into the lcore cache and the receive side allocates buffers from the lcore cache. The transmit side typically frees 32 buffers resulting in 32*8=256B of stores to lcore cache. The receive side allocates 32 buffers and stores them in the receive side software ring, resulting in 32*8=256B of stores and 256B of load from the lcore cache.

This patch proposes a mechanism to avoid freeing to/allocating from the lcore cache. i.e. the receive side will free the buffers from transmit side directly into it's software ring. This will avoid the 256B of loads and stores introduced by the lcore cache. It also frees up the cache lines used by the lcore cache.

However, this solution poses several constraints:

1)The receive queue needs to know which transmit queue it should take the buffers from. The application logic decides which transmit port to use to send out the packets. In many use cases the NIC might have a single port ([1], [2], [3]), in which case a given transmit queue is always mapped to a single receive queue (1:1 Rx queue: Tx queue). This is easy to configure.

Quoted 6 lines

If the NIC has 2 ports (there are several references), then we will have
1:2 (RX queue: TX queue) mapping which is still easy to configure. However, if this is generalized to 'N' ports, the configuration can be long. More over the PMD would have to scan a list of transmit queues to pull the buffers from.

Quoted 7 lines

Just to re-iterate some generic concerns about this proposal:
- We effectively link RX and TX queues - when this feature is enabled, user can't stop TX queue without stopping linked RX queue first. Right now user is free to start/stop any queues at his will. If that feature will allow to link queues from different ports, then even ports will become dependent and user will have to pay extra care when managing such ports.

Quoted 5 lines

[Feifei] When direct rearm enabled, there are two path for thread to choose. If there are enough Tx freed buffers, Rx can put buffers from Tx.
Otherwise, Rx will put buffers from mempool as usual. Thus, users do not need to pay much attention managing ports.

Quoted 9 lines

What I am talking about: right now different port or different queues of the same port can be treated as independent entities:
in general user is free to start/stop (and even reconfigure in some cases) one entity without need to stop other entity.
I.E user can stop and re-configure TX queue while keep receiving packets from RX queue.
With direct re-arm enabled, I think it wouldn't be possible any more:
before stopping/reconfiguring TX queue user would have make sure that corresponding RX queue wouldn't be used by datapath.

Quoted 1 line

I am trying to understand the problem better. For the TX queue to be stopped, the user must have blocked the data plane from accessing the TX queue.

Surely it is user responsibility tnot to call tx_burst()
for stopped/released queue.
The problem is that while TX for that queue is stopped,
RX for related queue still can continue.
So rx_burst() will try to read/modify TX queue data,
that might be already freed, or simultaneously modified by control path.

Again, it all can be mitigated by carefully re-designing and modifying control and data-path inside user app -
by doing extra checks and synchronizations, etc.
But from practical point - I presume most of users simply would avoid using this feature due all potential problems it might cause.

Quoted 2 lines

Like Feifei says, the RX side has the normal packet allocation path still available. Also this sounds like a corner case to me, we can handle this through checks in the queue_stop API.

Depends.
if it would be allowed to link queues only from the same port, then yes, extra checks for queue-stop might be enough. As right now DPDK doesn't allow user to change number of queues without dev_stop() first.
Though if it would be allowed to link queues from different ports, then situation will be much worse.
Right now ports are totally independent entities (except some special cases like link-bonding, etc.).
As one port can keep doing RX/TX, second one can be stopped, re-confgured, even detached, and newly attached device might re-use same port number.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

- very limited usage scenario - it will have a positive effect only when we have a fixed forwarding mapping: all (or nearly all) packets from the RX queue are forwarded into the same TX queue.

Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

[Feifei] Although the usage scenario is limited, this usage scenario has a wide range of applications, such as NIC with one port.

yes, there are NICs with one port, but no guarantee there wouldn't be several such NICs within the system.

What I see in my interactions is, a single NIC/DPU is under utilized for a 2 socket system. Some are adding more sockets to the system to better utilize the DPU. The NIC bandwidth continues to grow significantly. I do not think there will be a multi-DPU per server scenario.

Interesting... from my experience it is visa-versa:
in many cases 200Gb/s is not that much these days
to saturate modern 2 socket x86 server.
Though I suppose a lot depends on particular HW and actual workload.

Quoted 79 lines
Quoted 25 lines
Quoted 8 lines

Furtrhermore, I think this is a tradeoff between performance and flexibility.
Our goal is to achieve best performance, this means we need to give up some flexibility decisively. For example of 'FAST_FREE Mode', it deletes most of the buffer check (refcnt > 1, external buffer, chain buffer), chooses a shorest path, and then achieve significant performance

improvement.

Quoted 12 lines
Quoted 11 lines

Wonder did you had a chance to consider mempool-cache ZC API, similar to one we have for the ring?
It would allow us on TX free path to avoid copying mbufs to temporary array on the stack.
Instead we can put them straight from TX SW ring to the mempool cache. That should save extra store/load for mbuf and might help to achieve some performance gain without by-passing mempool.
It probably wouldn't be as fast as what you proposing, but might be fast enough to consider as alternative.
Again, it would be a generic one, so we can avoid all these implications and limitations.

Quoted 3 lines

[Feifei] I think this is a good try. However, the most important thing is that if we can bypass the mempool decisively to pursue the significant performance gains.

Quoted 5 lines

I understand the intention, and I personally think this is wrong and dangerous attitude.
We have mempool abstraction in place for very good reason.
So we need to try to improve mempool performance (and API if necessary) at first place, not to avoid it and break our own rules and recommendations.

The abstraction can be thought of at a higher level. i.e. the driver manages the buffer allocation/free and is hidden from the application. The application does not need to be aware of how these changes are implemented.

Quoted 38 lines
Quoted 8 lines

For ZC, there maybe a problem for it in i40e. The reason for that put Tx buffers into temporary is that i40e_tx_entry includes buffer pointer and index.
Thus we cannot put Tx SW_ring entry into mempool directly, we need to firstlt extract mbuf pointer. Finally, though we use ZC, we still can't avoid using a temporary stack to extract Tx buffer pointers.

When talking about ZC API for mempool cache I meant something like:
void ** mempool_cache_put_zc_start(struct rte_mempool_cache *mc, uint32_t *nb_elem, uint32_t flags); void mempool_cache_put_zc_finish(struct rte_mempool_cache *mc, uint32_t nb_elem); i.e. _start_ will return user a pointer inside mp-cache where to put free elems and max number of slots that can be safely filled.
_finish_ will update mc->len.
As an example:

/* expect to free N mbufs */
uint32_t n = N;
void **p = mempool_cache_put_zc_start(mc, &n, ...);

/* free up to n elems */
for (i = 0; i != n; i++) {

    /* get next free mbuf from somewhere */
    mb = extract_and_prefree_mbuf(...);

    /* no more free mbufs for now */
    if (mb == NULL)
       break;

    p[i] = mb;
}

/* finalize ZC put, with _i_ freed elems */ mempool_cache_put_zc_finish(mc, i);

Quoted 8 lines

That way, I think we can overcome the issue with i40e_tx_entry you mentioned above. Plus it might be useful in other similar places.

Another alternative is obviously to split i40e_tx_entry into two structs (one for mbuf, second for its metadata) and have a separate array for each of them.
Though with that approach we need to make sure no perf drops will be introduced, plus probably more code changes will be required.

Commit '5171b4ee6b6" already does this (in a different way), but just for AVX512. Unfortunately, it does not record any performance improvements. We could port this to Arm NEON and look at the performance.

#35 lore raw

<snip>

Quoted 60 lines
Quoted 60 lines
Quoted 60 lines

[[email protected] appears similar to someone who previously sent you email, but may not be that person. Learn why this could be a risk at https://aka.ms/LearnAboutSenderIdentification.]

16/05/2022 07:10, Feifei Wang пишет:

Quoted 38 lines
Quoted 30 lines
Quoted 23 lines

Currently, the transmit side frees the buffers into the lcore cache and the receive side allocates buffers from the lcore cache. The transmit side typically frees 32 buffers resulting in 32*8=256B of stores to lcore cache. The receive side allocates 32 buffers and stores them in the receive side software ring, resulting in 32*8=256B of stores and 256B of load from the lcore cache.

This patch proposes a mechanism to avoid freeing to/allocating from the lcore cache. i.e. the receive side will free the buffers from transmit side directly into it's software ring. This will avoid the 256B of loads and stores introduced by the lcore cache. It also frees up the cache lines used by the lcore cache.

However, this solution poses several constraints:

1)The receive queue needs to know which transmit queue it should take the buffers from. The application logic decides which transmit port to use to send out the packets. In many use cases the NIC might have a single port ([1], [2], [3]), in which case a given transmit queue is always mapped to a single receive queue (1:1 Rx queue: Tx queue). This is easy to configure.

Quoted 6 lines

If the NIC has 2 ports (there are several references), then we will have
1:2 (RX queue: TX queue) mapping which is still easy to configure. However, if this is generalized to 'N' ports, the configuration can be long. More over the PMD would have to scan a list of transmit queues to pull the buffers from.

Quoted 7 lines

Just to re-iterate some generic concerns about this proposal:
- We effectively link RX and TX queues - when this feature is enabled, user can't stop TX queue without stopping linked RX queue first. Right now user is free to start/stop any queues at his will. If that feature will allow to link queues from different ports, then even ports will become dependent and user will have to pay extra care when managing such ports.

Quoted 5 lines

[Feifei] When direct rearm enabled, there are two path for thread to choose. If there are enough Tx freed buffers, Rx can put buffers from Tx.
Otherwise, Rx will put buffers from mempool as usual. Thus, users do not need to pay much attention managing ports.

What I am talking about: right now different port or different queues of the same port can be treated as independent entities:
in general user is free to start/stop (and even reconfigure in some cases) one entity without need to stop other entity.
I.E user can stop and re-configure TX queue while keep receiving packets from RX queue.
With direct re-arm enabled, I think it wouldn't be possible any more:
before stopping/reconfiguring TX queue user would have make sure that corresponding RX queue wouldn't be used by datapath.

Quoted 8 lines
Quoted 1 line

I am trying to understand the problem better. For the TX queue to be stopped,

the user must have blocked the data plane from accessing the TX queue.

Surely it is user responsibility tnot to call tx_burst() for stopped/released queue. The problem is that while TX for that queue is stopped, RX for related queue still can continue.
So rx_burst() will try to read/modify TX queue data, that might be already freed, or simultaneously modified by control path.

Understood, agree on the issue

Quoted 5 lines

Again, it all can be mitigated by carefully re-designing and modifying control and data-path inside user app - by doing extra checks and synchronizations, etc. But from practical point - I presume most of users simply would avoid using this feature due all potential problems it might cause.

That is subjective, it all depends on the performance improvements users see in their application. IMO, the performance improvement seen with this patch is worth few changes.

Quoted 5 lines
Quoted 3 lines

Like Feifei says, the RX side has the normal packet allocation path still available. Also this sounds like a corner case to me, we can handle this through checks in

the queue_stop API.

Quoted 11 lines

Depends.
if it would be allowed to link queues only from the same port, then yes, extra checks for queue-stop might be enough.
As right now DPDK doesn't allow user to change number of queues without dev_stop() first.
Though if it would be allowed to link queues from different ports, then situation will be much worse.
Right now ports are totally independent entities (except some special cases like link-bonding, etc.).
As one port can keep doing RX/TX, second one can be stopped, re-confgured, even detached, and newly attached device might re-use same port number.

I see this as a similar restriction to the one discussed above. Do you see any issues if we enforce this with checks?

Quoted 110 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

- very limited usage scenario - it will have a positive effect only when we have a fixed forwarding mapping: all (or nearly all) packets from the RX queue are forwarded into the same TX queue.

Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

[Feifei] Although the usage scenario is limited, this usage scenario has a wide range of applications, such as NIC with one port.

yes, there are NICs with one port, but no guarantee there wouldn't be several such NICs within the system.

What I see in my interactions is, a single NIC/DPU is under utilized for a 2

socket system. Some are adding more sockets to the system to better utilize the DPU. The NIC bandwidth continues to grow significantly. I do not think there will be a multi-DPU per server scenario.

Interesting... from my experience it is visa-versa:
in many cases 200Gb/s is not that much these days to saturate modern 2 socket x86 server.
Though I suppose a lot depends on particular HW and actual workload.

Quoted 26 lines
Quoted 26 lines
Quoted 9 lines

Furtrhermore, I think this is a tradeoff between performance and flexibility.
Our goal is to achieve best performance, this means we need to give up some flexibility decisively. For example of 'FAST_FREE Mode', it deletes most of the buffer check (refcnt > 1, external buffer, chain buffer), chooses a shorest path, and then achieve significant performance

improvement.

Quoted 15 lines
Quoted 11 lines

Wonder did you had a chance to consider mempool-cache ZC API, similar to one we have for the ring?
It would allow us on TX free path to avoid copying mbufs to temporary array on the stack.
Instead we can put them straight from TX SW ring to the mempool cache. That should save extra store/load for mbuf and might help to achieve some performance gain without by-passing mempool. It probably wouldn't be as fast as what you proposing, but might be fast enough to consider as alternative.
Again, it would be a generic one, so we can avoid all these implications and limitations.

[Feifei] I think this is a good try. However, the most important thing is that if we can bypass the mempool decisively to pursue the significant performance gains.

Quoted 5 lines
Quoted 5 lines

I understand the intention, and I personally think this is wrong and dangerous attitude.
We have mempool abstraction in place for very good reason. So we need to try to improve mempool performance (and API if necessary) at first place, not to avoid it and break our own rules and

recommendations.

Quoted 1 line

The abstraction can be thought of at a higher level. i.e. the driver manages the

buffer allocation/free and is hidden from the application. The application does not need to be aware of how these changes are implemented.

Quoted 43 lines
Quoted 43 lines
Quoted 9 lines

For ZC, there maybe a problem for it in i40e. The reason for that put Tx buffers into temporary is that i40e_tx_entry includes buffer pointer and index.
Thus we cannot put Tx SW_ring entry into mempool directly, we need to firstlt extract mbuf pointer. Finally, though we use ZC, we still can't avoid using a temporary stack to extract Tx buffer pointers.

When talking about ZC API for mempool cache I meant something like:
void ** mempool_cache_put_zc_start(struct rte_mempool_cache *mc, uint32_t *nb_elem, uint32_t flags); void mempool_cache_put_zc_finish(struct
rte_mempool_cache *mc, uint32_t nb_elem); i.e. _start_ will return user a pointer inside mp-cache where to put free elems and max number of slots that can be safely filled.
_finish_ will update mc->len.
As an example:

/* expect to free N mbufs */
uint32_t n = N;
void **p = mempool_cache_put_zc_start(mc, &n, ...);

/* free up to n elems */
for (i = 0; i != n; i++) {

    /* get next free mbuf from somewhere */
    mb = extract_and_prefree_mbuf(...);

    /* no more free mbufs for now */
    if (mb == NULL)
       break;

    p[i] = mb;
}

/* finalize ZC put, with _i_ freed elems */
mempool_cache_put_zc_finish(mc, i);

That way, I think we can overcome the issue with i40e_tx_entry you mentioned above. Plus it might be useful in other similar places.

Quoted 6 lines
Quoted 5 lines

Another alternative is obviously to split i40e_tx_entry into two structs (one for mbuf, second for its metadata) and have a separate array for each of them.
Though with that approach we need to make sure no perf drops will be introduced, plus probably more code changes will be required.

Commit '5171b4ee6b6" already does this (in a different way), but just for

AVX512. Unfortunately, it does not record any performance improvements. We could port this to Arm NEON and look at the performance.

#36 lore raw
Quoted 73 lines

<snip>

Quoted 60 lines
Quoted 60 lines
Quoted 60 lines

[[email protected] appears similar to someone who previously sent you email, but may not be that person. Learn why this could be a risk at https://aka.ms/LearnAboutSenderIdentification.]

16/05/2022 07:10, Feifei Wang пишет:

Quoted 38 lines
Quoted 30 lines
Quoted 23 lines

Currently, the transmit side frees the buffers into the lcore cache and the receive side allocates buffers from the lcore cache. The transmit side typically frees 32 buffers resulting in 32*8=256B of stores to lcore cache. The receive side allocates 32 buffers and stores them in the receive side software ring, resulting in 32*8=256B of stores and 256B of load from the lcore cache.

This patch proposes a mechanism to avoid freeing to/allocating from the lcore cache. i.e. the receive side will free the buffers from transmit side directly into it's software ring. This will avoid the 256B of loads and stores introduced by the lcore cache. It also frees up the cache lines used by the lcore cache.

However, this solution poses several constraints:

1)The receive queue needs to know which transmit queue it should take the buffers from. The application logic decides which transmit port to use to send out the packets. In many use cases the NIC might have a single port ([1], [2], [3]), in which case a given transmit queue is always mapped to a single receive queue (1:1 Rx queue: Tx queue). This is easy to configure.

Quoted 6 lines

If the NIC has 2 ports (there are several references), then we will have
1:2 (RX queue: TX queue) mapping which is still easy to configure. However, if this is generalized to 'N' ports, the configuration can be long. More over the PMD would have to scan a list of transmit queues to pull the buffers from.

Quoted 7 lines

Just to re-iterate some generic concerns about this proposal:
- We effectively link RX and TX queues - when this feature is enabled, user can't stop TX queue without stopping linked RX queue first. Right now user is free to start/stop any queues at his will. If that feature will allow to link queues from different ports, then even ports will become dependent and user will have to pay extra care when managing such ports.

Quoted 5 lines

[Feifei] When direct rearm enabled, there are two path for thread to choose. If there are enough Tx freed buffers, Rx can put buffers from Tx.
Otherwise, Rx will put buffers from mempool as usual. Thus, users do not need to pay much attention managing ports.

What I am talking about: right now different port or different queues of the same port can be treated as independent entities:
in general user is free to start/stop (and even reconfigure in some cases) one entity without need to stop other entity.
I.E user can stop and re-configure TX queue while keep receiving packets from RX queue.
With direct re-arm enabled, I think it wouldn't be possible any more:
before stopping/reconfiguring TX queue user would have make sure that corresponding RX queue wouldn't be used by datapath.

Quoted 8 lines
Quoted 1 line

I am trying to understand the problem better. For the TX queue to be stopped,

the user must have blocked the data plane from accessing the TX queue.

Surely it is user responsibility tnot to call tx_burst() for stopped/released queue. The problem is that while TX for that queue is stopped, RX for related queue still can continue.
So rx_burst() will try to read/modify TX queue data, that might be already freed, or simultaneously modified by control path.

Understood, agree on the issue

Quoted 1 line
Quoted 6 lines
Quoted 4 lines

Again, it all can be mitigated by carefully re-designing and modifying control and data-path inside user app - by doing extra checks and synchronizations, etc. But from practical point - I presume most of users simply would avoid using this feature due all potential problems it might cause.

That is subjective, it all depends on the performance improvements users see in their application. IMO, the performance improvement seen with this patch is worth few changes.

Yes, it is subjective till some extent, though my feeling that it might end-up being sort of synthetic improvement used only by some show-case benchmarks.
From my perspective, it would be much more plausible, if we can introduce some sort of generic improvement, that doesn't impose all these extra constraints and implications. Like one, discussed below in that thread with ZC mempool approach.

Quoted 16 lines
Quoted 5 lines
Quoted 3 lines

Like Feifei says, the RX side has the normal packet allocation path still available. Also this sounds like a corner case to me, we can handle this through checks in

the queue_stop API.

Quoted 11 lines

Depends.
if it would be allowed to link queues only from the same port, then yes, extra checks for queue-stop might be enough.
As right now DPDK doesn't allow user to change number of queues without dev_stop() first.
Though if it would be allowed to link queues from different ports, then situation will be much worse.
Right now ports are totally independent entities (except some special cases like link-bonding, etc.).
As one port can keep doing RX/TX, second one can be stopped, re-confgured, even detached, and newly attached device might re-use same port number.

Quoted 1 line

I see this as a similar restriction to the one discussed above.

Yes, they are similar in principal, though I think that the case with queues from different port would make things much more complex.

Quoted 1 line

Do you see any issues if we enforce this with checks?

Hard to tell straightway, a lot will depend how smart such implementation would be.
Usually DPDK tends not to avoid heavy synchronizations within its data-path functions.

Quoted 111 lines
Quoted 110 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

- very limited usage scenario - it will have a positive effect only when we have a fixed forwarding mapping: all (or nearly all) packets from the RX queue are forwarded into the same TX queue.

Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

[Feifei] Although the usage scenario is limited, this usage scenario has a wide range of applications, such as NIC with one port.

yes, there are NICs with one port, but no guarantee there wouldn't be several such NICs within the system.

What I see in my interactions is, a single NIC/DPU is under utilized for a 2

socket system. Some are adding more sockets to the system to better utilize the DPU. The NIC bandwidth continues to grow significantly. I do not think there will be a multi-DPU per server scenario.

Interesting... from my experience it is visa-versa:
in many cases 200Gb/s is not that much these days to saturate modern 2 socket x86 server.
Though I suppose a lot depends on particular HW and actual workload.

Quoted 26 lines
Quoted 26 lines
Quoted 9 lines

Furtrhermore, I think this is a tradeoff between performance and flexibility.
Our goal is to achieve best performance, this means we need to give up some flexibility decisively. For example of 'FAST_FREE Mode', it deletes most of the buffer check (refcnt > 1, external buffer, chain buffer), chooses a shorest path, and then achieve significant performance

improvement.

Quoted 15 lines
Quoted 11 lines

Wonder did you had a chance to consider mempool-cache ZC API, similar to one we have for the ring?
It would allow us on TX free path to avoid copying mbufs to temporary array on the stack.
Instead we can put them straight from TX SW ring to the mempool cache. That should save extra store/load for mbuf and might help to achieve some performance gain without by-passing mempool. It probably wouldn't be as fast as what you proposing, but might be fast enough to consider as alternative.
Again, it would be a generic one, so we can avoid all these implications and limitations.

[Feifei] I think this is a good try. However, the most important thing is that if we can bypass the mempool decisively to pursue the significant performance gains.

Quoted 5 lines
Quoted 5 lines

I understand the intention, and I personally think this is wrong and dangerous attitude.
We have mempool abstraction in place for very good reason. So we need to try to improve mempool performance (and API if necessary) at first place, not to avoid it and break our own rules and

recommendations.

Quoted 1 line

The abstraction can be thought of at a higher level. i.e. the driver manages the

buffer allocation/free and is hidden from the application. The application does not need to be aware of how these changes are implemented.

Quoted 43 lines
Quoted 43 lines
Quoted 9 lines

For ZC, there maybe a problem for it in i40e. The reason for that put Tx buffers into temporary is that i40e_tx_entry includes buffer pointer and index.
Thus we cannot put Tx SW_ring entry into mempool directly, we need to firstlt extract mbuf pointer. Finally, though we use ZC, we still can't avoid using a temporary stack to extract Tx buffer pointers.

When talking about ZC API for mempool cache I meant something like:
void ** mempool_cache_put_zc_start(struct rte_mempool_cache *mc, uint32_t *nb_elem, uint32_t flags); void mempool_cache_put_zc_finish(struct
rte_mempool_cache *mc, uint32_t nb_elem); i.e. _start_ will return user a pointer inside mp-cache where to put free elems and max number of slots that can be safely filled.
_finish_ will update mc->len.
As an example:

/* expect to free N mbufs */
uint32_t n = N;
void **p = mempool_cache_put_zc_start(mc, &n, ...);

/* free up to n elems */
for (i = 0; i != n; i++) {

     /* get next free mbuf from somewhere */
     mb = extract_and_prefree_mbuf(...);

     /* no more free mbufs for now */
     if (mb == NULL)
        break;

     p[i] = mb;
}

/* finalize ZC put, with _i_ freed elems */
mempool_cache_put_zc_finish(mc, i);

That way, I think we can overcome the issue with i40e_tx_entry you mentioned above. Plus it might be useful in other similar places.

Quoted 6 lines
Quoted 5 lines

Another alternative is obviously to split i40e_tx_entry into two structs (one for mbuf, second for its metadata) and have a separate array for each of them.
Though with that approach we need to make sure no perf drops will be introduced, plus probably more code changes will be required.

Commit '5171b4ee6b6" already does this (in a different way), but just for

AVX512. Unfortunately, it does not record any performance improvements. We could port this to Arm NEON and look at the performance.

#37 lore raw
Quoted 107 lines

From: Konstantin Ananyev [mailto:[email protected]]
Sent: Saturday, 4 June 2022 01.32

Quoted 5 lines

<snip>

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

[[email protected] appears similar to someone who previously sent you email, but may not be that person. Learn why

this

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

could be a risk at https://aka.ms/LearnAboutSenderIdentification.]

16/05/2022 07:10, Feifei Wang пишет:

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Currently, the transmit side frees the buffers into the lcore cache and the receive side allocates buffers from the lcore

cache.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

The transmit side typically frees 32 buffers resulting in 32*8=256B of stores to lcore cache. The receive side allocates

32

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

buffers and stores them in the receive side software ring, resulting in 32*8=256B of stores and 256B of load from the

lcore cache.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

This patch proposes a mechanism to avoid freeing to/allocating from the lcore cache. i.e. the receive side will free the

buffers

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

from transmit side directly into it's software ring. This will avoid the 256B of loads and stores introduced by the lcore

cache.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

It also frees up the cache lines used by the lcore cache.

However, this solution poses several constraints:

1)The receive queue needs to know which transmit queue it

should

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

take the buffers from. The application logic decides which transmit port to use to send out the packets. In many use cases the NIC might have a single port ([1], [2], [3]), in which case

a

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

given transmit queue is always mapped to a single receive queue (1:1 Rx queue: Tx queue). This is easy to configure.

If the NIC has 2 ports (there are several references), then we will have
1:2 (RX queue: TX queue) mapping which is still easy to

configure.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

However, if this is generalized to 'N' ports, the configuration can be long. More over the PMD would have to scan a list of transmit queues to pull the buffers from.

Just to re-iterate some generic concerns about this proposal:
- We effectively link RX and TX queues - when this feature

is enabled,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

user can't stop TX queue without stopping linked RX queue

first.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Right now user is free to start/stop any queues at his

will.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

If that feature will allow to link queues from different

ports,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

then even ports will become dependent and user will have

to pay extra

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

care when managing such ports.

[Feifei] When direct rearm enabled, there are two path for thread

to

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

choose. If there are enough Tx freed buffers, Rx can put buffers from Tx.
Otherwise, Rx will put buffers from mempool as usual. Thus, users

do

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

not need to pay much attention managing ports.

What I am talking about: right now different port or different

queues

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

of the same port can be treated as independent entities:
in general user is free to start/stop (and even reconfigure in

some

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

cases) one entity without need to stop other entity. I.E user can stop and re-configure TX queue while keep receiving packets from RX queue.
With direct re-arm enabled, I think it wouldn't be possible any

more:

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

before stopping/reconfiguring TX queue user would have make sure

that

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line

corresponding RX queue wouldn't be used by datapath.

I am trying to understand the problem better. For the TX queue to

be stopped,

Quoted 1 line
Quoted 1 line

the user must have blocked the data plane from accessing the TX

queue.

Quoted 2 lines
Quoted 2 lines

Surely it is user responsibility tnot to call tx_burst() for

stopped/released queue.

Quoted 1 line
Quoted 1 line

The problem is that while TX for that queue is stopped, RX for

related queue still

Quoted 2 lines
Quoted 2 lines

can continue.
So rx_burst() will try to read/modify TX queue data, that might be

already freed,

Quoted 5 lines
Quoted 1 line

or simultaneously modified by control path.

Understood, agree on the issue

Quoted 2 lines

Again, it all can be mitigated by carefully re-designing and

modifying control and

Quoted 1 line
Quoted 1 line

data-path inside user app - by doing extra checks and

synchronizations, etc.

Quoted 1 line
Quoted 1 line

But from practical point - I presume most of users simply would

avoid using this

Quoted 1 line
Quoted 1 line

feature due all potential problems it might cause.

Quoted 8 lines
Quoted 1 line

That is subjective, it all depends on the performance improvements

users see in their application.

Quoted 1 line

IMO, the performance improvement seen with this patch is worth few

changes.

Yes, it is subjective till some extent, though my feeling that it might end-up being sort of synthetic improvement used only by some show-case benchmarks.

I believe that one specific important use case has already been mentioned, so I don't think this is a benchmark only feature.

Quoted 5 lines

From my perspective, it would be much more plausible,
if we can introduce some sort of generic improvement, that doesn't impose all these extra constraints and implications. Like one, discussed below in that thread with ZC mempool approach.

Considering this feature from a high level perspective, I agree with Konstantin's concerns, so I'll also support his views.

If this patch is supposed to be a generic feature, please add support for it in all NIC PMDs, not just one. (Regardless if the feature is defined as 1:1 mapping or N:M mapping.) It is purely software, so it should be available for all PMDs, not just your favorite hardware! Consider the "fast mbuf free" feature, which is pure software; why is that feature not implemented in all PMDs?

A secondary point I'm making here is that this specific feature will lead to an enormous amount of copy-paste code, instead of a generic library function easily available for all PMDs.

Quoted 30 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Like Feifei says, the RX side has the normal packet allocation path

still available.

Quoted 1 line
Quoted 1 line
Quoted 1 line

Also this sounds like a corner case to me, we can handle this

through checks in

Quoted 4 lines
Quoted 4 lines

the queue_stop API.

Depends.
if it would be allowed to link queues only from the same port, then

yes, extra

Quoted 2 lines
Quoted 2 lines

checks for queue-stop might be enough.
As right now DPDK doesn't allow user to change number of queues

without

Quoted 2 lines
Quoted 2 lines

dev_stop() first.
Though if it would be allowed to link queues from different ports,

then situation

Quoted 2 lines
Quoted 2 lines

will be much worse.
Right now ports are totally independent entities (except some

special cases like

Quoted 2 lines
Quoted 2 lines

link-bonding, etc.).
As one port can keep doing RX/TX, second one can be stopped, re-

confgured,

Quoted 1 line
Quoted 1 line

even detached, and newly attached device might re-use same port

number.

Quoted 1 line

I see this as a similar restriction to the one discussed above.

Yes, they are similar in principal, though I think that the case with queues from different port would make things much more complex.

Quoted 6 lines
Quoted 1 line

Do you see any issues if we enforce this with checks?

Hard to tell straightway, a lot will depend how smart such implementation would be.
Usually DPDK tends not to avoid heavy synchronizations within its data-path functions.

Certainly! Implementing more and more of such features in the PMDs will lead to longer and longer data plane code paths in the PMDs. It is the "salami method", where each small piece makes no performance difference, but they all add up, and eventually the sum of them does impact the performance of the general use case negatively.

Quoted 141 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

- very limited usage scenario - it will have a positive effect

only

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

when we have a fixed forwarding mapping: all (or nearly

all) packets

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

from the RX queue are forwarded into the same TX queue.

[Feifei] Although the usage scenario is limited, this usage

scenario

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

has a wide range of applications, such as NIC with one port.

yes, there are NICs with one port, but no guarantee there wouldn't

be

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line

several such NICs within the system.

What I see in my interactions is, a single NIC/DPU is under

utilized for a 2

Quoted 1 line
Quoted 1 line

socket system. Some are adding more sockets to the system to better

utilize the

Quoted 1 line
Quoted 1 line

DPU. The NIC bandwidth continues to grow significantly. I do not

think there will

Quoted 5 lines
Quoted 5 lines

be a multi-DPU per server scenario.

Interesting... from my experience it is visa-versa:
in many cases 200Gb/s is not that much these days to saturate modern

2 socket

Quoted 8 lines
Quoted 8 lines

x86 server.
Though I suppose a lot depends on particular HW and actual workload.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Furtrhermore, I think this is a tradeoff between performance and flexibility.
Our goal is to achieve best performance, this means we need to

give

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

up some flexibility decisively. For example of 'FAST_FREE Mode',

it

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

deletes most of the buffer check (refcnt > 1, external buffer,

chain

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

buffer), chooses a shorest path, and then achieve significant performance

improvement.

Quoted 5 lines
Quoted 5 lines

Wonder did you had a chance to consider mempool-cache ZC API, similar to one we have for the ring?
It would allow us on TX free path to avoid copying mbufs to temporary array on the stack.
Instead we can put them straight from TX SW ring to the mempool

cache.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

That should save extra store/load for mbuf and might help to achieve some performance gain without by-passing mempool. It probably wouldn't be as fast as what you proposing, but might

be

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

fast enough to consider as alternative.
Again, it would be a generic one, so we can avoid all these implications and limitations.

[Feifei] I think this is a good try. However, the most important thing is that if we can bypass the mempool decisively to pursue

the

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

significant performance gains.

I understand the intention, and I personally think this is wrong

and

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

dangerous attitude.
We have mempool abstraction in place for very good reason. So we need to try to improve mempool performance (and API if necessary) at first place, not to avoid it and break our own rules

and

Quoted 2 lines
Quoted 2 lines

recommendations.

Quoted 1 line

The abstraction can be thought of at a higher level. i.e. the

driver manages the

Quoted 1 line
Quoted 1 line

buffer allocation/free and is hidden from the application. The

application does

Quoted 6 lines
Quoted 6 lines

not need to be aware of how these changes are implemented.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

For ZC, there maybe a problem for it in i40e. The reason for that put Tx buffers into temporary is that i40e_tx_entry includes

buffer

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

pointer and index.
Thus we cannot put Tx SW_ring entry into mempool directly, we

need

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

to firstlt extract mbuf pointer. Finally, though we use ZC, we

still

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

can't avoid using a temporary stack to extract Tx buffer

pointers.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

When talking about ZC API for mempool cache I meant something

like:

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

void ** mempool_cache_put_zc_start(struct rte_mempool_cache *mc, uint32_t *nb_elem, uint32_t flags); void mempool_cache_put_zc_finish(struct
rte_mempool_cache *mc, uint32_t nb_elem); i.e. _start_ will return user a pointer inside mp-cache where to put free elems and max

number

Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines

of slots that can be safely filled.
_finish_ will update mc->len.
As an example:

/* expect to free N mbufs */
uint32_t n = N;
void **p = mempool_cache_put_zc_start(mc, &n, ...);

/* free up to n elems */
for (i = 0; i != n; i++) {

     /* get next free mbuf from somewhere */
     mb = extract_and_prefree_mbuf(...);

     /* no more free mbufs for now */
     if (mb == NULL)
        break;

     p[i] = mb;
}
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

/* finalize ZC put, with _i_ freed elems */
mempool_cache_put_zc_finish(mc, i);

That way, I think we can overcome the issue with i40e_tx_entry you mentioned above. Plus it might be useful in other similar places.

Another alternative is obviously to split i40e_tx_entry into two structs (one for mbuf, second for its metadata) and have a

separate

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

array for each of them.
Though with that approach we need to make sure no perf drops will

be

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line

introduced, plus probably more code changes will be required.

Commit '5171b4ee6b6" already does this (in a different way), but

just for

Quoted 1 line
Quoted 1 line

AVX512. Unfortunately, it does not record any performance

improvements. We

Quoted 2 lines
Quoted 1 line

could port this to Arm NEON and look at the performance.

#38 lore raw

<snip>

Quoted 111 lines
Quoted 111 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

[[email protected] appears similar to someone who previously sent you email, but may not be that person. Learn why

this

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

could be a risk at
https://aka.ms/LearnAboutSenderIdentification.]

16/05/2022 07:10, Feifei Wang пишет:

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Currently, the transmit side frees the buffers into the lcore cache and the receive side allocates buffers from the lcore

cache.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

The transmit side typically frees 32 buffers resulting in 32*8=256B of stores to lcore cache. The receive side allocates

32

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

buffers and stores them in the receive side software ring, resulting in 32*8=256B of stores and 256B of load from the

lcore cache.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

This patch proposes a mechanism to avoid freeing to/allocating from the lcore cache. i.e. the receive side will free the

buffers

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

from transmit side directly into it's software ring. This will avoid the 256B of loads and stores introduced by the lcore

cache.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

It also frees up the cache lines used by the lcore cache.

However, this solution poses several constraints:

1)The receive queue needs to know which transmit queue it

should

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

take the buffers from. The application logic decides which transmit port to use to send out the packets. In many use cases the NIC might have a single port ([1], [2], [3]), in which case

a

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

given transmit queue is always mapped to a single receive queue
(1:1 Rx queue: Tx queue). This is easy to configure.

If the NIC has 2 ports (there are several references), then we will have
1:2 (RX queue: TX queue) mapping which is still easy to

configure.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

However, if this is generalized to 'N' ports, the configuration can be long. More over the PMD would have to scan a list of transmit queues to pull the buffers from.

Just to re-iterate some generic concerns about this proposal:
- We effectively link RX and TX queues - when this feature

is enabled,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

user can't stop TX queue without stopping linked RX queue

first.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Right now user is free to start/stop any queues at his

will.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

If that feature will allow to link queues from different

ports,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

then even ports will become dependent and user will have

to pay extra

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

care when managing such ports.

[Feifei] When direct rearm enabled, there are two path for thread

to

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

choose. If there are enough Tx freed buffers, Rx can put buffers from Tx.
Otherwise, Rx will put buffers from mempool as usual. Thus, users

do

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

not need to pay much attention managing ports.

What I am talking about: right now different port or different

queues

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

of the same port can be treated as independent entities:
in general user is free to start/stop (and even reconfigure in

some

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

cases) one entity without need to stop other entity. I.E user can stop and re-configure TX queue while keep receiving packets from RX queue.
With direct re-arm enabled, I think it wouldn't be possible any

more:

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

before stopping/reconfiguring TX queue user would have make sure

that

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line

corresponding RX queue wouldn't be used by datapath.

I am trying to understand the problem better. For the TX queue to

be stopped,

Quoted 1 line
Quoted 1 line

the user must have blocked the data plane from accessing the TX

queue.

Quoted 2 lines
Quoted 2 lines

Surely it is user responsibility tnot to call tx_burst() for

stopped/released queue.

Quoted 1 line
Quoted 1 line

The problem is that while TX for that queue is stopped, RX for

related queue still

Quoted 2 lines
Quoted 2 lines

can continue.
So rx_burst() will try to read/modify TX queue data, that might be

already freed,

Quoted 5 lines
Quoted 1 line

or simultaneously modified by control path.

Understood, agree on the issue

Quoted 2 lines

Again, it all can be mitigated by carefully re-designing and

modifying control and

Quoted 1 line
Quoted 1 line

data-path inside user app - by doing extra checks and

synchronizations, etc.

Quoted 1 line
Quoted 1 line

But from practical point - I presume most of users simply would

avoid using this

Quoted 2 lines
Quoted 1 line

feature due all potential problems it might cause.

That is subjective, it all depends on the performance improvements

users see in their application.

Quoted 1 line

IMO, the performance improvement seen with this patch is worth few

Quoted 8 lines
Quoted 5 lines

changes.

Yes, it is subjective till some extent, though my feeling that it might end-up being sort of synthetic improvement used only by some show-case benchmarks.

I believe that one specific important use case has already been mentioned, so I don't think this is a benchmark only feature.

+1

Quoted 9 lines
Quoted 6 lines

From my perspective, it would be much more plausible, if we can
introduce some sort of generic improvement, that doesn't impose all these extra constraints and implications.
Like one, discussed below in that thread with ZC mempool approach.

Considering this feature from a high level perspective, I agree with Konstantin's concerns, so I'll also support his views.

We did hack the ZC mempool approach [1], level of improvement is pretty small compared with this patch.

[1] http://patches.dpdk.org/project/dpdk/patch/[email protected]/

Quoted 6 lines

If this patch is supposed to be a generic feature, please add support for it in all NIC PMDs, not just one. (Regardless if the feature is defined as 1:1 mapping or N:M mapping.) It is purely software, so it should be available for all PMDs, not just your favorite hardware! Consider the "fast mbuf free" feature, which is pure software; why is that feature not implemented in all PMDs?

Agree, it is good to have it supported in all the drivers. We do not have a favorite hardware, just picked a PMD which we are more familiar with. We do plan to implement in other prominent PMDs.

Quoted 4 lines

A secondary point I'm making here is that this specific feature will lead to an enormous amount of copy-paste code, instead of a generic library function easily available for all PMDs.

Are you talking about the i40e driver code in specific? If yes, agree we should avoid copy-paste and we will look to reduce that.

Quoted 39 lines
Quoted 32 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Like Feifei says, the RX side has the normal packet allocation path

still available.

Quoted 1 line
Quoted 1 line
Quoted 1 line

Also this sounds like a corner case to me, we can handle this

through checks in

Quoted 4 lines
Quoted 4 lines

the queue_stop API.

Depends.
if it would be allowed to link queues only from the same port, then

yes, extra

Quoted 2 lines
Quoted 2 lines

checks for queue-stop might be enough.
As right now DPDK doesn't allow user to change number of queues

without

Quoted 2 lines
Quoted 2 lines

dev_stop() first.
Though if it would be allowed to link queues from different ports,

then situation

Quoted 2 lines
Quoted 2 lines

will be much worse.
Right now ports are totally independent entities (except some

special cases like

Quoted 2 lines
Quoted 2 lines

link-bonding, etc.).
As one port can keep doing RX/TX, second one can be stopped, re-

confgured,

Quoted 1 line
Quoted 1 line

even detached, and newly attached device might re-use same port

number.

Quoted 1 line

I see this as a similar restriction to the one discussed above.

Yes, they are similar in principal, though I think that the case with queues from different port would make things much more complex.

Quoted 6 lines
Quoted 1 line

Do you see any issues if we enforce this with checks?

Hard to tell straightway, a lot will depend how smart such implementation would be.
Usually DPDK tends not to avoid heavy synchronizations within its data-path functions.

Quoted 5 lines

Certainly! Implementing more and more of such features in the PMDs will lead to longer and longer data plane code paths in the PMDs. It is the "salami method", where each small piece makes no performance difference, but they all add up, and eventually the sum of them does impact the performance of the general use case negatively.

It would be good to have a test running in UNH that shows the performance trend.

Quoted 148 lines
Quoted 148 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

- very limited usage scenario - it will have a positive effect

only

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

when we have a fixed forwarding mapping: all (or nearly

all) packets

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

from the RX queue are forwarded into the same TX queue.

[Feifei] Although the usage scenario is limited, this usage

scenario

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

has a wide range of applications, such as NIC with one port.

yes, there are NICs with one port, but no guarantee there wouldn't

be

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line

several such NICs within the system.

What I see in my interactions is, a single NIC/DPU is under

utilized for a 2

Quoted 1 line
Quoted 1 line

socket system. Some are adding more sockets to the system to better

utilize the

Quoted 1 line
Quoted 1 line

DPU. The NIC bandwidth continues to grow significantly. I do not

think there will

Quoted 6 lines
Quoted 6 lines

be a multi-DPU per server scenario.

Interesting... from my experience it is visa-versa:
in many cases 200Gb/s is not that much these days to saturate modern

2 socket

Quoted 8 lines
Quoted 8 lines

x86 server.
Though I suppose a lot depends on particular HW and actual workload.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Furtrhermore, I think this is a tradeoff between performance and flexibility.
Our goal is to achieve best performance, this means we need to

give

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

up some flexibility decisively. For example of 'FAST_FREE Mode',

it

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

deletes most of the buffer check (refcnt > 1, external buffer,

chain

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

buffer), chooses a shorest path, and then achieve significant performance

improvement.

Quoted 5 lines
Quoted 5 lines

Wonder did you had a chance to consider mempool-cache ZC API, similar to one we have for the ring?
It would allow us on TX free path to avoid copying mbufs to temporary array on the stack.
Instead we can put them straight from TX SW ring to the mempool

cache.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

That should save extra store/load for mbuf and might help to achieve some performance gain without by-passing mempool. It probably wouldn't be as fast as what you proposing, but might

be

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

fast enough to consider as alternative.
Again, it would be a generic one, so we can avoid all these implications and limitations.

[Feifei] I think this is a good try. However, the most important thing is that if we can bypass the mempool decisively to pursue

the

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

significant performance gains.

I understand the intention, and I personally think this is wrong

and

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

dangerous attitude.
We have mempool abstraction in place for very good reason. So we need to try to improve mempool performance (and API if necessary) at first place, not to avoid it and break our own rules

and

Quoted 2 lines
Quoted 2 lines

recommendations.

Quoted 1 line

The abstraction can be thought of at a higher level. i.e. the

driver manages the

Quoted 1 line
Quoted 1 line

buffer allocation/free and is hidden from the application. The

application does

Quoted 7 lines
Quoted 7 lines

not need to be aware of how these changes are implemented.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

For ZC, there maybe a problem for it in i40e. The reason for that put Tx buffers into temporary is that i40e_tx_entry includes

buffer

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

pointer and index.
Thus we cannot put Tx SW_ring entry into mempool directly, we

need

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

to firstlt extract mbuf pointer. Finally, though we use ZC, we

still

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

can't avoid using a temporary stack to extract Tx buffer

pointers.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

When talking about ZC API for mempool cache I meant something

like:

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

void ** mempool_cache_put_zc_start(struct rte_mempool_cache *mc, uint32_t *nb_elem, uint32_t flags); void mempool_cache_put_zc_finish(struct
rte_mempool_cache *mc, uint32_t nb_elem); i.e. _start_ will return user a pointer inside mp-cache where to put free elems and max

number

Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines

of slots that can be safely filled.
_finish_ will update mc->len.
As an example:

/* expect to free N mbufs */
uint32_t n = N;
void **p = mempool_cache_put_zc_start(mc, &n, ...);

/* free up to n elems */
for (i = 0; i != n; i++) {

     /* get next free mbuf from somewhere */
     mb = extract_and_prefree_mbuf(...);

     /* no more free mbufs for now */
     if (mb == NULL)
        break;

     p[i] = mb;
}
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

/* finalize ZC put, with _i_ freed elems */
mempool_cache_put_zc_finish(mc, i);

That way, I think we can overcome the issue with i40e_tx_entry you mentioned above. Plus it might be useful in other similar places.

Another alternative is obviously to split i40e_tx_entry into two structs (one for mbuf, second for its metadata) and have a

separate

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

array for each of them.
Though with that approach we need to make sure no perf drops will

be

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line

introduced, plus probably more code changes will be required.

Commit '5171b4ee6b6" already does this (in a different way), but

just for

Quoted 1 line
Quoted 1 line

AVX512. Unfortunately, it does not record any performance

improvements. We

Quoted 2 lines
Quoted 1 line

could port this to Arm NEON and look at the performance.

#39 lore raw
Quoted 169 lines

From: Honnappa Nagarahalli [mailto:[email protected]]
Sent: Wednesday, 29 June 2022 23.58

<snip>

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

16/05/2022 07:10, Feifei Wang пишет:

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Currently, the transmit side frees the buffers into the

lcore

Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

cache and the receive side allocates buffers from the lcore

cache.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

The transmit side typically frees 32 buffers resulting in 32*8=256B of stores to lcore cache. The receive side

allocates

Quoted 6 lines
Quoted 6 lines

32

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

buffers and stores them in the receive side software ring, resulting in 32*8=256B of stores and 256B of load from the

lcore cache.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

This patch proposes a mechanism to avoid freeing

to/allocating

Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

from the lcore cache. i.e. the receive side will free the

buffers

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

from transmit side directly into it's software ring. This

will

Quoted 12 lines
Quoted 12 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

avoid the 256B of loads and stores introduced by the lcore

cache.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

It also frees up the cache lines used by the lcore cache.

However, this solution poses several constraints:

1)The receive queue needs to know which transmit queue it

should

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

take the buffers from. The application logic decides which transmit port to use to send out the packets. In many use cases the NIC might have a single port ([1], [2], [3]), in which case

Quoted 6 lines
Quoted 6 lines

a

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

given transmit queue is always mapped to a single receive queue
(1:1 Rx queue: Tx queue). This is easy to configure.

If the NIC has 2 ports (there are several references), then

we

Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

will have
1:2 (RX queue: TX queue) mapping which is still easy to

configure.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

However, if this is generalized to 'N' ports, the configuration can be long. More over the PMD would have to scan a list of transmit queues to pull the buffers from.

Just to re-iterate some generic concerns about this

proposal:

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

- We effectively link RX and TX queues - when this

feature

Quoted 2 lines
Quoted 2 lines

is enabled,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

user can't stop TX queue without stopping linked RX

queue

Quoted 4 lines
Quoted 4 lines

first.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Right now user is free to start/stop any queues at his

will.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

If that feature will allow to link queues from

different

Quoted 2 lines
Quoted 2 lines

ports,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

then even ports will become dependent and user will

have

Quoted 7 lines
Quoted 7 lines

to pay extra

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

care when managing such ports.

[Feifei] When direct rearm enabled, there are two path for thread

to

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

choose. If there are enough Tx freed buffers, Rx can put

buffers

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

from Tx.
Otherwise, Rx will put buffers from mempool as usual. Thus, users

do

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line

not need to pay much attention managing ports.

Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

What I am talking about: right now different port or different

queues

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

of the same port can be treated as independent entities:
in general user is free to start/stop (and even reconfigure in

some

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

cases) one entity without need to stop other entity. I.E user can stop and re-configure TX queue while keep

receiving

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

packets from RX queue.
With direct re-arm enabled, I think it wouldn't be possible

any

Quoted 2 lines
Quoted 2 lines

more:

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

before stopping/reconfiguring TX queue user would have make

sure

Quoted 3 lines
Quoted 3 lines

that

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line

corresponding RX queue wouldn't be used by datapath.

I am trying to understand the problem better. For the TX queue

to

Quoted 10 lines
Quoted 10 lines

be stopped,

Quoted 1 line
Quoted 1 line

the user must have blocked the data plane from accessing the TX

queue.

Quoted 2 lines
Quoted 2 lines

Surely it is user responsibility tnot to call tx_burst() for

stopped/released queue.

Quoted 1 line
Quoted 1 line

The problem is that while TX for that queue is stopped, RX for

related queue still

Quoted 2 lines
Quoted 2 lines

can continue.
So rx_burst() will try to read/modify TX queue data, that might

be

Quoted 5 lines
Quoted 5 lines

already freed,

Quoted 4 lines
Quoted 1 line

or simultaneously modified by control path.

Understood, agree on the issue

Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line

Again, it all can be mitigated by carefully re-designing and

modifying control and

Quoted 1 line
Quoted 1 line

data-path inside user app - by doing extra checks and

synchronizations, etc.

Quoted 1 line
Quoted 1 line

But from practical point - I presume most of users simply would

avoid using this

Quoted 2 lines
Quoted 1 line

feature due all potential problems it might cause.

That is subjective, it all depends on the performance

improvements

Quoted 2 lines
Quoted 2 lines

users see in their application.

Quoted 1 line

IMO, the performance improvement seen with this patch is worth

few

Quoted 7 lines
Quoted 5 lines

changes.

Yes, it is subjective till some extent, though my feeling that it might end-up being sort of synthetic improvement used only by some show-case benchmarks.

I believe that one specific important use case has already been

mentioned, so I

Quoted 1 line

don't think this is a benchmark only feature.

+1

Quoted 8 lines
Quoted 6 lines

From my perspective, it would be much more plausible, if we can
introduce some sort of generic improvement, that doesn't impose all these extra constraints and implications.
Like one, discussed below in that thread with ZC mempool approach.

Considering this feature from a high level perspective, I agree with

Konstantin's

Quoted 1 line

concerns, so I'll also support his views.

We did hack the ZC mempool approach [1], level of improvement is pretty small compared with this patch.

[1] http://patches.dpdk.org/project/dpdk/patch/20220613055136.1949784-
[email protected]/

Quoted 2 lines

If this patch is supposed to be a generic feature, please add support

for it in all

Quoted 1 line

NIC PMDs, not just one. (Regardless if the feature is defined as 1:1

mapping or

Quoted 1 line

N:M mapping.) It is purely software, so it should be available for

all PMDs, not

Quoted 1 line

just your favorite hardware! Consider the "fast mbuf free" feature,

which is

Quoted 1 line

pure software; why is that feature not implemented in all PMDs?

Agree, it is good to have it supported in all the drivers. We do not have a favorite hardware, just picked a PMD which we are more familiar with. We do plan to implement in other prominent PMDs.

Quoted 1 line
Quoted 7 lines
Quoted 1 line

A secondary point I'm making here is that this specific feature will

lead to an

Quoted 1 line

enormous amount of copy-paste code, instead of a generic library

function

Quoted 1 line

easily available for all PMDs.

Are you talking about the i40e driver code in specific? If yes, agree we should avoid copy-paste and we will look to reduce that.

Yes, I am talking about the code that needs to be copied into all prominent PMDs. Perhaps you can move the majority of it into a common directory, if not in a generic library, so the modification per PMD becomes smaller. (I see the same copy-paste issue with the "fast mbuf free" feature, if to be supported by other than the i40e PMD.)

Please note that I do not expect you to implement this feature in other PMDs than you need. I was trying to make the point that implementing a software feature in a PMD requires copy-pasting to other PMDs, which can require a big effort; while implementing it in a library and calling the library from the PMDs require a smaller effort per PMD. I intentionally phrased it somewhat provokingly, and was lucky not to offend anyone. :-)

Quoted 47 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Like Feifei says, the RX side has the normal packet allocation path

Quoted 7 lines
Quoted 7 lines

still available.

Quoted 1 line
Quoted 1 line
Quoted 1 line

Also this sounds like a corner case to me, we can handle this

through checks in

Quoted 4 lines
Quoted 4 lines

the queue_stop API.

Depends.
if it would be allowed to link queues only from the same port,

then

Quoted 6 lines
Quoted 6 lines

yes, extra

Quoted 2 lines
Quoted 2 lines

checks for queue-stop might be enough.
As right now DPDK doesn't allow user to change number of queues

without

Quoted 2 lines
Quoted 2 lines

dev_stop() first.
Though if it would be allowed to link queues from different

ports,

Quoted 4 lines
Quoted 4 lines

then situation

Quoted 2 lines
Quoted 2 lines

will be much worse.
Right now ports are totally independent entities (except some

special cases like

Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines

link-bonding, etc.).
As one port can keep doing RX/TX, second one can be stopped, re-

confgured,

Quoted 1 line
Quoted 1 line

even detached, and newly attached device might re-use same port

number.

Quoted 1 line

I see this as a similar restriction to the one discussed above.

Yes, they are similar in principal, though I think that the case

with

Quoted 10 lines
Quoted 8 lines

queues from different port would make things much more complex.

Quoted 1 line

Do you see any issues if we enforce this with checks?

Hard to tell straightway, a lot will depend how smart such implementation would be.
Usually DPDK tends not to avoid heavy synchronizations within its data-path functions.

Certainly! Implementing more and more of such features in the PMDs

will lead

Quoted 1 line

to longer and longer data plane code paths in the PMDs. It is the

Quoted 8 lines

"salami

Quoted 1 line

method", where each small piece makes no performance difference, but

they all

Quoted 1 line

add up, and eventually the sum of them does impact the performance of

the

Quoted 1 line

general use case negatively.

It would be good to have a test running in UNH that shows the performance trend.

+1

Quoted 168 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

- very limited usage scenario - it will have a positive

effect

Quoted 9 lines
Quoted 9 lines

only

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

when we have a fixed forwarding mapping: all (or nearly

all) packets

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

from the RX queue are forwarded into the same TX queue.

[Feifei] Although the usage scenario is limited, this usage

scenario

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line

has a wide range of applications, such as NIC with one port.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

yes, there are NICs with one port, but no guarantee there wouldn't

be

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line

several such NICs within the system.

What I see in my interactions is, a single NIC/DPU is under

utilized for a 2

Quoted 1 line
Quoted 1 line

socket system. Some are adding more sockets to the system to

better

Quoted 6 lines
Quoted 6 lines

utilize the

Quoted 1 line
Quoted 1 line

DPU. The NIC bandwidth continues to grow significantly. I do not

think there will

Quoted 3 lines
Quoted 3 lines

be a multi-DPU per server scenario.

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

Interesting... from my experience it is visa-versa:
in many cases 200Gb/s is not that much these days to saturate modern

2 socket

Quoted 2 lines
Quoted 2 lines

x86 server.
Though I suppose a lot depends on particular HW and actual

workload.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Furtrhermore, I think this is a tradeoff between performance

and

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

flexibility.
Our goal is to achieve best performance, this means we need

to

Quoted 2 lines
Quoted 2 lines

give

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

up some flexibility decisively. For example of 'FAST_FREE

Mode',

Quoted 2 lines
Quoted 2 lines

it

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

deletes most of the buffer check (refcnt > 1, external

buffer,

Quoted 5 lines
Quoted 5 lines

chain

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

buffer), chooses a shorest path, and then achieve significant performance

improvement.

Quoted 1 line
Quoted 1 line

Wonder did you had a chance to consider mempool-cache ZC

API,

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

similar to one we have for the ring?
It would allow us on TX free path to avoid copying mbufs to temporary array on the stack.
Instead we can put them straight from TX SW ring to the

mempool

Quoted 11 lines
Quoted 11 lines

cache.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

That should save extra store/load for mbuf and might help to achieve some performance gain without by-passing mempool. It probably wouldn't be as fast as what you proposing, but might

be

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

fast enough to consider as alternative.
Again, it would be a generic one, so we can avoid all these implications and limitations.

[Feifei] I think this is a good try. However, the most

important

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

thing is that if we can bypass the mempool decisively to

pursue

Quoted 4 lines
Quoted 4 lines

the

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

significant performance gains.

I understand the intention, and I personally think this is

wrong

Quoted 16 lines
Quoted 16 lines

and

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

dangerous attitude.
We have mempool abstraction in place for very good reason. So we need to try to improve mempool performance (and API if necessary) at first place, not to avoid it and break our own rules

and

Quoted 2 lines
Quoted 2 lines

recommendations.

Quoted 1 line

The abstraction can be thought of at a higher level. i.e. the

driver manages the

Quoted 1 line
Quoted 1 line

buffer allocation/free and is hidden from the application. The

application does

Quoted 4 lines
Quoted 4 lines

not need to be aware of how these changes are implemented.

Quoted 3 lines
Quoted 3 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

For ZC, there maybe a problem for it in i40e. The reason for that put Tx buffers into temporary is that i40e_tx_entry includes

buffer

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

pointer and index.
Thus we cannot put Tx SW_ring entry into mempool directly, we

need

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

to firstlt extract mbuf pointer. Finally, though we use ZC,

we

Quoted 7 lines
Quoted 7 lines

still

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

can't avoid using a temporary stack to extract Tx buffer

pointers.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

When talking about ZC API for mempool cache I meant something

like:

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

void ** mempool_cache_put_zc_start(struct rte_mempool_cache

*mc,

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

uint32_t *nb_elem, uint32_t flags); void mempool_cache_put_zc_finish(struct rte_mempool_cache *mc, uint32_t nb_elem); i.e. _start_ will return user a pointer inside mp-cache where to put free elems

and

Quoted 20 lines
Quoted 20 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

max

number

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines

of slots that can be safely filled.
_finish_ will update mc->len.
As an example:

/* expect to free N mbufs */
uint32_t n = N;
void **p = mempool_cache_put_zc_start(mc, &n, ...);

/* free up to n elems */
for (i = 0; i != n; i++) {

     /* get next free mbuf from somewhere */
     mb = extract_and_prefree_mbuf(...);

     /* no more free mbufs for now */
     if (mb == NULL)
        break;
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
     p[i] = mb;
}

/* finalize ZC put, with _i_ freed elems */
mempool_cache_put_zc_finish(mc, i);

That way, I think we can overcome the issue with i40e_tx_entry you mentioned above. Plus it might be useful in other similar

places.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Another alternative is obviously to split i40e_tx_entry into

two

Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

structs (one for mbuf, second for its metadata) and have a

separate

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

array for each of them.
Though with that approach we need to make sure no perf drops

will

Quoted 3 lines
Quoted 3 lines

be

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line

introduced, plus probably more code changes will be required.

Commit '5171b4ee6b6" already does this (in a different way),

but

Quoted 6 lines
Quoted 6 lines

just for

Quoted 1 line
Quoted 1 line

AVX512. Unfortunately, it does not record any performance

improvements. We

Quoted 2 lines
Quoted 1 line

could port this to Arm NEON and look at the performance.

#40 lore raw

<snip>

Quoted 175 lines
Quoted 175 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

16/05/2022 07:10, Feifei Wang пишет:

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Currently, the transmit side frees the buffers into the

lcore

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

cache and the receive side allocates buffers from the lcore

cache.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

The transmit side typically frees 32 buffers resulting in 32*8=256B of stores to lcore cache. The receive side

allocates

Quoted 6 lines
Quoted 6 lines

32

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

buffers and stores them in the receive side software ring, resulting in 32*8=256B of stores and 256B of load from the

lcore cache.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

This patch proposes a mechanism to avoid freeing

to/allocating

Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

from the lcore cache. i.e. the receive side will free the

buffers

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

from transmit side directly into it's software ring. This

will

Quoted 12 lines
Quoted 12 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

avoid the 256B of loads and stores introduced by the lcore

cache.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

It also frees up the cache lines used by the lcore cache.

However, this solution poses several constraints:

1)The receive queue needs to know which transmit queue it

should

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

take the buffers from. The application logic decides which transmit port to use to send out the packets. In many use cases the NIC might have a single port ([1], [2], [3]), in which case

Quoted 7 lines
Quoted 7 lines

a

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

given transmit queue is always mapped to a single receive queue
(1:1 Rx queue: Tx queue). This is easy to configure.

If the NIC has 2 ports (there are several references), then

we

Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

will have
1:2 (RX queue: TX queue) mapping which is still easy to

configure.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

However, if this is generalized to 'N' ports, the configuration can be long. More over the PMD would have to scan a list of transmit queues to pull the buffers from.

Just to re-iterate some generic concerns about this

proposal:

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

- We effectively link RX and TX queues - when this

feature

Quoted 2 lines
Quoted 2 lines

is enabled,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

user can't stop TX queue without stopping linked RX

queue

Quoted 5 lines
Quoted 5 lines

first.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Right now user is free to start/stop any queues at
his

will.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

If that feature will allow to link queues from

different

Quoted 2 lines
Quoted 2 lines

ports,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

then even ports will become dependent and user will

have

Quoted 7 lines
Quoted 7 lines

to pay extra

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

care when managing such ports.

[Feifei] When direct rearm enabled, there are two path for thread

to

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

choose. If there are enough Tx freed buffers, Rx can put

buffers

Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

from Tx.
Otherwise, Rx will put buffers from mempool as usual. Thus, users

do

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

not need to pay much attention managing ports.

What I am talking about: right now different port or different

Quoted 7 lines
Quoted 7 lines

queues

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

of the same port can be treated as independent entities:
in general user is free to start/stop (and even reconfigure in

some

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

cases) one entity without need to stop other entity. I.E user can stop and re-configure TX queue while keep

receiving

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

packets from RX queue.
With direct re-arm enabled, I think it wouldn't be possible

any

Quoted 2 lines
Quoted 2 lines

more:

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

before stopping/reconfiguring TX queue user would have make

sure

Quoted 3 lines
Quoted 3 lines

that

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line

corresponding RX queue wouldn't be used by datapath.

I am trying to understand the problem better. For the TX queue

to

Quoted 10 lines
Quoted 10 lines

be stopped,

Quoted 1 line
Quoted 1 line

the user must have blocked the data plane from accessing the TX

queue.

Quoted 2 lines
Quoted 2 lines

Surely it is user responsibility tnot to call tx_burst() for

stopped/released queue.

Quoted 1 line
Quoted 1 line

The problem is that while TX for that queue is stopped, RX for

related queue still

Quoted 2 lines
Quoted 2 lines

can continue.
So rx_burst() will try to read/modify TX queue data, that might

be

Quoted 5 lines
Quoted 5 lines

already freed,

Quoted 4 lines
Quoted 1 line

or simultaneously modified by control path.

Understood, agree on the issue

Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line

Again, it all can be mitigated by carefully re-designing and

modifying control and

Quoted 1 line
Quoted 1 line

data-path inside user app - by doing extra checks and

synchronizations, etc.

Quoted 1 line
Quoted 1 line

But from practical point - I presume most of users simply would

avoid using this

Quoted 2 lines
Quoted 1 line

feature due all potential problems it might cause.

That is subjective, it all depends on the performance

improvements

Quoted 2 lines
Quoted 2 lines

users see in their application.

Quoted 1 line

IMO, the performance improvement seen with this patch is worth

few

Quoted 7 lines
Quoted 5 lines

changes.

Yes, it is subjective till some extent, though my feeling that it might end-up being sort of synthetic improvement used only by some show-case benchmarks.

I believe that one specific important use case has already been

mentioned, so I

Quoted 1 line

don't think this is a benchmark only feature.

+1

Quoted 8 lines
Quoted 6 lines

From my perspective, it would be much more plausible, if we can
introduce some sort of generic improvement, that doesn't impose all these extra constraints and implications.
Like one, discussed below in that thread with ZC mempool approach.

Considering this feature from a high level perspective, I agree with

Konstantin's

Quoted 1 line

concerns, so I'll also support his views.

We did hack the ZC mempool approach [1], level of improvement is pretty small compared with this patch.

[1] http://patches.dpdk.org/project/dpdk/patch/20220613055136.1949784-
[email protected]/

Quoted 3 lines

If this patch is supposed to be a generic feature, please add support

for it in all

Quoted 1 line

NIC PMDs, not just one. (Regardless if the feature is defined as 1:1

mapping or

Quoted 1 line

N:M mapping.) It is purely software, so it should be available for

all PMDs, not

Quoted 1 line

just your favorite hardware! Consider the "fast mbuf free" feature,

which is

Quoted 1 line

pure software; why is that feature not implemented in all PMDs?

Agree, it is good to have it supported in all the drivers. We do not have a favorite hardware, just picked a PMD which we are more familiar with. We do plan to implement in other prominent PMDs.

Quoted 2 lines

A secondary point I'm making here is that this specific feature will

lead to an

Quoted 1 line

enormous amount of copy-paste code, instead of a generic library

function

Quoted 1 line

easily available for all PMDs.

Quoted 8 lines
Quoted 2 lines

Are you talking about the i40e driver code in specific? If yes, agree we should avoid copy-paste and we will look to reduce that.

Yes, I am talking about the code that needs to be copied into all prominent PMDs. Perhaps you can move the majority of it into a common directory, if not in a generic library, so the modification per PMD becomes smaller. (I see the same copy-paste issue with the "fast mbuf free" feature, if to be supported by other than the i40e PMD.)

The current abstraction does not allow for common code at this (lower) level across all the PMDs. If we look at "fast free", it is accessing the device private structure for the list of buffers to free. If it needs to be common code, this needs to be lifted up along with other dependent configuration thresholds etc.

Quoted 239 lines

Please note that I do not expect you to implement this feature in other PMDs than you need. I was trying to make the point that implementing a software feature in a PMD requires copy-pasting to other PMDs, which can require a big effort; while implementing it in a library and calling the library from the PMDs require a smaller effort per PMD. I intentionally phrased it somewhat provokingly, and was lucky not to offend anyone. :-)

Quoted 49 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Like Feifei says, the RX side has the normal packet allocation path

Quoted 7 lines
Quoted 7 lines

still available.

Quoted 1 line
Quoted 1 line
Quoted 1 line

Also this sounds like a corner case to me, we can handle this

through checks in

Quoted 4 lines
Quoted 4 lines

the queue_stop API.

Depends.
if it would be allowed to link queues only from the same port,

then

Quoted 6 lines
Quoted 6 lines

yes, extra

Quoted 2 lines
Quoted 2 lines

checks for queue-stop might be enough.
As right now DPDK doesn't allow user to change number of queues

without

Quoted 2 lines
Quoted 2 lines

dev_stop() first.
Though if it would be allowed to link queues from different

ports,

Quoted 7 lines
Quoted 7 lines

then situation

Quoted 2 lines
Quoted 2 lines

will be much worse.
Right now ports are totally independent entities (except some

special cases like

Quoted 3 lines
Quoted 3 lines

link-bonding, etc.).
As one port can keep doing RX/TX, second one can be stopped, re-

Quoted 6 lines
Quoted 6 lines

confgured,

Quoted 1 line
Quoted 1 line

even detached, and newly attached device might re-use same port

number.

Quoted 1 line

I see this as a similar restriction to the one discussed above.

Yes, they are similar in principal, though I think that the case

with

Quoted 10 lines
Quoted 8 lines

queues from different port would make things much more complex.

Quoted 1 line

Do you see any issues if we enforce this with checks?

Hard to tell straightway, a lot will depend how smart such implementation would be.
Usually DPDK tends not to avoid heavy synchronizations within its data-path functions.

Certainly! Implementing more and more of such features in the PMDs

will lead

Quoted 1 line

to longer and longer data plane code paths in the PMDs. It is the

"salami

Quoted 8 lines
Quoted 1 line

method", where each small piece makes no performance difference, but

they all

Quoted 2 lines

add up, and eventually the sum of them does impact the performance of

the

Quoted 1 line

general use case negatively.

It would be good to have a test running in UNH that shows the performance trend.

+1

Quoted 172 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

- very limited usage scenario - it will have a positive

effect

Quoted 10 lines
Quoted 10 lines

only

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

when we have a fixed forwarding mapping: all (or
nearly

all) packets

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

from the RX queue are forwarded into the same TX queue.

[Feifei] Although the usage scenario is limited, this usage

scenario

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line

has a wide range of applications, such as NIC with one port.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

yes, there are NICs with one port, but no guarantee there wouldn't

be

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line

several such NICs within the system.

What I see in my interactions is, a single NIC/DPU is under

utilized for a 2

Quoted 1 line
Quoted 1 line

socket system. Some are adding more sockets to the system to

better

Quoted 7 lines
Quoted 7 lines

utilize the

Quoted 2 lines
Quoted 2 lines

DPU. The NIC bandwidth continues to grow significantly. I do not

think there will

Quoted 3 lines
Quoted 3 lines

be a multi-DPU per server scenario.

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

Interesting... from my experience it is visa-versa:
in many cases 200Gb/s is not that much these days to saturate modern

2 socket

Quoted 2 lines
Quoted 2 lines

x86 server.
Though I suppose a lot depends on particular HW and actual

workload.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Furtrhermore, I think this is a tradeoff between performance

and

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

flexibility.
Our goal is to achieve best performance, this means we need

to

Quoted 2 lines
Quoted 2 lines

give

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

up some flexibility decisively. For example of 'FAST_FREE

Mode',

Quoted 2 lines
Quoted 2 lines

it

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

deletes most of the buffer check (refcnt > 1, external

buffer,

Quoted 5 lines
Quoted 5 lines

chain

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

buffer), chooses a shorest path, and then achieve
significant performance

improvement.

Quoted 1 line
Quoted 1 line

Wonder did you had a chance to consider mempool-cache ZC

API,

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

similar to one we have for the ring?
It would allow us on TX free path to avoid copying mbufs to temporary array on the stack.
Instead we can put them straight from TX SW ring to the

mempool

Quoted 11 lines
Quoted 11 lines

cache.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

That should save extra store/load for mbuf and might help to achieve some performance gain without by-passing mempool. It probably wouldn't be as fast as what you proposing, but might

be

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

fast enough to consider as alternative.
Again, it would be a generic one, so we can avoid all these implications and limitations.

[Feifei] I think this is a good try. However, the most

important

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

thing is that if we can bypass the mempool decisively to

pursue

Quoted 4 lines
Quoted 4 lines

the

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

significant performance gains.

I understand the intention, and I personally think this is

wrong

Quoted 19 lines
Quoted 19 lines

and

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

dangerous attitude.
We have mempool abstraction in place for very good reason. So we need to try to improve mempool performance (and API if necessary) at first place, not to avoid it and break our own rules

and

Quoted 2 lines
Quoted 2 lines

recommendations.

Quoted 1 line

The abstraction can be thought of at a higher level. i.e. the

driver manages the

Quoted 1 line
Quoted 1 line

buffer allocation/free and is hidden from the application. The

application does

Quoted 7 lines
Quoted 7 lines

not need to be aware of how these changes are implemented.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

For ZC, there maybe a problem for it in i40e. The reason for that put Tx buffers into temporary is that i40e_tx_entry includes

Quoted 6 lines
Quoted 6 lines

buffer

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

pointer and index.
Thus we cannot put Tx SW_ring entry into mempool directly, we

need

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

to firstlt extract mbuf pointer. Finally, though we use ZC,

we

Quoted 7 lines
Quoted 7 lines

still

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

can't avoid using a temporary stack to extract Tx buffer

pointers.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

When talking about ZC API for mempool cache I meant something

like:

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

void ** mempool_cache_put_zc_start(struct rte_mempool_cache

*mc,

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

uint32_t *nb_elem, uint32_t flags); void mempool_cache_put_zc_finish(struct rte_mempool_cache *mc, uint32_t nb_elem); i.e. _start_ will return user a pointer inside mp-cache where to put free elems

and

Quoted 23 lines
Quoted 23 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

max

number

Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines

of slots that can be safely filled.
_finish_ will update mc->len.
As an example:

/* expect to free N mbufs */
uint32_t n = N;
void **p = mempool_cache_put_zc_start(mc, &n, ...);

/* free up to n elems */
for (i = 0; i != n; i++) {

     /* get next free mbuf from somewhere */
     mb = extract_and_prefree_mbuf(...);

     /* no more free mbufs for now */
     if (mb == NULL)
        break;

     p[i] = mb;
}
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

/* finalize ZC put, with _i_ freed elems */
mempool_cache_put_zc_finish(mc, i);

That way, I think we can overcome the issue with i40e_tx_entry you mentioned above. Plus it might be useful in other similar

places.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Another alternative is obviously to split i40e_tx_entry into

two

Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

structs (one for mbuf, second for its metadata) and have a

separate

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

array for each of them.
Though with that approach we need to make sure no perf drops

will

Quoted 3 lines
Quoted 3 lines

be

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line

introduced, plus probably more code changes will be required.

Commit '5171b4ee6b6" already does this (in a different way),

but

Quoted 6 lines
Quoted 6 lines

just for

Quoted 1 line
Quoted 1 line

AVX512. Unfortunately, it does not record any performance

improvements. We

Quoted 2 lines
Quoted 1 line

could port this to Arm NEON and look at the performance.

#41 lore raw
Quoted 229 lines

From: Honnappa Nagarahalli [mailto:[email protected]]
Sent: Friday, 1 July 2022 21.31

<snip>

Quoted 9 lines
Quoted 9 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

16/05/2022 07:10, Feifei Wang пишет:

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Currently, the transmit side frees the buffers into the

lcore

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

cache and the receive side allocates buffers from the lcore

cache.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

The transmit side typically frees 32 buffers resulting

in

Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

32*8=256B of stores to lcore cache. The receive side

allocates

Quoted 2 lines
Quoted 2 lines

32

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

buffers and stores them in the receive side software

ring,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

resulting in 32*8=256B of stores and 256B of load from

the

Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

lcore cache.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

This patch proposes a mechanism to avoid freeing

to/allocating

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

from the lcore cache. i.e. the receive side will free

the

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

buffers

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

from transmit side directly into it's software ring.

This

Quoted 2 lines
Quoted 2 lines

will

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

avoid the 256B of loads and stores introduced by the

lcore

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

cache.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

It also frees up the cache lines used by the lcore

cache.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

However, this solution poses several constraints:

1)The receive queue needs to know which transmit queue

it

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

should

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

take the buffers from. The application logic decides

which

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

transmit port to use to send out the packets. In many

use

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

cases the NIC might have a single port ([1], [2], [3]),

in

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

which case

a

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

given transmit queue is always mapped to a single

receive

Quoted 11 lines
Quoted 11 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

queue
(1:1 Rx queue: Tx queue). This is easy to configure.

If the NIC has 2 ports (there are several references), then

we

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

will have
1:2 (RX queue: TX queue) mapping which is still easy to

configure.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

However, if this is generalized to 'N' ports, the configuration can be long. More over the PMD would have

to

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

scan a list of transmit queues to pull the buffers

from.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Just to re-iterate some generic concerns about this

proposal:

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

- We effectively link RX and TX queues - when this

feature

Quoted 2 lines
Quoted 2 lines

is enabled,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

user can't stop TX queue without stopping linked

RX

Quoted 9 lines
Quoted 9 lines

queue

Quoted 5 lines
Quoted 5 lines

first.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Right now user is free to start/stop any queues at
his

will.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

If that feature will allow to link queues from

different

Quoted 2 lines
Quoted 2 lines

ports,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

then even ports will become dependent and user

will

Quoted 5 lines
Quoted 5 lines

have

Quoted 4 lines
Quoted 4 lines

to pay extra

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

care when managing such ports.

[Feifei] When direct rearm enabled, there are two path

for

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

thread

to

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

choose. If there are enough Tx freed buffers, Rx can put

buffers

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

from Tx.
Otherwise, Rx will put buffers from mempool as usual.

Thus,

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

users

do

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

not need to pay much attention managing ports.

What I am talking about: right now different port or different

queues

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

of the same port can be treated as independent entities:
in general user is free to start/stop (and even

reconfigure

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

in

some

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

cases) one entity without need to stop other entity. I.E user can stop and re-configure TX queue while keep

receiving

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

packets from RX queue.
With direct re-arm enabled, I think it wouldn't be

possible

Quoted 3 lines
Quoted 3 lines

any

Quoted 2 lines
Quoted 2 lines

more:

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

before stopping/reconfiguring TX queue user would have

make

Quoted 4 lines
Quoted 4 lines

sure

Quoted 3 lines
Quoted 3 lines

that

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line

corresponding RX queue wouldn't be used by datapath.

I am trying to understand the problem better. For the TX

queue

Quoted 3 lines
Quoted 3 lines

to

Quoted 2 lines
Quoted 2 lines

be stopped,

Quoted 1 line
Quoted 1 line

the user must have blocked the data plane from accessing the

TX

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

queue.

Quoted 2 lines
Quoted 2 lines

Surely it is user responsibility tnot to call tx_burst() for

stopped/released queue.

Quoted 1 line
Quoted 1 line

The problem is that while TX for that queue is stopped, RX

for

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

related queue still

Quoted 2 lines
Quoted 2 lines

can continue.
So rx_burst() will try to read/modify TX queue data, that

might

Quoted 11 lines
Quoted 11 lines

be

Quoted 10 lines
Quoted 10 lines

already freed,

Quoted 5 lines
Quoted 1 line

or simultaneously modified by control path.

Understood, agree on the issue

Quoted 2 lines

Again, it all can be mitigated by carefully re-designing and

modifying control and

Quoted 1 line
Quoted 1 line

data-path inside user app - by doing extra checks and

synchronizations, etc.

Quoted 1 line
Quoted 1 line

But from practical point - I presume most of users simply

would

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

avoid using this

Quoted 2 lines
Quoted 1 line

feature due all potential problems it might cause.

That is subjective, it all depends on the performance

improvements

Quoted 2 lines
Quoted 2 lines

users see in their application.

Quoted 1 line

IMO, the performance improvement seen with this patch is

worth

Quoted 4 lines
Quoted 4 lines

few

Quoted 3 lines
Quoted 3 lines

changes.

Yes, it is subjective till some extent, though my feeling that

it

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

might end-up being sort of synthetic improvement used only by

some

Quoted 9 lines
Quoted 9 lines
Quoted 3 lines
Quoted 1 line

show-case benchmarks.

I believe that one specific important use case has already been

mentioned, so I

Quoted 1 line

don't think this is a benchmark only feature.

+1

Quoted 2 lines
Quoted 2 lines

From my perspective, it would be much more plausible, if we

can

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

introduce some sort of generic improvement, that doesn't impose all these extra constraints and implications.
Like one, discussed below in that thread with ZC mempool

approach.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

Considering this feature from a high level perspective, I agree

with

Quoted 6 lines
Quoted 6 lines

Konstantin's

Quoted 1 line

concerns, so I'll also support his views.

We did hack the ZC mempool approach [1], level of improvement is pretty small compared with this patch.

[1]

http://patches.dpdk.org/project/dpdk/patch/20220613055136.1949784-

Quoted 7 lines
Quoted 7 lines

[email protected]/

Quoted 3 lines

If this patch is supposed to be a generic feature, please add support

for it in all

Quoted 1 line

NIC PMDs, not just one. (Regardless if the feature is defined as

1:1

Quoted 2 lines
Quoted 2 lines

mapping or

Quoted 1 line

N:M mapping.) It is purely software, so it should be available

for

Quoted 2 lines
Quoted 2 lines

all PMDs, not

Quoted 1 line

just your favorite hardware! Consider the "fast mbuf free"

feature,

Quoted 3 lines
Quoted 3 lines

which is

Quoted 1 line

pure software; why is that feature not implemented in all PMDs?

Agree, it is good to have it supported in all the drivers. We do

not

Quoted 1 line
Quoted 1 line

have a favorite hardware, just picked a PMD which we are more

familiar

Quoted 4 lines
Quoted 4 lines

with. We do plan to implement in other prominent PMDs.

Quoted 2 lines

A secondary point I'm making here is that this specific feature

will

Quoted 5 lines
Quoted 5 lines

lead to an

Quoted 1 line

enormous amount of copy-paste code, instead of a generic library

function

Quoted 1 line

easily available for all PMDs.

Are you talking about the i40e driver code in specific? If yes,

agree

Quoted 3 lines
Quoted 1 line

we should avoid copy-paste and we will look to reduce that.

Yes, I am talking about the code that needs to be copied into all

prominent

Quoted 1 line

PMDs. Perhaps you can move the majority of it into a common

directory, if not

Quoted 1 line

in a generic library, so the modification per PMD becomes smaller. (I

see the

Quoted 8 lines
Quoted 1 line

same copy-paste issue with the "fast mbuf free" feature, if to be

supported by

Quoted 1 line

other than the i40e PMD.)

The current abstraction does not allow for common code at this (lower) level across all the PMDs. If we look at "fast free", it is accessing the device private structure for the list of buffers to free. If it needs to be common code, this needs to be lifted up along with other dependent configuration thresholds etc.

Exactly. The "direct re-arm" feature has some of the same design properties as "fast free": They are both purely software features, and they are both embedded deeply in the PMD code.

I don't know if it is possible, but perhaps we could redesign the private buffer structures of the PMDs, so the first part of it is common across PMDs, and the following part is truly private. The common part would obviously hold the mbuf pointer. That way, a common library could manipulate the public part.

Or perhaps some other means could be provided for a common library to manipulate the common parts of a private structure, e.g. a common fast_free function could take a few parameters to work on private buffer structures: void * buffer_array, size_t element_size, size_t mbuf_offset_in_element, unsigned int num_elements.

Quoted 12 lines
Quoted 3 lines

Please note that I do not expect you to implement this feature in

other PMDs

Quoted 1 line

than you need. I was trying to make the point that implementing a

software

Quoted 1 line

feature in a PMD requires copy-pasting to other PMDs, which can

require a big

Quoted 1 line

effort; while implementing it in a library and calling the library

from the PMDs

Quoted 2 lines

require a smaller effort per PMD. I intentionally phrased it somewhat provokingly, and was lucky not to offend anyone. :-)

Unless we find a better solution, copy-paste code across PMDs seems to be the only way to implement such features. And I agree that they should not be blocked due to "code complexity" or "copy-paste" arguments, if they are impossible to implement in a more "correct" way.

The DPDK community should accept such contributions to the common code, and be grateful that they don't just go into private forks of DPDK!

#42 lore raw
In reply to Konstantin Ananyev
Quoted 147 lines

-----邮件原件-----
发件人: Konstantin Ananyev <[email protected]> 发送时间: Tuesday, May 24, 2022 9:26 AM
收件人: Feifei Wang <[email protected]>
抄送: nd <[email protected]>; [email protected]; Ruifeng Wang <[email protected]>; Honnappa Nagarahalli <[email protected]>
主题: Re: [PATCH v1 0/5] Direct re-arming of buffers on receive side

[[email protected] appears similar to someone who previously sent you email, but may not be that person. Learn why this could be a risk at
https://aka.ms/LearnAboutSenderIdentification.]

16/05/2022 07:10, Feifei Wang пишет:

Quoted 38 lines
Quoted 30 lines
Quoted 23 lines

Currently, the transmit side frees the buffers into the lcore cache and the receive side allocates buffers from the lcore cache. The transmit side typically frees 32 buffers resulting in 32*8=256B of stores to lcore cache. The receive side allocates 32 buffers and stores them in the receive side software ring, resulting in 32*8=256B of stores and 256B of load from the lcore cache.

This patch proposes a mechanism to avoid freeing to/allocating from the lcore cache. i.e. the receive side will free the buffers from transmit side directly into it's software ring. This will avoid the 256B of loads and stores introduced by the lcore cache. It also frees up the cache lines used by the lcore cache.

However, this solution poses several constraints:

1)The receive queue needs to know which transmit queue it should take the buffers from. The application logic decides which transmit port to use to send out the packets. In many use cases the NIC might have a single port ([1], [2], [3]), in which case a given transmit queue is always mapped to a single receive queue (1:1 Rx queue: Tx queue). This is easy to configure.

Quoted 6 lines

If the NIC has 2 ports (there are several references), then we will have
1:2 (RX queue: TX queue) mapping which is still easy to configure. However, if this is generalized to 'N' ports, the configuration can be long. More over the PMD would have to scan a list of transmit queues to pull the buffers from.

Quoted 7 lines

Just to re-iterate some generic concerns about this proposal:
- We effectively link RX and TX queues - when this feature is enabled, user can't stop TX queue without stopping linked RX queue first. Right now user is free to start/stop any queues at his will. If that feature will allow to link queues from different ports, then even ports will become dependent and user will have to pay extra care when managing such ports.

Quoted 5 lines

[Feifei] When direct rearm enabled, there are two path for thread to choose. If there are enough Tx freed buffers, Rx can put buffers from Tx.
Otherwise, Rx will put buffers from mempool as usual. Thus, users do not need to pay much attention managing ports.

What I am talking about: right now different port or different queues of the same port can be treated as independent entities:
in general user is free to start/stop (and even reconfigure in some cases) one entity without need to stop other entity.
I.E user can stop and re-configure TX queue while keep receiving packets from RX queue.
With direct re-arm enabled, I think it wouldn't be possible any more:
before stopping/reconfiguring TX queue user would have make sure that corresponding RX queue wouldn't be used by datapath.

Quoted 7 lines
Quoted 4 lines

- very limited usage scenario - it will have a positive effect only when we have a fixed forwarding mapping: all (or nearly all) packets from the RX queue are forwarded into the same TX queue.

[Feifei] Although the usage scenario is limited, this usage scenario has a wide range of applications, such as NIC with one port.

yes, there are NICs with one port, but no guarantee there wouldn't be several such NICs within the system.

Quoted 6 lines

Furtrhermore, I think this is a tradeoff between performance and flexibility.
Our goal is to achieve best performance, this means we need to give up some flexibility decisively. For example of 'FAST_FREE Mode', it deletes most of the buffer check (refcnt > 1, external buffer, chain buffer), chooses a shorest path, and then achieve significant performance

improvement.

Quoted 12 lines
Quoted 11 lines

Wonder did you had a chance to consider mempool-cache ZC API, similar to one we have for the ring?
It would allow us on TX free path to avoid copying mbufs to temporary array on the stack.
Instead we can put them straight from TX SW ring to the mempool cache. That should save extra store/load for mbuf and might help to achieve some performance gain without by-passing mempool.
It probably wouldn't be as fast as what you proposing, but might be fast enough to consider as alternative.
Again, it would be a generic one, so we can avoid all these implications and limitations.

Quoted 3 lines

[Feifei] I think this is a good try. However, the most important thing is that if we can bypass the mempool decisively to pursue the significant performance gains.

I understand the intention, and I personally think this is wrong and dangerous attitude.
We have mempool abstraction in place for very good reason.
So we need to try to improve mempool performance (and API if necessary) at first place, not to avoid it and break our own rules and recommendations.

Quoted 6 lines

For ZC, there maybe a problem for it in i40e. The reason for that put Tx buffers into temporary is that i40e_tx_entry includes buffer pointer and index.
Thus we cannot put Tx SW_ring entry into mempool directly, we need to firstlt extract mbuf pointer. Finally, though we use ZC, we still can't avoid using a temporary stack to extract Tx buffer pointers.

When talking about ZC API for mempool cache I meant something like:
void ** mempool_cache_put_zc_start(struct rte_mempool_cache *mc, uint32_t *nb_elem, uint32_t flags); void mempool_cache_put_zc_finish(struct rte_mempool_cache *mc, uint32_t nb_elem); i.e. _start_ will return user a pointer inside mp-cache where to put free elems and max number of slots that can be safely filled. _finish_ will update mc->len.
As an example:

/* expect to free N mbufs */
uint32_t n = N;
void **p = mempool_cache_put_zc_start(mc, &n, ...);

/* free up to n elems */
for (i = 0; i != n; i++) {

   /* get next free mbuf from somewhere */
   mb = extract_and_prefree_mbuf(...);

   /* no more free mbufs for now */
   if (mb == NULL)
      break;

   p[i] = mb;
}

/* finalize ZC put, with _i_ freed elems */ mempool_cache_put_zc_finish(mc, i);

Quoted 7 lines

That way, I think we can overcome the issue with i40e_tx_entry you mentioned above. Plus it might be useful in other similar places.

Another alternative is obviously to split i40e_tx_entry into two structs (one for mbuf, second for its metadata) and have a separate array for each of them. Though with that approach we need to make sure no perf drops will be introduced, plus probably more code changes will be required.

[Feifei] I just uploaded a RFC patch to the community:
http://patches.dpdk.org/project/dpdk/patch/[email protected]/
I do not use ZC api, and refer to i40e avx512 path, directly put mempool cache out of API. You can take a look when you are free. Thanks very much.

Quoted 4 lines
#43 lore raw

v3 cover Direct re-arming of buffers on receive side

In reply to Feifei Wang

Currently, the transmit side frees the buffers into the lcore cache and the receive side allocates buffers from the lcore cache. The transmit side typically frees 32 buffers resulting in 32*8=256B of stores to lcore cache. The receive side allocates 32 buffers and stores them in the receive side software ring, resulting in 32*8=256B of stores and 256B of load from the lcore cache.

This patch proposes a mechanism to avoid freeing to/allocating from the lcore cache. i.e. the receive side will free the buffers from transmit side directly into its software ring. This will avoid the 256B of loads and stores introduced by the lcore cache. It also frees up the cache lines used by the lcore cache.

However, this solution poses several constraints:

1)The receive queue needs to know which transmit queue it should take the buffers from. The application logic decides which transmit port to use to send out the packets. In many use cases the NIC might have a single port ([1], [2], [3]), in which case a given transmit queue is always mapped to a single receive queue (1:1 Rx queue: Tx queue). This is easy to configure.

If the NIC has 2 ports (there are several references), then we will have 1:2 (RX queue: TX queue) mapping which is still easy to configure. However, if this is generalized to 'N' ports, the configuration can be long. More over the PMD would have to scan a list of transmit queues to pull the buffers from.

2)The other factor that needs to be considered is 'run-to-completion' vs 'pipeline' models. In the run-to-completion model, the receive side and the transmit side are running on the same lcore serially. In the pipeline model. The receive side and transmit side might be running on different lcores in parallel. This requires locking. This is not supported at this point.

3)Tx and Rx buffers must be from the same mempool. And we also must ensure Tx buffer free number is equal to Rx buffer free number. Thus, 'tx_next_dd' can be updated correctly in direct-rearm mode. This is due to tx_next_dd is a variable to compute tx sw-ring free location. Its value will be one more round than the position where next time free starts.

Current status in this patch:
1)Two APIs are added for users to enable direct-rearm mode:
In control plane, users can call 'rte_eth_rx_queue_rearm_data_get' to get Rx sw_ring pointer and its rxq_info.
(This avoid Tx load Rx data directly);

In data plane, users can call 'rte_eth_dev_direct_rearm' to rearm Rx buffers and free Tx buffers at the same time. Specifically, in this API, there are two separated API for Rx and Tx.
For Tx, 'rte_eth_tx_fill_sw_ring' can fill a given sw_ring by Tx freed buffers. For Rx, 'rte_eth_rx_flush_descriptor' can flush its descriptors based on the rearm buffers.
Thus, this can separate Rx and Tx operation, and user can even re-arm RX queue not from the same driver's TX queue, but from different sources too.
-----------------------------------------------------------------------
control plane:
rte_eth_rx_queue_rearm_data_get(*rxq_rearm_data);
data plane:
loop {
rte_eth_dev_direct_rearm(*rxq_rearm_data){

			rte_eth_tx_fill_sw_ring{
     				for (i = 0; i <= 32; i++) {
       					sw_ring.mbuf[i] = tx.mbuf[i];
				}
    			}

			rte_eth_rx_flush_descriptor{
				for (i = 0; i <= 32; i++) {
					flush descs[i];
				}
			}
		}
		rte_eth_rx_burst;
		rte_eth_tx_burst;

}
-----------------------------------------------------------------------
2)The i40e driver is changed to do the direct re-arm of the receive
side.
3)The ixgbe driver is changed to do the direct re-arm of the receive
side.

Testing status:
(1) dpdk l3fwd test with multiple drivers:

    port 0: 82599 NIC   port 1: XL710 NIC

-------------------------------------------------------------
Without fast free With fast free
Thunderx2: +9.44% +7.14%
-------------------------------------------------------------

(2) dpdk l3fwd test with same driver:

    port 0 && 1: XL710 NIC

-------------------------------------------------------------
*Direct rearm with exposing rx_sw_ring:
Without fast free With fast free
Ampere altra: +14.98% +15.77%
n1sdp: +6.47% +0.52%
-------------------------------------------------------------

(3) VPP test with same driver:

    port 0 && 1: XL710 NIC

-------------------------------------------------------------
*Direct rearm with exposing rx_sw_ring:
Ampere altra: +4.59%
n1sdp: +5.4%
-------------------------------------------------------------

Reference:
[1] https://store.nvidia.com/en-us/networking/store/product/MCX623105AN-CDAT/NVIDIAMCX623105ANCDATConnectX6DxENAdapterCard100GbECryptoDisabled/
[2] https://www.intel.com/content/www/us/en/products/sku/192561/intel-ethernet-network-adapter-e810cqda1/specifications.html
[3] https://www.broadcom.com/products/ethernet-connectivity/network-adapters/100gb-nic-ocp/n1100g

V2:
1. Use data-plane API to enable direct-rearm (Konstantin, Honnappa)
2. Add 'txq_data_get' API to get txq info for Rx (Konstantin)
3. Use input parameter to enable direct rearm in l3fwd (Konstantin)
4. Add condition detection for direct rearm API (Morten, Andrew Rybchenko)

V3:
1. Seperate Rx and Tx operation with two APIs in direct-rearm (Konstantin)
2. Delete L3fwd change for direct rearm (Jerin)
3. enable direct rearm in ixgbe driver in Arm

Feifei Wang (3):
ethdev: enable direct rearm with separate API net/i40e: enable direct rearm with separate API net/ixgbe: enable direct rearm with separate API

 drivers/net/i40e/i40e_ethdev.c            |   1 +
 drivers/net/i40e/i40e_ethdev.h            |   2 +
 drivers/net/i40e/i40e_rxtx.c              |  19 +++
 drivers/net/i40e/i40e_rxtx.h              |   4 +
 drivers/net/i40e/i40e_rxtx_vec_common.h   |  54 +++++++
 drivers/net/i40e/i40e_rxtx_vec_neon.c     |  42 ++++++
 drivers/net/ixgbe/ixgbe_ethdev.c          |   1 +
 drivers/net/ixgbe/ixgbe_ethdev.h          |   3 +
 drivers/net/ixgbe/ixgbe_rxtx.c            |  19 +++
 drivers/net/ixgbe/ixgbe_rxtx.h            |   4 +
 drivers/net/ixgbe/ixgbe_rxtx_vec_common.h |  48 ++++++
 drivers/net/ixgbe/ixgbe_rxtx_vec_neon.c   |  52 +++++++
 lib/ethdev/ethdev_driver.h                |  10 ++
 lib/ethdev/ethdev_private.c               |   2 +
 lib/ethdev/rte_ethdev.c                   |  52 +++++++
 lib/ethdev/rte_ethdev.h                   | 174 ++++++++++++++++++++++
 lib/ethdev/rte_ethdev_core.h              |  11 ++
 lib/ethdev/version.map                    |   6 +
 18 files changed, 504 insertions(+)
Signature · 1 line
2.25.1
#44 lore raw

PATCH v3 1/3 ethdev: enable direct rearm with separate API

Add 'tx_fill_sw_ring' and 'rx_flush_descriptor' API into direct rearm mode for separate Rx and Tx Operation. And this can support different multiple sources in direct rearm mode. For examples, Rx driver is ixgbe, and Tx driver is i40e.

---
 lib/ethdev/ethdev_driver.h   |  10 ++
 lib/ethdev/ethdev_private.c  |   2 +
 lib/ethdev/rte_ethdev.c      |  52 +++++++++++
 lib/ethdev/rte_ethdev.h      | 174 +++++++++++++++++++++++++++++++++++
 lib/ethdev/rte_ethdev_core.h |  11 +++
 lib/ethdev/version.map       |   6 ++
 6 files changed, 255 insertions(+)
lib/ethdev/ethdev_driver.h +10 -0
index 6a550cfc83..bc539ec862 100644
@@ -59,6 +59,10 @@ struct rte_eth_dev {
 	eth_rx_descriptor_status_t rx_descriptor_status;
 	/** Check the status of a Tx descriptor */
 	eth_tx_descriptor_status_t tx_descriptor_status;
+	/** Fill Rx sw-ring with Tx buffers in direct rearm mode */
+	eth_tx_fill_sw_ring_t tx_fill_sw_ring;
+	/** Flush Rx descriptor in direct rearm mode */
+	eth_rx_flush_descriptor_t rx_flush_descriptor;
 
 	/**
 	 * Device data that is shared between primary and secondary processes
@@ -504,6 +508,10 @@ typedef void (*eth_rxq_info_get_t)(struct rte_eth_dev *dev,
 typedef void (*eth_txq_info_get_t)(struct rte_eth_dev *dev,
 	uint16_t tx_queue_id, struct rte_eth_txq_info *qinfo);
 
+/**< @internal Get rearm data for a receive queue of an Ethernet device. */
+typedef void (*eth_rxq_rearm_data_get_t)(struct rte_eth_dev *dev,
+	uint16_t tx_queue_id, struct rte_eth_rxq_rearm_data *rxq_rearm_data);
+
 typedef int (*eth_burst_mode_get_t)(struct rte_eth_dev *dev,
 	uint16_t queue_id, struct rte_eth_burst_mode *mode);
 
@@ -1215,6 +1223,8 @@ struct eth_dev_ops {
 	eth_rxq_info_get_t         rxq_info_get;
 	/** Retrieve Tx queue information */
 	eth_txq_info_get_t         txq_info_get;
+	/** Get Rx queue rearm data */
+	eth_rxq_rearm_data_get_t   rxq_rearm_data_get;
 	eth_burst_mode_get_t       rx_burst_mode_get; /**< Get Rx burst mode */
 	eth_burst_mode_get_t       tx_burst_mode_get; /**< Get Tx burst mode */
 	eth_fw_version_get_t       fw_version_get; /**< Get firmware version */
lib/ethdev/ethdev_private.c +2 -0
index 48090c879a..c5dd5e30f6 100644
@@ -276,6 +276,8 @@ eth_dev_fp_ops_setup(struct rte_eth_fp_ops *fpo,
 	fpo->rx_queue_count = dev->rx_queue_count;
 	fpo->rx_descriptor_status = dev->rx_descriptor_status;
 	fpo->tx_descriptor_status = dev->tx_descriptor_status;
+	fpo->tx_fill_sw_ring = dev->tx_fill_sw_ring;
+	fpo->rx_flush_descriptor = dev->rx_flush_descriptor;
 
 	fpo->rxq.data = dev->data->rx_queues;
 	fpo->rxq.clbk = (void **)(uintptr_t)dev->post_rx_burst_cbs;
lib/ethdev/rte_ethdev.c +52 -0

64 lines not on this page · raw message

lib/ethdev/rte_ethdev.h +174 -0

198 lines not on this page · raw message

lib/ethdev/rte_ethdev_core.h +11 -0
index dcf8adab92..5ecb57f6f0 100644
@@ -56,6 +56,13 @@ typedef int (*eth_rx_descriptor_status_t)(void *rxq, uint16_t offset);
 /** @internal Check the status of a Tx descriptor */
 typedef int (*eth_tx_descriptor_status_t)(void *txq, uint16_t offset);
 
+/** @internal Fill Rx sw-ring with Tx buffers in direct rearm mode */
+typedef int (*eth_tx_fill_sw_ring_t)(void *txq,
+		struct rte_eth_rxq_rearm_data *rxq_rearm_data);
+
+/** @internal Flush Rx descriptor in direct rearm mode */
+typedef int (*eth_rx_flush_descriptor_t)(void *rxq, uint16_t nb_rearm);
+
 /**
  * @internal
  * Structure used to hold opaque pointers to internal ethdev Rx/Tx
@@ -90,6 +97,8 @@ struct rte_eth_fp_ops {
 	eth_rx_queue_count_t rx_queue_count;
 	/** Check the status of a Rx descriptor. */
 	eth_rx_descriptor_status_t rx_descriptor_status;
+	/** Flush Rx descriptor in direct rearm mode */
+	eth_rx_flush_descriptor_t rx_flush_descriptor;
 	/** Rx queues data. */
 	struct rte_ethdev_qdata rxq;
 	uintptr_t reserved1[3];
@@ -106,6 +115,8 @@ struct rte_eth_fp_ops {
 	eth_tx_prep_t tx_pkt_prepare;
 	/** Check the status of a Tx descriptor. */
 	eth_tx_descriptor_status_t tx_descriptor_status;
+	/** Fill Rx sw-ring with Tx buffers in direct rearm mode */
+	eth_tx_fill_sw_ring_t tx_fill_sw_ring;
 	/** Tx queues data. */
 	struct rte_ethdev_qdata txq;
 	uintptr_t reserved2[3];
lib/ethdev/version.map +6 -0
index 17201fbe0f..f39f02a69b 100644
@@ -298,6 +298,12 @@ EXPERIMENTAL {
 	rte_flow_get_q_aged_flows;
 	rte_mtr_meter_policy_get;
 	rte_mtr_meter_profile_get;
+
+	# added in 23.03
+	rte_eth_dev_direct_rearm;
+	rte_eth_rx_flush_descriptor;
+	rte_eth_rx_queue_rearm_data_get;
+	rte_eth_tx_fill_sw_ring;
 };
 
 INTERNAL {
Signature · 1 line
2.25.1
#45 lore raw

RE: [PATCH v3 1/3] ethdev: enable direct rearm with separate API

Quoted 9 lines

From: Feifei Wang [mailto:[email protected]]
Sent: Wednesday, 4 January 2023 08.31

Add 'tx_fill_sw_ring' and 'rx_flush_descriptor' API into direct rearm mode for separate Rx and Tx Operation. And this can support different multiple sources in direct rearm mode. For examples, Rx driver is ixgbe,
and Tx driver is i40e.

Quoted 6 lines

---

This feature looks very promising for performance. I am pleased to see progress on it.

Please confirm that the fast path functions are still thread safe, i.e. one EAL thread may be calling rte_eth_rx_burst() while another EAL thread is calling rte_eth_tx_burst().

A few more comments below.

Quoted 13 lines
 lib/ethdev/ethdev_driver.h   |  10 ++
 lib/ethdev/ethdev_private.c  |   2 +
 lib/ethdev/rte_ethdev.c      |  52 +++++++++++
 lib/ethdev/rte_ethdev.h      | 174 +++++++++++++++++++++++++++++++++++
 lib/ethdev/rte_ethdev_core.h |  11 +++
 lib/ethdev/version.map       |   6 ++
 6 files changed, 255 insertions(+)
lib/ethdev/ethdev_driver.h +0 -0
index 6a550cfc83..bc539ec862 100644

lib/ethdev/ethdev_driver.h struct rte_eth_dev { · line 63

Quoted 5 lines
lib/ethdev/ethdev_driver.h +2 -0
@@ -59,6 +59,10 @@ struct rte_eth_dev {
 	eth_rx_descriptor_status_t rx_descriptor_status;
 	/** Check the status of a Tx descriptor */
 	eth_tx_descriptor_status_t tx_descriptor_status;
+	/** Fill Rx sw-ring with Tx buffers in direct rearm mode */
+	eth_tx_fill_sw_ring_t tx_fill_sw_ring;

What is "Rx sw-ring"? Please confirm that this is not an Intel PMD specific term and/or implementation detail, e.g. by providing a conceptual implementation for a non-Intel PMD, e.g. mlx5.

Please note: I do not request the ability to rearm between drivers from different vendors, I only request that the public ethdev API uses generic terms and concepts, so any NIC vendor can implement the direct-rearm functions in their PMDs.

lib/ethdev/ethdev_driver.h struct rte_eth_dev { · line 65

Quoted 2 lines
lib/ethdev/ethdev_driver.h +2 -0
+	/** Flush Rx descriptor in direct rearm mode */
+	eth_rx_flush_descriptor_t rx_flush_descriptor;

descriptor -> descriptors. There are more than one. Both in comment and function name.

[...]

Quoted 14 lines
lib/ethdev/rte_ethdev.h +4 -0
index c129ca1eaf..381c3d535f 100644
@@ -1818,6 +1818,17 @@ struct rte_eth_txq_info {
 	uint8_t queue_state;        /**< one of RTE_ETH_QUEUE_STATE_*. */
 } __rte_cache_min_aligned;
 
+/**
+ * @internal
+ * Structure used to hold pointers to internal ethdev Rx rearm data.
+ * The main purpose is to load Rx queue rearm data in direct rearm

mode.
+ */

lib/ethdev/rte_ethdev.h struct rte_eth_txq_info { · line 1830

Quoted 5 lines
lib/ethdev/rte_ethdev.h +5 -0
+struct rte_eth_rxq_rearm_data {
+	void *rx_sw_ring;
+	uint16_t *rearm_start;
+	uint16_t *rearm_nb;
+} __rte_cache_min_aligned;

Please add descriptions to the fields in this structure.

#46 lore raw

回复: [PATCH v3 1/3] ethdev: enable direct rearm with separate API

Hi, Morten

Quoted 26 lines

-----邮件原件-----
发件人: Morten Brørup <[email protected]>
发送时间: Wednesday, January 4, 2023 4:22 PM
收件人: Feifei Wang <[email protected]>; [email protected]; Ferruh Yigit <[email protected]>; Andrew Rybchenko <[email protected]>
抄送: [email protected]; [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; Ruifeng Wang <[email protected]>
主题: RE: [PATCH v3 1/3] ethdev: enable direct rearm with separate API

Quoted 8 lines

From: Feifei Wang [mailto:[email protected]]
Sent: Wednesday, 4 January 2023 08.31

Add 'tx_fill_sw_ring' and 'rx_flush_descriptor' API into direct rearm mode for separate Rx and Tx Operation. And this can support different multiple sources in direct rearm mode. For examples, Rx driver is ixgbe, and Tx driver is i40e.

Quoted 6 lines

---

Quoted 3 lines

This feature looks very promising for performance. I am pleased to see progress on it.

Thanks very much for your reviewing.

Quoted 4 lines

Please confirm that the fast path functions are still thread safe, i.e. one EAL thread may be calling rte_eth_rx_burst() while another EAL thread is calling rte_eth_tx_burst().

For the multiple threads safe, like we say in cover letter, current direct-rearm support Rx and Tx in the same thread. If we consider multiple threads like 'pipeline model', there need to add 'lock' in the data path which can decrease the performance.
Thus, the first step we do is try to enable direct-rearm in the single thread, and then we will consider to enable direct rearm in multiple threads and improve the performance.

Quoted 11 lines

A few more comments below.

Quoted 4 lines
 lib/ethdev/ethdev_driver.h   |  10 ++
 lib/ethdev/ethdev_private.c  |   2 +
 lib/ethdev/rte_ethdev.c      |  52 +++++++++++
 lib/ethdev/rte_ethdev.h      | 174

+++++++++++++++++++++++++++++++++++

Quoted 4 lines
 lib/ethdev/rte_ethdev_core.h |  11 +++
 lib/ethdev/version.map       |   6 ++
 6 files changed, 255 insertions(+)
Quoted 14 lines

lib/ethdev/ethdev_driver.h struct rte_eth_dev { · line 63

Quoted 10 lines
lib/ethdev/ethdev_driver.h +2 -0
index 6a550cfc83..bc539ec862 100644
@@ -59,6 +59,10 @@ struct rte_eth_dev {
 	eth_rx_descriptor_status_t rx_descriptor_status;
 	/** Check the status of a Tx descriptor */
 	eth_tx_descriptor_status_t tx_descriptor_status;
+	/** Fill Rx sw-ring with Tx buffers in direct rearm mode */
+	eth_tx_fill_sw_ring_t tx_fill_sw_ring;

What is "Rx sw-ring"? Please confirm that this is not an Intel PMD specific term and/or implementation detail, e.g. by providing a conceptual implementation for a non-Intel PMD, e.g. mlx5.

Rx sw_ring is used to store mbufs in intel PMD. This is the same as 'rxq->elts' in mlx5. Agree with that we need to providing a conceptual implementation for all PMDs.

Quoted 5 lines

Please note: I do not request the ability to rearm between drivers from different vendors, I only request that the public ethdev API uses generic terms and concepts, so any NIC vendor can implement the direct-rearm functions in their PMDs.

Agree with this. This is also our consideration.
In the future plan, we plan to implement this function in different PMDs.

Quoted 6 lines

lib/ethdev/ethdev_driver.h struct rte_eth_dev { · line 65

Quoted 3 lines
lib/ethdev/ethdev_driver.h +2 -0
+	/** Flush Rx descriptor in direct rearm mode */
+	eth_rx_flush_descriptor_t rx_flush_descriptor;

descriptor -> descriptors. There are more than one. Both in comment and function name.

Agree.

Quoted 11 lines

[...]

Quoted 8 lines
lib/ethdev/rte_ethdev.h index renamed from lib/ethdev/rte_ethdev.h +0 -0

c129ca1eaf..381c3d535f 100644

lib/ethdev/rte_ethdev.h +0 -0
@@ -1818,6 +1818,17 @@ struct rte_eth_txq_info {
 	uint8_t queue_state;        /**< one of RTE_ETH_QUEUE_STATE_*. */
 } __rte_cache_min_aligned;
 
Quoted 13 lines

lib/ethdev/rte_ethdev.h struct rte_eth_txq_info { · line 1830

Quoted 11 lines
lib/ethdev/rte_ethdev.h +4 -0
+/**
+ * @internal
+ * Structure used to hold pointers to internal ethdev Rx rearm data.
+ * The main purpose is to load Rx queue rearm data in direct rearm

mode.
+ */

lib/ethdev/rte_ethdev.h +5 -0
+struct rte_eth_rxq_rearm_data {
+	void *rx_sw_ring;
+	uint16_t *rearm_start;
+	uint16_t *rearm_nb;
+} __rte_cache_min_aligned;

Please add descriptions to the fields in this structure.

Agree.

#47 lore raw

RE: [PATCH v3 1/3] ethdev: enable direct rearm with separate API

Quoted 37 lines

From: Feifei Wang [mailto:[email protected]]
Sent: Wednesday, 4 January 2023 09.51

Hi, Morten

Quoted 7 lines

发件人: Morten Brørup <[email protected]>
发送时间: Wednesday, January 4, 2023 4:22 PM

Quoted 4 lines

From: Feifei Wang [mailto:[email protected]]
Sent: Wednesday, 4 January 2023 08.31

Add 'tx_fill_sw_ring' and 'rx_flush_descriptor' API into direct

rearm

Quoted 1 line
Quoted 1 line

mode for separate Rx and Tx Operation. And this can support

different

Quoted 11 lines
Quoted 9 lines

multiple sources in direct rearm mode. For examples, Rx driver is ixgbe, and Tx driver is i40e.

---

This feature looks very promising for performance. I am pleased to

see

Quoted 2 lines

progress on it.

Thanks very much for your reviewing.

Quoted 1 line

Please confirm that the fast path functions are still thread safe,

i.e. one EAL

Quoted 1 line

thread may be calling rte_eth_rx_burst() while another EAL thread is

calling

Quoted 2 lines

rte_eth_tx_burst().

Quoted 8 lines

For the multiple threads safe, like we say in cover letter, current direct-rearm support
Rx and Tx in the same thread. If we consider multiple threads like 'pipeline model', there
need to add 'lock' in the data path which can decrease the performance. Thus, the first step we do is try to enable direct-rearm in the single thread, and then we will consider
to enable direct rearm in multiple threads and improve the performance.

OK, doing it in steps is a good idea for a feature like this - makes it easier to understand and review.

When proceeding to add support for the "pipeline model", perhaps the lockless principles from the rte_ring can be used in this feature too.

From a high level perspective, I'm somewhat worried that releasing a "work-in-progress" version of this feature in some DPDK version will cause API/ABI breakage discussions when progressing to the next steps of the implementation to make the feature more complete. Not only support for thread safety across simultaneous RX and TX, but also support for multiple mbuf pools per RX queue [1]. Marking the functions experimental should alleviate such discussions, but there is a risk of pushback to not break the API/ABI anyway.

[1]: https://elixir.bootlin.com/dpdk/v22.11.1/source/lib/ethdev/rte_ethdev.h#L1105

[...]

Quoted 10 lines
Quoted 10 lines

lib/ethdev/ethdev_driver.h struct rte_eth_dev { · line 63

Quoted 8 lines
lib/ethdev/ethdev_driver.h +2 -0
@@ -59,6 +59,10 @@ struct rte_eth_dev {
 	eth_rx_descriptor_status_t rx_descriptor_status;
 	/** Check the status of a Tx descriptor */
 	eth_tx_descriptor_status_t tx_descriptor_status;
+	/** Fill Rx sw-ring with Tx buffers in direct rearm mode */
+	eth_tx_fill_sw_ring_t tx_fill_sw_ring;

What is "Rx sw-ring"? Please confirm that this is not an Intel PMD

Quoted 6 lines

specific

Quoted 2 lines

term and/or implementation detail, e.g. by providing a conceptual implementation for a non-Intel PMD, e.g. mlx5.

Rx sw_ring is used to store mbufs in intel PMD. This is the same as 'rxq->elts'
in mlx5.

Sounds good.

Then all we need is consensus on a generic name for this, unless "Rx sw-ring" already is the generic name. (I'm not a PMD developer, so I might be completely off track here.) Naming is often debatable, so I'll stop talking about it now - I only wanted to highlight that we should avoid vendor-specific terms in public APIs intended to be implemented by multiple vendors. On the other hand... if no other vendors raise their voices before merging into the DPDK main repository, they forfeit their right to complain about it. ;-)

Quoted 2 lines

Agree with that we need to providing a conceptual
implementation for all PMDs.

My main point is that we should ensure that the feature is not too tightly coupled with the way Intel PMDs implement mbuf handling. Providing a conceptual implementation for a non-Intel PMD is one way of checking this.

The actual implementation in other PMDs could be left up to the various NIC vendors.

#48 lore raw

回复: [PATCH v3 1/3] ethdev: enable direct rearm with separate API

Sorry for my delayed reply.

Quoted 71 lines

-----邮件原件-----
发件人: Morten Brørup <[email protected]>
发送时间: Wednesday, January 4, 2023 6:11 PM
收件人: Feifei Wang <[email protected]>; [email protected]; Ferruh Yigit <[email protected]>; Andrew Rybchenko <[email protected]>
抄送: [email protected]; [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; Ruifeng Wang <[email protected]>; nd <[email protected]>
主题: RE: [PATCH v3 1/3] ethdev: enable direct rearm with separate API

Quoted 37 lines

From: Feifei Wang [mailto:[email protected]]
Sent: Wednesday, 4 January 2023 09.51

Hi, Morten

Quoted 7 lines

发件人: Morten Brørup <[email protected]>
发送时间: Wednesday, January 4, 2023 4:22 PM

Quoted 4 lines

From: Feifei Wang [mailto:[email protected]]
Sent: Wednesday, 4 January 2023 08.31

Add 'tx_fill_sw_ring' and 'rx_flush_descriptor' API into direct

rearm

Quoted 1 line
Quoted 1 line

mode for separate Rx and Tx Operation. And this can support

different

Quoted 11 lines
Quoted 9 lines

multiple sources in direct rearm mode. For examples, Rx driver is ixgbe, and Tx driver is i40e.

---

This feature looks very promising for performance. I am pleased to

see

Quoted 2 lines

progress on it.

Thanks very much for your reviewing.

Quoted 1 line

Please confirm that the fast path functions are still thread safe,

i.e. one EAL

Quoted 1 line

thread may be calling rte_eth_rx_burst() while another EAL thread is

calling

Quoted 2 lines

rte_eth_tx_burst().

Quoted 7 lines

For the multiple threads safe, like we say in cover letter, current direct-rearm support Rx and Tx in the same thread. If we consider multiple threads like 'pipeline model', there need to add 'lock' in the data path which can decrease the performance.
Thus, the first step we do is try to enable direct-rearm in the single thread, and then we will consider to enable direct rearm in multiple threads and improve the performance.

OK, doing it in steps is a good idea for a feature like this - makes it easier to understand and review.

When proceeding to add support for the "pipeline model", perhaps the lockless principles from the rte_ring can be used in this feature too.

From a high level perspective, I'm somewhat worried that releasing a "work- in-progress" version of this feature in some DPDK version will cause API/ABI breakage discussions when progressing to the next steps of the implementation to make the feature more complete. Not only support for thread safety across simultaneous RX and TX, but also support for multiple mbuf pools per RX queue [1]. Marking the functions experimental should alleviate such discussions, but there is a risk of pushback to not break the API/ABI anyway.

Quoted 4 lines

[1]:
https://elixir.bootlin.com/dpdk/v22.11.1/source/lib/ethdev/rte_ethdev.h#L1
105

[Feifei] I think the subsequent upgrade does not significantly damage the stability of the API we currently define.

For thread safety across simultaneous RX and TX, in the future, the lockless operation change will happen in the pmd layer, such as CAS load/store for rxq queue index of pmd. Thus, this can not affect the stability of the upper API.

For multiple mbuf pools per RX queue, direct-rearm just put Tx buffers into Rx buffers, and it do not care which mempool the buffer coming from.
From different mempool buffers eventually freed into their respective sources in the no FAST_FREE path.
I think this is a mistake in cover letter. Previous direct-rearm can just support FAST_FREE so it constraint that buffer should be from the same mempool. Now, the latest version can support no_FAST_FREE path, but we forget to make change in cover letter.

Quoted 32 lines

[...]

Quoted 10 lines
Quoted 10 lines

lib/ethdev/ethdev_driver.h struct rte_eth_dev { · line 63

Quoted 8 lines
lib/ethdev/ethdev_driver.h +2 -0
@@ -59,6 +59,10 @@ struct rte_eth_dev {
 	eth_rx_descriptor_status_t rx_descriptor_status;
 	/** Check the status of a Tx descriptor */
 	eth_tx_descriptor_status_t tx_descriptor_status;
+	/** Fill Rx sw-ring with Tx buffers in direct rearm mode */
+	eth_tx_fill_sw_ring_t tx_fill_sw_ring;

What is "Rx sw-ring"? Please confirm that this is not an Intel PMD

Quoted 6 lines

specific

Quoted 2 lines

term and/or implementation detail, e.g. by providing a conceptual implementation for a non-Intel PMD, e.g. mlx5.

Rx sw_ring is used to store mbufs in intel PMD. This is the same as 'rxq->elts'
in mlx5.

Sounds good.

Then all we need is consensus on a generic name for this, unless "Rx sw-ring" already is the generic name. (I'm not a PMD developer, so I might be completely off track here.) Naming is often debatable, so I'll stop talking about it now - I only wanted to highlight that we should avoid vendor- specific terms in public APIs intended to be implemented by multiple vendors. On the other hand... if no other vendors raise their voices before merging into the DPDK main repository, they forfeit their right to complain about it. ;-)

Quoted 2 lines

Agree with that we need to providing a conceptual implementation for all PMDs.

Quoted 6 lines

My main point is that we should ensure that the feature is not too tightly coupled with the way Intel PMDs implement mbuf handling. Providing a conceptual implementation for a non-Intel PMD is one way of checking this.

The actual implementation in other PMDs could be left up to the various NIC vendors.

Yes. And we will rename our API to make it suitable for all vendors:
rte_eth_direct_rearm -> rte_eth_buf_cycle (upper API for direct rearm)
rte_eth_tx_fill_sw_ring -> rte_eth_tx_buf_stash (Tx queue fill Rx ring buffer )
rte_eth_rx_flush_descriptor -> rte_eth_rx_descriptors_refill (Rx queue flush its descriptors)

rte_eth_rxq_rearm_data {
	void *rx_sw_ring;
	uint16_t *rearm_start;
	uint16_t *rearm_nb;
}

->

struct *rxq_recycle_info {
	rte_mbuf **buf_ring;
	uint16_t *offset = (uint16 *)(&rq-<ci);
	uint16_t *end;
	uint16_t ring_size; 

}

#49 lore raw

Re: [PATCH v3 1/3] ethdev: enable direct rearm with separate API

In reply to Morten Brørup
Quoted 9 lines · Morten Brørup
Quoted 9 lines

From: Feifei Wang [mailto:[email protected]]
Sent: Wednesday, 4 January 2023 08.31

Add 'tx_fill_sw_ring' and 'rx_flush_descriptor' API into direct rearm mode for separate Rx and Tx Operation. And this can support different multiple sources in direct rearm mode. For examples, Rx driver is ixgbe,
and Tx driver is i40e.

Quoted 9 lines
Quoted 6 lines

---

This feature looks very promising for performance. I am pleased to see progress on it.

Hi Morten,

Yes it brings some performance, but not to generic use case, only to specific and constraint use case.

And changes are relatively invasive comparing the usecase it supports, like it adds new two inline datapath functions and a new dev_ops.

I am worried the unnecessary complexity and possible regressions in the fundamental and simple parts of the project, with a good intention to gain a few percentage performance in a specific usecase, can hurt the project.

I can see this is compared to MBUF_FAST_FREE feature, but MBUF_FAST_FREE is just an offload benefiting from existing offload infrastructure, which requires very small update and logically change in application and simple to implement in the drivers. So, they are not same from complexity perspective.

Briefly, I am not comfortable with this change, I would like to see an explicit approval and code review from techboard to proceed.

Quoted 56 lines

Please confirm that the fast path functions are still thread safe, i.e. one EAL thread may be calling rte_eth_rx_burst() while another EAL thread is calling rte_eth_tx_burst().

A few more comments below.

Quoted 13 lines
 lib/ethdev/ethdev_driver.h   |  10 ++
 lib/ethdev/ethdev_private.c  |   2 +
 lib/ethdev/rte_ethdev.c      |  52 +++++++++++
 lib/ethdev/rte_ethdev.h      | 174 +++++++++++++++++++++++++++++++++++
 lib/ethdev/rte_ethdev_core.h |  11 +++
 lib/ethdev/version.map       |   6 ++
 6 files changed, 255 insertions(+)
lib/ethdev/ethdev_driver.h +0 -0
index 6a550cfc83..bc539ec862 100644

lib/ethdev/ethdev_driver.h struct rte_eth_dev { · line 63

Quoted 5 lines
lib/ethdev/ethdev_driver.h +2 -0
@@ -59,6 +59,10 @@ struct rte_eth_dev {
 	eth_rx_descriptor_status_t rx_descriptor_status;
 	/** Check the status of a Tx descriptor */
 	eth_tx_descriptor_status_t tx_descriptor_status;
+	/** Fill Rx sw-ring with Tx buffers in direct rearm mode */
+	eth_tx_fill_sw_ring_t tx_fill_sw_ring;

What is "Rx sw-ring"? Please confirm that this is not an Intel PMD specific term and/or implementation detail, e.g. by providing a conceptual implementation for a non-Intel PMD, e.g. mlx5.

Please note: I do not request the ability to rearm between drivers from different vendors, I only request that the public ethdev API uses generic terms and concepts, so any NIC vendor can implement the direct-rearm functions in their PMDs.

lib/ethdev/ethdev_driver.h struct rte_eth_dev { · line 65

Quoted 2 lines
lib/ethdev/ethdev_driver.h +2 -0
+	/** Flush Rx descriptor in direct rearm mode */
+	eth_rx_flush_descriptor_t rx_flush_descriptor;

descriptor -> descriptors. There are more than one. Both in comment and function name.

[...]

Quoted 14 lines
lib/ethdev/rte_ethdev.h +4 -0
index c129ca1eaf..381c3d535f 100644
@@ -1818,6 +1818,17 @@ struct rte_eth_txq_info {
 	uint8_t queue_state;        /**< one of RTE_ETH_QUEUE_STATE_*. */
 } __rte_cache_min_aligned;
 
+/**
+ * @internal
+ * Structure used to hold pointers to internal ethdev Rx rearm data.
+ * The main purpose is to load Rx queue rearm data in direct rearm

mode.
+ */

lib/ethdev/rte_ethdev.h struct rte_eth_txq_info { · line 1830

Quoted 5 lines
lib/ethdev/rte_ethdev.h +5 -0
+struct rte_eth_rxq_rearm_data {
+	void *rx_sw_ring;
+	uint16_t *rearm_start;
+	uint16_t *rearm_nb;
+} __rte_cache_min_aligned;

Please add descriptions to the fields in this structure.

#50 lore raw

RE: [PATCH v3 1/3] ethdev: enable direct rearm with separate API

Quoted 20 lines

From: Ferruh Yigit [mailto:[email protected]]
Sent: Monday, 6 March 2023 13.49

Quoted 16 lines · Morten Brørup
Quoted 15 lines

From: Feifei Wang [mailto:[email protected]]
Sent: Wednesday, 4 January 2023 08.31

Add 'tx_fill_sw_ring' and 'rx_flush_descriptor' API into direct rearm mode for separate Rx and Tx Operation. And this can support different multiple sources in direct rearm mode. For examples, Rx driver is ixgbe,
and Tx driver is i40e.

---

Quoted 8 lines
Quoted 1 line

This feature looks very promising for performance. I am pleased to see

progress on it.

Quoted 1 line

Hi Morten,

Yes it brings some performance, but not to generic use case, only to specific and constraint use case.

I got the impression that the supported use case is a prominent and important use case.

This is the primary argument for considering such a complex non-generic feature.

Quoted 10 lines

And changes are relatively invasive comparing the usecase it supports, like it adds new two inline datapath functions and a new dev_ops.

I am worried the unnecessary complexity and possible regressions in the fundamental and simple parts of the project, with a good intention to gain a few percentage performance in a specific usecase, can hurt the project.

Quoted 8 lines

I can see this is compared to MBUF_FAST_FREE feature, but MBUF_FAST_FREE is just an offload benefiting from existing offload infrastructure, which requires very small update and logically change in application and simple to implement in the drivers. So, they are not same from complexity perspective.

Briefly, I am not comfortable with this change, I would like to see an explicit approval and code review from techboard to proceed.

I agree that the complexity is very high, and thus requires extra consideration. Your suggested techboard review and approval process seems like a good solution.

And the performance benefit of direct rearm should be compared to the performance using the new zero-copy mempool API.

-Morten

#51 lore raw

回复: [PATCH v3 1/3] ethdev: enable direct rearm with separate API

Hi, Morten, Ferruh

Thanks very much for your reviewing.

Whatever the worries about direct rearm or comments to improve direct rearm, these urge us to learn more and think more. I think it is beneficial and a good achievement for this exploration.

I will update the latest version for techboard code review. During this time, I need some time to do performance test for the latest version.
So it should not catch up with this week's meeting and will be done before the techboard meeting in two weeks.

Best Regards
Feifei

Quoted 73 lines

-----邮件原件-----
发件人: Morten Brørup <[email protected]>
发送时间: Monday, March 6, 2023 9:26 PM
收件人: Ferruh Yigit <[email protected]>; Feifei Wang <[email protected]>; [email protected]; Andrew Rybchenko <[email protected]>; [email protected]
抄送: [email protected]; [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; Ruifeng Wang <[email protected]>
主题: RE: [PATCH v3 1/3] ethdev: enable direct rearm with separate API

Quoted 21 lines

From: Ferruh Yigit [mailto:[email protected]]
Sent: Monday, 6 March 2023 13.49

Quoted 15 lines · Morten Brørup
Quoted 8 lines

From: Feifei Wang [mailto:[email protected]]
Sent: Wednesday, 4 January 2023 08.31

Add 'tx_fill_sw_ring' and 'rx_flush_descriptor' API into direct rearm mode for separate Rx and Tx Operation. And this can support different multiple sources in direct rearm mode. For examples, Rx driver is ixgbe, and Tx driver is i40e.

Quoted 6 lines

---

Quoted 2 lines

This feature looks very promising for performance. I am pleased to see

Quoted 7 lines

progress on it.

Quoted 1 line

Hi Morten,

Yes it brings some performance, but not to generic use case, only to specific and constraint use case.

I got the impression that the supported use case is a prominent and important use case.

This is the primary argument for considering such a complex non-generic feature.

Quoted 10 lines

And changes are relatively invasive comparing the usecase it supports, like it adds new two inline datapath functions and a new dev_ops.

I am worried the unnecessary complexity and possible regressions in the fundamental and simple parts of the project, with a good intention to gain a few percentage performance in a specific usecase, can hurt the project.

Quoted 8 lines

I can see this is compared to MBUF_FAST_FREE feature, but MBUF_FAST_FREE is just an offload benefiting from existing offload infrastructure, which requires very small update and logically change in application and simple to implement in the drivers. So, they are not same from complexity perspective.

Briefly, I am not comfortable with this change, I would like to see an explicit approval and code review from techboard to proceed.

I agree that the complexity is very high, and thus requires extra consideration. Your suggested techboard review and approval process seems like a good solution.

And the performance benefit of direct rearm should be compared to the performance using the new zero-copy mempool API.

-Morten

#52 lore raw

Re: [PATCH v3 1/3] ethdev: enable direct rearm with separate API

In reply to Morten Brørup
Quoted 24 lines · Morten Brørup
Quoted 24 lines

From: Ferruh Yigit [mailto:[email protected]]
Sent: Monday, 6 March 2023 13.49

Quoted 9 lines · Morten Brørup
Quoted 9 lines

From: Feifei Wang [mailto:[email protected]]
Sent: Wednesday, 4 January 2023 08.31

Add 'tx_fill_sw_ring' and 'rx_flush_descriptor' API into direct rearm mode for separate Rx and Tx Operation. And this can support different multiple sources in direct rearm mode. For examples, Rx driver is ixgbe,
and Tx driver is i40e.

Quoted 8 lines
Quoted 6 lines

---

This feature looks very promising for performance. I am pleased to see

progress on it.

Quoted 1 line
Quoted 7 lines
Quoted 4 lines

Hi Morten,

Yes it brings some performance, but not to generic use case, only to specific and constraint use case.

I got the impression that the supported use case is a prominent and important use case.

Can you please give real life samples for this use case, other than just showing better performance number in the test bench? This helps to understand the reasoning better.

Quoted 27 lines

This is the primary argument for considering such a complex non-generic feature.

Quoted 10 lines

And changes are relatively invasive comparing the usecase it supports, like it adds new two inline datapath functions and a new dev_ops.

I am worried the unnecessary complexity and possible regressions in the fundamental and simple parts of the project, with a good intention to gain a few percentage performance in a specific usecase, can hurt the project.

Quoted 8 lines

I can see this is compared to MBUF_FAST_FREE feature, but MBUF_FAST_FREE is just an offload benefiting from existing offload infrastructure, which requires very small update and logically change in application and simple to implement in the drivers. So, they are not same from complexity perspective.

Briefly, I am not comfortable with this change, I would like to see an explicit approval and code review from techboard to proceed.

I agree that the complexity is very high, and thus requires extra consideration. Your suggested techboard review and approval process seems like a good solution.

And the performance benefit of direct rearm should be compared to the performance using the new zero-copy mempool API.

-Morten

#53 lore raw

RE: [PATCH v3 1/3] ethdev: enable direct rearm with separate API

<snip>

Quoted 31 lines · Morten Brørup
Quoted 29 lines
Quoted 28 lines

From: Ferruh Yigit [mailto:[email protected]]
Sent: Monday, 6 March 2023 13.49

Quoted 15 lines · Morten Brørup
Quoted 8 lines

From: Feifei Wang [mailto:[email protected]]
Sent: Wednesday, 4 January 2023 08.31

Add 'tx_fill_sw_ring' and 'rx_flush_descriptor' API into direct rearm mode for separate Rx and Tx Operation. And this can support different multiple sources in direct rearm mode. For examples, Rx driver is ixgbe, and Tx driver is i40e.

Quoted 6 lines

---

Quoted 2 lines

This feature looks very promising for performance. I am pleased to see

progress on it.

Quoted 1 line

Hi Morten,

Yes it brings some performance, but not to generic use case, only to specific and constraint use case.

Quoted 7 lines
Quoted 1 line

I got the impression that the supported use case is a prominent and important

use case.

Quoted 1 line

Can you please give real life samples for this use case, other than just showing better performance number in the test bench? This helps to understand the reasoning better.

The very first patch started off with a constrained but prominent use case. Though, DPU based PCIe cards running DPDK applications with 1 or max 2 ports being used in tons of data centers is not a secret anymore and not a small use case that can be ignored.
However, the design of the patch has changed significantly from then. Now the solution can be applied to any generic use case that uses run-to-completion model of DPDK. i.e. the mapping of the RX and TX ports can be done dynamically in the data plane threads. There is no need of static configuration from control plane.

On the test bench, we need to make up our mind. When we see improvements, we say it is just a test bench. On other occasions when the test bench does not show any improvements (but improvements are shown by other metrics), we say the test bench does not show any improvements.

Quoted 3 lines
Quoted 2 lines

This is the primary argument for considering such a complex non-generic

feature.

I am not sure what is the complexity here, can you please elaborate?
I see other patches/designs (ex: proactive error recovery) which are way more complex to understand and comprehend.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

And changes are relatively invasive comparing the usecase it supports, like it adds new two inline datapath functions and a new dev_ops.

I am worried the unnecessary complexity and possible regressions in the fundamental and simple parts of the project, with a good intention to gain a few percentage performance in a specific usecase, can hurt the project.

I agree that we are touching some fundamental parts of the project. But, we also need to realize that those fundamental parts were not developed on architectures that have joined the project way later. Similarly, the use cases have evolved significantly from the original intended use cases. We cannot hold on to those fundamental designs if they affect the performance on other architectures while addressing prominent new use cases. Please note that this patch does not break any existing features or affect their performance in any negative way. The generic and originally intended use cases can benefit from this feature.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

I can see this is compared to MBUF_FAST_FREE feature, but MBUF_FAST_FREE is just an offload benefiting from existing offload infrastructure, which requires very small update and logically change in application and simple to implement in the drivers. So, they are not same from complexity perspective.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Briefly, I am not comfortable with this change, I would like to see an explicit approval and code review from techboard to proceed.

I agree that the complexity is very high, and thus requires extra consideration.

Your suggested techboard review and approval process seems like a good solution.

We can add to the agenda for the next Techboard meeting.

Quoted 6 lines
Quoted 2 lines

And the performance benefit of direct rearm should be compared to the

performance using the new zero-copy mempool API.

Quoted 3 lines

-Morten

#54 lore raw

RE: [PATCH v3 1/3] ethdev: enable direct rearm with separate API

Quoted 36 lines
Quoted 36 lines · Morten Brørup
Quoted 24 lines
Quoted 24 lines

From: Ferruh Yigit [mailto:[email protected]]
Sent: Monday, 6 March 2023 13.49

Quoted 15 lines · Morten Brørup
Quoted 8 lines

From: Feifei Wang [mailto:[email protected]]
Sent: Wednesday, 4 January 2023 08.31

Add 'tx_fill_sw_ring' and 'rx_flush_descriptor' API into direct rearm mode for separate Rx and Tx Operation. And this can support different multiple sources in direct rearm mode. For examples, Rx driver is ixgbe, and Tx driver is i40e.

Quoted 6 lines

---

Quoted 2 lines

This feature looks very promising for performance. I am pleased to see

progress on it.

Quoted 1 line
Quoted 6 lines
Quoted 4 lines

Hi Morten,

Yes it brings some performance, but not to generic use case, only to specific and constraint use case.

I got the impression that the supported use case is a prominent and important

use case.

Quoted 1 line
Quoted 8 lines
Quoted 3 lines

Can you please give real life samples for this use case, other than just showing better performance number in the test bench? This helps to understand the reasoning better.

The very first patch started off with a constrained but prominent use case. Though, DPU based PCIe cards running DPDK applications with 1 or max 2 ports being used in tons of data centers is not a secret anymore and not a small use case that can be ignored. However, the design of the patch has changed significantly from then. Now the solution can be applied to any generic use case that uses run-to-completion model of DPDK. i.e. the mapping of the RX and TX ports can be done dynamically in the data plane threads. There is no need of static configuration from control plane.

+1 to this statement.
I think the authors did a good job to make it generic enough, so it can be used for many different cases, plus AFAIU it doesn't introduce new implicit restrictions to the user.
Again, this feature is totally optional, so users are free to ignore it.
Personally I do not see any good reason why we shouldn't accept this feature into DPDK. Off-course with more code reviews, extra testing, docs updates, etc..

Quoted 49 lines

On the test bench, we need to make up our mind. When we see improvements, we say it is just a test bench. On other occasions when the test bench does not show any improvements (but improvements are shown by other metrics), we say the test bench does not show any improvements.

Quoted 3 lines
Quoted 2 lines

This is the primary argument for considering such a complex non-generic

feature.

I am not sure what is the complexity here, can you please elaborate?
I see other patches/designs (ex: proactive error recovery) which are way more complex to understand and comprehend.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

And changes are relatively invasive comparing the usecase it supports, like it adds new two inline datapath functions and a new dev_ops.

I am worried the unnecessary complexity and possible regressions in the fundamental and simple parts of the project, with a good intention to gain a few percentage performance in a specific usecase, can hurt the project.

I agree that we are touching some fundamental parts of the project. But, we also need to realize that those fundamental parts were not developed on architectures that have joined the project way later. Similarly, the use cases have evolved significantly from the original intended use cases. We cannot hold on to those fundamental designs if they affect the performance on other architectures while addressing prominent new use cases.
Please note that this patch does not break any existing features or affect their performance in any negative way. The generic and originally intended use cases can benefit from this feature.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

I can see this is compared to MBUF_FAST_FREE feature, but MBUF_FAST_FREE is just an offload benefiting from existing offload infrastructure, which requires very small update and logically change in application and simple to implement in the drivers. So, they are not same from complexity perspective.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Briefly, I am not comfortable with this change, I would like to see an explicit approval and code review from techboard to proceed.

I agree that the complexity is very high, and thus requires extra consideration.

Your suggested techboard review and approval process seems like a good solution.

We can add to the agenda for the next Techboard meeting.

Quoted 6 lines
Quoted 2 lines

And the performance benefit of direct rearm should be compared to the

performance using the new zero-copy mempool API.

Quoted 3 lines

-Morten

#55 lore raw

Re: [PATCH v3 1/3] ethdev: enable direct rearm with separate API

In reply to Honnappa Nagarahalli
Quoted 43 lines · Honnappa Nagarahalli

<snip>

Quoted 31 lines · Morten Brørup
Quoted 29 lines
Quoted 28 lines

From: Ferruh Yigit [mailto:[email protected]]
Sent: Monday, 6 March 2023 13.49

Quoted 15 lines · Morten Brørup
Quoted 8 lines

From: Feifei Wang [mailto:[email protected]]
Sent: Wednesday, 4 January 2023 08.31

Add 'tx_fill_sw_ring' and 'rx_flush_descriptor' API into direct rearm mode for separate Rx and Tx Operation. And this can support different multiple sources in direct rearm mode. For examples, Rx driver is ixgbe, and Tx driver is i40e.

Quoted 6 lines

---

Quoted 2 lines

This feature looks very promising for performance. I am pleased to see

progress on it.

Quoted 1 line

Hi Morten,

Yes it brings some performance, but not to generic use case, only to specific and constraint use case.

Quoted 7 lines
Quoted 1 line

I got the impression that the supported use case is a prominent and important

use case.

Quoted 1 line

Can you please give real life samples for this use case, other than just showing better performance number in the test bench? This helps to understand the reasoning better.

The very first patch started off with a constrained but prominent use case. Though, DPU based PCIe cards running DPDK applications with 1 or max 2 ports being used in tons of data centers is not a secret anymore and not a small use case that can be ignored.
However, the design of the patch has changed significantly from then. Now the solution can be applied to any generic use case that uses run-to-completion model of DPDK. i.e. the mapping of the RX and TX ports can be done dynamically in the data plane threads. There is no need of static configuration from control plane.

Quoted 6 lines

On the test bench, we need to make up our mind. When we see improvements, we say it is just a test bench. On other occasions when the test bench does not show any improvements (but improvements are shown by other metrics), we say the test bench does not show any improvements.

Quoted 3 lines
Quoted 2 lines

This is the primary argument for considering such a complex non-generic

feature.

I am not sure what is the complexity here, can you please elaborate?

I am considering from user perspective.

OK, DPDK is already low level, but ethdev has only a handful of datapath APIs (6 of them), and main ones are easy to comprehend:
rte_eth_rx_burst(port_id, queue_id, rx_pkts, nb_pkts); rte_eth_tx_burst(port_id, queue_id, tx_pkts, nb_pkts);

They (magically) Rx/Tx buffers, easy to grasp.

Maybe rte_eth_tx_prepare() is a little less obvious (why/when to use it), but still I believe simple.

Whoever looks to these APIs can figure out how to use in the application.

The other three is related to the descriptors and I am not sure about their use-case, I assume they are mostly good for debugging.

But now we are adding new datapath APIs:
rte_eth_tx_fill_sw_ring(port_id, queue_id, rxq_rearm_data); rte_eth_rx_flush_descriptor(port_id, queue_id, nb_rearm);

When you talk about SW ring and re-arming descriptors I believe you will loose most of the users already, driver developers will know what it is, you will know what that is, but people who are not close to the Ethernet HW won't.

And these APIs will be very visible, not like one of many control plane dev_ops. So this can confuse users who are not familiar with details.

Usage of these APIs comes with restrictions, it is possible that at some percentage of users will miss these restrictions or miss-understand them and will have issues.

Or many may be intimidated by them and stay away from using these APIs, leaving them as a burden to maintain, to test, to fix. That is why I think a real life usecase is needed, in that case at least we will know some consumers will fix or let us know when they get broken.

It may be possible to hide details under driver and user only set an offload flag, similar to FAST_FREE, but in that case feature will loose flexibility and it will be even more specific, perhaps making it less useful.

Quoted 36 lines

I see other patches/designs (ex: proactive error recovery) which are way more complex to understand and comprehend.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

And changes are relatively invasive comparing the usecase it supports, like it adds new two inline datapath functions and a new dev_ops.

I am worried the unnecessary complexity and possible regressions in the fundamental and simple parts of the project, with a good intention to gain a few percentage performance in a specific usecase, can hurt the project.

I agree that we are touching some fundamental parts of the project. But, we also need to realize that those fundamental parts were not developed on architectures that have joined the project way later. Similarly, the use cases have evolved significantly from the original intended use cases. We cannot hold on to those fundamental designs if they affect the performance on other architectures while addressing prominent new use cases. Please note that this patch does not break any existing features or affect their performance in any negative way. The generic and originally intended use cases can benefit from this feature.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

I can see this is compared to MBUF_FAST_FREE feature, but MBUF_FAST_FREE is just an offload benefiting from existing offload infrastructure, which requires very small update and logically change in application and simple to implement in the drivers. So, they are not same from complexity perspective.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Briefly, I am not comfortable with this change, I would like to see an explicit approval and code review from techboard to proceed.

I agree that the complexity is very high, and thus requires extra consideration.

Your suggested techboard review and approval process seems like a good solution.

We can add to the agenda for the next Techboard meeting.

Quoted 6 lines
Quoted 2 lines

And the performance benefit of direct rearm should be compared to the

performance using the new zero-copy mempool API.

Quoted 3 lines

-Morten

#56 lore raw

RE: [PATCH v3 1/3] ethdev: enable direct rearm with separate API

Quoted 73 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Tuesday, March 7, 2023 2:41 PM
To: Honnappa Nagarahalli <[email protected]>; Morten Brørup <[email protected]>; Feifei Wang <[email protected]>; [email protected]; Andrew Rybchenko <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; nd <[email protected]>; Ruifeng Wang <[email protected]>
Subject: Re: [PATCH v3 1/3] ethdev: enable direct rearm with separate API

Quoted 11 lines · Honnappa Nagarahalli

<snip>

Quoted 9 lines · Morten Brørup
Quoted 7 lines
Quoted 7 lines

From: Ferruh Yigit [mailto:[email protected]]
Sent: Monday, 6 March 2023 13.49

Quoted 3 lines · Morten Brørup
Quoted 3 lines

From: Feifei Wang [mailto:[email protected]]
Sent: Wednesday, 4 January 2023 08.31

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Add 'tx_fill_sw_ring' and 'rx_flush_descriptor' API into direct rearm mode for separate Rx and Tx Operation. And this can support different multiple sources in direct rearm mode. For examples, Rx driver is ixgbe, and Tx driver is i40e.

Suggested-by Honnappa Nagarahalli

<[email protected]>

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Reviewed-by Honnappa Nagarahalli

<[email protected]>

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 11 lines
Quoted 4 lines
Quoted 1 line

---

This feature looks very promising for performance. I am pleased to see

progress on it.

Quoted 1 line

Hi Morten,

Yes it brings some performance, but not to generic use case, only to specific and constraint use case.

I got the impression that the supported use case is a prominent and important

Quoted 7 lines
Quoted 6 lines

use case.

Quoted 1 line

Can you please give real life samples for this use case, other than just showing better performance number in the test bench? This helps to understand the reasoning better.

The very first patch started off with a constrained but prominent use case.

Though, DPU based PCIe cards running DPDK applications with 1 or max 2 ports being used in tons of data centers is not a secret anymore and not a small use case that can be ignored.

Quoted 1 line

However, the design of the patch has changed significantly from then. Now

the solution can be applied to any generic use case that uses run-to-completion model of DPDK. i.e. the mapping of the RX and TX ports can be done dynamically in the data plane threads. There is no need of static configuration from control plane.

Quoted 2 lines

On the test bench, we need to make up our mind. When we see

improvements, we say it is just a test bench. On other occasions when the test bench does not show any improvements (but improvements are shown by other metrics), we say the test bench does not show any improvements.

Quoted 2 lines
Quoted 2 lines
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines

This is the primary argument for considering such a complex non-generic

feature.

I am not sure what is the complexity here, can you please elaborate?

I am considering from user perspective.

Thanks for clarifying Ferruh.

Quoted 7 lines

OK, DPDK is already low level, but ethdev has only a handful of datapath APIs (6 of them), and main ones are easy to comprehend:
rte_eth_rx_burst(port_id, queue_id, rx_pkts, nb_pkts); rte_eth_tx_burst(port_id, queue_id, tx_pkts, nb_pkts);

They (magically) Rx/Tx buffers, easy to grasp.

I think the pktmbuf pool part is missed here. The user needs to create a pktmbuf pool by calling rte_pktmbuf_pool_create and has to pass the cache_size parameter. This requires the user to understand what is a cache, why it is required and how it affects the performance.
There are further complexities associated with pktmbuf pool - creating a pool with external pinned memory, creating a pool with ops name etc. So, practically, the user needs to be aware of more details than just the RX and TX functions.

Quoted 10 lines

Maybe rte_eth_tx_prepare() is a little less obvious (why/when to use it), but still I believe simple.

Whoever looks to these APIs can figure out how to use in the application.

The other three is related to the descriptors and I am not sure about their use- case, I assume they are mostly good for debugging.

Quoted 7 lines

But now we are adding new datapath APIs:
rte_eth_tx_fill_sw_ring(port_id, queue_id, rxq_rearm_data); rte_eth_rx_flush_descriptor(port_id, queue_id, nb_rearm);

When you talk about SW ring and re-arming descriptors I believe you will loose most of the users already, driver developers will know what it is, you will know what that is, but people who are not close to the Ethernet HW won't.

Agree, the names could be better. I personally do not want to separate out these two APIs as I do not think a use case (receiving and transmitting pkts across NICs of different types) exists to keep them separate. But, we did this based on feedback and to maintain a cleaner separation between RX and TX path. We will try to propose new names for these.

Quoted 7 lines

And these APIs will be very visible, not like one of many control plane dev_ops. So this can confuse users who are not familiar with details.

Usage of these APIs comes with restrictions, it is possible that at some percentage of users will miss these restrictions or miss-understand them and will have issues.

Agreed, there are several features already with restrictions.

Quoted 9 lines

Or many may be intimidated by them and stay away from using these APIs, leaving them as a burden to maintain, to test, to fix. That is why I think a real life usecase is needed, in that case at least we will know some consumers will fix or let us know when they get broken.

It may be possible to hide details under driver and user only set an offload flag, similar to FAST_FREE, but in that case feature will loose flexibility and it will be even more specific, perhaps making it less useful.

Agree.

Quoted 49 lines
Quoted 3 lines

I see other patches/designs (ex: proactive error recovery) which are way more

complex to understand and comprehend.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

And changes are relatively invasive comparing the usecase it supports, like it adds new two inline datapath functions and a new

dev_ops.

Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

I am worried the unnecessary complexity and possible regressions in the fundamental and simple parts of the project, with a good intention to gain a few percentage performance in a specific usecase, can hurt the project.

I agree that we are touching some fundamental parts of the project. But, we

also need to realize that those fundamental parts were not developed on architectures that have joined the project way later. Similarly, the use cases have evolved significantly from the original intended use cases. We cannot hold on to those fundamental designs if they affect the performance on other architectures while addressing prominent new use cases.

Quoted 1 line

Please note that this patch does not break any existing features or affect their

performance in any negative way. The generic and originally intended use cases can benefit from this feature.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

I can see this is compared to MBUF_FAST_FREE feature, but MBUF_FAST_FREE is just an offload benefiting from existing offload infrastructure, which requires very small update and logically change in application and simple to implement in the drivers. So, they are not same from complexity perspective.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

Briefly, I am not comfortable with this change, I would like to see an explicit approval and code review from techboard to proceed.

I agree that the complexity is very high, and thus requires extra

consideration.

Quoted 11 lines
Quoted 2 lines

Your suggested techboard review and approval process seems like a good solution.

We can add to the agenda for the next Techboard meeting.

Quoted 7 lines
Quoted 3 lines

And the performance benefit of direct rearm should be compared to the

performance using the new zero-copy mempool API.

Quoted 3 lines

-Morten

#57 lore raw

Re: [PATCH v3 1/3] ethdev: enable direct rearm with separate API

In reply to Feifei Wang

Hi Feifei,

Quoted 4 lines

Add 'tx_fill_sw_ring' and 'rx_flush_descriptor' API into direct rearm mode for separate Rx and Tx Operation. And this can support different multiple sources in direct rearm mode. For examples, Rx driver is ixgbe, and Tx driver is i40e.

Thanks for your effort and thanks for taking comments provided into consideration. That approach looks much better then previous ones.
Few nits below.
Konstantin

Quoted 137 lines
---
  lib/ethdev/ethdev_driver.h   |  10 ++
  lib/ethdev/ethdev_private.c  |   2 +
  lib/ethdev/rte_ethdev.c      |  52 +++++++++++
  lib/ethdev/rte_ethdev.h      | 174 +++++++++++++++++++++++++++++++++++
  lib/ethdev/rte_ethdev_core.h |  11 +++
  lib/ethdev/version.map       |   6 ++
  6 files changed, 255 insertions(+)
lib/ethdev/ethdev_driver.h +10 -0
index 6a550cfc83..bc539ec862 100644
@@ -59,6 +59,10 @@ struct rte_eth_dev {
  	eth_rx_descriptor_status_t rx_descriptor_status;
  	/** Check the status of a Tx descriptor */
  	eth_tx_descriptor_status_t tx_descriptor_status;
+	/** Fill Rx sw-ring with Tx buffers in direct rearm mode */
+	eth_tx_fill_sw_ring_t tx_fill_sw_ring;
+	/** Flush Rx descriptor in direct rearm mode */
+	eth_rx_flush_descriptor_t rx_flush_descriptor;
 
  	/**
  	 * Device data that is shared between primary and secondary processes
@@ -504,6 +508,10 @@ typedef void (*eth_rxq_info_get_t)(struct rte_eth_dev *dev,
  typedef void (*eth_txq_info_get_t)(struct rte_eth_dev *dev,
  	uint16_t tx_queue_id, struct rte_eth_txq_info *qinfo);
 
+/**< @internal Get rearm data for a receive queue of an Ethernet device. */
+typedef void (*eth_rxq_rearm_data_get_t)(struct rte_eth_dev *dev,
+	uint16_t tx_queue_id, struct rte_eth_rxq_rearm_data *rxq_rearm_data);
+
  typedef int (*eth_burst_mode_get_t)(struct rte_eth_dev *dev,
  	uint16_t queue_id, struct rte_eth_burst_mode *mode);
 
@@ -1215,6 +1223,8 @@ struct eth_dev_ops {
  	eth_rxq_info_get_t         rxq_info_get;
  	/** Retrieve Tx queue information */
  	eth_txq_info_get_t         txq_info_get;
+	/** Get Rx queue rearm data */
+	eth_rxq_rearm_data_get_t   rxq_rearm_data_get;
  	eth_burst_mode_get_t       rx_burst_mode_get; /**< Get Rx burst mode */
  	eth_burst_mode_get_t       tx_burst_mode_get; /**< Get Tx burst mode */
  	eth_fw_version_get_t       fw_version_get; /**< Get firmware version */
lib/ethdev/ethdev_private.c +2 -0
index 48090c879a..c5dd5e30f6 100644
@@ -276,6 +276,8 @@ eth_dev_fp_ops_setup(struct rte_eth_fp_ops *fpo,
  	fpo->rx_queue_count = dev->rx_queue_count;
  	fpo->rx_descriptor_status = dev->rx_descriptor_status;
  	fpo->tx_descriptor_status = dev->tx_descriptor_status;
+	fpo->tx_fill_sw_ring = dev->tx_fill_sw_ring;
+	fpo->rx_flush_descriptor = dev->rx_flush_descriptor;
 
  	fpo->rxq.data = dev->data->rx_queues;
  	fpo->rxq.clbk = (void **)(uintptr_t)dev->post_rx_burst_cbs;
lib/ethdev/rte_ethdev.c +52 -0
index 5d5e18db1e..2af5cb42fe 100644
@@ -3282,6 +3282,21 @@ rte_eth_dev_set_rx_queue_stats_mapping(uint16_t port_id, uint16_t rx_queue_id,
  						stat_idx, STAT_QMAP_RX));
  }
 
+int
+rte_eth_dev_direct_rearm(uint16_t rx_port_id, uint16_t rx_queue_id,
+		uint16_t tx_port_id, uint16_t tx_rx_queue_id,
+		struct rte_eth_rxq_rearm_data *rxq_rearm_data)
+{
+	int nb_rearm = 0;
+
+	nb_rearm = rte_eth_tx_fill_sw_ring(tx_port_id, tx_rx_queue_id, rxq_rearm_data);
+
+	if (nb_rearm > 0)
+		return rte_eth_rx_flush_descriptor(rx_port_id, rx_queue_id, nb_rearm);
+
+	return 0;
+}
+
  int
  rte_eth_dev_fw_version_get(uint16_t port_id, char *fw_version, size_t fw_size)
  {
@@ -5323,6 +5338,43 @@ rte_eth_tx_queue_info_get(uint16_t port_id, uint16_t queue_id,
  	return 0;
  }
 
+int
+rte_eth_rx_queue_rearm_data_get(uint16_t port_id, uint16_t queue_id,
+		struct rte_eth_rxq_rearm_data *rxq_rearm_data)
+{
+	struct rte_eth_dev *dev;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+	dev = &rte_eth_devices[port_id];
+
+	if (queue_id >= dev->data->nb_rx_queues) {
+		RTE_ETHDEV_LOG(ERR, "Invalid Rx queue_id=%u\n", queue_id);
+		return -EINVAL;
+	}
+
+	if (rxq_rearm_data == NULL) {
+		RTE_ETHDEV_LOG(ERR, "Cannot get ethdev port %u Rx queue %u rearm data to NULL\n",
+			port_id, queue_id);
+		return -EINVAL;
+	}
+
+	if (dev->data->rx_queues == NULL ||
+			dev->data->rx_queues[queue_id] == NULL) {
+		RTE_ETHDEV_LOG(ERR,
+			   "Rx queue %"PRIu16" of device with port_id=%"
+			   PRIu16" has not been setup\n",
+			   queue_id, port_id);
+		return -EINVAL;
+	}
+
+	if (*dev->dev_ops->rxq_rearm_data_get == NULL)
+		return -ENOTSUP;
+
+	dev->dev_ops->rxq_rearm_data_get(dev, queue_id, rxq_rearm_data);
+
+	return 0;
+}
+
  int
  rte_eth_rx_burst_mode_get(uint16_t port_id, uint16_t queue_id,
  			  struct rte_eth_burst_mode *mode)
lib/ethdev/rte_ethdev.h +0 -0
index c129ca1eaf..381c3d535f 100644

lib/ethdev/rte_ethdev.h struct rte_eth_txq_info { · line 1825

Quoted 8 lines
lib/ethdev/rte_ethdev.h +5 -0
@@ -1818,6 +1818,17 @@ struct rte_eth_txq_info {
  	uint8_t queue_state;        /**< one of RTE_ETH_QUEUE_STATE_*. */
  } __rte_cache_min_aligned;
 
+/**
+ * @internal
+ * Structure used to hold pointers to internal ethdev Rx rearm data.
+ * The main purpose is to load Rx queue rearm data in direct rearm mode.
+ */

I think this structure owes a lot more expalantion. What each fields suppose to do and what are the constraints, etc.

In general, more doc will be needed to explain that feature.

lib/ethdev/rte_ethdev.h struct rte_eth_txq_info { · line 1827

Quoted 2 lines
lib/ethdev/rte_ethdev.h +2 -0
+struct rte_eth_rxq_rearm_data {
+	void *rx_sw_ring;

That's misleading, we always suppose to store mbufs ptrs here, so why not be direct:
struct rte_mbuf **rx_sw_ring;

lib/ethdev/rte_ethdev.h struct rte_eth_txq_info { · line 1829

Quoted 2 lines
lib/ethdev/rte_ethdev.h +2 -0
+	uint16_t *rearm_start;
+	uint16_t *rearm_nb;

I know that for Intel NICs uint16_t is sufficient, wonder would it always be for other vendors?
Another thing to consider the case when ring position wrapping? Again I know that it is not required for Intel NICs, but would it be sufficient for API that supposed to be general?

Quoted 26 lines
lib/ethdev/rte_ethdev.h +19 -0
+} __rte_cache_min_aligned;
+
  /* Generic Burst mode flag definition, values can be ORed. */
 
  /**
@@ -3184,6 +3195,34 @@ int rte_eth_dev_set_rx_queue_stats_mapping(uint16_t port_id,
  					   uint16_t rx_queue_id,
  					   uint8_t stat_idx);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Directly put Tx freed buffers into Rx sw-ring and flush desc.
+ *
+ * @param rx_port_id
+ *   Port identifying the receive side.
+ * @param rx_queue_id
+ *   The index of the receive queue identifying the receive side.
+ *   The value must be in the range [0, nb_rx_queue - 1] previously supplied
+ *   to rte_eth_dev_configure().
+ * @param tx_port_id
+ *   Port identifying the transmit side.
+ * @param tx_queue_id
+ *   The index of the transmit queue identifying the transmit side.
+ *   The value must be in the range [0, nb_tx_queue - 1] previously supplied

lib/ethdev/rte_ethdev.h int rte_eth_dev_set_rx_queue_stats_mapping(uint16_t port_id, · line 3224

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+ *   to rte_eth_dev_configure().
+ * @param rxq_rearm_data
+ *   A pointer to a structure of type *rte_eth_txq_rearm_data* to be filled.
+ * @return
+ *   - 0: Success
+ */
+__rte_experimental
+int rte_eth_dev_direct_rearm(uint16_t rx_port_id, uint16_t rx_queue_id,
+		uint16_t tx_port_id, uint16_t tx_queue_id,
+		struct rte_eth_rxq_rearm_data *rxq_rearm_data);

I think we need one more parameter for that function 'uint16_t offset' or so. So _rearm_ will start to populate rx_sw_ring from *rearm_start + offset position. That way we can support populating from different sources. Or should 'offset' be part of truct rte_eth_rxq_rearm_data?

Quoted 206 lines
lib/ethdev/rte_ethdev.h +136 -0
+
  /**
   * Retrieve the Ethernet address of an Ethernet device.
   *
@@ -4782,6 +4821,27 @@ int rte_eth_rx_queue_info_get(uint16_t port_id, uint16_t queue_id,
  int rte_eth_tx_queue_info_get(uint16_t port_id, uint16_t queue_id,
  	struct rte_eth_txq_info *qinfo);
 
+/**
+ * Get rearm data about given ports's Rx queue.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param queue_id
+ *   The Rx queue on the Ethernet device for which rearm data
+ *   will be got.
+ * @param rxq_rearm_data
+ *   A pointer to a structure of type *rte_eth_txq_rearm_data* to be filled.
+ *
+ * @return
+ *   - 0: Success
+ *   - -ENODEV:  If *port_id* is invalid.
+ *   - -ENOTSUP: routine is not supported by the device PMD.
+ *   - -EINVAL:  The queue_id is out of range.
+ */
+__rte_experimental
+int rte_eth_rx_queue_rearm_data_get(uint16_t port_id, uint16_t queue_id,
+		struct rte_eth_rxq_rearm_data *rxq_rearm_data);
+
  /**
   * Retrieve information about the Rx packet burst mode.
   *
@@ -6103,6 +6163,120 @@ static inline int rte_eth_tx_descriptor_status(uint16_t port_id,
  	return (*p->tx_descriptor_status)(qd, offset);
  }
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Fill Rx sw-ring with Tx buffers in direct rearm mode.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param queue_id
+ *   The index of the transmit queue.
+ *   The value must be in the range [0, nb_tx_queue - 1] previously supplied
+ *   to rte_eth_dev_configure().
+ * @param rxq_rearm_data
+ *   A pointer to a structure of type *rte_eth_rxq_rearm_data* to be filled with
+ *   the rearm data of a receive queue.
+ * @return
+ *   - The number buffers correct to be filled in the Rx sw-ring.
+ *   - (-EINVAL) bad port or queue.
+ *   - (-ENODEV) bad port.
+ *   - (-ENOTSUP) if the device does not support this function.
+ *
+ */
+__rte_experimental
+static inline int rte_eth_tx_fill_sw_ring(uint16_t port_id,
+	uint16_t queue_id, struct rte_eth_rxq_rearm_data *rxq_rearm_data)
+{
+	struct rte_eth_fp_ops *p;
+	void *qd;
+
+#ifdef RTE_ETHDEV_DEBUG_TX
+	if (port_id >= RTE_MAX_ETHPORTS ||
+			queue_id >= RTE_MAX_QUEUES_PER_PORT) {
+		RTE_ETHDEV_LOG(ERR,
+			"Invalid port_id=%u or queue_id=%u\n",
+			port_id, queue_id);
+		return -EINVAL;
+	}
+#endif
+
+	p = &rte_eth_fp_ops[port_id];
+	qd = p->txq.data[queue_id];
+
+#ifdef RTE_ETHDEV_DEBUG_TX
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, 0);
+
+	if (qd == NULL) {
+		RTE_ETHDEV_LOG(ERR, "Invalid Tx queue_id=%u for port_id=%u\n",
+			queue_id, port_id);
+		return -ENODEV;
+	}
+#endif
+
+	if (p->tx_fill_sw_ring == NULL)
+		return -ENOTSUP;
+
+	return p->tx_fill_sw_ring(qd, rxq_rearm_data);
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ *  Flush Rx descriptor in direct rearm mode.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param queue_id
+ *   The index of the receive queue.
+ *   The value must be in the range [0, nb_rx_queue - 1] previously supplied
+ *   to rte_eth_dev_configure().
+ *@param nb_rearm
+ *   The number of Rx sw-ring buffers need to be flushed.
+ * @return
+ *   - (0) if successful.
+ *   - (-EINVAL) bad port or queue.
+ *   - (-ENODEV) bad port.
+ *   - (-ENOTSUP) if the device does not support this function.
+ */
+__rte_experimental
+static inline int rte_eth_rx_flush_descriptor(uint16_t port_id,
+	uint16_t queue_id, uint16_t nb_rearm)
+{
+	struct rte_eth_fp_ops *p;
+	void *qd;
+
+#ifdef RTE_ETHDEV_DEBUG_RX
+	if (port_id >= RTE_MAX_ETHPORTS ||
+			queue_id >= RTE_MAX_QUEUES_PER_PORT) {
+		RTE_ETHDEV_LOG(ERR,
+			"Invalid port_id=%u or queue_id=%u\n",
+			port_id, queue_id);
+		return -EINVAL;
+	}
+#endif
+
+	p = &rte_eth_fp_ops[port_id];
+	qd = p->rxq.data[queue_id];
+
+#ifdef RTE_ETHDEV_DEBUG_RX
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, 0);
+
+	if (qd == NULL) {
+		RTE_ETHDEV_LOG(ERR, "Invalid Rx queue_id=%u for port_id=%u\n",
+			queue_id, port_id);
+		return -ENODEV;
+	}
+#endif
+
+	if (p->rx_flush_descriptor == NULL)
+		return -ENOTSUP;
+
+	return p->rx_flush_descriptor(qd, nb_rearm);
+}
+
  /**
   * @internal
   * Helper routine for rte_eth_tx_burst().
lib/ethdev/rte_ethdev_core.h +11 -0
index dcf8adab92..5ecb57f6f0 100644
@@ -56,6 +56,13 @@ typedef int (*eth_rx_descriptor_status_t)(void *rxq, uint16_t offset);
  /** @internal Check the status of a Tx descriptor */
  typedef int (*eth_tx_descriptor_status_t)(void *txq, uint16_t offset);
 
+/** @internal Fill Rx sw-ring with Tx buffers in direct rearm mode */
+typedef int (*eth_tx_fill_sw_ring_t)(void *txq,
+		struct rte_eth_rxq_rearm_data *rxq_rearm_data);
+
+/** @internal Flush Rx descriptor in direct rearm mode */
+typedef int (*eth_rx_flush_descriptor_t)(void *rxq, uint16_t nb_rearm);
+
  /**
   * @internal
   * Structure used to hold opaque pointers to internal ethdev Rx/Tx
@@ -90,6 +97,8 @@ struct rte_eth_fp_ops {
  	eth_rx_queue_count_t rx_queue_count;
  	/** Check the status of a Rx descriptor. */
  	eth_rx_descriptor_status_t rx_descriptor_status;
+	/** Flush Rx descriptor in direct rearm mode */
+	eth_rx_flush_descriptor_t rx_flush_descriptor;
  	/** Rx queues data. */
  	struct rte_ethdev_qdata rxq;
  	uintptr_t reserved1[3];
@@ -106,6 +115,8 @@ struct rte_eth_fp_ops {
  	eth_tx_prep_t tx_pkt_prepare;
  	/** Check the status of a Tx descriptor. */
  	eth_tx_descriptor_status_t tx_descriptor_status;
+	/** Fill Rx sw-ring with Tx buffers in direct rearm mode */
+	eth_tx_fill_sw_ring_t tx_fill_sw_ring;
  	/** Tx queues data. */
  	struct rte_ethdev_qdata txq;
  	uintptr_t reserved2[3];
lib/ethdev/version.map +6 -0
index 17201fbe0f..f39f02a69b 100644
@@ -298,6 +298,12 @@ EXPERIMENTAL {
  	rte_flow_get_q_aged_flows;
  	rte_mtr_meter_policy_get;
  	rte_mtr_meter_profile_get;
+
+	# added in 23.03
+	rte_eth_dev_direct_rearm;
+	rte_eth_rx_flush_descriptor;
+	rte_eth_rx_queue_rearm_data_get;
+	rte_eth_tx_fill_sw_ring;
  };
 
  INTERNAL {
#58 lore raw

回复: [PATCH v3 1/3] ethdev: enable direct rearm with separate API

Hi, Konstantin

Thanks for your reviewing and sorry for my delayed response. For your comments, we put forward several improvement plans below.

Best Regards
Feifei

Quoted 188 lines

-----邮件原件-----
发件人: Konstantin Ananyev <[email protected]> 发送时间: Thursday, February 2, 2023 10:33 PM
收件人: Feifei Wang <[email protected]>; [email protected]; Ferruh Yigit <[email protected]>; Andrew Rybchenko <[email protected]>
抄送: [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; Ruifeng Wang <[email protected]>
主题: Re: [PATCH v3 1/3] ethdev: enable direct rearm with separate API

Hi Feifei,

Quoted 4 lines

Add 'tx_fill_sw_ring' and 'rx_flush_descriptor' API into direct rearm mode for separate Rx and Tx Operation. And this can support different multiple sources in direct rearm mode. For examples, Rx driver is ixgbe, and Tx driver is i40e.

Thanks for your effort and thanks for taking comments provided into consideration.
That approach looks much better then previous ones.
Few nits below.
Konstantin

Quoted 10 lines
---
  lib/ethdev/ethdev_driver.h   |  10 ++
  lib/ethdev/ethdev_private.c  |   2 +
  lib/ethdev/rte_ethdev.c      |  52 +++++++++++
  lib/ethdev/rte_ethdev.h      | 174

+++++++++++++++++++++++++++++++++++

Quoted 19 lines
  lib/ethdev/rte_ethdev_core.h |  11 +++
  lib/ethdev/version.map       |   6 ++
  6 files changed, 255 insertions(+)
lib/ethdev/ethdev_driver.h +4 -0
index 6a550cfc83..bc539ec862 100644
@@ -59,6 +59,10 @@ struct rte_eth_dev {
  	eth_rx_descriptor_status_t rx_descriptor_status;
  	/** Check the status of a Tx descriptor */
  	eth_tx_descriptor_status_t tx_descriptor_status;
+	/** Fill Rx sw-ring with Tx buffers in direct rearm mode */
+	eth_tx_fill_sw_ring_t tx_fill_sw_ring;
+	/** Flush Rx descriptor in direct rearm mode */
+	eth_rx_flush_descriptor_t rx_flush_descriptor;
 
  	/**
  	 * Device data that is shared between primary and secondary
Quoted 1 line

processes @@ -504,6 +508,10 @@ typedef void

(*eth_rxq_info_get_t)(struct rte_eth_dev *dev,

lib/ethdev/ethdev_driver.h

Quoted 5 lines

typedef void (*eth_txq_info_get_t)(struct rte_eth_dev *dev,
uint16_t tx_queue_id, struct rte_eth_txq_info *qinfo);

lib/ethdev/ethdev_driver.h +2 -0
+/**< @internal Get rearm data for a receive queue of an Ethernet
+device. */ typedef void (*eth_rxq_rearm_data_get_t)(struct rte_eth_dev

*dev,

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1228

Quoted 13 lines
lib/ethdev/ethdev_driver.h +5 -0
+	uint16_t tx_queue_id, struct rte_eth_rxq_rearm_data
+*rxq_rearm_data);
+
  typedef int (*eth_burst_mode_get_t)(struct rte_eth_dev *dev,
  	uint16_t queue_id, struct rte_eth_burst_mode *mode);
@@ -1215,6 +1223,8 @@ struct eth_dev_ops {
  	eth_rxq_info_get_t         rxq_info_get;
  	/** Retrieve Tx queue information */
  	eth_txq_info_get_t         txq_info_get;
+	/** Get Rx queue rearm data */
+	eth_rxq_rearm_data_get_t   rxq_rearm_data_get;
  	eth_burst_mode_get_t       rx_burst_mode_get; /**< Get Rx burst

mode */

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1229

Quoted 1 line
lib/ethdev/ethdev_driver.h +0 -0
  	eth_burst_mode_get_t       tx_burst_mode_get; /**< Get Tx burst

mode */

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1230

Quoted 1 line
lib/ethdev/ethdev_driver.h +0 -0
  	eth_fw_version_get_t       fw_version_get; /**< Get firmware version

*/

lib/ethdev/rte_ethdev.c

Quoted 18 lines
lib/ethdev/ethdev_private.c +2 -0
index 48090c879a..c5dd5e30f6 100644
@@ -276,6 +276,8 @@ eth_dev_fp_ops_setup(struct rte_eth_fp_ops *fpo,
  	fpo->rx_queue_count = dev->rx_queue_count;
  	fpo->rx_descriptor_status = dev->rx_descriptor_status;
  	fpo->tx_descriptor_status = dev->tx_descriptor_status;
+	fpo->tx_fill_sw_ring = dev->tx_fill_sw_ring;
+	fpo->rx_flush_descriptor = dev->rx_flush_descriptor;
 
  	fpo->rxq.data = dev->data->rx_queues;
  	fpo->rxq.clbk = (void **)(uintptr_t)dev->post_rx_burst_cbs;
lib/ethdev/rte_ethdev.c index renamed from lib/ethdev/rte_ethdev.c +0 -0

5d5e18db1e..2af5cb42fe 100644

lib/ethdev/rte_ethdev.c +0 -0
@@ -3282,6 +3282,21 @@

rte_eth_dev_set_rx_queue_stats_mapping(uint16_t port_id, uint16_t rx_queue_id,

Quoted 11 lines
lib/ethdev/rte_ethdev.c +8 -0
  						stat_idx, STAT_QMAP_RX));
  }
 
+int
+rte_eth_dev_direct_rearm(uint16_t rx_port_id, uint16_t rx_queue_id,
+		uint16_t tx_port_id, uint16_t tx_rx_queue_id,
+		struct rte_eth_rxq_rearm_data *rxq_rearm_data) {
+	int nb_rearm = 0;
+
+	nb_rearm = rte_eth_tx_fill_sw_ring(tx_port_id, tx_rx_queue_id,
+rxq_rearm_data);

lib/ethdev/rte_ethdev.c rte_eth_dev_set_rx_queue_stats_mapping(uint16_t port_id, uint16_t rx_queue_id,

Quoted 10 lines
lib/ethdev/rte_ethdev.c +8 -0
+
+	if (nb_rearm > 0)
+		return rte_eth_rx_flush_descriptor(rx_port_id, rx_queue_id,
+nb_rearm);
+
+	return 0;
+}
+
  int
  rte_eth_dev_fw_version_get(uint16_t port_id, char *fw_version, size_t

fw_size)

lib/ethdev/rte_ethdev.c rte_eth_tx_queue_info_get(uint16_t port_id,

Quoted 2 lines
lib/ethdev/rte_ethdev.c +0 -0
  {
@@ -5323,6 +5338,43 @@ rte_eth_tx_queue_info_get(uint16_t port_id,

uint16_t queue_id,

lib/ethdev/rte_ethdev.c rte_eth_tx_queue_info_get(uint16_t port_id, uint16_t queue_id,

Quoted 13 lines
lib/ethdev/rte_ethdev.c +10 -0
  	return 0;
  }
 
+int
+rte_eth_rx_queue_rearm_data_get(uint16_t port_id, uint16_t queue_id,
+		struct rte_eth_rxq_rearm_data *rxq_rearm_data) {
+	struct rte_eth_dev *dev;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+	dev = &rte_eth_devices[port_id];
+
+	if (queue_id >= dev->data->nb_rx_queues) {
+		RTE_ETHDEV_LOG(ERR, "Invalid Rx queue_id=%u\n",

queue_id);

lib/ethdev/rte_ethdev.c rte_eth_tx_queue_info_get(uint16_t port_id, uint16_t queue_id,

Quoted 5 lines
lib/ethdev/rte_ethdev.c +5 -0
+		return -EINVAL;
+	}
+
+	if (rxq_rearm_data == NULL) {
+		RTE_ETHDEV_LOG(ERR, "Cannot get ethdev port %u Rx

queue %u rearm data to NULL\n",

Quoted 7 lines
lib/ethdev/rte_ethdev.c +7 -0
+			port_id, queue_id);
+		return -EINVAL;
+	}
+
+	if (dev->data->rx_queues == NULL ||
+			dev->data->rx_queues[queue_id] == NULL) {
+		RTE_ETHDEV_LOG(ERR,

lib/ethdev/rte_ethdev.c rte_eth_tx_queue_info_get(uint16_t port_id, uint16_t queue_id,

Quoted 10 lines
lib/ethdev/rte_ethdev.c +10 -0
+			   "Rx queue %"PRIu16" of device with port_id=%"
+			   PRIu16" has not been setup\n",
+			   queue_id, port_id);
+		return -EINVAL;
+	}
+
+	if (*dev->dev_ops->rxq_rearm_data_get == NULL)
+		return -ENOTSUP;
+
+	dev->dev_ops->rxq_rearm_data_get(dev, queue_id,

rxq_rearm_data);

Quoted 12 lines
lib/ethdev/rte_ethdev.c +4 -0
+
+	return 0;
+}
+
  int
  rte_eth_rx_burst_mode_get(uint16_t port_id, uint16_t queue_id,
  			  struct rte_eth_burst_mode *mode) diff --git

a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h index c129ca1eaf..381c3d535f 100644

lib/ethdev/rte_ethdev.h struct rte_eth_txq_info { · line 1825

Quoted 8 lines
lib/ethdev/rte_ethdev.h +5 -0
@@ -1818,6 +1818,17 @@ struct rte_eth_txq_info {
  	uint8_t queue_state;        /**< one of RTE_ETH_QUEUE_STATE_*. */
  } __rte_cache_min_aligned;
 
+/**
+ * @internal
+ * Structure used to hold pointers to internal ethdev Rx rearm data.
+ * The main purpose is to load Rx queue rearm data in direct rearm mode.
+ */

I think this structure owes a lot more expalantion. What each fields suppose to do and what are the constraints, etc.

Quoted 1 line

In general, more doc will be needed to explain that feature.

You are right. We will add more explanation for it and also update the doc To explain the feature.

Quoted 7 lines

lib/ethdev/rte_ethdev.h struct rte_eth_txq_info { · line 1827

Quoted 2 lines
lib/ethdev/rte_ethdev.h +2 -0
+struct rte_eth_rxq_rearm_data {
+	void *rx_sw_ring;

That's misleading, we always suppose to store mbufs ptrs here, so why not be direct:
struct rte_mbuf **rx_sw_ring;

Agree.

Quoted 9 lines

lib/ethdev/rte_ethdev.h struct rte_eth_txq_info { · line 1829

Quoted 2 lines
lib/ethdev/rte_ethdev.h +2 -0
+	uint16_t *rearm_start;
+	uint16_t *rearm_nb;

I know that for Intel NICs uint16_t is sufficient, wonder would it always be for other vendors?
Another thing to consider the case when ring position wrapping? Again I know that it is not required for Intel NICs, but would it be sufficient for API that supposed to be general?

For this, we re-define this structure:

rte_eth_rxq_rearm_data {
	void *rx_sw_ring;
	uint16_t *rearm_start;
	uint16_t *rearm_nb;
}
->
struct *rxq_recycle_info {
	rte_mbuf **buf_ring;
	uint16_t *offset = (uint16 *)(&rq->ci);
	uint16_t *end;
	uint16_t ring_size; 

}
For the new structure, *offset is a pointer for rearm-start index of Rx buffer ring (consumer index). *end is a pointer for rearm-end index Of Rx buffer ring (producer index).

1. we look up different pmds, some pmds using 'uint_16t' as index size like intel PMD, some pmds using 'uint32_t' as index size like MLX5 or thunderx PMD.
For pmd using 'uint32_t', rearm starts at 'buf_ring[offset & (ring_size -1)]', and 'uint16_t' is enough for ring size.

2. Good question. In general path, there is a constraint that 'nb_rearm < ring_size - rq->ci', This can ensure no ring wrapping in rearm. Thus in direct-rearm, we will refer to this to solve ring wrapping.

Quoted 40 lines

lib/ethdev/rte_ethdev.h int

Quoted 7 lines
lib/ethdev/rte_ethdev.h +2 -0
+} __rte_cache_min_aligned;
+
  /* Generic Burst mode flag definition, values can be ORed. */
 
  /**
@@ -3184,6 +3195,34 @@ int

rte_eth_dev_set_rx_queue_stats_mapping(uint16_t port_id,

Quoted 5 lines
lib/ethdev/rte_ethdev.h +2 -0
  					   uint16_t rx_queue_id,
  					   uint8_t stat_idx);
 
+/**
+ * @warning

lib/ethdev/rte_ethdev.h int rte_eth_dev_set_rx_queue_stats_mapping(uint16_t port_id,

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior
+notice
+ *
+ * Directly put Tx freed buffers into Rx sw-ring and flush desc.
+ *
+ * @param rx_port_id
+ *   Port identifying the receive side.
+ * @param rx_queue_id
+ *   The index of the receive queue identifying the receive side.
+ *   The value must be in the range [0, nb_rx_queue - 1] previously

supplied

lib/ethdev/rte_ethdev.h int rte_eth_dev_set_rx_queue_stats_mapping(uint16_t port_id,

Quoted 6 lines
lib/ethdev/rte_ethdev.h +6 -0
+ *   to rte_eth_dev_configure().
+ * @param tx_port_id
+ *   Port identifying the transmit side.
+ * @param tx_queue_id
+ *   The index of the transmit queue identifying the transmit side.
+ *   The value must be in the range [0, nb_tx_queue - 1] previously

supplied

Quoted 9 lines
lib/ethdev/rte_ethdev.h +9 -0
+ *   to rte_eth_dev_configure().
+ * @param rxq_rearm_data
+ *   A pointer to a structure of type *rte_eth_txq_rearm_data* to be filled.
+ * @return
+ *   - 0: Success
+ */
+__rte_experimental
+int rte_eth_dev_direct_rearm(uint16_t rx_port_id, uint16_t rx_queue_id,
+		uint16_t tx_port_id, uint16_t tx_queue_id,
Quoted 9 lines

lib/ethdev/rte_ethdev.h int · line 3225

Quoted 1 line
lib/ethdev/rte_ethdev.h +1 -0
+		struct rte_eth_rxq_rearm_data *rxq_rearm_data);

I think we need one more parameter for that function 'uint16_t offset' or so.
So _rearm_ will start to populate rx_sw_ring from *rearm_start + offset position. That way we can support populating from different sources. Or should 'offset' be part of truct rte_eth_rxq_rearm_data?

Agree, please see above, we will do the change in the structure.

Quoted 215 lines

lib/ethdev/rte_ethdev.h int rte_eth_rx_queue_info_get(uint16_t

Quoted 5 lines
lib/ethdev/rte_ethdev.h +1 -0
+
  /**
   * Retrieve the Ethernet address of an Ethernet device.
   *
@@ -4782,6 +4821,27 @@ int rte_eth_rx_queue_info_get(uint16_t

port_id, uint16_t queue_id,

Quoted 12 lines
lib/ethdev/rte_ethdev.h +9 -0
  int rte_eth_tx_queue_info_get(uint16_t port_id, uint16_t queue_id,
  	struct rte_eth_txq_info *qinfo);
 
+/**
+ * Get rearm data about given ports's Rx queue.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param queue_id
+ *   The Rx queue on the Ethernet device for which rearm data
+ *   will be got.
+ * @param rxq_rearm_data

lib/ethdev/rte_ethdev.h int rte_eth_rx_queue_info_get(uint16_t port_id, uint16_t queue_id,

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+ *   A pointer to a structure of type *rte_eth_txq_rearm_data* to be filled.
+ *
+ * @return
+ *   - 0: Success
+ *   - -ENODEV:  If *port_id* is invalid.
+ *   - -ENOTSUP: routine is not supported by the device PMD.
+ *   - -EINVAL:  The queue_id is out of range.
+ */
+__rte_experimental
+int rte_eth_rx_queue_rearm_data_get(uint16_t port_id, uint16_t

queue_id,

lib/ethdev/rte_ethdev.h static inline int

Quoted 6 lines
lib/ethdev/rte_ethdev.h +2 -0
+		struct rte_eth_rxq_rearm_data *rxq_rearm_data);
+
  /**
   * Retrieve information about the Rx packet burst mode.
   *
@@ -6103,6 +6163,120 @@ static inline int

rte_eth_tx_descriptor_status(uint16_t port_id,

Quoted 5 lines
lib/ethdev/rte_ethdev.h +2 -0
  	return (*p->tx_descriptor_status)(qd, offset);
  }
 
+/**
+ * @warning

lib/ethdev/rte_ethdev.h static inline int rte_eth_tx_descriptor_status(uint16_t port_id,

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior
+notice
+ *
+ * Fill Rx sw-ring with Tx buffers in direct rearm mode.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param queue_id
+ *   The index of the transmit queue.
+ *   The value must be in the range [0, nb_tx_queue - 1] previously

supplied

lib/ethdev/rte_ethdev.h int rte_eth_dev_set_rx_queue_stats_mapping(uint16_t port_id,

Quoted 3 lines
lib/ethdev/rte_ethdev.h +3 -0
+ *   to rte_eth_dev_configure().
+ * @param rxq_rearm_data
+ *   A pointer to a structure of type *rte_eth_rxq_rearm_data* to be filled

with

lib/ethdev/rte_ethdev.h static inline int rte_eth_tx_descriptor_status(uint16_t port_id, · line 6190

Quoted 11 lines
lib/ethdev/rte_ethdev.h +11 -0
+ *   the rearm data of a receive queue.
+ * @return
+ *   - The number buffers correct to be filled in the Rx sw-ring.
+ *   - (-EINVAL) bad port or queue.
+ *   - (-ENODEV) bad port.
+ *   - (-ENOTSUP) if the device does not support this function.
+ *
+ */
+__rte_experimental
+static inline int rte_eth_tx_fill_sw_ring(uint16_t port_id,
+	uint16_t queue_id, struct rte_eth_rxq_rearm_data *rxq_rearm_data)

{

Quoted 11 lines
lib/ethdev/rte_ethdev.h +11 -0
+	struct rte_eth_fp_ops *p;
+	void *qd;
+
+#ifdef RTE_ETHDEV_DEBUG_TX
+	if (port_id >= RTE_MAX_ETHPORTS ||
+			queue_id >= RTE_MAX_QUEUES_PER_PORT) {
+		RTE_ETHDEV_LOG(ERR,
+			"Invalid port_id=%u or queue_id=%u\n",
+			port_id, queue_id);
+		return -EINVAL;
+	}

lib/ethdev/rte_ethdev.h static inline int rte_eth_tx_descriptor_status(uint16_t port_id,

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+#endif
+
+	p = &rte_eth_fp_ops[port_id];
+	qd = p->txq.data[queue_id];
+
+#ifdef RTE_ETHDEV_DEBUG_TX
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, 0);
+
+	if (qd == NULL) {
+		RTE_ETHDEV_LOG(ERR, "Invalid Tx queue_id=%u for

port_id=%u\n",

Quoted 12 lines
lib/ethdev/rte_ethdev.h +12 -0
+			queue_id, port_id);
+		return -ENODEV;
+	}
+#endif
+
+	if (p->tx_fill_sw_ring == NULL)
+		return -ENOTSUP;
+
+	return p->tx_fill_sw_ring(qd, rxq_rearm_data); }
+
+/**
+ * @warning

lib/ethdev/rte_ethdev.h static inline int rte_eth_tx_descriptor_status(uint16_t port_id,

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior
+notice
+ *
+ *  Flush Rx descriptor in direct rearm mode.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param queue_id
+ *   The index of the receive queue.
+ *   The value must be in the range [0, nb_rx_queue - 1] previously

supplied

Quoted 23 lines
lib/ethdev/rte_ethdev.h +23 -0
+ *   to rte_eth_dev_configure().
+ *@param nb_rearm
+ *   The number of Rx sw-ring buffers need to be flushed.
+ * @return
+ *   - (0) if successful.
+ *   - (-EINVAL) bad port or queue.
+ *   - (-ENODEV) bad port.
+ *   - (-ENOTSUP) if the device does not support this function.
+ */
+__rte_experimental
+static inline int rte_eth_rx_flush_descriptor(uint16_t port_id,
+	uint16_t queue_id, uint16_t nb_rearm) {
+	struct rte_eth_fp_ops *p;
+	void *qd;
+
+#ifdef RTE_ETHDEV_DEBUG_RX
+	if (port_id >= RTE_MAX_ETHPORTS ||
+			queue_id >= RTE_MAX_QUEUES_PER_PORT) {
+		RTE_ETHDEV_LOG(ERR,
+			"Invalid port_id=%u or queue_id=%u\n",
+			port_id, queue_id);
+		return -EINVAL;
+	}

lib/ethdev/rte_ethdev.h static inline int rte_eth_tx_descriptor_status(uint16_t port_id,

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+#endif
+
+	p = &rte_eth_fp_ops[port_id];
+	qd = p->rxq.data[queue_id];
+
+#ifdef RTE_ETHDEV_DEBUG_RX
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, 0);
+
+	if (qd == NULL) {
+		RTE_ETHDEV_LOG(ERR, "Invalid Rx queue_id=%u for

port_id=%u\n",

lib/ethdev/rte_ethdev_core.h typedef int (*eth_rx_descriptor_status_t)(void *rxq,

Quoted 18 lines
lib/ethdev/rte_ethdev.h +10 -0
+			queue_id, port_id);
+		return -ENODEV;
+	}
+#endif
+
+	if (p->rx_flush_descriptor == NULL)
+		return -ENOTSUP;
+
+	return p->rx_flush_descriptor(qd, nb_rearm); }
+
  /**
   * @internal
   * Helper routine for rte_eth_tx_burst().
(quoted diff) +0 -0

b/lib/ethdev/rte_ethdev_core.h index dcf8adab92..5ecb57f6f0 100644

lib/ethdev/rte_ethdev_core.h +0 -0
@@ -56,6 +56,13 @@ typedef int (*eth_rx_descriptor_status_t)(void *rxq,

uint16_t offset);

Quoted 49 lines
lib/ethdev/rte_ethdev_core.h +7 -0
  /** @internal Check the status of a Tx descriptor */
  typedef int (*eth_tx_descriptor_status_t)(void *txq, uint16_t
 offset);
 
+/** @internal Fill Rx sw-ring with Tx buffers in direct rearm mode */
+typedef int (*eth_tx_fill_sw_ring_t)(void *txq,
+		struct rte_eth_rxq_rearm_data *rxq_rearm_data);
+
+/** @internal Flush Rx descriptor in direct rearm mode */ typedef int
+(*eth_rx_flush_descriptor_t)(void *rxq, uint16_t nb_rearm);
+
  /**
   * @internal

* Structure used to hold opaque pointers to internal ethdev Rx/Tx

lib/ethdev/rte_ethdev_core.h +4 -0
@@ -90,6 +97,8 @@ struct rte_eth_fp_ops {
  	eth_rx_queue_count_t rx_queue_count;
  	/** Check the status of a Rx descriptor. */
  	eth_rx_descriptor_status_t rx_descriptor_status;
+	/** Flush Rx descriptor in direct rearm mode */
+	eth_rx_flush_descriptor_t rx_flush_descriptor;
  	/** Rx queues data. */
  	struct rte_ethdev_qdata rxq;
  	uintptr_t reserved1[3];
@@ -106,6 +115,8 @@ struct rte_eth_fp_ops {
  	eth_tx_prep_t tx_pkt_prepare;
  	/** Check the status of a Tx descriptor. */
  	eth_tx_descriptor_status_t tx_descriptor_status;
+	/** Fill Rx sw-ring with Tx buffers in direct rearm mode */
+	eth_tx_fill_sw_ring_t tx_fill_sw_ring;
  	/** Tx queues data. */
  	struct rte_ethdev_qdata txq;
  	uintptr_t reserved2[3];
lib/ethdev/version.map index renamed from lib/ethdev/version.map +0 -0

17201fbe0f..f39f02a69b 100644

lib/ethdev/version.map +6 -0
@@ -298,6 +298,12 @@ EXPERIMENTAL {
  	rte_flow_get_q_aged_flows;
  	rte_mtr_meter_policy_get;
  	rte_mtr_meter_profile_get;
+
+	# added in 23.03
+	rte_eth_dev_direct_rearm;
+	rte_eth_rx_flush_descriptor;
+	rte_eth_rx_queue_rearm_data_get;
+	rte_eth_tx_fill_sw_ring;
  };
 
  INTERNAL {
#59 lore raw

RE: [PATCH v3 1/3] ethdev: enable direct rearm with separate API

Hi Feifei ,

Quoted 27 lines
Quoted 9 lines

lib/ethdev/rte_ethdev.h struct rte_eth_txq_info { · line 1829

Quoted 2 lines
lib/ethdev/rte_ethdev.h +2 -0
+	uint16_t *rearm_start;
+	uint16_t *rearm_nb;

I know that for Intel NICs uint16_t is sufficient, wonder would it always be for other vendors?
Another thing to consider the case when ring position wrapping? Again I know that it is not required for Intel NICs, but would it be sufficient for API that supposed to be general?

For this, we re-define this structure:

rte_eth_rxq_rearm_data {
	void *rx_sw_ring;
	uint16_t *rearm_start;
	uint16_t *rearm_nb;
}
->
struct *rxq_recycle_info {
	rte_mbuf **buf_ring;
	uint16_t *offset = (uint16 *)(&rq->ci);
	uint16_t *end;
	uint16_t ring_size;

}
For the new structure, *offset is a pointer for rearm-start index of Rx buffer ring (consumer index). *end is a pointer for rearm-end index Of Rx buffer ring (producer index).

Quoted 4 lines

1. we look up different pmds, some pmds using 'uint_16t' as index size like intel PMD, some pmds using 'uint32_t' as index size like MLX5 or thunderx PMD.
For pmd using 'uint32_t', rearm starts at 'buf_ring[offset & (ring_size -1)]', and 'uint16_t' is enough for ring size.

Sounds like a smart idea to me.

Quoted 4 lines

2. Good question. In general path, there is a constraint that 'nb_rearm < ring_size - rq->ci', This can ensure no ring wrapping in rearm. Thus in direct-rearm, we will refer to this to solve ring wrapping.

Should work, I think...
Just need not to forget to document it :)

#60 lore raw

回复: [PATCH v3 1/3] ethdev: enable direct rearm with separate API

Quoted 56 lines

-----邮件原件-----
发件人: Konstantin Ananyev <[email protected]> 发送时间: Tuesday, February 28, 2023 3:32 AM
收件人: Feifei Wang <[email protected]>; Konstantin Ananyev <[email protected]>; [email protected]; Ferruh Yigit <[email protected]>; Andrew Rybchenko <[email protected]>
抄送: [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; Ruifeng Wang <[email protected]>; nd <[email protected]>
主题: RE: [PATCH v3 1/3] ethdev: enable direct rearm with separate API

Hi Feifei ,

Quoted 22 lines
Quoted 9 lines

lib/ethdev/rte_ethdev.h struct rte_eth_txq_info { · line 1829

Quoted 2 lines
lib/ethdev/rte_ethdev.h +2 -0
+	uint16_t *rearm_start;
+	uint16_t *rearm_nb;

I know that for Intel NICs uint16_t is sufficient, wonder would it always be for other vendors?
Another thing to consider the case when ring position wrapping? Again I know that it is not required for Intel NICs, but would it be sufficient for API that supposed to be general?

For this, we re-define this structure:

rte_eth_rxq_rearm_data {
	void *rx_sw_ring;
	uint16_t *rearm_start;
	uint16_t *rearm_nb;
}
->
struct *rxq_recycle_info {
	rte_mbuf **buf_ring;
	uint16_t *offset = (uint16 *)(&rq->ci);
	uint16_t *end;
	uint16_t ring_size;
Quoted 7 lines

}
For the new structure, *offset is a pointer for rearm-start index of Rx buffer ring (consumer index). *end is a pointer for rearm-end index Of Rx buffer ring (producer index).

1. we look up different pmds, some pmds using 'uint_16t' as index size like intel PMD, some pmds using 'uint32_t' as index size like MLX5 or

thunderx PMD.

Quoted 1 line

For pmd using 'uint32_t', rearm starts at 'buf_ring[offset & (ring_size -1)]',

and 'uint16_t'

Quoted 1 line

is enough for ring size.

Sounds like a smart idea to me.

Quoted 1 line
Quoted 6 lines
Quoted 3 lines

2. Good question. In general path, there is a constraint that 'nb_rearm < ring_size - rq->ci', This can ensure no ring wrapping in rearm. Thus in direct-rearm, we will refer to this to solve ring wrapping.

Should work, I think...
Just need not to forget to document it :)

Agree, we need to doc this.

#61 lore raw

RE: [PATCH v3 1/3] ethdev: enable direct rearm with separate API

In reply to Konstantin Ananyev

Hi Feifei,

Quoted 35 lines

From: Konstantin Ananyev [mailto:[email protected]]
Sent: Monday, 27 February 2023 20.32

Hi Feifei ,

Quoted 4 lines
Quoted 4 lines

lib/ethdev/rte_ethdev.h struct rte_eth_txq_info { · line 1829

Quoted 2 lines
lib/ethdev/rte_ethdev.h +2 -0
+	uint16_t *rearm_start;
+	uint16_t *rearm_nb;

I know that for Intel NICs uint16_t is sufficient, wonder would it always

be

Quoted 3 lines
Quoted 3 lines

for other vendors?
Another thing to consider the case when ring position wrapping? Again I know that it is not required for Intel NICs, but would it be

sufficient

Quoted 20 lines
Quoted 2 lines

for API that supposed to be general?

For this, we re-define this structure:

rte_eth_rxq_rearm_data {
	void *rx_sw_ring;
	uint16_t *rearm_start;
	uint16_t *rearm_nb;
}
->
struct *rxq_recycle_info {
	rte_mbuf **buf_ring;
	uint16_t *offset = (uint16 *)(&rq->ci);
	uint16_t *end;
	uint16_t ring_size;

}
For the new structure, *offset is a pointer for rearm-start index of Rx buffer ring (consumer index). *end is a pointer for rearm-end index Of Rx buffer ring (producer index).

Quoted 8 lines
Quoted 1 line

1. we look up different pmds, some pmds using 'uint_16t' as index size like

intel PMD,

Quoted 2 lines

some pmds using 'uint32_t' as index size like MLX5 or thunderx PMD. For pmd using 'uint32_t', rearm starts at 'buf_ring[offset & (ring_size -

1)]', and 'uint16_t'

Quoted 1 line

is enough for ring size.

Sounds like a smart idea to me.

When configuring an Ethernet device queue, the nb_rx/tx_desc parameter to rte_eth_rx/tx_queue_setup() is uint16_t, so I agree that uint16_t should suffice here too.

I had the following thought, but am not sure. So please take this comment for consideration only:

I think the "& (ring_size -1)" is superfluous, unless a PMD allows its index pointer to exceed the ring size, and performs the same "& (ring_size -1)" when using the index pointer to access its ring.

And if a PMD uses the index pointer like that (i.e. exceeding the ring size), you would need the same wrap protection for a 16 bit index pointer.

Quoted 11 lines
Quoted 4 lines

2. Good question. In general path, there is a constraint that 'nb_rearm <

ring_size - rq->ci',

Quoted 1 line

This can ensure no ring wrapping in rearm. Thus in direct-rearm, we will

refer to this to

Quoted 1 line

solve ring wrapping.

Should work, I think...
Just need not to forget to document it :)

It is this constraint (the guarantee that there is no ring wrapping in a rearm burst) that makes me think that the "& (ring_size -1)" is superfluous.

#62 lore raw

回复: [PATCH v3 1/3] ethdev: enable direct rearm with separate API

Hi, Morten

Quoted 85 lines

-----邮件原件-----
发件人: Morten Brørup <[email protected]>
发送时间: Tuesday, February 28, 2023 4:09 PM
收件人: Konstantin Ananyev <[email protected]>; Feifei Wang <[email protected]>; Konstantin Ananyev <[email protected]>; [email protected]; Ferruh Yigit <[email protected]>; Andrew Rybchenko <[email protected]>
抄送: [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; Ruifeng Wang <[email protected]>; nd <[email protected]>
主题: RE: [PATCH v3 1/3] ethdev: enable direct rearm with separate API

Hi Feifei,

Quoted 39 lines

From: Konstantin Ananyev [mailto:[email protected]]
Sent: Monday, 27 February 2023 20.32

Hi Feifei ,

Quoted 5 lines
Quoted 5 lines

lib/ethdev/rte_ethdev.h struct rte_eth_txq_info { · line 1829

Quoted 2 lines
lib/ethdev/rte_ethdev.h +2 -0
+	uint16_t *rearm_start;
+	uint16_t *rearm_nb;

I know that for Intel NICs uint16_t is sufficient, wonder would it always

be

Quoted 4 lines
Quoted 4 lines

for other vendors?
Another thing to consider the case when ring position wrapping? Again I know that it is not required for Intel NICs, but would it be

sufficient

Quoted 15 lines
Quoted 2 lines

for API that supposed to be general?

For this, we re-define this structure:

rte_eth_rxq_rearm_data {
	void *rx_sw_ring;
	uint16_t *rearm_start;
	uint16_t *rearm_nb;
}
->
struct *rxq_recycle_info {
	rte_mbuf **buf_ring;
	uint16_t *offset = (uint16 *)(&rq->ci);
	uint16_t *end;
	uint16_t ring_size;
Quoted 7 lines

}
For the new structure, *offset is a pointer for rearm-start index of Rx buffer ring (consumer index). *end is a pointer for rearm-end index Of Rx buffer ring (producer index).

1. we look up different pmds, some pmds using 'uint_16t' as index size like

Quoted 8 lines

intel PMD,

Quoted 3 lines

some pmds using 'uint32_t' as index size like MLX5 or thunderx PMD. For pmd using 'uint32_t', rearm starts at 'buf_ring[offset & (ring_size -

1)]', and 'uint16_t'

Quoted 1 line

is enough for ring size.

Sounds like a smart idea to me.

When configuring an Ethernet device queue, the nb_rx/tx_desc parameter to rte_eth_rx/tx_queue_setup() is uint16_t, so I agree that uint16_t should suffice here too.

I had the following thought, but am not sure. So please take this comment for consideration only:

I think the "& (ring_size -1)" is superfluous, unless a PMD allows its index pointer to exceed the ring size, and performs the same "& (ring_size -1)" when using the index pointer to access its ring.

And if a PMD uses the index pointer like that (i.e. exceeding the ring size), you would need the same wrap protection for a 16 bit index pointer.

Quoted 8 lines
Quoted 5 lines

2. Good question. In general path, there is a constraint that 'nb_rearm <

ring_size - rq->ci',

Quoted 2 lines

This can ensure no ring wrapping in rearm. Thus in direct-rearm, we will

Quoted 8 lines
Quoted 5 lines

refer to this to

Quoted 1 line

solve ring wrapping.

Should work, I think...
Just need not to forget to document it :)

It is this constraint (the guarantee that there is no ring wrapping in a rearm burst) that makes me think that the "& (ring_size -1)" is superfluous.

Actually there is some misleading here, I can explain about this.

(ring_size -1 ) is for some pmds whose index can exceed the ring_size, such as MLX5:
uint16_t *offset = &mlx5_rxq->rq_ci, rq_ci is an index which can exceed the ring_size.

#63 lore raw

PATCH v3 2/3 net/i40e: enable direct rearm with separate API

In reply to Feifei Wang

Add internal API to separate direct rearm operations between Rx and Tx.

---
 drivers/net/i40e/i40e_ethdev.c          |  1 +
 drivers/net/i40e/i40e_ethdev.h          |  2 +
 drivers/net/i40e/i40e_rxtx.c            | 19 +++++++++
 drivers/net/i40e/i40e_rxtx.h            |  4 ++
 drivers/net/i40e/i40e_rxtx_vec_common.h | 54 +++++++++++++++++++++++++
 drivers/net/i40e/i40e_rxtx_vec_neon.c   | 42 +++++++++++++++++++
 6 files changed, 122 insertions(+)
drivers/net/i40e/i40e_ethdev.c +1 -0
index 7726a89d99..29c1ce2470 100644
@@ -497,6 +497,7 @@ static const struct eth_dev_ops i40e_eth_dev_ops = {
 	.flow_ops_get                 = i40e_dev_flow_ops_get,
 	.rxq_info_get                 = i40e_rxq_info_get,
 	.txq_info_get                 = i40e_txq_info_get,
+	.rxq_rearm_data_get           = i40e_rxq_rearm_data_get,
 	.rx_burst_mode_get            = i40e_rx_burst_mode_get,
 	.tx_burst_mode_get            = i40e_tx_burst_mode_get,
 	.timesync_enable              = i40e_timesync_enable,
drivers/net/i40e/i40e_ethdev.h +2 -0
index fe943a45ff..6a6a2a6d3c 100644
@@ -1352,6 +1352,8 @@ void i40e_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 	struct rte_eth_rxq_info *qinfo);
 void i40e_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 	struct rte_eth_txq_info *qinfo);
+void i40e_rxq_rearm_data_get(struct rte_eth_dev *dev, uint16_t queue_id,
+	struct rte_eth_rxq_rearm_data *rxq_rearm_data);
 int i40e_rx_burst_mode_get(struct rte_eth_dev *dev, uint16_t queue_id,
 			   struct rte_eth_burst_mode *mode);
 int i40e_tx_burst_mode_get(struct rte_eth_dev *dev, uint16_t queue_id,
drivers/net/i40e/i40e_rxtx.c +19 -0
index 788ffb51c2..d8d801acaf 100644
@@ -3197,6 +3197,19 @@ i40e_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 	qinfo->conf.offloads = txq->offloads;
 }
 
+void
+i40e_rxq_rearm_data_get(struct rte_eth_dev *dev, uint16_t queue_id,
+	struct rte_eth_rxq_rearm_data *rxq_rearm_data)
+{
+	struct i40e_rx_queue *rxq;
+
+	rxq = dev->data->rx_queues[queue_id];
+
+	rxq_rearm_data->rx_sw_ring = rxq->sw_ring;
+	rxq_rearm_data->rearm_start = &rxq->rxrearm_start;
+	rxq_rearm_data->rearm_nb = &rxq->rxrearm_nb;
+}
+
 #ifdef RTE_ARCH_X86
 static inline bool
 get_avx_supported(bool request_avx512)
@@ -3321,6 +3334,9 @@ i40e_set_rx_function(struct rte_eth_dev *dev)
 			PMD_INIT_LOG(DEBUG, "Using Vector Rx (port %d).",
 				     dev->data->port_id);
 			dev->rx_pkt_burst = i40e_recv_pkts_vec;
+#ifdef RTE_ARCH_ARM64
+			dev->rx_flush_descriptor = i40e_rx_flush_descriptor_vec;
+#endif
 		}
 #endif /* RTE_ARCH_X86 */
 	} else if (!dev->data->scattered_rx && ad->rx_bulk_alloc_allowed) {
@@ -3484,6 +3500,9 @@ i40e_set_tx_function(struct rte_eth_dev *dev)
 			PMD_INIT_LOG(DEBUG, "Using Vector Tx (port %d).",
 				     dev->data->port_id);
 			dev->tx_pkt_burst = i40e_xmit_pkts_vec;
+#ifdef RTE_ARCH_ARM64
+			dev->tx_fill_sw_ring = i40e_tx_fill_sw_ring;
+#endif
 #endif /* RTE_ARCH_X86 */
 		} else {
 			PMD_INIT_LOG(DEBUG, "Simple tx finally be used.");
drivers/net/i40e/i40e_rxtx.h +4 -0
index 5e6eecc501..8a29bd89df 100644
@@ -233,6 +233,10 @@ uint32_t i40e_dev_rx_queue_count(void *rx_queue);
 int i40e_dev_rx_descriptor_status(void *rx_queue, uint16_t offset);
 int i40e_dev_tx_descriptor_status(void *tx_queue, uint16_t offset);
 
+int i40e_tx_fill_sw_ring(void *tx_queue,
+		struct rte_eth_rxq_rearm_data *rxq_rearm_data);
+int i40e_rx_flush_descriptor_vec(void *rx_queue, uint16_t nb_rearm);
+
 uint16_t i40e_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,
 			    uint16_t nb_pkts);
 uint16_t i40e_recv_scattered_pkts_vec(void *rx_queue,
drivers/net/i40e/i40e_rxtx_vec_common.h +54 -0

60 lines not on this page · raw message

drivers/net/i40e/i40e_rxtx_vec_neon.c +42 -0

48 lines not on this page · raw message

Signature · 1 line
2.25.1
#64 lore raw

Re: [PATCH v3 2/3] net/i40e: enable direct rearm with separate API

04/01/2023 07:30, Feifei Wang пишет:

Quoted 76 lines

Add internal API to separate direct rearm operations between Rx and Tx.

---
  drivers/net/i40e/i40e_ethdev.c          |  1 +
  drivers/net/i40e/i40e_ethdev.h          |  2 +
  drivers/net/i40e/i40e_rxtx.c            | 19 +++++++++
  drivers/net/i40e/i40e_rxtx.h            |  4 ++
  drivers/net/i40e/i40e_rxtx_vec_common.h | 54 +++++++++++++++++++++++++
  drivers/net/i40e/i40e_rxtx_vec_neon.c   | 42 +++++++++++++++++++
  6 files changed, 122 insertions(+)
drivers/net/i40e/i40e_ethdev.c +1 -0
index 7726a89d99..29c1ce2470 100644
@@ -497,6 +497,7 @@ static const struct eth_dev_ops i40e_eth_dev_ops = {
  	.flow_ops_get                 = i40e_dev_flow_ops_get,
  	.rxq_info_get                 = i40e_rxq_info_get,
  	.txq_info_get                 = i40e_txq_info_get,
+	.rxq_rearm_data_get           = i40e_rxq_rearm_data_get,
  	.rx_burst_mode_get            = i40e_rx_burst_mode_get,
  	.tx_burst_mode_get            = i40e_tx_burst_mode_get,
  	.timesync_enable              = i40e_timesync_enable,
drivers/net/i40e/i40e_ethdev.h +2 -0
index fe943a45ff..6a6a2a6d3c 100644
@@ -1352,6 +1352,8 @@ void i40e_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
  	struct rte_eth_rxq_info *qinfo);
  void i40e_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
  	struct rte_eth_txq_info *qinfo);
+void i40e_rxq_rearm_data_get(struct rte_eth_dev *dev, uint16_t queue_id,
+	struct rte_eth_rxq_rearm_data *rxq_rearm_data);
  int i40e_rx_burst_mode_get(struct rte_eth_dev *dev, uint16_t queue_id,
  			   struct rte_eth_burst_mode *mode);
  int i40e_tx_burst_mode_get(struct rte_eth_dev *dev, uint16_t queue_id,
drivers/net/i40e/i40e_rxtx.c +16 -0
index 788ffb51c2..d8d801acaf 100644
@@ -3197,6 +3197,19 @@ i40e_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
  	qinfo->conf.offloads = txq->offloads;
  }
 
+void
+i40e_rxq_rearm_data_get(struct rte_eth_dev *dev, uint16_t queue_id,
+	struct rte_eth_rxq_rearm_data *rxq_rearm_data)
+{
+	struct i40e_rx_queue *rxq;
+
+	rxq = dev->data->rx_queues[queue_id];
+
+	rxq_rearm_data->rx_sw_ring = rxq->sw_ring;
+	rxq_rearm_data->rearm_start = &rxq->rxrearm_start;
+	rxq_rearm_data->rearm_nb = &rxq->rxrearm_nb;
+}
+
  #ifdef RTE_ARCH_X86
  static inline bool
  get_avx_supported(bool request_avx512)
@@ -3321,6 +3334,9 @@ i40e_set_rx_function(struct rte_eth_dev *dev)
  			PMD_INIT_LOG(DEBUG, "Using Vector Rx (port %d).",
  				     dev->data->port_id);
  			dev->rx_pkt_burst = i40e_recv_pkts_vec;
+#ifdef RTE_ARCH_ARM64
+			dev->rx_flush_descriptor = i40e_rx_flush_descriptor_vec;
+#endif
  		}
  #endif /* RTE_ARCH_X86 */
  	} else if (!dev->data->scattered_rx && ad->rx_bulk_alloc_allowed) {

drivers/net/i40e/i40e_rxtx.c i40e_set_tx_function(struct rte_eth_dev *dev) · line 3505

Quoted 6 lines
drivers/net/i40e/i40e_rxtx.c +3 -0
@@ -3484,6 +3500,9 @@ i40e_set_tx_function(struct rte_eth_dev *dev)
  			PMD_INIT_LOG(DEBUG, "Using Vector Tx (port %d).",
  				     dev->data->port_id);
  			dev->tx_pkt_burst = i40e_xmit_pkts_vec;
+#ifdef RTE_ARCH_ARM64
+			dev->tx_fill_sw_ring = i40e_tx_fill_sw_ring;
+#endif

As I can see tx_fill_sw_ring() is non ARM specific, any reason to guard it with #ifdef ARM? Actually same ask for rx_flush_descriptor() - can we have generic version too?

Quoted 136 lines
drivers/net/i40e/i40e_rxtx.c +0 -0
  #endif /* RTE_ARCH_X86 */
  		} else {
  			PMD_INIT_LOG(DEBUG, "Simple tx finally be used.");
drivers/net/i40e/i40e_rxtx.h +4 -0
index 5e6eecc501..8a29bd89df 100644
@@ -233,6 +233,10 @@ uint32_t i40e_dev_rx_queue_count(void *rx_queue);
  int i40e_dev_rx_descriptor_status(void *rx_queue, uint16_t offset);
  int i40e_dev_tx_descriptor_status(void *tx_queue, uint16_t offset);
 
+int i40e_tx_fill_sw_ring(void *tx_queue,
+		struct rte_eth_rxq_rearm_data *rxq_rearm_data);
+int i40e_rx_flush_descriptor_vec(void *rx_queue, uint16_t nb_rearm);
+
  uint16_t i40e_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,
  			    uint16_t nb_pkts);
  uint16_t i40e_recv_scattered_pkts_vec(void *rx_queue,
drivers/net/i40e/i40e_rxtx_vec_common.h +54 -0
index fe1a6ec75e..eb96301a43 100644
@@ -146,6 +146,60 @@ i40e_tx_free_bufs(struct i40e_tx_queue *txq)
  	return txq->tx_rs_thresh;
  }
 
+int
+i40e_tx_fill_sw_ring(void *tx_queue,
+		struct rte_eth_rxq_rearm_data *rxq_rearm_data)
+{
+	struct i40e_tx_queue *txq = tx_queue;
+	struct i40e_tx_entry *txep;
+	void **rxep;
+	struct rte_mbuf *m;
+	int i, n;
+	int nb_rearm = 0;
+
+	if (*rxq_rearm_data->rearm_nb < txq->tx_rs_thresh ||
+			txq->nb_tx_free > txq->tx_free_thresh)
+		return 0;
+
+	/* check DD bits on threshold descriptor */
+	if ((txq->tx_ring[txq->tx_next_dd].cmd_type_offset_bsz &
+			rte_cpu_to_le_64(I40E_TXD_QW1_DTYPE_MASK)) !=
+			rte_cpu_to_le_64(I40E_TX_DESC_DTYPE_DESC_DONE))
+		return 0;
+
+	n = txq->tx_rs_thresh;
+
+	/* first buffer to free from S/W ring is at index
+	 * tx_next_dd - (tx_rs_thresh-1)
+	 */
+	txep = &txq->sw_ring[txq->tx_next_dd - (n - 1)];
+	rxep = rxq_rearm_data->rx_sw_ring;
+	rxep += *rxq_rearm_data->rearm_start;
+
+	if (txq->offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) {
+		/* directly put mbufs from Tx to Rx */
+		for (i = 0; i < n; i++, rxep++, txep++)
+			*rxep = txep[0].mbuf;
+	} else {
+		for (i = 0; i < n; i++, rxep++) {
+			m = rte_pktmbuf_prefree_seg(txep[i].mbuf);
+			if (m != NULL) {
+				*rxep = m;
+				nb_rearm++;
+			}
+		}
+		n = nb_rearm;
+	}
+
+	/* update counters for Tx */
+	txq->nb_tx_free = (uint16_t)(txq->nb_tx_free + txq->tx_rs_thresh);
+	txq->tx_next_dd = (uint16_t)(txq->tx_next_dd + txq->tx_rs_thresh);
+	if (txq->tx_next_dd >= txq->nb_tx_desc)
+		txq->tx_next_dd = (uint16_t)(txq->tx_rs_thresh - 1);
+
+	return n;
+}
+
  static __rte_always_inline void
  tx_backlog_entry(struct i40e_tx_entry *txep,
  		 struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
drivers/net/i40e/i40e_rxtx_vec_neon.c +42 -0
index 12e6f1cbcb..1509d3223b 100644
@@ -739,6 +739,48 @@ i40e_xmit_fixed_burst_vec(void *__rte_restrict tx_queue,
  	return nb_pkts;
  }
 
+int
+i40e_rx_flush_descriptor_vec(void *rx_queue, uint16_t nb_rearm)
+{
+	struct i40e_rx_queue *rxq = rx_queue;
+	struct i40e_rx_entry *rxep;
+	volatile union i40e_rx_desc *rxdp;
+	uint16_t rx_id;
+	uint64x2_t dma_addr;
+	uint64_t paddr;
+	uint16_t i;
+
+	rxdp = rxq->rx_ring + rxq->rxrearm_start;
+	rxep = &rxq->sw_ring[rxq->rxrearm_start];
+
+	for (i = 0; i < nb_rearm; i++) {
+		/* Initialize rxdp descs */
+		paddr = (rxep[i].mbuf)->buf_iova + RTE_PKTMBUF_HEADROOM;
+		dma_addr = vdupq_n_u64(paddr);
+		/* flush desc with pa dma_addr */
+		vst1q_u64((uint64_t *)&rxdp++->read, dma_addr);
+	}
+
+	/* Update the descriptor initializer index */
+	rxq->rxrearm_start += nb_rearm;
+	rx_id = rxq->rxrearm_start - 1;
+
+	if (unlikely(rxq->rxrearm_start >= rxq->nb_rx_desc)) {
+		rxq->rxrearm_start = rxq->rxrearm_start - rxq->nb_rx_desc;
+		if (!rxq->rxrearm_start)
+			rx_id = rxq->nb_rx_desc - 1;
+		else
+			rx_id = rxq->rxrearm_start - 1;
+	}
+	rxq->rxrearm_nb -= nb_rearm;
+
+	rte_io_wmb();
+	/* Update the tail pointer on the NIC */
+	I40E_PCI_REG_WRITE_RELAXED(rxq->qrx_tail, rx_id);
+
+	return 0;
+}
+
  void __rte_cold
  i40e_rx_queue_release_mbufs_vec(struct i40e_rx_queue *rxq)
  {
#65 lore raw

回复: [PATCH v3 2/3] net/i40e: enable direct rearm with separate API

Quoted 88 lines

-----邮件原件-----
发件人: Konstantin Ananyev <[email protected]> 发送时间: Thursday, February 2, 2023 10:38 PM
收件人: Feifei Wang <[email protected]>; Yuying Zhang <[email protected]>; Beilei Xing <[email protected]>; Ruifeng Wang <[email protected]>
抄送: [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>
主题: Re: [PATCH v3 2/3] net/i40e: enable direct rearm with separate API

04/01/2023 07:30, Feifei Wang пишет:

Quoted 7 lines

Add internal API to separate direct rearm operations between Rx and Tx.

Quoted 6 lines
---
  drivers/net/i40e/i40e_ethdev.c          |  1 +
  drivers/net/i40e/i40e_ethdev.h          |  2 +
  drivers/net/i40e/i40e_rxtx.c            | 19 +++++++++
  drivers/net/i40e/i40e_rxtx.h            |  4 ++
  drivers/net/i40e/i40e_rxtx_vec_common.h | 54

+++++++++++++++++++++++++

drivers/net/i40e/i40e_ethdev.c static const struct eth_dev_ops i40e_eth_dev_ops

Quoted 8 lines
  drivers/net/i40e/i40e_rxtx_vec_neon.c   | 42 +++++++++++++++++++
  6 files changed, 122 insertions(+)
(quoted diff) +0 -0

b/drivers/net/i40e/i40e_ethdev.c index 7726a89d99..29c1ce2470 100644

drivers/net/i40e/i40e_ethdev.c +0 -0
@@ -497,6 +497,7 @@ static const struct eth_dev_ops i40e_eth_dev_ops

= {

drivers/net/i40e/i40e_ethdev.h void i40e_rxq_info_get(struct rte_eth_dev *dev,

Quoted 12 lines
drivers/net/i40e/i40e_ethdev.c +1 -0
  	.flow_ops_get                 = i40e_dev_flow_ops_get,
  	.rxq_info_get                 = i40e_rxq_info_get,
  	.txq_info_get                 = i40e_txq_info_get,
+	.rxq_rearm_data_get           = i40e_rxq_rearm_data_get,
  	.rx_burst_mode_get            = i40e_rx_burst_mode_get,
  	.tx_burst_mode_get            = i40e_tx_burst_mode_get,
  	.timesync_enable              = i40e_timesync_enable,
(quoted diff) +0 -0

b/drivers/net/i40e/i40e_ethdev.h index fe943a45ff..6a6a2a6d3c 100644

drivers/net/i40e/i40e_ethdev.h +0 -0
@@ -1352,6 +1352,8 @@ void i40e_rxq_info_get(struct rte_eth_dev *dev,

uint16_t queue_id,

drivers/net/i40e/i40e_ethdev.h void i40e_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,

Quoted 4 lines
drivers/net/i40e/i40e_ethdev.h +1 -0
  	struct rte_eth_rxq_info *qinfo);
  void i40e_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
  	struct rte_eth_txq_info *qinfo);
+void i40e_rxq_rearm_data_get(struct rte_eth_dev *dev, uint16_t

queue_id,

drivers/net/i40e/i40e_rxtx.c i40e_txq_info_get(struct rte_eth_dev *dev,

Quoted 9 lines
drivers/net/i40e/i40e_ethdev.h +1 -0
+	struct rte_eth_rxq_rearm_data *rxq_rearm_data);
  int i40e_rx_burst_mode_get(struct rte_eth_dev *dev, uint16_t queue_id,
  			   struct rte_eth_burst_mode *mode);
  int i40e_tx_burst_mode_get(struct rte_eth_dev *dev, uint16_t

queue_id, diff --git a/drivers/net/i40e/i40e_rxtx.c b/drivers/net/i40e/i40e_rxtx.c index 788ffb51c2..d8d801acaf 100644

drivers/net/i40e/i40e_rxtx.c +0 -0
@@ -3197,6 +3197,19 @@ i40e_txq_info_get(struct rte_eth_dev *dev,

uint16_t queue_id,

Quoted 21 lines
drivers/net/i40e/i40e_rxtx.c +11 -0
  	qinfo->conf.offloads = txq->offloads;
  }
 
+void
+i40e_rxq_rearm_data_get(struct rte_eth_dev *dev, uint16_t queue_id,
+	struct rte_eth_rxq_rearm_data *rxq_rearm_data) {
+	struct i40e_rx_queue *rxq;
+
+	rxq = dev->data->rx_queues[queue_id];
+
+	rxq_rearm_data->rx_sw_ring = rxq->sw_ring;
+	rxq_rearm_data->rearm_start = &rxq->rxrearm_start;
+	rxq_rearm_data->rearm_nb = &rxq->rxrearm_nb; }
+
  #ifdef RTE_ARCH_X86
  static inline bool
  get_avx_supported(bool request_avx512) @@ -3321,6 +3334,9 @@

i40e_set_rx_function(struct rte_eth_dev *dev)
PMD_INIT_LOG(DEBUG, "Using Vector Rx (port %d).",
dev->data->port_id);
dev->rx_pkt_burst = i40e_recv_pkts_vec;

drivers/net/i40e/i40e_rxtx.c

Quoted 2 lines
drivers/net/i40e/i40e_rxtx.c +2 -0
+#ifdef RTE_ARCH_ARM64
+			dev->rx_flush_descriptor =

i40e_rx_flush_descriptor_vec; #endif

Quoted 13 lines

drivers/net/i40e/i40e_rxtx.c i40e_set_tx_function(struct rte_eth_dev *dev) · line 3504

Quoted 9 lines
  		}
  #endif /* RTE_ARCH_X86 */
  	} else if (!dev->data->scattered_rx && ad->rx_bulk_alloc_allowed) {
drivers/net/i40e/i40e_rxtx.c +2 -0
@@ -3484,6 +3500,9 @@ i40e_set_tx_function(struct rte_eth_dev *dev)
  			PMD_INIT_LOG(DEBUG, "Using Vector Tx (port %d).",
  				     dev->data->port_id);
  			dev->tx_pkt_burst = i40e_xmit_pkts_vec;
+#ifdef RTE_ARCH_ARM64
+			dev->tx_fill_sw_ring = i40e_tx_fill_sw_ring; #endif

As I can see tx_fill_sw_ring() is non ARM specific, any reason to guard it with #ifdef ARM?
Actually same ask for rx_flush_descriptor() - can we have generic version too?

Here we consider direct-rearm not enable in other architecture. Agree with that we need to have generic version to avoid this, I will update in the next version.

Quoted 142 lines

drivers/net/i40e/i40e_rxtx.c i40e_set_tx_function(struct rte_eth_dev *dev) · line 3508

Quoted 4 lines
drivers/net/i40e/i40e_rxtx.c +0 -0
  #endif /* RTE_ARCH_X86 */
  		} else {
  			PMD_INIT_LOG(DEBUG, "Simple tx finally be used.");

diff --git

drivers/net/i40e/i40e_rxtx.h uint32_t i40e_dev_rx_queue_count(void

Quoted 5 lines

a/drivers/net/i40e/i40e_rxtx.h b/drivers/net/i40e/i40e_rxtx.h index 5e6eecc501..8a29bd89df 100644

drivers/net/i40e/i40e_rxtx.h +0 -0
@@ -233,6 +233,10 @@ uint32_t i40e_dev_rx_queue_count(void

*rx_queue);

Quoted 27 lines
drivers/net/i40e/i40e_rxtx.h +4 -0
  int i40e_dev_rx_descriptor_status(void *rx_queue, uint16_t offset);
  int i40e_dev_tx_descriptor_status(void *tx_queue, uint16_t offset);
 
+int i40e_tx_fill_sw_ring(void *tx_queue,
+		struct rte_eth_rxq_rearm_data *rxq_rearm_data); int
+i40e_rx_flush_descriptor_vec(void *rx_queue, uint16_t nb_rearm);
+
  uint16_t i40e_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,
  			    uint16_t nb_pkts);
  uint16_t i40e_recv_scattered_pkts_vec(void *rx_queue, diff --git

a/drivers/net/i40e/i40e_rxtx_vec_common.h
b/drivers/net/i40e/i40e_rxtx_vec_common.h
index fe1a6ec75e..eb96301a43 100644

drivers/net/i40e/i40e_rxtx_vec_common.h +8 -0
@@ -146,6 +146,60 @@ i40e_tx_free_bufs(struct i40e_tx_queue *txq)
  	return txq->tx_rs_thresh;
  }
 
+int
+i40e_tx_fill_sw_ring(void *tx_queue,
+		struct rte_eth_rxq_rearm_data *rxq_rearm_data) {
+	struct i40e_tx_queue *txq = tx_queue;
+	struct i40e_tx_entry *txep;
+	void **rxep;
+	struct rte_mbuf *m;
+	int i, n;

drivers/net/i40e/i40e_rxtx_vec_common.h i40e_tx_free_bufs(struct i40e_tx_queue *txq) · line 166

Quoted 10 lines
drivers/net/i40e/i40e_rxtx_vec_common.h +10 -0
+	int nb_rearm = 0;
+
+	if (*rxq_rearm_data->rearm_nb < txq->tx_rs_thresh ||
+			txq->nb_tx_free > txq->tx_free_thresh)
+		return 0;
+
+	/* check DD bits on threshold descriptor */
+	if ((txq->tx_ring[txq->tx_next_dd].cmd_type_offset_bsz &
+			rte_cpu_to_le_64(I40E_TXD_QW1_DTYPE_MASK)) !=
+
	rte_cpu_to_le_64(I40E_TX_DESC_DTYPE_DESC_DONE))

drivers/net/i40e/i40e_rxtx_vec_neon.c i40e_xmit_fixed_burst_vec(void *__rte_restrict

Quoted 44 lines
drivers/net/i40e/i40e_rxtx_vec_common.h +35 -0
+		return 0;
+
+	n = txq->tx_rs_thresh;
+
+	/* first buffer to free from S/W ring is at index
+	 * tx_next_dd - (tx_rs_thresh-1)
+	 */
+	txep = &txq->sw_ring[txq->tx_next_dd - (n - 1)];
+	rxep = rxq_rearm_data->rx_sw_ring;
+	rxep += *rxq_rearm_data->rearm_start;
+
+	if (txq->offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) {
+		/* directly put mbufs from Tx to Rx */
+		for (i = 0; i < n; i++, rxep++, txep++)
+			*rxep = txep[0].mbuf;
+	} else {
+		for (i = 0; i < n; i++, rxep++) {
+			m = rte_pktmbuf_prefree_seg(txep[i].mbuf);
+			if (m != NULL) {
+				*rxep = m;
+				nb_rearm++;
+			}
+		}
+		n = nb_rearm;
+	}
+
+	/* update counters for Tx */
+	txq->nb_tx_free = (uint16_t)(txq->nb_tx_free + txq->tx_rs_thresh);
+	txq->tx_next_dd = (uint16_t)(txq->tx_next_dd + txq->tx_rs_thresh);
+	if (txq->tx_next_dd >= txq->nb_tx_desc)
+		txq->tx_next_dd = (uint16_t)(txq->tx_rs_thresh - 1);
+
+	return n;
+}
+
  static __rte_always_inline void
  tx_backlog_entry(struct i40e_tx_entry *txep,
  		 struct rte_mbuf **tx_pkts, uint16_t nb_pkts) diff --git

a/drivers/net/i40e/i40e_rxtx_vec_neon.c
b/drivers/net/i40e/i40e_rxtx_vec_neon.c
index 12e6f1cbcb..1509d3223b 100644

drivers/net/i40e/i40e_rxtx_vec_neon.c +0 -0
@@ -739,6 +739,48 @@ i40e_xmit_fixed_burst_vec(void *__rte_restrict

tx_queue,

Quoted 9 lines
drivers/net/i40e/i40e_rxtx_vec_neon.c +6 -0
  	return nb_pkts;
  }
 
+int
+i40e_rx_flush_descriptor_vec(void *rx_queue, uint16_t nb_rearm) {
+	struct i40e_rx_queue *rxq = rx_queue;
+	struct i40e_rx_entry *rxep;
+	volatile union i40e_rx_desc *rxdp;
+	uint16_t rx_id;

drivers/net/i40e/i40e_rxtx_vec_neon.c i40e_xmit_fixed_burst_vec(void *__rte_restrict tx_queue,

Quoted 10 lines
drivers/net/i40e/i40e_rxtx_vec_neon.c +10 -0
+	uint64x2_t dma_addr;
+	uint64_t paddr;
+	uint16_t i;
+
+	rxdp = rxq->rx_ring + rxq->rxrearm_start;
+	rxep = &rxq->sw_ring[rxq->rxrearm_start];
+
+	for (i = 0; i < nb_rearm; i++) {
+		/* Initialize rxdp descs */
+		paddr = (rxep[i].mbuf)->buf_iova +

RTE_PKTMBUF_HEADROOM;

Quoted 28 lines
drivers/net/i40e/i40e_rxtx_vec_neon.c +25 -0
+		dma_addr = vdupq_n_u64(paddr);
+		/* flush desc with pa dma_addr */
+		vst1q_u64((uint64_t *)&rxdp++->read, dma_addr);
+	}
+
+	/* Update the descriptor initializer index */
+	rxq->rxrearm_start += nb_rearm;
+	rx_id = rxq->rxrearm_start - 1;
+
+	if (unlikely(rxq->rxrearm_start >= rxq->nb_rx_desc)) {
+		rxq->rxrearm_start = rxq->rxrearm_start - rxq->nb_rx_desc;
+		if (!rxq->rxrearm_start)
+			rx_id = rxq->nb_rx_desc - 1;
+		else
+			rx_id = rxq->rxrearm_start - 1;
+	}
+	rxq->rxrearm_nb -= nb_rearm;
+
+	rte_io_wmb();
+	/* Update the tail pointer on the NIC */
+	I40E_PCI_REG_WRITE_RELAXED(rxq->qrx_tail, rx_id);
+
+	return 0;
+}
+
  void __rte_cold
  i40e_rx_queue_release_mbufs_vec(struct i40e_rx_queue *rxq)
  {
#66 lore raw

RE: [PATCH v3 2/3] net/i40e: enable direct rearm with separate API

Quoted 31 lines
Quoted 31 lines
Quoted 10 lines
drivers/net/i40e/i40e_rxtx_vec_common.h +8 -0
+int
+i40e_tx_fill_sw_ring(void *tx_queue,
+		struct rte_eth_rxq_rearm_data *rxq_rearm_data) {
+	struct i40e_tx_queue *txq = tx_queue;
+	struct i40e_tx_entry *txep;
+	void **rxep;
+	struct rte_mbuf *m;
+	int i, n;

drivers/net/i40e/i40e_rxtx_vec_common.h i40e_tx_free_bufs(struct i40e_tx_queue *txq) · line 169

Quoted 10 lines
drivers/net/i40e/i40e_rxtx_vec_common.h +10 -0
+	int nb_rearm = 0;
+
+	if (*rxq_rearm_data->rearm_nb < txq->tx_rs_thresh ||
+			txq->nb_tx_free > txq->tx_free_thresh)
+		return 0;
+
+	/* check DD bits on threshold descriptor */
+	if ((txq->tx_ring[txq->tx_next_dd].cmd_type_offset_bsz &
+			rte_cpu_to_le_64(I40E_TXD_QW1_DTYPE_MASK)) !=
+
	rte_cpu_to_le_64(I40E_TX_DESC_DTYPE_DESC_DONE))
Quoted 10 lines
drivers/net/i40e/i40e_rxtx_vec_common.h +10 -0
+		return 0;
+
+	n = txq->tx_rs_thresh;
+
+	/* first buffer to free from S/W ring is at index
+	 * tx_next_dd - (tx_rs_thresh-1)
+	 */
+	txep = &txq->sw_ring[txq->tx_next_dd - (n - 1)];
+	rxep = rxq_rearm_data->rx_sw_ring;
+	rxep += *rxq_rearm_data->rearm_start;
Quoted 8 lines
Quoted 8 lines

drivers/net/i40e/i40e_rxtx_vec_common.h i40e_tx_free_bufs(struct i40e_tx_queue *txq) · line 185

Quoted 8 lines
drivers/net/i40e/i40e_rxtx_vec_common.h +8 -0
+
+	if (txq->offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) {
+		/* directly put mbufs from Tx to Rx */
+		for (i = 0; i < n; i++, rxep++, txep++)
+			*rxep = txep[0].mbuf;
+	} else {
+		for (i = 0; i < n; i++, rxep++) {
+			m = rte_pktmbuf_prefree_seg(txep[i].mbuf);

One thing I forgot to ask:
What would happen if this mbuf belongs to different mempool (not one that we specify at rx_queue_setup())?
Do we need to check it here?
Or would it be upper layer constraint?
Or...?

Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
drivers/net/i40e/i40e_rxtx_vec_common.h +17 -0
+			if (m != NULL) {
+				*rxep = m;
+				nb_rearm++;
+			}
+		}
+		n = nb_rearm;
+	}
+
+	/* update counters for Tx */
+	txq->nb_tx_free = (uint16_t)(txq->nb_tx_free + txq->tx_rs_thresh);
+	txq->tx_next_dd = (uint16_t)(txq->tx_next_dd + txq->tx_rs_thresh);
+	if (txq->tx_next_dd >= txq->nb_tx_desc)
+		txq->tx_next_dd = (uint16_t)(txq->tx_rs_thresh - 1);
+
+	return n;
+}
+
#67 lore raw

回复: [PATCH v3 2/3] net/i40e: enable direct rearm with separate API

Quoted 51 lines

-----邮件原件-----
发件人: Konstantin Ananyev <[email protected]> 发送时间: Tuesday, February 28, 2023 3:36 AM
收件人: Feifei Wang <[email protected]>; Konstantin Ananyev <[email protected]>; Yuying Zhang <[email protected]>; Beilei Xing <[email protected]>; Ruifeng Wang <[email protected]>
抄送: [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
主题: RE: [PATCH v3 2/3] net/i40e: enable direct rearm with separate API

Quoted 36 lines
Quoted 36 lines
Quoted 8 lines
drivers/net/i40e/i40e_rxtx_vec_common.h +8 -0
+int
+i40e_tx_fill_sw_ring(void *tx_queue,
+		struct rte_eth_rxq_rearm_data *rxq_rearm_data) {
+	struct i40e_tx_queue *txq = tx_queue;
+	struct i40e_tx_entry *txep;
+	void **rxep;
+	struct rte_mbuf *m;
+	int i, n;

drivers/net/i40e/i40e_rxtx_vec_common.h i40e_tx_free_bufs(struct i40e_tx_queue *txq) · line 169

Quoted 10 lines
drivers/net/i40e/i40e_rxtx_vec_common.h +10 -0
+	int nb_rearm = 0;
+
+	if (*rxq_rearm_data->rearm_nb < txq->tx_rs_thresh ||
+			txq->nb_tx_free > txq->tx_free_thresh)
+		return 0;
+
+	/* check DD bits on threshold descriptor */
+	if ((txq->tx_ring[txq->tx_next_dd].cmd_type_offset_bsz &
+			rte_cpu_to_le_64(I40E_TXD_QW1_DTYPE_MASK)) !=
+
	rte_cpu_to_le_64(I40E_TX_DESC_DTYPE_DESC_DONE))
Quoted 17 lines
drivers/net/i40e/i40e_rxtx_vec_common.h +17 -0
+		return 0;
+
+	n = txq->tx_rs_thresh;
+
+	/* first buffer to free from S/W ring is at index
+	 * tx_next_dd - (tx_rs_thresh-1)
+	 */
+	txep = &txq->sw_ring[txq->tx_next_dd - (n - 1)];
+	rxep = rxq_rearm_data->rx_sw_ring;
+	rxep += *rxq_rearm_data->rearm_start;
+
+	if (txq->offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) {
+		/* directly put mbufs from Tx to Rx */
+		for (i = 0; i < n; i++, rxep++, txep++)
+			*rxep = txep[0].mbuf;
+	} else {
+		for (i = 0; i < n; i++, rxep++) {
Quoted 9 lines
Quoted 1 line
Quoted 1 line

drivers/net/i40e/i40e_rxtx_vec_common.h i40e_tx_free_bufs(struct i40e_tx_queue *txq) · line 185

Quoted 1 line
drivers/net/i40e/i40e_rxtx_vec_common.h +1 -0
+			m = rte_pktmbuf_prefree_seg(txep[i].mbuf);

One thing I forgot to ask:
What would happen if this mbuf belongs to different mempool (not one that we specify at rx_queue_setup())?
Do we need to check it here?
Or would it be upper layer constraint?
Or...?

First, 'different mempool' is valid for no FAST_FREE path in tx_free_buffers.

If buffers belong to different mempool, we can have an example here:
Buffer 1 from mempool 1, its recycle path is:
-----------------------------------------------------------------------------------------
1. queue_setup: rearm from mempool 1 into Rx sw-ring
2. rte_eth_Rx_burst: used by user app (Rx)
3. rte_eth_Tx_burst: mount on Tx sw-ring
4. rte_eth_direct_rearm: free into Rx sw-ring:
or
tx_free_buffers: free into mempool 1 (no fast_free path)
-----------------------------------------------------------------------------------------

Buffer 2 from mempool 2, its recycle path is:
-----------------------------------------------------------------------------------------
1. queue_setup: rearm from mempool 2 into Rx sw-ring
2. rte_eth_Rx_burst: used by user app (Rx)
3. rte_eth_Tx_burst: mount on Tx sw-ring
4. rte_eth_direct_rearm: free into Rx sw-ring
or
tx_free_buffers: free into mempool 2 (no fast_free_path)
-----------------------------------------------------------------------------------------

Thus, buffers from Tx different mempools are the same for Rx. The difference point is that they will be freed into different mempool if the thread uses generic free buffers. I think this cannot affect direct-rearm mode, and we do not need to check this.

Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
drivers/net/i40e/i40e_rxtx_vec_common.h +17 -0
+			if (m != NULL) {
+				*rxep = m;
+				nb_rearm++;
+			}
+		}
+		n = nb_rearm;
+	}
+
+	/* update counters for Tx */
+	txq->nb_tx_free = (uint16_t)(txq->nb_tx_free + txq->tx_rs_thresh);
+	txq->tx_next_dd = (uint16_t)(txq->tx_next_dd + txq->tx_rs_thresh);
+	if (txq->tx_next_dd >= txq->nb_tx_desc)
+		txq->tx_next_dd = (uint16_t)(txq->tx_rs_thresh - 1);
+
+	return n;
+}
+
#68 lore raw

RE: [PATCH v3 2/3] net/i40e: enable direct rearm with separate API

Quoted 84 lines
Quoted 51 lines

-----邮件原件-----
发件人: Konstantin Ananyev <[email protected]> 发送时间: Tuesday, February 28, 2023 3:36 AM
收件人: Feifei Wang <[email protected]>; Konstantin Ananyev <[email protected]>; Yuying Zhang <[email protected]>; Beilei Xing <[email protected]>; Ruifeng Wang <[email protected]>
抄送: [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
主题: RE: [PATCH v3 2/3] net/i40e: enable direct rearm with separate API

Quoted 36 lines
Quoted 36 lines
Quoted 8 lines
drivers/net/i40e/i40e_rxtx_vec_common.h +8 -0
+int
+i40e_tx_fill_sw_ring(void *tx_queue,
+		struct rte_eth_rxq_rearm_data *rxq_rearm_data) {
+	struct i40e_tx_queue *txq = tx_queue;
+	struct i40e_tx_entry *txep;
+	void **rxep;
+	struct rte_mbuf *m;
+	int i, n;

drivers/net/i40e/i40e_rxtx_vec_common.h i40e_tx_free_bufs(struct i40e_tx_queue *txq) · line 169

Quoted 10 lines
drivers/net/i40e/i40e_rxtx_vec_common.h +10 -0
+	int nb_rearm = 0;
+
+	if (*rxq_rearm_data->rearm_nb < txq->tx_rs_thresh ||
+			txq->nb_tx_free > txq->tx_free_thresh)
+		return 0;
+
+	/* check DD bits on threshold descriptor */
+	if ((txq->tx_ring[txq->tx_next_dd].cmd_type_offset_bsz &
+			rte_cpu_to_le_64(I40E_TXD_QW1_DTYPE_MASK)) !=
+
	rte_cpu_to_le_64(I40E_TX_DESC_DTYPE_DESC_DONE))
Quoted 17 lines
drivers/net/i40e/i40e_rxtx_vec_common.h +17 -0
+		return 0;
+
+	n = txq->tx_rs_thresh;
+
+	/* first buffer to free from S/W ring is at index
+	 * tx_next_dd - (tx_rs_thresh-1)
+	 */
+	txep = &txq->sw_ring[txq->tx_next_dd - (n - 1)];
+	rxep = rxq_rearm_data->rx_sw_ring;
+	rxep += *rxq_rearm_data->rearm_start;
+
+	if (txq->offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) {
+		/* directly put mbufs from Tx to Rx */
+		for (i = 0; i < n; i++, rxep++, txep++)
+			*rxep = txep[0].mbuf;
+	} else {
+		for (i = 0; i < n; i++, rxep++) {
Quoted 9 lines
Quoted 1 line
Quoted 1 line

drivers/net/i40e/i40e_rxtx_vec_common.h i40e_tx_free_bufs(struct i40e_tx_queue *txq) · line 185

Quoted 1 line
drivers/net/i40e/i40e_rxtx_vec_common.h +1 -0
+			m = rte_pktmbuf_prefree_seg(txep[i].mbuf);

One thing I forgot to ask:
What would happen if this mbuf belongs to different mempool (not one that we specify at rx_queue_setup())?
Do we need to check it here?
Or would it be upper layer constraint?
Or...?

First, 'different mempool' is valid for no FAST_FREE path in tx_free_buffers.

If buffers belong to different mempool, we can have an example here:
Buffer 1 from mempool 1, its recycle path is:
-----------------------------------------------------------------------------------------
1. queue_setup: rearm from mempool 1 into Rx sw-ring
2. rte_eth_Rx_burst: used by user app (Rx)
3. rte_eth_Tx_burst: mount on Tx sw-ring
4. rte_eth_direct_rearm: free into Rx sw-ring:
or
tx_free_buffers: free into mempool 1 (no fast_free path)
-----------------------------------------------------------------------------------------

Buffer 2 from mempool 2, its recycle path is:
-----------------------------------------------------------------------------------------
1. queue_setup: rearm from mempool 2 into Rx sw-ring
2. rte_eth_Rx_burst: used by user app (Rx)
3. rte_eth_Tx_burst: mount on Tx sw-ring
4. rte_eth_direct_rearm: free into Rx sw-ring
or
tx_free_buffers: free into mempool 2 (no fast_free_path)
-----------------------------------------------------------------------------------------

Quoted 3 lines

Thus, buffers from Tx different mempools are the same for Rx. The difference point is that they will be freed into different mempool if the thread uses generic free buffers. I think this cannot affect direct-rearm mode, and we do not need to check this.

I understand that it should work even with multiple mempools.
What I am trying to say - user may not want to use mbufs from particular mempool for RX (while it is still ok to use it for TX).
Let say user can have a separate mempool with small data-buffers (less then normal MTU) to send some 'special' paclets, or even use this memppol with small buffers for zero-copy updating of packet L2/L3 headers, etc.
Or it could be some 'special' user provided mempool.
That's why I wonder should we allow only mbufs from mempool that is assigned to that RX queue.

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
drivers/net/i40e/i40e_rxtx_vec_common.h +17 -0
+			if (m != NULL) {
+				*rxep = m;
+				nb_rearm++;
+			}
+		}
+		n = nb_rearm;
+	}
+
+	/* update counters for Tx */
+	txq->nb_tx_free = (uint16_t)(txq->nb_tx_free + txq->tx_rs_thresh);
+	txq->tx_next_dd = (uint16_t)(txq->tx_next_dd + txq->tx_rs_thresh);
+	if (txq->tx_next_dd >= txq->nb_tx_desc)
+		txq->tx_next_dd = (uint16_t)(txq->tx_rs_thresh - 1);
+
+	return n;
+}
+
#69 lore raw

回复: [PATCH v3 2/3] net/i40e: enable direct rearm with separate API

Quoted 108 lines

-----邮件原件-----
发件人: Konstantin Ananyev <[email protected]> 发送时间: Tuesday, March 7, 2023 7:01 PM
收件人: Feifei Wang <[email protected]>; Konstantin Ananyev <[email protected]>; Yuying Zhang <[email protected]>; Beilei Xing <[email protected]>; Ruifeng Wang <[email protected]>
抄送: [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>; nd <[email protected]> 主题: RE: [PATCH v3 2/3] net/i40e: enable direct rearm with separate API

Quoted 23 lines
Quoted 23 lines

-----邮件原件-----
发件人: Konstantin Ananyev <[email protected]> 发送时间: Tuesday, February 28, 2023 3:36 AM
收件人: Feifei Wang <[email protected]>; Konstantin Ananyev <[email protected]>; Yuying Zhang <[email protected]>; Beilei Xing <[email protected]>; Ruifeng Wang <[email protected]>
抄送: [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
主题: RE: [PATCH v3 2/3] net/i40e: enable direct rearm with separate API

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
drivers/net/i40e/i40e_rxtx_vec_common.h +9 -0
+int
+i40e_tx_fill_sw_ring(void *tx_queue,
+		struct rte_eth_rxq_rearm_data *rxq_rearm_data) {
+	struct i40e_tx_queue *txq = tx_queue;
+	struct i40e_tx_entry *txep;
+	void **rxep;
+	struct rte_mbuf *m;
+	int i, n;
+	int nb_rearm = 0;
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

drivers/net/i40e/i40e_rxtx_vec_common.h i40e_tx_free_bufs(struct i40e_tx_queue *txq) · line 169

Quoted 8 lines
drivers/net/i40e/i40e_rxtx_vec_common.h +8 -0
+
+	if (*rxq_rearm_data->rearm_nb < txq->tx_rs_thresh ||
+			txq->nb_tx_free > txq->tx_free_thresh)
+		return 0;
+
+	/* check DD bits on threshold descriptor */
+	if ((txq->tx_ring[txq->tx_next_dd].cmd_type_offset_bsz &
+
	rte_cpu_to_le_64(I40E_TXD_QW1_DTYPE_MASK)) !=
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 1 line

+

	rte_cpu_to_le_64(I40E_TX_DESC_DTYPE_DESC_DONE))

drivers/net/i40e/i40e_rxtx_vec_common.h i40e_tx_free_bufs(struct i40e_tx_queue *txq)

Quoted 12 lines
drivers/net/i40e/i40e_rxtx_vec_common.h +12 -0
+		return 0;
+
+	n = txq->tx_rs_thresh;
+
+	/* first buffer to free from S/W ring is at index
+	 * tx_next_dd - (tx_rs_thresh-1)
+	 */
+	txep = &txq->sw_ring[txq->tx_next_dd - (n - 1)];
+	rxep = rxq_rearm_data->rx_sw_ring;
+	rxep += *rxq_rearm_data->rearm_start;
+
+	if (txq->offloads &

RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) {

Quoted 40 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/net/i40e/i40e_rxtx_vec_common.h +5 -0
+		/* directly put mbufs from Tx to Rx */
+		for (i = 0; i < n; i++, rxep++, txep++)
+			*rxep = txep[0].mbuf;
+	} else {
+		for (i = 0; i < n; i++, rxep++) {
Quoted 9 lines
Quoted 1 line
Quoted 1 line

drivers/net/i40e/i40e_rxtx_vec_common.h i40e_tx_free_bufs(struct i40e_tx_queue *txq) · line 185

Quoted 1 line
drivers/net/i40e/i40e_rxtx_vec_common.h +1 -0
+			m = rte_pktmbuf_prefree_seg(txep[i].mbuf);

One thing I forgot to ask:
What would happen if this mbuf belongs to different mempool (not one that we specify at rx_queue_setup())?
Do we need to check it here?
Or would it be upper layer constraint?
Or...?

First, 'different mempool' is valid for no FAST_FREE path in tx_free_buffers.

If buffers belong to different mempool, we can have an example here:
Buffer 1 from mempool 1, its recycle path is:
----------------------------------------------------------------------
------------------- 1. queue_setup: rearm from mempool 1 into Rx sw-ring 2. rte_eth_Rx_burst: used by user app (Rx) 3.
rte_eth_Tx_burst: mount on Tx sw-ring 4. rte_eth_direct_rearm: free into Rx sw-ring:
or
tx_free_buffers: free into mempool 1 (no fast_free path)
----------------------------------------------------------------------
-------------------

Buffer 2 from mempool 2, its recycle path is:
----------------------------------------------------------------------
------------------- 1. queue_setup: rearm from mempool 2 into Rx sw-ring 2. rte_eth_Rx_burst: used by user app (Rx) 3.
rte_eth_Tx_burst: mount on Tx sw-ring 4. rte_eth_direct_rearm: free into Rx sw-ring
or
tx_free_buffers: free into mempool 2 (no fast_free_path)
----------------------------------------------------------------------
-------------------

Quoted 2 lines

Thus, buffers from Tx different mempools are the same for Rx. The difference point is that they will be freed into different mempool if the

thread uses generic free buffers.

Quoted 1 line

I think this cannot affect direct-rearm mode, and we do not need to check

this.

Quoted 9 lines

I understand that it should work even with multiple mempools. What I am trying to say - user may not want to use mbufs from particular mempool for RX (while it is still ok to use it for TX).
Let say user can have a separate mempool with small data-buffers (less then normal MTU) to send some 'special' paclets, or even use this memppol with small buffers for zero-copy updating of packet L2/L3 headers, etc. Or it could be some 'special' user provided mempool.
That's why I wonder should we allow only mbufs from mempool that is assigned to that RX queue.

Sorry for my misleading. If I understand correctly this time, you means a special mempool. Maybe its buffer size is very small and this Tx buffer is generated from control plane.

However, if we recycle this Tx buffer into Rx buffer ring, there maybe some error due to its size is so small.

Thus we can only allow general buffers which is valid for Rx buffer ring. Furthermore, this should be user's responsibility to ensure the Tx recycling buffers should be valid. If we check this in the data plane, it will cost a lot of CPU cycles. At last, what we can do is to add constraint in the notes to remind users.

Quoted 21 lines
Quoted 21 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines

drivers/net/i40e/i40e_rxtx_vec_common.h i40e_tx_free_bufs(struct i40e_tx_queue *txq)

Quoted 12 lines
drivers/net/i40e/i40e_rxtx_vec_common.h +10 -0
+			if (m != NULL) {
+				*rxep = m;
+				nb_rearm++;
+			}
+		}
+		n = nb_rearm;
+	}
+
+	/* update counters for Tx */
+	txq->nb_tx_free = (uint16_t)(txq->nb_tx_free + txq-

tx_rs_thresh);

Quoted 1 line
Quoted 1 line
Quoted 1 line

Review

Quoted 1 line
(quoted diff) +1 -0
+	txq->tx_next_dd = (uint16_t)(txq->tx_next_dd + txq-

tx_rs_thresh);

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
drivers/net/i40e/i40e_rxtx_vec_common.h +6 -0
+	if (txq->tx_next_dd >= txq->nb_tx_desc)
+		txq->tx_next_dd = (uint16_t)(txq->tx_rs_thresh - 1);
+
+	return n;
+}
+
#70 lore raw

Re: 回复: [PATCH v3 2/3] net/i40e: enable direct rearm with separate API

Quoted 93 lines
Quoted 80 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
(quoted diff) +9 -0
+int
+i40e_tx_fill_sw_ring(void *tx_queue,
+		struct rte_eth_rxq_rearm_data *rxq_rearm_data) {
+	struct i40e_tx_queue *txq = tx_queue;
+	struct i40e_tx_entry *txep;
+	void **rxep;
+	struct rte_mbuf *m;
+	int i, n;
+	int nb_rearm = 0;
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Review

Quoted 8 lines
(quoted diff) +8 -0
+
+	if (*rxq_rearm_data->rearm_nb < txq->tx_rs_thresh ||
+			txq->nb_tx_free > txq->tx_free_thresh)
+		return 0;
+
+	/* check DD bits on threshold descriptor */
+	if ((txq->tx_ring[txq->tx_next_dd].cmd_type_offset_bsz &
+
	rte_cpu_to_le_64(I40E_TXD_QW1_DTYPE_MASK)) !=
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 1 line

+

	rte_cpu_to_le_64(I40E_TX_DESC_DTYPE_DESC_DONE))

Review

Quoted 12 lines
(quoted diff) +12 -0
+		return 0;
+
+	n = txq->tx_rs_thresh;
+
+	/* first buffer to free from S/W ring is at index
+	 * tx_next_dd - (tx_rs_thresh-1)
+	 */
+	txep = &txq->sw_ring[txq->tx_next_dd - (n - 1)];
+	rxep = rxq_rearm_data->rx_sw_ring;
+	rxep += *rxq_rearm_data->rearm_start;
+
+	if (txq->offloads &

RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) {

Quoted 40 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
(quoted diff) +5 -0
+		/* directly put mbufs from Tx to Rx */
+		for (i = 0; i < n; i++, rxep++, txep++)
+			*rxep = txep[0].mbuf;
+	} else {
+		for (i = 0; i < n; i++, rxep++) {
Quoted 9 lines
Quoted 1 line
Quoted 1 line

Review

Quoted 1 line
(quoted diff) +1 -0
+			m = rte_pktmbuf_prefree_seg(txep[i].mbuf);

One thing I forgot to ask:
What would happen if this mbuf belongs to different mempool (not one that we specify at rx_queue_setup())?
Do we need to check it here?
Or would it be upper layer constraint?
Or...?

First, 'different mempool' is valid for no FAST_FREE path in tx_free_buffers.

If buffers belong to different mempool, we can have an example here:
Buffer 1 from mempool 1, its recycle path is:
----------------------------------------------------------------------
------------------- 1. queue_setup: rearm from mempool 1 into Rx sw-ring 2. rte_eth_Rx_burst: used by user app (Rx) 3.
rte_eth_Tx_burst: mount on Tx sw-ring 4. rte_eth_direct_rearm: free into Rx sw-ring:
or
tx_free_buffers: free into mempool 1 (no fast_free path)
----------------------------------------------------------------------
-------------------

Buffer 2 from mempool 2, its recycle path is:
----------------------------------------------------------------------
------------------- 1. queue_setup: rearm from mempool 2 into Rx sw-ring 2. rte_eth_Rx_burst: used by user app (Rx) 3.
rte_eth_Tx_burst: mount on Tx sw-ring 4. rte_eth_direct_rearm: free into Rx sw-ring
or
tx_free_buffers: free into mempool 2 (no fast_free_path)
----------------------------------------------------------------------
-------------------

Quoted 2 lines

Thus, buffers from Tx different mempools are the same for Rx. The difference point is that they will be freed into different mempool if the

thread uses generic free buffers.

Quoted 1 line

I think this cannot affect direct-rearm mode, and we do not need to check

this.

Quoted 9 lines

I understand that it should work even with multiple mempools. What I am trying to say - user may not want to use mbufs from particular mempool for RX (while it is still ok to use it for TX).
Let say user can have a separate mempool with small data-buffers (less then normal MTU) to send some 'special' paclets, or even use this memppol with small buffers for zero-copy updating of packet L2/L3 headers, etc. Or it could be some 'special' user provided mempool.
That's why I wonder should we allow only mbufs from mempool that is assigned to that RX queue.

Sorry for my misleading. If I understand correctly this time, you means a special mempool. Maybe its buffer size is very small and this Tx buffer is generated from control plane.

Quoted 6 lines

However, if we recycle this Tx buffer into Rx buffer ring, there maybe some error due to its size is so small.

Thus we can only allow general buffers which is valid for Rx buffer ring. Furthermore, this should be user's responsibility to ensure the Tx recycling buffers should be valid. If we check this in the data plane, it will cost a lot of CPU cycles. At last, what we can do is to add constraint in the notes to remind users.

As I thought: in theory we can add 'struct rte_mempool *mp' into rte_eth_rxq_rearm_data.
And then:
if (mbuf->pool == rxq_rearm_data->mp)
/* put mbuf into rearm buffer */
else
/* free mbuf */
For the 'proper' config (when txq contains mbufs from expected mempool) the overhead will be minimal.
In other case it might be higher, but still would work and no need for extra limitations.

Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines

Review

Quoted 12 lines
(quoted diff) +10 -0
+			if (m != NULL) {
+				*rxep = m;
+				nb_rearm++;
+			}
+		}
+		n = nb_rearm;
+	}
+
+	/* update counters for Tx */
+	txq->nb_tx_free = (uint16_t)(txq->nb_tx_free + txq-

tx_rs_thresh);

Quoted 1 line
Quoted 1 line
Quoted 1 line

Review

Quoted 1 line
(quoted diff) +1 -0
+	txq->tx_next_dd = (uint16_t)(txq->tx_next_dd + txq-

tx_rs_thresh);

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
(quoted diff) +6 -0
+	if (txq->tx_next_dd >= txq->nb_tx_desc)
+		txq->tx_next_dd = (uint16_t)(txq->tx_rs_thresh - 1);
+
+	return n;
+}
+
#71 lore raw

RE: 回复: [PATCH v3 2/3] net/i40e: enable direct rearm with separate API

Quoted 125 lines

-----Original Message-----
From: Konstantin Ananyev <[email protected]>
Sent: Monday, March 20, 2023 12:11 AM
To: Feifei Wang <[email protected]>; Konstantin Ananyev <[email protected]>; Yuying Zhang <[email protected]>; Beilei Xing <[email protected]>; Ruifeng Wang <[email protected]>
Cc: [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>
Subject: Re: 回复: [PATCH v3 2/3] net/i40e: enable direct rearm with separate API

Quoted 48 lines
Quoted 48 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
(quoted diff) +9 -0
+int
+i40e_tx_fill_sw_ring(void *tx_queue,
+		struct rte_eth_rxq_rearm_data *rxq_rearm_data) {
+	struct i40e_tx_queue *txq = tx_queue;
+	struct i40e_tx_entry *txep;
+	void **rxep;
+	struct rte_mbuf *m;
+	int i, n;
+	int nb_rearm = 0;
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Review

Quoted 8 lines
(quoted diff) +8 -0
+
+	if (*rxq_rearm_data->rearm_nb < txq->tx_rs_thresh ||
+			txq->nb_tx_free > txq->tx_free_thresh)
+		return 0;
+
+	/* check DD bits on threshold descriptor */
+	if ((txq->tx_ring[txq->tx_next_dd].cmd_type_offset_bsz &
+
	rte_cpu_to_le_64(I40E_TXD_QW1_DTYPE_MASK)) !=
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 1 line

+

	rte_cpu_to_le_64(I40E_TX_DESC_DTYPE_DESC_DONE))

Review

Quoted 12 lines
(quoted diff) +12 -0
+		return 0;
+
+	n = txq->tx_rs_thresh;
+
+	/* first buffer to free from S/W ring is at index
+	 * tx_next_dd - (tx_rs_thresh-1)
+	 */
+	txep = &txq->sw_ring[txq->tx_next_dd - (n - 1)];
+	rxep = rxq_rearm_data->rx_sw_ring;
+	rxep += *rxq_rearm_data->rearm_start;
+
+	if (txq->offloads &

RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) {

Quoted 15 lines
Quoted 14 lines
Quoted 6 lines
Quoted 6 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+		/* directly put mbufs from Tx to Rx */
+		for (i = 0; i < n; i++, rxep++, txep++)
+			*rxep = txep[0].mbuf;
+	} else {
+		for (i = 0; i < n; i++, rxep++) {
+			m = rte_pktmbuf_prefree_seg(txep[i].mbuf);

One thing I forgot to ask:
What would happen if this mbuf belongs to different mempool (not one that we specify at rx_queue_setup())?
Do we need to check it here?
Or would it be upper layer constraint?
Or...?

Quoted 1 line
Quoted 1 line
Quoted 1 line

First, 'different mempool' is valid for no FAST_FREE path in

tx_free_buffers.

Quoted 34 lines
Quoted 34 lines
Quoted 28 lines

If buffers belong to different mempool, we can have an example here:
Buffer 1 from mempool 1, its recycle path is:
--------------------------------------------------------------------
--
------------------- 1. queue_setup: rearm from mempool 1 into Rx sw-ring 2. rte_eth_Rx_burst: used by user app (Rx) 3.
rte_eth_Tx_burst: mount on Tx sw-ring 4. rte_eth_direct_rearm: free into Rx sw-ring:
or
tx_free_buffers: free into mempool 1 (no fast_free path)
--------------------------------------------------------------------
--
-------------------

Buffer 2 from mempool 2, its recycle path is:
--------------------------------------------------------------------
--
------------------- 1. queue_setup: rearm from mempool 2 into Rx sw-ring 2. rte_eth_Rx_burst: used by user app (Rx) 3.
rte_eth_Tx_burst: mount on Tx sw-ring 4. rte_eth_direct_rearm: free into Rx sw-ring
or
tx_free_buffers: free into mempool 2 (no fast_free_path)
--------------------------------------------------------------------
--
-------------------

Quoted 3 lines

Thus, buffers from Tx different mempools are the same for Rx. The difference point is that they will be freed into different mempool if the

thread uses generic free buffers.

Quoted 2 lines

I think this cannot affect direct-rearm mode, and we do not need to check

Quoted 8 lines
Quoted 8 lines

this.

I understand that it should work even with multiple mempools. What I am trying to say - user may not want to use mbufs from particular mempool for RX (while it is still ok to use it for TX). Let say user can have a separate mempool with small data-buffers (less then normal MTU) to send some 'special' paclets, or even use this memppol with small buffers for zero-copy updating of packet L2/L3

headers, etc.

Quoted 6 lines
Quoted 3 lines

Or it could be some 'special' user provided mempool. That's why I wonder should we allow only mbufs from mempool that is assigned to that RX queue.

Sorry for my misleading. If I understand correctly this time, you means a special mempool. Maybe its buffer size is very small and this Tx

buffer is generated from control plane.

Quoted 7 lines

However, if we recycle this Tx buffer into Rx buffer ring, there maybe some error due to its size is so small.

Thus we can only allow general buffers which is valid for Rx buffer ring. Furthermore, this should be user's responsibility to ensure the Tx recycling buffers should be valid. If we check this in the data plane, it will

cost a lot of CPU cycles. At last, what we can do is to add constraint in the notes to remind users.

Quoted 11 lines

As I thought: in theory we can add 'struct rte_mempool *mp'
into rte_eth_rxq_rearm_data.
And then:
if (mbuf->pool == rxq_rearm_data->mp)
/* put mbuf into rearm buffer */
else
/* free mbuf */
For the 'proper' config (when txq contains mbufs from expected mempool) the overhead will be minimal.
In other case it might be higher, but still would work and no need for extra limitations.

It's a good idea. And try to test performance with this change, there is currently no performance degradation. Thus, I add this check in the latest version.

Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 4 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

Review

Quoted 10 lines
(quoted diff) +10 -0
+			if (m != NULL) {
+				*rxep = m;
+				nb_rearm++;
+			}
+		}
+		n = nb_rearm;
+	}
+
+	/* update counters for Tx */
+	txq->nb_tx_free = (uint16_t)(txq->nb_tx_free + txq-

tx_rs_thresh);

Quoted 1 line
Quoted 1 line
Quoted 1 line

Review

Quoted 1 line
(quoted diff) +1 -0
+	txq->tx_next_dd = (uint16_t)(txq->tx_next_dd + txq-

tx_rs_thresh);

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
(quoted diff) +6 -0
+	if (txq->tx_next_dd >= txq->nb_tx_desc)
+		txq->tx_next_dd = (uint16_t)(txq->tx_rs_thresh - 1);
+
+	return n;
+}
+
#72 lore raw

PATCH v3 3/3 net/ixgbe: enable direct rearm with separate API

In reply to Feifei Wang

Add internal API to separate direct rearm operations between Rx and Tx.

---
 drivers/net/ixgbe/ixgbe_ethdev.c          |  1 +
 drivers/net/ixgbe/ixgbe_ethdev.h          |  3 ++
 drivers/net/ixgbe/ixgbe_rxtx.c            | 19 +++++++++
 drivers/net/ixgbe/ixgbe_rxtx.h            |  4 ++
 drivers/net/ixgbe/ixgbe_rxtx_vec_common.h | 48 +++++++++++++++++++++
 drivers/net/ixgbe/ixgbe_rxtx_vec_neon.c   | 52 +++++++++++++++++++++++
 6 files changed, 127 insertions(+)
drivers/net/ixgbe/ixgbe_ethdev.c +1 -0
index ae9f65b334..e5383d7dbc 100644
@@ -542,6 +542,7 @@ static const struct eth_dev_ops ixgbe_eth_dev_ops = {
 	.set_mc_addr_list     = ixgbe_dev_set_mc_addr_list,
 	.rxq_info_get         = ixgbe_rxq_info_get,
 	.txq_info_get         = ixgbe_txq_info_get,
+	.rxq_rearm_data_get   = ixgbe_rxq_rearm_data_get,
 	.timesync_enable      = ixgbe_timesync_enable,
 	.timesync_disable     = ixgbe_timesync_disable,
 	.timesync_read_rx_timestamp = ixgbe_timesync_read_rx_timestamp,
drivers/net/ixgbe/ixgbe_ethdev.h +3 -0
index 48290af512..2a8ae5af7a 100644
@@ -625,6 +625,9 @@ void ixgbe_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 void ixgbe_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 	struct rte_eth_txq_info *qinfo);
 
+void ixgbe_rxq_rearm_data_get(struct rte_eth_dev *dev, uint16_t queue_id,
+	struct rte_eth_rxq_rearm_data *rxq_rearm_data);
+
 int ixgbevf_dev_rx_init(struct rte_eth_dev *dev);
 
 void ixgbevf_dev_tx_init(struct rte_eth_dev *dev);
drivers/net/ixgbe/ixgbe_rxtx.c +19 -0
index c9d6ca9efe..2d7fe710e4 100644
@@ -2559,6 +2559,9 @@ ixgbe_set_tx_function(struct rte_eth_dev *dev, struct ixgbe_tx_queue *txq)
 					ixgbe_txq_vec_setup(txq) == 0)) {
 			PMD_INIT_LOG(DEBUG, "Vector tx enabled.");
 			dev->tx_pkt_burst = ixgbe_xmit_pkts_vec;
+#ifdef RTE_ARCH_ARM64
+			dev->tx_fill_sw_ring = ixgbe_tx_fill_sw_ring;
+#endif
 		} else
 		dev->tx_pkt_burst = ixgbe_xmit_pkts_simple;
 	} else {
@@ -4853,6 +4856,9 @@ ixgbe_set_rx_function(struct rte_eth_dev *dev)
 			     dev->data->port_id);
 
 		dev->rx_pkt_burst = ixgbe_recv_pkts_vec;
+#ifdef RTE_ARCH_ARM64
+		dev->rx_flush_descriptor = ixgbe_rx_flush_descriptor_vec;
+#endif
 	} else if (adapter->rx_bulk_alloc_allowed) {
 		PMD_INIT_LOG(DEBUG, "Rx Burst Bulk Alloc Preconditions are "
 				    "satisfied. Rx Burst Bulk Alloc function "
@@ -5623,6 +5629,19 @@ ixgbe_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 	qinfo->conf.tx_deferred_start = txq->tx_deferred_start;
 }
 
+void
+ixgbe_rxq_rearm_data_get(struct rte_eth_dev *dev, uint16_t queue_id,
+	struct rte_eth_rxq_rearm_data *rxq_rearm_data)
+{
+	struct ixgbe_rx_queue *rxq;
+
+	rxq = dev->data->rx_queues[queue_id];
+
+	rxq_rearm_data->rx_sw_ring = rxq->sw_ring;
+	rxq_rearm_data->rearm_start = &rxq->rxrearm_start;
+	rxq_rearm_data->rearm_nb = &rxq->rxrearm_nb;
+}
+
 /*
  * [VF] Initializes Receive Unit.
  */
drivers/net/ixgbe/ixgbe_rxtx.h +4 -0
index 668a5b9814..7c90426f49 100644
@@ -295,6 +295,10 @@ int ixgbe_dev_tx_done_cleanup(void *tx_queue, uint32_t free_cnt);
 extern const uint32_t ptype_table[IXGBE_PACKET_TYPE_MAX];
 extern const uint32_t ptype_table_tn[IXGBE_PACKET_TYPE_TN_MAX];
 
+int ixgbe_tx_fill_sw_ring(void *tx_queue,
+		struct rte_eth_rxq_rearm_data *rxq_rearm_data);
+int ixgbe_rx_flush_descriptor_vec(void *rx_queue, uint16_t nb_rearm);
+
 uint16_t ixgbe_xmit_fixed_burst_vec(void *tx_queue, struct rte_mbuf **tx_pkts,
 				    uint16_t nb_pkts);
 int ixgbe_txq_vec_setup(struct ixgbe_tx_queue *txq);
drivers/net/ixgbe/ixgbe_rxtx_vec_common.h +48 -0

54 lines not on this page · raw message

drivers/net/ixgbe/ixgbe_rxtx_vec_neon.c +52 -0

58 lines not on this page · raw message

Signature · 1 line
2.25.1
#74 lore raw

Hi Feifei,

Quoted 4 lines

+ping konstantin,

Would you please give some comments for this patch series? Thanks very much.

Sure, will have a look in next few days.
Apologies for the delay.

#75 lore raw

That's all right. Thanks very much for your attention~

Quoted 16 lines

-----邮件原件-----
发件人: Konstantin Ananyev <[email protected]> 发送时间: Wednesday, February 1, 2023 9:11 AM
收件人: Feifei Wang <[email protected]>
抄送: [email protected]; nd <[email protected]>
主题: Re: 回复: [PATCH v3 0/3] Direct re-arming of buffers on receive side

Hi Feifei,

Quoted 4 lines

+ping konstantin,

Would you please give some comments for this patch series? Thanks very much.

Sure, will have a look in next few days.
Apologies for the delay.

#76 lore raw
In reply to Feifei Wang
Quoted 10 lines

From: Feifei Wang [mailto:[email protected]]
Sent: Wednesday, 4 January 2023 08.31

Currently, the transmit side frees the buffers into the lcore cache and the receive side allocates buffers from the lcore cache. The transmit side typically frees 32 buffers resulting in 32*8=256B of stores to lcore cache. The receive side allocates 32 buffers and stores them in the receive side software ring, resulting in 32*8=256B of stores and 256B of load from the lcore cache.

Quoted 5 lines

This patch proposes a mechanism to avoid freeing to/allocating from the lcore cache. i.e. the receive side will free the buffers from transmit side directly into its software ring. This will avoid the 256B of loads and stores introduced by the lcore cache. It also frees up the cache lines used by the lcore cache.

I am starting to wonder if we have been adding unnecessary feature creep in order to make this feature too generic.

Could you please describe some of the most important high-volume use cases from real life? It would help setting the scope correctly.

Quoted 146 lines

However, this solution poses several constraints:

1)The receive queue needs to know which transmit queue it should take the buffers from. The application logic decides which transmit port to use to send out the packets. In many use cases the NIC might have a single port ([1], [2], [3]), in which case a given transmit queue is always mapped to a single receive queue (1:1 Rx queue: Tx queue). This is easy to configure.

If the NIC has 2 ports (there are several references), then we will have 1:2 (RX queue: TX queue) mapping which is still easy to configure. However, if this is generalized to 'N' ports, the configuration can be long. More over the PMD would have to scan a list of transmit queues to pull the buffers from.

2)The other factor that needs to be considered is 'run-to-completion' vs 'pipeline' models. In the run-to-completion model, the receive side and the transmit side are running on the same lcore serially. In the pipeline model. The receive side and transmit side might be running on different lcores in parallel. This requires locking. This is not supported at this point.

3)Tx and Rx buffers must be from the same mempool. And we also must ensure Tx buffer free number is equal to Rx buffer free number. Thus, 'tx_next_dd' can be updated correctly in direct-rearm mode. This is due to tx_next_dd is a variable to compute tx sw-ring free location. Its value will be one more round than the position where next time free starts.

Current status in this patch:
1)Two APIs are added for users to enable direct-rearm mode:
In control plane, users can call 'rte_eth_rx_queue_rearm_data_get' to get Rx sw_ring pointer and its rxq_info.
(This avoid Tx load Rx data directly);

In data plane, users can call 'rte_eth_dev_direct_rearm' to rearm Rx buffers and free Tx buffers at the same time. Specifically, in this API, there are two separated API for Rx and Tx.
For Tx, 'rte_eth_tx_fill_sw_ring' can fill a given sw_ring by Tx freed
buffers.
For Rx, 'rte_eth_rx_flush_descriptor' can flush its descriptors based on the rearm buffers.
Thus, this can separate Rx and Tx operation, and user can even re-arm RX queue not from the same driver's TX queue, but from different sources too.
-----------------------------------------------------------------------
control plane:
rte_eth_rx_queue_rearm_data_get(*rxq_rearm_data);
data plane:
loop {
rte_eth_dev_direct_rearm(*rxq_rearm_data){

			rte_eth_tx_fill_sw_ring{
     				for (i = 0; i <= 32; i++) {
       					sw_ring.mbuf[i] = tx.mbuf[i];
				}
    			}

			rte_eth_rx_flush_descriptor{
				for (i = 0; i <= 32; i++) {
					flush descs[i];
				}
			}
		}
		rte_eth_rx_burst;
		rte_eth_tx_burst;

}
-----------------------------------------------------------------------
2)The i40e driver is changed to do the direct re-arm of the receive
side.
3)The ixgbe driver is changed to do the direct re-arm of the receive
side.

Testing status:
(1) dpdk l3fwd test with multiple drivers:

    port 0: 82599 NIC   port 1: XL710 NIC

-------------------------------------------------------------
Without fast free With fast free
Thunderx2: +9.44% +7.14%
-------------------------------------------------------------

(2) dpdk l3fwd test with same driver:

    port 0 && 1: XL710 NIC

-------------------------------------------------------------
*Direct rearm with exposing rx_sw_ring:
Without fast free With fast free
Ampere altra: +14.98% +15.77%
n1sdp: +6.47% +0.52%
-------------------------------------------------------------

(3) VPP test with same driver:

    port 0 && 1: XL710 NIC

-------------------------------------------------------------
*Direct rearm with exposing rx_sw_ring:
Ampere altra: +4.59%
n1sdp: +5.4%
-------------------------------------------------------------

Reference:
[1] https://store.nvidia.com/en-us/networking/store/product/MCX623105AN-
CDAT/NVIDIAMCX623105ANCDATConnectX6DxENAdapterCard100GbECryptoDisabled/
[2] https://www.intel.com/content/www/us/en/products/sku/192561/intel-
ethernet-network-adapter-e810cqda1/specifications.html
[3] https://www.broadcom.com/products/ethernet-connectivity/network-
adapters/100gb-nic-ocp/n1100g

V2:
1. Use data-plane API to enable direct-rearm (Konstantin, Honnappa)
2. Add 'txq_data_get' API to get txq info for Rx (Konstantin)
3. Use input parameter to enable direct rearm in l3fwd (Konstantin)
4. Add condition detection for direct rearm API (Morten, Andrew Rybchenko)

V3:
1. Seperate Rx and Tx operation with two APIs in direct-rearm (Konstantin)
2. Delete L3fwd change for direct rearm (Jerin)
3. enable direct rearm in ixgbe driver in Arm

Feifei Wang (3):
ethdev: enable direct rearm with separate API net/i40e: enable direct rearm with separate API net/ixgbe: enable direct rearm with separate API

 drivers/net/i40e/i40e_ethdev.c            |   1 +
 drivers/net/i40e/i40e_ethdev.h            |   2 +
 drivers/net/i40e/i40e_rxtx.c              |  19 +++
 drivers/net/i40e/i40e_rxtx.h              |   4 +
 drivers/net/i40e/i40e_rxtx_vec_common.h   |  54 +++++++
 drivers/net/i40e/i40e_rxtx_vec_neon.c     |  42 ++++++
 drivers/net/ixgbe/ixgbe_ethdev.c          |   1 +
 drivers/net/ixgbe/ixgbe_ethdev.h          |   3 +
 drivers/net/ixgbe/ixgbe_rxtx.c            |  19 +++
 drivers/net/ixgbe/ixgbe_rxtx.h            |   4 +
 drivers/net/ixgbe/ixgbe_rxtx_vec_common.h |  48 ++++++
 drivers/net/ixgbe/ixgbe_rxtx_vec_neon.c   |  52 +++++++
 lib/ethdev/ethdev_driver.h                |  10 ++
 lib/ethdev/ethdev_private.c               |   2 +
 lib/ethdev/rte_ethdev.c                   |  52 +++++++
 lib/ethdev/rte_ethdev.h                   | 174 ++++++++++++++++++++++
 lib/ethdev/rte_ethdev_core.h              |  11 ++
 lib/ethdev/version.map                    |   6 +
 18 files changed, 504 insertions(+)

--
2.25.1

#77 lore raw
Quoted 22 lines

-----Original Message-----
From: Morten Brørup <[email protected]>
Sent: Wednesday, March 22, 2023 7:57 AM
To: Feifei Wang <[email protected]>
Cc: [email protected]; [email protected]; nd <[email protected]>; [email protected]; Yuying Zhang <[email protected]>; Beilei Xing <[email protected]>; Ruifeng Wang <[email protected]>; Honnappa Nagarahalli <[email protected]>
Subject: RE: [PATCH v3 0/3] Direct re-arming of buffers on receive side

Quoted 10 lines

From: Feifei Wang [mailto:[email protected]]
Sent: Wednesday, 4 January 2023 08.31

Currently, the transmit side frees the buffers into the lcore cache and the receive side allocates buffers from the lcore cache. The transmit side typically frees 32 buffers resulting in 32*8=256B of stores to lcore cache. The receive side allocates 32 buffers and stores them in the receive side software ring, resulting in 32*8=256B of stores and 256B of load from the lcore cache.

Quoted 8 lines
Quoted 5 lines

This patch proposes a mechanism to avoid freeing to/allocating from the lcore cache. i.e. the receive side will free the buffers from transmit side directly into its software ring. This will avoid the 256B of loads and stores introduced by the lcore cache. It also frees up the cache lines used by the lcore cache.

I am starting to wonder if we have been adding unnecessary feature creep in order to make this feature too generic.

Can you please elaborate on the feature creep you are thinking? The features have been the same since the first implementation, but it is made more generic.

Quoted 3 lines

Could you please describe some of the most important high-volume use cases from real life? It would help setting the scope correctly.

The use cases have been discussed several times already.

Quoted 152 lines
Quoted 101 lines

However, this solution poses several constraints:

1)The receive queue needs to know which transmit queue it should take the buffers from. The application logic decides which transmit port to use to send out the packets. In many use cases the NIC might have a single port ([1], [2], [3]), in which case a given transmit queue is always mapped to a single receive queue (1:1 Rx queue: Tx queue). This is easy to configure.

If the NIC has 2 ports (there are several references), then we will have
1:2 (RX queue: TX queue) mapping which is still easy to configure. However, if this is generalized to 'N' ports, the configuration can be long. More over the PMD would have to scan a list of transmit queues to pull the buffers from.

2)The other factor that needs to be considered is 'run-to-completion' vs 'pipeline' models. In the run-to-completion model, the receive side and the transmit side are running on the same lcore serially. In the pipeline model. The receive side and transmit side might be running on different lcores in parallel. This requires locking. This is not supported at this point.

3)Tx and Rx buffers must be from the same mempool. And we also must ensure Tx buffer free number is equal to Rx buffer free number. Thus, 'tx_next_dd' can be updated correctly in direct-rearm mode. This is due to tx_next_dd is a variable to compute tx sw-ring free location. Its value will be one more round than the position where next time free starts.

Current status in this patch:
1)Two APIs are added for users to enable direct-rearm mode:
In control plane, users can call 'rte_eth_rx_queue_rearm_data_get' to get Rx sw_ring pointer and its rxq_info.
(This avoid Tx load Rx data directly);

In data plane, users can call 'rte_eth_dev_direct_rearm' to rearm Rx buffers and free Tx buffers at the same time. Specifically, in this API, there are two separated API for Rx and Tx.
For Tx, 'rte_eth_tx_fill_sw_ring' can fill a given sw_ring by Tx
freed buffers.
For Rx, 'rte_eth_rx_flush_descriptor' can flush its descriptors based on the rearm buffers.
Thus, this can separate Rx and Tx operation, and user can even re-arm RX queue not from the same driver's TX queue, but from different sources too.
-----------------------------------------------------------------------
control plane:
rte_eth_rx_queue_rearm_data_get(*rxq_rearm_data);
data plane:
loop {
rte_eth_dev_direct_rearm(*rxq_rearm_data){

			rte_eth_tx_fill_sw_ring{
     				for (i = 0; i <= 32; i++) {
       					sw_ring.mbuf[i] = tx.mbuf[i];
				}
    			}

			rte_eth_rx_flush_descriptor{
				for (i = 0; i <= 32; i++) {
					flush descs[i];
				}
			}
		}
		rte_eth_rx_burst;
		rte_eth_tx_burst;

}
----------------------------------------------------------------------
- 2)The i40e driver is changed to do the direct re-arm of the receive side.
3)The ixgbe driver is changed to do the direct re-arm of the receive
side.

Testing status:
(1) dpdk l3fwd test with multiple drivers:

    port 0: 82599 NIC   port 1: XL710 NIC

-------------------------------------------------------------
Without fast free With fast free
Thunderx2: +9.44% +7.14%
-------------------------------------------------------------

(2) dpdk l3fwd test with same driver:

    port 0 && 1: XL710 NIC

-------------------------------------------------------------
*Direct rearm with exposing rx_sw_ring:
Without fast free With fast free
Ampere altra: +14.98% +15.77%
n1sdp: +6.47% +0.52%
-------------------------------------------------------------

(3) VPP test with same driver:

    port 0 && 1: XL710 NIC

-------------------------------------------------------------
*Direct rearm with exposing rx_sw_ring:
Ampere altra: +4.59%
n1sdp: +5.4%
-------------------------------------------------------------

Quoted 4 lines

Reference:
[1]
https://store.nvidia.com/en-us/networking/store/product/MCX623105AN-

CDAT/NVIDIAMCX623105ANCDATConnectX6DxENAdapterCard100GbECrypto Disabled

Quoted 45 lines

/ [2]
https://www.intel.com/content/www/us/en/products/sku/192561/intel-
ethernet-network-adapter-e810cqda1/specifications.html
[3] https://www.broadcom.com/products/ethernet-connectivity/network-
adapters/100gb-nic-ocp/n1100g

V2:
1. Use data-plane API to enable direct-rearm (Konstantin, Honnappa) 2. Add 'txq_data_get' API to get txq info for Rx (Konstantin) 3. Use input parameter to enable direct rearm in l3fwd (Konstantin) 4. Add condition detection for direct rearm API (Morten, Andrew Rybchenko)

V3:
1. Seperate Rx and Tx operation with two APIs in direct-rearm (Konstantin) 2. Delete L3fwd change for direct rearm (Jerin) 3. enable direct rearm in ixgbe driver in Arm

Feifei Wang (3):
ethdev: enable direct rearm with separate API net/i40e: enable direct rearm with separate API net/ixgbe: enable direct rearm with separate API

 drivers/net/i40e/i40e_ethdev.c            |   1 +
 drivers/net/i40e/i40e_ethdev.h            |   2 +
 drivers/net/i40e/i40e_rxtx.c              |  19 +++
 drivers/net/i40e/i40e_rxtx.h              |   4 +
 drivers/net/i40e/i40e_rxtx_vec_common.h   |  54 +++++++
 drivers/net/i40e/i40e_rxtx_vec_neon.c     |  42 ++++++
 drivers/net/ixgbe/ixgbe_ethdev.c          |   1 +
 drivers/net/ixgbe/ixgbe_ethdev.h          |   3 +
 drivers/net/ixgbe/ixgbe_rxtx.c            |  19 +++
 drivers/net/ixgbe/ixgbe_rxtx.h            |   4 +
 drivers/net/ixgbe/ixgbe_rxtx_vec_common.h |  48 ++++++
 drivers/net/ixgbe/ixgbe_rxtx_vec_neon.c   |  52 +++++++
 lib/ethdev/ethdev_driver.h                |  10 ++
 lib/ethdev/ethdev_private.c               |   2 +
 lib/ethdev/rte_ethdev.c                   |  52 +++++++
 lib/ethdev/rte_ethdev.h                   | 174 ++++++++++++++++++++++
 lib/ethdev/rte_ethdev_core.h              |  11 ++
 lib/ethdev/version.map                    |   6 +
 18 files changed, 504 insertions(+)

--
2.25.1

#78 lore raw
Quoted 22 lines

From: Honnappa Nagarahalli [mailto:[email protected]]
Sent: Wednesday, 22 March 2023 14.42

Quoted 19 lines

From: Morten Brørup <[email protected]>
Sent: Wednesday, March 22, 2023 7:57 AM

Quoted 15 lines

From: Feifei Wang [mailto:[email protected]]
Sent: Wednesday, 4 January 2023 08.31

Currently, the transmit side frees the buffers into the lcore cache and the receive side allocates buffers from the lcore cache. The transmit side typically frees 32 buffers resulting in 32*8=256B of stores to lcore cache. The receive side allocates 32 buffers and stores them in the receive side software ring, resulting in 32*8=256B of stores and 256B of load from the lcore cache.

This patch proposes a mechanism to avoid freeing to/allocating from the lcore cache. i.e. the receive side will free the buffers from transmit side directly into its software ring. This will avoid the 256B of loads and stores introduced by the lcore cache. It also frees up the cache lines used by the lcore cache.

Quoted 5 lines
Quoted 2 lines

I am starting to wonder if we have been adding unnecessary feature creep in order to make this feature too generic.

Can you please elaborate on the feature creep you are thinking? The features have been the same since the first implementation, but it is made more generic.

Maybe not "features" as such; but the API has evolved, and perhaps we could simplify both the API and the implementation if we narrowed the scope.

I'm not saying that what we have is bad or too complex; I'm only asking to consider if there are opportunities to take a step back and simplify some things.

Quoted 5 lines
Quoted 4 lines

Could you please describe some of the most important high-volume use cases from real life? It would help setting the scope correctly.

The use cases have been discussed several times already.

Yes, but they should be mentioned in the patch cover letter - and later on in the documentation.

It will help limiting the scope while developing this feature. And it will make it easier for application developers to relate to the feature and determine if it is relevant for their application.

Quoted 154 lines
Quoted 153 lines
Quoted 102 lines

However, this solution poses several constraints:

1)The receive queue needs to know which transmit queue it should take the buffers from. The application logic decides which transmit port to use to send out the packets. In many use cases the NIC might have a single port ([1], [2], [3]), in which case a given transmit queue is always mapped to a single receive queue (1:1 Rx queue: Tx queue). This is easy to configure.

If the NIC has 2 ports (there are several references), then we will have
1:2 (RX queue: TX queue) mapping which is still easy to configure. However, if this is generalized to 'N' ports, the configuration can be long. More over the PMD would have to scan a list of transmit queues to pull the buffers from.

2)The other factor that needs to be considered is 'run-to-completion' vs 'pipeline' models. In the run-to-completion model, the receive side and the transmit side are running on the same lcore serially. In the pipeline model. The receive side and transmit side might be running on different lcores in parallel. This requires locking. This is not supported at this point.

3)Tx and Rx buffers must be from the same mempool. And we also must ensure Tx buffer free number is equal to Rx buffer free number. Thus, 'tx_next_dd' can be updated correctly in direct-rearm mode. This is due to tx_next_dd is a variable to compute tx sw-ring free location. Its value will be one more round than the position where next time free starts.

Current status in this patch:
1)Two APIs are added for users to enable direct-rearm mode:
In control plane, users can call 'rte_eth_rx_queue_rearm_data_get' to get Rx sw_ring pointer and its rxq_info.
(This avoid Tx load Rx data directly);

In data plane, users can call 'rte_eth_dev_direct_rearm' to rearm Rx buffers and free Tx buffers at the same time. Specifically, in this API, there are two separated API for Rx and Tx.
For Tx, 'rte_eth_tx_fill_sw_ring' can fill a given sw_ring by Tx
freed buffers.
For Rx, 'rte_eth_rx_flush_descriptor' can flush its descriptors based on the rearm buffers.
Thus, this can separate Rx and Tx operation, and user can even re-arm RX queue not from the same driver's TX queue, but from different sources too.
-----------------------------------------------------------------------
control plane:
rte_eth_rx_queue_rearm_data_get(*rxq_rearm_data);
data plane:
loop {
rte_eth_dev_direct_rearm(*rxq_rearm_data){

			rte_eth_tx_fill_sw_ring{
     				for (i = 0; i <= 32; i++) {
       					sw_ring.mbuf[i] = tx.mbuf[i];
				}
    			}

			rte_eth_rx_flush_descriptor{
				for (i = 0; i <= 32; i++) {
					flush descs[i];
				}
			}
		}
		rte_eth_rx_burst;
		rte_eth_tx_burst;

}
----------------------------------------------------------------------
- 2)The i40e driver is changed to do the direct re-arm of the receive side.
3)The ixgbe driver is changed to do the direct re-arm of the receive
side.

Testing status:
(1) dpdk l3fwd test with multiple drivers:

    port 0: 82599 NIC   port 1: XL710 NIC

-------------------------------------------------------------
Without fast free With fast free
Thunderx2: +9.44% +7.14%
-------------------------------------------------------------

(2) dpdk l3fwd test with same driver:

    port 0 && 1: XL710 NIC

-------------------------------------------------------------
*Direct rearm with exposing rx_sw_ring:
Without fast free With fast free
Ampere altra: +14.98% +15.77%
n1sdp: +6.47% +0.52%
-------------------------------------------------------------

(3) VPP test with same driver:

    port 0 && 1: XL710 NIC

-------------------------------------------------------------
*Direct rearm with exposing rx_sw_ring:
Ampere altra: +4.59%
n1sdp: +5.4%
-------------------------------------------------------------

Quoted 4 lines

Reference:
[1]
https://store.nvidia.com/en-us/networking/store/product/MCX623105AN-

CDAT/NVIDIAMCX623105ANCDATConnectX6DxENAdapterCard100GbECrypto Disabled

Quoted 45 lines

/ [2]
https://www.intel.com/content/www/us/en/products/sku/192561/intel-
ethernet-network-adapter-e810cqda1/specifications.html
[3] https://www.broadcom.com/products/ethernet-connectivity/network-
adapters/100gb-nic-ocp/n1100g

V2:
1. Use data-plane API to enable direct-rearm (Konstantin, Honnappa) 2. Add 'txq_data_get' API to get txq info for Rx (Konstantin) 3. Use input parameter to enable direct rearm in l3fwd (Konstantin) 4. Add condition detection for direct rearm API (Morten, Andrew Rybchenko)

V3:
1. Seperate Rx and Tx operation with two APIs in direct-rearm (Konstantin) 2. Delete L3fwd change for direct rearm (Jerin) 3. enable direct rearm in ixgbe driver in Arm

Feifei Wang (3):
ethdev: enable direct rearm with separate API net/i40e: enable direct rearm with separate API net/ixgbe: enable direct rearm with separate API

 drivers/net/i40e/i40e_ethdev.c            |   1 +
 drivers/net/i40e/i40e_ethdev.h            |   2 +
 drivers/net/i40e/i40e_rxtx.c              |  19 +++
 drivers/net/i40e/i40e_rxtx.h              |   4 +
 drivers/net/i40e/i40e_rxtx_vec_common.h   |  54 +++++++
 drivers/net/i40e/i40e_rxtx_vec_neon.c     |  42 ++++++
 drivers/net/ixgbe/ixgbe_ethdev.c          |   1 +
 drivers/net/ixgbe/ixgbe_ethdev.h          |   3 +
 drivers/net/ixgbe/ixgbe_rxtx.c            |  19 +++
 drivers/net/ixgbe/ixgbe_rxtx.h            |   4 +
 drivers/net/ixgbe/ixgbe_rxtx_vec_common.h |  48 ++++++
 drivers/net/ixgbe/ixgbe_rxtx_vec_neon.c   |  52 +++++++
 lib/ethdev/ethdev_driver.h                |  10 ++
 lib/ethdev/ethdev_private.c               |   2 +
 lib/ethdev/rte_ethdev.c                   |  52 +++++++
 lib/ethdev/rte_ethdev.h                   | 174 ++++++++++++++++++++++
 lib/ethdev/rte_ethdev_core.h              |  11 ++
 lib/ethdev/version.map                    |   6 +
 18 files changed, 504 insertions(+)

--
2.25.1

#79 lore raw

v8 cover Recycle mbufs from Tx queue into Rx queue

In reply to Feifei Wang

Currently, the transmit side frees the buffers into the lcore cache and the receive side allocates buffers from the lcore cache. The transmit side typically frees 32 buffers resulting in 32*8=256B of stores to lcore cache. The receive side allocates 32 buffers and stores them in the receive side software ring, resulting in 32*8=256B of stores and 256B of load from the lcore cache.

This patch proposes a mechanism to avoid freeing to/allocating from the lcore cache. i.e. the receive side will free the buffers from transmit side directly into its software ring. This will avoid the 256B of loads and stores introduced by the lcore cache. It also frees up the cache lines used by the lcore cache. And we can call this mode as mbufs recycle mode.

In the latest version, mbufs recycle mode is packaged as a separate API.
This allows for the users to change rxq/txq pairing in real time in data plane, according to the analysis of the packet flow by the application, for example:
-----------------------------------------------------------------------
Step 1: upper application analyse the flow direction
Step 2: recycle_rxq_info = rte_eth_recycle_rx_queue_info_get(rx_portid, rx_queueid) Step 3: rte_eth_recycle_mbufs(rx_portid, rx_queueid, tx_portid, tx_queueid, recycle_rxq_info); Step 4: rte_eth_rx_burst(rx_portid,rx_queueid);
Step 5: rte_eth_tx_burst(tx_portid,tx_queueid);
-----------------------------------------------------------------------
Above can support user to change rxq/txq pairing at run-time and user does not need to know the direction of flow in advance. This can effectively expand mbufs recycle mode's use scenarios.

Furthermore, mbufs recycle mode is no longer limited to the same pmd,
it can support moving mbufs between different vendor pmds, even can put the mbufs anywhere into your Rx mbuf ring as long as the address of the mbuf ring can be provided. In the latest version, we enable mbufs recycle mode in i40e pmd and ixgbe pmd, and also try to use i40e driver in Rx, ixgbe driver in Tx, and then achieve 7-9% performance improvement by mbufs recycle mode.

Difference between mbuf recycle, ZC API used in mempool and general path
For general path:
Rx: 32 pkts memcpy from mempool cache to rx_sw_ring
Tx: 32 pkts memcpy from tx_sw_ring to temporary variable + 32 pkts memcpy from temporary variable to mempool cache
For ZC API used in mempool:
Rx: 32 pkts memcpy from mempool cache to rx_sw_ring
Tx: 32 pkts memcpy from tx_sw_ring to zero-copy mempool cache
Refer link: http://patches.dpdk.org/project/dpdk/patch/[email protected]/
For mbufs recycle:
Rx/Tx: 32 pkts memcpy from tx_sw_ring to rx_sw_ring
Thus we can see in the one loop, compared to general path, mbufs recycle mode reduces 32+32=64 pkts memcpy; Compared to ZC API used in mempool, we can see mbufs recycle mode reduce 32 pkts memcpy in each loop.
So, mbufs recycle has its own benefits.

Testing status:
(1) dpdk l3fwd test with multiple drivers:

    port 0: 82599 NIC   port 1: XL710 NIC

-------------------------------------------------------------
Without fast free With fast free
Thunderx2: +7.53% +13.54%
-------------------------------------------------------------

(2) dpdk l3fwd test with same driver:

    port 0 && 1: XL710 NIC

-------------------------------------------------------------
Without fast free With fast free
Ampere altra: +12.61% +11.42%
n1sdp: +8.30% +3.85%
x86-sse: +8.43% +3.72%
-------------------------------------------------------------

(3) Performance comparison with ZC_mempool used
port 0 && 1: XL710 NIC
with fast free
-------------------------------------------------------------
With recycle buffer With zc_mempool
Ampere altra: 11.42% 3.54%
-------------------------------------------------------------

Furthermore, we add recycle_mbuf engine in testpmd. Due to XL710 NIC has I/O bottleneck in testpmd in ampere altra, we can not see throughput change compared with I/O fwd engine. However, using record cmd in testpmd:
'$set record-burst-stats on'
we can see the ratio of 'Rx/Tx burst size of 32' is reduced. This indicate mbufs recycle can save CPU cycles.

V2:
1. Use data-plane API to enable direct-rearm (Konstantin, Honnappa)
2. Add 'txq_data_get' API to get txq info for Rx (Konstantin)
3. Use input parameter to enable direct rearm in l3fwd (Konstantin)
4. Add condition detection for direct rearm API (Morten, Andrew Rybchenko)

V3:
1. Seperate Rx and Tx operation with two APIs in direct-rearm (Konstantin)
2. Delete L3fwd change for direct rearm (Jerin)
3. enable direct rearm in ixgbe driver in Arm

v4:
1. Rename direct-rearm as buffer recycle. Based on this, function name and variable name are changed to let this mode more general for all drivers. (Konstantin, Morten)
2. Add ring wrapping check (Konstantin)

v5:
1. some change for ethdev API (Morten)
2. add support for avx2, sse, altivec path

v6:
1. fix ixgbe build issue in ppc
2. remove 'recycle_tx_mbufs_reuse' and 'recycle_rx_descriptors_refill' API wrapper (Tech Board meeting)
3. add recycle_mbufs engine in testpmd (Tech Board meeting)
4. add namespace in the functions related to mbufs recycle(Ferruh)

v7:
1. move 'rxq/txq data' pointers to the beginning of eth_dev structure, in order to keep them in the same cache line as rx/tx_burst function pointers (Morten)
2. add the extra description for 'rte_eth_recycle_mbufs' to show it can support feeding 1 Rx queue from 2 Tx queues in the same thread (Konstantin)
3. For i40e/ixgbe driver, make the previous copied buffers as invalid if there are Tx buffers refcnt > 1 or from unexpected mempool (Konstantin)
4. add check for the return value of 'rte_eth_recycle_rx_queue_info_get' in testpmd fwd engine (Morten)

v8:
1. add arm/x86 build option to fix ixgbe build issue in ppc

Feifei Wang (4):
ethdev: add API for mbufs recycle mode
net/i40e: implement mbufs recycle mode
net/ixgbe: implement mbufs recycle mode
app/testpmd: add recycle mbufs engine

 app/test-pmd/meson.build                      |   1 +
 app/test-pmd/recycle_mbufs.c                  |  58 ++++++
 app/test-pmd/testpmd.c                        |   1 +
 app/test-pmd/testpmd.h                        |   3 +
 doc/guides/rel_notes/release_23_11.rst        |  15 ++
 doc/guides/testpmd_app_ug/run_app.rst         |   1 +
 doc/guides/testpmd_app_ug/testpmd_funcs.rst   |   5 +-
 drivers/net/i40e/i40e_ethdev.c                |   1 +
 drivers/net/i40e/i40e_ethdev.h                |   2 +
 .../net/i40e/i40e_recycle_mbufs_vec_common.c  | 147 ++++++++++++++
 drivers/net/i40e/i40e_rxtx.c                  |  32 ++++
 drivers/net/i40e/i40e_rxtx.h                  |   4 +
 drivers/net/i40e/meson.build                  |   1 +
 drivers/net/ixgbe/ixgbe_ethdev.c              |   1 +
 drivers/net/ixgbe/ixgbe_ethdev.h              |   3 +
 .../ixgbe/ixgbe_recycle_mbufs_vec_common.c    | 143 ++++++++++++++
 drivers/net/ixgbe/ixgbe_rxtx.c                |  37 +++-
 drivers/net/ixgbe/ixgbe_rxtx.h                |   4 +
 drivers/net/ixgbe/meson.build                 |   3 +
 lib/ethdev/ethdev_driver.h                    |  10 +
 lib/ethdev/ethdev_private.c                   |   2 +
 lib/ethdev/rte_ethdev.c                       |  31 +++
 lib/ethdev/rte_ethdev.h                       | 181 ++++++++++++++++++
 lib/ethdev/rte_ethdev_core.h                  |  23 ++-
 lib/ethdev/version.map                        |   4 +
 25 files changed, 704 insertions(+), 9 deletions(-)
 create mode 100644 app/test-pmd/recycle_mbufs.c
 create mode 100644 drivers/net/i40e/i40e_recycle_mbufs_vec_common.c
 create mode 100644 drivers/net/ixgbe/ixgbe_recycle_mbufs_vec_common.c
Signature · 1 line
2.25.1
#80 lore raw

PATCH v8 1/4 ethdev: add API for mbufs recycle mode

Add 'rte_eth_recycle_rx_queue_info_get' and 'rte_eth_recycle_mbufs' APIs to recycle used mbufs from a transmit queue of an Ethernet device, and move these mbufs into a mbuf ring for a receive queue of an Ethernet device. This can bypass mempool 'put/get' operations hence saving CPU cycles.

For each recycling mbufs, the rte_eth_recycle_mbufs() function performs the following operations:
- Copy used *rte_mbuf* buffer pointers from Tx mbuf ring into Rx mbuf ring.
- Replenish the Rx descriptors with the recycling *rte_mbuf* mbufs freed from the Tx mbuf ring.

---
 doc/guides/rel_notes/release_23_11.rst |  15 ++
 lib/ethdev/ethdev_driver.h             |  10 ++
 lib/ethdev/ethdev_private.c            |   2 +
 lib/ethdev/rte_ethdev.c                |  31 +++++
 lib/ethdev/rte_ethdev.h                | 181 +++++++++++++++++++++++++
 lib/ethdev/rte_ethdev_core.h           |  23 +++-
 lib/ethdev/version.map                 |   4 +
 7 files changed, 260 insertions(+), 6 deletions(-)
doc/guides/rel_notes/release_23_11.rst +15 -0
index 6b4dd21fd0..fd16d267ae 100644
@@ -55,6 +55,13 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Add mbufs recycling support. **
+
+  Added ``rte_eth_recycle_rx_queue_info_get`` and ``rte_eth_recycle_mbufs``
+  APIs which allow the user to copy used mbufs from the Tx mbuf ring
+  into the Rx mbuf ring. This feature supports the case that the Rx Ethernet
+  device is different from the Tx Ethernet device with respective driver
+  callback functions in ``rte_eth_recycle_mbufs``.
 
 Removed Items
 -------------
@@ -100,6 +107,14 @@ ABI Changes
    Also, make sure to start the actual text at the margin.
    =======================================================
 
+* ethdev: Added ``recycle_tx_mbufs_reuse`` and ``recycle_rx_descriptors_refill``
+  fields to ``rte_eth_dev`` structure.
+
+* ethdev: Structure ``rte_eth_fp_ops`` was affected to add
+  ``recycle_tx_mbufs_reuse`` and ``recycle_rx_descriptors_refill``
+  fields, to move ``rxq`` and ``txq`` fields, to change the size of
+  ``reserved1`` and ``reserved2`` fields.
+
 
 Known Issues
 ------------
lib/ethdev/ethdev_driver.h +10 -0
index 980f837ab6..b0c55a8523 100644
@@ -58,6 +58,10 @@ struct rte_eth_dev {
 	eth_rx_descriptor_status_t rx_descriptor_status;
 	/** Check the status of a Tx descriptor */
 	eth_tx_descriptor_status_t tx_descriptor_status;
+	/** Pointer to PMD transmit mbufs reuse function */
+	eth_recycle_tx_mbufs_reuse_t recycle_tx_mbufs_reuse;
+	/** Pointer to PMD receive descriptors refill function */
+	eth_recycle_rx_descriptors_refill_t recycle_rx_descriptors_refill;
 
 	/**
 	 * Device data that is shared between primary and secondary processes
@@ -507,6 +511,10 @@ typedef void (*eth_rxq_info_get_t)(struct rte_eth_dev *dev,
 typedef void (*eth_txq_info_get_t)(struct rte_eth_dev *dev,
 	uint16_t tx_queue_id, struct rte_eth_txq_info *qinfo);
 
+typedef void (*eth_recycle_rxq_info_get_t)(struct rte_eth_dev *dev,
+	uint16_t rx_queue_id,
+	struct rte_eth_recycle_rxq_info *recycle_rxq_info);
+
 typedef int (*eth_burst_mode_get_t)(struct rte_eth_dev *dev,
 	uint16_t queue_id, struct rte_eth_burst_mode *mode);
 
@@ -1250,6 +1258,8 @@ struct eth_dev_ops {
 	eth_rxq_info_get_t         rxq_info_get;
 	/** Retrieve Tx queue information */
 	eth_txq_info_get_t         txq_info_get;
+	/** Retrieve mbufs recycle Rx queue information */
+	eth_recycle_rxq_info_get_t recycle_rxq_info_get;
 	eth_burst_mode_get_t       rx_burst_mode_get; /**< Get Rx burst mode */
 	eth_burst_mode_get_t       tx_burst_mode_get; /**< Get Tx burst mode */
 	eth_fw_version_get_t       fw_version_get; /**< Get firmware version */
lib/ethdev/ethdev_private.c +2 -0
index 14ec8c6ccf..f8ab64f195 100644
@@ -277,6 +277,8 @@ eth_dev_fp_ops_setup(struct rte_eth_fp_ops *fpo,
 	fpo->rx_queue_count = dev->rx_queue_count;
 	fpo->rx_descriptor_status = dev->rx_descriptor_status;
 	fpo->tx_descriptor_status = dev->tx_descriptor_status;
+	fpo->recycle_tx_mbufs_reuse = dev->recycle_tx_mbufs_reuse;
+	fpo->recycle_rx_descriptors_refill = dev->recycle_rx_descriptors_refill;
 
 	fpo->rxq.data = dev->data->rx_queues;
 	fpo->rxq.clbk = (void **)(uintptr_t)dev->post_rx_burst_cbs;
lib/ethdev/rte_ethdev.c +31 -0
index 0840d2b594..ea89a101a1 100644
@@ -5876,6 +5876,37 @@ rte_eth_tx_queue_info_get(uint16_t port_id, uint16_t queue_id,
 	return 0;
 }
 
+int
+rte_eth_recycle_rx_queue_info_get(uint16_t port_id, uint16_t queue_id,
+		struct rte_eth_recycle_rxq_info *recycle_rxq_info)
+{
+	struct rte_eth_dev *dev;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+	dev = &rte_eth_devices[port_id];
+
+	if (queue_id >= dev->data->nb_rx_queues) {
+		RTE_ETHDEV_LOG(ERR, "Invalid Rx queue_id=%u\n", queue_id);
+		return -EINVAL;
+	}
+
+	if (dev->data->rx_queues == NULL ||
+			dev->data->rx_queues[queue_id] == NULL) {
+		RTE_ETHDEV_LOG(ERR,
+			   "Rx queue %"PRIu16" of device with port_id=%"
+			   PRIu16" has not been setup\n",
+			   queue_id, port_id);
+		return -EINVAL;
+	}
+
+	if (*dev->dev_ops->recycle_rxq_info_get == NULL)
+		return -ENOTSUP;
+
+	dev->dev_ops->recycle_rxq_info_get(dev, queue_id, recycle_rxq_info);
+
+	return 0;
+}
+
 int
 rte_eth_rx_burst_mode_get(uint16_t port_id, uint16_t queue_id,
 			  struct rte_eth_burst_mode *mode)
lib/ethdev/rte_ethdev.h +181 -0

199 lines not on this page · raw message

lib/ethdev/rte_ethdev_core.h +17 -6
index 46e9721e07..a24ad7a6b2 100644
@@ -55,6 +55,13 @@ typedef int (*eth_rx_descriptor_status_t)(void *rxq, uint16_t offset);
 /** @internal Check the status of a Tx descriptor */
 typedef int (*eth_tx_descriptor_status_t)(void *txq, uint16_t offset);
 
+/** @internal Copy used mbufs from Tx mbuf ring into Rx mbuf ring */
+typedef uint16_t (*eth_recycle_tx_mbufs_reuse_t)(void *txq,
+		struct rte_eth_recycle_rxq_info *recycle_rxq_info);
+
+/** @internal Refill Rx descriptors with the recycling mbufs */
+typedef void (*eth_recycle_rx_descriptors_refill_t)(void *rxq, uint16_t nb);
+
 /**
  * @internal
  * Structure used to hold opaque pointers to internal ethdev Rx/Tx
@@ -83,15 +90,17 @@ struct rte_eth_fp_ops {
 	 * Rx fast-path functions and related data.
 	 * 64-bit systems: occupies first 64B line
 	 */
+	/** Rx queues data. */
+	struct rte_ethdev_qdata rxq;
 	/** PMD receive function. */
 	eth_rx_burst_t rx_pkt_burst;
 	/** Get the number of used Rx descriptors. */
 	eth_rx_queue_count_t rx_queue_count;
 	/** Check the status of a Rx descriptor. */
 	eth_rx_descriptor_status_t rx_descriptor_status;
-	/** Rx queues data. */
-	struct rte_ethdev_qdata rxq;
-	uintptr_t reserved1[3];
+	/** Refill Rx descriptors with the recycling mbufs. */
+	eth_recycle_rx_descriptors_refill_t recycle_rx_descriptors_refill;
+	uintptr_t reserved1[2];
 	/**@}*/
 
 	/**@{*/
@@ -99,15 +108,17 @@ struct rte_eth_fp_ops {
 	 * Tx fast-path functions and related data.
 	 * 64-bit systems: occupies second 64B line
 	 */
+	/** Tx queues data. */
+	struct rte_ethdev_qdata txq;
 	/** PMD transmit function. */
 	eth_tx_burst_t tx_pkt_burst;
 	/** PMD transmit prepare function. */
 	eth_tx_prep_t tx_pkt_prepare;
 	/** Check the status of a Tx descriptor. */
 	eth_tx_descriptor_status_t tx_descriptor_status;
-	/** Tx queues data. */
-	struct rte_ethdev_qdata txq;
-	uintptr_t reserved2[3];
+	/** Copy used mbufs from Tx mbuf ring into Rx. */
+	eth_recycle_tx_mbufs_reuse_t recycle_tx_mbufs_reuse;
+	uintptr_t reserved2[2];
 	/**@}*/
 
 } __rte_cache_aligned;
lib/ethdev/version.map +4 -0
index b965d6aa52..e52c1563b4 100644
@@ -312,6 +312,10 @@ EXPERIMENTAL {
 	rte_flow_async_action_list_handle_query_update;
 	rte_flow_async_actions_update;
 	rte_flow_restore_info_dynflag;
+
+	# added in 23.11
+	rte_eth_recycle_mbufs;
+	rte_eth_recycle_rx_queue_info_get;
 };
 
 INTERNAL {
Signature · 1 line
2.25.1
#81 lore raw

PATCH v8 2/4 net/i40e: implement mbufs recycle mode

In reply to Feifei Wang

Define specific function implementation for i40e driver. Currently, mbufs recycle mode can support 128bit vector path and avx2 path. And can be enabled both in fast free and no fast free mode.

---
 drivers/net/i40e/i40e_ethdev.c                |   1 +
 drivers/net/i40e/i40e_ethdev.h                |   2 +
 .../net/i40e/i40e_recycle_mbufs_vec_common.c  | 147 ++++++++++++++++++
 drivers/net/i40e/i40e_rxtx.c                  |  32 ++++
 drivers/net/i40e/i40e_rxtx.h                  |   4 +
 drivers/net/i40e/meson.build                  |   1 +
 6 files changed, 187 insertions(+)
 create mode 100644 drivers/net/i40e/i40e_recycle_mbufs_vec_common.c
drivers/net/i40e/i40e_ethdev.c +1 -0
index 8271bbb394..50ba9aac94 100644
@@ -496,6 +496,7 @@ static const struct eth_dev_ops i40e_eth_dev_ops = {
 	.flow_ops_get                 = i40e_dev_flow_ops_get,
 	.rxq_info_get                 = i40e_rxq_info_get,
 	.txq_info_get                 = i40e_txq_info_get,
+	.recycle_rxq_info_get         = i40e_recycle_rxq_info_get,
 	.rx_burst_mode_get            = i40e_rx_burst_mode_get,
 	.tx_burst_mode_get            = i40e_tx_burst_mode_get,
 	.timesync_enable              = i40e_timesync_enable,
drivers/net/i40e/i40e_ethdev.h +2 -0
index 6f65d5e0ac..af758798e1 100644
@@ -1355,6 +1355,8 @@ void i40e_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 	struct rte_eth_rxq_info *qinfo);
 void i40e_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 	struct rte_eth_txq_info *qinfo);
+void i40e_recycle_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
+	struct rte_eth_recycle_rxq_info *recycle_rxq_info);
 int i40e_rx_burst_mode_get(struct rte_eth_dev *dev, uint16_t queue_id,
 			   struct rte_eth_burst_mode *mode);
 int i40e_tx_burst_mode_get(struct rte_eth_dev *dev, uint16_t queue_id,
drivers/net/i40e/i40e_recycle_mbufs_vec_common.c new file +147 -0

147 lines not on this page · raw message

drivers/net/i40e/i40e_rxtx.c +32 -0
index b4f65b58fa..a9c9eb331c 100644
@@ -3199,6 +3199,30 @@ i40e_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 	qinfo->conf.offloads = txq->offloads;
 }
 
+void
+i40e_recycle_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
+	struct rte_eth_recycle_rxq_info *recycle_rxq_info)
+{
+	struct i40e_rx_queue *rxq;
+	struct i40e_adapter *ad =
+		I40E_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private);
+
+	rxq = dev->data->rx_queues[queue_id];
+
+	recycle_rxq_info->mbuf_ring = (void *)rxq->sw_ring;
+	recycle_rxq_info->mp = rxq->mp;
+	recycle_rxq_info->mbuf_ring_size = rxq->nb_rx_desc;
+	recycle_rxq_info->receive_tail = &rxq->rx_tail;
+
+	if (ad->rx_vec_allowed) {
+		recycle_rxq_info->refill_requirement = RTE_I40E_RXQ_REARM_THRESH;
+		recycle_rxq_info->refill_head = &rxq->rxrearm_start;
+	} else {
+		recycle_rxq_info->refill_requirement = rxq->rx_free_thresh;
+		recycle_rxq_info->refill_head = &rxq->rx_free_trigger;
+	}
+}
+
 #ifdef RTE_ARCH_X86
 static inline bool
 get_avx_supported(bool request_avx512)
@@ -3293,6 +3317,8 @@ i40e_set_rx_function(struct rte_eth_dev *dev)
 				dev->rx_pkt_burst = ad->rx_use_avx2 ?
 					i40e_recv_scattered_pkts_vec_avx2 :
 					i40e_recv_scattered_pkts_vec;
+				dev->recycle_rx_descriptors_refill =
+					i40e_recycle_rx_descriptors_refill_vec;
 			}
 		} else {
 			if (ad->rx_use_avx512) {
@@ -3311,9 +3337,12 @@ i40e_set_rx_function(struct rte_eth_dev *dev)
 				dev->rx_pkt_burst = ad->rx_use_avx2 ?
 					i40e_recv_pkts_vec_avx2 :
 					i40e_recv_pkts_vec;
+				dev->recycle_rx_descriptors_refill =
+					i40e_recycle_rx_descriptors_refill_vec;
 			}
 		}
 #else /* RTE_ARCH_X86 */
+		dev->recycle_rx_descriptors_refill = i40e_recycle_rx_descriptors_refill_vec;
 		if (dev->data->scattered_rx) {
 			PMD_INIT_LOG(DEBUG,
 				     "Using Vector Scattered Rx (port %d).",
@@ -3481,15 +3510,18 @@ i40e_set_tx_function(struct rte_eth_dev *dev)
 				dev->tx_pkt_burst = ad->tx_use_avx2 ?
 						    i40e_xmit_pkts_vec_avx2 :
 						    i40e_xmit_pkts_vec;
+				dev->recycle_tx_mbufs_reuse = i40e_recycle_tx_mbufs_reuse_vec;
 			}
 #else /* RTE_ARCH_X86 */
 			PMD_INIT_LOG(DEBUG, "Using Vector Tx (port %d).",
 				     dev->data->port_id);
 			dev->tx_pkt_burst = i40e_xmit_pkts_vec;
+			dev->recycle_tx_mbufs_reuse = i40e_recycle_tx_mbufs_reuse_vec;
 #endif /* RTE_ARCH_X86 */
 		} else {
 			PMD_INIT_LOG(DEBUG, "Simple tx finally be used.");
 			dev->tx_pkt_burst = i40e_xmit_pkts_simple;
+			dev->recycle_tx_mbufs_reuse = i40e_recycle_tx_mbufs_reuse_vec;
 		}
 		dev->tx_pkt_prepare = i40e_simple_prep_pkts;
 	} else {
drivers/net/i40e/i40e_rxtx.h +4 -0
index a8686224e5..b191f23e1f 100644
@@ -236,6 +236,10 @@ uint32_t i40e_dev_rx_queue_count(void *rx_queue);
 int i40e_dev_rx_descriptor_status(void *rx_queue, uint16_t offset);
 int i40e_dev_tx_descriptor_status(void *tx_queue, uint16_t offset);
 
+uint16_t i40e_recycle_tx_mbufs_reuse_vec(void *tx_queue,
+		struct rte_eth_recycle_rxq_info *recycle_rxq_info);
+void i40e_recycle_rx_descriptors_refill_vec(void *rx_queue, uint16_t nb_mbufs);
+
 uint16_t i40e_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,
 			    uint16_t nb_pkts);
 uint16_t i40e_recv_scattered_pkts_vec(void *rx_queue,
drivers/net/i40e/meson.build +1 -0
index 8e53b87a65..3b1a233c84 100644
@@ -34,6 +34,7 @@ sources = files(
         'i40e_tm.c',
         'i40e_hash.c',
         'i40e_vf_representor.c',
+	'i40e_recycle_mbufs_vec_common.c',
         'rte_pmd_i40e.c',
 )
 
Signature · 1 line
2.25.1
#82 lore raw

PATCH v8 3/4 net/ixgbe: implement mbufs recycle mode

In reply to Feifei Wang

Define specific function implementation for ixgbe driver. Currently, recycle buffer mode can support 128bit vector path. And can be enabled both in fast free and no fast free mode.

---
 drivers/net/ixgbe/ixgbe_ethdev.c              |   1 +
 drivers/net/ixgbe/ixgbe_ethdev.h              |   3 +
 .../ixgbe/ixgbe_recycle_mbufs_vec_common.c    | 143 ++++++++++++++++++
 drivers/net/ixgbe/ixgbe_rxtx.c                |  37 ++++-
 drivers/net/ixgbe/ixgbe_rxtx.h                |   4 +
 drivers/net/ixgbe/meson.build                 |   3 +
 6 files changed, 189 insertions(+), 2 deletions(-)
 create mode 100644 drivers/net/ixgbe/ixgbe_recycle_mbufs_vec_common.c
drivers/net/ixgbe/ixgbe_ethdev.c +1 -0
index 14a7d571e0..ea4c9dd561 100644
@@ -543,6 +543,7 @@ static const struct eth_dev_ops ixgbe_eth_dev_ops = {
 	.set_mc_addr_list     = ixgbe_dev_set_mc_addr_list,
 	.rxq_info_get         = ixgbe_rxq_info_get,
 	.txq_info_get         = ixgbe_txq_info_get,
+	.recycle_rxq_info_get = ixgbe_recycle_rxq_info_get,
 	.timesync_enable      = ixgbe_timesync_enable,
 	.timesync_disable     = ixgbe_timesync_disable,
 	.timesync_read_rx_timestamp = ixgbe_timesync_read_rx_timestamp,
drivers/net/ixgbe/ixgbe_ethdev.h +3 -0
index 1291e9099c..22fc3be3d8 100644
@@ -626,6 +626,9 @@ void ixgbe_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 void ixgbe_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 	struct rte_eth_txq_info *qinfo);
 
+void ixgbe_recycle_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
+		struct rte_eth_recycle_rxq_info *recycle_rxq_info);
+
 int ixgbevf_dev_rx_init(struct rte_eth_dev *dev);
 
 void ixgbevf_dev_tx_init(struct rte_eth_dev *dev);
drivers/net/ixgbe/ixgbe_recycle_mbufs_vec_common.c new file +143 -0

143 lines not on this page · raw message

drivers/net/ixgbe/ixgbe_rxtx.c +35 -2
index 954ef241a0..90b0a7004f 100644
@@ -2552,6 +2552,9 @@ ixgbe_set_tx_function(struct rte_eth_dev *dev, struct ixgbe_tx_queue *txq)
 				(rte_eal_process_type() != RTE_PROC_PRIMARY ||
 					ixgbe_txq_vec_setup(txq) == 0)) {
 			PMD_INIT_LOG(DEBUG, "Vector tx enabled.");
+#if defined(RTE_ARCH_X86) || defined(RTE_ARCH_ARM)
+			dev->recycle_tx_mbufs_reuse = ixgbe_recycle_tx_mbufs_reuse_vec;
+#endif
 			dev->tx_pkt_burst = ixgbe_xmit_pkts_vec;
 		} else
 		dev->tx_pkt_burst = ixgbe_xmit_pkts_simple;
@@ -4890,7 +4893,10 @@ ixgbe_set_rx_function(struct rte_eth_dev *dev)
 			PMD_INIT_LOG(DEBUG, "Using Vector Scattered Rx "
 					    "callback (port=%d).",
 				     dev->data->port_id);
-
+#if defined(RTE_ARCH_X86) || defined(RTE_ARCH_ARM)
+			dev->recycle_rx_descriptors_refill =
+				ixgbe_recycle_rx_descriptors_refill_vec;
+#endif
 			dev->rx_pkt_burst = ixgbe_recv_scattered_pkts_vec;
 		} else if (adapter->rx_bulk_alloc_allowed) {
 			PMD_INIT_LOG(DEBUG, "Using a Scattered with bulk "
@@ -4919,7 +4925,9 @@ ixgbe_set_rx_function(struct rte_eth_dev *dev)
 				    "burst size no less than %d (port=%d).",
 			     RTE_IXGBE_DESCS_PER_LOOP,
 			     dev->data->port_id);
-
+#if defined(RTE_ARCH_X86) || defined(RTE_ARCH_ARM)
+		dev->recycle_rx_descriptors_refill = ixgbe_recycle_rx_descriptors_refill_vec;
+#endif
 		dev->rx_pkt_burst = ixgbe_recv_pkts_vec;
 	} else if (adapter->rx_bulk_alloc_allowed) {
 		PMD_INIT_LOG(DEBUG, "Rx Burst Bulk Alloc Preconditions are "
@@ -5691,6 +5699,31 @@ ixgbe_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 	qinfo->conf.tx_deferred_start = txq->tx_deferred_start;
 }
 
+void
+ixgbe_recycle_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
+	struct rte_eth_recycle_rxq_info *recycle_rxq_info)
+{
+	struct ixgbe_rx_queue *rxq;
+	struct ixgbe_adapter *adapter = dev->data->dev_private;
+
+	rxq = dev->data->rx_queues[queue_id];
+
+	recycle_rxq_info->mbuf_ring = (void *)rxq->sw_ring;
+	recycle_rxq_info->mp = rxq->mb_pool;
+	recycle_rxq_info->mbuf_ring_size = rxq->nb_rx_desc;
+	recycle_rxq_info->receive_tail = &rxq->rx_tail;
+
+	if (adapter->rx_vec_allowed) {
+#if defined(RTE_ARCH_X86) || defined(RTE_ARCH_ARM)
+		recycle_rxq_info->refill_requirement = RTE_IXGBE_RXQ_REARM_THRESH;
+		recycle_rxq_info->refill_head = &rxq->rxrearm_start;
+#endif
+	} else {
+		recycle_rxq_info->refill_requirement = rxq->rx_free_thresh;
+		recycle_rxq_info->refill_head = &rxq->rx_free_trigger;
+	}
+}
+
 /*
  * [VF] Initializes Receive Unit.
  */
drivers/net/ixgbe/ixgbe_rxtx.h +4 -0
index 668a5b9814..ee89c89929 100644
@@ -295,6 +295,10 @@ int ixgbe_dev_tx_done_cleanup(void *tx_queue, uint32_t free_cnt);
 extern const uint32_t ptype_table[IXGBE_PACKET_TYPE_MAX];
 extern const uint32_t ptype_table_tn[IXGBE_PACKET_TYPE_TN_MAX];
 
+uint16_t ixgbe_recycle_tx_mbufs_reuse_vec(void *tx_queue,
+		struct rte_eth_recycle_rxq_info *recycle_rxq_info);
+void ixgbe_recycle_rx_descriptors_refill_vec(void *rx_queue, uint16_t nb_mbufs);
+
 uint16_t ixgbe_xmit_fixed_burst_vec(void *tx_queue, struct rte_mbuf **tx_pkts,
 				    uint16_t nb_pkts);
 int ixgbe_txq_vec_setup(struct ixgbe_tx_queue *txq);
drivers/net/ixgbe/meson.build +3 -0
index a18908ef7c..52e77190f7 100644
@@ -17,6 +17,7 @@ sources = files(
         'ixgbe_rxtx.c',
         'ixgbe_tm.c',
         'ixgbe_vf_representor.c',
+	'ixgbe_recycle_mbufs_vec_common.c',
         'rte_pmd_ixgbe.c',
 )
 
@@ -26,11 +27,13 @@ deps += ['hash', 'security']
 
 if arch_subdir == 'x86'
     sources += files('ixgbe_rxtx_vec_sse.c')
+    sources += files('ixgbe_recycle_mbufs_vec_common.c')
     if is_windows and cc.get_id() != 'clang'
         cflags += ['-fno-asynchronous-unwind-tables']
     endif
 elif arch_subdir == 'arm'
     sources += files('ixgbe_rxtx_vec_neon.c')
+    sources += files('ixgbe_recycle_mbufs_vec_common.c')
 endif
 
 includes += include_directories('base')
Signature · 1 line
2.25.1
#83 lore raw

PATCH v8 4/4 app/testpmd: add recycle mbufs engine

In reply to Feifei Wang

Add recycle mbufs engine for testpmd. This engine forward pkts with I/O forward mode. But enable mbufs recycle feature to recycle used txq mbufs for rxq mbuf ring, which can bypass mempool path and save CPU cycles.

---
 app/test-pmd/meson.build                    |  1 +
 app/test-pmd/recycle_mbufs.c                | 58 +++++++++++++++++++++
 app/test-pmd/testpmd.c                      |  1 +
 app/test-pmd/testpmd.h                      |  3 ++
 doc/guides/testpmd_app_ug/run_app.rst       |  1 +
 doc/guides/testpmd_app_ug/testpmd_funcs.rst |  5 +-
 6 files changed, 68 insertions(+), 1 deletion(-)
 create mode 100644 app/test-pmd/recycle_mbufs.c
app/test-pmd/meson.build +1 -0
index d2e3f60892..6e5f067274 100644
@@ -22,6 +22,7 @@ sources = files(
         'macswap.c',
         'noisy_vnf.c',
         'parameters.c',
+	'recycle_mbufs.c',
         'rxonly.c',
         'shared_rxq_fwd.c',
         'testpmd.c',
app/test-pmd/recycle_mbufs.c new file +58 -0

58 lines not on this page · raw message

app/test-pmd/testpmd.c +1 -0
index 938ca035d4..5d0f9ca119 100644
@@ -199,6 +199,7 @@ struct fwd_engine * fwd_engines[] = {
 	&icmp_echo_engine,
 	&noisy_vnf_engine,
 	&five_tuple_swap_fwd_engine,
+	&recycle_mbufs_engine,
 #ifdef RTE_LIBRTE_IEEE1588
 	&ieee1588_fwd_engine,
 #endif
app/test-pmd/testpmd.h +3 -0
index f1df6a8faf..0eb8d7883a 100644
@@ -188,6 +188,8 @@ struct fwd_stream {
 	struct pkt_burst_stats rx_burst_stats;
 	struct pkt_burst_stats tx_burst_stats;
 	struct fwd_lcore *lcore; /**< Lcore being scheduled. */
+	/**< Rx queue information for recycling mbufs */
+	struct rte_eth_recycle_rxq_info recycle_rxq_info;
 };
 
 /**
@@ -449,6 +451,7 @@ extern struct fwd_engine csum_fwd_engine;
 extern struct fwd_engine icmp_echo_engine;
 extern struct fwd_engine noisy_vnf_engine;
 extern struct fwd_engine five_tuple_swap_fwd_engine;
+extern struct fwd_engine recycle_mbufs_engine;
 #ifdef RTE_LIBRTE_IEEE1588
 extern struct fwd_engine ieee1588_fwd_engine;
 #endif
doc/guides/testpmd_app_ug/run_app.rst +1 -0
index 6e9c552e76..24a086401e 100644
@@ -232,6 +232,7 @@ The command line options are:
        noisy
        5tswap
        shared-rxq
+       recycle_mbufs
 
 *   ``--rss-ip``
 
doc/guides/testpmd_app_ug/testpmd_funcs.rst +4 -1
index a182479ab2..aef4de3e0e 100644
@@ -318,7 +318,7 @@ set fwd
 Set the packet forwarding mode::
 
    testpmd> set fwd (io|mac|macswap|flowgen| \
-                     rxonly|txonly|csum|icmpecho|noisy|5tswap|shared-rxq) (""|retry)
+                     rxonly|txonly|csum|icmpecho|noisy|5tswap|shared-rxq|recycle_mbufs) (""|retry)
 
 ``retry`` can be specified for forwarding engines except ``rx_only``.
 
@@ -364,6 +364,9 @@ The available information categories are:
 * ``shared-rxq``: Receive only for shared Rx queue.
   Resolve packet source port from mbuf and update stream statistics accordingly.
 
+* ``recycle_mbufs``:  Recycle Tx queue used mbufs for Rx queue mbuf ring.
+  This mode uses fast path mbuf recycle feature and forwards packets in I/O mode.
+
 Example::
 
    testpmd> set fwd rxonly
Signature · 1 line
2.25.1
#84 lore raw

v9 cover Recycle mbufs from Tx queue into Rx queue

In reply to Feifei Wang

Currently, the transmit side frees the buffers into the lcore cache and the receive side allocates buffers from the lcore cache. The transmit side typically frees 32 buffers resulting in 32*8=256B of stores to lcore cache. The receive side allocates 32 buffers and stores them in the receive side software ring, resulting in 32*8=256B of stores and 256B of load from the lcore cache.

This patch proposes a mechanism to avoid freeing to/allocating from the lcore cache. i.e. the receive side will free the buffers from transmit side directly into its software ring. This will avoid the 256B of loads and stores introduced by the lcore cache. It also frees up the cache lines used by the lcore cache. And we can call this mode as mbufs recycle mode.

In the latest version, mbufs recycle mode is packaged as a separate API.
This allows for the users to change rxq/txq pairing in real time in data plane, according to the analysis of the packet flow by the application, for example:
-----------------------------------------------------------------------
Step 1: upper application analyse the flow direction
Step 2: recycle_rxq_info = rte_eth_recycle_rx_queue_info_get(rx_portid, rx_queueid) Step 3: rte_eth_recycle_mbufs(rx_portid, rx_queueid, tx_portid, tx_queueid, recycle_rxq_info); Step 4: rte_eth_rx_burst(rx_portid,rx_queueid);
Step 5: rte_eth_tx_burst(tx_portid,tx_queueid);
-----------------------------------------------------------------------
Above can support user to change rxq/txq pairing at run-time and user does not need to know the direction of flow in advance. This can effectively expand mbufs recycle mode's use scenarios.

Furthermore, mbufs recycle mode is no longer limited to the same pmd,
it can support moving mbufs between different vendor pmds, even can put the mbufs anywhere into your Rx mbuf ring as long as the address of the mbuf ring can be provided. In the latest version, we enable mbufs recycle mode in i40e pmd and ixgbe pmd, and also try to use i40e driver in Rx, ixgbe driver in Tx, and then achieve 7-9% performance improvement by mbufs recycle mode.

Difference between mbuf recycle, ZC API used in mempool and general path
For general path:
Rx: 32 pkts memcpy from mempool cache to rx_sw_ring
Tx: 32 pkts memcpy from tx_sw_ring to temporary variable + 32 pkts memcpy from temporary variable to mempool cache
For ZC API used in mempool:
Rx: 32 pkts memcpy from mempool cache to rx_sw_ring
Tx: 32 pkts memcpy from tx_sw_ring to zero-copy mempool cache
Refer link: http://patches.dpdk.org/project/dpdk/patch/[email protected]/
For mbufs recycle:
Rx/Tx: 32 pkts memcpy from tx_sw_ring to rx_sw_ring
Thus we can see in the one loop, compared to general path, mbufs recycle mode reduces 32+32=64 pkts memcpy; Compared to ZC API used in mempool, we can see mbufs recycle mode reduce 32 pkts memcpy in each loop.
So, mbufs recycle has its own benefits.

Testing status:
(1) dpdk l3fwd test with multiple drivers:

    port 0: 82599 NIC   port 1: XL710 NIC

-------------------------------------------------------------
Without fast free With fast free
Thunderx2: +7.53% +13.54%
-------------------------------------------------------------

(2) dpdk l3fwd test with same driver:

    port 0 && 1: XL710 NIC

-------------------------------------------------------------
Without fast free With fast free
Ampere altra: +12.61% +11.42%
n1sdp: +8.30% +3.85%
x86-sse: +8.43% +3.72%
-------------------------------------------------------------

(3) Performance comparison with ZC_mempool used
port 0 && 1: XL710 NIC
with fast free
-------------------------------------------------------------
With recycle buffer With zc_mempool
Ampere altra: 11.42% 3.54%
-------------------------------------------------------------

Furthermore, we add recycle_mbuf engine in testpmd. Due to XL710 NIC has I/O bottleneck in testpmd in ampere altra, we can not see throughput change compared with I/O fwd engine. However, using record cmd in testpmd:
'$set record-burst-stats on'
we can see the ratio of 'Rx/Tx burst size of 32' is reduced. This indicate mbufs recycle can save CPU cycles.

V2:
1. Use data-plane API to enable direct-rearm (Konstantin, Honnappa)
2. Add 'txq_data_get' API to get txq info for Rx (Konstantin)
3. Use input parameter to enable direct rearm in l3fwd (Konstantin)
4. Add condition detection for direct rearm API (Morten, Andrew Rybchenko)

V3:
1. Seperate Rx and Tx operation with two APIs in direct-rearm (Konstantin)
2. Delete L3fwd change for direct rearm (Jerin)
3. enable direct rearm in ixgbe driver in Arm

v4:
1. Rename direct-rearm as buffer recycle. Based on this, function name and variable name are changed to let this mode more general for all drivers. (Konstantin, Morten)
2. Add ring wrapping check (Konstantin)

v5:
1. some change for ethdev API (Morten)
2. add support for avx2, sse, altivec path

v6:
1. fix ixgbe build issue in ppc
2. remove 'recycle_tx_mbufs_reuse' and 'recycle_rx_descriptors_refill' API wrapper (Tech Board meeting)
3. add recycle_mbufs engine in testpmd (Tech Board meeting)
4. add namespace in the functions related to mbufs recycle(Ferruh)

v7:
1. move 'rxq/txq data' pointers to the beginning of eth_dev structure, in order to keep them in the same cache line as rx/tx_burst function pointers (Morten)
2. add the extra description for 'rte_eth_recycle_mbufs' to show it can support feeding 1 Rx queue from 2 Tx queues in the same thread (Konstantin)
3. For i40e/ixgbe driver, make the previous copied buffers as invalid if there are Tx buffers refcnt > 1 or from unexpected mempool (Konstantin)
4. add check for the return value of 'rte_eth_recycle_rx_queue_info_get' in testpmd fwd engine (Morten)

v8:
1. add arm/x86 build option to fix ixgbe build issue in ppc

v9:
1. delete duplicate file name for ixgbe

Feifei Wang (4):
ethdev: add API for mbufs recycle mode
net/i40e: implement mbufs recycle mode
net/ixgbe: implement mbufs recycle mode
app/testpmd: add recycle mbufs engine

 app/test-pmd/meson.build                      |   1 +
 app/test-pmd/recycle_mbufs.c                  |  58 ++++++
 app/test-pmd/testpmd.c                        |   1 +
 app/test-pmd/testpmd.h                        |   3 +
 doc/guides/rel_notes/release_23_11.rst        |  15 ++
 doc/guides/testpmd_app_ug/run_app.rst         |   1 +
 doc/guides/testpmd_app_ug/testpmd_funcs.rst   |   5 +-
 drivers/net/i40e/i40e_ethdev.c                |   1 +
 drivers/net/i40e/i40e_ethdev.h                |   2 +
 .../net/i40e/i40e_recycle_mbufs_vec_common.c  | 147 ++++++++++++++
 drivers/net/i40e/i40e_rxtx.c                  |  32 ++++
 drivers/net/i40e/i40e_rxtx.h                  |   4 +
 drivers/net/i40e/meson.build                  |   1 +
 drivers/net/ixgbe/ixgbe_ethdev.c              |   1 +
 drivers/net/ixgbe/ixgbe_ethdev.h              |   3 +
 .../ixgbe/ixgbe_recycle_mbufs_vec_common.c    | 143 ++++++++++++++
 drivers/net/ixgbe/ixgbe_rxtx.c                |  37 +++-
 drivers/net/ixgbe/ixgbe_rxtx.h                |   4 +
 drivers/net/ixgbe/meson.build                 |   2 +
 lib/ethdev/ethdev_driver.h                    |  10 +
 lib/ethdev/ethdev_private.c                   |   2 +
 lib/ethdev/rte_ethdev.c                       |  31 +++
 lib/ethdev/rte_ethdev.h                       | 181 ++++++++++++++++++
 lib/ethdev/rte_ethdev_core.h                  |  23 ++-
 lib/ethdev/version.map                        |   4 +
 25 files changed, 703 insertions(+), 9 deletions(-)
 create mode 100644 app/test-pmd/recycle_mbufs.c
 create mode 100644 drivers/net/i40e/i40e_recycle_mbufs_vec_common.c
 create mode 100644 drivers/net/ixgbe/ixgbe_recycle_mbufs_vec_common.c
Signature · 1 line
2.25.1
#85 lore raw

PATCH v9 1/4 ethdev: add API for mbufs recycle mode

Add 'rte_eth_recycle_rx_queue_info_get' and 'rte_eth_recycle_mbufs' APIs to recycle used mbufs from a transmit queue of an Ethernet device, and move these mbufs into a mbuf ring for a receive queue of an Ethernet device. This can bypass mempool 'put/get' operations hence saving CPU cycles.

For each recycling mbufs, the rte_eth_recycle_mbufs() function performs the following operations:
- Copy used *rte_mbuf* buffer pointers from Tx mbuf ring into Rx mbuf ring.
- Replenish the Rx descriptors with the recycling *rte_mbuf* mbufs freed from the Tx mbuf ring.

---
 doc/guides/rel_notes/release_23_11.rst |  15 ++
 lib/ethdev/ethdev_driver.h             |  10 ++
 lib/ethdev/ethdev_private.c            |   2 +
 lib/ethdev/rte_ethdev.c                |  31 +++++
 lib/ethdev/rte_ethdev.h                | 181 +++++++++++++++++++++++++
 lib/ethdev/rte_ethdev_core.h           |  23 +++-
 lib/ethdev/version.map                 |   4 +
 7 files changed, 260 insertions(+), 6 deletions(-)
doc/guides/rel_notes/release_23_11.rst +15 -0
index 6b4dd21fd0..fd16d267ae 100644
@@ -55,6 +55,13 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Add mbufs recycling support. **
+
+  Added ``rte_eth_recycle_rx_queue_info_get`` and ``rte_eth_recycle_mbufs``
+  APIs which allow the user to copy used mbufs from the Tx mbuf ring
+  into the Rx mbuf ring. This feature supports the case that the Rx Ethernet
+  device is different from the Tx Ethernet device with respective driver
+  callback functions in ``rte_eth_recycle_mbufs``.
 
 Removed Items
 -------------
@@ -100,6 +107,14 @@ ABI Changes
    Also, make sure to start the actual text at the margin.
    =======================================================
 
+* ethdev: Added ``recycle_tx_mbufs_reuse`` and ``recycle_rx_descriptors_refill``
+  fields to ``rte_eth_dev`` structure.
+
+* ethdev: Structure ``rte_eth_fp_ops`` was affected to add
+  ``recycle_tx_mbufs_reuse`` and ``recycle_rx_descriptors_refill``
+  fields, to move ``rxq`` and ``txq`` fields, to change the size of
+  ``reserved1`` and ``reserved2`` fields.
+
 
 Known Issues
 ------------
lib/ethdev/ethdev_driver.h +10 -0
index 980f837ab6..b0c55a8523 100644
@@ -58,6 +58,10 @@ struct rte_eth_dev {
 	eth_rx_descriptor_status_t rx_descriptor_status;
 	/** Check the status of a Tx descriptor */
 	eth_tx_descriptor_status_t tx_descriptor_status;
+	/** Pointer to PMD transmit mbufs reuse function */
+	eth_recycle_tx_mbufs_reuse_t recycle_tx_mbufs_reuse;
+	/** Pointer to PMD receive descriptors refill function */
+	eth_recycle_rx_descriptors_refill_t recycle_rx_descriptors_refill;
 
 	/**
 	 * Device data that is shared between primary and secondary processes
@@ -507,6 +511,10 @@ typedef void (*eth_rxq_info_get_t)(struct rte_eth_dev *dev,
 typedef void (*eth_txq_info_get_t)(struct rte_eth_dev *dev,
 	uint16_t tx_queue_id, struct rte_eth_txq_info *qinfo);
 
+typedef void (*eth_recycle_rxq_info_get_t)(struct rte_eth_dev *dev,
+	uint16_t rx_queue_id,
+	struct rte_eth_recycle_rxq_info *recycle_rxq_info);
+
 typedef int (*eth_burst_mode_get_t)(struct rte_eth_dev *dev,
 	uint16_t queue_id, struct rte_eth_burst_mode *mode);
 
@@ -1250,6 +1258,8 @@ struct eth_dev_ops {
 	eth_rxq_info_get_t         rxq_info_get;
 	/** Retrieve Tx queue information */
 	eth_txq_info_get_t         txq_info_get;
+	/** Retrieve mbufs recycle Rx queue information */
+	eth_recycle_rxq_info_get_t recycle_rxq_info_get;
 	eth_burst_mode_get_t       rx_burst_mode_get; /**< Get Rx burst mode */
 	eth_burst_mode_get_t       tx_burst_mode_get; /**< Get Tx burst mode */
 	eth_fw_version_get_t       fw_version_get; /**< Get firmware version */
lib/ethdev/ethdev_private.c +2 -0
index 14ec8c6ccf..f8ab64f195 100644
@@ -277,6 +277,8 @@ eth_dev_fp_ops_setup(struct rte_eth_fp_ops *fpo,
 	fpo->rx_queue_count = dev->rx_queue_count;
 	fpo->rx_descriptor_status = dev->rx_descriptor_status;
 	fpo->tx_descriptor_status = dev->tx_descriptor_status;
+	fpo->recycle_tx_mbufs_reuse = dev->recycle_tx_mbufs_reuse;
+	fpo->recycle_rx_descriptors_refill = dev->recycle_rx_descriptors_refill;
 
 	fpo->rxq.data = dev->data->rx_queues;
 	fpo->rxq.clbk = (void **)(uintptr_t)dev->post_rx_burst_cbs;
lib/ethdev/rte_ethdev.c +31 -0
index 0840d2b594..ea89a101a1 100644
@@ -5876,6 +5876,37 @@ rte_eth_tx_queue_info_get(uint16_t port_id, uint16_t queue_id,
 	return 0;
 }
 
+int
+rte_eth_recycle_rx_queue_info_get(uint16_t port_id, uint16_t queue_id,
+		struct rte_eth_recycle_rxq_info *recycle_rxq_info)
+{
+	struct rte_eth_dev *dev;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+	dev = &rte_eth_devices[port_id];
+
+	if (queue_id >= dev->data->nb_rx_queues) {
+		RTE_ETHDEV_LOG(ERR, "Invalid Rx queue_id=%u\n", queue_id);
+		return -EINVAL;
+	}
+
+	if (dev->data->rx_queues == NULL ||
+			dev->data->rx_queues[queue_id] == NULL) {
+		RTE_ETHDEV_LOG(ERR,
+			   "Rx queue %"PRIu16" of device with port_id=%"
+			   PRIu16" has not been setup\n",
+			   queue_id, port_id);
+		return -EINVAL;
+	}
+
+	if (*dev->dev_ops->recycle_rxq_info_get == NULL)
+		return -ENOTSUP;
+
+	dev->dev_ops->recycle_rxq_info_get(dev, queue_id, recycle_rxq_info);
+
+	return 0;
+}
+
 int
 rte_eth_rx_burst_mode_get(uint16_t port_id, uint16_t queue_id,
 			  struct rte_eth_burst_mode *mode)
lib/ethdev/rte_ethdev.h +181 -0

199 lines not on this page · raw message

lib/ethdev/rte_ethdev_core.h +17 -6
index 46e9721e07..a24ad7a6b2 100644
@@ -55,6 +55,13 @@ typedef int (*eth_rx_descriptor_status_t)(void *rxq, uint16_t offset);
 /** @internal Check the status of a Tx descriptor */
 typedef int (*eth_tx_descriptor_status_t)(void *txq, uint16_t offset);
 
+/** @internal Copy used mbufs from Tx mbuf ring into Rx mbuf ring */
+typedef uint16_t (*eth_recycle_tx_mbufs_reuse_t)(void *txq,
+		struct rte_eth_recycle_rxq_info *recycle_rxq_info);
+
+/** @internal Refill Rx descriptors with the recycling mbufs */
+typedef void (*eth_recycle_rx_descriptors_refill_t)(void *rxq, uint16_t nb);
+
 /**
  * @internal
  * Structure used to hold opaque pointers to internal ethdev Rx/Tx
@@ -83,15 +90,17 @@ struct rte_eth_fp_ops {
 	 * Rx fast-path functions and related data.
 	 * 64-bit systems: occupies first 64B line
 	 */
+	/** Rx queues data. */
+	struct rte_ethdev_qdata rxq;
 	/** PMD receive function. */
 	eth_rx_burst_t rx_pkt_burst;
 	/** Get the number of used Rx descriptors. */
 	eth_rx_queue_count_t rx_queue_count;
 	/** Check the status of a Rx descriptor. */
 	eth_rx_descriptor_status_t rx_descriptor_status;
-	/** Rx queues data. */
-	struct rte_ethdev_qdata rxq;
-	uintptr_t reserved1[3];
+	/** Refill Rx descriptors with the recycling mbufs. */
+	eth_recycle_rx_descriptors_refill_t recycle_rx_descriptors_refill;
+	uintptr_t reserved1[2];
 	/**@}*/
 
 	/**@{*/
@@ -99,15 +108,17 @@ struct rte_eth_fp_ops {
 	 * Tx fast-path functions and related data.
 	 * 64-bit systems: occupies second 64B line
 	 */
+	/** Tx queues data. */
+	struct rte_ethdev_qdata txq;
 	/** PMD transmit function. */
 	eth_tx_burst_t tx_pkt_burst;
 	/** PMD transmit prepare function. */
 	eth_tx_prep_t tx_pkt_prepare;
 	/** Check the status of a Tx descriptor. */
 	eth_tx_descriptor_status_t tx_descriptor_status;
-	/** Tx queues data. */
-	struct rte_ethdev_qdata txq;
-	uintptr_t reserved2[3];
+	/** Copy used mbufs from Tx mbuf ring into Rx. */
+	eth_recycle_tx_mbufs_reuse_t recycle_tx_mbufs_reuse;
+	uintptr_t reserved2[2];
 	/**@}*/
 
 } __rte_cache_aligned;
lib/ethdev/version.map +4 -0
index b965d6aa52..e52c1563b4 100644
@@ -312,6 +312,10 @@ EXPERIMENTAL {
 	rte_flow_async_action_list_handle_query_update;
 	rte_flow_async_actions_update;
 	rte_flow_restore_info_dynflag;
+
+	# added in 23.11
+	rte_eth_recycle_mbufs;
+	rte_eth_recycle_rx_queue_info_get;
 };
 
 INTERNAL {
Signature · 1 line
2.25.1