Skip to contentSkip to conversation

Direct re-arming of buffers on receive side

PATCH Superseded Feifei Wang 4 participants 17 messages Sep 27, 2022 – Jan 4, 2023 Network drivers

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover Direct re-arming of buffers on receive side

Currently, the transmit side frees the buffers into the lcore cache and the receive side allocates buffers from the lcore cache. The transmit side typically frees 32 buffers resulting in 32*8=256B of stores to lcore cache. The receive side allocates 32 buffers and stores them in the receive side software ring, resulting in 32*8=256B of stores and 256B of load from the lcore cache.

This patch proposes a mechanism to avoid freeing to/allocating from the lcore cache. i.e. the receive side will free the buffers from transmit side directly into it's software ring. This will avoid the 256B of loads and stores introduced by the lcore cache. It also frees up the cache lines used by the lcore cache.

However, this solution poses several constraints:

1)The receive queue needs to know which transmit queue it should take the buffers from. The application logic decides which transmit port to use to send out the packets. In many use cases the NIC might have a single port ([1], [2], [3]), in which case a given transmit queue is always mapped to a single receive queue (1:1 Rx queue: Tx queue). This is easy to configure.

If the NIC has 2 ports (there are several references), then we will have 1:2 (RX queue: TX queue) mapping which is still easy to configure. However, if this is generalized to 'N' ports, the configuration can be long. More over the PMD would have to scan a list of transmit queues to pull the buffers from.

2)The other factor that needs to be considered is 'run-to-completion' vs 'pipeline' models. In the run-to-completion model, the receive side and the transmit side are running on the same lcore serially. In the pipeline model. The receive side and transmit side might be running on different lcores in parallel. This requires locking. This is not supported at this point.

3)Tx and Rx buffers must be from the same mempool. And we also must ensure Tx buffer free number is equal to Rx buffer free number:
(txq->tx_rs_thresh == RTE_I40E_RXQ_REARM_THRESH)
Thus, 'tx_next_dd' can be updated correctly in direct-rearm mode. This is due to tx_next_dd is a variable to compute tx sw-ring free location. Its value will be one more round than the position where next time free starts.

Current status in this patch:
1)Two APIs are added for users to enable direct-rearm mode:
In control plane, users can call 'rte_eth_txq_data_get' to get Tx sw_ring pointer and its txq_info (This avoid Rx load Tx data directly);

In data plane, users can call 'rte_eth_rx_direct_rearm' to rearm Rx buffers and free Tx buffers at the same time (Currently it supports 1:1 (rxq:txq) mapping:)
-----------------------------------------------------------------------
control plane:
rte_eth_txq_data_get(*txq_data);
data plane:
loop {

  		rte_eth_rx_direct_rearm(*txq_data){
     			for (i = 0; i <= 32; i++) {
       				rx.mbuf[i] = tx.mbuf[i];
       				initialize descs[i];
    			}
		}
		rte_eth_rx_burst;
		rte_eth_tx_burst;

}
-----------------------------------------------------------------------
2)The i40e driver is changed to do the direct re-arm of the receive
side.
3)L3fwd application is modified to enable direct rearm mode. Users can
enable direct-rearm and map queues by input parameters.

Testing status:
1.The testing results for L3fwd are as follows:
-------------------------------------------------------------------
enabled direct rearm
-------------------------------------------------------------------
Arm:
N1SDP(neon path):
without fast-free mode with fast-free mode

	+15.09%				+4.2%

Ampere Altra(neon path):
without fast-free mode with fast-free mode

	+10.9%				+14.6%

-------------------------------------------------------------------

2.The testing results for VPP-L3fwd are as follows:
-------------------------------------------------------------------
Arm:
N1SDP(neon path):
with direct re-arm mode enabled

	+4.5%

Ampere Altra(neon path):
with direct re-arm mode enabled

        +6.5%

-------------------------------------------------------------------

Reference:
[1] https://store.nvidia.com/en-us/networking/store/product/MCX623105AN-CDAT/NVIDIAMCX623105ANCDATConnectX6DxENAdapterCard100GbECryptoDisabled/
[2] https://www.intel.com/content/www/us/en/products/sku/192561/intel-ethernet-network-adapter-e810cqda1/specifications.html
[3] https://www.broadcom.com/products/ethernet-connectivity/network-adapters/100gb-nic-ocp/n1100g

V2:
1. Use data-plane API to enable direct-rearm (Konstantin, Honnappa)
2. Add 'txq_data_get' API to get txq info for Rx (Konstantin)
3. Use input parameter to enable direct rearm in l3fwd (Konstantin)
4. Add condition detection for direct rearm API (Morten, Andrew Rybchenko)

Feifei Wang (3):
ethdev: add API for direct rearm mode
net/i40e: enable direct rearm mode
examples/l3fwd: enable direct rearm mode

 drivers/net/i40e/i40e_ethdev.c        |  1 +
 drivers/net/i40e/i40e_ethdev.h        |  2 +
 drivers/net/i40e/i40e_rxtx.c          | 19 ++++++
 drivers/net/i40e/i40e_rxtx.h          |  2 +
 drivers/net/i40e/i40e_rxtx_vec_neon.c | 93 ++++++++++++++++++++++++++
 examples/l3fwd/l3fwd.h                | 12 ++++
 examples/l3fwd/l3fwd_lpm.c            | 22 +++++++
 examples/l3fwd/main.c                 | 94 +++++++++++++++++++++++++-
 lib/ethdev/ethdev_driver.h            |  9 +++
 lib/ethdev/ethdev_private.c           |  1 +
 lib/ethdev/rte_ethdev.c               | 37 +++++++++++
 lib/ethdev/rte_ethdev.h               | 95 +++++++++++++++++++++++++++
 lib/ethdev/rte_ethdev_core.h          |  5 ++
 lib/ethdev/version.map                |  4 ++
 14 files changed, 395 insertions(+), 1 deletion(-)
Signature · 1 line
2.25.1
#1 lore raw

PATCH v2 1/3 ethdev: add API for direct rearm mode

Add API for enabling direct rearm mode and for mapping RX and TX queues. Currently, the API supports 1:1(txq : rxq) mapping.

Furthermore, to avoid Rx load Tx data directly, add API called 'rte_eth_txq_data_get' to get Tx sw_ring and its information.

---
 lib/ethdev/ethdev_driver.h   |  9 ++++
 lib/ethdev/ethdev_private.c  |  1 +
 lib/ethdev/rte_ethdev.c      | 37 ++++++++++++++
 lib/ethdev/rte_ethdev.h      | 95 ++++++++++++++++++++++++++++++++++++
 lib/ethdev/rte_ethdev_core.h |  5 ++
 lib/ethdev/version.map       |  4 ++
 6 files changed, 151 insertions(+)
lib/ethdev/ethdev_driver.h +9 -0
index 47a55a419e..14f52907c1 100644
@@ -58,6 +58,8 @@ struct rte_eth_dev {
 	eth_rx_descriptor_status_t rx_descriptor_status;
 	/** Check the status of a Tx descriptor */
 	eth_tx_descriptor_status_t tx_descriptor_status;
+	/**  Use Tx mbufs for Rx to rearm */
+	eth_rx_direct_rearm_t rx_direct_rearm;
 
 	/**
 	 * Device data that is shared between primary and secondary processes
@@ -486,6 +488,11 @@ typedef int (*eth_rx_enable_intr_t)(struct rte_eth_dev *dev,
 typedef int (*eth_rx_disable_intr_t)(struct rte_eth_dev *dev,
 				    uint16_t rx_queue_id);
 
+/**< @internal Get Tx information of a transmit queue of an Ethernet device. */
+typedef void (*eth_txq_data_get_t)(struct rte_eth_dev *dev,
+				      uint16_t tx_queue_id,
+				      struct rte_eth_txq_data *txq_data);
+
 /** @internal Release memory resources allocated by given Rx/Tx queue. */
 typedef void (*eth_queue_release_t)(struct rte_eth_dev *dev,
 				    uint16_t queue_id);
@@ -1138,6 +1145,8 @@ struct eth_dev_ops {
 	eth_rxq_info_get_t         rxq_info_get;
 	/** Retrieve Tx queue information */
 	eth_txq_info_get_t         txq_info_get;
+	/** Get the address where Tx data is stored */
+	eth_txq_data_get_t         txq_data_get;
 	eth_burst_mode_get_t       rx_burst_mode_get; /**< Get Rx burst mode */
 	eth_burst_mode_get_t       tx_burst_mode_get; /**< Get Tx burst mode */
 	eth_fw_version_get_t       fw_version_get; /**< Get firmware version */
lib/ethdev/ethdev_private.c +1 -0
index 48090c879a..bfe16c7d77 100644
@@ -276,6 +276,7 @@ eth_dev_fp_ops_setup(struct rte_eth_fp_ops *fpo,
 	fpo->rx_queue_count = dev->rx_queue_count;
 	fpo->rx_descriptor_status = dev->rx_descriptor_status;
 	fpo->tx_descriptor_status = dev->tx_descriptor_status;
+	fpo->rx_direct_rearm = dev->rx_direct_rearm;
 
 	fpo->rxq.data = dev->data->rx_queues;
 	fpo->rxq.clbk = (void **)(uintptr_t)dev->post_rx_burst_cbs;
lib/ethdev/rte_ethdev.c +37 -0
index 0c2c1088c0..0dccec2e4b 100644
@@ -1648,6 +1648,43 @@ rte_eth_dev_is_removed(uint16_t port_id)
 	return ret;
 }
 
+int
+rte_eth_tx_queue_data_get(uint16_t port_id, uint16_t queue_id,
+			struct rte_eth_txq_data *txq_data)
+{
+	struct rte_eth_dev *dev;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+	dev = &rte_eth_devices[port_id];
+
+	if (queue_id >= dev->data->nb_tx_queues) {
+		RTE_ETHDEV_LOG(ERR, "Invalid Tx queue_id=%u\n", queue_id);
+		return -EINVAL;
+	}
+
+	if (txq_data == NULL) {
+		RTE_ETHDEV_LOG(ERR, "Cannot get ethdev port %u Tx queue %u data to NULL\n",
+			port_id, queue_id);
+		return -EINVAL;
+	}
+
+	if (dev->data->tx_queues == NULL ||
+			dev->data->tx_queues[queue_id] == NULL) {
+		RTE_ETHDEV_LOG(ERR,
+			   "Tx queue %"PRIu16" of device with port_id=%"
+			   PRIu16" has not been setup\n",
+			   queue_id, port_id);
+		return -EINVAL;
+	}
+
+	if (*dev->dev_ops->txq_data_get == NULL)
+		return -ENOTSUP;
+
+	dev->dev_ops->txq_data_get(dev, queue_id, txq_data);
+
+	return 0;
+}
+
 static int
 rte_eth_rx_queue_check_split(const struct rte_eth_rxseg_split *rx_seg,
 			     uint16_t n_seg, uint32_t *mbp_buf_size,
lib/ethdev/rte_ethdev.h +95 -0

113 lines not on this page · raw message

lib/ethdev/rte_ethdev_core.h +5 -0
index dcf8adab92..6a328e2481 100644
@@ -56,6 +56,9 @@ typedef int (*eth_rx_descriptor_status_t)(void *rxq, uint16_t offset);
 /** @internal Check the status of a Tx descriptor */
 typedef int (*eth_tx_descriptor_status_t)(void *txq, uint16_t offset);
 
+/** @internal Put Tx buffers into Rx sw-ring and rearm descs */
+typedef uint16_t (*eth_rx_direct_rearm_t)(void *rxq, struct rte_eth_txq_data *txq_data);
+
 /**
  * @internal
  * Structure used to hold opaque pointers to internal ethdev Rx/Tx
@@ -90,6 +93,8 @@ struct rte_eth_fp_ops {
 	eth_rx_queue_count_t rx_queue_count;
 	/** Check the status of a Rx descriptor. */
 	eth_rx_descriptor_status_t rx_descriptor_status;
+	/** Put Tx buffers into Rx sw-ring and rearm descs */
+	eth_rx_direct_rearm_t rx_direct_rearm;
 	/** Rx queues data. */
 	struct rte_ethdev_qdata rxq;
 	uintptr_t reserved1[3];
lib/ethdev/version.map +4 -0
index 03f52fee91..69e4c376d3 100644
@@ -285,6 +285,10 @@ EXPERIMENTAL {
 	rte_mtr_color_in_protocol_priority_get;
 	rte_mtr_color_in_protocol_set;
 	rte_mtr_meter_vlan_table_update;
+
+	# added in 22.11
+	rte_eth_tx_queue_data_get;
+	rte_eth_rx_direct_rearm;
 };
 
 INTERNAL {
Signature · 1 line
2.25.1
#2 lore raw

Re: [PATCH v2 1/3] ethdev: add API for direct rearm mode

27/09/2022 03:47, Feifei Wang пишет:

Quoted 128 lines

Add API for enabling direct rearm mode and for mapping RX and TX queues. Currently, the API supports 1:1(txq : rxq) mapping.

Furthermore, to avoid Rx load Tx data directly, add API called 'rte_eth_txq_data_get' to get Tx sw_ring and its information.

---
  lib/ethdev/ethdev_driver.h   |  9 ++++
  lib/ethdev/ethdev_private.c  |  1 +
  lib/ethdev/rte_ethdev.c      | 37 ++++++++++++++
  lib/ethdev/rte_ethdev.h      | 95 ++++++++++++++++++++++++++++++++++++
  lib/ethdev/rte_ethdev_core.h |  5 ++
  lib/ethdev/version.map       |  4 ++
  6 files changed, 151 insertions(+)
lib/ethdev/ethdev_driver.h +9 -0
index 47a55a419e..14f52907c1 100644
@@ -58,6 +58,8 @@ struct rte_eth_dev {
  	eth_rx_descriptor_status_t rx_descriptor_status;
  	/** Check the status of a Tx descriptor */
  	eth_tx_descriptor_status_t tx_descriptor_status;
+	/**  Use Tx mbufs for Rx to rearm */
+	eth_rx_direct_rearm_t rx_direct_rearm;
 
  	/**
  	 * Device data that is shared between primary and secondary processes
@@ -486,6 +488,11 @@ typedef int (*eth_rx_enable_intr_t)(struct rte_eth_dev *dev,
  typedef int (*eth_rx_disable_intr_t)(struct rte_eth_dev *dev,
  				    uint16_t rx_queue_id);
 
+/**< @internal Get Tx information of a transmit queue of an Ethernet device. */
+typedef void (*eth_txq_data_get_t)(struct rte_eth_dev *dev,
+				      uint16_t tx_queue_id,
+				      struct rte_eth_txq_data *txq_data);
+
  /** @internal Release memory resources allocated by given Rx/Tx queue. */
  typedef void (*eth_queue_release_t)(struct rte_eth_dev *dev,
  				    uint16_t queue_id);
@@ -1138,6 +1145,8 @@ struct eth_dev_ops {
  	eth_rxq_info_get_t         rxq_info_get;
  	/** Retrieve Tx queue information */
  	eth_txq_info_get_t         txq_info_get;
+	/** Get the address where Tx data is stored */
+	eth_txq_data_get_t         txq_data_get;
  	eth_burst_mode_get_t       rx_burst_mode_get; /**< Get Rx burst mode */
  	eth_burst_mode_get_t       tx_burst_mode_get; /**< Get Tx burst mode */
  	eth_fw_version_get_t       fw_version_get; /**< Get firmware version */
lib/ethdev/ethdev_private.c +1 -0
index 48090c879a..bfe16c7d77 100644
@@ -276,6 +276,7 @@ eth_dev_fp_ops_setup(struct rte_eth_fp_ops *fpo,
  	fpo->rx_queue_count = dev->rx_queue_count;
  	fpo->rx_descriptor_status = dev->rx_descriptor_status;
  	fpo->tx_descriptor_status = dev->tx_descriptor_status;
+	fpo->rx_direct_rearm = dev->rx_direct_rearm;
 
  	fpo->rxq.data = dev->data->rx_queues;
  	fpo->rxq.clbk = (void **)(uintptr_t)dev->post_rx_burst_cbs;
lib/ethdev/rte_ethdev.c +37 -0
index 0c2c1088c0..0dccec2e4b 100644
@@ -1648,6 +1648,43 @@ rte_eth_dev_is_removed(uint16_t port_id)
  	return ret;
  }
 
+int
+rte_eth_tx_queue_data_get(uint16_t port_id, uint16_t queue_id,
+			struct rte_eth_txq_data *txq_data)
+{
+	struct rte_eth_dev *dev;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+	dev = &rte_eth_devices[port_id];
+
+	if (queue_id >= dev->data->nb_tx_queues) {
+		RTE_ETHDEV_LOG(ERR, "Invalid Tx queue_id=%u\n", queue_id);
+		return -EINVAL;
+	}
+
+	if (txq_data == NULL) {
+		RTE_ETHDEV_LOG(ERR, "Cannot get ethdev port %u Tx queue %u data to NULL\n",
+			port_id, queue_id);
+		return -EINVAL;
+	}
+
+	if (dev->data->tx_queues == NULL ||
+			dev->data->tx_queues[queue_id] == NULL) {
+		RTE_ETHDEV_LOG(ERR,
+			   "Tx queue %"PRIu16" of device with port_id=%"
+			   PRIu16" has not been setup\n",
+			   queue_id, port_id);
+		return -EINVAL;
+	}
+
+	if (*dev->dev_ops->txq_data_get == NULL)
+		return -ENOTSUP;
+
+	dev->dev_ops->txq_data_get(dev, queue_id, txq_data);
+
+	return 0;
+}
+
  static int
  rte_eth_rx_queue_check_split(const struct rte_eth_rxseg_split *rx_seg,
  			     uint16_t n_seg, uint32_t *mbp_buf_size,
lib/ethdev/rte_ethdev.h +6 -0
index 2e783536c1..daf7f05d62 100644
@@ -1949,6 +1949,23 @@ struct rte_eth_txq_info {
  	uint8_t queue_state;        /**< one of RTE_ETH_QUEUE_STATE_*. */
  } __rte_cache_min_aligned;
 
+/**
+ * @internal
+ * Structure used to hold pointers to internal ethdev Tx data.
+ * The main purpose is to load and store Tx queue data in direct rearm mode.
+ */
+struct rte_eth_txq_data {

lib/ethdev/rte_ethdev.h struct rte_eth_txq_info { · line 1967

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+	uint64_t *offloads;
+	void *tx_sw_ring;
+	volatile void *tx_ring;
+	uint16_t *tx_next_dd;
+	uint16_t *nb_tx_free;
+	uint16_t nb_tx_desc;
+	uint16_t tx_rs_thresh;
+	uint16_t tx_free_thresh;
+} __rte_cache_min_aligned;
+

first of all it is not clear why this struct has to be in public header, why it can't be in on of ethdev 'private' headers.
Second it looks like a snippet from private txq fields for some Intel (and alike) PMDs (i40e, ice, etc.).
How it supposed to to be universal and be applicable
for any PMD that decides to implement this new API?

Quoted 74 lines
lib/ethdev/rte_ethdev.h +60 -0
  /* Generic Burst mode flag definition, values can be ORed. */
 
  /**
@@ -4718,6 +4735,27 @@ int rte_eth_remove_rx_callback(uint16_t port_id, uint16_t queue_id,
  int rte_eth_remove_tx_callback(uint16_t port_id, uint16_t queue_id,
  		const struct rte_eth_rxtx_callback *user_cb);
 
+/**
+ * Get the address which Tx data is stored.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param queue_id
+ *   The Tx queue on the Ethernet device for which information
+ *   will be retrieved.
+ * @param txq_data
+ *   A pointer to a structure of type *rte_eth_txq_data* to be filled.
+ *
+ * @return
+ *   - 0: Success
+ *   - -ENODEV:  If *port_id* is invalid.
+ *   - -ENOTSUP: routine is not supported by the device PMD.
+ *   - -EINVAL:  The queue_id is out of range.
+ */
+__rte_experimental
+int rte_eth_tx_queue_data_get(uint16_t port_id, uint16_t queue_id,
+		struct rte_eth_txq_data *txq_data);
+
  /**
   * Retrieve information about given port's Rx queue.
   *
@@ -6209,6 +6247,63 @@ rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id,
  	return rte_eth_tx_buffer_flush(port_id, queue_id, buffer);
  }
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Put Tx buffers into Rx sw-ring and rearm descs.
+ *
+ * @param port_id
+ *   Port identifying the receive side.
+ * @param queue_id
+ *   The index of the transmit queue identifying the receive side.
+ *   The value must be in the range [0, nb_rx_queue - 1] previously supplied
+ *   to rte_eth_dev_configure().
+ * @param txq_data
+ *   A pointer to a structure of type *rte_eth_txq_data* to be filled.
+ * @return
+ *   The number of direct-rearmed buffers.
+ */
+__rte_experimental
+static __rte_always_inline uint16_t
+rte_eth_rx_direct_rearm(uint16_t port_id, uint16_t queue_id,
+		struct rte_eth_txq_data *txq_data)
+{
+	uint16_t nb_rearm;
+	struct rte_eth_fp_ops *p;
+	void *qd;
+
+#ifdef RTE_ETHDEV_DEBUG_RX
+	if (port_id >= RTE_MAX_ETHPORTS ||
+			queue_id >= RTE_MAX_QUEUES_PER_PORT) {
+		RTE_ETHDEV_LOG(ERR,
+			"Invalid port_id=%u or queue_id=%u\n",
+			port_id, queue_id);
+		return 0;
+	}
+#endif
+
+	p = &rte_eth_fp_ops[port_id];
+	qd = p->rxq.data[queue_id];
+

lib/ethdev/rte_ethdev.h rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id, · line 6298

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+#ifdef RTE_ETHDEV_DEBUG_RX
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, 0);
+
+	if (qd == NULL) {
+		RTE_ETHDEV_LOG(ERR, "Invalid Rx queue_id=%u for port_id=%u\n",
+			queue_id, port_id);
+		return 0;
+	}
+
+	if (!p->rx_direct_rearm)

This check should be done always (unconditionally).
it is not a mandatory function for the driver (it can safely skip to implement it).

lib/ethdev/rte_ethdev.h rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id, · line 6299

Quoted 1 line
lib/ethdev/rte_ethdev.h +1 -0
+		return -ENOTSUP;

This function returns uint16_t, why signed integers here?

lib/ethdev/rte_ethdev.h rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id, · line 6302

Quoted 3 lines
lib/ethdev/rte_ethdev.h +3 -0
+#endif
+
+	nb_rearm = p->rx_direct_rearm(qd, txq_data);

So rx_direct_rearm() function knows how to extract data from TX queue? As I understand that is possible only in one case:
rx_direct_rearm() has full knowledge and acess of txq internals, etc. That means that rxq and txq have to belong to the same driver and device type.

First of all, I still think it is not the best design choice.
If we going ahead with introducing this feature, it better be as generic as possible. Plus it mixes TX and RX code-paths together,
while it would be much better to to keep them independent as they
are right now.

Another thing with such approach - even for the same PMD,
both TXQ and RXQ can have different internal data format and
behavior logic (depending on port/queue configuration).
So rx_direct_rearm() function selection have to be done
based on both RXQ and TXQ config.
So instead of rte_eth_tx_queue_data_get(), you'll probably need:
eth_rx_direct_rearm_t rte_eth_get_rx_direct_rearm_func(rx_port, rx_queue, tx_port, tx_queue); Then, it will be user responsibility to store it somewhere
and call periodically:

control_path:

	...
	rearm_func = rte_eth_get_rx_direct_rearm_func(rxport, rxqueue,
		 txport, txqueue);
data-path:
	while(...) {
		rearm_func(rxport, txport, rxqueue, txqueue);
		rte_eth_rx_burst(rxport, rxqueue, ....);
		rte_eth_tx_burst(txport, txqueue, ....);
	}

In that case there seems absolutely no point to introduce struct rte_eth_txq_data. rx_direct_rearm() accesses TXQ private data directly anyway.

Another way - make rte_eth_txq_data totally opaque and allow PMD to store there some data that will help it to distinguish expected TXQ format. That will allow PMD to keep rx_direct_rearm() the same for all supported TXQ formats (it will make decision internally based on data stored in txq_data).
Though in that case you'll probably need one more dev-op to free txq_data.

Quoted 45 lines
lib/ethdev/rte_ethdev.h +4 -0
+
+	return nb_rearm;
+}
+
  #ifdef __cplusplus
  }
  #endif
lib/ethdev/rte_ethdev_core.h +5 -0
index dcf8adab92..6a328e2481 100644
@@ -56,6 +56,9 @@ typedef int (*eth_rx_descriptor_status_t)(void *rxq, uint16_t offset);
  /** @internal Check the status of a Tx descriptor */
  typedef int (*eth_tx_descriptor_status_t)(void *txq, uint16_t offset);
 
+/** @internal Put Tx buffers into Rx sw-ring and rearm descs */
+typedef uint16_t (*eth_rx_direct_rearm_t)(void *rxq, struct rte_eth_txq_data *txq_data);
+
  /**
   * @internal
   * Structure used to hold opaque pointers to internal ethdev Rx/Tx
@@ -90,6 +93,8 @@ struct rte_eth_fp_ops {
  	eth_rx_queue_count_t rx_queue_count;
  	/** Check the status of a Rx descriptor. */
  	eth_rx_descriptor_status_t rx_descriptor_status;
+	/** Put Tx buffers into Rx sw-ring and rearm descs */
+	eth_rx_direct_rearm_t rx_direct_rearm;
  	/** Rx queues data. */
  	struct rte_ethdev_qdata rxq;
  	uintptr_t reserved1[3];
lib/ethdev/version.map +4 -0
index 03f52fee91..69e4c376d3 100644
@@ -285,6 +285,10 @@ EXPERIMENTAL {
  	rte_mtr_color_in_protocol_priority_get;
  	rte_mtr_color_in_protocol_set;
  	rte_mtr_meter_vlan_table_update;
+
+	# added in 22.11
+	rte_eth_tx_queue_data_get;
+	rte_eth_rx_direct_rearm;
  };
 
  INTERNAL {
#3 lore raw

回复: [PATCH v2 1/3] ethdev: add API for direct rearm mode

Quoted 267 lines

-----邮件原件-----
发件人: Konstantin Ananyev <[email protected]> 发送时间: Monday, October 3, 2022 4:58 PM
收件人: Feifei Wang <[email protected]>; [email protected]; Ferruh Yigit <[email protected]>; Andrew Rybchenko <[email protected]>; Ray Kinsella <[email protected]> 抄送: [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; Ruifeng Wang <[email protected]>
主题: Re: [PATCH v2 1/3] ethdev: add API for direct rearm mode

27/09/2022 03:47, Feifei Wang пишет:

Quoted 11 lines

Add API for enabling direct rearm mode and for mapping RX and TX queues. Currently, the API supports 1:1(txq : rxq) mapping.

Furthermore, to avoid Rx load Tx data directly, add API called 'rte_eth_txq_data_get' to get Tx sw_ring and its information.

Quoted 5 lines
---
  lib/ethdev/ethdev_driver.h   |  9 ++++
  lib/ethdev/ethdev_private.c  |  1 +
  lib/ethdev/rte_ethdev.c      | 37 ++++++++++++++
  lib/ethdev/rte_ethdev.h      | 95

++++++++++++++++++++++++++++++++++++

Quoted 17 lines
  lib/ethdev/rte_ethdev_core.h |  5 ++
  lib/ethdev/version.map       |  4 ++
  6 files changed, 151 insertions(+)
lib/ethdev/ethdev_driver.h +2 -0
index 47a55a419e..14f52907c1 100644
@@ -58,6 +58,8 @@ struct rte_eth_dev {
  	eth_rx_descriptor_status_t rx_descriptor_status;
  	/** Check the status of a Tx descriptor */
  	eth_tx_descriptor_status_t tx_descriptor_status;
+	/**  Use Tx mbufs for Rx to rearm */
+	eth_rx_direct_rearm_t rx_direct_rearm;
 
  	/**
  	 * Device data that is shared between primary and secondary
Quoted 1 line

processes @@ -486,6 +488,11 @@ typedef int

(*eth_rx_enable_intr_t)(struct rte_eth_dev *dev,

lib/ethdev/ethdev_driver.h typedef int (*eth_rx_enable_intr_t)(struct rte_eth_dev *dev,

Quoted 9 lines

typedef int (*eth_rx_disable_intr_t)(struct rte_eth_dev *dev,
uint16_t rx_queue_id);

lib/ethdev/ethdev_driver.h +5 -0
+/**< @internal Get Tx information of a transmit queue of an Ethernet
+device. */ typedef void (*eth_txq_data_get_t)(struct rte_eth_dev *dev,
+				      uint16_t tx_queue_id,
+				      struct rte_eth_txq_data *txq_data);
+
  /** @internal Release memory resources allocated by given Rx/Tx queue.

*/

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1150

Quoted 9 lines

typedef void (*eth_queue_release_t)(struct rte_eth_dev *dev,
uint16_t queue_id);

lib/ethdev/ethdev_driver.h +2 -0
@@ -1138,6 +1145,8 @@ struct eth_dev_ops {
  	eth_rxq_info_get_t         rxq_info_get;
  	/** Retrieve Tx queue information */
  	eth_txq_info_get_t         txq_info_get;
+	/** Get the address where Tx data is stored */
+	eth_txq_data_get_t         txq_data_get;
  	eth_burst_mode_get_t       rx_burst_mode_get; /**< Get Rx burst

mode */

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1151

Quoted 1 line
lib/ethdev/ethdev_driver.h +0 -0
  	eth_burst_mode_get_t       tx_burst_mode_get; /**< Get Tx burst

mode */

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1152

Quoted 1 line
lib/ethdev/ethdev_driver.h +0 -0
  	eth_fw_version_get_t       fw_version_get; /**< Get firmware

version */

lib/ethdev/ethdev_private.c eth_dev_fp_ops_setup(struct rte_eth_fp_ops

Quoted 5 lines
lib/ethdev/ethdev_private.c +0 -0
index 48090c879a..bfe16c7d77 100644
@@ -276,6 +276,7 @@ eth_dev_fp_ops_setup(struct rte_eth_fp_ops

*fpo,

Quoted 15 lines
lib/ethdev/ethdev_private.c +1 -0
  	fpo->rx_queue_count = dev->rx_queue_count;
  	fpo->rx_descriptor_status = dev->rx_descriptor_status;
  	fpo->tx_descriptor_status = dev->tx_descriptor_status;
+	fpo->rx_direct_rearm = dev->rx_direct_rearm;
 
  	fpo->rxq.data = dev->data->rx_queues;
  	fpo->rxq.clbk = (void **)(uintptr_t)dev->post_rx_burst_cbs;
lib/ethdev/rte_ethdev.c index renamed from lib/ethdev/rte_ethdev.c +0 -0

0c2c1088c0..0dccec2e4b 100644

lib/ethdev/rte_ethdev.c +0 -0
@@ -1648,6 +1648,43 @@ rte_eth_dev_is_removed(uint16_t port_id)
  	return ret;
  }
 

lib/ethdev/rte_ethdev.c rte_eth_dev_is_removed(uint16_t port_id) · line 1660

Quoted 10 lines
lib/ethdev/rte_ethdev.c +10 -0
+int
+rte_eth_tx_queue_data_get(uint16_t port_id, uint16_t queue_id,
+			struct rte_eth_txq_data *txq_data) {
+	struct rte_eth_dev *dev;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+	dev = &rte_eth_devices[port_id];
+
+	if (queue_id >= dev->data->nb_tx_queues) {
+		RTE_ETHDEV_LOG(ERR, "Invalid Tx queue_id=%u\n",

queue_id);

lib/ethdev/rte_ethdev.c rte_eth_dev_is_removed(uint16_t port_id)

Quoted 5 lines
lib/ethdev/rte_ethdev.c +5 -0
+		return -EINVAL;
+	}
+
+	if (txq_data == NULL) {
+		RTE_ETHDEV_LOG(ERR, "Cannot get ethdev port %u Tx

queue %u data to NULL\n",

Quoted 29 lines
lib/ethdev/rte_ethdev.c +21 -0
+			port_id, queue_id);
+		return -EINVAL;
+	}
+
+	if (dev->data->tx_queues == NULL ||
+			dev->data->tx_queues[queue_id] == NULL) {
+		RTE_ETHDEV_LOG(ERR,
+			   "Tx queue %"PRIu16" of device with port_id=%"
+			   PRIu16" has not been setup\n",
+			   queue_id, port_id);
+		return -EINVAL;
+	}
+
+	if (*dev->dev_ops->txq_data_get == NULL)
+		return -ENOTSUP;
+
+	dev->dev_ops->txq_data_get(dev, queue_id, txq_data);
+
+	return 0;
+}
+
  static int
  rte_eth_rx_queue_check_split(const struct rte_eth_rxseg_split *rx_seg,
  			     uint16_t n_seg, uint32_t *mbp_buf_size, diff --git

a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h index 2e783536c1..daf7f05d62 100644

lib/ethdev/rte_ethdev.h struct rte_eth_txq_info { · line 1955

Quoted 7 lines
lib/ethdev/rte_ethdev.h +4 -0
@@ -1949,6 +1949,23 @@ struct rte_eth_txq_info {
  	uint8_t queue_state;        /**< one of RTE_ETH_QUEUE_STATE_*. */
  } __rte_cache_min_aligned;
 
+/**
+ * @internal
+ * Structure used to hold pointers to internal ethdev Tx data.
+ * The main purpose is to load and store Tx queue data in direct rearm

mode.

lib/ethdev/rte_ethdev.h struct rte_eth_txq_info { · line 1967

Quoted 12 lines
lib/ethdev/rte_ethdev.h +12 -0
+ */
+struct rte_eth_txq_data {
+	uint64_t *offloads;
+	void *tx_sw_ring;
+	volatile void *tx_ring;
+	uint16_t *tx_next_dd;
+	uint16_t *nb_tx_free;
+	uint16_t nb_tx_desc;
+	uint16_t tx_rs_thresh;
+	uint16_t tx_free_thresh;
+} __rte_cache_min_aligned;
+

first of all it is not clear why this struct has to be in public header, why it can't be in on of ethdev 'private' headers.
Second it looks like a snippet from private txq fields for some Intel (and alike) PMDs (i40e, ice, etc.).
How it supposed to to be universal and be applicable for any PMD that decides to implement this new API?

lib/ethdev/rte_ethdev.h int rte_eth_remove_rx_callback(uint16_t

Quoted 4 lines
lib/ethdev/rte_ethdev.h +0 -0
  /* Generic Burst mode flag definition, values can be ORed. */
 
  /**
@@ -4718,6 +4735,27 @@ int rte_eth_remove_rx_callback(uint16_t

port_id, uint16_t queue_id,

lib/ethdev/rte_ethdev.h rte_eth_tx_buffer(uint16_t port_id, uint16_t

Quoted 28 lines
lib/ethdev/rte_ethdev.h +21 -0
  int rte_eth_remove_tx_callback(uint16_t port_id, uint16_t queue_id,
  		const struct rte_eth_rxtx_callback *user_cb);
 
+/**
+ * Get the address which Tx data is stored.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param queue_id
+ *   The Tx queue on the Ethernet device for which information
+ *   will be retrieved.
+ * @param txq_data
+ *   A pointer to a structure of type *rte_eth_txq_data* to be filled.
+ *
+ * @return
+ *   - 0: Success
+ *   - -ENODEV:  If *port_id* is invalid.
+ *   - -ENOTSUP: routine is not supported by the device PMD.
+ *   - -EINVAL:  The queue_id is out of range.
+ */
+__rte_experimental
+int rte_eth_tx_queue_data_get(uint16_t port_id, uint16_t queue_id,
+		struct rte_eth_txq_data *txq_data);
+
  /**
   * Retrieve information about given port's Rx queue.
   *
@@ -6209,6 +6247,63 @@ rte_eth_tx_buffer(uint16_t port_id, uint16_t

queue_id,

Quoted 5 lines
lib/ethdev/rte_ethdev.h +2 -0
  	return rte_eth_tx_buffer_flush(port_id, queue_id, buffer);
  }
 
+/**
+ * @warning

lib/ethdev/rte_ethdev.h rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id,

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior
+notice
+ *
+ * Put Tx buffers into Rx sw-ring and rearm descs.
+ *
+ * @param port_id
+ *   Port identifying the receive side.
+ * @param queue_id
+ *   The index of the transmit queue identifying the receive side.
+ *   The value must be in the range [0, nb_rx_queue - 1] previously

supplied

Quoted 23 lines
lib/ethdev/rte_ethdev.h +23 -0
+ *   to rte_eth_dev_configure().
+ * @param txq_data
+ *   A pointer to a structure of type *rte_eth_txq_data* to be filled.
+ * @return
+ *   The number of direct-rearmed buffers.
+ */
+__rte_experimental
+static __rte_always_inline uint16_t
+rte_eth_rx_direct_rearm(uint16_t port_id, uint16_t queue_id,
+		struct rte_eth_txq_data *txq_data)
+{
+	uint16_t nb_rearm;
+	struct rte_eth_fp_ops *p;
+	void *qd;
+
+#ifdef RTE_ETHDEV_DEBUG_RX
+	if (port_id >= RTE_MAX_ETHPORTS ||
+			queue_id >= RTE_MAX_QUEUES_PER_PORT) {
+		RTE_ETHDEV_LOG(ERR,
+			"Invalid port_id=%u or queue_id=%u\n",
+			port_id, queue_id);
+		return 0;
+	}

lib/ethdev/rte_ethdev.h rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id,

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+#endif
+
+	p = &rte_eth_fp_ops[port_id];
+	qd = p->rxq.data[queue_id];
+
+#ifdef RTE_ETHDEV_DEBUG_RX
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, 0);
+
+	if (qd == NULL) {
+		RTE_ETHDEV_LOG(ERR, "Invalid Rx queue_id=%u for

port_id=%u\n",

lib/ethdev/rte_ethdev.h rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id, · line 6298

Quoted 5 lines
lib/ethdev/rte_ethdev.h +5 -0
+			queue_id, port_id);
+		return 0;
+	}
+
+	if (!p->rx_direct_rearm)

This check should be done always (unconditionally).
it is not a mandatory function for the driver (it can safely skip to implement it).

lib/ethdev/rte_ethdev.h rte_eth_tx_buffer(uint16_t port_id, uint16_t · line 6300

Quoted 1 line
lib/ethdev/rte_ethdev.h +1 -0
+		return -ENOTSUP;

This function returns uint16_t, why signed integers here?

Quoted 10 lines

lib/ethdev/rte_ethdev.h rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id, · line 6302

Quoted 3 lines
lib/ethdev/rte_ethdev.h +3 -0
+#endif
+
+	nb_rearm = p->rx_direct_rearm(qd, txq_data);

So rx_direct_rearm() function knows how to extract data from TX queue? As I understand that is possible only in one case:
rx_direct_rearm() has full knowledge and acess of txq internals, etc. That means that rxq and txq have to belong to the same driver and device type.

Thanks for the comments, and I have some questions for this.

Quoted 9 lines

First of all, I still think it is not the best design choice. If we going ahead with introducing this feature, it better be as generic as possible.
Plus it mixes TX and RX code-paths together, while it would be much better to to keep them independent as they are right now.

Another thing with such approach - even for the same PMD, both TXQ and RXQ can have different internal data format and behavior logic (depending on port/queue configuration).

1. Here TXQ and RXQ have different internal format means the queue type and descs can be different, right? If I understand correctly, based on your first strategy, is it means we will need different 'rearm_func' for different queue type in the same PMD?

Quoted 19 lines

So rx_direct_rearm() function selection have to be done based on both RXQ and TXQ config.
So instead of rte_eth_tx_queue_data_get(), you'll probably need:
eth_rx_direct_rearm_t rte_eth_get_rx_direct_rearm_func(rx_port, rx_queue, tx_port, tx_queue);
Then, it will be user responsibility to store it somewhere and call periodically:

control_path:

	...
	rearm_func = rte_eth_get_rx_direct_rearm_func(rxport, rxqueue,
		 txport, txqueue);
data-path:
	while(...) {
		rearm_func(rxport, txport, rxqueue, txqueue);
		rte_eth_rx_burst(rxport, rxqueue, ....);
		rte_eth_tx_burst(txport, txqueue, ....);
	}
Quoted 3 lines

In that case there seems absolutely no point to introduce struct rte_eth_txq_data. rx_direct_rearm() accesses TXQ private data directly anyway.

2. This is a very good proposal and it will be our first choice. Before working on it, I have a few questions about how to implement 'rearm_func'.
Like you say above, mixed Rx and Tx path code in 'rearm_func' means the hard-code is mixed like:

rearm_func(...) {
     ...
    txep = &txq->sw_ring[txq->tx_next_dd - (txq->tx_rs_thresh - 1)];
    for (...) {
       rxep[i].mbuf = txep[i].mbuf;
       mb0 = txep[i].mbuf;
       paddr = mb0->buf_iova + RTE_PKTMBUF_HEADROOM;
      dma_addr0 = vdupq_n_u64(paddr);
      vst1q_u64((uint64_t *)&rxdp++->read, dma_addr0);
    }
}

Is my understanding is right?

Quoted 7 lines

Another way - make rte_eth_txq_data totally opaque and allow PMD to store there some data that will help it to distinguish expected TXQ format. That will allow PMD to keep rx_direct_rearm() the same for all supported TXQ formats (it will make decision internally based on data stored in txq_data).
Though in that case you'll probably need one more dev-op to free txq_data.

#4 lore raw

Re: 回复: [PATCH v2 1/3] ethdev: add API for direct rearm mode

Quoted 274 lines
Quoted 256 lines
Quoted 13 lines

Add API for enabling direct rearm mode and for mapping RX and TX queues. Currently, the API supports 1:1(txq : rxq) mapping.

Furthermore, to avoid Rx load Tx data directly, add API called 'rte_eth_txq_data_get' to get Tx sw_ring and its information.

Quoted 5 lines
---
   lib/ethdev/ethdev_driver.h   |  9 ++++
   lib/ethdev/ethdev_private.c  |  1 +
   lib/ethdev/rte_ethdev.c      | 37 ++++++++++++++
   lib/ethdev/rte_ethdev.h      | 95

++++++++++++++++++++++++++++++++++++

Quoted 17 lines
   lib/ethdev/rte_ethdev_core.h |  5 ++
   lib/ethdev/version.map       |  4 ++
   6 files changed, 151 insertions(+)
lib/ethdev/ethdev_driver.h +2 -0
index 47a55a419e..14f52907c1 100644
@@ -58,6 +58,8 @@ struct rte_eth_dev {
   	eth_rx_descriptor_status_t rx_descriptor_status;
   	/** Check the status of a Tx descriptor */
   	eth_tx_descriptor_status_t tx_descriptor_status;
+	/**  Use Tx mbufs for Rx to rearm */
+	eth_rx_direct_rearm_t rx_direct_rearm;
 
   	/**
   	 * Device data that is shared between primary and secondary
Quoted 1 line

processes @@ -486,6 +488,11 @@ typedef int

(*eth_rx_enable_intr_t)(struct rte_eth_dev *dev,

lib/ethdev/ethdev_driver.h typedef int (*eth_rx_enable_intr_t)(struct rte_eth_dev *dev,

Quoted 9 lines

typedef int (*eth_rx_disable_intr_t)(struct rte_eth_dev *dev,
uint16_t rx_queue_id);

lib/ethdev/ethdev_driver.h +5 -0
+/**< @internal Get Tx information of a transmit queue of an Ethernet
+device. */ typedef void (*eth_txq_data_get_t)(struct rte_eth_dev *dev,
+				      uint16_t tx_queue_id,
+				      struct rte_eth_txq_data *txq_data);
+
   /** @internal Release memory resources allocated by given Rx/Tx queue.

*/

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1150

Quoted 9 lines

typedef void (*eth_queue_release_t)(struct rte_eth_dev *dev,
uint16_t queue_id);

lib/ethdev/ethdev_driver.h +2 -0
@@ -1138,6 +1145,8 @@ struct eth_dev_ops {
   	eth_rxq_info_get_t         rxq_info_get;
   	/** Retrieve Tx queue information */
   	eth_txq_info_get_t         txq_info_get;
+	/** Get the address where Tx data is stored */
+	eth_txq_data_get_t         txq_data_get;
   	eth_burst_mode_get_t       rx_burst_mode_get; /**< Get Rx burst

mode */

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1151

Quoted 1 line
lib/ethdev/ethdev_driver.h +0 -0
   	eth_burst_mode_get_t       tx_burst_mode_get; /**< Get Tx burst

mode */

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1152

Quoted 1 line
lib/ethdev/ethdev_driver.h +0 -0
   	eth_fw_version_get_t       fw_version_get; /**< Get firmware

version */

lib/ethdev/ethdev_private.c eth_dev_fp_ops_setup(struct rte_eth_fp_ops

Quoted 5 lines
lib/ethdev/ethdev_private.c +0 -0
index 48090c879a..bfe16c7d77 100644
@@ -276,6 +276,7 @@ eth_dev_fp_ops_setup(struct rte_eth_fp_ops

*fpo,

Quoted 15 lines
lib/ethdev/ethdev_private.c +1 -0
   	fpo->rx_queue_count = dev->rx_queue_count;
   	fpo->rx_descriptor_status = dev->rx_descriptor_status;
   	fpo->tx_descriptor_status = dev->tx_descriptor_status;
+	fpo->rx_direct_rearm = dev->rx_direct_rearm;
 
   	fpo->rxq.data = dev->data->rx_queues;
   	fpo->rxq.clbk = (void **)(uintptr_t)dev->post_rx_burst_cbs;
lib/ethdev/rte_ethdev.c index renamed from lib/ethdev/rte_ethdev.c +0 -0

0c2c1088c0..0dccec2e4b 100644

lib/ethdev/rte_ethdev.c +0 -0
@@ -1648,6 +1648,43 @@ rte_eth_dev_is_removed(uint16_t port_id)
   	return ret;
   }
 

lib/ethdev/rte_ethdev.c rte_eth_dev_is_removed(uint16_t port_id) · line 1660

Quoted 10 lines
lib/ethdev/rte_ethdev.c +10 -0
+int
+rte_eth_tx_queue_data_get(uint16_t port_id, uint16_t queue_id,
+			struct rte_eth_txq_data *txq_data) {
+	struct rte_eth_dev *dev;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+	dev = &rte_eth_devices[port_id];
+
+	if (queue_id >= dev->data->nb_tx_queues) {
+		RTE_ETHDEV_LOG(ERR, "Invalid Tx queue_id=%u\n",

queue_id);

lib/ethdev/rte_ethdev.c rte_eth_dev_is_removed(uint16_t port_id)

Quoted 5 lines
lib/ethdev/rte_ethdev.c +5 -0
+		return -EINVAL;
+	}
+
+	if (txq_data == NULL) {
+		RTE_ETHDEV_LOG(ERR, "Cannot get ethdev port %u Tx

queue %u data to NULL\n",

Quoted 29 lines
lib/ethdev/rte_ethdev.c +21 -0
+			port_id, queue_id);
+		return -EINVAL;
+	}
+
+	if (dev->data->tx_queues == NULL ||
+			dev->data->tx_queues[queue_id] == NULL) {
+		RTE_ETHDEV_LOG(ERR,
+			   "Tx queue %"PRIu16" of device with port_id=%"
+			   PRIu16" has not been setup\n",
+			   queue_id, port_id);
+		return -EINVAL;
+	}
+
+	if (*dev->dev_ops->txq_data_get == NULL)
+		return -ENOTSUP;
+
+	dev->dev_ops->txq_data_get(dev, queue_id, txq_data);
+
+	return 0;
+}
+
   static int
   rte_eth_rx_queue_check_split(const struct rte_eth_rxseg_split *rx_seg,
   			     uint16_t n_seg, uint32_t *mbp_buf_size, diff --git

a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h index 2e783536c1..daf7f05d62 100644

lib/ethdev/rte_ethdev.h struct rte_eth_txq_info { · line 1955

Quoted 7 lines
lib/ethdev/rte_ethdev.h +4 -0
@@ -1949,6 +1949,23 @@ struct rte_eth_txq_info {
   	uint8_t queue_state;        /**< one of RTE_ETH_QUEUE_STATE_*. */
   } __rte_cache_min_aligned;
 
+/**
+ * @internal
+ * Structure used to hold pointers to internal ethdev Tx data.
+ * The main purpose is to load and store Tx queue data in direct rearm

mode.

lib/ethdev/rte_ethdev.h struct rte_eth_txq_info { · line 1967

Quoted 12 lines
lib/ethdev/rte_ethdev.h +12 -0
+ */
+struct rte_eth_txq_data {
+	uint64_t *offloads;
+	void *tx_sw_ring;
+	volatile void *tx_ring;
+	uint16_t *tx_next_dd;
+	uint16_t *nb_tx_free;
+	uint16_t nb_tx_desc;
+	uint16_t tx_rs_thresh;
+	uint16_t tx_free_thresh;
+} __rte_cache_min_aligned;
+

first of all it is not clear why this struct has to be in public header, why it can't be in on of ethdev 'private' headers.
Second it looks like a snippet from private txq fields for some Intel (and alike) PMDs (i40e, ice, etc.).
How it supposed to to be universal and be applicable for any PMD that decides to implement this new API?

lib/ethdev/rte_ethdev.h int rte_eth_remove_rx_callback(uint16_t

Quoted 4 lines
lib/ethdev/rte_ethdev.h +0 -0
   /* Generic Burst mode flag definition, values can be ORed. */
 
   /**
@@ -4718,6 +4735,27 @@ int rte_eth_remove_rx_callback(uint16_t

port_id, uint16_t queue_id,

lib/ethdev/rte_ethdev.h rte_eth_tx_buffer(uint16_t port_id, uint16_t

Quoted 28 lines
lib/ethdev/rte_ethdev.h +21 -0
   int rte_eth_remove_tx_callback(uint16_t port_id, uint16_t queue_id,
   		const struct rte_eth_rxtx_callback *user_cb);
 
+/**
+ * Get the address which Tx data is stored.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param queue_id
+ *   The Tx queue on the Ethernet device for which information
+ *   will be retrieved.
+ * @param txq_data
+ *   A pointer to a structure of type *rte_eth_txq_data* to be filled.
+ *
+ * @return
+ *   - 0: Success
+ *   - -ENODEV:  If *port_id* is invalid.
+ *   - -ENOTSUP: routine is not supported by the device PMD.
+ *   - -EINVAL:  The queue_id is out of range.
+ */
+__rte_experimental
+int rte_eth_tx_queue_data_get(uint16_t port_id, uint16_t queue_id,
+		struct rte_eth_txq_data *txq_data);
+
   /**
    * Retrieve information about given port's Rx queue.
    *
@@ -6209,6 +6247,63 @@ rte_eth_tx_buffer(uint16_t port_id, uint16_t

queue_id,

Quoted 5 lines
lib/ethdev/rte_ethdev.h +2 -0
   	return rte_eth_tx_buffer_flush(port_id, queue_id, buffer);
   }
 
+/**
+ * @warning

lib/ethdev/rte_ethdev.h rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id,

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior
+notice
+ *
+ * Put Tx buffers into Rx sw-ring and rearm descs.
+ *
+ * @param port_id
+ *   Port identifying the receive side.
+ * @param queue_id
+ *   The index of the transmit queue identifying the receive side.
+ *   The value must be in the range [0, nb_rx_queue - 1] previously

supplied

Quoted 23 lines
lib/ethdev/rte_ethdev.h +23 -0
+ *   to rte_eth_dev_configure().
+ * @param txq_data
+ *   A pointer to a structure of type *rte_eth_txq_data* to be filled.
+ * @return
+ *   The number of direct-rearmed buffers.
+ */
+__rte_experimental
+static __rte_always_inline uint16_t
+rte_eth_rx_direct_rearm(uint16_t port_id, uint16_t queue_id,
+		struct rte_eth_txq_data *txq_data)
+{
+	uint16_t nb_rearm;
+	struct rte_eth_fp_ops *p;
+	void *qd;
+
+#ifdef RTE_ETHDEV_DEBUG_RX
+	if (port_id >= RTE_MAX_ETHPORTS ||
+			queue_id >= RTE_MAX_QUEUES_PER_PORT) {
+		RTE_ETHDEV_LOG(ERR,
+			"Invalid port_id=%u or queue_id=%u\n",
+			port_id, queue_id);
+		return 0;
+	}

lib/ethdev/rte_ethdev.h rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id,

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+#endif
+
+	p = &rte_eth_fp_ops[port_id];
+	qd = p->rxq.data[queue_id];
+
+#ifdef RTE_ETHDEV_DEBUG_RX
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, 0);
+
+	if (qd == NULL) {
+		RTE_ETHDEV_LOG(ERR, "Invalid Rx queue_id=%u for

port_id=%u\n",

lib/ethdev/rte_ethdev.h rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id, · line 6298

Quoted 5 lines
lib/ethdev/rte_ethdev.h +5 -0
+			queue_id, port_id);
+		return 0;
+	}
+
+	if (!p->rx_direct_rearm)

This check should be done always (unconditionally).
it is not a mandatory function for the driver (it can safely skip to implement it).

lib/ethdev/rte_ethdev.h rte_eth_tx_buffer(uint16_t port_id, uint16_t · line 6300

Quoted 1 line
lib/ethdev/rte_ethdev.h +1 -0
+		return -ENOTSUP;

This function returns uint16_t, why signed integers here?

Quoted 10 lines

lib/ethdev/rte_ethdev.h rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id, · line 6302

Quoted 3 lines
lib/ethdev/rte_ethdev.h +3 -0
+#endif
+
+	nb_rearm = p->rx_direct_rearm(qd, txq_data);

So rx_direct_rearm() function knows how to extract data from TX queue? As I understand that is possible only in one case:
rx_direct_rearm() has full knowledge and acess of txq internals, etc. That means that rxq and txq have to belong to the same driver and device type.

Thanks for the comments, and I have some questions for this.

Quoted 6 lines

First of all, I still think it is not the best design choice. If we going ahead with introducing this feature, it better be as generic as possible.
Plus it mixes TX and RX code-paths together, while it would be much better to to keep them independent as they are right now.

Quoted 7 lines
Quoted 3 lines

Another thing with such approach - even for the same PMD, both TXQ and RXQ can have different internal data format and behavior logic (depending on port/queue configuration).

1. Here TXQ and RXQ have different internal format means the queue type and descs can be different, right? If I understand correctly, based on your first strategy, is it means we will need different 'rearm_func' for different queue type in the same PMD?

Yes, I think so.
If let say we have some PMD where depending on the config,
there cpuld be 2 different RXQ formats: rxq_a and rxq_b,
and 2 different txq formats: txq_c, txq_d.
Then assuming PMD would like to support direct-rearm mode for all four combinations, it needs 4 different rearm functions:

rearm_txq_c_to_rxq_a()
rearm_txq_c_to_rxq_b()
rearm_txq_d_to_rxq_a()
rearm_txq_d_to_rxq_b()

Quoted 38 lines
Quoted 20 lines

So rx_direct_rearm() function selection have to be done based on both RXQ and TXQ config.
So instead of rte_eth_tx_queue_data_get(), you'll probably need:
eth_rx_direct_rearm_t rte_eth_get_rx_direct_rearm_func(rx_port, rx_queue, tx_port, tx_queue);
Then, it will be user responsibility to store it somewhere and call periodically:

control_path:

	...
	rearm_func = rte_eth_get_rx_direct_rearm_func(rxport, rxqueue,
		 txport, txqueue);
data-path:
	while(...) {
		rearm_func(rxport, txport, rxqueue, txqueue);
		rte_eth_rx_burst(rxport, rxqueue, ....);
		rte_eth_tx_burst(txport, txqueue, ....);
	}
Quoted 3 lines

In that case there seems absolutely no point to introduce struct rte_eth_txq_data. rx_direct_rearm() accesses TXQ private data directly anyway.

2. This is a very good proposal and it will be our first choice. Before working on it, I have a few questions about how to implement 'rearm_func'.
Like you say above, mixed Rx and Tx path code in 'rearm_func' means the hard-code is mixed like:

rearm_func(...) {
      ...
     txep = &txq->sw_ring[txq->tx_next_dd - (txq->tx_rs_thresh - 1)];
     for (...) {
        rxep[i].mbuf = txep[i].mbuf;
        mb0 = txep[i].mbuf;
        paddr = mb0->buf_iova + RTE_PKTMBUF_HEADROOM;
       dma_addr0 = vdupq_n_u64(paddr);
       vst1q_u64((uint64_t *)&rxdp++->read, dma_addr0);
     }
}
Quoted 1 line

Is my understanding is right?

Sorry, I don't understand the question.
Can you probably elaborate a bit?

Quoted 9 lines
Quoted 8 lines

Another way - make rte_eth_txq_data totally opaque and allow PMD to store there some data that will help it to distinguish expected TXQ format. That will allow PMD to keep rx_direct_rearm() the same for all supported TXQ formats (it will make decision internally based on data stored in txq_data).
Though in that case you'll probably need one more dev-op to free txq_data.

#5 lore raw

回复: 回复: [PATCH v2 1/3] ethdev: add API for direct rearm mode

Quoted 306 lines

-----邮件原件-----
发件人: Konstantin Ananyev <[email protected]> 发送时间: Wednesday, October 12, 2022 6:21 AM
收件人: Feifei Wang <[email protected]>; [email protected]; Ferruh Yigit <[email protected]>; Andrew Rybchenko <[email protected]>; Ray Kinsella <[email protected]> 抄送: [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; Ruifeng Wang <[email protected]>
主题: Re: 回复: [PATCH v2 1/3] ethdev: add API for direct rearm mode

Quoted 36 lines
Quoted 36 lines
Quoted 11 lines

Add API for enabling direct rearm mode and for mapping RX and TX queues. Currently, the API supports 1:1(txq : rxq) mapping.

Furthermore, to avoid Rx load Tx data directly, add API called 'rte_eth_txq_data_get' to get Tx sw_ring and its information.

Quoted 5 lines
---
   lib/ethdev/ethdev_driver.h   |  9 ++++
   lib/ethdev/ethdev_private.c  |  1 +
   lib/ethdev/rte_ethdev.c      | 37 ++++++++++++++
   lib/ethdev/rte_ethdev.h      | 95

++++++++++++++++++++++++++++++++++++

Quoted 17 lines
   lib/ethdev/rte_ethdev_core.h |  5 ++
   lib/ethdev/version.map       |  4 ++
   6 files changed, 151 insertions(+)
lib/ethdev/ethdev_driver.h +2 -0
index 47a55a419e..14f52907c1 100644
@@ -58,6 +58,8 @@ struct rte_eth_dev {
   	eth_rx_descriptor_status_t rx_descriptor_status;
   	/** Check the status of a Tx descriptor */
   	eth_tx_descriptor_status_t tx_descriptor_status;
+	/**  Use Tx mbufs for Rx to rearm */
+	eth_rx_direct_rearm_t rx_direct_rearm;
 
   	/**
   	 * Device data that is shared between primary and secondary
Quoted 1 line

processes @@ -486,6 +488,11 @@ typedef int

(*eth_rx_enable_intr_t)(struct rte_eth_dev *dev,

Quoted 5 lines
Quoted 5 lines

lib/ethdev/ethdev_driver.h

Quoted 5 lines

typedef int (*eth_rx_disable_intr_t)(struct rte_eth_dev *dev,
uint16_t rx_queue_id);

lib/ethdev/ethdev_driver.h +2 -0
+/**< @internal Get Tx information of a transmit queue of an
+Ethernet device. */ typedef void (*eth_txq_data_get_t)(struct

rte_eth_dev *dev,

Quoted 4 lines
Quoted 4 lines

lib/ethdev/ethdev_driver.h typedef int (*eth_rx_enable_intr_t)(struct rte_eth_dev *dev,

Quoted 4 lines
lib/ethdev/ethdev_driver.h +3 -0
+				      uint16_t tx_queue_id,
+				      struct rte_eth_txq_data *txq_data);
+
   /** @internal Release memory resources allocated by given Rx/Tx

queue.

Quoted 68 lines
Quoted 68 lines

*/

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1150

Quoted 9 lines

typedef void (*eth_queue_release_t)(struct rte_eth_dev *dev,
uint16_t queue_id);

lib/ethdev/ethdev_driver.h +2 -0
@@ -1138,6 +1145,8 @@ struct eth_dev_ops {
   	eth_rxq_info_get_t         rxq_info_get;
   	/** Retrieve Tx queue information */
   	eth_txq_info_get_t         txq_info_get;
+	/** Get the address where Tx data is stored */
+	eth_txq_data_get_t         txq_data_get;
   	eth_burst_mode_get_t       rx_burst_mode_get; /**< Get Rx burst

mode */

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1151

Quoted 1 line
lib/ethdev/ethdev_driver.h +0 -0
   	eth_burst_mode_get_t       tx_burst_mode_get; /**< Get Tx burst

mode */

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1152

Quoted 1 line
lib/ethdev/ethdev_driver.h +0 -0
   	eth_fw_version_get_t       fw_version_get; /**< Get firmware

version */

lib/ethdev/ethdev_private.c eth_dev_fp_ops_setup(struct rte_eth_fp_ops

Quoted 5 lines
(quoted diff) +0 -0

b/lib/ethdev/ethdev_private.c index 48090c879a..bfe16c7d77 100644

lib/ethdev/ethdev_private.c +0 -0
@@ -276,6 +276,7 @@ eth_dev_fp_ops_setup(struct rte_eth_fp_ops

*fpo,

Quoted 15 lines
lib/ethdev/ethdev_private.c +1 -0
   	fpo->rx_queue_count = dev->rx_queue_count;
   	fpo->rx_descriptor_status = dev->rx_descriptor_status;
   	fpo->tx_descriptor_status = dev->tx_descriptor_status;
+	fpo->rx_direct_rearm = dev->rx_direct_rearm;
 
   	fpo->rxq.data = dev->data->rx_queues;
   	fpo->rxq.clbk = (void **)(uintptr_t)dev->post_rx_burst_cbs;
lib/ethdev/rte_ethdev.c index renamed from lib/ethdev/rte_ethdev.c +0 -0

0c2c1088c0..0dccec2e4b 100644

lib/ethdev/rte_ethdev.c +0 -0
@@ -1648,6 +1648,43 @@ rte_eth_dev_is_removed(uint16_t port_id)
   	return ret;
   }
 

lib/ethdev/rte_ethdev.c rte_eth_dev_is_removed(uint16_t port_id) · line 1660

Quoted 10 lines
lib/ethdev/rte_ethdev.c +10 -0
+int
+rte_eth_tx_queue_data_get(uint16_t port_id, uint16_t queue_id,
+			struct rte_eth_txq_data *txq_data) {
+	struct rte_eth_dev *dev;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+	dev = &rte_eth_devices[port_id];
+
+	if (queue_id >= dev->data->nb_tx_queues) {
+		RTE_ETHDEV_LOG(ERR, "Invalid Tx queue_id=%u\n",

queue_id);

lib/ethdev/rte_ethdev.c rte_eth_dev_is_removed(uint16_t port_id)

Quoted 5 lines
lib/ethdev/rte_ethdev.c +5 -0
+		return -EINVAL;
+	}
+
+	if (txq_data == NULL) {
+		RTE_ETHDEV_LOG(ERR, "Cannot get ethdev port %u Tx

queue %u data to NULL\n",

Quoted 15 lines
lib/ethdev/rte_ethdev.c +15 -0
+			port_id, queue_id);
+		return -EINVAL;
+	}
+
+	if (dev->data->tx_queues == NULL ||
+			dev->data->tx_queues[queue_id] == NULL) {
+		RTE_ETHDEV_LOG(ERR,
+			   "Tx queue %"PRIu16" of device with port_id=%"
+			   PRIu16" has not been setup\n",
+			   queue_id, port_id);
+		return -EINVAL;
+	}
+
+	if (*dev->dev_ops->txq_data_get == NULL)
+		return -ENOTSUP;
Quoted 8 lines
Quoted 8 lines

lib/ethdev/rte_ethdev.c rte_eth_dev_is_removed(uint16_t port_id)

Quoted 8 lines
lib/ethdev/rte_ethdev.c +6 -0
+
+	dev->dev_ops->txq_data_get(dev, queue_id, txq_data);
+
+	return 0;
+}
+
   static int
   rte_eth_rx_queue_check_split(const struct rte_eth_rxseg_split

*rx_seg,

Quoted 119 lines
Quoted 119 lines
Quoted 6 lines
lib/ethdev/rte_ethdev.c +0 -0
   			     uint16_t n_seg, uint32_t *mbp_buf_size, diff --git

a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h index 2e783536c1..daf7f05d62 100644

lib/ethdev/rte_ethdev.h struct rte_eth_txq_info { · line 1956

Quoted 8 lines
lib/ethdev/rte_ethdev.h +5 -0
@@ -1949,6 +1949,23 @@ struct rte_eth_txq_info {
   	uint8_t queue_state;        /**< one of RTE_ETH_QUEUE_STATE_*. */
   } __rte_cache_min_aligned;
 
+/**
+ * @internal
+ * Structure used to hold pointers to internal ethdev Tx data.
+ * The main purpose is to load and store Tx queue data in direct
+rearm

mode.

lib/ethdev/rte_ethdev.h struct rte_eth_txq_info { · line 1967

Quoted 12 lines
lib/ethdev/rte_ethdev.h +12 -0
+ */
+struct rte_eth_txq_data {
+	uint64_t *offloads;
+	void *tx_sw_ring;
+	volatile void *tx_ring;
+	uint16_t *tx_next_dd;
+	uint16_t *nb_tx_free;
+	uint16_t nb_tx_desc;
+	uint16_t tx_rs_thresh;
+	uint16_t tx_free_thresh;
+} __rte_cache_min_aligned;
+

first of all it is not clear why this struct has to be in public header, why it can't be in on of ethdev 'private' headers. Second it looks like a snippet from private txq fields for some Intel (and alike) PMDs (i40e, ice, etc.).
How it supposed to to be universal and be applicable for any PMD that decides to implement this new API?

lib/ethdev/rte_ethdev.h int rte_eth_remove_rx_callback(uint16_t

Quoted 4 lines
lib/ethdev/rte_ethdev.h +0 -0
   /* Generic Burst mode flag definition, values can be ORed. */
 
   /**
@@ -4718,6 +4735,27 @@ int rte_eth_remove_rx_callback(uint16_t

port_id, uint16_t queue_id,

lib/ethdev/rte_ethdev.h rte_eth_tx_buffer(uint16_t port_id, uint16_t

Quoted 28 lines
lib/ethdev/rte_ethdev.h +21 -0
   int rte_eth_remove_tx_callback(uint16_t port_id, uint16_t queue_id,
   		const struct rte_eth_rxtx_callback *user_cb);
 
+/**
+ * Get the address which Tx data is stored.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param queue_id
+ *   The Tx queue on the Ethernet device for which information
+ *   will be retrieved.
+ * @param txq_data
+ *   A pointer to a structure of type *rte_eth_txq_data* to be filled.
+ *
+ * @return
+ *   - 0: Success
+ *   - -ENODEV:  If *port_id* is invalid.
+ *   - -ENOTSUP: routine is not supported by the device PMD.
+ *   - -EINVAL:  The queue_id is out of range.
+ */
+__rte_experimental
+int rte_eth_tx_queue_data_get(uint16_t port_id, uint16_t queue_id,
+		struct rte_eth_txq_data *txq_data);
+
   /**
    * Retrieve information about given port's Rx queue.
    *
@@ -6209,6 +6247,63 @@ rte_eth_tx_buffer(uint16_t port_id, uint16_t

queue_id,

Quoted 5 lines
lib/ethdev/rte_ethdev.h +2 -0
   	return rte_eth_tx_buffer_flush(port_id, queue_id, buffer);
   }
 
+/**
+ * @warning

lib/ethdev/rte_ethdev.h rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id,

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+ * @b EXPERIMENTAL: this API may change, or be removed, without
+prior notice
+ *
+ * Put Tx buffers into Rx sw-ring and rearm descs.
+ *
+ * @param port_id
+ *   Port identifying the receive side.
+ * @param queue_id
+ *   The index of the transmit queue identifying the receive side.
+ *   The value must be in the range [0, nb_rx_queue - 1] previously

supplied

Quoted 22 lines
lib/ethdev/rte_ethdev.h +22 -0
+ *   to rte_eth_dev_configure().
+ * @param txq_data
+ *   A pointer to a structure of type *rte_eth_txq_data* to be filled.
+ * @return
+ *   The number of direct-rearmed buffers.
+ */
+__rte_experimental
+static __rte_always_inline uint16_t
+rte_eth_rx_direct_rearm(uint16_t port_id, uint16_t queue_id,
+		struct rte_eth_txq_data *txq_data) {
+	uint16_t nb_rearm;
+	struct rte_eth_fp_ops *p;
+	void *qd;
+
+#ifdef RTE_ETHDEV_DEBUG_RX
+	if (port_id >= RTE_MAX_ETHPORTS ||
+			queue_id >= RTE_MAX_QUEUES_PER_PORT) {
+		RTE_ETHDEV_LOG(ERR,
+			"Invalid port_id=%u or queue_id=%u\n",
+			port_id, queue_id);
+		return 0;
+	}

lib/ethdev/rte_ethdev.h rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id,

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+#endif
+
+	p = &rte_eth_fp_ops[port_id];
+	qd = p->rxq.data[queue_id];
+
+#ifdef RTE_ETHDEV_DEBUG_RX
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, 0);
+
+	if (qd == NULL) {
+		RTE_ETHDEV_LOG(ERR, "Invalid Rx queue_id=%u for

port_id=%u\n",

Quoted 8 lines
Quoted 8 lines

lib/ethdev/rte_ethdev.h rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id, · line 6298

Quoted 5 lines
lib/ethdev/rte_ethdev.h +5 -0
+			queue_id, port_id);
+		return 0;
+	}
+
+	if (!p->rx_direct_rearm)

This check should be done always (unconditionally).
it is not a mandatory function for the driver (it can safely skip to

implement it).

Quoted 24 lines
Quoted 6 lines

lib/ethdev/rte_ethdev.h rte_eth_tx_buffer(uint16_t port_id, uint16_t · line 6299

Quoted 2 lines
lib/ethdev/rte_ethdev.h +1 -0
+		return -ENOTSUP;

This function returns uint16_t, why signed integers here?

Quoted 10 lines

lib/ethdev/rte_ethdev.h rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id, · line 6302

Quoted 3 lines
lib/ethdev/rte_ethdev.h +3 -0
+#endif
+
+	nb_rearm = p->rx_direct_rearm(qd, txq_data);

So rx_direct_rearm() function knows how to extract data from TX queue? As I understand that is possible only in one case:
rx_direct_rearm() has full knowledge and acess of txq internals, etc. That means that rxq and txq have to belong to the same driver and device type.

Thanks for the comments, and I have some questions for this.

Quoted 6 lines

First of all, I still think it is not the best design choice. If we going ahead with introducing this feature, it better be as generic as possible.
Plus it mixes TX and RX code-paths together, while it would be much better to to keep them independent as they are right now.

Quoted 7 lines
Quoted 3 lines

Another thing with such approach - even for the same PMD, both TXQ and RXQ can have different internal data format and behavior logic (depending on port/queue configuration).

1. Here TXQ and RXQ have different internal format means the queue type and descs can be different, right? If I understand correctly, based on your first strategy, is it means we will need different 'rearm_func' for different queue type in the same PMD?

Yes, I think so.
If let say we have some PMD where depending on the config, there cpuld be 2 different RXQ formats: rxq_a and rxq_b, and 2 different txq formats: txq_c, txq_d.
Then assuming PMD would like to support direct-rearm mode for all four combinations, it needs 4 different rearm functions:

Quoted 5 lines

rearm_txq_c_to_rxq_a()
rearm_txq_c_to_rxq_b()
rearm_txq_d_to_rxq_a()
rearm_txq_d_to_rxq_b()

Thank you for your detailed explanation, I can understand this.

Quoted 44 lines
Quoted 8 lines
Quoted 8 lines

So rx_direct_rearm() function selection have to be done based on both RXQ and TXQ config.
So instead of rte_eth_tx_queue_data_get(), you'll probably need:
eth_rx_direct_rearm_t rte_eth_get_rx_direct_rearm_func(rx_port, rx_queue, tx_port, tx_queue);
Then, it will be user responsibility to store it somewhere and call

periodically:

Quoted 13 lines
Quoted 13 lines

control_path:

	...
	rearm_func = rte_eth_get_rx_direct_rearm_func(rxport, rxqueue,
		 txport, txqueue);
data-path:
	while(...) {
		rearm_func(rxport, txport, rxqueue, txqueue);
		rte_eth_rx_burst(rxport, rxqueue, ....);
		rte_eth_tx_burst(txport, txqueue, ....);
	}
Quoted 5 lines
Quoted 3 lines

In that case there seems absolutely no point to introduce struct rte_eth_txq_data. rx_direct_rearm() accesses TXQ private data directly anyway.

2. This is a very good proposal and it will be our first choice. Before working on it, I have a few questions about how to implement

'rearm_func'.

Quoted 13 lines

Like you say above, mixed Rx and Tx path code in 'rearm_func' means the hard-code is mixed like:

rearm_func(...) {
      ...
     txep = &txq->sw_ring[txq->tx_next_dd - (txq->tx_rs_thresh - 1)];
     for (...) {
        rxep[i].mbuf = txep[i].mbuf;
        mb0 = txep[i].mbuf;
        paddr = mb0->buf_iova + RTE_PKTMBUF_HEADROOM;
       dma_addr0 = vdupq_n_u64(paddr);
       vst1q_u64((uint64_t *)&rxdp++->read, dma_addr0);
     }
}
Quoted 1 line

Is my understanding is right?

Quoted 2 lines

Sorry, I don't understand the question.
Can you probably elaborate a bit?

Sorry for my unclear expression.

I mean if we need two func which contains tx and rx paths code respectively in rearm_func, like:

rearm_func(...) {
         rte_tx_fill_sw_ring;
         rte_rx_rearm_descs;
}

Or just mixed tx and rx path code like I said before. I prefer 'rx and tx hard code mixed', because from the performance perspective, this can reduce the cost of function calls.

Quoted 11 lines
Quoted 9 lines
Quoted 9 lines

Another way - make rte_eth_txq_data totally opaque and allow PMD to store there some data that will help it to distinguish expected TXQ format. That will allow PMD to keep rx_direct_rearm() the same for all supported TXQ formats (it will make decision internally based on data stored in txq_data).
Though in that case you'll probably need one more dev-op to free

txq_data.

Quoted 1 line
#6 lore raw

Re: 回复: 回复: [PATCH v2 1/3] ethdev: add API for direct rearm mode

12/10/2022 14:38, Feifei Wang пишет:

Quoted 361 lines
Quoted 306 lines

-----邮件原件-----
发件人: Konstantin Ananyev <[email protected]> 发送时间: Wednesday, October 12, 2022 6:21 AM
收件人: Feifei Wang <[email protected]>; [email protected]; Ferruh Yigit <[email protected]>; Andrew Rybchenko <[email protected]>; Ray Kinsella <[email protected]> 抄送: [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; Ruifeng Wang <[email protected]>
主题: Re: 回复: [PATCH v2 1/3] ethdev: add API for direct rearm mode

Quoted 36 lines
Quoted 36 lines
Quoted 11 lines

Add API for enabling direct rearm mode and for mapping RX and TX queues. Currently, the API supports 1:1(txq : rxq) mapping.

Furthermore, to avoid Rx load Tx data directly, add API called 'rte_eth_txq_data_get' to get Tx sw_ring and its information.

Quoted 5 lines
---
    lib/ethdev/ethdev_driver.h   |  9 ++++
    lib/ethdev/ethdev_private.c  |  1 +
    lib/ethdev/rte_ethdev.c      | 37 ++++++++++++++
    lib/ethdev/rte_ethdev.h      | 95

++++++++++++++++++++++++++++++++++++

Quoted 17 lines
    lib/ethdev/rte_ethdev_core.h |  5 ++
    lib/ethdev/version.map       |  4 ++
    6 files changed, 151 insertions(+)
lib/ethdev/ethdev_driver.h +2 -0
index 47a55a419e..14f52907c1 100644
@@ -58,6 +58,8 @@ struct rte_eth_dev {
    	eth_rx_descriptor_status_t rx_descriptor_status;
    	/** Check the status of a Tx descriptor */
    	eth_tx_descriptor_status_t tx_descriptor_status;
+	/**  Use Tx mbufs for Rx to rearm */
+	eth_rx_direct_rearm_t rx_direct_rearm;
 
    	/**
    	 * Device data that is shared between primary and secondary
Quoted 1 line

processes @@ -486,6 +488,11 @@ typedef int

(*eth_rx_enable_intr_t)(struct rte_eth_dev *dev,

Quoted 5 lines
Quoted 5 lines

lib/ethdev/ethdev_driver.h

Quoted 5 lines

typedef int (*eth_rx_disable_intr_t)(struct rte_eth_dev *dev,
uint16_t rx_queue_id);

lib/ethdev/ethdev_driver.h +2 -0
+/**< @internal Get Tx information of a transmit queue of an
+Ethernet device. */ typedef void (*eth_txq_data_get_t)(struct

rte_eth_dev *dev,

Quoted 4 lines
Quoted 4 lines

lib/ethdev/ethdev_driver.h typedef int (*eth_rx_enable_intr_t)(struct rte_eth_dev *dev,

Quoted 4 lines
lib/ethdev/ethdev_driver.h +3 -0
+				      uint16_t tx_queue_id,
+				      struct rte_eth_txq_data *txq_data);
+
    /** @internal Release memory resources allocated by given Rx/Tx

queue.

Quoted 68 lines
Quoted 68 lines

*/

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1150

Quoted 9 lines

typedef void (*eth_queue_release_t)(struct rte_eth_dev *dev,
uint16_t queue_id);

lib/ethdev/ethdev_driver.h +2 -0
@@ -1138,6 +1145,8 @@ struct eth_dev_ops {
    	eth_rxq_info_get_t         rxq_info_get;
    	/** Retrieve Tx queue information */
    	eth_txq_info_get_t         txq_info_get;
+	/** Get the address where Tx data is stored */
+	eth_txq_data_get_t         txq_data_get;
    	eth_burst_mode_get_t       rx_burst_mode_get; /**< Get Rx burst

mode */

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1151

Quoted 1 line
lib/ethdev/ethdev_driver.h +0 -0
    	eth_burst_mode_get_t       tx_burst_mode_get; /**< Get Tx burst

mode */

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1152

Quoted 1 line
lib/ethdev/ethdev_driver.h +0 -0
    	eth_fw_version_get_t       fw_version_get; /**< Get firmware

version */

lib/ethdev/ethdev_private.c eth_dev_fp_ops_setup(struct rte_eth_fp_ops

Quoted 5 lines
(quoted diff) +0 -0

b/lib/ethdev/ethdev_private.c index 48090c879a..bfe16c7d77 100644

lib/ethdev/ethdev_private.c +0 -0
@@ -276,6 +276,7 @@ eth_dev_fp_ops_setup(struct rte_eth_fp_ops

*fpo,

Quoted 15 lines
lib/ethdev/ethdev_private.c +1 -0
    	fpo->rx_queue_count = dev->rx_queue_count;
    	fpo->rx_descriptor_status = dev->rx_descriptor_status;
    	fpo->tx_descriptor_status = dev->tx_descriptor_status;
+	fpo->rx_direct_rearm = dev->rx_direct_rearm;
 
    	fpo->rxq.data = dev->data->rx_queues;
    	fpo->rxq.clbk = (void **)(uintptr_t)dev->post_rx_burst_cbs;
lib/ethdev/rte_ethdev.c index renamed from lib/ethdev/rte_ethdev.c +0 -0

0c2c1088c0..0dccec2e4b 100644

lib/ethdev/rte_ethdev.c +0 -0
@@ -1648,6 +1648,43 @@ rte_eth_dev_is_removed(uint16_t port_id)
    	return ret;
    }
 

lib/ethdev/rte_ethdev.c rte_eth_dev_is_removed(uint16_t port_id) · line 1660

Quoted 10 lines
lib/ethdev/rte_ethdev.c +10 -0
+int
+rte_eth_tx_queue_data_get(uint16_t port_id, uint16_t queue_id,
+			struct rte_eth_txq_data *txq_data) {
+	struct rte_eth_dev *dev;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+	dev = &rte_eth_devices[port_id];
+
+	if (queue_id >= dev->data->nb_tx_queues) {
+		RTE_ETHDEV_LOG(ERR, "Invalid Tx queue_id=%u\n",

queue_id);

lib/ethdev/rte_ethdev.c rte_eth_dev_is_removed(uint16_t port_id)

Quoted 5 lines
lib/ethdev/rte_ethdev.c +5 -0
+		return -EINVAL;
+	}
+
+	if (txq_data == NULL) {
+		RTE_ETHDEV_LOG(ERR, "Cannot get ethdev port %u Tx

queue %u data to NULL\n",

Quoted 15 lines
lib/ethdev/rte_ethdev.c +15 -0
+			port_id, queue_id);
+		return -EINVAL;
+	}
+
+	if (dev->data->tx_queues == NULL ||
+			dev->data->tx_queues[queue_id] == NULL) {
+		RTE_ETHDEV_LOG(ERR,
+			   "Tx queue %"PRIu16" of device with port_id=%"
+			   PRIu16" has not been setup\n",
+			   queue_id, port_id);
+		return -EINVAL;
+	}
+
+	if (*dev->dev_ops->txq_data_get == NULL)
+		return -ENOTSUP;
Quoted 8 lines
Quoted 8 lines

lib/ethdev/rte_ethdev.c rte_eth_dev_is_removed(uint16_t port_id)

Quoted 8 lines
lib/ethdev/rte_ethdev.c +6 -0
+
+	dev->dev_ops->txq_data_get(dev, queue_id, txq_data);
+
+	return 0;
+}
+
    static int
    rte_eth_rx_queue_check_split(const struct rte_eth_rxseg_split

*rx_seg,

Quoted 119 lines
Quoted 119 lines
Quoted 6 lines
lib/ethdev/rte_ethdev.c +0 -0
    			     uint16_t n_seg, uint32_t *mbp_buf_size, diff --git

a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h index 2e783536c1..daf7f05d62 100644

lib/ethdev/rte_ethdev.h struct rte_eth_txq_info { · line 1956

Quoted 8 lines
lib/ethdev/rte_ethdev.h +5 -0
@@ -1949,6 +1949,23 @@ struct rte_eth_txq_info {
    	uint8_t queue_state;        /**< one of RTE_ETH_QUEUE_STATE_*. */
    } __rte_cache_min_aligned;
 
+/**
+ * @internal
+ * Structure used to hold pointers to internal ethdev Tx data.
+ * The main purpose is to load and store Tx queue data in direct
+rearm

mode.

lib/ethdev/rte_ethdev.h struct rte_eth_txq_info { · line 1967

Quoted 12 lines
lib/ethdev/rte_ethdev.h +12 -0
+ */
+struct rte_eth_txq_data {
+	uint64_t *offloads;
+	void *tx_sw_ring;
+	volatile void *tx_ring;
+	uint16_t *tx_next_dd;
+	uint16_t *nb_tx_free;
+	uint16_t nb_tx_desc;
+	uint16_t tx_rs_thresh;
+	uint16_t tx_free_thresh;
+} __rte_cache_min_aligned;
+

first of all it is not clear why this struct has to be in public header, why it can't be in on of ethdev 'private' headers. Second it looks like a snippet from private txq fields for some Intel (and alike) PMDs (i40e, ice, etc.).
How it supposed to to be universal and be applicable for any PMD that decides to implement this new API?

lib/ethdev/rte_ethdev.h int rte_eth_remove_rx_callback(uint16_t

Quoted 4 lines
lib/ethdev/rte_ethdev.h +0 -0
    /* Generic Burst mode flag definition, values can be ORed. */
 
    /**
@@ -4718,6 +4735,27 @@ int rte_eth_remove_rx_callback(uint16_t

port_id, uint16_t queue_id,

lib/ethdev/rte_ethdev.h rte_eth_tx_buffer(uint16_t port_id, uint16_t

Quoted 28 lines
lib/ethdev/rte_ethdev.h +21 -0
    int rte_eth_remove_tx_callback(uint16_t port_id, uint16_t queue_id,
    		const struct rte_eth_rxtx_callback *user_cb);
 
+/**
+ * Get the address which Tx data is stored.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param queue_id
+ *   The Tx queue on the Ethernet device for which information
+ *   will be retrieved.
+ * @param txq_data
+ *   A pointer to a structure of type *rte_eth_txq_data* to be filled.
+ *
+ * @return
+ *   - 0: Success
+ *   - -ENODEV:  If *port_id* is invalid.
+ *   - -ENOTSUP: routine is not supported by the device PMD.
+ *   - -EINVAL:  The queue_id is out of range.
+ */
+__rte_experimental
+int rte_eth_tx_queue_data_get(uint16_t port_id, uint16_t queue_id,
+		struct rte_eth_txq_data *txq_data);
+
    /**
     * Retrieve information about given port's Rx queue.
     *
@@ -6209,6 +6247,63 @@ rte_eth_tx_buffer(uint16_t port_id, uint16_t

queue_id,

Quoted 5 lines
lib/ethdev/rte_ethdev.h +2 -0
    	return rte_eth_tx_buffer_flush(port_id, queue_id, buffer);
    }
 
+/**
+ * @warning

lib/ethdev/rte_ethdev.h rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id,

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+ * @b EXPERIMENTAL: this API may change, or be removed, without
+prior notice
+ *
+ * Put Tx buffers into Rx sw-ring and rearm descs.
+ *
+ * @param port_id
+ *   Port identifying the receive side.
+ * @param queue_id
+ *   The index of the transmit queue identifying the receive side.
+ *   The value must be in the range [0, nb_rx_queue - 1] previously

supplied

Quoted 22 lines
lib/ethdev/rte_ethdev.h +22 -0
+ *   to rte_eth_dev_configure().
+ * @param txq_data
+ *   A pointer to a structure of type *rte_eth_txq_data* to be filled.
+ * @return
+ *   The number of direct-rearmed buffers.
+ */
+__rte_experimental
+static __rte_always_inline uint16_t
+rte_eth_rx_direct_rearm(uint16_t port_id, uint16_t queue_id,
+		struct rte_eth_txq_data *txq_data) {
+	uint16_t nb_rearm;
+	struct rte_eth_fp_ops *p;
+	void *qd;
+
+#ifdef RTE_ETHDEV_DEBUG_RX
+	if (port_id >= RTE_MAX_ETHPORTS ||
+			queue_id >= RTE_MAX_QUEUES_PER_PORT) {
+		RTE_ETHDEV_LOG(ERR,
+			"Invalid port_id=%u or queue_id=%u\n",
+			port_id, queue_id);
+		return 0;
+	}

lib/ethdev/rte_ethdev.h rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id,

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+#endif
+
+	p = &rte_eth_fp_ops[port_id];
+	qd = p->rxq.data[queue_id];
+
+#ifdef RTE_ETHDEV_DEBUG_RX
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, 0);
+
+	if (qd == NULL) {
+		RTE_ETHDEV_LOG(ERR, "Invalid Rx queue_id=%u for

port_id=%u\n",

Quoted 8 lines
Quoted 8 lines

lib/ethdev/rte_ethdev.h rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id, · line 6298

Quoted 5 lines
lib/ethdev/rte_ethdev.h +5 -0
+			queue_id, port_id);
+		return 0;
+	}
+
+	if (!p->rx_direct_rearm)

This check should be done always (unconditionally).
it is not a mandatory function for the driver (it can safely skip to

implement it).

Quoted 24 lines
Quoted 6 lines

lib/ethdev/rte_ethdev.h rte_eth_tx_buffer(uint16_t port_id, uint16_t · line 6299

Quoted 2 lines
lib/ethdev/rte_ethdev.h +1 -0
+		return -ENOTSUP;

This function returns uint16_t, why signed integers here?

Quoted 10 lines

lib/ethdev/rte_ethdev.h rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id, · line 6302

Quoted 3 lines
lib/ethdev/rte_ethdev.h +3 -0
+#endif
+
+	nb_rearm = p->rx_direct_rearm(qd, txq_data);

So rx_direct_rearm() function knows how to extract data from TX queue? As I understand that is possible only in one case:
rx_direct_rearm() has full knowledge and acess of txq internals, etc. That means that rxq and txq have to belong to the same driver and device type.

Thanks for the comments, and I have some questions for this.

Quoted 6 lines

First of all, I still think it is not the best design choice. If we going ahead with introducing this feature, it better be as generic as possible.
Plus it mixes TX and RX code-paths together, while it would be much better to to keep them independent as they are right now.

Quoted 7 lines
Quoted 3 lines

Another thing with such approach - even for the same PMD, both TXQ and RXQ can have different internal data format and behavior logic (depending on port/queue configuration).

1. Here TXQ and RXQ have different internal format means the queue type and descs can be different, right? If I understand correctly, based on your first strategy, is it means we will need different 'rearm_func' for different queue type in the same PMD?

Yes, I think so.
If let say we have some PMD where depending on the config, there cpuld be 2 different RXQ formats: rxq_a and rxq_b, and 2 different txq formats: txq_c, txq_d.
Then assuming PMD would like to support direct-rearm mode for all four combinations, it needs 4 different rearm functions:

Quoted 5 lines

rearm_txq_c_to_rxq_a()
rearm_txq_c_to_rxq_b()
rearm_txq_d_to_rxq_a()
rearm_txq_d_to_rxq_b()

Thank you for your detailed explanation, I can understand this.

Quoted 44 lines
Quoted 8 lines
Quoted 8 lines

So rx_direct_rearm() function selection have to be done based on both RXQ and TXQ config.
So instead of rte_eth_tx_queue_data_get(), you'll probably need:
eth_rx_direct_rearm_t rte_eth_get_rx_direct_rearm_func(rx_port, rx_queue, tx_port, tx_queue);
Then, it will be user responsibility to store it somewhere and call

periodically:

Quoted 13 lines
Quoted 13 lines

control_path:

	...
	rearm_func = rte_eth_get_rx_direct_rearm_func(rxport, rxqueue,
		 txport, txqueue);
data-path:
	while(...) {
		rearm_func(rxport, txport, rxqueue, txqueue);
		rte_eth_rx_burst(rxport, rxqueue, ....);
		rte_eth_tx_burst(txport, txqueue, ....);
	}
Quoted 5 lines
Quoted 3 lines

In that case there seems absolutely no point to introduce struct rte_eth_txq_data. rx_direct_rearm() accesses TXQ private data directly anyway.

2. This is a very good proposal and it will be our first choice. Before working on it, I have a few questions about how to implement

'rearm_func'.

Quoted 13 lines

Like you say above, mixed Rx and Tx path code in 'rearm_func' means the hard-code is mixed like:

rearm_func(...) {
       ...
      txep = &txq->sw_ring[txq->tx_next_dd - (txq->tx_rs_thresh - 1)];
      for (...) {
         rxep[i].mbuf = txep[i].mbuf;
         mb0 = txep[i].mbuf;
         paddr = mb0->buf_iova + RTE_PKTMBUF_HEADROOM;
        dma_addr0 = vdupq_n_u64(paddr);
        vst1q_u64((uint64_t *)&rxdp++->read, dma_addr0);
      }
}
Quoted 1 line

Is my understanding is right?

Quoted 2 lines

Sorry, I don't understand the question.
Can you probably elaborate a bit?

Sorry for my unclear expression.

Quoted 8 lines

I mean if we need two func which contains tx and rx paths code respectively in rearm_func, like:

rearm_func(...) {
          rte_tx_fill_sw_ring;
          rte_rx_rearm_descs;
}

Or just mixed tx and rx path code like I said before. I prefer 'rx and tx hard code mixed', because from the performance perspective, this can reduce the cost of function calls.

I suppose it depends on what we choose:
If we restrict it in a way that rxq and txq have to belong to the same PMD, then I suppose this decision could be left to each particular PMD.
If we'd like to allow rearm to work accross different PMDs (i.e. it would allow to rearm mlx with ice and visa-versa), then yes we need PMDs somehow to expose rx_sw_ring abstraction to each other.
My preference would be the second one - as it will make this feature more flexible and would help to adopt it more widely. Though the first one is probably easier to implement,
and as I udnerstand you are leaning towards the first one.

Konstantin

Quoted 12 lines
Quoted 11 lines
Quoted 9 lines
Quoted 9 lines

Another way - make rte_eth_txq_data totally opaque and allow PMD to store there some data that will help it to distinguish expected TXQ format. That will allow PMD to keep rx_direct_rearm() the same for all supported TXQ formats (it will make decision internally based on data stored in txq_data).
Though in that case you'll probably need one more dev-op to free

txq_data.

Quoted 1 line
#7 lore raw

回复: 回复: 回复: [PATCH v2 1/3] ethdev: add API for direct rearm mode

Quoted 418 lines

-----邮件原件-----
发件人: Konstantin Ananyev <[email protected]> 发送时间: Thursday, October 13, 2022 5:49 PM
收件人: Feifei Wang <[email protected]>; [email protected]; Ferruh Yigit <[email protected]>; Andrew Rybchenko <[email protected]>; Ray Kinsella <[email protected]> 抄送: [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; Ruifeng Wang <[email protected]>
主题: Re: 回复: 回复: [PATCH v2 1/3] ethdev: add API for direct rearm mode

12/10/2022 14:38, Feifei Wang пишет:

Quoted 6 lines
Quoted 6 lines

-----邮件原件-----
发件人: Konstantin Ananyev <[email protected]> 发送时间: Wednesday, October 12, 2022 6:21 AM
收件人: Feifei Wang <[email protected]>; [email protected];

Ferruh

Quoted 4 lines
Quoted 4 lines

Yigit <[email protected]>; Andrew Rybchenko <[email protected]>; Ray Kinsella <[email protected]> 抄送: [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; Ruifeng Wang

<[email protected]>

Quoted 11 lines
Quoted 11 lines

主题: Re: 回复: [PATCH v2 1/3] ethdev: add API for direct rearm mode

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Add API for enabling direct rearm mode and for mapping RX and TX queues. Currently, the API supports 1:1(txq : rxq) mapping.

Furthermore, to avoid Rx load Tx data directly, add API called 'rte_eth_txq_data_get' to get Tx sw_ring and its information.

Suggested-by Honnappa Nagarahalli

<[email protected]>

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Reviewed-by Honnappa Nagarahalli

<[email protected]>

Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 5 lines
---
    lib/ethdev/ethdev_driver.h   |  9 ++++
    lib/ethdev/ethdev_private.c  |  1 +
    lib/ethdev/rte_ethdev.c      | 37 ++++++++++++++
    lib/ethdev/rte_ethdev.h      | 95

++++++++++++++++++++++++++++++++++++

Quoted 17 lines
    lib/ethdev/rte_ethdev_core.h |  5 ++
    lib/ethdev/version.map       |  4 ++
    6 files changed, 151 insertions(+)
(quoted diff) +0 -0

b/lib/ethdev/ethdev_driver.h index 47a55a419e..14f52907c1 100644

lib/ethdev/ethdev_driver.h +2 -0
@@ -58,6 +58,8 @@ struct rte_eth_dev {
    	eth_rx_descriptor_status_t rx_descriptor_status;
    	/** Check the status of a Tx descriptor */
    	eth_tx_descriptor_status_t tx_descriptor_status;
+	/**  Use Tx mbufs for Rx to rearm */
+	eth_rx_direct_rearm_t rx_direct_rearm;
 
    	/**
    	 * Device data that is shared between primary and secondary
Quoted 1 line

processes @@ -486,6 +488,11 @@ typedef int

(*eth_rx_enable_intr_t)(struct rte_eth_dev *dev,

Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines

lib/ethdev/ethdev_driver.h

Quoted 5 lines

typedef int (*eth_rx_disable_intr_t)(struct rte_eth_dev *dev,
uint16_t rx_queue_id);

lib/ethdev/ethdev_driver.h +2 -0
+/**< @internal Get Tx information of a transmit queue of an
+Ethernet device. */ typedef void (*eth_txq_data_get_t)(struct

rte_eth_dev *dev,

Quoted 2 lines
Quoted 2 lines

lib/ethdev/ethdev_driver.h

Quoted 2 lines
lib/ethdev/ethdev_driver.h +2 -0
+				      uint16_t tx_queue_id,
+				      struct rte_eth_txq_data

*txq_data);

Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

+

    /** @internal Release memory resources allocated by given

Rx/Tx

queue.

Quoted 1 line
Quoted 1 line

*/

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1150

Quoted 9 lines

typedef void (*eth_queue_release_t)(struct rte_eth_dev *dev,
uint16_t queue_id);

lib/ethdev/ethdev_driver.h +2 -0
@@ -1138,6 +1145,8 @@ struct eth_dev_ops {
    	eth_rxq_info_get_t         rxq_info_get;
    	/** Retrieve Tx queue information */
    	eth_txq_info_get_t         txq_info_get;
+	/** Get the address where Tx data is stored */
+	eth_txq_data_get_t         txq_data_get;
    	eth_burst_mode_get_t       rx_burst_mode_get; /**< Get Rx

burst

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

mode */

Quoted 1 line
lib/ethdev/ethdev_driver.h +0 -0
    	eth_burst_mode_get_t       tx_burst_mode_get; /**< Get Tx

burst

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

mode */

Quoted 1 line
lib/ethdev/ethdev_driver.h +0 -0
    	eth_fw_version_get_t       fw_version_get; /**< Get

firmware

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

version */

lib/ethdev/ethdev_private.c eth_dev_fp_ops_setup(struct

Quoted 5 lines
(quoted diff) +0 -0

b/lib/ethdev/ethdev_private.c index 48090c879a..bfe16c7d77 100644

lib/ethdev/ethdev_private.c +0 -0
@@ -276,6 +276,7 @@ eth_dev_fp_ops_setup(struct

rte_eth_fp_ops

Quoted 27 lines
Quoted 27 lines
Quoted 27 lines
Quoted 27 lines

*fpo,

Quoted 15 lines
lib/ethdev/ethdev_private.c +1 -0
    	fpo->rx_queue_count = dev->rx_queue_count;
    	fpo->rx_descriptor_status = dev->rx_descriptor_status;
    	fpo->tx_descriptor_status = dev->tx_descriptor_status;
+	fpo->rx_direct_rearm = dev->rx_direct_rearm;
 
    	fpo->rxq.data = dev->data->rx_queues;
    	fpo->rxq.clbk = (void **)(uintptr_t)dev->post_rx_burst_cbs;
lib/ethdev/rte_ethdev.c +0 -0
index 0c2c1088c0..0dccec2e4b 100644
@@ -1648,6 +1648,43 @@ rte_eth_dev_is_removed(uint16_t port_id)
    	return ret;
    }
 

lib/ethdev/rte_ethdev.c rte_eth_dev_is_removed(uint16_t port_id) · line 1660

Quoted 10 lines
lib/ethdev/rte_ethdev.c +10 -0
+int
+rte_eth_tx_queue_data_get(uint16_t port_id, uint16_t queue_id,
+			struct rte_eth_txq_data *txq_data) {
+	struct rte_eth_dev *dev;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+	dev = &rte_eth_devices[port_id];
+
+	if (queue_id >= dev->data->nb_tx_queues) {
+		RTE_ETHDEV_LOG(ERR, "Invalid Tx queue_id=%u\n",

queue_id);

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/ethdev/rte_ethdev.c rte_eth_dev_is_removed(uint16_t port_id)

Quoted 5 lines
lib/ethdev/rte_ethdev.c +5 -0
+		return -EINVAL;
+	}
+
+	if (txq_data == NULL) {
+		RTE_ETHDEV_LOG(ERR, "Cannot get ethdev port %u

Tx

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

queue %u data to NULL\n",

lib/ethdev/rte_ethdev.c rte_eth_dev_is_removed(uint16_t port_id)

Quoted 8 lines
lib/ethdev/rte_ethdev.c +8 -0
+			port_id, queue_id);
+		return -EINVAL;
+	}
+
+	if (dev->data->tx_queues == NULL ||
+			dev->data->tx_queues[queue_id] == NULL) {
+		RTE_ETHDEV_LOG(ERR,
+			   "Tx queue %"PRIu16" of device with

port_id=%"

Quoted 16 lines
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
lib/ethdev/rte_ethdev.c +7 -0
+			   PRIu16" has not been setup\n",
+			   queue_id, port_id);
+		return -EINVAL;
+	}
+
+	if (*dev->dev_ops->txq_data_get == NULL)
+		return -ENOTSUP;
Quoted 8 lines
Quoted 8 lines

lib/ethdev/rte_ethdev.c rte_eth_dev_is_removed(uint16_t port_id)

Quoted 8 lines
lib/ethdev/rte_ethdev.c +6 -0
+
+	dev->dev_ops->txq_data_get(dev, queue_id, txq_data);
+
+	return 0;
+}
+
    static int
    rte_eth_rx_queue_check_split(const struct rte_eth_rxseg_split

*rx_seg,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/ethdev/rte_ethdev.c rte_eth_dev_is_removed(uint16_t port_id) · line 1690

Quoted 1 line
lib/ethdev/rte_ethdev.c +0 -0
    			     uint16_t n_seg, uint32_t *mbp_buf_size,

diff --git

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

lib/ethdev/rte_ethdev.h struct rte_eth_txq_info { · line 1949

Quoted 6 lines

a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h index 2e783536c1..daf7f05d62 100644

lib/ethdev/rte_ethdev.h +0 -0
@@ -1949,6 +1949,23 @@ struct rte_eth_txq_info {
    	uint8_t queue_state;        /**< one of

RTE_ETH_QUEUE_STATE_*. */

Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Quoted 29 lines

lib/ethdev/rte_ethdev.h struct rte_eth_txq_info {

Quoted 7 lines
lib/ethdev/rte_ethdev.h +5 -0
    } __rte_cache_min_aligned;
 
+/**
+ * @internal
+ * Structure used to hold pointers to internal ethdev Tx data.
+ * The main purpose is to load and store Tx queue data in direct
+rearm

mode.

lib/ethdev/rte_ethdev.h struct rte_eth_txq_info { · line 1967

Quoted 12 lines
lib/ethdev/rte_ethdev.h +12 -0
+ */
+struct rte_eth_txq_data {
+	uint64_t *offloads;
+	void *tx_sw_ring;
+	volatile void *tx_ring;
+	uint16_t *tx_next_dd;
+	uint16_t *nb_tx_free;
+	uint16_t nb_tx_desc;
+	uint16_t tx_rs_thresh;
+	uint16_t tx_free_thresh;
+} __rte_cache_min_aligned;
+

first of all it is not clear why this struct has to be in public header, why it can't be in on of ethdev 'private' headers. Second it looks like a snippet from private txq fields for some Intel (and alike) PMDs (i40e, ice, etc.).
How it supposed to to be universal and be applicable for any PMD that decides to implement this new API?

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

lib/ethdev/rte_ethdev.h int rte_eth_remove_rx_callback(uint16_t

Quoted 4 lines
lib/ethdev/rte_ethdev.h +0 -0
    /* Generic Burst mode flag definition, values can be ORed. */
 
    /**
@@ -4718,6 +4735,27 @@ int rte_eth_remove_rx_callback(uint16_t

port_id, uint16_t queue_id,

Quoted 1 line
lib/ethdev/rte_ethdev.h +0 -0
    int rte_eth_remove_tx_callback(uint16_t port_id, uint16_t

queue_id,

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
lib/ethdev/rte_ethdev.h +11 -0
    		const struct rte_eth_rxtx_callback *user_cb);
 
+/**
+ * Get the address which Tx data is stored.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param queue_id
+ *   The Tx queue on the Ethernet device for which information
+ *   will be retrieved.
+ * @param txq_data
+ *   A pointer to a structure of type *rte_eth_txq_data* to be filled.
+ *
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/ethdev/rte_ethdev.h int rte_eth_remove_rx_callback(uint16_t port_id, uint16_t queue_id,

Quoted 8 lines
lib/ethdev/rte_ethdev.h +8 -0
+ * @return
+ *   - 0: Success
+ *   - -ENODEV:  If *port_id* is invalid.
+ *   - -ENOTSUP: routine is not supported by the device PMD.
+ *   - -EINVAL:  The queue_id is out of range.
+ */
+__rte_experimental
+int rte_eth_tx_queue_data_get(uint16_t port_id, uint16_t

queue_id,

Quoted 33 lines
Quoted 33 lines
Quoted 33 lines
Quoted 33 lines
Quoted 7 lines
lib/ethdev/rte_ethdev.h +2 -0
+		struct rte_eth_txq_data *txq_data);
+
    /**
     * Retrieve information about given port's Rx queue.
     *
@@ -6209,6 +6247,63 @@ rte_eth_tx_buffer(uint16_t port_id,

uint16_t

queue_id,

Quoted 5 lines
    	return rte_eth_tx_buffer_flush(port_id, queue_id, buffer);
    }
lib/ethdev/rte_ethdev.h +2 -0
+/**
+ * @warning

lib/ethdev/rte_ethdev.h rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id,

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+ * @b EXPERIMENTAL: this API may change, or be removed, without
+prior notice
+ *
+ * Put Tx buffers into Rx sw-ring and rearm descs.
+ *
+ * @param port_id
+ *   Port identifying the receive side.
+ * @param queue_id
+ *   The index of the transmit queue identifying the receive side.
+ *   The value must be in the range [0, nb_rx_queue - 1] previously

supplied

Quoted 9 lines

+ * to rte_eth_dev_configure().
+ * @param txq_data
+ * A pointer to a structure of type *rte_eth_txq_data* to be filled.
+ * @return
+ * The number of direct-rearmed buffers.
+ */

lib/ethdev/rte_ethdev.h +3 -0
+__rte_experimental
+static __rte_always_inline uint16_t
+rte_eth_rx_direct_rearm(uint16_t port_id, uint16_t queue_id,
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/ethdev/rte_ethdev.h rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id,

Quoted 8 lines
lib/ethdev/rte_ethdev.h +8 -0
+		struct rte_eth_txq_data *txq_data) {
+	uint16_t nb_rearm;
+	struct rte_eth_fp_ops *p;
+	void *qd;
+
+#ifdef RTE_ETHDEV_DEBUG_RX
+	if (port_id >= RTE_MAX_ETHPORTS ||
+			queue_id >= RTE_MAX_QUEUES_PER_PORT)

{

Quoted 32 lines
Quoted 32 lines
Quoted 16 lines
Quoted 16 lines
Quoted 5 lines
lib/ethdev/rte_ethdev.h +5 -0
+		RTE_ETHDEV_LOG(ERR,
+			"Invalid port_id=%u or queue_id=%u\n",
+			port_id, queue_id);
+		return 0;
+	}

lib/ethdev/rte_ethdev.h rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id,

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+#endif
+
+	p = &rte_eth_fp_ops[port_id];
+	qd = p->rxq.data[queue_id];
+
+#ifdef RTE_ETHDEV_DEBUG_RX
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, 0);
+
+	if (qd == NULL) {
+		RTE_ETHDEV_LOG(ERR, "Invalid Rx queue_id=%u for

port_id=%u\n",

Quoted 9 lines
Quoted 9 lines

lib/ethdev/rte_ethdev.h rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id, · line 6298

Quoted 5 lines
lib/ethdev/rte_ethdev.h +5 -0
+			queue_id, port_id);
+		return 0;
+	}
+
+	if (!p->rx_direct_rearm)

This check should be done always (unconditionally).
it is not a mandatory function for the driver (it can safely skip to

implement it).

Quoted 6 lines
Quoted 6 lines

lib/ethdev/rte_ethdev.h

Quoted 2 lines
lib/ethdev/rte_ethdev.h +1 -0
+		return -ENOTSUP;

This function returns uint16_t, why signed integers here?

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/ethdev/rte_ethdev.h rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id, · line 6302

Quoted 3 lines
lib/ethdev/rte_ethdev.h +3 -0
+#endif
+
+	nb_rearm = p->rx_direct_rearm(qd, txq_data);

So rx_direct_rearm() function knows how to extract data from TX

queue?

Quoted 80 lines
Quoted 29 lines
Quoted 13 lines
Quoted 5 lines

As I understand that is possible only in one case:
rx_direct_rearm() has full knowledge and acess of txq internals, etc. That means that rxq and txq have to belong to the same driver and device type.

Thanks for the comments, and I have some questions for this.

Quoted 6 lines

First of all, I still think it is not the best design choice. If we going ahead with introducing this feature, it better be as generic as possible.
Plus it mixes TX and RX code-paths together, while it would be much better to to keep them independent as they are right now.

Quoted 7 lines
Quoted 3 lines

Another thing with such approach - even for the same PMD, both TXQ and RXQ can have different internal data format and behavior logic (depending on port/queue configuration).

1. Here TXQ and RXQ have different internal format means the queue type and descs can be different, right? If I understand correctly, based on your first strategy, is it means we will need different 'rearm_func' for different queue type in the same PMD?

Yes, I think so.
If let say we have some PMD where depending on the config, there cpuld be
2 different RXQ formats: rxq_a and rxq_b, and 2 different txq
formats: txq_c, txq_d.
Then assuming PMD would like to support direct-rearm mode for all four combinations, it needs 4 different rearm functions:

Quoted 5 lines

rearm_txq_c_to_rxq_a()
rearm_txq_c_to_rxq_b()
rearm_txq_d_to_rxq_a()
rearm_txq_d_to_rxq_b()

Thank you for your detailed explanation, I can understand this.

Quoted 44 lines
Quoted 8 lines
Quoted 8 lines

So rx_direct_rearm() function selection have to be done based on both RXQ and TXQ config.
So instead of rte_eth_tx_queue_data_get(), you'll probably need:
eth_rx_direct_rearm_t rte_eth_get_rx_direct_rearm_func(rx_port, rx_queue, tx_port, tx_queue);
Then, it will be user responsibility to store it somewhere and call

periodically:

Quoted 13 lines
Quoted 13 lines

control_path:

	...
	rearm_func = rte_eth_get_rx_direct_rearm_func(rxport, rxqueue,
		 txport, txqueue);
data-path:
	while(...) {
		rearm_func(rxport, txport, rxqueue, txqueue);
		rte_eth_rx_burst(rxport, rxqueue, ....);
		rte_eth_tx_burst(txport, txqueue, ....);
	}
Quoted 5 lines
Quoted 3 lines

In that case there seems absolutely no point to introduce struct rte_eth_txq_data. rx_direct_rearm() accesses TXQ private data directly anyway.

2. This is a very good proposal and it will be our first choice. Before working on it, I have a few questions about how to implement

'rearm_func'.

Quoted 14 lines

Like you say above, mixed Rx and Tx path code in 'rearm_func' means the hard-code is mixed like:

rearm_func(...) {
       ...
      txep = &txq->sw_ring[txq->tx_next_dd - (txq->tx_rs_thresh - 1)];
      for (...) {
         rxep[i].mbuf = txep[i].mbuf;
         mb0 = txep[i].mbuf;
         paddr = mb0->buf_iova + RTE_PKTMBUF_HEADROOM;
        dma_addr0 = vdupq_n_u64(paddr);
        vst1q_u64((uint64_t *)&rxdp++->read, dma_addr0);
      }
}

Is my understanding is right?

Quoted 6 lines
Quoted 2 lines

Sorry, I don't understand the question.
Can you probably elaborate a bit?

Sorry for my unclear expression.

I mean if we need two func which contains tx and rx paths code

respectively in rearm_func, like:

Quoted 7 lines
rearm_func(...) {
          rte_tx_fill_sw_ring;
          rte_rx_rearm_descs;
}

Or just mixed tx and rx path code like I said before. I prefer 'rx and tx hard code mixed', because from the performance perspective, this can

reduce the cost of function calls.

I suppose it depends on what we choose:
If we restrict it in a way that rxq and txq have to belong to the same PMD, then I suppose this decision could be left to each particular PMD. If we'd like to allow rearm to work accross different PMDs (i.e. it would allow to rearm mlx with ice and visa-versa), then yes we need PMDs somehow to expose rx_sw_ring abstraction to each other.
My preference would be the second one - as it will make this feature more flexible and would help to adopt it more widely.
Though the first one is probably easier to implement, and as I udnerstand you are leaning towards the first one.

Quoted 1 line

Konstantin

Hi, Konstantin

1. After further consideration, I think we should give up 'rte_eth_get_rx_direct_rearm' functions. And just keep one API 'rte_eth_direct_rearm' which contains different queue types path in the pmd driver.

This is due to that application can dynamically configure queue-mapping in the data path,
and thus expand direct-rearm usage scenarios based on this way. For example, consider a flow which received by one rx_queue(rxq_1) and sent by two different types tx_queues( type_a txq_1 and type_b txq_2), Users can dynamically map direct_rearm according to tx path.
--------------------------------------------------------------------------------------
rte_eth_rx_burst(rxq_1);

%routing table lookup to decide tx queue%
txq_n = txq_1 or txq_2?

rte_eth_tx_burst(txq_n);
rte_eth_direct_rearm(txq_n);
--------------------------------------------------------------------------------------
Thus, this can avoid repeatedly calling 'rte_eth_get_rx_direct_rearm' in the data path to reduce performance.

Furthermore, rte_eth_direct_rearm can be implemented as:
--------------------------------------------------------------------------------------
rte_eth_direct_rearm = i40e_eth_direct_rearm;

i40e_eth_direct_rearm {
	rearm_queue_config_a path;
	rearm_queue_config_b path;
	rearm_queue_config_c path;
	rearm_queue_config_d path;
}

--------------------------------------------------------------------------------------
This implementation refers to 'rte_eth_rx_burst' and 'rte_eth_tx_burst'. If there will be different queue types, the implementation will be in pmd layer.

2. I'm not sure whether cross pmd usage is realistic.
Consider the tradeoff between performance and different pmd map, maybe we can provide two different direct rearm mode for users. One is mixed Rx and Tx path and is used for the same pmd. The other is separate Rx and Tx path, and is used for different pmd.

Best Regards
Feifei

Quoted 16 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Another way - make rte_eth_txq_data totally opaque and allow PMD

to

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

store there some data that will help it to distinguish expected TXQ

format.

Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines

That will allow PMD to keep rx_direct_rearm() the same for all supported TXQ formats (it will make decision internally based on data stored in txq_data).
Though in that case you'll probably need one more dev-op to free

txq_data.

Quoted 1 line
#8 lore raw

Re: 回复: 回复: 回复: [PATCH v2 1/3] ethdev: add API for direct rearm mode

Hi Feifei,

Quoted 392 lines
Quoted 388 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Add API for enabling direct rearm mode and for mapping RX and TX queues. Currently, the API supports 1:1(txq : rxq) mapping.

Furthermore, to avoid Rx load Tx data directly, add API called 'rte_eth_txq_data_get' to get Tx sw_ring and its information.

Suggested-by Honnappa Nagarahalli

<[email protected]>

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Reviewed-by Honnappa Nagarahalli

<[email protected]>

Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 5 lines
---
     lib/ethdev/ethdev_driver.h   |  9 ++++
     lib/ethdev/ethdev_private.c  |  1 +
     lib/ethdev/rte_ethdev.c      | 37 ++++++++++++++
     lib/ethdev/rte_ethdev.h      | 95

++++++++++++++++++++++++++++++++++++

Quoted 17 lines
     lib/ethdev/rte_ethdev_core.h |  5 ++
     lib/ethdev/version.map       |  4 ++
     6 files changed, 151 insertions(+)
(quoted diff) +0 -0

b/lib/ethdev/ethdev_driver.h index 47a55a419e..14f52907c1 100644

lib/ethdev/ethdev_driver.h +2 -0
@@ -58,6 +58,8 @@ struct rte_eth_dev {
     	eth_rx_descriptor_status_t rx_descriptor_status;
     	/** Check the status of a Tx descriptor */
     	eth_tx_descriptor_status_t tx_descriptor_status;
+	/**  Use Tx mbufs for Rx to rearm */
+	eth_rx_direct_rearm_t rx_direct_rearm;
 
     	/**
     	 * Device data that is shared between primary and secondary
Quoted 1 line

processes @@ -486,6 +488,11 @@ typedef int

(*eth_rx_enable_intr_t)(struct rte_eth_dev *dev,

Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines

lib/ethdev/ethdev_driver.h

Quoted 5 lines

typedef int (*eth_rx_disable_intr_t)(struct rte_eth_dev *dev,
uint16_t rx_queue_id);

lib/ethdev/ethdev_driver.h +2 -0
+/**< @internal Get Tx information of a transmit queue of an
+Ethernet device. */ typedef void (*eth_txq_data_get_t)(struct

rte_eth_dev *dev,

Quoted 2 lines
Quoted 2 lines

lib/ethdev/ethdev_driver.h

Quoted 2 lines
lib/ethdev/ethdev_driver.h +2 -0
+				      uint16_t tx_queue_id,
+				      struct rte_eth_txq_data

*txq_data);

Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

+

     /** @internal Release memory resources allocated by given

Rx/Tx

queue.

Quoted 1 line
Quoted 1 line

*/

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1150

Quoted 9 lines

typedef void (*eth_queue_release_t)(struct rte_eth_dev *dev,
uint16_t queue_id);

lib/ethdev/ethdev_driver.h +2 -0
@@ -1138,6 +1145,8 @@ struct eth_dev_ops {
     	eth_rxq_info_get_t         rxq_info_get;
     	/** Retrieve Tx queue information */
     	eth_txq_info_get_t         txq_info_get;
+	/** Get the address where Tx data is stored */
+	eth_txq_data_get_t         txq_data_get;
     	eth_burst_mode_get_t       rx_burst_mode_get; /**< Get Rx

burst

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

mode */

Quoted 1 line
lib/ethdev/ethdev_driver.h +0 -0
     	eth_burst_mode_get_t       tx_burst_mode_get; /**< Get Tx

burst

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

mode */

Quoted 1 line
lib/ethdev/ethdev_driver.h +0 -0
     	eth_fw_version_get_t       fw_version_get; /**< Get

firmware

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

version */

lib/ethdev/ethdev_private.c eth_dev_fp_ops_setup(struct

Quoted 5 lines
(quoted diff) +0 -0

b/lib/ethdev/ethdev_private.c index 48090c879a..bfe16c7d77 100644

lib/ethdev/ethdev_private.c +0 -0
@@ -276,6 +276,7 @@ eth_dev_fp_ops_setup(struct

rte_eth_fp_ops

Quoted 27 lines
Quoted 27 lines
Quoted 27 lines
Quoted 27 lines

*fpo,

Quoted 15 lines
lib/ethdev/ethdev_private.c +1 -0
     	fpo->rx_queue_count = dev->rx_queue_count;
     	fpo->rx_descriptor_status = dev->rx_descriptor_status;
     	fpo->tx_descriptor_status = dev->tx_descriptor_status;
+	fpo->rx_direct_rearm = dev->rx_direct_rearm;
 
     	fpo->rxq.data = dev->data->rx_queues;
     	fpo->rxq.clbk = (void **)(uintptr_t)dev->post_rx_burst_cbs;
lib/ethdev/rte_ethdev.c +0 -0
index 0c2c1088c0..0dccec2e4b 100644
@@ -1648,6 +1648,43 @@ rte_eth_dev_is_removed(uint16_t port_id)
     	return ret;
     }
 

lib/ethdev/rte_ethdev.c rte_eth_dev_is_removed(uint16_t port_id) · line 1660

Quoted 10 lines
lib/ethdev/rte_ethdev.c +10 -0
+int
+rte_eth_tx_queue_data_get(uint16_t port_id, uint16_t queue_id,
+			struct rte_eth_txq_data *txq_data) {
+	struct rte_eth_dev *dev;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+	dev = &rte_eth_devices[port_id];
+
+	if (queue_id >= dev->data->nb_tx_queues) {
+		RTE_ETHDEV_LOG(ERR, "Invalid Tx queue_id=%u\n",

queue_id);

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/ethdev/rte_ethdev.c rte_eth_dev_is_removed(uint16_t port_id) · line 1665

Quoted 5 lines
lib/ethdev/rte_ethdev.c +5 -0
+		return -EINVAL;
+	}
+
+	if (txq_data == NULL) {
+		RTE_ETHDEV_LOG(ERR, "Cannot get ethdev port %u

Tx

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

queue %u data to NULL\n",

lib/ethdev/rte_ethdev.c rte_eth_dev_is_removed(uint16_t port_id) · line 1673

Quoted 8 lines
lib/ethdev/rte_ethdev.c +8 -0
+			port_id, queue_id);
+		return -EINVAL;
+	}
+
+	if (dev->data->tx_queues == NULL ||
+			dev->data->tx_queues[queue_id] == NULL) {
+		RTE_ETHDEV_LOG(ERR,
+			   "Tx queue %"PRIu16" of device with

port_id=%"

Quoted 16 lines
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
lib/ethdev/rte_ethdev.c +7 -0
+			   PRIu16" has not been setup\n",
+			   queue_id, port_id);
+		return -EINVAL;
+	}
+
+	if (*dev->dev_ops->txq_data_get == NULL)
+		return -ENOTSUP;
Quoted 8 lines
Quoted 8 lines

lib/ethdev/rte_ethdev.c rte_eth_dev_is_removed(uint16_t port_id) · line 1688

Quoted 8 lines
lib/ethdev/rte_ethdev.c +6 -0
+
+	dev->dev_ops->txq_data_get(dev, queue_id, txq_data);
+
+	return 0;
+}
+
     static int
     rte_eth_rx_queue_check_split(const struct rte_eth_rxseg_split

*rx_seg,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/ethdev/rte_ethdev.c rte_eth_dev_is_removed(uint16_t port_id) · line 1689

Quoted 1 line
lib/ethdev/rte_ethdev.c +0 -0
     			     uint16_t n_seg, uint32_t *mbp_buf_size,

diff --git

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

lib/ethdev/rte_ethdev.h struct rte_eth_txq_info { · line 1949

Quoted 6 lines

a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h index 2e783536c1..daf7f05d62 100644

lib/ethdev/rte_ethdev.h +0 -0
@@ -1949,6 +1949,23 @@ struct rte_eth_txq_info {
     	uint8_t queue_state;        /**< one of

RTE_ETH_QUEUE_STATE_*. */

Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Quoted 29 lines

lib/ethdev/rte_ethdev.h struct rte_eth_txq_info { · line 1956

Quoted 7 lines
lib/ethdev/rte_ethdev.h +5 -0
     } __rte_cache_min_aligned;
 
+/**
+ * @internal
+ * Structure used to hold pointers to internal ethdev Tx data.
+ * The main purpose is to load and store Tx queue data in direct
+rearm

mode.

lib/ethdev/rte_ethdev.h struct rte_eth_txq_info { · line 1968

Quoted 12 lines
lib/ethdev/rte_ethdev.h +12 -0
+ */
+struct rte_eth_txq_data {
+	uint64_t *offloads;
+	void *tx_sw_ring;
+	volatile void *tx_ring;
+	uint16_t *tx_next_dd;
+	uint16_t *nb_tx_free;
+	uint16_t nb_tx_desc;
+	uint16_t tx_rs_thresh;
+	uint16_t tx_free_thresh;
+} __rte_cache_min_aligned;
+

first of all it is not clear why this struct has to be in public header, why it can't be in on of ethdev 'private' headers. Second it looks like a snippet from private txq fields for some Intel (and alike) PMDs (i40e, ice, etc.).
How it supposed to to be universal and be applicable for any PMD that decides to implement this new API?

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

lib/ethdev/rte_ethdev.h int rte_eth_remove_rx_callback(uint16_t

Quoted 4 lines
lib/ethdev/rte_ethdev.h +0 -0
     /* Generic Burst mode flag definition, values can be ORed. */
 
     /**
@@ -4718,6 +4735,27 @@ int rte_eth_remove_rx_callback(uint16_t

port_id, uint16_t queue_id,

Quoted 1 line
lib/ethdev/rte_ethdev.h +0 -0
     int rte_eth_remove_tx_callback(uint16_t port_id, uint16_t

queue_id,

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
lib/ethdev/rte_ethdev.h +11 -0
     		const struct rte_eth_rxtx_callback *user_cb);
 
+/**
+ * Get the address which Tx data is stored.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param queue_id
+ *   The Tx queue on the Ethernet device for which information
+ *   will be retrieved.
+ * @param txq_data
+ *   A pointer to a structure of type *rte_eth_txq_data* to be filled.
+ *
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/ethdev/rte_ethdev.h int rte_eth_remove_rx_callback(uint16_t · line 4756

Quoted 8 lines
lib/ethdev/rte_ethdev.h +8 -0
+ * @return
+ *   - 0: Success
+ *   - -ENODEV:  If *port_id* is invalid.
+ *   - -ENOTSUP: routine is not supported by the device PMD.
+ *   - -EINVAL:  The queue_id is out of range.
+ */
+__rte_experimental
+int rte_eth_tx_queue_data_get(uint16_t port_id, uint16_t

queue_id,

Quoted 33 lines
Quoted 33 lines
Quoted 33 lines
Quoted 33 lines
Quoted 7 lines
lib/ethdev/rte_ethdev.h +2 -0
+		struct rte_eth_txq_data *txq_data);
+
     /**
      * Retrieve information about given port's Rx queue.
      *
@@ -6209,6 +6247,63 @@ rte_eth_tx_buffer(uint16_t port_id,

uint16_t

queue_id,

Quoted 5 lines
     	return rte_eth_tx_buffer_flush(port_id, queue_id, buffer);
     }
lib/ethdev/rte_ethdev.h +2 -0
+/**
+ * @warning

lib/ethdev/rte_ethdev.h

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+ * @b EXPERIMENTAL: this API may change, or be removed, without
+prior notice
+ *
+ * Put Tx buffers into Rx sw-ring and rearm descs.
+ *
+ * @param port_id
+ *   Port identifying the receive side.
+ * @param queue_id
+ *   The index of the transmit queue identifying the receive side.
+ *   The value must be in the range [0, nb_rx_queue - 1] previously

supplied

Quoted 9 lines

+ * to rte_eth_dev_configure().
+ * @param txq_data
+ * A pointer to a structure of type *rte_eth_txq_data* to be filled.
+ * @return
+ * The number of direct-rearmed buffers.
+ */

lib/ethdev/rte_ethdev.h +3 -0
+__rte_experimental
+static __rte_always_inline uint16_t
+rte_eth_rx_direct_rearm(uint16_t port_id, uint16_t queue_id,
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/ethdev/rte_ethdev.h

Quoted 8 lines
lib/ethdev/rte_ethdev.h +8 -0
+		struct rte_eth_txq_data *txq_data) {
+	uint16_t nb_rearm;
+	struct rte_eth_fp_ops *p;
+	void *qd;
+
+#ifdef RTE_ETHDEV_DEBUG_RX
+	if (port_id >= RTE_MAX_ETHPORTS ||
+			queue_id >= RTE_MAX_QUEUES_PER_PORT)

{

Quoted 32 lines
Quoted 32 lines
Quoted 16 lines
Quoted 16 lines
Quoted 5 lines
lib/ethdev/rte_ethdev.h +5 -0
+		RTE_ETHDEV_LOG(ERR,
+			"Invalid port_id=%u or queue_id=%u\n",
+			port_id, queue_id);
+		return 0;
+	}

lib/ethdev/rte_ethdev.h

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+#endif
+
+	p = &rte_eth_fp_ops[port_id];
+	qd = p->rxq.data[queue_id];
+
+#ifdef RTE_ETHDEV_DEBUG_RX
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, 0);
+
+	if (qd == NULL) {
+		RTE_ETHDEV_LOG(ERR, "Invalid Rx queue_id=%u for

port_id=%u\n",

Quoted 9 lines
Quoted 9 lines

lib/ethdev/rte_ethdev.h

Quoted 5 lines
lib/ethdev/rte_ethdev.h +5 -0
+			queue_id, port_id);
+		return 0;
+	}
+
+	if (!p->rx_direct_rearm)

This check should be done always (unconditionally).
it is not a mandatory function for the driver (it can safely skip to

implement it).

Quoted 6 lines
Quoted 6 lines

lib/ethdev/rte_ethdev.h

Quoted 2 lines
lib/ethdev/rte_ethdev.h +1 -0
+		return -ENOTSUP;

This function returns uint16_t, why signed integers here?

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/ethdev/rte_ethdev.h

Quoted 3 lines
lib/ethdev/rte_ethdev.h +3 -0
+#endif
+
+	nb_rearm = p->rx_direct_rearm(qd, txq_data);

So rx_direct_rearm() function knows how to extract data from TX

queue?

Quoted 80 lines
Quoted 29 lines
Quoted 13 lines
Quoted 5 lines

As I understand that is possible only in one case:
rx_direct_rearm() has full knowledge and acess of txq internals, etc. That means that rxq and txq have to belong to the same driver and device type.

Thanks for the comments, and I have some questions for this.

Quoted 6 lines

First of all, I still think it is not the best design choice. If we going ahead with introducing this feature, it better be as generic as possible.
Plus it mixes TX and RX code-paths together, while it would be much better to to keep them independent as they are right now.

Quoted 7 lines
Quoted 3 lines

Another thing with such approach - even for the same PMD, both TXQ and RXQ can have different internal data format and behavior logic (depending on port/queue configuration).

1. Here TXQ and RXQ have different internal format means the queue type and descs can be different, right? If I understand correctly, based on your first strategy, is it means we will need different 'rearm_func' for different queue type in the same PMD?

Yes, I think so.
If let say we have some PMD where depending on the config, there cpuld be
2 different RXQ formats: rxq_a and rxq_b, and 2 different txq
formats: txq_c, txq_d.
Then assuming PMD would like to support direct-rearm mode for all four combinations, it needs 4 different rearm functions:

Quoted 5 lines

rearm_txq_c_to_rxq_a()
rearm_txq_c_to_rxq_b()
rearm_txq_d_to_rxq_a()
rearm_txq_d_to_rxq_b()

Thank you for your detailed explanation, I can understand this.

Quoted 44 lines
Quoted 8 lines
Quoted 8 lines

So rx_direct_rearm() function selection have to be done based on both RXQ and TXQ config.
So instead of rte_eth_tx_queue_data_get(), you'll probably need:
eth_rx_direct_rearm_t rte_eth_get_rx_direct_rearm_func(rx_port, rx_queue, tx_port, tx_queue);
Then, it will be user responsibility to store it somewhere and call

periodically:

Quoted 13 lines
Quoted 13 lines

control_path:

	...
	rearm_func = rte_eth_get_rx_direct_rearm_func(rxport, rxqueue,
		 txport, txqueue);
data-path:
	while(...) {
		rearm_func(rxport, txport, rxqueue, txqueue);
		rte_eth_rx_burst(rxport, rxqueue, ....);
		rte_eth_tx_burst(txport, txqueue, ....);
	}
Quoted 5 lines
Quoted 3 lines

In that case there seems absolutely no point to introduce struct rte_eth_txq_data. rx_direct_rearm() accesses TXQ private data directly anyway.

2. This is a very good proposal and it will be our first choice. Before working on it, I have a few questions about how to implement

'rearm_func'.

Quoted 14 lines

Like you say above, mixed Rx and Tx path code in 'rearm_func' means the hard-code is mixed like:

rearm_func(...) {
        ...
       txep = &txq->sw_ring[txq->tx_next_dd - (txq->tx_rs_thresh - 1)];
       for (...) {
          rxep[i].mbuf = txep[i].mbuf;
          mb0 = txep[i].mbuf;
          paddr = mb0->buf_iova + RTE_PKTMBUF_HEADROOM;
         dma_addr0 = vdupq_n_u64(paddr);
         vst1q_u64((uint64_t *)&rxdp++->read, dma_addr0);
       }
}

Is my understanding is right?

Quoted 6 lines
Quoted 2 lines

Sorry, I don't understand the question.
Can you probably elaborate a bit?

Sorry for my unclear expression.

I mean if we need two func which contains tx and rx paths code

respectively in rearm_func, like:

Quoted 7 lines
rearm_func(...) {
           rte_tx_fill_sw_ring;
           rte_rx_rearm_descs;
}

Or just mixed tx and rx path code like I said before. I prefer 'rx and tx hard code mixed', because from the performance perspective, this can

reduce the cost of function calls.

I suppose it depends on what we choose:
If we restrict it in a way that rxq and txq have to belong to the same PMD, then I suppose this decision could be left to each particular PMD. If we'd like to allow rearm to work accross different PMDs (i.e. it would allow to rearm mlx with ice and visa-versa), then yes we need PMDs somehow to expose rx_sw_ring abstraction to each other.
My preference would be the second one - as it will make this feature more flexible and would help to adopt it more widely.
Though the first one is probably easier to implement, and as I udnerstand you are leaning towards the first one.

Quoted 1 line

Konstantin

Hi, Konstantin

Quoted 7 lines

1. After further consideration, I think we should give up 'rte_eth_get_rx_direct_rearm' functions. And just keep one API 'rte_eth_direct_rearm' which contains different queue types path in the pmd driver.

This is due to that application can dynamically configure queue-mapping in the data path,
and thus expand direct-rearm usage scenarios based on this way. For example, consider a flow which received by one rx_queue(rxq_1) and sent by two different types tx_queues( type_a txq_1 and type_b txq_2), Users can dynamically map direct_rearm according to tx path.

That's good point, indeed for such case special function pointer doesn't look very conveniet.
I still think that the best way to deal with multiple sources would be
to expose some sort of rx_sw_ring abstraction.
Then it could be obtaiend once and passed as a paramter to actual rearm function.
In that case, user can even re-arm RX queue not from TX queue only,
but from other sources too (freed mbufs, etc.).
As I understand what you suggest is one unified function:
int rte_eth_rx_direct_rearm(rx_port, rx_queue, tx_port, tx_queue), correct?
I am not against it in principle, but does it mean at each invocation,
inside PMD layer, first thing you would have to do -
check that rx_queue and tx_queue are from the same device type and select proper behaviour (for different queue configs)?

Quoted 47 lines

--------------------------------------------------------------------------------------
rte_eth_rx_burst(rxq_1);

%routing table lookup to decide tx queue%
txq_n = txq_1 or txq_2?

rte_eth_tx_burst(txq_n);
rte_eth_direct_rearm(txq_n);
--------------------------------------------------------------------------------------
Thus, this can avoid repeatedly calling 'rte_eth_get_rx_direct_rearm' in the data path to reduce performance.

Furthermore, rte_eth_direct_rearm can be implemented as:
--------------------------------------------------------------------------------------
rte_eth_direct_rearm = i40e_eth_direct_rearm;

i40e_eth_direct_rearm {
	rearm_queue_config_a path;
	rearm_queue_config_b path;
	rearm_queue_config_c path;
	rearm_queue_config_d path;
}

--------------------------------------------------------------------------------------
This implementation refers to 'rte_eth_rx_burst' and 'rte_eth_tx_burst'. If there will be different queue types, the implementation will be in pmd layer.

2. I'm not sure whether cross pmd usage is realistic.
Consider the tradeoff between performance and different pmd map, maybe we can provide two different direct rearm mode for users. One is mixed Rx and Tx path and is used for the same pmd. The other is separate Rx and Tx path, and is used for different pmd.
Best Regards
Feifei

Quoted 16 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Another way - make rte_eth_txq_data totally opaque and allow PMD

to

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

store there some data that will help it to distinguish expected TXQ

format.

Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines

That will allow PMD to keep rx_direct_rearm() the same for all supported TXQ formats (it will make decision internally based on data stored in txq_data).
Though in that case you'll probably need one more dev-op to free

txq_data.

Quoted 1 line
#9 lore raw

回复: 回复: 回复: 回复: [PATCH v2 1/3] ethdev: add API for direct rearm mode

Hi, Konstantin

Quoted 437 lines

-----邮件原件-----
发件人: Konstantin Ananyev <[email protected]> 发送时间: Tuesday, November 1, 2022 12:37 AM
收件人: Feifei Wang <[email protected]>; [email protected]; Ferruh Yigit <[email protected]>; Andrew Rybchenko <[email protected]>; Ray Kinsella <[email protected]> 抄送: [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; Ruifeng Wang <[email protected]>
主题: Re: 回复: 回复: 回复: [PATCH v2 1/3] ethdev: add API for direct rearm mode

Hi Feifei,

Quoted 18 lines
Quoted 18 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Add API for enabling direct rearm mode and for mapping RX and TX queues. Currently, the API supports 1:1(txq : rxq) mapping.

Furthermore, to avoid Rx load Tx data directly, add API called 'rte_eth_txq_data_get' to get Tx sw_ring and its information.

Suggested-by Honnappa Nagarahalli

<[email protected]>

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Reviewed-by Honnappa Nagarahalli

<[email protected]>

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
---
     lib/ethdev/ethdev_driver.h   |  9 ++++
     lib/ethdev/ethdev_private.c  |  1 +
     lib/ethdev/rte_ethdev.c      | 37 ++++++++++++++
     lib/ethdev/rte_ethdev.h      | 95
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

++++++++++++++++++++++++++++++++++++

Quoted 6 lines
     lib/ethdev/rte_ethdev_core.h |  5 ++
     lib/ethdev/version.map       |  4 ++
     6 files changed, 151 insertions(+)
(quoted diff) +0 -0

b/lib/ethdev/ethdev_driver.h index 47a55a419e..14f52907c1

100644

Quoted 22 lines
Quoted 22 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 11 lines
lib/ethdev/ethdev_driver.h +2 -0
@@ -58,6 +58,8 @@ struct rte_eth_dev {
     	eth_rx_descriptor_status_t rx_descriptor_status;
     	/** Check the status of a Tx descriptor */
     	eth_tx_descriptor_status_t tx_descriptor_status;
+	/**  Use Tx mbufs for Rx to rearm */
+	eth_rx_direct_rearm_t rx_direct_rearm;
 
     	/**
     	 * Device data that is shared between primary and
Quoted 1 line

secondary processes @@ -486,6 +488,11 @@ typedef int

(*eth_rx_enable_intr_t)(struct rte_eth_dev *dev,

Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines

lib/ethdev/ethdev_driver.h

Quoted 5 lines

typedef int (*eth_rx_disable_intr_t)(struct rte_eth_dev *dev,
uint16_t rx_queue_id);

lib/ethdev/ethdev_driver.h +2 -0
+/**< @internal Get Tx information of a transmit queue of an
+Ethernet device. */ typedef void (*eth_txq_data_get_t)(struct

rte_eth_dev *dev,

Quoted 2 lines
Quoted 2 lines

lib/ethdev/ethdev_driver.h

Quoted 2 lines
lib/ethdev/ethdev_driver.h +2 -0
+				      uint16_t tx_queue_id,
+				      struct rte_eth_txq_data

*txq_data);

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

+

     /** @internal Release memory resources allocated by given

Rx/Tx

queue.

Quoted 3 lines
Quoted 3 lines

*/

Quoted 2 lines

typedef void (*eth_queue_release_t)(struct rte_eth_dev *dev,
uint16_t queue_id); @@ -1138,6

+1145,8 @@ struct

Quoted 28 lines
Quoted 28 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

lib/ethdev/ethdev_driver.h

Quoted 7 lines
eth_dev_ops {
     	eth_rxq_info_get_t         rxq_info_get;
     	/** Retrieve Tx queue information */
     	eth_txq_info_get_t         txq_info_get;
lib/ethdev/ethdev_driver.h +2 -0
+	/** Get the address where Tx data is stored */
+	eth_txq_data_get_t         txq_data_get;
     	eth_burst_mode_get_t       rx_burst_mode_get; /**< Get Rx

burst

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

mode */

Quoted 1 line

eth_burst_mode_get_t tx_burst_mode_get; /**< Get Tx

burst

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

mode */

Quoted 1 line

eth_fw_version_get_t fw_version_get; /**< Get

firmware

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

version */

lib/ethdev/ethdev_private.c eth_dev_fp_ops_setup(struct

Quoted 6 lines
(quoted diff) +0 -0

b/lib/ethdev/ethdev_private.c index 48090c879a..bfe16c7d77 100644

lib/ethdev/ethdev_private.c +0 -0
@@ -276,6 +276,7 @@ eth_dev_fp_ops_setup(struct

rte_eth_fp_ops

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

*fpo,

Quoted 5 lines
lib/ethdev/ethdev_private.c +1 -0
     	fpo->rx_queue_count = dev->rx_queue_count;
     	fpo->rx_descriptor_status = dev->rx_descriptor_status;
     	fpo->tx_descriptor_status = dev->tx_descriptor_status;
+	fpo->rx_direct_rearm = dev->rx_direct_rearm;
 
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/ethdev/rte_ethdev.c rte_eth_dev_is_removed(uint16_t

Quoted 8 lines
lib/ethdev/ethdev_private.c +0 -0
     	fpo->rxq.data = dev->data->rx_queues;
     	fpo->rxq.clbk = (void

**)(uintptr_t)dev->post_rx_burst_cbs;

lib/ethdev/rte_ethdev.c +0 -0
index 0c2c1088c0..0dccec2e4b 100644
@@ -1648,6 +1648,43 @@ rte_eth_dev_is_removed(uint16_t

port_id)

Quoted 85 lines
Quoted 85 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines

lib/ethdev/rte_ethdev.c rte_eth_dev_is_removed(uint16_t · line 1660

Quoted 13 lines
lib/ethdev/rte_ethdev.c +10 -0
     	return ret;
     }
 
+int
+rte_eth_tx_queue_data_get(uint16_t port_id, uint16_t queue_id,
+			struct rte_eth_txq_data *txq_data) {
+	struct rte_eth_dev *dev;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+	dev = &rte_eth_devices[port_id];
+
+	if (queue_id >= dev->data->nb_tx_queues) {
+		RTE_ETHDEV_LOG(ERR, "Invalid Tx queue_id=%u\n",

queue_id);

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/ethdev/rte_ethdev.c rte_eth_dev_is_removed(uint16_t · line 1665

Quoted 5 lines
lib/ethdev/rte_ethdev.c +5 -0
+		return -EINVAL;
+	}
+
+	if (txq_data == NULL) {
+		RTE_ETHDEV_LOG(ERR, "Cannot get ethdev port %u

Tx

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

queue %u data to NULL\n",

lib/ethdev/rte_ethdev.c rte_eth_dev_is_removed(uint16_t · line 1673

Quoted 8 lines
lib/ethdev/rte_ethdev.c +8 -0
+			port_id, queue_id);
+		return -EINVAL;
+	}
+
+	if (dev->data->tx_queues == NULL ||
+			dev->data->tx_queues[queue_id] == NULL) {
+		RTE_ETHDEV_LOG(ERR,
+			   "Tx queue %"PRIu16" of device with

port_id=%"

Quoted 17 lines
Quoted 17 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
lib/ethdev/rte_ethdev.c +8 -0
+			   PRIu16" has not been setup\n",
+			   queue_id, port_id);
+		return -EINVAL;
+	}
+
+	if (*dev->dev_ops->txq_data_get == NULL)
+		return -ENOTSUP;
+
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
lib/ethdev/rte_ethdev.c +5 -0
+	dev->dev_ops->txq_data_get(dev, queue_id, txq_data);
+
+	return 0;
+}
+
     static int
     rte_eth_rx_queue_check_split(const struct

rte_eth_rxseg_split

*rx_seg,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
     			     uint16_t n_seg, uint32_t *mbp_buf_size,

diff --git

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

lib/ethdev/rte_ethdev.h struct rte_eth_txq_info { · line 1949

Quoted 6 lines

a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h index 2e783536c1..daf7f05d62 100644

lib/ethdev/rte_ethdev.h +0 -0
@@ -1949,6 +1949,23 @@ struct rte_eth_txq_info {
     	uint8_t queue_state;        /**< one of

RTE_ETH_QUEUE_STATE_*. */

Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Quoted 29 lines

lib/ethdev/rte_ethdev.h struct rte_eth_txq_info { · line 1956

Quoted 7 lines
lib/ethdev/rte_ethdev.h +5 -0
     } __rte_cache_min_aligned;
 
+/**
+ * @internal
+ * Structure used to hold pointers to internal ethdev Tx data.
+ * The main purpose is to load and store Tx queue data in
+direct rearm

mode.

lib/ethdev/rte_ethdev.h struct rte_eth_txq_info { · line 1968

Quoted 12 lines
lib/ethdev/rte_ethdev.h +12 -0
+ */
+struct rte_eth_txq_data {
+	uint64_t *offloads;
+	void *tx_sw_ring;
+	volatile void *tx_ring;
+	uint16_t *tx_next_dd;
+	uint16_t *nb_tx_free;
+	uint16_t nb_tx_desc;
+	uint16_t tx_rs_thresh;
+	uint16_t tx_free_thresh;
+} __rte_cache_min_aligned;
+

first of all it is not clear why this struct has to be in public header, why it can't be in on of ethdev 'private' headers. Second it looks like a snippet from private txq fields for some Intel (and alike) PMDs (i40e, ice, etc.).
How it supposed to to be universal and be applicable for any PMD that decides to implement this new API?

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/ethdev/rte_ethdev.h int

Quoted 5 lines
lib/ethdev/rte_ethdev.h +0 -0
     /* Generic Burst mode flag definition, values can be ORed.
 */
 
     /**
lib/ethdev/rte_ethdev.h +0 -0
@@ -4718,6 +4735,27 @@ int

rte_eth_remove_rx_callback(uint16_t

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

port_id, uint16_t queue_id,

Quoted 1 line
lib/ethdev/rte_ethdev.h +0 -0
     int rte_eth_remove_tx_callback(uint16_t port_id, uint16_t

queue_id,

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
lib/ethdev/rte_ethdev.h +0 -0
     		const struct rte_eth_rxtx_callback *user_cb);
 
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

lib/ethdev/rte_ethdev.h int · line 4747

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+/**
+ * Get the address which Tx data is stored.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param queue_id
+ *   The Tx queue on the Ethernet device for which information
+ *   will be retrieved.
+ * @param txq_data
+ *   A pointer to a structure of type *rte_eth_txq_data* to be

filled.

Quoted 18 lines
Quoted 18 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

lib/ethdev/rte_ethdev.h int · line 4756

Quoted 9 lines
lib/ethdev/rte_ethdev.h +9 -0
+ *
+ * @return
+ *   - 0: Success
+ *   - -ENODEV:  If *port_id* is invalid.
+ *   - -ENOTSUP: routine is not supported by the device PMD.
+ *   - -EINVAL:  The queue_id is out of range.
+ */
+__rte_experimental
+int rte_eth_tx_queue_data_get(uint16_t port_id, uint16_t

queue_id,

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
lib/ethdev/rte_ethdev.h +2 -0
+		struct rte_eth_txq_data *txq_data);
+
     /**
      * Retrieve information about given port's Rx queue.
      *
@@ -6209,6 +6247,63 @@ rte_eth_tx_buffer(uint16_t port_id,

uint16_t

queue_id,

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
     	return rte_eth_tx_buffer_flush(port_id, queue_id, buffer);
     }

+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed,

without

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

+prior notice
+ *
+ * Put Tx buffers into Rx sw-ring and rearm descs.
+ *
+ * @param port_id
+ * Port identifying the receive side.
+ * @param queue_id
+ * The index of the transmit queue identifying the receive side.
+ * The value must be in the range [0, nb_rx_queue - 1] previously

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

supplied

Quoted 3 lines

+ * to rte_eth_dev_configure().
+ * @param txq_data
+ * A pointer to a structure of type *rte_eth_txq_data* to be

filled.

Quoted 98 lines
Quoted 98 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

+ * @return
+ * The number of direct-rearmed buffers.
+ */

lib/ethdev/rte_ethdev.h +3 -0
+__rte_experimental
+static __rte_always_inline uint16_t
+rte_eth_rx_direct_rearm(uint16_t port_id, uint16_t queue_id,
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/ethdev/rte_ethdev.h

Quoted 8 lines
lib/ethdev/rte_ethdev.h +8 -0
+		struct rte_eth_txq_data *txq_data) {
+	uint16_t nb_rearm;
+	struct rte_eth_fp_ops *p;
+	void *qd;
+
+#ifdef RTE_ETHDEV_DEBUG_RX
+	if (port_id >= RTE_MAX_ETHPORTS ||
+			queue_id >= RTE_MAX_QUEUES_PER_PORT)

{

Quoted 32 lines
Quoted 32 lines
Quoted 16 lines
Quoted 16 lines
Quoted 5 lines
lib/ethdev/rte_ethdev.h +5 -0
+		RTE_ETHDEV_LOG(ERR,
+			"Invalid port_id=%u or queue_id=%u\n",
+			port_id, queue_id);
+		return 0;
+	}

lib/ethdev/rte_ethdev.h

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+#endif
+
+	p = &rte_eth_fp_ops[port_id];
+	qd = p->rxq.data[queue_id];
+
+#ifdef RTE_ETHDEV_DEBUG_RX
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, 0);
+
+	if (qd == NULL) {
+		RTE_ETHDEV_LOG(ERR, "Invalid Rx queue_id=%u for

port_id=%u\n",

Quoted 9 lines
Quoted 9 lines

lib/ethdev/rte_ethdev.h

Quoted 5 lines
lib/ethdev/rte_ethdev.h +5 -0
+			queue_id, port_id);
+		return 0;
+	}
+
+	if (!p->rx_direct_rearm)

This check should be done always (unconditionally).
it is not a mandatory function for the driver (it can safely skip to

implement it).

Quoted 6 lines
Quoted 6 lines

lib/ethdev/rte_ethdev.h

Quoted 2 lines
lib/ethdev/rte_ethdev.h +1 -0
+		return -ENOTSUP;

This function returns uint16_t, why signed integers here?

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/ethdev/rte_ethdev.h

Quoted 3 lines
lib/ethdev/rte_ethdev.h +3 -0
+#endif
+
+	nb_rearm = p->rx_direct_rearm(qd, txq_data);

So rx_direct_rearm() function knows how to extract data from TX

queue?

Quoted 45 lines
Quoted 29 lines
Quoted 13 lines
Quoted 5 lines

As I understand that is possible only in one case:
rx_direct_rearm() has full knowledge and acess of txq internals, etc. That means that rxq and txq have to belong to the same driver and device type.

Thanks for the comments, and I have some questions for this.

Quoted 6 lines

First of all, I still think it is not the best design choice. If we going ahead with introducing this feature, it better be as generic as possible.
Plus it mixes TX and RX code-paths together, while it would be much better to to keep them independent as they are right now.

Quoted 7 lines
Quoted 3 lines

Another thing with such approach - even for the same PMD, both TXQ and RXQ can have different internal data format and behavior logic (depending on port/queue configuration).

1. Here TXQ and RXQ have different internal format means the queue type and descs can be different, right? If I understand correctly, based on your first strategy, is it means we will need different 'rearm_func' for different queue type in the same PMD?

Yes, I think so.
If let say we have some PMD where depending on the config, there cpuld be
2 different RXQ formats: rxq_a and rxq_b, and 2 different txq
formats: txq_c, txq_d.
Then assuming PMD would like to support direct-rearm mode for all four combinations, it needs 4 different rearm functions:

Quoted 5 lines

rearm_txq_c_to_rxq_a()
rearm_txq_c_to_rxq_b()
rearm_txq_d_to_rxq_a()
rearm_txq_d_to_rxq_b()

Thank you for your detailed explanation, I can understand this.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

So rx_direct_rearm() function selection have to be done based on both RXQ and TXQ config.
So instead of rte_eth_tx_queue_data_get(), you'll probably need:
eth_rx_direct_rearm_t rte_eth_get_rx_direct_rearm_func(rx_port, rx_queue, tx_port, tx_queue);
Then, it will be user responsibility to store it somewhere and call

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

periodically:

Quoted 4 lines
Quoted 4 lines

control_path:

	...
	rearm_func = rte_eth_get_rx_direct_rearm_func(rxport,

rxqueue,

Quoted 44 lines
Quoted 44 lines
Quoted 32 lines
Quoted 32 lines
Quoted 9 lines
Quoted 9 lines
		 txport, txqueue);
data-path:
	while(...) {
		rearm_func(rxport, txport, rxqueue, txqueue);
		rte_eth_rx_burst(rxport, rxqueue, ....);
		rte_eth_tx_burst(txport, txqueue, ....);
	}
Quoted 6 lines
Quoted 3 lines

In that case there seems absolutely no point to introduce struct rte_eth_txq_data. rx_direct_rearm() accesses TXQ private data directly anyway.

2. This is a very good proposal and it will be our first choice. Before working on it, I have a few questions about how to implement

'rearm_func'.

Quoted 14 lines

Like you say above, mixed Rx and Tx path code in 'rearm_func' means the hard-code is mixed like:

rearm_func(...) {
        ...
       txep = &txq->sw_ring[txq->tx_next_dd - (txq->tx_rs_thresh - 1)];
       for (...) {
          rxep[i].mbuf = txep[i].mbuf;
          mb0 = txep[i].mbuf;
          paddr = mb0->buf_iova + RTE_PKTMBUF_HEADROOM;
         dma_addr0 = vdupq_n_u64(paddr);
         vst1q_u64((uint64_t *)&rxdp++->read, dma_addr0);
       }
}

Is my understanding is right?

Quoted 6 lines
Quoted 2 lines

Sorry, I don't understand the question.
Can you probably elaborate a bit?

Sorry for my unclear expression.

I mean if we need two func which contains tx and rx paths code

respectively in rearm_func, like:

Quoted 5 lines
rearm_func(...) {
           rte_tx_fill_sw_ring;
           rte_rx_rearm_descs;
}
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

Or just mixed tx and rx path code like I said before. I prefer 'rx and tx hard code mixed', because from the performance perspective, this can

reduce the cost of function calls.

I suppose it depends on what we choose:
If we restrict it in a way that rxq and txq have to belong to the same PMD, then I suppose this decision could be left to each particular

PMD.

Quoted 8 lines
Quoted 8 lines

If we'd like to allow rearm to work accross different PMDs (i.e. it would allow to rearm mlx with ice and visa-versa), then yes we need PMDs somehow to expose rx_sw_ring abstraction to each other. My preference would be the second one - as it will make this feature more flexible and would help to adopt it more widely. Though the first one is probably easier to implement, and as I udnerstand you are leaning towards the first one.

Quoted 5 lines
Quoted 1 line

Konstantin

Hi, Konstantin

1. After further consideration, I think we should give up

'rte_eth_get_rx_direct_rearm' functions.

Quoted 1 line

And just keep one API 'rte_eth_direct_rearm' which contains different

queue types path in the pmd driver.

Quoted 5 lines

This is due to that application can dynamically configure queue-mapping in the data path, and thus expand direct-rearm usage scenarios based on this way. For example, consider a flow which received by one rx_queue(rxq_1) and sent by two different types

tx_queues( type_a txq_1 and type_b txq_2), Users can dynamically map direct_rearm according to tx path.

Quoted 14 lines

That's good point, indeed for such case special function pointer doesn't look very conveniet.
I still think that the best way to deal with multiple sources would be to expose some sort of rx_sw_ring abstraction.
Then it could be obtaiend once and passed as a paramter to actual rearm function.
In that case, user can even re-arm RX queue not from TX queue only, but from other sources too (freed mbufs, etc.).
As I understand what you suggest is one unified function:
int rte_eth_rx_direct_rearm(rx_port, rx_queue, tx_port, tx_queue), correct? I am not against it in principle, but does it mean at each invocation, inside PMD layer, first thing you would have to do - check that rx_queue and tx_queue are from the same device type and select proper behaviour (for different queue configs)?

Sorry for my late reply. We just used direct-rearm to deal with multiple sources by expose
some sort of rx_sw_ring abstraction. (For example, Rx: ixgbe Tx:i40e):
http://patches.dpdk.org/project/dpdk/cover/[email protected]/
Following are the test results:
(1) dpdk l3fwd test with multiple drivers: port 0: 82599 NIC port 1: XL710 NIC
-------------------------------------------------------------------------------------------------------
Without fast free With fast free
Thunderx2: +9.44% +7.14%
-------------------------------------------------------------------------------------------------------

(2) dpdk l3fwd test with same driver: port 0 && 1 XL710 NIC
-------------------------------------------------------------------------------------------------------
*Direct rearm with exposing rx_sw_ring:
Without fast free With fast free
Ampere altra: +14.98% +15.77%
n1sdp: +6.47% +0.52%

*Direct rearm with one unified function (community patch):
Without fast free With fast free
Ampere altra: +17.18% +16.92%
n1sdp: +15.09% +4.2%
-------------------------------------------------------------------------------------------------------

(3) VPP test with same driver: port 0 && 1 XL710 NIC
-------------------------------------------------------------------------------------------------------
*Direct rearm with exposing rx_sw_ring:
Ampere altra: +4.59%
n1sdp: +5.4%

*Direct rearm with one unified function (community patch):
Ampere altra: +4.50%
n1sdp: +4.5%
-------------------------------------------------------------------------------------------------------
According to the test results, we can see direct-rearm mode by expose rx_sw_ring can improve the performance significantly for the same driver and different driver.

Compared with direct-rearm with one unified function in community, there will be some performance degradation in n1sdp for new direct-rearm mode. This is due to we separate fill sw_ring and flush descriptors operation which may increase cache-misses. For other servers, performance is Ok. I think performance degradation in n1sdp can be achieved and new scheme can bring better expansibility.

Best Regards
Feifei

Quoted 54 lines
Quoted 22 lines

----------------------------------------------------------------------
----------------
rte_eth_rx_burst(rxq_1);

%routing table lookup to decide tx queue% txq_n = txq_1 or txq_2?

rte_eth_tx_burst(txq_n);
rte_eth_direct_rearm(txq_n);
----------------------------------------------------------------------
---------------- Thus, this can avoid repeatedly calling 'rte_eth_get_rx_direct_rearm' in the data path to reduce performance.

Furthermore, rte_eth_direct_rearm can be implemented as:
----------------------------------------------------------------------
---------------- rte_eth_direct_rearm = i40e_eth_direct_rearm;

i40e_eth_direct_rearm {
	rearm_queue_config_a path;
	rearm_queue_config_b path;
	rearm_queue_config_c path;
	rearm_queue_config_d path;
}
Quoted 7 lines

----------------------------------------------------------------------
---------------- This implementation refers to 'rte_eth_rx_burst' and 'rte_eth_tx_burst'. If there will be different queue types, the implementation will be in pmd layer.

2. I'm not sure whether cross pmd usage is realistic. Consider the tradeoff between performance and different pmd map,

maybe

Quoted 1 line

we can provide two different direct rearm mode for users. One is mixed Rx

and Tx path and is used for the same pmd.

Quoted 9 lines

The other is separate Rx and Tx path, and is used for different pmd. Best Regards
Feifei

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Another way - make rte_eth_txq_data totally opaque and allow

PMD

Quoted 12 lines
Quoted 11 lines

to

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

store there some data that will help it to distinguish expected TXQ

format.

Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines

That will allow PMD to keep rx_direct_rearm() the same for all supported TXQ formats (it will make decision internally based on data stored in txq_data).
Though in that case you'll probably need one more dev-op to free

txq_data.

Quoted 1 line
#10 lore raw

PATCH v2 2/3 net/i40e: enable direct rearm mode

In reply to Feifei Wang

For i40e driver, enable direct re-arm mode. This patch supports the case of mapping Rx/Tx queues from the same single lcore.

---
 drivers/net/i40e/i40e_ethdev.c        |  1 +
 drivers/net/i40e/i40e_ethdev.h        |  2 +
 drivers/net/i40e/i40e_rxtx.c          | 19 ++++++
 drivers/net/i40e/i40e_rxtx.h          |  2 +
 drivers/net/i40e/i40e_rxtx_vec_neon.c | 93 +++++++++++++++++++++++++++
 5 files changed, 117 insertions(+)
drivers/net/i40e/i40e_ethdev.c +1 -0
index 65e689df32..649ec06f31 100644
@@ -497,6 +497,7 @@ static const struct eth_dev_ops i40e_eth_dev_ops = {
 	.flow_ops_get                 = i40e_dev_flow_ops_get,
 	.rxq_info_get                 = i40e_rxq_info_get,
 	.txq_info_get                 = i40e_txq_info_get,
+	.txq_data_get                 = i40e_txq_data_get,
 	.rx_burst_mode_get            = i40e_rx_burst_mode_get,
 	.tx_burst_mode_get            = i40e_tx_burst_mode_get,
 	.timesync_enable              = i40e_timesync_enable,
drivers/net/i40e/i40e_ethdev.h +2 -0
index fe943a45ff..ee13730917 100644
@@ -1352,6 +1352,8 @@ void i40e_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 	struct rte_eth_rxq_info *qinfo);
 void i40e_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 	struct rte_eth_txq_info *qinfo);
+void i40e_txq_data_get(struct rte_eth_dev *dev, uint16_t queue_id,
+	struct rte_eth_txq_data *txq_data);
 int i40e_rx_burst_mode_get(struct rte_eth_dev *dev, uint16_t queue_id,
 			   struct rte_eth_burst_mode *mode);
 int i40e_tx_burst_mode_get(struct rte_eth_dev *dev, uint16_t queue_id,
drivers/net/i40e/i40e_rxtx.c +19 -0
index 788ffb51c2..be92ccd38a 100644
@@ -3197,6 +3197,24 @@ i40e_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 	qinfo->conf.offloads = txq->offloads;
 }
 
+void
+i40e_txq_data_get(struct rte_eth_dev *dev, uint16_t queue_id,
+			struct rte_eth_txq_data *txq_data)
+{
+	struct i40e_tx_queue *txq;
+
+	txq = dev->data->tx_queues[queue_id];
+
+	txq_data->offloads = &txq->offloads;
+	txq_data->tx_sw_ring = txq->sw_ring;
+	txq_data->tx_ring = txq->tx_ring;
+	txq_data->tx_next_dd = &txq->tx_next_dd;
+	txq_data->nb_tx_free = &txq->nb_tx_free;
+	txq_data->nb_tx_desc = txq->nb_tx_desc;
+	txq_data->tx_rs_thresh = txq->tx_rs_thresh;
+	txq_data->tx_free_thresh = txq->tx_free_thresh;
+}
+
 #ifdef RTE_ARCH_X86
 static inline bool
 get_avx_supported(bool request_avx512)
@@ -3321,6 +3339,7 @@ i40e_set_rx_function(struct rte_eth_dev *dev)
 			PMD_INIT_LOG(DEBUG, "Using Vector Rx (port %d).",
 				     dev->data->port_id);
 			dev->rx_pkt_burst = i40e_recv_pkts_vec;
+			dev->rx_direct_rearm = i40e_direct_rearm_vec;
 		}
 #endif /* RTE_ARCH_X86 */
 	} else if (!dev->data->scattered_rx && ad->rx_bulk_alloc_allowed) {
drivers/net/i40e/i40e_rxtx.h +2 -0
index 5e6eecc501..7a8fa2d1e8 100644
@@ -216,6 +216,8 @@ uint16_t i40e_simple_prep_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 			       uint16_t nb_pkts);
 uint16_t i40e_prep_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 		uint16_t nb_pkts);
+uint16_t i40e_direct_rearm_vec(void *rx_queue,
+			  struct rte_eth_txq_data *txq_data);
 int i40e_tx_queue_init(struct i40e_tx_queue *txq);
 int i40e_rx_queue_init(struct i40e_rx_queue *rxq);
 void i40e_free_tx_resources(struct i40e_tx_queue *txq);
drivers/net/i40e/i40e_rxtx_vec_neon.c +93 -0

96 lines not on this page · raw message

Signature · 1 line
2.25.1
#11 lore raw

PATCH v2 3/3 examples/l3fwd: enable direct rearm mode

In reply to Feifei Wang

Enable direct rearm mode in l3fwd. Users can use parameters:
'--direct-rearm=(rx_portid,rx_queueid,tx_portid,tx_queueid)' to enable direct rearm.

---
 examples/l3fwd/l3fwd.h     | 12 +++++
 examples/l3fwd/l3fwd_lpm.c | 22 +++++++++
 examples/l3fwd/main.c      | 94 +++++++++++++++++++++++++++++++++++++-
 3 files changed, 127 insertions(+), 1 deletion(-)
examples/l3fwd/l3fwd.h +12 -0
index 40b5f32a9e..db097e344c 100644
@@ -57,6 +57,10 @@
 #endif
 #define HASH_ENTRY_NUMBER_DEFAULT	16
 
+/* MAX number of direct rearm mapping entry */
+#define MAX_DIRECT_REARM_ENTRY_NUMBER   16
+#define MAX_DIRECT_REARM_QUEUE_PER_PORT 8
+
 struct parm_cfg {
 	const char *rule_ipv4_name;
 	const char *rule_ipv6_name;
@@ -114,6 +118,14 @@ extern struct parm_cfg parm_config;
 
 extern struct acl_algorithms acl_alg[];
 
+/* Used in direct rearm mode */
+extern bool enabled_direct_rearm;
+extern uint8_t direct_rearm_entry_number;
+extern bool queue_enabled_direct_rearm[RTE_MAX_ETHPORTS][MAX_DIRECT_REARM_QUEUE_PER_PORT];
+extern uint16_t direct_rearm_map_tx_port[RTE_MAX_ETHPORTS][MAX_DIRECT_REARM_QUEUE_PER_PORT];
+extern uint16_t direct_rearm_map_tx_queue[RTE_MAX_ETHPORTS][MAX_DIRECT_REARM_QUEUE_PER_PORT];
+extern uint8_t direct_rearm_entry_idx[RTE_MAX_ETHPORTS][MAX_DIRECT_REARM_QUEUE_PER_PORT];
+
 /* Send burst of packets on an output interface */
 static inline int
 send_burst(struct lcore_conf *qconf, uint16_t n, uint16_t port)
examples/l3fwd/l3fwd_lpm.c +22 -0
index 22d7f61a42..973fe70aae 100644
@@ -150,6 +150,8 @@ lpm_main_loop(__rte_unused void *dummy)
 	int i, nb_rx;
 	uint16_t portid;
 	uint8_t queueid;
+	uint8_t idx;
+	struct rte_eth_txq_data txq_data[MAX_DIRECT_REARM_ENTRY_NUMBER];
 	struct lcore_conf *qconf;
 	const uint64_t drain_tsc = (rte_get_tsc_hz() + US_PER_S - 1) /
 		US_PER_S * BURST_TX_DRAIN_US;
@@ -175,6 +177,20 @@ lpm_main_loop(__rte_unused void *dummy)
 			lcore_id, portid, queueid);
 	}
 
+	if (enabled_direct_rearm) {
+		uint16_t tx_portid;
+		uint8_t tx_queueid;
+
+		for (i = 0; i < n_rx_q; i++) {
+			portid = qconf->rx_queue_list[i].port_id;
+			queueid = qconf->rx_queue_list[i].queue_id;
+			tx_portid = direct_rearm_map_tx_port[portid][queueid];
+			tx_queueid = direct_rearm_map_tx_queue[portid][queueid];
+			idx = direct_rearm_entry_idx[portid][queueid];
+			rte_eth_tx_queue_data_get(tx_portid, tx_queueid, &(txq_data[idx]));
+		}
+	}
+
 	cur_tsc = rte_rdtsc();
 	prev_tsc = cur_tsc;
 
@@ -205,6 +221,12 @@ lpm_main_loop(__rte_unused void *dummy)
 		for (i = 0; i < n_rx_q; ++i) {
 			portid = qconf->rx_queue_list[i].port_id;
 			queueid = qconf->rx_queue_list[i].queue_id;
+
+			if (queue_enabled_direct_rearm[portid][queueid]) {
+				idx = direct_rearm_entry_idx[portid][queueid];
+				rte_eth_rx_direct_rearm(portid, queueid, &(txq_data[idx]));
+			}
+
 			nb_rx = rte_eth_rx_burst(portid, queueid, pkts_burst,
 				MAX_PKT_BURST);
 			if (nb_rx == 0)
examples/l3fwd/main.c +93 -1

148 lines not on this page · raw message

Signature · 1 line
2.25.1
#12 lore raw

Re: [PATCH v2 3/3] examples/l3fwd: enable direct rearm mode

Quoted 52 lines · Feifei Wang

Enable direct rearm mode in l3fwd. Users can use parameters:
'--direct-rearm=(rx_portid,rx_queueid,tx_portid,tx_queueid)' to enable direct rearm.

---
 examples/l3fwd/l3fwd.h     | 12 +++++
 examples/l3fwd/l3fwd_lpm.c | 22 +++++++++
 examples/l3fwd/main.c      | 94 +++++++++++++++++++++++++++++++++++++-
 3 files changed, 127 insertions(+), 1 deletion(-)
examples/l3fwd/l3fwd.h +12 -0
index 40b5f32a9e..db097e344c 100644
@@ -57,6 +57,10 @@
 #endif
 #define HASH_ENTRY_NUMBER_DEFAULT      16
 
+/* MAX number of direct rearm mapping entry */
+#define MAX_DIRECT_REARM_ENTRY_NUMBER   16
+#define MAX_DIRECT_REARM_QUEUE_PER_PORT 8
+
 struct parm_cfg {
        const char *rule_ipv4_name;
        const char *rule_ipv6_name;
@@ -114,6 +118,14 @@ extern struct parm_cfg parm_config;
 
 extern struct acl_algorithms acl_alg[];
 
+/* Used in direct rearm mode */
+extern bool enabled_direct_rearm;
+extern uint8_t direct_rearm_entry_number;
+extern bool queue_enabled_direct_rearm[RTE_MAX_ETHPORTS][MAX_DIRECT_REARM_QUEUE_PER_PORT];
+extern uint16_t direct_rearm_map_tx_port[RTE_MAX_ETHPORTS][MAX_DIRECT_REARM_QUEUE_PER_PORT];
+extern uint16_t direct_rearm_map_tx_queue[RTE_MAX_ETHPORTS][MAX_DIRECT_REARM_QUEUE_PER_PORT];
+extern uint8_t direct_rearm_entry_idx[RTE_MAX_ETHPORTS][MAX_DIRECT_REARM_QUEUE_PER_PORT];
+
 /* Send burst of packets on an output interface */
 static inline int
 send_burst(struct lcore_conf *qconf, uint16_t n, uint16_t port)
examples/l3fwd/l3fwd_lpm.c +0 -0
index 22d7f61a42..973fe70aae 100644
@@ -150,6 +150,8 @@ lpm_main_loop(__rte_unused void *dummy)
 

examples/l3fwd/l3fwd_lpm.c lpm_main_loop(__rte_unused void *dummy) · line 225

Quoted 5 lines
examples/l3fwd/l3fwd_lpm.c +2 -0
@@ -205,6 +221,12 @@ lpm_main_loop(__rte_unused void *dummy)
                for (i = 0; i < n_rx_q; ++i) {
                        portid = qconf->rx_queue_list[i].port_id;
                        queueid = qconf->rx_queue_list[i].queue_id;
+
+                       if (queue_enabled_direct_rearm[portid][queueid]) {

IMO, We should not change fastpath code that impacts performance on other targets for a
feature which has a very constraint use case. Also need for expressing [1], kind of defeat the purpose
LPM table populated.

IMO, If we need to add a test case for this API, testpmd would be an ideal place with a dedicated fwd_engine just for this.

[1]

Quoted 1 line
examples/l3fwd/l3fwd_lpm.c +1 -0
+               "  --direct-rearm (rx_port, rx_queue, tx_port, tx_queue): Put Tx queue sw-ring buffers into Rx queue\n"
#13 lore raw

回复: [PATCH v2 3/3] examples/l3fwd: enable direct rearm mode

Quoted 88 lines

-----邮件原件-----
发件人: Jerin Jacob <[email protected]>
发送时间: Friday, September 30, 2022 7:56 PM
收件人: Feifei Wang <[email protected]>
抄送: [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; Ruifeng Wang <[email protected]>
主题: Re: [PATCH v2 3/3] examples/l3fwd: enable direct rearm mode

Quoted 5 lines · Feifei Wang

Enable direct rearm mode in l3fwd. Users can use parameters:
'--direct-rearm=(rx_portid,rx_queueid,tx_portid,tx_queueid)' to enable direct rearm.

Quoted 8 lines
---
 examples/l3fwd/l3fwd.h     | 12 +++++
 examples/l3fwd/l3fwd_lpm.c | 22 +++++++++
 examples/l3fwd/main.c      | 94

+++++++++++++++++++++++++++++++++++++-

Quoted 18 lines
 3 files changed, 127 insertions(+), 1 deletion(-)
examples/l3fwd/l3fwd.h index renamed from examples/l3fwd/l3fwd.h +0 -0

40b5f32a9e..db097e344c 100644

examples/l3fwd/l3fwd.h +4 -0
@@ -57,6 +57,10 @@
 #endif
 #define HASH_ENTRY_NUMBER_DEFAULT      16
 
+/* MAX number of direct rearm mapping entry */
+#define MAX_DIRECT_REARM_ENTRY_NUMBER   16
+#define MAX_DIRECT_REARM_QUEUE_PER_PORT 8
+
 struct parm_cfg {
        const char *rule_ipv4_name;
        const char *rule_ipv6_name;

examples/l3fwd/l3fwd.h extern struct parm_cfg parm_config; · line 124

Quoted 7 lines
examples/l3fwd/l3fwd.h +3 -0
@@ -114,6 +118,14 @@ extern struct parm_cfg parm_config;
 
 extern struct acl_algorithms acl_alg[];
 
+/* Used in direct rearm mode */
+extern bool enabled_direct_rearm;
+extern uint8_t direct_rearm_entry_number; extern bool
 

+queue_enabled_direct_rearm[RTE_MAX_ETHPORTS][MAX_DIRECT_REAR M_QUEUE_P

examples/l3fwd/l3fwd.h extern struct parm_cfg parm_config; · line 126

Quoted 2 lines
examples/l3fwd/l3fwd.h +1 -0
+ER_PORT]; extern uint16_t
 

+direct_rearm_map_tx_port[RTE_MAX_ETHPORTS][MAX_DIRECT_REARM_ QUEUE_PER

examples/l3fwd/l3fwd.h extern struct parm_cfg parm_config; · line 128

Quoted 2 lines
examples/l3fwd/l3fwd.h +1 -0
+_PORT]; extern uint16_t
 

+direct_rearm_map_tx_queue[RTE_MAX_ETHPORTS][MAX_DIRECT_REAR M_QUEUE_PE

examples/l3fwd/l3fwd.h extern struct parm_cfg parm_config; · line 129

Quoted 2 lines
examples/l3fwd/l3fwd.h +1 -0
+R_PORT]; extern uint8_t

+direct_rearm_entry_idx[RTE_MAX_ETHPORTS][MAX_DIRECT_REARM_QUE UE_PER_P

Quoted 11 lines
examples/l3fwd/l3fwd.h +2 -0
+ORT];
+

/* Send burst of packets on an output interface */ static inline int
send_burst(struct lcore_conf *qconf, uint16_t n, uint16_t port) diff --git a/examples/l3fwd/l3fwd_lpm.c b/examples/l3fwd/l3fwd_lpm.c index 22d7f61a42..973fe70aae 100644

examples/l3fwd/l3fwd_lpm.c +0 -0
@@ -150,6 +150,8 @@ lpm_main_loop(__rte_unused void *dummy)
 

examples/l3fwd/l3fwd_lpm.c lpm_main_loop(__rte_unused void *dummy) · line 226

Quoted 6 lines
examples/l3fwd/l3fwd_lpm.c +3 -0
@@ -205,6 +221,12 @@ lpm_main_loop(__rte_unused void *dummy)
                for (i = 0; i < n_rx_q; ++i) {
                        portid = qconf->rx_queue_list[i].port_id;
                        queueid = qconf->rx_queue_list[i].queue_id;
+
+                       if
+ (queue_enabled_direct_rearm[portid][queueid]) {

IMO, We should not change fastpath code that impacts performance on other targets for a feature which has a very constraint use case. Also need for expressing [1], kind of defeat the purpose LPM table populated.

Quoted 6 lines

IMO, If we need to add a test case for this API, testpmd would be an ideal place with a dedicated fwd_engine just for this.

[1]

examples/l3fwd/l3fwd_lpm.c lpm_main_loop(__rte_unused void *dummy) · line 227

Quoted 1 line
examples/l3fwd/l3fwd_lpm.c +1 -0
+               "  --direct-rearm (rx_port, rx_queue, tx_port, tx_queue): Put Tx

queue sw-ring buffers into Rx queue\n"

Thanks very much for your comments. This patch here is mainly to facilitate verification test and show how to use direct-rearm API.

After 'direct-rearm' matures, we will consider to enable direct-rearm in testpmd or other application in dpdk.

#14 lore raw

Re: [PATCH v2 3/3] examples/l3fwd: enable direct rearm mode

Quoted 89 lines · Feifei Wang
Quoted 89 lines

-----邮件原件-----
发件人: Jerin Jacob <[email protected]>
发送时间: Friday, September 30, 2022 7:56 PM
收件人: Feifei Wang <[email protected]>
抄送: [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; Ruifeng Wang <[email protected]>
主题: Re: [PATCH v2 3/3] examples/l3fwd: enable direct rearm mode

Quoted 5 lines · Feifei Wang

Enable direct rearm mode in l3fwd. Users can use parameters:
'--direct-rearm=(rx_portid,rx_queueid,tx_portid,tx_queueid)' to enable direct rearm.

Quoted 8 lines
---
 examples/l3fwd/l3fwd.h     | 12 +++++
 examples/l3fwd/l3fwd_lpm.c | 22 +++++++++
 examples/l3fwd/main.c      | 94

+++++++++++++++++++++++++++++++++++++-

Quoted 18 lines
 3 files changed, 127 insertions(+), 1 deletion(-)
examples/l3fwd/l3fwd.h index renamed from examples/l3fwd/l3fwd.h +0 -0

40b5f32a9e..db097e344c 100644

examples/l3fwd/l3fwd.h +4 -0
@@ -57,6 +57,10 @@
 #endif
 #define HASH_ENTRY_NUMBER_DEFAULT      16
 
+/* MAX number of direct rearm mapping entry */
+#define MAX_DIRECT_REARM_ENTRY_NUMBER   16
+#define MAX_DIRECT_REARM_QUEUE_PER_PORT 8
+
 struct parm_cfg {
        const char *rule_ipv4_name;
        const char *rule_ipv6_name;

examples/l3fwd/l3fwd.h extern struct parm_cfg parm_config; · line 124

Quoted 7 lines
examples/l3fwd/l3fwd.h +3 -0
@@ -114,6 +118,14 @@ extern struct parm_cfg parm_config;
 
 extern struct acl_algorithms acl_alg[];
 
+/* Used in direct rearm mode */
+extern bool enabled_direct_rearm;
+extern uint8_t direct_rearm_entry_number; extern bool
 

+queue_enabled_direct_rearm[RTE_MAX_ETHPORTS][MAX_DIRECT_REAR M_QUEUE_P

examples/l3fwd/l3fwd.h extern struct parm_cfg parm_config; · line 126

Quoted 2 lines
examples/l3fwd/l3fwd.h +1 -0
+ER_PORT]; extern uint16_t
 

+direct_rearm_map_tx_port[RTE_MAX_ETHPORTS][MAX_DIRECT_REARM_ QUEUE_PER

examples/l3fwd/l3fwd.h extern struct parm_cfg parm_config; · line 128

Quoted 2 lines
examples/l3fwd/l3fwd.h +1 -0
+_PORT]; extern uint16_t
 

+direct_rearm_map_tx_queue[RTE_MAX_ETHPORTS][MAX_DIRECT_REAR M_QUEUE_PE

examples/l3fwd/l3fwd.h extern struct parm_cfg parm_config; · line 129

Quoted 2 lines
examples/l3fwd/l3fwd.h +1 -0
+R_PORT]; extern uint8_t

+direct_rearm_entry_idx[RTE_MAX_ETHPORTS][MAX_DIRECT_REARM_QUE UE_PER_P

Quoted 11 lines
examples/l3fwd/l3fwd.h +2 -0
+ORT];
+

/* Send burst of packets on an output interface */ static inline int
send_burst(struct lcore_conf *qconf, uint16_t n, uint16_t port) diff --git a/examples/l3fwd/l3fwd_lpm.c b/examples/l3fwd/l3fwd_lpm.c index 22d7f61a42..973fe70aae 100644

examples/l3fwd/l3fwd_lpm.c +0 -0
@@ -150,6 +150,8 @@ lpm_main_loop(__rte_unused void *dummy)
 

examples/l3fwd/l3fwd_lpm.c lpm_main_loop(__rte_unused void *dummy) · line 226

Quoted 6 lines
examples/l3fwd/l3fwd_lpm.c +3 -0
@@ -205,6 +221,12 @@ lpm_main_loop(__rte_unused void *dummy)
                for (i = 0; i < n_rx_q; ++i) {
                        portid = qconf->rx_queue_list[i].port_id;
                        queueid = qconf->rx_queue_list[i].queue_id;
+
+                       if
+ (queue_enabled_direct_rearm[portid][queueid]) {

IMO, We should not change fastpath code that impacts performance on other targets for a feature which has a very constraint use case. Also need for expressing [1], kind of defeat the purpose LPM table populated.

Quoted 8 lines
Quoted 6 lines

IMO, If we need to add a test case for this API, testpmd would be an ideal place with a dedicated fwd_engine just for this.

[1]

examples/l3fwd/l3fwd_lpm.c lpm_main_loop(__rte_unused void *dummy) · line 227

Quoted 1 line
examples/l3fwd/l3fwd_lpm.c +1 -0
+               "  --direct-rearm (rx_port, rx_queue, tx_port, tx_queue): Put Tx

queue sw-ring buffers into Rx queue\n"

Thanks very much for your comments. This patch here is mainly to facilitate verification test and show how to use direct-rearm API.

IMO, it should be other way around. for PMD and ethdev API verification, Please use testpmd.

Quoted 3 lines

After 'direct-rearm' matures, we will consider to enable direct-rearm in testpmd or other application in dpdk.

#15 lore raw
In reply to Feifei Wang
Quoted 112 lines

-----邮件原件-----
发件人: Feifei Wang <[email protected]>
发送时间: Tuesday, September 27, 2022 10:48 AM
抄送: [email protected]; nd <[email protected]>; Feifei Wang <[email protected]>
主题: [PATCH v2 0/3] Direct re-arming of buffers on receive side

Currently, the transmit side frees the buffers into the lcore cache and the receive side allocates buffers from the lcore cache. The transmit side typically frees 32 buffers resulting in 32*8=256B of stores to lcore cache. The receive side allocates 32 buffers and stores them in the receive side software ring, resulting in 32*8=256B of stores and 256B of load from the lcore cache.

This patch proposes a mechanism to avoid freeing to/allocating from the lcore cache. i.e. the receive side will free the buffers from transmit side directly into it's software ring. This will avoid the 256B of loads and stores introduced by the lcore cache. It also frees up the cache lines used by the lcore cache.

However, this solution poses several constraints:

1)The receive queue needs to know which transmit queue it should take the buffers from. The application logic decides which transmit port to use to send out the packets. In many use cases the NIC might have a single port ([1], [2], [3]), in which case a given transmit queue is always mapped to a single receive queue (1:1 Rx queue: Tx queue). This is easy to configure.

If the NIC has 2 ports (there are several references), then we will have 1:2 (RX queue: TX queue) mapping which is still easy to configure. However, if this is generalized to 'N' ports, the configuration can be long. More over the PMD would have to scan a list of transmit queues to pull the buffers from.

2)The other factor that needs to be considered is 'run-to-completion' vs 'pipeline' models. In the run-to-completion model, the receive side and the transmit side are running on the same lcore serially. In the pipeline model. The receive side and transmit side might be running on different lcores in parallel. This requires locking. This is not supported at this point.

3)Tx and Rx buffers must be from the same mempool. And we also must ensure Tx buffer free number is equal to Rx buffer free number:
(txq->tx_rs_thresh == RTE_I40E_RXQ_REARM_THRESH) Thus, 'tx_next_dd' can be updated correctly in direct-rearm mode. This is due to tx_next_dd is a variable to compute tx sw-ring free location.
Its value will be one more round than the position where next time free starts.

Current status in this patch:
1)Two APIs are added for users to enable direct-rearm mode:
In control plane, users can call 'rte_eth_txq_data_get' to get Tx sw_ring pointer and its txq_info (This avoid Rx load Tx data directly);

In data plane, users can call 'rte_eth_rx_direct_rearm' to rearm Rx buffers and free Tx buffers at the same time (Currently it supports 1:1 (rxq:txq) mapping:)
-----------------------------------------------------------------------
control plane:
rte_eth_txq_data_get(*txq_data);
data plane:
loop {

  		rte_eth_rx_direct_rearm(*txq_data){
     			for (i = 0; i <= 32; i++) {
       				rx.mbuf[i] = tx.mbuf[i];
       				initialize descs[i];
    			}
		}
		rte_eth_rx_burst;
		rte_eth_tx_burst;

}
-----------------------------------------------------------------------
2)The i40e driver is changed to do the direct re-arm of the receive
side.
3)L3fwd application is modified to enable direct rearm mode. Users can
enable direct-rearm and map queues by input parameters.

Testing status:
1.The testing results for L3fwd are as follows:
-------------------------------------------------------------------
enabled direct rearm
-------------------------------------------------------------------
Arm:
N1SDP(neon path):
without fast-free mode with fast-free mode

	+15.09%				+4.2%

Ampere Altra(neon path):
without fast-free mode with fast-free mode

	+10.9%				+14.6%

-------------------------------------------------------------------

2.The testing results for VPP-L3fwd are as follows:
-------------------------------------------------------------------
Arm:
N1SDP(neon path):
with direct re-arm mode enabled

	+4.5%

Ampere Altra(neon path):
with direct re-arm mode enabled

        +6.5%

-------------------------------------------------------------------

Reference:
[1] https://store.nvidia.com/en-
us/networking/store/product/MCX623105AN- CDAT/NVIDIAMCX623105ANCDATConnectX6DxENAdapterCard100GbECrypt oDisabled/
[2] https://www.intel.com/content/www/us/en/products/sku/192561/intel-
ethernet-network-adapter-e810cqda1/specifications.html
[3] https://www.broadcom.com/products/ethernet-connectivity/network-
adapters/100gb-nic-ocp/n1100g

Quoted 6 lines

V2:
1. Use data-plane API to enable direct-rearm (Konstantin, Honnappa) 2. Add 'txq_data_get' API to get txq info for Rx (Konstantin) 3. Use input parameter to enable direct rearm in l3fwd (Konstantin) 4. Add condition detection for direct rearm API (Morten, Andrew Rybchenko)

PING

Hi,

Would you please give some comments for this version? Thanks very much.

Best Regards
Feifei

#16 lore raw
Quoted 118 lines · Feifei Wang
Quoted 112 lines

-----邮件原件-----
发件人: Feifei Wang <[email protected]>
发送时间: Tuesday, September 27, 2022 10:48 AM
抄送: [email protected]; nd <[email protected]>; Feifei Wang <[email protected]>
主题: [PATCH v2 0/3] Direct re-arming of buffers on receive side

Currently, the transmit side frees the buffers into the lcore cache and the receive side allocates buffers from the lcore cache. The transmit side typically frees 32 buffers resulting in 32*8=256B of stores to lcore cache. The receive side allocates 32 buffers and stores them in the receive side software ring, resulting in 32*8=256B of stores and 256B of load from the lcore cache.

This patch proposes a mechanism to avoid freeing to/allocating from the lcore cache. i.e. the receive side will free the buffers from transmit side directly into it's software ring. This will avoid the 256B of loads and stores introduced by the lcore cache. It also frees up the cache lines used by the lcore cache.

However, this solution poses several constraints:

1)The receive queue needs to know which transmit queue it should take the buffers from. The application logic decides which transmit port to use to send out the packets. In many use cases the NIC might have a single port ([1], [2], [3]), in which case a given transmit queue is always mapped to a single receive queue (1:1 Rx queue: Tx queue). This is easy to configure.

If the NIC has 2 ports (there are several references), then we will have 1:2 (RX queue: TX queue) mapping which is still easy to configure. However, if this is generalized to 'N' ports, the configuration can be long. More over the PMD would have to scan a list of transmit queues to pull the buffers from.

2)The other factor that needs to be considered is 'run-to-completion' vs 'pipeline' models. In the run-to-completion model, the receive side and the transmit side are running on the same lcore serially. In the pipeline model. The receive side and transmit side might be running on different lcores in parallel. This requires locking. This is not supported at this point.

3)Tx and Rx buffers must be from the same mempool. And we also must ensure Tx buffer free number is equal to Rx buffer free number:
(txq->tx_rs_thresh == RTE_I40E_RXQ_REARM_THRESH) Thus, 'tx_next_dd' can be updated correctly in direct-rearm mode. This is due to tx_next_dd is a variable to compute tx sw-ring free location.
Its value will be one more round than the position where next time free starts.

Current status in this patch:
1)Two APIs are added for users to enable direct-rearm mode:
In control plane, users can call 'rte_eth_txq_data_get' to get Tx sw_ring pointer and its txq_info (This avoid Rx load Tx data directly);

In data plane, users can call 'rte_eth_rx_direct_rearm' to rearm Rx buffers and free Tx buffers at the same time (Currently it supports 1:1 (rxq:txq) mapping:)
-----------------------------------------------------------------------
control plane:

   	rte_eth_txq_data_get(*txq_data);
   data plane:
   	loop {
   		rte_eth_rx_direct_rearm(*txq_data){
      			for (i = 0; i <= 32; i++) {
        				rx.mbuf[i] = tx.mbuf[i];
        				initialize descs[i];
     			}
		}
		rte_eth_rx_burst;
		rte_eth_tx_burst;
   	}

-----------------------------------------------------------------------
2)The i40e driver is changed to do the direct re-arm of the receive
side.
3)L3fwd application is modified to enable direct rearm mode. Users can
enable direct-rearm and map queues by input parameters.

Testing status:
1.The testing results for L3fwd are as follows:
-------------------------------------------------------------------
enabled direct rearm
-------------------------------------------------------------------
Arm:
N1SDP(neon path):
without fast-free mode with fast-free mode

	+15.09%				+4.2%

Ampere Altra(neon path):
without fast-free mode with fast-free mode

	+10.9%				+14.6%

-------------------------------------------------------------------

2.The testing results for VPP-L3fwd are as follows:
-------------------------------------------------------------------
Arm:
N1SDP(neon path):
with direct re-arm mode enabled

	+4.5%

Ampere Altra(neon path):
with direct re-arm mode enabled

         +6.5%

-------------------------------------------------------------------

Reference:
[1] https://store.nvidia.com/en-
us/networking/store/product/MCX623105AN- CDAT/NVIDIAMCX623105ANCDATConnectX6DxENAdapterCard100GbECrypt oDisabled/
[2] https://www.intel.com/content/www/us/en/products/sku/192561/intel-
ethernet-network-adapter-e810cqda1/specifications.html
[3] https://www.broadcom.com/products/ethernet-connectivity/network-
adapters/100gb-nic-ocp/n1100g

Quoted 6 lines

V2:
1. Use data-plane API to enable direct-rearm (Konstantin, Honnappa) 2. Add 'txq_data_get' API to get txq info for Rx (Konstantin) 3. Use input parameter to enable direct rearm in l3fwd (Konstantin) 4. Add condition detection for direct rearm API (Morten, Andrew Rybchenko)

Quoted 7 lines

PING

Hi,

Would you please give some comments for this version? Thanks very much.

Hi Feifei,

Because of the restrictions this feature has, although it gives a performance improvement on some specific tests I believe it doesn't have much practical usage, so I have constraints about the feature.