Skip to contentSkip to conversation

vhost: introduce DMA vchannel unconfiguration

PATCH Superseded [email protected] 4 participants 43 messages Aug 14, 2022 – Oct 26, 2022

lore
Share & tools
lore
#0 lore raw

cover vhost: introduce DMA vchannel unconfiguration

From: Xuan Ding <[email protected]>

This patchset introduces a new API rte_vhost_async_dma_unconfigure() to help user to manually free the DMA vchannel finished to use.

Note: this API should be called after async channel unregister.

Xuan Ding (2):
vhost: introduce DMA vchannel unconfiguration
example/vhost: unconfigure DMA vchannel

 doc/guides/prog_guide/vhost_lib.rst    |  5 ++++
 doc/guides/rel_notes/release_22_11.rst |  2 ++
 examples/vhost/main.c                  |  7 +++++
 lib/vhost/rte_vhost_async.h            | 17 ++++++++++++
 lib/vhost/version.map                  |  3 +++
 lib/vhost/vhost.c                      | 37 ++++++++++++++++++++++++++
 6 files changed, 71 insertions(+)
Signature · 1 line
2.17.1
#1 lore raw

PATCH 1/2 vhost: introduce DMA vchannel unconfiguration

From: Xuan Ding <[email protected]>

This patch adds a new API rte_vhost_async_dma_unconfigure() to unconfigure DMA vchannels in vhost async data path.

---
 doc/guides/prog_guide/vhost_lib.rst    |  5 ++++
 doc/guides/rel_notes/release_22_11.rst |  2 ++
 lib/vhost/rte_vhost_async.h            | 17 ++++++++++++
 lib/vhost/version.map                  |  3 +++
 lib/vhost/vhost.c                      | 37 ++++++++++++++++++++++++++
 5 files changed, 64 insertions(+)
doc/guides/prog_guide/vhost_lib.rst +5 -0
index bad4d819e1..78b1943d29 100644
@@ -323,6 +323,11 @@ The following is an overview of some key Vhost API functions:
   Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET,
   VDPA_DEVICE_TYPE_BLK.
 
+* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)``
+
+  Clear DMA vChannels finished to use. This function needs to
+  be called after async unregister.
+
 Vhost-user Implementations
 --------------------------
 
doc/guides/rel_notes/release_22_11.rst +2 -0
index 8c021cf050..e94c006e39 100644
@@ -55,6 +55,8 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added vhost API to unconfigure DMA vchannels.**
+  Added an API which helps to unconfigure DMA vchannels.
 
 Removed Items
 -------------
lib/vhost/rte_vhost_async.h +17 -0
index 1db2a10124..0442e027fd 100644
@@ -266,6 +266,23 @@ rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id,
 	struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count,
 	int *nr_inflight, int16_t dma_id, uint16_t vchan_id);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice.
+ *
+ * Unconfigure DMA vChannels in asynchronous data path.
+ *
+ * @param dma_id
+ *  the identifier of DMA device
+ * @param vchan_id
+ *  the identifier of virtual DMA channel
+ * @return
+ *  0 on success, and -1 on failure
+ */
+__rte_experimental
+int
+rte_vhost_async_dma_unconfigure(int16_t dma_id, uint16_t vchan_id);
+
 #ifdef __cplusplus
 }
 #endif
lib/vhost/version.map +3 -0
index 18574346d5..013a6bcc42 100644
@@ -96,6 +96,9 @@ EXPERIMENTAL {
 	rte_vhost_async_try_dequeue_burst;
 	rte_vhost_driver_get_vdpa_dev_type;
 	rte_vhost_clear_queue;
+
+	# added in 22.11
+	rte_vhost_async_dma_unconfigure;
 };
 
 INTERNAL {
lib/vhost/vhost.c +37 -0
index 60cb05a0ff..d215c917a2 100644
@@ -1863,6 +1863,43 @@ rte_vhost_async_channel_unregister_thread_unsafe(int vid, uint16_t queue_id)
 	return 0;
 }
 
+int
+rte_vhost_async_dma_unconfigure(int16_t dma_id, uint16_t vchan_id)
+{
+	struct rte_dma_info info;
+	void *pkts_cmpl_flag_addr;
+
+	if (!rte_dma_is_valid(dma_id)) {
+		VHOST_LOG_CONFIG("dma", ERR, "DMA %d is not found.\n", dma_id);
+		return -1;
+	}
+
+	if (rte_dma_info_get(dma_id, &info) != 0) {
+		VHOST_LOG_CONFIG("dma", ERR, "Fail to get DMA %d information.\n", dma_id);
+		return -1;
+	}
+
+	if (vchan_id >= info.max_vchans) {
+		VHOST_LOG_CONFIG("dma", ERR, "Invalid DMA %d vChannel %u.\n", dma_id, vchan_id);
+		return -1;
+	}
+
+	pkts_cmpl_flag_addr = dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr;
+	if (pkts_cmpl_flag_addr) {
+		rte_free(pkts_cmpl_flag_addr);
+		pkts_cmpl_flag_addr = NULL;
+	}
+
+	if (dma_copy_track[dma_id].vchans) {
+		rte_free(dma_copy_track[dma_id].vchans);
+		dma_copy_track[dma_id].vchans = NULL;
+	}
+
+	dma_copy_track[dma_id].nr_vchans--;
+
+	return 0;
+}
+
 int
 rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id)
 {
Signature · 1 line
2.17.1
#2 lore raw

PATCH 2/2 example/vhost: unconfigure DMA vchannel

In reply to [email protected]

From: Xuan Ding <[email protected]>

This patch uses rte_vhost_async_dma_unconfigure() API to manually free 'dma_coy_track' array rather than wait for the program to finish before being freed.

---
 examples/vhost/main.c | 7 +++++++
 1 file changed, 7 insertions(+)
examples/vhost/main.c +7 -0
index 7e1666f42a..1dba9724c2 100644
@@ -2060,6 +2060,13 @@ main(int argc, char *argv[])
 	RTE_LCORE_FOREACH_WORKER(lcore_id)
 		rte_eal_wait_lcore(lcore_id);
 
+	for (i = 0; i < dma_count; i++) {
+		if (rte_vhost_async_dma_unconfigure(dmas_id[i], 0) < 0) {
+			RTE_LOG(ERR, VHOST_PORT, "Failed to unconfigure DMA in vhost.\n");
+			rte_exit(EXIT_FAILURE, "Cannot use given DMA device\n");
+		}
+	}
+
 	/* clean up the EAL */
 	rte_eal_cleanup();
 
Signature · 1 line
2.17.1
#3 lore raw

v2 cover vhost: introduce DMA vchannel unconfiguration

In reply to [email protected]

From: Xuan Ding <[email protected]>

This patchset introduces a new API rte_vhost_async_dma_unconfigure() to help user to manually free the DMA vchannel finished to use.

Note: this API should be called after async channel unregister.

v2:
* Add spinlock protection.
* Fix a memory leak issue.
* Refine the doc.

Xuan Ding (2):
vhost: introduce DMA vchannel unconfiguration
examples/vhost: unconfigure DMA vchannel

 doc/guides/prog_guide/vhost_lib.rst    |  5 ++
 doc/guides/rel_notes/release_22_11.rst |  2 +
 examples/vhost/main.c                  |  7 +++
 lib/vhost/rte_vhost_async.h            | 17 +++++++
 lib/vhost/version.map                  |  3 ++
 lib/vhost/vhost.c                      | 69 ++++++++++++++++++++++++--
 6 files changed, 98 insertions(+), 5 deletions(-)
Signature · 1 line
2.17.1
#4 lore raw

PATCH v2 1/2 vhost: introduce DMA vchannel unconfiguration

From: Xuan Ding <[email protected]>

This patch adds a new API rte_vhost_async_dma_unconfigure() to unconfigure DMA vchannels in vhost async data path.

Lock protection are also added to protect DMA vchannels configuration and unconfiguration from concurrent calls.

---
 doc/guides/prog_guide/vhost_lib.rst    |  5 ++
 doc/guides/rel_notes/release_22_11.rst |  2 +
 lib/vhost/rte_vhost_async.h            | 17 +++++++
 lib/vhost/version.map                  |  3 ++
 lib/vhost/vhost.c                      | 69 ++++++++++++++++++++++++--
 5 files changed, 91 insertions(+), 5 deletions(-)
doc/guides/prog_guide/vhost_lib.rst +5 -0
index bad4d819e1..22764cbeaa 100644
@@ -323,6 +323,11 @@ The following is an overview of some key Vhost API functions:
   Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET,
   VDPA_DEVICE_TYPE_BLK.
 
+* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)``
+
+  Clear DMA vChannels finished to use. This function needs to
+  be called after the deregisterration of async path has been finished.
+
 Vhost-user Implementations
 --------------------------
 
doc/guides/rel_notes/release_22_11.rst +2 -0
index 8c021cf050..e94c006e39 100644
@@ -55,6 +55,8 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added vhost API to unconfigure DMA vchannels.**
+  Added an API which helps to unconfigure DMA vchannels.
 
 Removed Items
 -------------
lib/vhost/rte_vhost_async.h +17 -0
index 1db2a10124..0442e027fd 100644
@@ -266,6 +266,23 @@ rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id,
 	struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count,
 	int *nr_inflight, int16_t dma_id, uint16_t vchan_id);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice.
+ *
+ * Unconfigure DMA vChannels in asynchronous data path.
+ *
+ * @param dma_id
+ *  the identifier of DMA device
+ * @param vchan_id
+ *  the identifier of virtual DMA channel
+ * @return
+ *  0 on success, and -1 on failure
+ */
+__rte_experimental
+int
+rte_vhost_async_dma_unconfigure(int16_t dma_id, uint16_t vchan_id);
+
 #ifdef __cplusplus
 }
 #endif
lib/vhost/version.map +3 -0
index 18574346d5..013a6bcc42 100644
@@ -96,6 +96,9 @@ EXPERIMENTAL {
 	rte_vhost_async_try_dequeue_burst;
 	rte_vhost_driver_get_vdpa_dev_type;
 	rte_vhost_clear_queue;
+
+	# added in 22.11
+	rte_vhost_async_dma_unconfigure;
 };
 
 INTERNAL {
lib/vhost/vhost.c +64 -5

121 lines not on this page · raw message

Signature · 1 line
2.17.1
#5 lore raw
Quoted 37 lines

-----Original Message-----
From: Ding, Xuan <[email protected]>
Sent: Tuesday, September 6, 2022 1:22 PM
To: [email protected]; Xia, Chenbo <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; He, Xingguang <[email protected]>; Yang, YvonneX <[email protected]>; Jiang, Cheng1 <[email protected]>; Wang, YuanX <[email protected]>; Ma, WenwuX <[email protected]>; Ding, Xuan <[email protected]>
Subject: [PATCH v2 1/2] vhost: introduce DMA vchannel unconfiguration

From: Xuan Ding <[email protected]>

This patch adds a new API rte_vhost_async_dma_unconfigure() to unconfigure DMA vchannels in vhost async data path.

Lock protection are also added to protect DMA vchannels configuration and unconfiguration from concurrent calls.

---
 doc/guides/prog_guide/vhost_lib.rst    |  5 ++
 doc/guides/rel_notes/release_22_11.rst |  2 +
 lib/vhost/rte_vhost_async.h            | 17 +++++++
 lib/vhost/version.map                  |  3 ++
 lib/vhost/vhost.c                      | 69 ++++++++++++++++++++++++--
 5 files changed, 91 insertions(+), 5 deletions(-)
doc/guides/prog_guide/vhost_lib.rst +0 -0
index bad4d819e1..22764cbeaa 100644
@@ -323,6 +323,11 @@ The following is an overview of some key Vhost API

functions:
Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET, VDPA_DEVICE_TYPE_BLK.

doc/guides/prog_guide/vhost_lib.rst The following is an overview of some key Vhost API functions: · line 329

Quoted 4 lines
doc/guides/prog_guide/vhost_lib.rst +4 -0
+* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)``
+
+  Clear DMA vChannels finished to use. This function needs to
+  be called after the deregisterration of async path has been finished.

Deregistration

Quoted 10 lines
doc/guides/prog_guide/vhost_lib.rst +1 -0
+
 Vhost-user Implementations
 --------------------------
doc/guides/rel_notes/release_22_11.rst +0 -0
index 8c021cf050..e94c006e39 100644

doc/guides/rel_notes/release_22_11.rst New Features · line 59

Quoted 5 lines
doc/guides/rel_notes/release_22_11.rst +2 -0
@@ -55,6 +55,8 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added vhost API to unconfigure DMA vchannels.**
+  Added an API which helps to unconfigure DMA vchannels.

Added XXX for async vhost

Overall LGTM. It seems it needs some rebasing too.

Thanks,
Chenbo

Quoted 204 lines
doc/guides/rel_notes/release_22_11.rst +0 -0
 Removed Items
 -------------
lib/vhost/rte_vhost_async.h +3 -0
index 1db2a10124..0442e027fd 100644
@@ -266,6 +266,23 @@ rte_vhost_async_try_dequeue_burst(int vid, uint16_t
 queue_id,
 	struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t
 count,
 	int *nr_inflight, int16_t dma_id, uint16_t vchan_id);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior

notice.
+ *
+ * Unconfigure DMA vChannels in asynchronous data path.
+ *
+ * @param dma_id
+ * the identifier of DMA device
+ * @param vchan_id
+ * the identifier of virtual DMA channel
+ * @return
+ * 0 on success, and -1 on failure
+ */

lib/vhost/rte_vhost_async.h +4 -0
+__rte_experimental
+int
+rte_vhost_async_dma_unconfigure(int16_t dma_id, uint16_t vchan_id);
+
 #ifdef __cplusplus
 }
 #endif
lib/vhost/version.map +3 -0
index 18574346d5..013a6bcc42 100644
@@ -96,6 +96,9 @@ EXPERIMENTAL {
 	rte_vhost_async_try_dequeue_burst;
 	rte_vhost_driver_get_vdpa_dev_type;
 	rte_vhost_clear_queue;
+
+	# added in 22.11
+	rte_vhost_async_dma_unconfigure;
 };
 
 INTERNAL {
lib/vhost/vhost.c +5 -3
index 60cb05a0ff..273616da11 100644
@@ -23,6 +23,7 @@
 
 struct virtio_net *vhost_devices[RTE_MAX_VHOST_DEVICE];
 pthread_mutex_t vhost_dev_lock = PTHREAD_MUTEX_INITIALIZER;
+static rte_spinlock_t vhost_dma_lock = RTE_SPINLOCK_INITIALIZER;
 
 struct vhost_vq_stats_name_off {
 	char name[RTE_VHOST_STATS_NAME_SIZE];
@@ -1870,19 +1871,20 @@ rte_vhost_async_dma_configure(int16_t dma_id,
 uint16_t vchan_id)
 	void *pkts_cmpl_flag_addr;
 	uint16_t max_desc;
 
+	rte_spinlock_lock(&vhost_dma_lock);
 	if (!rte_dma_is_valid(dma_id)) {
 		VHOST_LOG_CONFIG("dma", ERR, "DMA %d is not found.\n", dma_id);
-		return -1;
+		goto error;
 	}
 
 	if (rte_dma_info_get(dma_id, &info) != 0) {
 		VHOST_LOG_CONFIG("dma", ERR, "Fail to get DMA %d
 information.\n", dma_id);
-		return -1;
+		goto error;
 	}
 
 	if (vchan_id >= info.max_vchans) {
 		VHOST_LOG_CONFIG("dma", ERR, "Invalid DMA %d vChannel %u.\n",
 dma_id, vchan_id);
-		return -1;
+		goto error;
lib/vhost/vhost.c +1 -1
 	}
 
 	if (!dma_copy_track[dma_id].vchans) {
@@ -1894,7 +1896,7 @@ rte_vhost_async_dma_configure(int16_t dma_id,
 uint16_t vchan_id)
 			VHOST_LOG_CONFIG("dma", ERR,
 				"Failed to allocate vchans for DMA %d
 vChannel %u.\n",
 				dma_id, vchan_id);
-			return -1;
+			goto error;
 		}
lib/vhost/vhost.c +1 -0
 		dma_copy_track[dma_id].vchans = vchans;
@@ -1903,6 +1905,7 @@ rte_vhost_async_dma_configure(int16_t dma_id,
 uint16_t vchan_id)
 	if (dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr) {
 		VHOST_LOG_CONFIG("dma", INFO, "DMA %d vChannel %u already
 registered.\n",
 			dma_id, vchan_id);
+		rte_spinlock_unlock(&vhost_dma_lock);
 		return 0;
lib/vhost/vhost.c +1 -1
 	}
@@ -1920,7 +1923,7 @@ rte_vhost_async_dma_configure(int16_t dma_id,
 uint16_t vchan_id)
 			rte_free(dma_copy_track[dma_id].vchans);
 			dma_copy_track[dma_id].vchans = NULL;
 		}
-		return -1;
+		goto error;
 	}
 

dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr =
pkts_cmpl_flag_addr;

lib/vhost/vhost.c +5 -0
@@ -1928,7 +1931,12 @@ rte_vhost_async_dma_configure(int16_t dma_id,
 uint16_t vchan_id)
 	dma_copy_track[dma_id].vchans[vchan_id].ring_mask = max_desc - 1;
 	dma_copy_track[dma_id].nr_vchans++;
 
+	rte_spinlock_unlock(&vhost_dma_lock);
 	return 0;
+
+error:
+	rte_spinlock_unlock(&vhost_dma_lock);
+	return -1;
 }
 

int

lib/vhost/vhost.c +20 -0
@@ -2117,5 +2125,56 @@ int rte_vhost_vring_stats_reset(int vid, uint16_t
 queue_id)
 	return 0;
 }
 
+int
+rte_vhost_async_dma_unconfigure(int16_t dma_id, uint16_t vchan_id)
+{
+	struct rte_dma_info info;
+	uint16_t max_desc;
+	int i;
+
+	rte_spinlock_lock(&vhost_dma_lock);
+	if (!rte_dma_is_valid(dma_id)) {
+		VHOST_LOG_CONFIG("dma", ERR, "DMA %d is not found.\n", dma_id);
+		goto error;
+	}
+
+	if (rte_dma_info_get(dma_id, &info) != 0) {
+		VHOST_LOG_CONFIG("dma", ERR, "Fail to get DMA %d
 information.\n", dma_id);
+		goto error;
+	}
+
+	if (vchan_id >= info.max_vchans) {
+		VHOST_LOG_CONFIG("dma", ERR, "Invalid DMA %d vChannel %u.\n",

dma_id, vchan_id);

lib/vhost/vhost.c +6 -0
+		goto error;
+	}
+
+	max_desc = info.max_desc;
+	for (i = 0; i < max_desc; i++) {
+		if

(dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr[i] != NULL) { +
rte_free(dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr
[i]);
+
dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr[i] =
NULL;

lib/vhost/vhost.c +4 -0
+		}
+	}
+
+	if (dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr !=

NULL) {
+
rte_free(dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr
);

lib/vhost/vhost.c +1 -0
+		dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr =

NULL;

lib/vhost/vhost.c +17 -1
+	}
+
+	if (dma_copy_track[dma_id].vchans != NULL) {
+		rte_free(dma_copy_track[dma_id].vchans);
+		dma_copy_track[dma_id].vchans = NULL;
+	}
+
+	dma_copy_track[dma_id].nr_vchans--;
+
+	rte_spinlock_unlock(&vhost_dma_lock);
+	return 0;
+
+error:
+	rte_spinlock_unlock(&vhost_dma_lock);
+	return -1;
+}
+
 RTE_LOG_REGISTER_SUFFIX(vhost_config_log_level, config, INFO);
 RTE_LOG_REGISTER_SUFFIX(vhost_data_log_level, data, WARNING);
--

2.17.1

author
#6 lore raw

Hi Chenbo,

Thanks for your comments, please see replies inline.

Quoted 49 lines

-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Monday, September 26, 2022 2:07 PM
To: Ding, Xuan <[email protected]>; [email protected]
Cc: [email protected]; Hu, Jiayu <[email protected]>; He, Xingguang <[email protected]>; Yang, YvonneX <[email protected]>; Jiang, Cheng1 <[email protected]>; Wang, YuanX <[email protected]>; Ma, WenwuX <[email protected]>
Subject: RE: [PATCH v2 1/2] vhost: introduce DMA vchannel unconfiguration

Quoted 39 lines

-----Original Message-----
From: Ding, Xuan <[email protected]>
Sent: Tuesday, September 6, 2022 1:22 PM
To: [email protected]; Xia, Chenbo <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; He, Xingguang <[email protected]>; Yang, YvonneX <[email protected]>; Jiang,
Cheng1 <[email protected]>; Wang, YuanX <[email protected]>; Ma, WenwuX <[email protected]>; Ding, Xuan <[email protected]>
Subject: [PATCH v2 1/2] vhost: introduce DMA vchannel unconfiguration

From: Xuan Ding <[email protected]>

This patch adds a new API rte_vhost_async_dma_unconfigure() to unconfigure DMA vchannels in vhost async data path.

Lock protection are also added to protect DMA vchannels configuration and unconfiguration from concurrent calls.

---
 doc/guides/prog_guide/vhost_lib.rst    |  5 ++
 doc/guides/rel_notes/release_22_11.rst |  2 +
 lib/vhost/rte_vhost_async.h            | 17 +++++++
 lib/vhost/version.map                  |  3 ++
 lib/vhost/vhost.c                      | 69 ++++++++++++++++++++++++--
 5 files changed, 91 insertions(+), 5 deletions(-)
doc/guides/prog_guide/vhost_lib.rst +0 -0
index bad4d819e1..22764cbeaa 100644
@@ -323,6 +323,11 @@ The following is an overview of some key Vhost

API
functions:
Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET, VDPA_DEVICE_TYPE_BLK.

Quoted 6 lines

doc/guides/prog_guide/vhost_lib.rst The following is an overview of some key Vhost API functions:

Quoted 4 lines
doc/guides/prog_guide/vhost_lib.rst +4 -0
+* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)``
+
+  Clear DMA vChannels finished to use. This function needs to  be
+ called after the deregisterration of async path has been finished.

Deregistration

Thanks for your catch.

Quoted 14 lines
Quoted 14 lines
doc/guides/prog_guide/vhost_lib.rst +1 -0
+
 Vhost-user Implementations
 --------------------------
doc/guides/rel_notes/release_22_11.rst +0 -0
index 8c021cf050..e94c006e39 100644
@@ -55,6 +55,8 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
Quoted 4 lines

doc/guides/rel_notes/release_22_11.rst New Features · line 59

Quoted 2 lines
doc/guides/rel_notes/release_22_11.rst +2 -0
+* **Added vhost API to unconfigure DMA vchannels.**
+  Added an API which helps to unconfigure DMA vchannels.

Added XXX for async vhost

Good idea.

Quoted 2 lines

Overall LGTM. It seems it needs some rebasing too.

I'm preparing v3 patch series, please see next version.

Regards,
Xuan

Quoted 221 lines

Thanks,
Chenbo

Quoted 53 lines
doc/guides/rel_notes/release_22_11.rst +0 -0
 Removed Items
 -------------
lib/vhost/rte_vhost_async.h +3 -0
index 1db2a10124..0442e027fd 100644
@@ -266,6 +266,23 @@ rte_vhost_async_try_dequeue_burst(int vid,
 uint16_t queue_id,
 	struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t
 count,
 	int *nr_inflight, int16_t dma_id, uint16_t vchan_id);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior

notice.
+ *
+ * Unconfigure DMA vChannels in asynchronous data path.
+ *
+ * @param dma_id
+ * the identifier of DMA device
+ * @param vchan_id
+ * the identifier of virtual DMA channel
+ * @return
+ * 0 on success, and -1 on failure
+ */

lib/vhost/rte_vhost_async.h +4 -0
+__rte_experimental
+int
+rte_vhost_async_dma_unconfigure(int16_t dma_id, uint16_t vchan_id);
+
 #ifdef __cplusplus
 }
 #endif
lib/vhost/version.map index renamed from lib/vhost/version.map +0 -0

18574346d5..013a6bcc42 100644

lib/vhost/version.map +3 -0
@@ -96,6 +96,9 @@ EXPERIMENTAL {
 	rte_vhost_async_try_dequeue_burst;
 	rte_vhost_driver_get_vdpa_dev_type;
 	rte_vhost_clear_queue;
+
+	# added in 22.11
+	rte_vhost_async_dma_unconfigure;
 };
 
 INTERNAL {
lib/vhost/vhost.c index renamed from lib/vhost/vhost.c +0 -0

60cb05a0ff..273616da11 100644

lib/vhost/vhost.c line 29

Quoted 7 lines
lib/vhost/vhost.c +1 -0
@@ -23,6 +23,7 @@
 
 struct virtio_net *vhost_devices[RTE_MAX_VHOST_DEVICE];
 pthread_mutex_t vhost_dev_lock = PTHREAD_MUTEX_INITIALIZER;
+static rte_spinlock_t vhost_dma_lock = RTE_SPINLOCK_INITIALIZER;
 
 struct vhost_vq_stats_name_off {
 	char name[RTE_VHOST_STATS_NAME_SIZE]; @@ -1870,19 +1871,20

@@

lib/vhost/vhost.c rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id)

Quoted 7 lines

rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id)

lib/vhost/vhost.c +1 -0
 	void *pkts_cmpl_flag_addr;
 	uint16_t max_desc;
 
+	rte_spinlock_lock(&vhost_dma_lock);
 	if (!rte_dma_is_valid(dma_id)) {
 		VHOST_LOG_CONFIG("dma", ERR, "DMA %d is not found.\n",

dma_id);

lib/vhost/vhost.c rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id)

Quoted 6 lines
lib/vhost/vhost.c +1 -1
-		return -1;
+		goto error;
 	}
 
 	if (rte_dma_info_get(dma_id, &info) != 0) {
 		VHOST_LOG_CONFIG("dma", ERR, "Fail to get DMA %d

information.\n",

lib/vhost/vhost.c rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id)

Quoted 7 lines

dma_id);

lib/vhost/vhost.c +1 -1
-		return -1;
+		goto error;
 	}
 
 	if (vchan_id >= info.max_vchans) {
 		VHOST_LOG_CONFIG("dma", ERR, "Invalid DMA %d

vChannel %u.\n",

lib/vhost/vhost.c

Quoted 9 lines

dma_id, vchan_id);

lib/vhost/vhost.c +1 -1
-		return -1;
+		goto error;
 	}
 
 	if (!dma_copy_track[dma_id].vchans) { @@ -1894,7 +1896,7 @@

rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id)

lib/vhost/vhost.c +0 -0
 			VHOST_LOG_CONFIG("dma", ERR,
 				"Failed to allocate vchans for DMA %d

vChannel %u.\n",

lib/vhost/vhost.c rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id)

Quoted 6 lines
lib/vhost/vhost.c +1 -1
 				dma_id, vchan_id);
-			return -1;
+			goto error;
 		}
 
 		dma_copy_track[dma_id].vchans = vchans; @@ -1903,6

+1905,7 @@

Quoted 2 lines

rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id)
if (dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr)

{

Quoted 1 line

VHOST_LOG_CONFIG("dma", INFO, "DMA %d vChannel %u

already

Quoted 15 lines

registered.\n",

lib/vhost/vhost.c +2 -1
 			dma_id, vchan_id);
+		rte_spinlock_unlock(&vhost_dma_lock);
 		return 0;
 	}
@@ -1920,7 +1923,7 @@ rte_vhost_async_dma_configure(int16_t dma_id,
 uint16_t vchan_id)
 			rte_free(dma_copy_track[dma_id].vchans);
 			dma_copy_track[dma_id].vchans = NULL;
 		}
-		return -1;
+		goto error;
 	}
 
Quoted 4 lines

dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr =
pkts_cmpl_flag_addr; @@ -1928,7 +1931,12 @@ rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id)
dma_copy_track[dma_id].vchans[vchan_id].ring_mask = max_desc -

1;

Quoted 16 lines
lib/vhost/vhost.c +5 -0
 	dma_copy_track[dma_id].nr_vchans++;
 
+	rte_spinlock_unlock(&vhost_dma_lock);
 	return 0;
+
+error:
+	rte_spinlock_unlock(&vhost_dma_lock);
+	return -1;
 }
 
 int
@@ -2117,5 +2125,56 @@ int rte_vhost_vring_stats_reset(int vid,

uint16_t
queue_id)

lib/vhost/vhost.c +0 -0
 	return 0;
 }

lib/vhost/vhost.c int rte_vhost_vring_stats_reset(int vid, uint16_t queue_id)

Quoted 10 lines
lib/vhost/vhost.c +9 -0
 
+int
+rte_vhost_async_dma_unconfigure(int16_t dma_id, uint16_t vchan_id) {
+	struct rte_dma_info info;
+	uint16_t max_desc;
+	int i;
+
+	rte_spinlock_lock(&vhost_dma_lock);
+	if (!rte_dma_is_valid(dma_id)) {
+		VHOST_LOG_CONFIG("dma", ERR, "DMA %d is not found.\n",

dma_id);

lib/vhost/vhost.c int rte_vhost_vring_stats_reset(int vid, uint16_t queue_id)

Quoted 11 lines
lib/vhost/vhost.c +5 -0
+		goto error;
+	}
+
+	if (rte_dma_info_get(dma_id, &info) != 0) {
+		VHOST_LOG_CONFIG("dma", ERR, "Fail to get DMA %d

information.\n", dma_id);

lib/vhost/vhost.c +5 -0
+		goto error;
+	}
+
+	if (vchan_id >= info.max_vchans) {
+		VHOST_LOG_CONFIG("dma", ERR, "Invalid DMA %d

vChannel %u.\n",

Quoted 11 lines

dma_id, vchan_id);

lib/vhost/vhost.c +6 -0
+		goto error;
+	}
+
+	max_desc = info.max_desc;
+	for (i = 0; i < max_desc; i++) {
+		if

(dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr[i] != NULL) {
+

rte_free(dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag
_addr

lib/vhost/vhost.c int rte_vhost_vring_stats_reset(int vid, uint16_t queue_id)

Quoted 8 lines

[i]);
+
dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr[i] =
NULL;

lib/vhost/vhost.c +4 -0
+		}
+	}
+
+	if

(dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr !=

Quoted 3 lines

NULL) {
+

rte_free(dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag
_addr

Quoted 2 lines

);
+

dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr =

Quoted 22 lines

NULL;

lib/vhost/vhost.c +17 -0
+	}
+
+	if (dma_copy_track[dma_id].vchans != NULL) {
+		rte_free(dma_copy_track[dma_id].vchans);
+		dma_copy_track[dma_id].vchans = NULL;
+	}
+
+	dma_copy_track[dma_id].nr_vchans--;
+
+	rte_spinlock_unlock(&vhost_dma_lock);
+	return 0;
+
+error:
+	rte_spinlock_unlock(&vhost_dma_lock);
+	return -1;
+}
+
 RTE_LOG_REGISTER_SUFFIX(vhost_config_log_level, config, INFO);

RTE_LOG_REGISTER_SUFFIX(vhost_data_log_level, data, WARNING); --
2.17.1

#7 lore raw

PATCH v2 2/2 examples/vhost: unconfigure DMA vchannel

In reply to [email protected]

From: Xuan Ding <[email protected]>

This patch applies rte_vhost_async_dma_unconfigure() API to manually free 'dma_copy_track' array instead of waiting until the program ends to be released.

---
 examples/vhost/main.c | 7 +++++++
 1 file changed, 7 insertions(+)
examples/vhost/main.c +7 -0
index 7e1666f42a..1754d9ee27 100644
@@ -1624,6 +1624,13 @@ destroy_device(int vid)
 		dma_bind[vid].dmas[VIRTIO_TXQ].async_enabled = false;
 	}
 
+	for (i = 0; i < dma_count; i++) {
+		if (rte_vhost_async_dma_unconfigure(dmas_id[i], 0) < 0) {
+			RTE_LOG(ERR, VHOST_PORT, "Failed to unconfigure DMA in vhost.\n");
+			rte_exit(EXIT_FAILURE, "Cannot use given DMA device\n");
+		}
+	}
+
 	rte_free(vdev);
 }
 
Signature · 1 line
2.17.1
#8 lore raw

v3 cover vhost: introduce DMA vchannel unconfiguration

In reply to [email protected]

From: Xuan Ding <[email protected]>

This patchset introduces a new API rte_vhost_async_dma_unconfigure() to help user to manually free the DMA vchannel finished to use.

Note: this API should be called after async channel unregister.

v3:
* Rebase to latest DPDK.
* Refine some descriptions in the doc.
* Fix one bug in the vhost example.

v2:
* Add spinlock protection.
* Fix a memory leak issue.
* Refine the doc.

Xuan Ding (2):
vhost: introduce DMA vchannel unconfiguration
examples/vhost: unconfigure DMA vchannel

 doc/guides/prog_guide/vhost_lib.rst    |  6 +++
 doc/guides/rel_notes/release_22_11.rst |  3 ++
 examples/vhost/main.c                  | 45 ++++++++++++-----
 examples/vhost/main.h                  |  1 +
 lib/vhost/rte_vhost_async.h            | 18 +++++++
 lib/vhost/version.map                  |  3 ++
 lib/vhost/vhost.c                      | 69 ++++++++++++++++++++++++--
 7 files changed, 127 insertions(+), 18 deletions(-)
Signature · 1 line
2.17.1
#9 lore raw

PATCH v3 1/2 vhost: introduce DMA vchannel unconfiguration

From: Xuan Ding <[email protected]>

This patch adds a new API rte_vhost_async_dma_unconfigure() to unconfigure DMA vchannels in vhost async data path.

Lock protection are also added to protect DMA vchannels configuration and unconfiguration from concurrent calls.

---
 doc/guides/prog_guide/vhost_lib.rst    |  6 +++
 doc/guides/rel_notes/release_22_11.rst |  3 ++
 lib/vhost/rte_vhost_async.h            | 18 +++++++
 lib/vhost/version.map                  |  3 ++
 lib/vhost/vhost.c                      | 69 ++++++++++++++++++++++++--
 5 files changed, 94 insertions(+), 5 deletions(-)
doc/guides/prog_guide/vhost_lib.rst +6 -0
index bad4d819e1..d3cef978d0 100644
@@ -323,6 +323,12 @@ The following is an overview of some key Vhost API functions:
   Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET,
   VDPA_DEVICE_TYPE_BLK.
 
+* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)``
+
+  Clean DMA vChannels finished to use. This function needs to
+  be called after the deregistration of async DMA vchannel
+  has been finished.
+
 Vhost-user Implementations
 --------------------------
 
doc/guides/rel_notes/release_22_11.rst +3 -0
index 684bf74596..a641d8a6b8 100644
@@ -67,6 +67,9 @@ New Features
 
   * Added support to set device link down/up.
 
+* **Added DMA vChannel unconfiguration for async vhost.**
+
+  * Added support to unconfigure DMA vChannels that have been unregistered.
 
 Removed Items
 -------------
lib/vhost/rte_vhost_async.h +18 -0
index 1db2a10124..6ee4f7258d 100644
@@ -266,6 +266,24 @@ rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id,
 	struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count,
 	int *nr_inflight, int16_t dma_id, uint16_t vchan_id);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice.
+ *
+ * Unconfigure DMA vChannels in asynchronous data path.
+ * This function should be called after the DMA vChannel has been unregistered.
+ *
+ * @param dma_id
+ *  the identifier of DMA device
+ * @param vchan_id
+ *  the identifier of virtual DMA channel
+ * @return
+ *  0 on success, and -1 on failure
+ */
+__rte_experimental
+int
+rte_vhost_async_dma_unconfigure(int16_t dma_id, uint16_t vchan_id);
+
 #ifdef __cplusplus
 }
 #endif
lib/vhost/version.map +3 -0
index 18574346d5..013a6bcc42 100644
@@ -96,6 +96,9 @@ EXPERIMENTAL {
 	rte_vhost_async_try_dequeue_burst;
 	rte_vhost_driver_get_vdpa_dev_type;
 	rte_vhost_clear_queue;
+
+	# added in 22.11
+	rte_vhost_async_dma_unconfigure;
 };
 
 INTERNAL {
lib/vhost/vhost.c +64 -5

121 lines not on this page · raw message

Signature · 1 line
2.17.1
#10 lore raw

PATCH v3 2/2 examples/vhost: unconfigure DMA vchannel

In reply to [email protected]

From: Xuan Ding <[email protected]>

This patch applies rte_vhost_async_dma_unconfigure() API to manually free DMA vchannels instead of waiting until the program ends to be released.

---
 examples/vhost/main.c | 45 ++++++++++++++++++++++++++++++-------------
 examples/vhost/main.h |  1 +
 2 files changed, 33 insertions(+), 13 deletions(-)
examples/vhost/main.c +32 -13
index 0fa4753e70..32f396d88a 100644
@@ -1558,6 +1558,28 @@ vhost_clear_queue(struct vhost_dev *vdev, uint16_t queue_id)
 	}
 }
 
+static void
+vhost_clear_async(struct vhost_dev *vdev, int vid, uint16_t queue_id)
+{
+	int16_t dma_id;
+	uint16_t ref_count;
+
+	if (dma_bind[vid].dmas[queue_id].async_enabled) {
+		vhost_clear_queue(vdev, queue_id);
+		rte_vhost_async_channel_unregister(vid, queue_id);
+		dma_bind[vid].dmas[queue_id].async_enabled = false;
+	}
+
+	dma_id = dma_bind[vid2socketid[vdev->vid]].dmas[queue_id].dev_id;
+	dma_bind[vid2socketid[vdev->vid]].dmas[queue_id].dma_ref_count--;
+	ref_count = dma_bind[vid2socketid[vdev->vid]].dmas[queue_id].dma_ref_count;
+
+	if (ref_count == 0 && dma_id != INVALID_DMA_ID) {
+		if (rte_vhost_async_dma_unconfigure(dma_id, 0) < 0)
+			RTE_LOG(ERR, VHOST_PORT, "Failed to unconfigure DMA in vhost.\n");
+	}
+}
+
 /*
  * Remove a device from the specific data core linked list and from the
  * main linked list. Synchronization  occurs through the use of the
@@ -1614,17 +1636,8 @@ destroy_device(int vid)
 		"(%d) device has been removed from data core\n",
 		vdev->vid);
 
-	if (dma_bind[vid].dmas[VIRTIO_RXQ].async_enabled) {
-		vhost_clear_queue(vdev, VIRTIO_RXQ);
-		rte_vhost_async_channel_unregister(vid, VIRTIO_RXQ);
-		dma_bind[vid].dmas[VIRTIO_RXQ].async_enabled = false;
-	}
-
-	if (dma_bind[vid].dmas[VIRTIO_TXQ].async_enabled) {
-		vhost_clear_queue(vdev, VIRTIO_TXQ);
-		rte_vhost_async_channel_unregister(vid, VIRTIO_TXQ);
-		dma_bind[vid].dmas[VIRTIO_TXQ].async_enabled = false;
-	}
+	vhost_clear_async(vdev, vid, VIRTIO_RXQ);
+	vhost_clear_async(vdev, vid, VIRTIO_TXQ);
 
 	rte_free(vdev);
 }
@@ -1673,14 +1686,19 @@ vhost_async_channel_register(int vid)
 
 	if (dma_bind[vid2socketid[vid]].dmas[VIRTIO_RXQ].dev_id != INVALID_DMA_ID) {
 		rx_ret = rte_vhost_async_channel_register(vid, VIRTIO_RXQ);
-		if (rx_ret == 0)
+		if (rx_ret == 0) {
 			dma_bind[vid2socketid[vid]].dmas[VIRTIO_RXQ].async_enabled = true;
+			dma_bind[vid2socketid[vid]].dmas[VIRTIO_RXQ].dma_ref_count++;
+		}
+
 	}
 
 	if (dma_bind[vid2socketid[vid]].dmas[VIRTIO_TXQ].dev_id != INVALID_DMA_ID) {
 		tx_ret = rte_vhost_async_channel_register(vid, VIRTIO_TXQ);
-		if (tx_ret == 0)
+		if (tx_ret == 0) {
 			dma_bind[vid2socketid[vid]].dmas[VIRTIO_TXQ].async_enabled = true;
+			dma_bind[vid2socketid[vid]].dmas[VIRTIO_TXQ].dma_ref_count++;
+		}
 	}
 
 	return rx_ret | tx_ret;
@@ -1886,6 +1904,7 @@ reset_dma(void)
 		for (j = 0; j < RTE_MAX_QUEUES_PER_PORT * 2; j++) {
 			dma_bind[i].dmas[j].dev_id = INVALID_DMA_ID;
 			dma_bind[i].dmas[j].async_enabled = false;
+			dma_bind[i].dmas[j].dma_ref_count = 0;
 		}
 	}
 
examples/vhost/main.h +1 -0
index 2fcb8376c5..2b2cf828d3 100644
@@ -96,6 +96,7 @@ struct dma_info {
 	struct rte_pci_addr addr;
 	int16_t dev_id;
 	bool async_enabled;
+	uint16_t dma_ref_count;
 };
 
 struct dma_for_vhost {
Signature · 1 line
2.17.1
#11 lore raw

RE: [PATCH v3 2/2] examples/vhost: unconfigure DMA vchannel

Quoted 37 lines

-----Original Message-----
From: Ding, Xuan <[email protected]>
Sent: Thursday, September 29, 2022 9:33 AM
To: [email protected]; Xia, Chenbo <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; He, Xingguang <[email protected]>; Yang, YvonneX <[email protected]>; Jiang, Cheng1 <[email protected]>; Wang, YuanX <[email protected]>; Ma, WenwuX <[email protected]>; Ding, Xuan <[email protected]>
Subject: [PATCH v3 2/2] examples/vhost: unconfigure DMA vchannel

From: Xuan Ding <[email protected]>

This patch applies rte_vhost_async_dma_unconfigure() API to manually free DMA vchannels instead of waiting until the program ends to be released.

---
 examples/vhost/main.c | 45 ++++++++++++++++++++++++++++++-------------
 examples/vhost/main.h |  1 +
 2 files changed, 33 insertions(+), 13 deletions(-)
examples/vhost/main.c +6 -0
index 0fa4753e70..32f396d88a 100644
@@ -1558,6 +1558,28 @@ vhost_clear_queue(struct vhost_dev *vdev, uint16_t
 queue_id)
 	}
 }
 
+static void
+vhost_clear_async(struct vhost_dev *vdev, int vid, uint16_t queue_id)
+{
+	int16_t dma_id;
+	uint16_t ref_count;
+

examples/vhost/main.c vhost_clear_queue(struct vhost_dev *vdev, uint16_t · line 1577

Quoted 10 lines
examples/vhost/main.c +9 -0
+	if (dma_bind[vid].dmas[queue_id].async_enabled) {
+		vhost_clear_queue(vdev, queue_id);
+		rte_vhost_async_channel_unregister(vid, queue_id);
+		dma_bind[vid].dmas[queue_id].async_enabled = false;
+	}
+
+	dma_id = dma_bind[vid2socketid[vdev->vid]].dmas[queue_id].dev_id;
+	dma_bind[vid2socketid[vdev->vid]].dmas[queue_id].dma_ref_count--;
+	ref_count = dma_bind[vid2socketid[vdev-
 >vid]].dmas[queue_id].dma_ref_count;

Above two lines could be combined like 'ref_count = --XXXX'

But I wonder is this dma_ref_count really needed? It's always 0/1 like Async_enabled, no? If it target at multi-port sharing same one dma case, it should not be implemented like this.

Thanks,
Chenbo

Quoted 81 lines
examples/vhost/main.c +7 -0
+
+	if (ref_count == 0 && dma_id != INVALID_DMA_ID) {
+		if (rte_vhost_async_dma_unconfigure(dma_id, 0) < 0)
+			RTE_LOG(ERR, VHOST_PORT, "Failed to unconfigure DMA in
 vhost.\n");
+	}
+}
+

/*
* Remove a device from the specific data core linked list and from the
* main linked list. Synchronization occurs through the use of the

examples/vhost/main.c +7 -13
@@ -1614,17 +1636,8 @@ destroy_device(int vid)
 		"(%d) device has been removed from data core\n",
 		vdev->vid);
 
-	if (dma_bind[vid].dmas[VIRTIO_RXQ].async_enabled) {
-		vhost_clear_queue(vdev, VIRTIO_RXQ);
-		rte_vhost_async_channel_unregister(vid, VIRTIO_RXQ);
-		dma_bind[vid].dmas[VIRTIO_RXQ].async_enabled = false;
-	}
-
-	if (dma_bind[vid].dmas[VIRTIO_TXQ].async_enabled) {
-		vhost_clear_queue(vdev, VIRTIO_TXQ);
-		rte_vhost_async_channel_unregister(vid, VIRTIO_TXQ);
-		dma_bind[vid].dmas[VIRTIO_TXQ].async_enabled = false;
-	}
+	vhost_clear_async(vdev, vid, VIRTIO_RXQ);
+	vhost_clear_async(vdev, vid, VIRTIO_TXQ);
 
 	rte_free(vdev);
 }
@@ -1673,14 +1686,19 @@ vhost_async_channel_register(int vid)
 
 	if (dma_bind[vid2socketid[vid]].dmas[VIRTIO_RXQ].dev_id !=
 INVALID_DMA_ID) {
 		rx_ret = rte_vhost_async_channel_register(vid, VIRTIO_RXQ);
-		if (rx_ret == 0)
+		if (rx_ret == 0) {
 
 	dma_bind[vid2socketid[vid]].dmas[VIRTIO_RXQ].async_enabled = true;
+
 	dma_bind[vid2socketid[vid]].dmas[VIRTIO_RXQ].dma_ref_count++;
+		}
+
 	}
 
 	if (dma_bind[vid2socketid[vid]].dmas[VIRTIO_TXQ].dev_id !=
 INVALID_DMA_ID) {
 		tx_ret = rte_vhost_async_channel_register(vid, VIRTIO_TXQ);
-		if (tx_ret == 0)
+		if (tx_ret == 0) {
	dma_bind[vid2socketid[vid]].dmas[VIRTIO_TXQ].async_enabled = true;
+
	dma_bind[vid2socketid[vid]].dmas[VIRTIO_TXQ].dma_ref_count++;
examples/vhost/main.c +2 -0
+		}
 	}
 
 	return rx_ret | tx_ret;
@@ -1886,6 +1904,7 @@ reset_dma(void)
 		for (j = 0; j < RTE_MAX_QUEUES_PER_PORT * 2; j++) {
 			dma_bind[i].dmas[j].dev_id = INVALID_DMA_ID;
 			dma_bind[i].dmas[j].async_enabled = false;
+			dma_bind[i].dmas[j].dma_ref_count = 0;
 		}
 	}
 
examples/vhost/main.h +1 -1
index 2fcb8376c5..2b2cf828d3 100644
@@ -96,6 +96,7 @@ struct dma_info {
 	struct rte_pci_addr addr;
 	int16_t dev_id;
 	bool async_enabled;
+	uint16_t dma_ref_count;
 };
 
 struct dma_for_vhost {
--

2.17.1

author
#12 lore raw

RE: [PATCH v3 2/2] examples/vhost: unconfigure DMA vchannel

Hi Chenbo,

Quoted 63 lines

-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Thursday, September 29, 2022 4:28 PM
To: Ding, Xuan <[email protected]>; [email protected]
Cc: [email protected]; Hu, Jiayu <[email protected]>; He, Xingguang <[email protected]>; Yang, YvonneX <[email protected]>; Jiang, Cheng1 <[email protected]>; Wang, YuanX <[email protected]>; Ma, WenwuX <[email protected]>
Subject: RE: [PATCH v3 2/2] examples/vhost: unconfigure DMA vchannel

Quoted 8 lines

-----Original Message-----
From: Ding, Xuan <[email protected]>
Sent: Thursday, September 29, 2022 9:33 AM
To: [email protected]; Xia, Chenbo <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; He, Xingguang <[email protected]>; Yang, YvonneX <[email protected]>; Jiang,
Cheng1 <[email protected]>; Wang, YuanX

<[email protected]>;

Quoted 1 line

Ma, WenwuX <[email protected]>; Ding, Xuan

<[email protected]>

Quoted 42 lines

Subject: [PATCH v3 2/2] examples/vhost: unconfigure DMA vchannel

From: Xuan Ding <[email protected]>

This patch applies rte_vhost_async_dma_unconfigure() API to manually free DMA vchannels instead of waiting until the program ends to be released.

---
 examples/vhost/main.c | 45
++++++++++++++++++++++++++++++-------------
 examples/vhost/main.h |  1 +
 2 files changed, 33 insertions(+), 13 deletions(-)
examples/vhost/main.c index renamed from examples/vhost/main.c +0 -0

0fa4753e70..32f396d88a 100644

examples/vhost/main.c +0 -0
@@ -1558,6 +1558,28 @@ vhost_clear_queue(struct vhost_dev *vdev,

uint16_t
queue_id)

examples/vhost/main.c +13 -0
 	}
 }
 
+static void
+vhost_clear_async(struct vhost_dev *vdev, int vid, uint16_t queue_id)
+{
+	int16_t dma_id;
+	uint16_t ref_count;
+
+	if (dma_bind[vid].dmas[queue_id].async_enabled) {
+		vhost_clear_queue(vdev, queue_id);
+		rte_vhost_async_channel_unregister(vid, queue_id);
+		dma_bind[vid].dmas[queue_id].async_enabled = false;
+	}
+
+	dma_id = dma_bind[vid2socketid[vdev-

vid]].dmas[queue_id].dev_id;

examples/vhost/main.c +1 -0
+	dma_bind[vid2socketid[vdev-

vid]].dmas[queue_id].dma_ref_count--;

examples/vhost/main.c +1 -0
+	ref_count = dma_bind[vid2socketid[vdev-
Quoted 7 lines
Quoted 1 line
Quoted 1 line

vid]].dmas[queue_id].dma_ref_count;

Above two lines could be combined like 'ref_count = --XXXX'

But I wonder is this dma_ref_count really needed? It's always 0/1 like Async_enabled, no? If it target at multi-port sharing same one dma case, it should not be implemented like this.

Yes, the dma_ref_count here is for multi-port sharing same one dma case. The intention was to unconfigure dma device when it is no longer used by any ports.
I will check the implementation.

Thanks,
Xuan

Quoted 90 lines

Thanks,
Chenbo

examples/vhost/main.c vhost_clear_queue(struct vhost_dev *vdev, uint16_t queue_id)

Quoted 4 lines
examples/vhost/main.c +4 -0
+
+	if (ref_count == 0 && dma_id != INVALID_DMA_ID) {
+		if (rte_vhost_async_dma_unconfigure(dma_id, 0) < 0)
+			RTE_LOG(ERR, VHOST_PORT, "Failed to unconfigure

DMA in

Quoted 28 lines

vhost.\n");

examples/vhost/main.c +5 -11
+	}
+}
+
 /*
  * Remove a device from the specific data core linked list and from the
  * main linked list. Synchronization  occurs through the use of the
@@ -1614,17 +1636,8 @@ destroy_device(int vid)
 		"(%d) device has been removed from data core\n",
 		vdev->vid);
 
-	if (dma_bind[vid].dmas[VIRTIO_RXQ].async_enabled) {
-		vhost_clear_queue(vdev, VIRTIO_RXQ);
-		rte_vhost_async_channel_unregister(vid, VIRTIO_RXQ);
-		dma_bind[vid].dmas[VIRTIO_RXQ].async_enabled = false;
-	}
-
-	if (dma_bind[vid].dmas[VIRTIO_TXQ].async_enabled) {
-		vhost_clear_queue(vdev, VIRTIO_TXQ);
-		rte_vhost_async_channel_unregister(vid, VIRTIO_TXQ);
-		dma_bind[vid].dmas[VIRTIO_TXQ].async_enabled = false;
-	}
+	vhost_clear_async(vdev, vid, VIRTIO_RXQ);
+	vhost_clear_async(vdev, vid, VIRTIO_TXQ);
 
 	rte_free(vdev);
 }

examples/vhost/main.c vhost_async_channel_register(int vid) · line 1689

Quoted 4 lines
examples/vhost/main.c +0 -0
@@ -1673,14 +1686,19 @@ vhost_async_channel_register(int vid)
 
 	if (dma_bind[vid2socketid[vid]].dmas[VIRTIO_RXQ].dev_id !=
 INVALID_DMA_ID) {
 		rx_ret = rte_vhost_async_channel_register(vid,

VIRTIO_RXQ);

examples/vhost/main.c vhost_async_channel_register(int vid)

Quoted 4 lines
examples/vhost/main.c +1 -1
-		if (rx_ret == 0)
+		if (rx_ret == 0) {
 
 	dma_bind[vid2socketid[vid]].dmas[VIRTIO_RXQ].async_enabled =

true;

examples/vhost/main.c vhost_async_channel_register(int vid) · line 1701

Quoted 9 lines
examples/vhost/main.c +3 -0
+
 	dma_bind[vid2socketid[vid]].dmas[VIRTIO_RXQ].dma_ref_count++;
+		}
+
 	}
 
 	if (dma_bind[vid2socketid[vid]].dmas[VIRTIO_TXQ].dev_id !=
 INVALID_DMA_ID) {
 		tx_ret = rte_vhost_async_channel_register(vid,

VIRTIO_TXQ);

Quoted 4 lines
examples/vhost/main.c +1 -1
-		if (tx_ret == 0)
+		if (tx_ret == 0) {

dma_bind[vid2socketid[vid]].dmas[VIRTIO_TXQ].async_enabled =

true;

Quoted 28 lines

+

	dma_bind[vid2socketid[vid]].dmas[VIRTIO_TXQ].dma_ref_count++;
examples/vhost/main.c +2 -0
+		}
 	}
 
 	return rx_ret | tx_ret;
@@ -1886,6 +1904,7 @@ reset_dma(void)
 		for (j = 0; j < RTE_MAX_QUEUES_PER_PORT * 2; j++) {
 			dma_bind[i].dmas[j].dev_id = INVALID_DMA_ID;
 			dma_bind[i].dmas[j].async_enabled = false;
+			dma_bind[i].dmas[j].dma_ref_count = 0;
 		}
 	}
 
examples/vhost/main.h index renamed from examples/vhost/main.h +0 -0

2fcb8376c5..2b2cf828d3 100644

examples/vhost/main.h +1 -1
@@ -96,6 +96,7 @@ struct dma_info {
 	struct rte_pci_addr addr;
 	int16_t dev_id;
 	bool async_enabled;
+	uint16_t dma_ref_count;
 };
 
 struct dma_for_vhost {
--

2.17.1

#13 lore raw

v4 cover vhost: introduce DMA vchannel unconfiguration

In reply to [email protected]

From: Xuan Ding <[email protected]>

This patchset introduces a new API rte_vhost_async_dma_unconfigure() to help user to manually free DMA vchannels finished to use.

Note: this API should be called after async channel unregister.

v4:
* Rebase to 22.11 rc1.
* Fix the usage of 'dma_ref_count' to make sure the specified DMA device is not used by any vhost ports before unconfiguration.

v3:
* Rebase to latest DPDK.
* Refine some descriptions in the doc.
* Fix one bug in the vhost example.

v2:
* Add spinlock protection.
* Fix a memory leak issue.
* Refine the doc.

Xuan Ding (2):
vhost: introduce DMA vchannel unconfiguration
examples/vhost: unconfigure DMA vchannel

 doc/guides/prog_guide/vhost_lib.rst    |  6 +++
 doc/guides/rel_notes/release_22_11.rst |  4 ++
 examples/vhost/main.c                  | 38 +++++++++-----
 lib/vhost/rte_vhost_async.h            | 18 +++++++
 lib/vhost/version.map                  |  3 ++
 lib/vhost/vhost.c                      | 69 ++++++++++++++++++++++++--
 6 files changed, 120 insertions(+), 18 deletions(-)
Signature · 1 line
2.17.1
#14 lore raw

PATCH v4 1/2 vhost: introduce DMA vchannel unconfiguration

From: Xuan Ding <[email protected]>

Add a new API rte_vhost_async_dma_unconfigure() to unconfigure DMA vchannels in vhost async data path. Lock protection are also added to protect DMA vchannels configuration and unconfiguration from concurrent calls.

---
 doc/guides/prog_guide/vhost_lib.rst    |  6 +++
 doc/guides/rel_notes/release_22_11.rst |  4 ++
 lib/vhost/rte_vhost_async.h            | 18 +++++++
 lib/vhost/version.map                  |  3 ++
 lib/vhost/vhost.c                      | 69 ++++++++++++++++++++++++--
 5 files changed, 95 insertions(+), 5 deletions(-)
doc/guides/prog_guide/vhost_lib.rst +6 -0
index bad4d819e1..d3cef978d0 100644
@@ -323,6 +323,12 @@ The following is an overview of some key Vhost API functions:
   Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET,
   VDPA_DEVICE_TYPE_BLK.
 
+* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)``
+
+  Clean DMA vChannels finished to use. This function needs to
+  be called after the deregistration of async DMA vchannel
+  has been finished.
+
 Vhost-user Implementations
 --------------------------
 
doc/guides/rel_notes/release_22_11.rst +4 -0
index 2da8bc9661..3be150122c 100644
@@ -236,6 +236,10 @@ New Features
 
      strings $dpdk_binary_or_driver | sed -n 's/^PMD_INFO_STRING= //p'
 
+* **Added DMA vChannel unconfiguration for async vhost.**
+
+  * Added support to unconfigure DMA vChannels that have been unregistered.
+
 
 Removed Items
 -------------
lib/vhost/rte_vhost_async.h +18 -0
index 1db2a10124..6ee4f7258d 100644
@@ -266,6 +266,24 @@ rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id,
 	struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count,
 	int *nr_inflight, int16_t dma_id, uint16_t vchan_id);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice.
+ *
+ * Unconfigure DMA vChannels in asynchronous data path.
+ * This function should be called after the DMA vChannel has been unregistered.
+ *
+ * @param dma_id
+ *  the identifier of DMA device
+ * @param vchan_id
+ *  the identifier of virtual DMA channel
+ * @return
+ *  0 on success, and -1 on failure
+ */
+__rte_experimental
+int
+rte_vhost_async_dma_unconfigure(int16_t dma_id, uint16_t vchan_id);
+
 #ifdef __cplusplus
 }
 #endif
lib/vhost/version.map +3 -0
index 7a00b65740..0b61870870 100644
@@ -94,6 +94,9 @@ EXPERIMENTAL {
 	rte_vhost_async_try_dequeue_burst;
 	rte_vhost_driver_get_vdpa_dev_type;
 	rte_vhost_clear_queue;
+
+	# added in 22.11
+	rte_vhost_async_dma_unconfigure;
 };
 
 INTERNAL {
lib/vhost/vhost.c +64 -5

121 lines not on this page · raw message

Signature · 1 line
2.17.1
#15 lore raw
Quoted 96 lines · On 10/13/22 08:40, [email protected] wrote:

From: Xuan Ding <[email protected]>

Add a new API rte_vhost_async_dma_unconfigure() to unconfigure DMA vchannels in vhost async data path. Lock protection are also added to protect DMA vchannels configuration and unconfiguration from concurrent calls.

---
  doc/guides/prog_guide/vhost_lib.rst    |  6 +++
  doc/guides/rel_notes/release_22_11.rst |  4 ++
  lib/vhost/rte_vhost_async.h            | 18 +++++++
  lib/vhost/version.map                  |  3 ++
  lib/vhost/vhost.c                      | 69 ++++++++++++++++++++++++--
  5 files changed, 95 insertions(+), 5 deletions(-)
doc/guides/prog_guide/vhost_lib.rst +6 -0
index bad4d819e1..d3cef978d0 100644
@@ -323,6 +323,12 @@ The following is an overview of some key Vhost API functions:
    Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET,
    VDPA_DEVICE_TYPE_BLK.
 
+* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)``
+
+  Clean DMA vChannels finished to use. This function needs to
+  be called after the deregistration of async DMA vchannel
+  has been finished.
+
  Vhost-user Implementations
  --------------------------
 
doc/guides/rel_notes/release_22_11.rst +4 -0
index 2da8bc9661..3be150122c 100644
@@ -236,6 +236,10 @@ New Features
 
       strings $dpdk_binary_or_driver | sed -n 's/^PMD_INFO_STRING= //p'
 
+* **Added DMA vChannel unconfiguration for async vhost.**
+
+  * Added support to unconfigure DMA vChannels that have been unregistered.
+
 
  Removed Items
  -------------
lib/vhost/rte_vhost_async.h +18 -0
index 1db2a10124..6ee4f7258d 100644
@@ -266,6 +266,24 @@ rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id,
  	struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count,
  	int *nr_inflight, int16_t dma_id, uint16_t vchan_id);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice.
+ *
+ * Unconfigure DMA vChannels in asynchronous data path.
+ * This function should be called after the DMA vChannel has been unregistered.
+ *
+ * @param dma_id
+ *  the identifier of DMA device
+ * @param vchan_id
+ *  the identifier of virtual DMA channel
+ * @return
+ *  0 on success, and -1 on failure
+ */
+__rte_experimental
+int
+rte_vhost_async_dma_unconfigure(int16_t dma_id, uint16_t vchan_id);
+
  #ifdef __cplusplus
  }
  #endif
lib/vhost/version.map +3 -0
index 7a00b65740..0b61870870 100644
@@ -94,6 +94,9 @@ EXPERIMENTAL {
  	rte_vhost_async_try_dequeue_burst;
  	rte_vhost_driver_get_vdpa_dev_type;
  	rte_vhost_clear_queue;
+
+	# added in 22.11
+	rte_vhost_async_dma_unconfigure;
  };
 
  INTERNAL {
lib/vhost/vhost.c +0 -0
index 8740aa2788..9fbc56229a 100644

lib/vhost/vhost.c line 26

Quoted 4 lines
lib/vhost/vhost.c +1 -0
@@ -23,6 +23,7 @@
 
  struct virtio_net *vhost_devices[RTE_MAX_VHOST_DEVICE];
  pthread_mutex_t vhost_dev_lock = PTHREAD_MUTEX_INITIALIZER;
+static rte_spinlock_t vhost_dma_lock = RTE_SPINLOCK_INITIALIZER;

I think a mutex would be more appropiate.

Thanks,
Maxime

author
#16 lore raw

Hi Maxime,

Quoted 118 lines

-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Thursday, October 13, 2022 4:02 PM
To: Ding, Xuan <[email protected]>; Xia, Chenbo <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; He, Xingguang <[email protected]>; Yang, YvonneX <[email protected]>; Jiang, Cheng1 <[email protected]>; Wang, YuanX <[email protected]>; Ma, WenwuX <[email protected]>
Subject: Re: [PATCH v4 1/2] vhost: introduce DMA vchannel unconfiguration

doc/guides/prog_guide/vhost_lib.rst The following is an overview of some key Vhost

Quoted 22 lines · On 10/13/22 08:40, [email protected] wrote:

From: Xuan Ding <[email protected]>

Add a new API rte_vhost_async_dma_unconfigure() to unconfigure DMA vchannels in vhost async data path. Lock protection are also added to protect DMA vchannels configuration and unconfiguration from concurrent calls.

---
  doc/guides/prog_guide/vhost_lib.rst    |  6 +++
  doc/guides/rel_notes/release_22_11.rst |  4 ++
  lib/vhost/rte_vhost_async.h            | 18 +++++++
  lib/vhost/version.map                  |  3 ++
  lib/vhost/vhost.c                      | 69 ++++++++++++++++++++++++--
  5 files changed, 95 insertions(+), 5 deletions(-)
doc/guides/prog_guide/vhost_lib.rst +0 -0
index bad4d819e1..d3cef978d0 100644
@@ -323,6 +323,12 @@ The following is an overview of some key Vhost

API functions:

Quoted 18 lines
doc/guides/prog_guide/vhost_lib.rst +6 -0
    Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET,
    VDPA_DEVICE_TYPE_BLK.
 
+* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)``
+
+  Clean DMA vChannels finished to use. This function needs to  be
+ called after the deregistration of async DMA vchannel  has been
+ finished.
+
  Vhost-user Implementations
  --------------------------
 
doc/guides/rel_notes/release_22_11.rst +0 -0
index 2da8bc9661..3be150122c 100644

doc/guides/rel_notes/release_22_11.rst New Features · line 241

Quoted 6 lines
doc/guides/rel_notes/release_22_11.rst +3 -0
@@ -236,6 +236,10 @@ New Features
 
       strings $dpdk_binary_or_driver | sed -n 's/^PMD_INFO_STRING= //p'
 
+* **Added DMA vChannel unconfiguration for async vhost.**
+
+  * Added support to unconfigure DMA vChannels that have been

unregistered.

lib/vhost/rte_vhost_async.h rte_vhost_async_try_dequeue_burst(int vid,

Quoted 9 lines
doc/guides/rel_notes/release_22_11.rst +1 -0
+
 
  Removed Items
  -------------
lib/vhost/rte_vhost_async.h +0 -0
index 1db2a10124..6ee4f7258d 100644
@@ -266,6 +266,24 @@ rte_vhost_async_try_dequeue_burst(int vid,

uint16_t queue_id,

lib/vhost/rte_vhost_async.h rte_vhost_async_try_dequeue_burst(int vid, · line 266

Quoted 1 line
lib/vhost/rte_vhost_async.h +0 -0
  	struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t

count,

lib/vhost/rte_vhost_async.h rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id,

Quoted 5 lines
lib/vhost/rte_vhost_async.h +3 -0
  	int *nr_inflight, int16_t dma_id, uint16_t vchan_id);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior

notice.

lib/vhost/rte_vhost_async.h rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id,

Quoted 3 lines
lib/vhost/rte_vhost_async.h +3 -0
+ *
+ * Unconfigure DMA vChannels in asynchronous data path.
+ * This function should be called after the DMA vChannel has been

unregistered.

Quoted 35 lines
lib/vhost/rte_vhost_async.h +12 -0
+ *
+ * @param dma_id
+ *  the identifier of DMA device
+ * @param vchan_id
+ *  the identifier of virtual DMA channel
+ * @return
+ *  0 on success, and -1 on failure
+ */
+__rte_experimental
+int
+rte_vhost_async_dma_unconfigure(int16_t dma_id, uint16_t vchan_id);
+
  #ifdef __cplusplus
  }
  #endif
lib/vhost/version.map index renamed from lib/vhost/version.map +0 -0

7a00b65740..0b61870870 100644

lib/vhost/version.map +3 -0
@@ -94,6 +94,9 @@ EXPERIMENTAL {
  	rte_vhost_async_try_dequeue_burst;
  	rte_vhost_driver_get_vdpa_dev_type;
  	rte_vhost_clear_queue;
+
+	# added in 22.11
+	rte_vhost_async_dma_unconfigure;
  };
 
  INTERNAL {
lib/vhost/vhost.c index renamed from lib/vhost/vhost.c +0 -0

8740aa2788..9fbc56229a 100644

lib/vhost/vhost.c +0 -0
@@ -23,6 +23,7 @@
 
Quoted 5 lines

lib/vhost/vhost.c line 26

Quoted 3 lines
lib/vhost/vhost.c +1 -0
  struct virtio_net *vhost_devices[RTE_MAX_VHOST_DEVICE];
  pthread_mutex_t vhost_dev_lock = PTHREAD_MUTEX_INITIALIZER;
+static rte_spinlock_t vhost_dma_lock = RTE_SPINLOCK_INITIALIZER;

I think a mutex would be more appropiate.

Thanks for your suggestion.
Maybe mutex is more efficient, I will switch to mutex in next version.

Regards,
Xuan

Quoted 3 lines

Thanks,
Maxime

#17 lore raw

PATCH v4 2/2 examples/vhost: unconfigure DMA vchannel

In reply to [email protected]

From: Xuan Ding <[email protected]>

This patch applies rte_vhost_async_dma_unconfigure() to manually free DMA vchannels. Before unconfiguration, need make sure the specified DMA device is no longer used by any vhost ports.

---
 examples/vhost/main.c | 38 +++++++++++++++++++++++++-------------
 1 file changed, 25 insertions(+), 13 deletions(-)
examples/vhost/main.c +25 -13
index ac78704d79..bfeb808dcc 100644
@@ -73,6 +73,7 @@ static int total_num_mbufs = NUM_MBUFS_DEFAULT;
 
 struct dma_for_vhost dma_bind[RTE_MAX_VHOST_DEVICE];
 int16_t dmas_id[RTE_DMADEV_DEFAULT_MAX];
+int16_t dma_ref_count[RTE_DMADEV_DEFAULT_MAX];
 static int dma_count;
 
 /* mask of enabled ports */
@@ -371,6 +372,7 @@ open_dma(const char *value)
 done:
 		(dma_info + socketid)->dmas[vring_id].dev_id = dev_id;
 		(dma_info + socketid)->async_flag |= async_flag;
+		dma_ref_count[dev_id]++;
 		i++;
 	}
 out:
@@ -1562,6 +1564,27 @@ vhost_clear_queue(struct vhost_dev *vdev, uint16_t queue_id)
 	}
 }
 
+static void
+vhost_clear_async(struct vhost_dev *vdev, int vid, uint16_t queue_id)
+{
+	int16_t dma_id;
+
+	if (dma_bind[vid].dmas[queue_id].async_enabled) {
+		vhost_clear_queue(vdev, queue_id);
+		rte_vhost_async_channel_unregister(vid, queue_id);
+		dma_bind[vid].dmas[queue_id].async_enabled = false;
+
+		dma_id = dma_bind[vid2socketid[vdev->vid]].dmas[queue_id].dev_id;
+		dma_ref_count[dma_id]--;
+
+		if (dma_ref_count[dma_id] == 0) {
+			if (rte_vhost_async_dma_unconfigure(dma_id, 0) < 0)
+				RTE_LOG(ERR, VHOST_CONFIG,
+				       "Failed to unconfigure DMA %d in vhost.\n", dma_id);
+		}
+	}
+}
+
 /*
  * Remove a device from the specific data core linked list and from the
  * main linked list. Synchronization  occurs through the use of the
@@ -1618,17 +1641,8 @@ destroy_device(int vid)
 		"(%d) device has been removed from data core\n",
 		vdev->vid);
 
-	if (dma_bind[vid].dmas[VIRTIO_RXQ].async_enabled) {
-		vhost_clear_queue(vdev, VIRTIO_RXQ);
-		rte_vhost_async_channel_unregister(vid, VIRTIO_RXQ);
-		dma_bind[vid].dmas[VIRTIO_RXQ].async_enabled = false;
-	}
-
-	if (dma_bind[vid].dmas[VIRTIO_TXQ].async_enabled) {
-		vhost_clear_queue(vdev, VIRTIO_TXQ);
-		rte_vhost_async_channel_unregister(vid, VIRTIO_TXQ);
-		dma_bind[vid].dmas[VIRTIO_TXQ].async_enabled = false;
-	}
+	vhost_clear_async(vdev, vid, VIRTIO_RXQ);
+	vhost_clear_async(vdev, vid, VIRTIO_TXQ);
 
 	rte_free(vdev);
 }
@@ -1690,8 +1704,6 @@ vhost_async_channel_register(int vid)
 	return rx_ret | tx_ret;
 }
 
-
-
 /*
  * A new device is added to a data core. First the device is added to the main linked list
  * and then allocated to a specific data core.
Signature · 1 line
2.17.1
#18 lore raw

Re: [PATCH v4 2/2] examples/vhost: unconfigure DMA vchannel

Quoted 32 lines · On 10/13/22 08:40, [email protected] wrote:

From: Xuan Ding <[email protected]>

This patch applies rte_vhost_async_dma_unconfigure() to manually free DMA vchannels. Before unconfiguration, need make sure the specified DMA device is no longer used by any vhost ports.

---
  examples/vhost/main.c | 38 +++++++++++++++++++++++++-------------
  1 file changed, 25 insertions(+), 13 deletions(-)
examples/vhost/main.c +2 -0
index ac78704d79..bfeb808dcc 100644
@@ -73,6 +73,7 @@ static int total_num_mbufs = NUM_MBUFS_DEFAULT;
 
  struct dma_for_vhost dma_bind[RTE_MAX_VHOST_DEVICE];
  int16_t dmas_id[RTE_DMADEV_DEFAULT_MAX];
+int16_t dma_ref_count[RTE_DMADEV_DEFAULT_MAX];
  static int dma_count;
 
  /* mask of enabled ports */
@@ -371,6 +372,7 @@ open_dma(const char *value)
  done:
  		(dma_info + socketid)->dmas[vring_id].dev_id = dev_id;
  		(dma_info + socketid)->async_flag |= async_flag;
+		dma_ref_count[dev_id]++;
  		i++;
  	}
  out:

examples/vhost/main.c vhost_clear_queue(struct vhost_dev *vdev, uint16_t queue_id) · line 1572

Quoted 9 lines
examples/vhost/main.c +6 -0
@@ -1562,6 +1564,27 @@ vhost_clear_queue(struct vhost_dev *vdev, uint16_t queue_id)
  	}
  }
 
+static void
+vhost_clear_async(struct vhost_dev *vdev, int vid, uint16_t queue_id)
+{
+	int16_t dma_id;
+
+	if (dma_bind[vid].dmas[queue_id].async_enabled) {

if (!dma_bind[vid].dmas[queue_id].async_enabled)

	return;

examples/vhost/main.c vhost_clear_queue(struct vhost_dev *vdev, uint16_t queue_id) · line 1580

Quoted 8 lines
examples/vhost/main.c +8 -0
+		vhost_clear_queue(vdev, queue_id);
+		rte_vhost_async_channel_unregister(vid, queue_id);
+		dma_bind[vid].dmas[queue_id].async_enabled = false;
+
+		dma_id = dma_bind[vid2socketid[vdev->vid]].dmas[queue_id].dev_id;
+		dma_ref_count[dma_id]--;
+
+		if (dma_ref_count[dma_id] == 0) {

if (dma_ref_count[dma_id] > 0)

	return;

Doing this should improve readability.

Quoted 39 lines
examples/vhost/main.c +9 -13
+			if (rte_vhost_async_dma_unconfigure(dma_id, 0) < 0)
+				RTE_LOG(ERR, VHOST_CONFIG,
+				       "Failed to unconfigure DMA %d in vhost.\n", dma_id);
+		}
+	}
+}
+
  /*
   * Remove a device from the specific data core linked list and from the
   * main linked list. Synchronization  occurs through the use of the
@@ -1618,17 +1641,8 @@ destroy_device(int vid)
  		"(%d) device has been removed from data core\n",
  		vdev->vid);
 
-	if (dma_bind[vid].dmas[VIRTIO_RXQ].async_enabled) {
-		vhost_clear_queue(vdev, VIRTIO_RXQ);
-		rte_vhost_async_channel_unregister(vid, VIRTIO_RXQ);
-		dma_bind[vid].dmas[VIRTIO_RXQ].async_enabled = false;
-	}
-
-	if (dma_bind[vid].dmas[VIRTIO_TXQ].async_enabled) {
-		vhost_clear_queue(vdev, VIRTIO_TXQ);
-		rte_vhost_async_channel_unregister(vid, VIRTIO_TXQ);
-		dma_bind[vid].dmas[VIRTIO_TXQ].async_enabled = false;
-	}
+	vhost_clear_async(vdev, vid, VIRTIO_RXQ);
+	vhost_clear_async(vdev, vid, VIRTIO_TXQ);
 
  	rte_free(vdev);
  }
@@ -1690,8 +1704,6 @@ vhost_async_channel_register(int vid)
  	return rx_ret | tx_ret;
  }
 
-
-
  /*
   * A new device is added to a data core. First the device is added to the main linked list
   * and then allocated to a specific data core.
author
#19 lore raw

RE: [PATCH v4 2/2] examples/vhost: unconfigure DMA vchannel

Quoted 72 lines

-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Thursday, October 13, 2022 4:07 PM
To: Ding, Xuan <[email protected]>; Xia, Chenbo <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; He, Xingguang <[email protected]>; Yang, YvonneX <[email protected]>; Jiang, Cheng1 <[email protected]>; Wang, YuanX <[email protected]>; Ma, WenwuX <[email protected]>
Subject: Re: [PATCH v4 2/2] examples/vhost: unconfigure DMA vchannel

examples/vhost/main.c vhost_clear_queue(struct vhost_dev *vdev,

Quoted 32 lines · On 10/13/22 08:40, [email protected] wrote:

From: Xuan Ding <[email protected]>

This patch applies rte_vhost_async_dma_unconfigure() to manually free DMA vchannels. Before unconfiguration, need make sure the specified DMA device is no longer used by any vhost ports.

---
  examples/vhost/main.c | 38 +++++++++++++++++++++++++-------------
  1 file changed, 25 insertions(+), 13 deletions(-)
examples/vhost/main.c index renamed from examples/vhost/main.c +0 -0

ac78704d79..bfeb808dcc 100644

examples/vhost/main.c +2 -0
@@ -73,6 +73,7 @@ static int total_num_mbufs = NUM_MBUFS_DEFAULT;
 
  struct dma_for_vhost dma_bind[RTE_MAX_VHOST_DEVICE];
  int16_t dmas_id[RTE_DMADEV_DEFAULT_MAX];
+int16_t dma_ref_count[RTE_DMADEV_DEFAULT_MAX];
  static int dma_count;
 
  /* mask of enabled ports */
@@ -371,6 +372,7 @@ open_dma(const char *value)
  done:
  		(dma_info + socketid)->dmas[vring_id].dev_id = dev_id;
  		(dma_info + socketid)->async_flag |= async_flag;
+		dma_ref_count[dev_id]++;
  		i++;
  	}
  out:
@@ -1562,6 +1564,27 @@ vhost_clear_queue(struct vhost_dev *vdev,

uint16_t queue_id)

examples/vhost/main.c vhost_clear_queue(struct vhost_dev *vdev, uint16_t queue_id) · line 1572

Quoted 9 lines
examples/vhost/main.c +6 -0
  	}
  }
 
+static void
+vhost_clear_async(struct vhost_dev *vdev, int vid, uint16_t queue_id)
+{
+	int16_t dma_id;
+
+	if (dma_bind[vid].dmas[queue_id].async_enabled) {

if (!dma_bind[vid].dmas[queue_id].async_enabled)

	return;

examples/vhost/main.c vhost_clear_queue(struct vhost_dev *vdev, uint16_t queue_id) · line 1580

Quoted 9 lines
examples/vhost/main.c +8 -0
+		vhost_clear_queue(vdev, queue_id);
+		rte_vhost_async_channel_unregister(vid, queue_id);
+		dma_bind[vid].dmas[queue_id].async_enabled = false;
+
+		dma_id = dma_bind[vid2socketid[vdev-
 vid]].dmas[queue_id].dev_id;
+		dma_ref_count[dma_id]--;
+
+		if (dma_ref_count[dma_id] == 0) {

if (dma_ref_count[dma_id] > 0)

Quoted 3 lines
	return;

Doing this should improve readability.

Good suggestion! Please see v5.

Thanks,
Xuan

Quoted 42 lines

examples/vhost/main.c vhost_clear_queue(struct vhost_dev *vdev, uint16_t queue_id)

Quoted 4 lines
examples/vhost/main.c +3 -0
+			if (rte_vhost_async_dma_unconfigure(dma_id, 0) < 0)
+				RTE_LOG(ERR, VHOST_CONFIG,
+				       "Failed to unconfigure DMA %d in

vhost.\n", dma_id);

Quoted 28 lines
examples/vhost/main.c +4 -0
+		}
+	}
+}
+
  /*
   * Remove a device from the specific data core linked list and from the

* main linked list. Synchronization occurs through the use of the

examples/vhost/main.c +2 -11
@@ -1618,17 +1641,8 @@ destroy_device(int vid)
  		"(%d) device has been removed from data core\n",
  		vdev->vid);
 
-	if (dma_bind[vid].dmas[VIRTIO_RXQ].async_enabled) {
-		vhost_clear_queue(vdev, VIRTIO_RXQ);
-		rte_vhost_async_channel_unregister(vid, VIRTIO_RXQ);
-		dma_bind[vid].dmas[VIRTIO_RXQ].async_enabled = false;
-	}
-
-	if (dma_bind[vid].dmas[VIRTIO_TXQ].async_enabled) {
-		vhost_clear_queue(vdev, VIRTIO_TXQ);
-		rte_vhost_async_channel_unregister(vid, VIRTIO_TXQ);
-		dma_bind[vid].dmas[VIRTIO_TXQ].async_enabled = false;
-	}
+	vhost_clear_async(vdev, vid, VIRTIO_RXQ);
+	vhost_clear_async(vdev, vid, VIRTIO_TXQ);
 
  	rte_free(vdev);
  }

examples/vhost/main.c vhost_async_channel_register(int vid) · line 1708

Quoted 7 lines
examples/vhost/main.c +0 -2
@@ -1690,8 +1704,6 @@ vhost_async_channel_register(int vid)
  	return rx_ret | tx_ret;
  }
 
-
-
  /*
   * A new device is added to a data core. First the device is added to the

main linked list

Quoted 1 line
examples/vhost/main.c +0 -0
   * and then allocated to a specific data core.
#20 lore raw

v5 cover vhost: introduce DMA vchannel unconfiguration

In reply to [email protected]

From: Xuan Ding <[email protected]>

This patchset introduces a new API rte_vhost_async_dma_unconfigure() to help user to manually free DMA vchannels finished to use.

Note: this API should be called after async channel unregister.

v5:
* Use mutex instead of spinlock.
* Improve code readability.

v4:
* Rebase to 22.11 rc1.
* Fix the usage of 'dma_ref_count' to make sure the specified DMA device is not used by any vhost ports before unconfiguration.

v3:
* Rebase to latest DPDK.
* Refine some descriptions in the doc.
* Fix one bug in the vhost example.

v2:
* Add spinlock protection.
* Fix a memory leak issue.
* Refine the doc.

Xuan Ding (2):
vhost: introduce DMA vchannel unconfiguration
examples/vhost: unconfigure DMA vchannel

 doc/guides/prog_guide/vhost_lib.rst    |  6 +++
 doc/guides/rel_notes/release_22_11.rst |  4 ++
 examples/vhost/main.c                  | 40 ++++++++++-----
 lib/vhost/rte_vhost_async.h            | 18 +++++++
 lib/vhost/version.map                  |  3 ++
 lib/vhost/vhost.c                      | 69 ++++++++++++++++++++++++--
 6 files changed, 122 insertions(+), 18 deletions(-)
Signature · 1 line
2.17.1
#21 lore raw

PATCH v5 1/2 vhost: introduce DMA vchannel unconfiguration

From: Xuan Ding <[email protected]>

Add a new API rte_vhost_async_dma_unconfigure() to unconfigure DMA vchannels in vhost async data path. Lock protection are also added to protect DMA vchannels configuration and unconfiguration from concurrent calls.

---
 doc/guides/prog_guide/vhost_lib.rst    |  6 +++
 doc/guides/rel_notes/release_22_11.rst |  4 ++
 lib/vhost/rte_vhost_async.h            | 18 +++++++
 lib/vhost/version.map                  |  3 ++
 lib/vhost/vhost.c                      | 69 ++++++++++++++++++++++++--
 5 files changed, 95 insertions(+), 5 deletions(-)
doc/guides/prog_guide/vhost_lib.rst +6 -0
index bad4d819e1..d3cef978d0 100644
@@ -323,6 +323,12 @@ The following is an overview of some key Vhost API functions:
   Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET,
   VDPA_DEVICE_TYPE_BLK.
 
+* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)``
+
+  Clean DMA vChannels finished to use. This function needs to
+  be called after the deregistration of async DMA vchannel
+  has been finished.
+
 Vhost-user Implementations
 --------------------------
 
doc/guides/rel_notes/release_22_11.rst +4 -0
index 2da8bc9661..3be150122c 100644
@@ -236,6 +236,10 @@ New Features
 
      strings $dpdk_binary_or_driver | sed -n 's/^PMD_INFO_STRING= //p'
 
+* **Added DMA vChannel unconfiguration for async vhost.**
+
+  * Added support to unconfigure DMA vChannels that have been unregistered.
+
 
 Removed Items
 -------------
lib/vhost/rte_vhost_async.h +18 -0
index 1db2a10124..6ee4f7258d 100644
@@ -266,6 +266,24 @@ rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id,
 	struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count,
 	int *nr_inflight, int16_t dma_id, uint16_t vchan_id);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice.
+ *
+ * Unconfigure DMA vChannels in asynchronous data path.
+ * This function should be called after the DMA vChannel has been unregistered.
+ *
+ * @param dma_id
+ *  the identifier of DMA device
+ * @param vchan_id
+ *  the identifier of virtual DMA channel
+ * @return
+ *  0 on success, and -1 on failure
+ */
+__rte_experimental
+int
+rte_vhost_async_dma_unconfigure(int16_t dma_id, uint16_t vchan_id);
+
 #ifdef __cplusplus
 }
 #endif
lib/vhost/version.map +3 -0
index 7a00b65740..0b61870870 100644
@@ -94,6 +94,9 @@ EXPERIMENTAL {
 	rte_vhost_async_try_dequeue_burst;
 	rte_vhost_driver_get_vdpa_dev_type;
 	rte_vhost_clear_queue;
+
+	# added in 22.11
+	rte_vhost_async_dma_unconfigure;
 };
 
 INTERNAL {
lib/vhost/vhost.c +64 -5

121 lines not on this page · raw message

Signature · 1 line
2.17.1
#22 lore raw

PATCH v5 2/2 examples/vhost: unconfigure DMA vchannel

In reply to [email protected]

From: Xuan Ding <[email protected]>

This patch applies rte_vhost_async_dma_unconfigure() to manually free DMA vchannels. Before unconfiguration, make sure the specified DMA device is no longer used by any vhost ports.

---
 examples/vhost/main.c | 40 +++++++++++++++++++++++++++-------------
 1 file changed, 27 insertions(+), 13 deletions(-)
examples/vhost/main.c +27 -13
index ac78704d79..baf7d49116 100644
@@ -73,6 +73,7 @@ static int total_num_mbufs = NUM_MBUFS_DEFAULT;
 
 struct dma_for_vhost dma_bind[RTE_MAX_VHOST_DEVICE];
 int16_t dmas_id[RTE_DMADEV_DEFAULT_MAX];
+int16_t dma_ref_count[RTE_DMADEV_DEFAULT_MAX];
 static int dma_count;
 
 /* mask of enabled ports */
@@ -371,6 +372,7 @@ open_dma(const char *value)
 done:
 		(dma_info + socketid)->dmas[vring_id].dev_id = dev_id;
 		(dma_info + socketid)->async_flag |= async_flag;
+		dma_ref_count[dev_id]++;
 		i++;
 	}
 out:
@@ -1562,6 +1564,29 @@ vhost_clear_queue(struct vhost_dev *vdev, uint16_t queue_id)
 	}
 }
 
+static void
+vhost_clear_async(struct vhost_dev *vdev, int vid, uint16_t queue_id)
+{
+	int16_t dma_id;
+
+	if (!dma_bind[vid].dmas[queue_id].async_enabled)
+		return;
+
+	vhost_clear_queue(vdev, queue_id);
+	rte_vhost_async_channel_unregister(vid, queue_id);
+	dma_bind[vid].dmas[queue_id].async_enabled = false;
+
+	dma_id = dma_bind[vid2socketid[vdev->vid]].dmas[queue_id].dev_id;
+	dma_ref_count[dma_id]--;
+
+	if (dma_ref_count[dma_id] > 0)
+		return;
+
+	if (rte_vhost_async_dma_unconfigure(dma_id, 0) < 0)
+		RTE_LOG(ERR, VHOST_CONFIG,
+			"Failed to unconfigure DMA %d in vhost.\n", dma_id);
+}
+
 /*
  * Remove a device from the specific data core linked list and from the
  * main linked list. Synchronization  occurs through the use of the
@@ -1618,17 +1643,8 @@ destroy_device(int vid)
 		"(%d) device has been removed from data core\n",
 		vdev->vid);
 
-	if (dma_bind[vid].dmas[VIRTIO_RXQ].async_enabled) {
-		vhost_clear_queue(vdev, VIRTIO_RXQ);
-		rte_vhost_async_channel_unregister(vid, VIRTIO_RXQ);
-		dma_bind[vid].dmas[VIRTIO_RXQ].async_enabled = false;
-	}
-
-	if (dma_bind[vid].dmas[VIRTIO_TXQ].async_enabled) {
-		vhost_clear_queue(vdev, VIRTIO_TXQ);
-		rte_vhost_async_channel_unregister(vid, VIRTIO_TXQ);
-		dma_bind[vid].dmas[VIRTIO_TXQ].async_enabled = false;
-	}
+	vhost_clear_async(vdev, vid, VIRTIO_RXQ);
+	vhost_clear_async(vdev, vid, VIRTIO_TXQ);
 
 	rte_free(vdev);
 }
@@ -1690,8 +1706,6 @@ vhost_async_channel_register(int vid)
 	return rx_ret | tx_ret;
 }
 
-
-
 /*
  * A new device is added to a data core. First the device is added to the main linked list
  * and then allocated to a specific data core.
Signature · 1 line
2.17.1
#23 lore raw

v6 cover vhost: introduce DMA vchannel unconfiguration

In reply to [email protected]

From: Xuan Ding <[email protected]>

This patchset introduces a new API rte_vhost_async_dma_unconfigure() to help user to manually free DMA vchannels finished to use.

v6:
* Move DMA unconfiguration to the end due to DMA devices maybe reused after destroy_device().
* Refine the doc to claim the DMA device should not be used in vhost after unconfiguration.

v5:
* Use mutex instead of spinlock.
* Improve code readability.

v4:
* Rebase to 22.11 rc1.
* Fix the usage of 'dma_ref_count' to make sure the specified DMA device is not used by any vhost ports before unconfiguration.

v3:
* Rebase to latest DPDK.
* Refine some descriptions in the doc.
* Fix one bug in the vhost example.

v2:
* Add spinlock protection.
* Fix a memory leak issue.
* Refine the doc.

Xuan Ding (2):
vhost: introduce DMA vchannel unconfiguration
examples/vhost: unconfigure DMA vchannel

 doc/guides/prog_guide/vhost_lib.rst    |  7 +++
 doc/guides/rel_notes/release_22_11.rst |  5 ++
 examples/vhost/main.c                  |  8 +++
 lib/vhost/rte_vhost_async.h            | 20 ++++++++
 lib/vhost/version.map                  |  3 ++
 lib/vhost/vhost.c                      | 69 ++++++++++++++++++++++++--
 6 files changed, 107 insertions(+), 5 deletions(-)
Signature · 1 line
2.17.1
#24 lore raw

PATCH v6 1/2 vhost: introduce DMA vchannel unconfiguration

From: Xuan Ding <[email protected]>

Add a new API rte_vhost_async_dma_unconfigure() to unconfigure DMA vchannels in vhost async data path. Lock protection are also added to protect DMA vchannels configuration and unconfiguration from concurrent calls.

---
 doc/guides/prog_guide/vhost_lib.rst    |  7 +++
 doc/guides/rel_notes/release_22_11.rst |  5 ++
 lib/vhost/rte_vhost_async.h            | 20 ++++++++
 lib/vhost/version.map                  |  3 ++
 lib/vhost/vhost.c                      | 69 ++++++++++++++++++++++++--
 5 files changed, 99 insertions(+), 5 deletions(-)
doc/guides/prog_guide/vhost_lib.rst +7 -0
index bad4d819e1..fbe841321f 100644
@@ -323,6 +323,13 @@ The following is an overview of some key Vhost API functions:
   Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET,
   VDPA_DEVICE_TYPE_BLK.
 
+* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)``
+
+  Clean DMA vChannels finished to use. This function needs to be called
+  after the deregistration of async DMA vchannel has been finished.
+  After this function is called, the specified DMA device should no
+  longer be used by the Vhost library.
+
 Vhost-user Implementations
 --------------------------
 
doc/guides/rel_notes/release_22_11.rst +5 -0
index 2da8bc9661..a1c5cdea7c 100644
@@ -236,6 +236,11 @@ New Features
 
      strings $dpdk_binary_or_driver | sed -n 's/^PMD_INFO_STRING= //p'
 
+* **Added DMA vChannel unconfiguration for async vhost.**
+
+  * Added support to unconfigure DMA vChannels that have been unregistered
+  and no longer used by Vhost library.
+
 
 Removed Items
 -------------
lib/vhost/rte_vhost_async.h +20 -0
index 1db2a10124..eac7eb568e 100644
@@ -266,6 +266,26 @@ rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id,
 	struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count,
 	int *nr_inflight, int16_t dma_id, uint16_t vchan_id);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice.
+ *
+ * Unconfigure DMA vChannels in Vhost asynchronous data path.
+ * This function should be called after the DMA vChannel has been unregistered.
+ * After this function is called, the specified DMA device should no longer
+ * be used by the Vhost library.
+ *
+ * @param dma_id
+ *  the identifier of DMA device
+ * @param vchan_id
+ *  the identifier of virtual DMA channel
+ * @return
+ *  0 on success, and -1 on failure
+ */
+__rte_experimental
+int
+rte_vhost_async_dma_unconfigure(int16_t dma_id, uint16_t vchan_id);
+
 #ifdef __cplusplus
 }
 #endif
lib/vhost/version.map +3 -0
index 7a00b65740..0b61870870 100644
@@ -94,6 +94,9 @@ EXPERIMENTAL {
 	rte_vhost_async_try_dequeue_burst;
 	rte_vhost_driver_get_vdpa_dev_type;
 	rte_vhost_clear_queue;
+
+	# added in 22.11
+	rte_vhost_async_dma_unconfigure;
 };
 
 INTERNAL {
lib/vhost/vhost.c +64 -5

121 lines not on this page · raw message

Signature · 1 line
2.17.1
#25 lore raw
Quoted 58 lines

-----Original Message-----
From: Ding, Xuan <[email protected]>
Sent: Tuesday, October 18, 2022 11:22 PM
To: [email protected]; Xia, Chenbo <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; He, Xingguang <[email protected]>; Yang, YvonneX <[email protected]>; Jiang, Cheng1 <[email protected]>; Wang, YuanX <[email protected]>; Ma, WenwuX <[email protected]>; Ding, Xuan <[email protected]>
Subject: [PATCH v6 1/2] vhost: introduce DMA vchannel unconfiguration

From: Xuan Ding <[email protected]>

Add a new API rte_vhost_async_dma_unconfigure() to unconfigure DMA vchannels in vhost async data path. Lock protection are also added to protect DMA vchannels configuration and unconfiguration from concurrent calls.

---
 doc/guides/prog_guide/vhost_lib.rst    |  7 +++
 doc/guides/rel_notes/release_22_11.rst |  5 ++
 lib/vhost/rte_vhost_async.h            | 20 ++++++++
 lib/vhost/version.map                  |  3 ++
 lib/vhost/vhost.c                      | 69 ++++++++++++++++++++++++--
 5 files changed, 99 insertions(+), 5 deletions(-)
doc/guides/prog_guide/vhost_lib.rst +0 -0
index bad4d819e1..fbe841321f 100644
@@ -323,6 +323,13 @@ The following is an overview of some key Vhost API

functions:
Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET, VDPA_DEVICE_TYPE_BLK.

doc/guides/prog_guide/vhost_lib.rst +7 -0
+* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)``
+
+  Clean DMA vChannels finished to use. This function needs to be called
+  after the deregistration of async DMA vchannel has been finished.
+  After this function is called, the specified DMA device should no
+  longer be used by the Vhost library.
+
 Vhost-user Implementations
 --------------------------
doc/guides/rel_notes/release_22_11.rst +3 -0
index 2da8bc9661..a1c5cdea7c 100644
@@ -236,6 +236,11 @@ New Features
 
      strings $dpdk_binary_or_driver | sed -n 's/^PMD_INFO_STRING= //p'
 
+* **Added DMA vChannel unconfiguration for async vhost.**
+
+  * Added support to unconfigure DMA vChannels that have been
Quoted 2 lines

unregistered
+ and no longer used by Vhost library.

Somehow this is making release notes build fail.

Maxime, is this because of the '*' before 'Added support'?

Thanks,
Chenbo

Quoted 210 lines
doc/guides/rel_notes/release_22_11.rst +1 -0
+
 
 Removed Items
 -------------
lib/vhost/rte_vhost_async.h +3 -0
index 1db2a10124..eac7eb568e 100644
@@ -266,6 +266,26 @@ rte_vhost_async_try_dequeue_burst(int vid, uint16_t
 queue_id,
 	struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t
 count,
 	int *nr_inflight, int16_t dma_id, uint16_t vchan_id);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior

notice.
+ *
+ * Unconfigure DMA vChannels in Vhost asynchronous data path.
+ * This function should be called after the DMA vChannel has been unregistered.
+ * After this function is called, the specified DMA device should no longer
+ * be used by the Vhost library.
+ *
+ * @param dma_id
+ * the identifier of DMA device
+ * @param vchan_id
+ * the identifier of virtual DMA channel
+ * @return
+ * 0 on success, and -1 on failure
+ */

lib/vhost/rte_vhost_async.h +4 -0
+__rte_experimental
+int
+rte_vhost_async_dma_unconfigure(int16_t dma_id, uint16_t vchan_id);
+
 #ifdef __cplusplus
 }
 #endif
lib/vhost/version.map +3 -0
index 7a00b65740..0b61870870 100644
@@ -94,6 +94,9 @@ EXPERIMENTAL {
 	rte_vhost_async_try_dequeue_burst;
 	rte_vhost_driver_get_vdpa_dev_type;
 	rte_vhost_clear_queue;
+
+	# added in 22.11
+	rte_vhost_async_dma_unconfigure;
 };
 
 INTERNAL {
lib/vhost/vhost.c +5 -3
index 8740aa2788..975c0d3297 100644
@@ -23,6 +23,7 @@
 
 struct virtio_net *vhost_devices[RTE_MAX_VHOST_DEVICE];
 pthread_mutex_t vhost_dev_lock = PTHREAD_MUTEX_INITIALIZER;
+pthread_mutex_t vhost_dma_lock = PTHREAD_MUTEX_INITIALIZER;
 
 struct vhost_vq_stats_name_off {
 	char name[RTE_VHOST_STATS_NAME_SIZE];
@@ -1844,19 +1845,20 @@ rte_vhost_async_dma_configure(int16_t dma_id,
 uint16_t vchan_id)
 	void *pkts_cmpl_flag_addr;
 	uint16_t max_desc;
 
+	pthread_mutex_lock(&vhost_dma_lock);
 	if (!rte_dma_is_valid(dma_id)) {
 		VHOST_LOG_CONFIG("dma", ERR, "DMA %d is not found.\n", dma_id);
-		return -1;
+		goto error;
 	}
 
 	if (rte_dma_info_get(dma_id, &info) != 0) {
 		VHOST_LOG_CONFIG("dma", ERR, "Fail to get DMA %d
 information.\n", dma_id);
-		return -1;
+		goto error;
 	}
 
 	if (vchan_id >= info.max_vchans) {
 		VHOST_LOG_CONFIG("dma", ERR, "Invalid DMA %d vChannel %u.\n",
 dma_id, vchan_id);
-		return -1;
+		goto error;
lib/vhost/vhost.c +1 -1
 	}
 
 	if (!dma_copy_track[dma_id].vchans) {
@@ -1868,7 +1870,7 @@ rte_vhost_async_dma_configure(int16_t dma_id,
 uint16_t vchan_id)
 			VHOST_LOG_CONFIG("dma", ERR,
 				"Failed to allocate vchans for DMA %d
 vChannel %u.\n",
 				dma_id, vchan_id);
-			return -1;
+			goto error;
 		}
lib/vhost/vhost.c +1 -0
 		dma_copy_track[dma_id].vchans = vchans;
@@ -1877,6 +1879,7 @@ rte_vhost_async_dma_configure(int16_t dma_id,
 uint16_t vchan_id)
 	if (dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr) {
 		VHOST_LOG_CONFIG("dma", INFO, "DMA %d vChannel %u already
 registered.\n",
 			dma_id, vchan_id);
+		pthread_mutex_unlock(&vhost_dma_lock);
 		return 0;
lib/vhost/vhost.c +1 -1
 	}
@@ -1894,7 +1897,7 @@ rte_vhost_async_dma_configure(int16_t dma_id,
 uint16_t vchan_id)
 			rte_free(dma_copy_track[dma_id].vchans);
 			dma_copy_track[dma_id].vchans = NULL;
 		}
-		return -1;
+		goto error;
 	}
 

dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr =
pkts_cmpl_flag_addr;

lib/vhost/vhost.c +5 -0
@@ -1902,7 +1905,12 @@ rte_vhost_async_dma_configure(int16_t dma_id,
 uint16_t vchan_id)
 	dma_copy_track[dma_id].vchans[vchan_id].ring_mask = max_desc - 1;
 	dma_copy_track[dma_id].nr_vchans++;
 
+	pthread_mutex_unlock(&vhost_dma_lock);
 	return 0;
+
+error:
+	pthread_mutex_unlock(&vhost_dma_lock);
+	return -1;
 }
 

int

lib/vhost/vhost.c +20 -0
@@ -2091,5 +2099,56 @@ int rte_vhost_vring_stats_reset(int vid, uint16_t
 queue_id)
 	return 0;
 }
 
+int
+rte_vhost_async_dma_unconfigure(int16_t dma_id, uint16_t vchan_id)
+{
+	struct rte_dma_info info;
+	uint16_t max_desc;
+	int i;
+
+	pthread_mutex_lock(&vhost_dma_lock);
+	if (!rte_dma_is_valid(dma_id)) {
+		VHOST_LOG_CONFIG("dma", ERR, "DMA %d is not found.\n", dma_id);
+		goto error;
+	}
+
+	if (rte_dma_info_get(dma_id, &info) != 0) {
+		VHOST_LOG_CONFIG("dma", ERR, "Fail to get DMA %d
 information.\n", dma_id);
+		goto error;
+	}
+
+	if (vchan_id >= info.max_vchans) {
+		VHOST_LOG_CONFIG("dma", ERR, "Invalid DMA %d vChannel %u.\n",

dma_id, vchan_id);

lib/vhost/vhost.c +6 -0
+		goto error;
+	}
+
+	max_desc = info.max_desc;
+	for (i = 0; i < max_desc; i++) {
+		if

(dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr[i] != NULL) { +
rte_free(dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr
[i]);
+
dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr[i] =
NULL;

lib/vhost/vhost.c +4 -0
+		}
+	}
+
+	if (dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr !=

NULL) {
+
rte_free(dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr
);

lib/vhost/vhost.c +1 -0
+		dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr =

NULL;

lib/vhost/vhost.c +17 -1
+	}
+
+	if (dma_copy_track[dma_id].vchans != NULL) {
+		rte_free(dma_copy_track[dma_id].vchans);
+		dma_copy_track[dma_id].vchans = NULL;
+	}
+
+	dma_copy_track[dma_id].nr_vchans--;
+
+	pthread_mutex_unlock(&vhost_dma_lock);
+	return 0;
+
+error:
+	pthread_mutex_unlock(&vhost_dma_lock);
+	return -1;
+}
+
 RTE_LOG_REGISTER_SUFFIX(vhost_config_log_level, config, INFO);
 RTE_LOG_REGISTER_SUFFIX(vhost_data_log_level, data, WARNING);
--

2.17.1

#26 lore raw

PATCH v6 2/2 examples/vhost: unconfigure DMA vchannel

In reply to [email protected]

From: Xuan Ding <[email protected]>

This patch applies rte_vhost_async_dma_unconfigure() to manually free DMA vchannels. Before unconfiguration, make sure the specified DMA device is no longer used by any vhost ports.

---
 examples/vhost/main.c | 8 ++++++++
 1 file changed, 8 insertions(+)
examples/vhost/main.c +8 -0
index ac78704d79..42e53a0f9a 100644
@@ -2066,6 +2066,14 @@ main(int argc, char *argv[])
 	RTE_LCORE_FOREACH_WORKER(lcore_id)
 		rte_eal_wait_lcore(lcore_id);
 
+	for (i = 0; i < dma_count; i++) {
+		if (rte_vhost_async_dma_unconfigure(dmas_id[i], 0) < 0) {
+			RTE_LOG(ERR, VHOST_PORT,
+				"Failed to unconfigure DMA %d in vhost.\n", dmas_id[i]);
+			rte_exit(EXIT_FAILURE, "Cannot use given DMA device\n");
+		}
+	}
+
 	/* clean up the EAL */
 	rte_eal_cleanup();
 
Signature · 1 line
2.17.1
Tested-by on v6 2/2
#27 lore raw

RE: [PATCH v6 2/2] examples/vhost: unconfigure DMA vchannel

Quoted 10 lines

-----Original Message-----
From: [email protected] <[email protected]>
Sent: Tuesday, October 18, 2022 11:22 PM
To: [email protected]; Xia, Chenbo <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; He, Xingguang <[email protected]>; Yang, YvonneX <[email protected]>; Jiang, Cheng1 <[email protected]>; Wang, YuanX <[email protected]>; Ma, WenwuX <[email protected]>; Ding, Xuan <[email protected]>
Subject: [PATCH v6 2/2] examples/vhost: unconfigure DMA vchannel

Quoted 8 lines

From: Xuan Ding <[email protected]>

This patch applies rte_vhost_async_dma_unconfigure() to manually free DMA vchannels. Before unconfiguration, make sure the specified DMA device is no longer used by any vhost ports.

---

#28 lore raw

v7 cover vhost: introduce DMA vchannel unconfiguration

In reply to [email protected]

From: Xuan Ding <[email protected]>

This patchset introduces a new API rte_vhost_async_dma_unconfigure() to help user to manually free DMA vchannels finished to use.

v7:
* Add inflight packets processing.
* Fix CI error.

v6:
* Move DMA unconfiguration to the end due to DMA devices maybe reused after destroy_device().
* Refine the doc to claim the DMA device should not be used in vhost after unconfiguration.

v5:
* Use mutex instead of spinlock.
* Improve code readability.

v4:
* Rebase to 22.11 rc1.
* Fix the usage of 'dma_ref_count' to make sure the specified DMA device is not used by any vhost ports before unconfiguration.

v3:
* Rebase to latest DPDK.
* Refine some descriptions in the doc.
* Fix one bug in the vhost example.

v2:
* Add spinlock protection.
* Fix a memory leak issue.
* Refine the doc.

Xuan Ding (2):
vhost: introduce DMA vchannel unconfiguration
examples/vhost: unconfigure DMA vchannel

 doc/guides/prog_guide/vhost_lib.rst    |  5 ++
 doc/guides/rel_notes/release_22_11.rst |  5 ++
 examples/vhost/main.c                  |  8 +++
 lib/vhost/rte_vhost_async.h            | 20 +++++++
 lib/vhost/version.map                  |  3 ++
 lib/vhost/vhost.c                      | 75 ++++++++++++++++++++++++--
 6 files changed, 111 insertions(+), 5 deletions(-)
Signature · 1 line
2.17.1
#29 lore raw

PATCH v7 1/2 vhost: introduce DMA vchannel unconfiguration

From: Xuan Ding <[email protected]>

Add a new API rte_vhost_async_dma_unconfigure() to unconfigure DMA vchannels in vhost async data path. Lock protection are also added to protect DMA vchannels configuration and unconfiguration from concurrent calls.

---
 doc/guides/prog_guide/vhost_lib.rst    |  5 ++
 doc/guides/rel_notes/release_22_11.rst |  5 ++
 lib/vhost/rte_vhost_async.h            | 20 +++++++
 lib/vhost/version.map                  |  3 ++
 lib/vhost/vhost.c                      | 75 ++++++++++++++++++++++++--
 5 files changed, 103 insertions(+), 5 deletions(-)
doc/guides/prog_guide/vhost_lib.rst +5 -0
index bad4d819e1..cdf0c0b026 100644
@@ -323,6 +323,11 @@ The following is an overview of some key Vhost API functions:
   Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET,
   VDPA_DEVICE_TYPE_BLK.
 
+* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)``
+
+  Clean DMA vChannels finished to use. After this function is called,
+  the specified DMA vChannel should no longer be used by the Vhost library.
+
 Vhost-user Implementations
 --------------------------
 
doc/guides/rel_notes/release_22_11.rst +5 -0
index 2da8bc9661..dc52d1ce2f 100644
@@ -225,6 +225,11 @@ New Features
   sysfs entries to adjust the minimum and maximum uncore frequency values,
   which works on Linux with Intel hardware only.
 
+* **Added DMA vChannel unconfiguration for async vhost.**
+
+  Added support to unconfigure DMA vChannels that are no longer used
+  by the Vhost library.
+
 * **Rewritten pmdinfo script.**
 
   The ``dpdk-pmdinfo.py`` script was rewritten to produce valid JSON only.
lib/vhost/rte_vhost_async.h +20 -0
index 1db2a10124..45c75be55c 100644
@@ -266,6 +266,26 @@ rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id,
 	struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count,
 	int *nr_inflight, int16_t dma_id, uint16_t vchan_id);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice.
+ *
+ * Unconfigure DMA vChannels in Vhost asynchronous data path.
+ * This function should be called when the specified DMA device is no longer
+ * used by the Vhost library. Before this function is called, make sure there
+ * does not exist in-flight packets in DMA vChannel.
+ *
+ * @param dma_id
+ *  the identifier of DMA device
+ * @param vchan_id
+ *  the identifier of virtual DMA channel
+ * @return
+ *  0 on success, and -1 on failure
+ */
+__rte_experimental
+int
+rte_vhost_async_dma_unconfigure(int16_t dma_id, uint16_t vchan_id);
+
 #ifdef __cplusplus
 }
 #endif
lib/vhost/version.map +3 -0
index 7a00b65740..0b61870870 100644
@@ -94,6 +94,9 @@ EXPERIMENTAL {
 	rte_vhost_async_try_dequeue_burst;
 	rte_vhost_driver_get_vdpa_dev_type;
 	rte_vhost_clear_queue;
+
+	# added in 22.11
+	rte_vhost_async_dma_unconfigure;
 };
 
 INTERNAL {
lib/vhost/vhost.c +70 -5

127 lines not on this page · raw message

Signature · 1 line
2.17.1
Reviewed-by on v7 1/2
#30 lore raw
Quoted 7 lines · On 10/20/22 11:11, [email protected] wrote:

From: Xuan Ding <[email protected]>

Add a new API rte_vhost_async_dma_unconfigure() to unconfigure DMA vchannels in vhost async data path. Lock protection are also added to protect DMA vchannels configuration and unconfiguration from concurrent calls.

Quoted 9 lines
---
  doc/guides/prog_guide/vhost_lib.rst    |  5 ++
  doc/guides/rel_notes/release_22_11.rst |  5 ++
  lib/vhost/rte_vhost_async.h            | 20 +++++++
  lib/vhost/version.map                  |  3 ++
  lib/vhost/vhost.c                      | 75 ++++++++++++++++++++++++--
  5 files changed, 103 insertions(+), 5 deletions(-)

Thanks,
Maxime

author
#31 lore raw

Hi Maxime,

Quoted 30 lines

-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Friday, October 21, 2022 4:10 PM
To: Ding, Xuan <[email protected]>; Xia, Chenbo <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; He, Xingguang <[email protected]>; Yang, YvonneX <[email protected]>; Jiang, Cheng1 <[email protected]>; Wang, YuanX <[email protected]>; Ma, WenwuX <[email protected]>
Subject: Re: [PATCH v7 1/2] vhost: introduce DMA vchannel unconfiguration

Quoted 7 lines · On 10/20/22 11:11, [email protected] wrote:

From: Xuan Ding <[email protected]>

Add a new API rte_vhost_async_dma_unconfigure() to unconfigure DMA vchannels in vhost async data path. Lock protection are also added to protect DMA vchannels configuration and unconfiguration from concurrent calls.

Quoted 9 lines
---
  doc/guides/prog_guide/vhost_lib.rst    |  5 ++
  doc/guides/rel_notes/release_22_11.rst |  5 ++
  lib/vhost/rte_vhost_async.h            | 20 +++++++
  lib/vhost/version.map                  |  3 ++
  lib/vhost/vhost.c                      | 75 ++++++++++++++++++++++++--
  5 files changed, 103 insertions(+), 5 deletions(-)
Quoted 1 line

This patch needs mirror changes.
Please help to review and wait until the v8 test passes before merging. Thanks a lot.

Regards,
Xuan

Quoted 3 lines

Thanks,
Maxime

#32 lore raw

PATCH v7 2/2 examples/vhost: unconfigure DMA vchannel

In reply to [email protected]

From: Xuan Ding <[email protected]>

This patch applies rte_vhost_async_dma_unconfigure() to manually free DMA vchannels. Before unconfiguration, make sure the specified DMA device is no longer used by any vhost ports.

---
 examples/vhost/main.c | 8 ++++++++
 1 file changed, 8 insertions(+)
examples/vhost/main.c +8 -0
index ac78704d79..42e53a0f9a 100644
@@ -2066,6 +2066,14 @@ main(int argc, char *argv[])
 	RTE_LCORE_FOREACH_WORKER(lcore_id)
 		rte_eal_wait_lcore(lcore_id);
 
+	for (i = 0; i < dma_count; i++) {
+		if (rte_vhost_async_dma_unconfigure(dmas_id[i], 0) < 0) {
+			RTE_LOG(ERR, VHOST_PORT,
+				"Failed to unconfigure DMA %d in vhost.\n", dmas_id[i]);
+			rte_exit(EXIT_FAILURE, "Cannot use given DMA device\n");
+		}
+	}
+
 	/* clean up the EAL */
 	rte_eal_cleanup();
 
Signature · 1 line
2.17.1
Reviewed-by on v7 2/2
#33 lore raw

Re: [PATCH v7 2/2] examples/vhost: unconfigure DMA vchannel

Quoted 20 lines · On 10/20/22 11:11, [email protected] wrote:

From: Xuan Ding <[email protected]>

This patch applies rte_vhost_async_dma_unconfigure() to manually free DMA vchannels. Before unconfiguration, make sure the specified DMA device is no longer used by any vhost ports.

---
  examples/vhost/main.c | 8 ++++++++
  1 file changed, 8 insertions(+)
examples/vhost/main.c +1 -0
index ac78704d79..42e53a0f9a 100644
@@ -2066,6 +2066,14 @@ main(int argc, char *argv[])
  	RTE_LCORE_FOREACH_WORKER(lcore_id)
  		rte_eal_wait_lcore(lcore_id);
 
+	for (i = 0; i < dma_count; i++) {

examples/vhost/main.c main(int argc, char *argv[]) · line 2079

Quoted 10 lines
examples/vhost/main.c +7 -0
+		if (rte_vhost_async_dma_unconfigure(dmas_id[i], 0) < 0) {
+			RTE_LOG(ERR, VHOST_PORT,
+				"Failed to unconfigure DMA %d in vhost.\n", dmas_id[i]);
+			rte_exit(EXIT_FAILURE, "Cannot use given DMA device\n");
+		}
+	}
+
  	/* clean up the EAL */
  	rte_eal_cleanup();
 

Thanks,
Maxime

#34 lore raw

v8 cover vhost: introduce DMA vchannel unconfiguration

In reply to [email protected]

From: Xuan Ding <[email protected]>

This patchset introduces a new API rte_vhost_async_dma_unconfigure() to help user to manually free DMA vChannels finished to use.

v8:
* Check inflight packets release virtual channel.

v7:
* Add inflight packets processing.
* Fix CI error.

v6:
* Move DMA unconfiguration to the end due to DMA devices maybe reused after destroy_device().
* Refine the doc to claim the DMA device should not be used in vhost after unconfiguration.

v5:
* Use mutex instead of spinlock.
* Improve code readability.

v4:
* Rebase to 22.11 rc1.
* Fix the usage of 'dma_ref_count' to make sure the specified DMA device is not used by any vhost ports before unconfiguration.

v3:
* Rebase to latest DPDK.
* Refine some descriptions in the doc.
* Fix one bug in the vhost example.

v2:
* Add spinlock protection.
* Fix a memory leak issue.
* Refine the doc.

Xuan Ding (2):
vhost: introduce DMA vchannel unconfiguration
examples/vhost: unconfigure DMA vchannel

 doc/guides/prog_guide/vhost_lib.rst    |  5 ++
 doc/guides/rel_notes/release_22_11.rst |  5 ++
 examples/vhost/main.c                  |  8 +++
 lib/vhost/rte_vhost_async.h            | 20 +++++++
 lib/vhost/version.map                  |  3 ++
 lib/vhost/vhost.c                      | 72 ++++++++++++++++++++++++--
 6 files changed, 108 insertions(+), 5 deletions(-)
Signature · 1 line
2.17.1
#35 lore raw

PATCH v8 1/2 vhost: introduce DMA vchannel unconfiguration

From: Xuan Ding <[email protected]>

Add a new API rte_vhost_async_dma_unconfigure() to unconfigure DMA vChannels in vhost async data path. Lock protection are also added to protect DMA vChannel configuration and unconfiguration from concurrent calls.

---
 doc/guides/prog_guide/vhost_lib.rst    |  5 ++
 doc/guides/rel_notes/release_22_11.rst |  5 ++
 lib/vhost/rte_vhost_async.h            | 20 +++++++
 lib/vhost/version.map                  |  3 ++
 lib/vhost/vhost.c                      | 72 ++++++++++++++++++++++++--
 5 files changed, 100 insertions(+), 5 deletions(-)
doc/guides/prog_guide/vhost_lib.rst +5 -0
index bad4d819e1..0d9eca1f7d 100644
@@ -323,6 +323,11 @@ The following is an overview of some key Vhost API functions:
   Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET,
   VDPA_DEVICE_TYPE_BLK.
 
+* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)``
+
+  Clean DMA vChannel finished to use. After this function is called,
+  the specified DMA vChannel should no longer be used by the Vhost library.
+
 Vhost-user Implementations
 --------------------------
 
doc/guides/rel_notes/release_22_11.rst +5 -0
index 2da8bc9661..bbd1c5aa9c 100644
@@ -225,6 +225,11 @@ New Features
   sysfs entries to adjust the minimum and maximum uncore frequency values,
   which works on Linux with Intel hardware only.
 
+* **Added DMA vChannel unconfiguration for async vhost.**
+
+  Added support to unconfigure DMA vChannel that is no longer used
+  by the Vhost library.
+
 * **Rewritten pmdinfo script.**
 
   The ``dpdk-pmdinfo.py`` script was rewritten to produce valid JSON only.
lib/vhost/rte_vhost_async.h +20 -0
index 1db2a10124..8f190dd44b 100644
@@ -266,6 +266,26 @@ rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id,
 	struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count,
 	int *nr_inflight, int16_t dma_id, uint16_t vchan_id);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice.
+ *
+ * Unconfigure DMA vChannel in Vhost asynchronous data path.
+ * This function should be called when the specified DMA vChannel is no longer
+ * used by the Vhost library. Before this function is called, make sure there
+ * does not exist in-flight packets in DMA vChannel.
+ *
+ * @param dma_id
+ *  the identifier of DMA device
+ * @param vchan_id
+ *  the identifier of virtual DMA channel
+ * @return
+ *  0 on success, and -1 on failure
+ */
+__rte_experimental
+int
+rte_vhost_async_dma_unconfigure(int16_t dma_id, uint16_t vchan_id);
+
 #ifdef __cplusplus
 }
 #endif
lib/vhost/version.map +3 -0
index 7a00b65740..0b61870870 100644
@@ -94,6 +94,9 @@ EXPERIMENTAL {
 	rte_vhost_async_try_dequeue_burst;
 	rte_vhost_driver_get_vdpa_dev_type;
 	rte_vhost_clear_queue;
+
+	# added in 22.11
+	rte_vhost_async_dma_unconfigure;
 };
 
 INTERNAL {
lib/vhost/vhost.c +67 -5

124 lines not on this page · raw message

Signature · 1 line
2.17.1
Reviewed-by on v8 1/2
#36 lore raw
Quoted 7 lines · On 10/25/22 10:25, [email protected] wrote:

From: Xuan Ding <[email protected]>

Add a new API rte_vhost_async_dma_unconfigure() to unconfigure DMA vChannels in vhost async data path. Lock protection are also added to protect DMA vChannel configuration and unconfiguration from concurrent calls.

Quoted 9 lines
---
  doc/guides/prog_guide/vhost_lib.rst    |  5 ++
  doc/guides/rel_notes/release_22_11.rst |  5 ++
  lib/vhost/rte_vhost_async.h            | 20 +++++++
  lib/vhost/version.map                  |  3 ++
  lib/vhost/vhost.c                      | 72 ++++++++++++++++++++++++--
  5 files changed, 100 insertions(+), 5 deletions(-)

Thanks,
Maxime

Reviewed-by on v8 1/2
#37 lore raw
In reply to [email protected]
Quoted 266 lines

-----Original Message-----
From: Ding, Xuan <[email protected]>
Sent: Tuesday, October 25, 2022 4:26 PM
To: [email protected]; Xia, Chenbo <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; He, Xingguang <[email protected]>; Ling, WeiX <[email protected]>; Jiang, Cheng1 <[email protected]>; Wang, YuanX <[email protected]>; Ma, WenwuX <[email protected]>; Ding, Xuan <[email protected]>
Subject: [PATCH v8 1/2] vhost: introduce DMA vchannel unconfiguration

From: Xuan Ding <[email protected]>

Add a new API rte_vhost_async_dma_unconfigure() to unconfigure DMA vChannels in vhost async data path. Lock protection are also added to protect DMA vChannel configuration and unconfiguration from concurrent calls.

---
 doc/guides/prog_guide/vhost_lib.rst    |  5 ++
 doc/guides/rel_notes/release_22_11.rst |  5 ++
 lib/vhost/rte_vhost_async.h            | 20 +++++++
 lib/vhost/version.map                  |  3 ++
 lib/vhost/vhost.c                      | 72 ++++++++++++++++++++++++--
 5 files changed, 100 insertions(+), 5 deletions(-)
doc/guides/prog_guide/vhost_lib.rst +0 -0
index bad4d819e1..0d9eca1f7d 100644
@@ -323,6 +323,11 @@ The following is an overview of some key Vhost API

functions:
Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET, VDPA_DEVICE_TYPE_BLK.

doc/guides/prog_guide/vhost_lib.rst +4 -0
+* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)``
+
+  Clean DMA vChannel finished to use. After this function is called,
+  the specified DMA vChannel should no longer be used by the Vhost

library.

doc/guides/prog_guide/vhost_lib.rst +1 -0
+
 Vhost-user Implementations
 --------------------------
doc/guides/rel_notes/release_22_11.rst +5 -0
index 2da8bc9661..bbd1c5aa9c 100644
@@ -225,6 +225,11 @@ New Features
   sysfs entries to adjust the minimum and maximum uncore frequency values,
   which works on Linux with Intel hardware only.
 
+* **Added DMA vChannel unconfiguration for async vhost.**
+
+  Added support to unconfigure DMA vChannel that is no longer used
+  by the Vhost library.
+
 * **Rewritten pmdinfo script.**
 
   The ``dpdk-pmdinfo.py`` script was rewritten to produce valid JSON only.
lib/vhost/rte_vhost_async.h +3 -0
index 1db2a10124..8f190dd44b 100644
@@ -266,6 +266,26 @@ rte_vhost_async_try_dequeue_burst(int vid, uint16_t
 queue_id,
 	struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t
 count,
 	int *nr_inflight, int16_t dma_id, uint16_t vchan_id);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior

notice.
+ *
+ * Unconfigure DMA vChannel in Vhost asynchronous data path.
+ * This function should be called when the specified DMA vChannel is no longer
+ * used by the Vhost library. Before this function is called, make sure there
+ * does not exist in-flight packets in DMA vChannel.
+ *
+ * @param dma_id
+ * the identifier of DMA device
+ * @param vchan_id
+ * the identifier of virtual DMA channel
+ * @return
+ * 0 on success, and -1 on failure
+ */

lib/vhost/rte_vhost_async.h +4 -0
+__rte_experimental
+int
+rte_vhost_async_dma_unconfigure(int16_t dma_id, uint16_t vchan_id);
+
 #ifdef __cplusplus
 }
 #endif
lib/vhost/version.map +3 -0
index 7a00b65740..0b61870870 100644
@@ -94,6 +94,9 @@ EXPERIMENTAL {
 	rte_vhost_async_try_dequeue_burst;
 	rte_vhost_driver_get_vdpa_dev_type;
 	rte_vhost_clear_queue;
+
+	# added in 22.11
+	rte_vhost_async_dma_unconfigure;
 };
 
 INTERNAL {
lib/vhost/vhost.c +6 -3
index 8740aa2788..1bb01c2a2e 100644
@@ -23,6 +23,7 @@
 
 struct virtio_net *vhost_devices[RTE_MAX_VHOST_DEVICE];
 pthread_mutex_t vhost_dev_lock = PTHREAD_MUTEX_INITIALIZER;
+pthread_mutex_t vhost_dma_lock = PTHREAD_MUTEX_INITIALIZER;
 
 struct vhost_vq_stats_name_off {
 	char name[RTE_VHOST_STATS_NAME_SIZE];
@@ -1844,19 +1845,21 @@ rte_vhost_async_dma_configure(int16_t dma_id,
 uint16_t vchan_id)
 	void *pkts_cmpl_flag_addr;
 	uint16_t max_desc;
 
+	pthread_mutex_lock(&vhost_dma_lock);
+
 	if (!rte_dma_is_valid(dma_id)) {
 		VHOST_LOG_CONFIG("dma", ERR, "DMA %d is not found.\n", dma_id);
-		return -1;
+		goto error;
 	}
 
 	if (rte_dma_info_get(dma_id, &info) != 0) {
 		VHOST_LOG_CONFIG("dma", ERR, "Fail to get DMA %d
 information.\n", dma_id);
-		return -1;
+		goto error;
 	}
 
 	if (vchan_id >= info.max_vchans) {
 		VHOST_LOG_CONFIG("dma", ERR, "Invalid DMA %d vChannel %u.\n",
 dma_id, vchan_id);
-		return -1;
+		goto error;
lib/vhost/vhost.c +1 -1
 	}
 
 	if (!dma_copy_track[dma_id].vchans) {
@@ -1868,7 +1871,7 @@ rte_vhost_async_dma_configure(int16_t dma_id,
 uint16_t vchan_id)
 			VHOST_LOG_CONFIG("dma", ERR,
 				"Failed to allocate vchans for DMA %d
 vChannel %u.\n",
 				dma_id, vchan_id);
-			return -1;
+			goto error;
 		}
lib/vhost/vhost.c +1 -0
 		dma_copy_track[dma_id].vchans = vchans;
@@ -1877,6 +1880,7 @@ rte_vhost_async_dma_configure(int16_t dma_id,
 uint16_t vchan_id)
 	if (dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr) {
 		VHOST_LOG_CONFIG("dma", INFO, "DMA %d vChannel %u already
 registered.\n",
 			dma_id, vchan_id);
+		pthread_mutex_unlock(&vhost_dma_lock);
 		return 0;
lib/vhost/vhost.c +1 -1
 	}
@@ -1894,7 +1898,7 @@ rte_vhost_async_dma_configure(int16_t dma_id,
 uint16_t vchan_id)
 			rte_free(dma_copy_track[dma_id].vchans);
 			dma_copy_track[dma_id].vchans = NULL;
 		}
-		return -1;
+		goto error;
 	}
 

dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr =
pkts_cmpl_flag_addr;

lib/vhost/vhost.c +5 -0
@@ -1902,7 +1906,12 @@ rte_vhost_async_dma_configure(int16_t dma_id,
 uint16_t vchan_id)
 	dma_copy_track[dma_id].vchans[vchan_id].ring_mask = max_desc - 1;
 	dma_copy_track[dma_id].nr_vchans++;
 
+	pthread_mutex_unlock(&vhost_dma_lock);
 	return 0;
+
+error:
+	pthread_mutex_unlock(&vhost_dma_lock);
+	return -1;
 }
 

int

lib/vhost/vhost.c +20 -0
@@ -2091,5 +2100,58 @@ int rte_vhost_vring_stats_reset(int vid, uint16_t
 queue_id)
 	return 0;
 }
 
+int
+rte_vhost_async_dma_unconfigure(int16_t dma_id, uint16_t vchan_id)
+{
+	struct rte_dma_info info;
+	struct rte_dma_stats stats = { 0 };
+
+	pthread_mutex_lock(&vhost_dma_lock);
+
+	if (!rte_dma_is_valid(dma_id)) {
+		VHOST_LOG_CONFIG("dma", ERR, "DMA %d is not found.\n", dma_id);
+		goto error;
+	}
+
+	if (rte_dma_info_get(dma_id, &info) != 0) {
+		VHOST_LOG_CONFIG("dma", ERR, "Fail to get DMA %d
 information.\n", dma_id);
+		goto error;
+	}
+
+	if (vchan_id >= info.max_vchans || !dma_copy_track[dma_id].vchans ||
+		!dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr)

{

lib/vhost/vhost.c +1 -0
+		VHOST_LOG_CONFIG("dma", ERR, "Invalid channel %d:%u.\n",

dma_id, vchan_id);

lib/vhost/vhost.c +6 -0
+		goto error;
+	}
+
+	if (rte_dma_stats_get(dma_id, vchan_id, &stats) != 0) {
+		VHOST_LOG_CONFIG("dma", ERR,
+				 "Failed to get stats for DMA %d vChannel %u.\n",

dma_id, vchan_id);

lib/vhost/vhost.c +6 -0
+		goto error;
+	}
+
+	if (stats.submitted - stats.completed != 0) {
+		VHOST_LOG_CONFIG("dma", ERR,
+				 "Do not unconfigure when there are inflight

packets.\n");

lib/vhost/vhost.c +4 -0
+		goto error;
+	}
+
+

rte_free(dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr
);

lib/vhost/vhost.c +16 -1
+	dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr = NULL;
+	dma_copy_track[dma_id].nr_vchans--;
+
+	if (dma_copy_track[dma_id].nr_vchans == 0) {
+		rte_free(dma_copy_track[dma_id].vchans);
+		dma_copy_track[dma_id].vchans = NULL;
+	}
+
+	pthread_mutex_unlock(&vhost_dma_lock);
+	return 0;
+
+error:
+	pthread_mutex_unlock(&vhost_dma_lock);
+	return -1;
+}
+
 RTE_LOG_REGISTER_SUFFIX(vhost_config_log_level, config, INFO);
 RTE_LOG_REGISTER_SUFFIX(vhost_data_log_level, data, WARNING);
--
Quoted 1 line

2.17.1

#38 lore raw

PATCH v8 2/2 examples/vhost: unconfigure DMA vchannel

In reply to [email protected]

From: Xuan Ding <[email protected]>

This patch applies rte_vhost_async_dma_unconfigure() to manually free DMA vChannels. Before unconfiguration, make sure the specified DMA vChannel is no longer used by any vhost ports.

---
 examples/vhost/main.c | 8 ++++++++
 1 file changed, 8 insertions(+)
examples/vhost/main.c +8 -0
index ac78704d79..42e53a0f9a 100644
@@ -2066,6 +2066,14 @@ main(int argc, char *argv[])
 	RTE_LCORE_FOREACH_WORKER(lcore_id)
 		rte_eal_wait_lcore(lcore_id);
 
+	for (i = 0; i < dma_count; i++) {
+		if (rte_vhost_async_dma_unconfigure(dmas_id[i], 0) < 0) {
+			RTE_LOG(ERR, VHOST_PORT,
+				"Failed to unconfigure DMA %d in vhost.\n", dmas_id[i]);
+			rte_exit(EXIT_FAILURE, "Cannot use given DMA device\n");
+		}
+	}
+
 	/* clean up the EAL */
 	rte_eal_cleanup();
 
Signature · 1 line
2.17.1
Tested-by on v8 2/2
#39 lore raw

RE: [PATCH v8 2/2] examples/vhost: unconfigure DMA vchannel

Quoted 10 lines

-----Original Message-----
From: Ding, Xuan <[email protected]>
Sent: Tuesday, October 25, 2022 4:26 PM
To: [email protected]; Xia, Chenbo <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; He, Xingguang <[email protected]>; Ling, WeiX <[email protected]>; Jiang, Cheng1 <[email protected]>; Wang, YuanX <[email protected]>; Ma, WenwuX <[email protected]>; Ding, Xuan <[email protected]>
Subject: [PATCH v8 2/2] examples/vhost: unconfigure DMA vchannel

Quoted 8 lines

From: Xuan Ding <[email protected]>

This patch applies rte_vhost_async_dma_unconfigure() to manually free DMA vChannels. Before unconfiguration, make sure the specified DMA vChannel is no longer used by any vhost ports.

---

Reviewed-by on v8 2/2
#40 lore raw

Re: [PATCH v8 2/2] examples/vhost: unconfigure DMA vchannel

In reply to [email protected]
Quoted 20 lines · On 10/25/22 10:25, [email protected] wrote:

From: Xuan Ding <[email protected]>

This patch applies rte_vhost_async_dma_unconfigure() to manually free DMA vChannels. Before unconfiguration, make sure the specified DMA vChannel is no longer used by any vhost ports.

---
  examples/vhost/main.c | 8 ++++++++
  1 file changed, 8 insertions(+)
examples/vhost/main.c +1 -0
index ac78704d79..42e53a0f9a 100644
@@ -2066,6 +2066,14 @@ main(int argc, char *argv[])
  	RTE_LCORE_FOREACH_WORKER(lcore_id)
  		rte_eal_wait_lcore(lcore_id);
 
+	for (i = 0; i < dma_count; i++) {

examples/vhost/main.c main(int argc, char *argv[]) · line 2079

Quoted 10 lines
examples/vhost/main.c +7 -0
+		if (rte_vhost_async_dma_unconfigure(dmas_id[i], 0) < 0) {
+			RTE_LOG(ERR, VHOST_PORT,
+				"Failed to unconfigure DMA %d in vhost.\n", dmas_id[i]);
+			rte_exit(EXIT_FAILURE, "Cannot use given DMA device\n");
+		}
+	}
+
  	/* clean up the EAL */
  	rte_eal_cleanup();
 

Thanks,
Maxime

Reviewed-by on v8 2/2
#41 lore raw

RE: [PATCH v8 2/2] examples/vhost: unconfigure DMA vchannel

In reply to [email protected]
Quoted 42 lines

-----Original Message-----
From: Ding, Xuan <[email protected]>
Sent: Tuesday, October 25, 2022 4:26 PM
To: [email protected]; Xia, Chenbo <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; He, Xingguang <[email protected]>; Ling, WeiX <[email protected]>; Jiang, Cheng1 <[email protected]>; Wang, YuanX <[email protected]>; Ma, WenwuX <[email protected]>; Ding, Xuan <[email protected]>
Subject: [PATCH v8 2/2] examples/vhost: unconfigure DMA vchannel

From: Xuan Ding <[email protected]>

This patch applies rte_vhost_async_dma_unconfigure() to manually free DMA vChannels. Before unconfiguration, make sure the specified DMA vChannel is no longer used by any vhost ports.

---
 examples/vhost/main.c | 8 ++++++++
 1 file changed, 8 insertions(+)
examples/vhost/main.c +8 -1
index ac78704d79..42e53a0f9a 100644
@@ -2066,6 +2066,14 @@ main(int argc, char *argv[])
 	RTE_LCORE_FOREACH_WORKER(lcore_id)
 		rte_eal_wait_lcore(lcore_id);
 
+	for (i = 0; i < dma_count; i++) {
+		if (rte_vhost_async_dma_unconfigure(dmas_id[i], 0) < 0) {
+			RTE_LOG(ERR, VHOST_PORT,
+				"Failed to unconfigure DMA %d in vhost.\n",
 dmas_id[i]);
+			rte_exit(EXIT_FAILURE, "Cannot use given DMA device\n");
+		}
+	}
+
 	/* clean up the EAL */
 	rte_eal_cleanup();
 
--
Quoted 1 line

2.17.1

#42 lore raw
In reply to [email protected]
Quoted 59 lines

-----Original Message-----
From: Ding, Xuan <[email protected]>
Sent: Tuesday, October 25, 2022 4:26 PM
To: [email protected]; Xia, Chenbo <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; He, Xingguang <[email protected]>; Ling, WeiX <[email protected]>; Jiang, Cheng1 <[email protected]>; Wang, YuanX <[email protected]>; Ma, WenwuX <[email protected]>; Ding, Xuan <[email protected]>
Subject: [PATCH v8 0/2] vhost: introduce DMA vchannel unconfiguration

From: Xuan Ding <[email protected]>

This patchset introduces a new API rte_vhost_async_dma_unconfigure() to help user to manually free DMA vChannels finished to use.

v8:
* Check inflight packets release virtual channel.

v7:
* Add inflight packets processing.
* Fix CI error.

v6:
* Move DMA unconfiguration to the end due to DMA devices maybe reused after destroy_device().
* Refine the doc to claim the DMA device should not be used in vhost after unconfiguration.

v5:
* Use mutex instead of spinlock.
* Improve code readability.

v4:
* Rebase to 22.11 rc1.
* Fix the usage of 'dma_ref_count' to make sure the specified DMA device is not used by any vhost ports before unconfiguration.

v3:
* Rebase to latest DPDK.
* Refine some descriptions in the doc.
* Fix one bug in the vhost example.

v2:
* Add spinlock protection.
* Fix a memory leak issue.
* Refine the doc.

Xuan Ding (2):
vhost: introduce DMA vchannel unconfiguration
examples/vhost: unconfigure DMA vchannel

 doc/guides/prog_guide/vhost_lib.rst    |  5 ++
 doc/guides/rel_notes/release_22_11.rst |  5 ++
 examples/vhost/main.c                  |  8 +++
 lib/vhost/rte_vhost_async.h            | 20 +++++++
 lib/vhost/version.map                  |  3 ++
 lib/vhost/vhost.c                      | 72 ++++++++++++++++++++++++--
 6 files changed, 108 insertions(+), 5 deletions(-)
Quoted 2 lines

--
2.17.1

Series applied to next-virtio/main, thanks