vhost: introduce DMA vchannel unconfiguration
cover vhost: introduce DMA vchannel unconfiguration
From: Xuan Ding <[email protected]>
This patchset introduces a new API rte_vhost_async_dma_unconfigure() to help user to manually free the DMA vchannel finished to use.
Note: this API should be called after async channel unregister.
Xuan Ding (2):
vhost: introduce DMA vchannel unconfiguration
example/vhost: unconfigure DMA vchannel
doc/guides/prog_guide/vhost_lib.rst | 5 ++++ doc/guides/rel_notes/release_22_11.rst | 2 ++ examples/vhost/main.c | 7 +++++ lib/vhost/rte_vhost_async.h | 17 ++++++++++++ lib/vhost/version.map | 3 +++ lib/vhost/vhost.c | 37 ++++++++++++++++++++++++++ 6 files changed, 71 insertions(+)
Signature · 1 line
2.17.1
PATCH 1/2 vhost: introduce DMA vchannel unconfiguration
From: Xuan Ding <[email protected]>
This patch adds a new API rte_vhost_async_dma_unconfigure() to unconfigure DMA vchannels in vhost async data path.
--- doc/guides/prog_guide/vhost_lib.rst | 5 ++++ doc/guides/rel_notes/release_22_11.rst | 2 ++ lib/vhost/rte_vhost_async.h | 17 ++++++++++++ lib/vhost/version.map | 3 +++ lib/vhost/vhost.c | 37 ++++++++++++++++++++++++++ 5 files changed, 64 insertions(+)
doc/guides/prog_guide/vhost_lib.rst +5 -0
@@ -323,6 +323,11 @@ The following is an overview of some key Vhost API functions: Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET, VDPA_DEVICE_TYPE_BLK. +* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)`` + + Clear DMA vChannels finished to use. This function needs to + be called after async unregister. + Vhost-user Implementations --------------------------
doc/guides/rel_notes/release_22_11.rst +2 -0
@@ -55,6 +55,8 @@ New Features Also, make sure to start the actual text at the margin. ======================================================= +* **Added vhost API to unconfigure DMA vchannels.** + Added an API which helps to unconfigure DMA vchannels. Removed Items -------------
lib/vhost/rte_vhost_async.h +17 -0
@@ -266,6 +266,23 @@ rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id, struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count, int *nr_inflight, int16_t dma_id, uint16_t vchan_id); +/** +* @warning +* @b EXPERIMENTAL: this API may change, or be removed, without prior notice. +* +* Unconfigure DMA vChannels in asynchronous data path. +* +* @param dma_id +* the identifier of DMA device +* @param vchan_id +* the identifier of virtual DMA channel +* @return +* 0 on success, and -1 on failure +*/ +__rte_experimental +int +rte_vhost_async_dma_unconfigure (int16_t dma_id,uint16_t vchan_id); + #ifdef __cplusplus }#endif
lib/vhost/version.map +3 -0
@@ -96,6 +96,9 @@ EXPERIMENTAL { rte_vhost_async_try_dequeue_burst; rte_vhost_driver_get_vdpa_dev_type; rte_vhost_clear_queue; + + # added in 22.11 + rte_vhost_async_dma_unconfigure; }; INTERNAL {
lib/vhost/vhost.c +37 -0
@@ -1863,6 +1863,43 @@ rte_vhost_async_channel_unregister_thread_unsafe(int vid, uint16_t queue_id)return 0 ; } +int +rte_vhost_async_dma_unconfigure (int16_t dma_id,uint16_t vchan_id) +{ +struct rte_dma_info info; +void *pkts_cmpl_flag_addr; + +if (!rte_dma_is_valid (dma_id)) { +VHOST_LOG_CONFIG ("dma" ,ERR ,"DMA %d is not found.\n" , dma_id); +return -1 ; + } + +if (rte_dma_info_get (dma_id, &info) !=0 ) { +VHOST_LOG_CONFIG ("dma" ,ERR ,"Fail to get DMA %d information.\n" , dma_id); +return -1 ; + } + +if (vchan_id >= info.max_vchans) { +VHOST_LOG_CONFIG ("dma" ,ERR ,"Invalid DMA %d vChannel %u.\n" , dma_id, vchan_id); +return -1 ; + } + + pkts_cmpl_flag_addr = dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr; +if (pkts_cmpl_flag_addr) { +rte_free (pkts_cmpl_flag_addr); + pkts_cmpl_flag_addr =NULL ; + } + +if (dma_copy_track[dma_id].vchans) { +rte_free (dma_copy_track[dma_id].vchans); + dma_copy_track[dma_id].vchans =NULL ; + } + + dma_copy_track[dma_id].nr_vchans--; + +return 0 ; +} +int rte_vhost_async_dma_configure (int16_t dma_id,uint16_t vchan_id) {
Signature · 1 line
2.17.1
PATCH 2/2 example/vhost: unconfigure DMA vchannel
In reply to [email protected]From: Xuan Ding <[email protected]>
This patch uses rte_vhost_async_dma_unconfigure() API to manually free 'dma_coy_track' array rather than wait for the program to finish before being freed.
--- examples/vhost/main.c | 7 +++++++ 1 file changed, 7 insertions(+)
examples/vhost/main.c +7 -0
@@ -2060,6 +2060,13 @@ main(int argc, char *argv[])RTE_LCORE_FOREACH_WORKER (lcore_id)rte_eal_wait_lcore (lcore_id); +for (i =0 ; i < dma_count; i++) { +if (rte_vhost_async_dma_unconfigure (dmas_id[i],0 ) <0 ) { +RTE_LOG (ERR ,VHOST_PORT ,"Failed to unconfigure DMA in vhost.\n" ); +rte_exit (EXIT_FAILURE ,"Cannot use given DMA device\n" ); + } + } +/* clean up the EAL */ rte_eal_cleanup ();
Signature · 1 line
2.17.1
v2 cover vhost: introduce DMA vchannel unconfiguration
In reply to [email protected]From: Xuan Ding <[email protected]>
This patchset introduces a new API rte_vhost_async_dma_unconfigure() to help user to manually free the DMA vchannel finished to use.
Note: this API should be called after async channel unregister.
v2:
* Add spinlock protection.
* Fix a memory leak issue.
* Refine the doc.
Xuan Ding (2):
vhost: introduce DMA vchannel unconfiguration
examples/vhost: unconfigure DMA vchannel
doc/guides/prog_guide/vhost_lib.rst | 5 ++ doc/guides/rel_notes/release_22_11.rst | 2 + examples/vhost/main.c | 7 +++ lib/vhost/rte_vhost_async.h | 17 +++++++ lib/vhost/version.map | 3 ++ lib/vhost/vhost.c | 69 ++++++++++++++++++++++++-- 6 files changed, 98 insertions(+), 5 deletions(-)
Signature · 1 line
2.17.1
PATCH v2 1/2 vhost: introduce DMA vchannel unconfiguration
From: Xuan Ding <[email protected]>
This patch adds a new API rte_vhost_async_dma_unconfigure() to unconfigure DMA vchannels in vhost async data path.
Lock protection are also added to protect DMA vchannels configuration and unconfiguration from concurrent calls.
--- doc/guides/prog_guide/vhost_lib.rst | 5 ++ doc/guides/rel_notes/release_22_11.rst | 2 + lib/vhost/rte_vhost_async.h | 17 +++++++ lib/vhost/version.map | 3 ++ lib/vhost/vhost.c | 69 ++++++++++++++++++++++++-- 5 files changed, 91 insertions(+), 5 deletions(-)
doc/guides/prog_guide/vhost_lib.rst +5 -0
@@ -323,6 +323,11 @@ The following is an overview of some key Vhost API functions: Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET, VDPA_DEVICE_TYPE_BLK. +* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)`` + + Clear DMA vChannels finished to use. This function needs to + be called after the deregisterration of async path has been finished. + Vhost-user Implementations --------------------------
doc/guides/rel_notes/release_22_11.rst +2 -0
@@ -55,6 +55,8 @@ New Features Also, make sure to start the actual text at the margin. ======================================================= +* **Added vhost API to unconfigure DMA vchannels.** + Added an API which helps to unconfigure DMA vchannels. Removed Items -------------
lib/vhost/rte_vhost_async.h +17 -0
@@ -266,6 +266,23 @@ rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id, struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count, int *nr_inflight, int16_t dma_id, uint16_t vchan_id); +/** +* @warning +* @b EXPERIMENTAL: this API may change, or be removed, without prior notice. +* +* Unconfigure DMA vChannels in asynchronous data path. +* +* @param dma_id +* the identifier of DMA device +* @param vchan_id +* the identifier of virtual DMA channel +* @return +* 0 on success, and -1 on failure +*/ +__rte_experimental +int +rte_vhost_async_dma_unconfigure (int16_t dma_id,uint16_t vchan_id); + #ifdef __cplusplus }#endif
lib/vhost/version.map +3 -0
@@ -96,6 +96,9 @@ EXPERIMENTAL { rte_vhost_async_try_dequeue_burst; rte_vhost_driver_get_vdpa_dev_type; rte_vhost_clear_queue; + + # added in 22.11 + rte_vhost_async_dma_unconfigure; }; INTERNAL {
lib/vhost/vhost.c +64 -5
121 lines not on this page · raw message
Signature · 1 line
2.17.1
Quoted 37 lines
-----Original Message-----
From: Ding, Xuan <[email protected]>
Sent: Tuesday, September 6, 2022 1:22 PM
To: [email protected]; Xia, Chenbo <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; He, Xingguang <[email protected]>; Yang, YvonneX <[email protected]>; Jiang, Cheng1 <[email protected]>; Wang, YuanX <[email protected]>; Ma, WenwuX <[email protected]>; Ding, Xuan <[email protected]>
Subject: [PATCH v2 1/2] vhost: introduce DMA vchannel unconfigurationFrom: Xuan Ding <[email protected]>
This patch adds a new API
rte_vhost_async_dma_unconfigure()to unconfigure DMA vchannels in vhost async data path.Lock protection are also added to protect DMA vchannels configuration and unconfiguration from concurrent calls.
Signed-off-by Xuan Ding <[email protected]>--- doc/guides/prog_guide/vhost_lib.rst | 5 ++ doc/guides/rel_notes/release_22_11.rst | 2 + lib/vhost/rte_vhost_async.h | 17 +++++++ lib/vhost/version.map | 3 ++ lib/vhost/vhost.c | 69 ++++++++++++++++++++++++-- 5 files changed, 91 insertions(+), 5 deletions(-)doc/guides/prog_guide/vhost_lib.rst +0
-0@@ -323,6 +323,11 @@ The following is an overview of some key Vhost APIfunctions:
Get device type of vDPA device, such asVDPA_DEVICE_TYPE_NET,VDPA_DEVICE_TYPE_BLK.
doc/guides/prog_guide/vhost_lib.rst The following is an overview of some key Vhost API functions: · line 329
Quoted 4 lines
doc/guides/prog_guide/vhost_lib.rst +4
-0+* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)`` + + Clear DMA vChannels finished to use. This function needs to + be called after the deregisterration of async path has been finished.
Deregistration
Quoted 10 lines
doc/guides/prog_guide/vhost_lib.rst +1
-0+ Vhost-user Implementations --------------------------doc/guides/rel_notes/release_22_11.rst +0
-0
doc/guides/rel_notes/release_22_11.rst New Features · line 59
Quoted 5 lines
doc/guides/rel_notes/release_22_11.rst +2
-0@@ -55,6 +55,8 @@ New Features Also, make sure to start the actual text at the margin. ======================================================= +* **Added vhost API to unconfigure DMA vchannels.** + Added an API which helps to unconfigure DMA vchannels.
Added XXX for async vhost
Overall LGTM. It seems it needs some rebasing too.
Thanks,
Chenbo
Quoted 204 lines
doc/guides/rel_notes/release_22_11.rst +0
-0Removed Items -------------lib/vhost/rte_vhost_async.h +3
-0@@ -266,6 +266,23 @@ rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id, struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count, int *nr_inflight, int16_t dma_id, uint16_t vchan_id); +/** +* @warning +* @b EXPERIMENTAL: this API may change, or be removed, without prior notice.
+ *
+ * Unconfigure DMA vChannels in asynchronous data path.
+ *
+ * @paramdma_id
+ * the identifier of DMA device
+ * @paramvchan_id
+ * the identifier of virtual DMA channel
+ * @return
+ * 0 on success, and -1 on failure
+ */lib/vhost/rte_vhost_async.h +4
-0+__rte_experimental +int +rte_vhost_async_dma_unconfigure (int16_t dma_id,uint16_t vchan_id); + #ifdef __cplusplus }#endif lib/vhost/version.map +3
-0@@ -96,6 +96,9 @@ EXPERIMENTAL { rte_vhost_async_try_dequeue_burst; rte_vhost_driver_get_vdpa_dev_type; rte_vhost_clear_queue; + + # added in 22.11 + rte_vhost_async_dma_unconfigure; }; INTERNAL {lib/vhost/vhost.c +5
-3@@ -23,6 +23,7 @@struct virtio_net *vhost_devices[RTE_MAX_VHOST_DEVICE ];pthread_mutex_t vhost_dev_lock =PTHREAD_MUTEX_INITIALIZER ; +static rte_spinlock_t vhost_dma_lock =RTE_SPINLOCK_INITIALIZER ;struct vhost_vq_stats_name_off {char name[RTE_VHOST_STATS_NAME_SIZE ];@@ -1870,19 +1871,20 @@ rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id)void *pkts_cmpl_flag_addr;uint16_t max_desc; +rte_spinlock_lock (&vhost_dma_lock);if (!rte_dma_is_valid (dma_id)) {VHOST_LOG_CONFIG ("dma" ,ERR ,"DMA %d is not found.\n" , dma_id); -return -1 ; + goto error; }if (rte_dma_info_get (dma_id, &info) !=0 ) { VHOST_LOG_CONFIG("dma", ERR, "Fail to get DMA %d information.\n", dma_id); -return -1 ; + goto error; }if (vchan_id >= info.max_vchans) {VHOST_LOG_CONFIG ("dma" ,ERR ,"Invalid DMA %d vChannel %u.\n" , dma_id, vchan_id); -return -1 ; + goto error;lib/vhost/vhost.c +1
-1}if (!dma_copy_track[dma_id].vchans) {@@ -1894,7 +1896,7 @@ rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id) VHOST_LOG_CONFIG("dma", ERR, "Failed to allocate vchans for DMA %d vChannel %u.\n", dma_id, vchan_id); -return -1 ; + goto error; }lib/vhost/vhost.c +1
-0dma_copy_track[dma_id].vchans = vchans;@@ -1903,6 +1905,7 @@ rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id)if (dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr) { VHOST_LOG_CONFIG("dma", INFO, "DMA %d vChannel %u already registered.\n", dma_id, vchan_id); +rte_spinlock_unlock (&vhost_dma_lock);return 0 ;lib/vhost/vhost.c +1
-1}@@ -1920,7 +1923,7 @@ rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id) rte_free(dma_copy_track[dma_id].vchans); dma_copy_track[dma_id].vchans =NULL ; } -return -1 ; + goto error; }dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr =
pkts_cmpl_flag_addr;lib/vhost/vhost.c +5
-0@@ -1928,7 +1931,12 @@ rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id) dma_copy_track[dma_id].vchans[vchan_id].ring_mask = max_desc - 1; dma_copy_track[dma_id].nr_vchans++; +rte_spinlock_unlock (&vhost_dma_lock);return 0 ; + +error: +rte_spinlock_unlock (&vhost_dma_lock); +return -1 ; }int
lib/vhost/vhost.c +20
-0@@ -2117,5 +2125,56 @@ int rte_vhost_vring_stats_reset(int vid, uint16_t queue_id)return 0 ; } +int +rte_vhost_async_dma_unconfigure (int16_t dma_id,uint16_t vchan_id) +{ +struct rte_dma_info info; +uint16_t max_desc; +int i; + +rte_spinlock_lock (&vhost_dma_lock); +if (!rte_dma_is_valid (dma_id)) { +VHOST_LOG_CONFIG ("dma" ,ERR ,"DMA %d is not found.\n" , dma_id); + goto error; + } + +if (rte_dma_info_get (dma_id, &info) !=0 ) { + VHOST_LOG_CONFIG("dma", ERR, "Fail to get DMA %d information.\n", dma_id); + goto error; + } + +if (vchan_id >= info.max_vchans) { + VHOST_LOG_CONFIG("dma", ERR, "Invalid DMA %d vChannel %u.\n",
dma_id,vchan_id);lib/vhost/vhost.c +6
-0+ goto error; + } + + max_desc = info.max_desc; +for (i =0 ; i < max_desc; i++) { + if(dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr[i] != NULL) { +
rte_free(dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr
[i]);
+
dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr[i] =
NULL;lib/vhost/vhost.c +4
-0+ } + } + + if (dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr !=NULL) {
+
rte_free(dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr
);lib/vhost/vhost.c +1
-0+ dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr =NULL;
lib/vhost/vhost.c +17
-1+ } + +if (dma_copy_track[dma_id].vchans !=NULL ) { +rte_free (dma_copy_track[dma_id].vchans); + dma_copy_track[dma_id].vchans =NULL ; + } + + dma_copy_track[dma_id].nr_vchans--; + +rte_spinlock_unlock (&vhost_dma_lock); +return 0 ; + +error: +rte_spinlock_unlock (&vhost_dma_lock); +return -1 ; +} +RTE_LOG_REGISTER_SUFFIX (vhost_config_log_level, config,INFO );RTE_LOG_REGISTER_SUFFIX (vhost_data_log_level, data,WARNING ); --2.17.1
Hi Chenbo,
Thanks for your comments, please see replies inline.
Quoted 49 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Monday, September 26, 2022 2:07 PM
To: Ding, Xuan <[email protected]>; [email protected]
Cc: [email protected]; Hu, Jiayu <[email protected]>; He, Xingguang <[email protected]>; Yang, YvonneX <[email protected]>; Jiang, Cheng1 <[email protected]>; Wang, YuanX <[email protected]>; Ma, WenwuX <[email protected]>
Subject: RE: [PATCH v2 1/2] vhost: introduce DMA vchannel unconfigurationQuoted 39 lines
-----Original Message-----
From: Ding, Xuan <[email protected]>
Sent: Tuesday, September 6, 2022 1:22 PM
To: [email protected]; Xia, Chenbo <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; He, Xingguang <[email protected]>; Yang, YvonneX <[email protected]>; Jiang,
Cheng1 <[email protected]>; Wang, YuanX <[email protected]>; Ma, WenwuX <[email protected]>; Ding, Xuan <[email protected]>
Subject: [PATCH v2 1/2] vhost: introduce DMA vchannel unconfigurationFrom: Xuan Ding <[email protected]>
This patch adds a new API
rte_vhost_async_dma_unconfigure()to unconfigure DMA vchannels in vhost async data path.Lock protection are also added to protect DMA vchannels configuration and unconfiguration from concurrent calls.
Signed-off-by Xuan Ding <[email protected]>--- doc/guides/prog_guide/vhost_lib.rst | 5 ++ doc/guides/rel_notes/release_22_11.rst | 2 + lib/vhost/rte_vhost_async.h | 17 +++++++ lib/vhost/version.map | 3 ++ lib/vhost/vhost.c | 69 ++++++++++++++++++++++++-- 5 files changed, 91 insertions(+), 5 deletions(-)doc/guides/prog_guide/vhost_lib.rst +0
-0@@ -323,6 +323,11 @@ The following is an overview of some key VhostAPI
functions:
Get device type of vDPA device, such asVDPA_DEVICE_TYPE_NET,VDPA_DEVICE_TYPE_BLK.
Quoted 6 lines
doc/guides/prog_guide/vhost_lib.rst The following is an overview of some key Vhost API functions:
Quoted 4 lines
doc/guides/prog_guide/vhost_lib.rst +4
-0+* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)`` + + Clear DMA vChannels finished to use. This function needs to be + called after the deregisterration of async path has been finished.Deregistration
Thanks for your catch.
Quoted 14 lines
Quoted 14 lines
doc/guides/prog_guide/vhost_lib.rst +1
-0+ Vhost-user Implementations --------------------------doc/guides/rel_notes/release_22_11.rst +0
-0@@ -55,6 +55,8 @@ New Features Also, make sure to start the actual text at the margin. =======================================================
Quoted 4 lines
doc/guides/rel_notes/release_22_11.rst New Features · line 59
Quoted 2 lines
doc/guides/rel_notes/release_22_11.rst +2
-0+* **Added vhost API to unconfigure DMA vchannels.** + Added an API which helps to unconfigure DMA vchannels.Added XXX for async vhost
Good idea.
Quoted 2 lines
Overall LGTM. It seems it needs some rebasing too.
I'm preparing v3 patch series, please see next version.
Regards,
Xuan
Quoted 221 lines
Thanks,
ChenboQuoted 53 lines
doc/guides/rel_notes/release_22_11.rst +0
-0Removed Items -------------lib/vhost/rte_vhost_async.h +3
-0@@ -266,6 +266,23 @@ rte_vhost_async_try_dequeue_burst(int vid,uint16_t queue_id, struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count, int *nr_inflight, int16_t dma_id, uint16_t vchan_id); +/** +* @warning +* @b EXPERIMENTAL: this API may change, or be removed, without prior notice.
+ *
+ * Unconfigure DMA vChannels in asynchronous data path.
+ *
+ * @paramdma_id
+ * the identifier of DMA device
+ * @paramvchan_id
+ * the identifier of virtual DMA channel
+ * @return
+ * 0 on success, and -1 on failure
+ */lib/vhost/rte_vhost_async.h +4
-0+__rte_experimental +int +rte_vhost_async_dma_unconfigure (int16_t dma_id,uint16_t vchan_id); + #ifdef __cplusplus }#endif lib/vhost/version.map index renamed from lib/vhost/version.map +0
-0
18574346d5..013a6bcc42100644lib/vhost/version.map +3
-0@@ -96,6 +96,9 @@ EXPERIMENTAL { rte_vhost_async_try_dequeue_burst; rte_vhost_driver_get_vdpa_dev_type; rte_vhost_clear_queue; + + # added in 22.11 + rte_vhost_async_dma_unconfigure; }; INTERNAL {lib/vhost/vhost.c index renamed from lib/vhost/vhost.c +0
-0
60cb05a0ff..273616da11100644lib/vhost/vhost.c line 29
Quoted 7 lines
lib/vhost/vhost.c +1
-0@@ -23,6 +23,7 @@struct virtio_net *vhost_devices[RTE_MAX_VHOST_DEVICE ];pthread_mutex_t vhost_dev_lock =PTHREAD_MUTEX_INITIALIZER ; +static rte_spinlock_t vhost_dma_lock =RTE_SPINLOCK_INITIALIZER ;struct vhost_vq_stats_name_off { char name[RTE_VHOST_STATS_NAME_SIZE]; @@ -1870,19 +1871,20@@
lib/vhost/vhost.c rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id)
Quoted 7 lines
rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id)lib/vhost/vhost.c +1
-0void *pkts_cmpl_flag_addr;uint16_t max_desc; +rte_spinlock_lock (&vhost_dma_lock);if (!rte_dma_is_valid (dma_id)) { VHOST_LOG_CONFIG("dma", ERR, "DMA %d is not found.\n",
dma_id);lib/vhost/vhost.c rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id)
Quoted 6 lines
lib/vhost/vhost.c +1
-1-return -1 ; + goto error; }if (rte_dma_info_get (dma_id, &info) !=0 ) { VHOST_LOG_CONFIG("dma", ERR, "Fail to get DMA %dinformation.\n",
lib/vhost/vhost.c rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id)
Quoted 7 lines
dma_id);lib/vhost/vhost.c +1
-1-return -1 ; + goto error; }if (vchan_id >= info.max_vchans) { VHOST_LOG_CONFIG("dma", ERR, "Invalid DMA %dvChannel %u.\n",
lib/vhost/vhost.c
Quoted 9 lines
dma_id,vchan_id);lib/vhost/vhost.c +1
-1-return -1 ; + goto error; } if (!dma_copy_track[dma_id].vchans) { @@ -1894,7 +1896,7 @@
rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id)lib/vhost/vhost.c +0
-0VHOST_LOG_CONFIG("dma", ERR, "Failed to allocate vchans for DMA %dvChannel %u.\n",
lib/vhost/vhost.c rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id)
Quoted 6 lines
lib/vhost/vhost.c +1
-1dma_id, vchan_id); -return -1 ; + goto error; } dma_copy_track[dma_id].vchans = vchans; @@ -1903,6+1905,7 @@
Quoted 2 lines
rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id)
if (dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr){
Quoted 1 line
VHOST_LOG_CONFIG("dma", INFO, "DMA %d vChannel %u
already
Quoted 15 lines
registered.\n",
lib/vhost/vhost.c +2
-1dma_id, vchan_id); +rte_spinlock_unlock (&vhost_dma_lock);return 0 ; }@@ -1920,7 +1923,7 @@ rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id) rte_free(dma_copy_track[dma_id].vchans); dma_copy_track[dma_id].vchans =NULL ; } -return -1 ; + goto error; }Quoted 4 lines
dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr =
pkts_cmpl_flag_addr; @@ -1928,7 +1931,12 @@rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id)
dma_copy_track[dma_id].vchans[vchan_id].ring_mask =max_desc-1;
Quoted 16 lines
lib/vhost/vhost.c +5
-0dma_copy_track[dma_id].nr_vchans++; +rte_spinlock_unlock (&vhost_dma_lock);return 0 ; + +error: +rte_spinlock_unlock (&vhost_dma_lock); +return -1 ; }int @@ -2117,5 +2125,56 @@ int rte_vhost_vring_stats_reset(int vid,
uint16_tqueue_id)lib/vhost/vhost.c +0
-0return 0 ; }lib/vhost/vhost.c int rte_vhost_vring_stats_reset(int vid, uint16_t queue_id)
Quoted 10 lines
lib/vhost/vhost.c +9
-0+int +rte_vhost_async_dma_unconfigure (int16_t dma_id,uint16_t vchan_id) { +struct rte_dma_info info; +uint16_t max_desc; +int i; + +rte_spinlock_lock (&vhost_dma_lock); +if (!rte_dma_is_valid (dma_id)) { + VHOST_LOG_CONFIG("dma", ERR, "DMA %d is not found.\n",
dma_id);lib/vhost/vhost.c int rte_vhost_vring_stats_reset(int vid, uint16_t queue_id)
Quoted 11 lines
lib/vhost/vhost.c +5
-0+ goto error; + } + +if (rte_dma_info_get (dma_id, &info) !=0 ) { + VHOST_LOG_CONFIG("dma", ERR, "Fail to get DMA %dinformation.\n",
dma_id);lib/vhost/vhost.c +5
-0+ goto error; + } + +if (vchan_id >= info.max_vchans) { + VHOST_LOG_CONFIG("dma", ERR, "Invalid DMA %dvChannel %u.\n",
Quoted 11 lines
dma_id,vchan_id);lib/vhost/vhost.c +6
-0+ goto error; + } + + max_desc = info.max_desc; +for (i =0 ; i < max_desc; i++) { + if(dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr[i] != NULL) {
+rte_free(dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag
_addrlib/vhost/vhost.c int rte_vhost_vring_stats_reset(int vid, uint16_t queue_id)
Quoted 8 lines
[i]);
+
dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr[i] =
NULL;lib/vhost/vhost.c +4
-0+ } + } + + if(dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr !=
Quoted 3 lines
NULL) {
+rte_free(dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag
_addrQuoted 2 lines
);
+dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr =
Quoted 22 lines
NULL;
lib/vhost/vhost.c +17
-0+ } + +if (dma_copy_track[dma_id].vchans !=NULL ) { +rte_free (dma_copy_track[dma_id].vchans); + dma_copy_track[dma_id].vchans =NULL ; + } + + dma_copy_track[dma_id].nr_vchans--; + +rte_spinlock_unlock (&vhost_dma_lock); +return 0 ; + +error: +rte_spinlock_unlock (&vhost_dma_lock); +return -1 ; +} +RTE_LOG_REGISTER_SUFFIX (vhost_config_log_level, config,INFO );
RTE_LOG_REGISTER_SUFFIX(vhost_data_log_level, data, WARNING); --
2.17.1
PATCH v2 2/2 examples/vhost: unconfigure DMA vchannel
In reply to [email protected]From: Xuan Ding <[email protected]>
This patch applies rte_vhost_async_dma_unconfigure() API to manually free 'dma_copy_track' array instead of waiting until the program ends to be released.
--- examples/vhost/main.c | 7 +++++++ 1 file changed, 7 insertions(+)
examples/vhost/main.c +7 -0
@@ -1624,6 +1624,13 @@ destroy_device(int vid) dma_bind[vid].dmas[VIRTIO_TXQ ].async_enabled = false; } +for (i =0 ; i < dma_count; i++) { +if (rte_vhost_async_dma_unconfigure (dmas_id[i],0 ) <0 ) { +RTE_LOG (ERR ,VHOST_PORT ,"Failed to unconfigure DMA in vhost.\n" ); +rte_exit (EXIT_FAILURE ,"Cannot use given DMA device\n" ); + } + } +rte_free (vdev); }
Signature · 1 line
2.17.1
v3 cover vhost: introduce DMA vchannel unconfiguration
In reply to [email protected]From: Xuan Ding <[email protected]>
This patchset introduces a new API rte_vhost_async_dma_unconfigure() to help user to manually free the DMA vchannel finished to use.
Note: this API should be called after async channel unregister.
v3:
* Rebase to latest DPDK.
* Refine some descriptions in the doc.
* Fix one bug in the vhost example.
v2:
* Add spinlock protection.
* Fix a memory leak issue.
* Refine the doc.
Xuan Ding (2):
vhost: introduce DMA vchannel unconfiguration
examples/vhost: unconfigure DMA vchannel
doc/guides/prog_guide/vhost_lib.rst | 6 +++ doc/guides/rel_notes/release_22_11.rst | 3 ++ examples/vhost/main.c | 45 ++++++++++++----- examples/vhost/main.h | 1 + lib/vhost/rte_vhost_async.h | 18 +++++++ lib/vhost/version.map | 3 ++ lib/vhost/vhost.c | 69 ++++++++++++++++++++++++-- 7 files changed, 127 insertions(+), 18 deletions(-)
Signature · 1 line
2.17.1
PATCH v3 1/2 vhost: introduce DMA vchannel unconfiguration
From: Xuan Ding <[email protected]>
This patch adds a new API rte_vhost_async_dma_unconfigure() to unconfigure DMA vchannels in vhost async data path.
Lock protection are also added to protect DMA vchannels configuration and unconfiguration from concurrent calls.
--- doc/guides/prog_guide/vhost_lib.rst | 6 +++ doc/guides/rel_notes/release_22_11.rst | 3 ++ lib/vhost/rte_vhost_async.h | 18 +++++++ lib/vhost/version.map | 3 ++ lib/vhost/vhost.c | 69 ++++++++++++++++++++++++-- 5 files changed, 94 insertions(+), 5 deletions(-)
doc/guides/prog_guide/vhost_lib.rst +6 -0
@@ -323,6 +323,12 @@ The following is an overview of some key Vhost API functions: Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET, VDPA_DEVICE_TYPE_BLK. +* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)`` + + Clean DMA vChannels finished to use. This function needs to + be called after the deregistration of async DMA vchannel + has been finished. + Vhost-user Implementations --------------------------
doc/guides/rel_notes/release_22_11.rst +3 -0
@@ -67,6 +67,9 @@ New Features * Added support to set device link down/up. +* **Added DMA vChannel unconfiguration for async vhost.** + + * Added support to unconfigure DMA vChannels that have been unregistered. Removed Items -------------
lib/vhost/rte_vhost_async.h +18 -0
@@ -266,6 +266,24 @@ rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id, struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count, int *nr_inflight, int16_t dma_id, uint16_t vchan_id); +/** +* @warning +* @b EXPERIMENTAL: this API may change, or be removed, without prior notice. +* +* Unconfigure DMA vChannels in asynchronous data path. +* This function should be called after the DMA vChannel has been unregistered. +* +* @param dma_id +* the identifier of DMA device +* @param vchan_id +* the identifier of virtual DMA channel +* @return +* 0 on success, and -1 on failure +*/ +__rte_experimental +int +rte_vhost_async_dma_unconfigure (int16_t dma_id,uint16_t vchan_id); + #ifdef __cplusplus }#endif
lib/vhost/version.map +3 -0
@@ -96,6 +96,9 @@ EXPERIMENTAL { rte_vhost_async_try_dequeue_burst; rte_vhost_driver_get_vdpa_dev_type; rte_vhost_clear_queue; + + # added in 22.11 + rte_vhost_async_dma_unconfigure; }; INTERNAL {
lib/vhost/vhost.c +64 -5
121 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v3 2/2 examples/vhost: unconfigure DMA vchannel
In reply to [email protected]From: Xuan Ding <[email protected]>
This patch applies rte_vhost_async_dma_unconfigure() API to manually free DMA vchannels instead of waiting until the program ends to be released.
--- examples/vhost/main.c | 45 ++++++++++++++++++++++++++++++------------- examples/vhost/main.h | 1 + 2 files changed, 33 insertions(+), 13 deletions(-)
examples/vhost/main.c +32 -13
@@ -1558,6 +1558,28 @@ vhost_clear_queue(struct vhost_dev *vdev, uint16_t queue_id) } } +static void +vhost_clear_async (struct vhost_dev *vdev,int vid,uint16_t queue_id) +{ +int16_t dma_id; +uint16_t ref_count; + +if (dma_bind[vid].dmas[queue_id].async_enabled) { +vhost_clear_queue (vdev, queue_id); +rte_vhost_async_channel_unregister (vid, queue_id); + dma_bind[vid].dmas[queue_id].async_enabled = false; + } + + dma_id = dma_bind[vid2socketid[vdev->vid]].dmas[queue_id].dev_id; + dma_bind[vid2socketid[vdev->vid]].dmas[queue_id].dma_ref_count--; + ref_count = dma_bind[vid2socketid[vdev->vid]].dmas[queue_id].dma_ref_count; + +if (ref_count ==0 && dma_id !=INVALID_DMA_ID ) { +if (rte_vhost_async_dma_unconfigure (dma_id,0 ) <0 ) +RTE_LOG (ERR ,VHOST_PORT ,"Failed to unconfigure DMA in vhost.\n" ); + } +} +/* * Remove a device from the specific data core linked list and from the * main linked list. Synchronization occurs through the use of the
@@ -1614,17 +1636,8 @@ destroy_device(int vid)"(%d) device has been removed from data core\n" , vdev->vid); -if (dma_bind[vid].dmas[VIRTIO_RXQ ].async_enabled) { -vhost_clear_queue (vdev,VIRTIO_RXQ ); -rte_vhost_async_channel_unregister (vid,VIRTIO_RXQ ); - dma_bind[vid].dmas[VIRTIO_RXQ ].async_enabled = false; - } - -if (dma_bind[vid].dmas[VIRTIO_TXQ ].async_enabled) { -vhost_clear_queue (vdev,VIRTIO_TXQ ); -rte_vhost_async_channel_unregister (vid,VIRTIO_TXQ ); - dma_bind[vid].dmas[VIRTIO_TXQ ].async_enabled = false; - } +vhost_clear_async (vdev, vid,VIRTIO_RXQ ); +vhost_clear_async (vdev, vid,VIRTIO_TXQ );rte_free (vdev); }
@@ -1673,14 +1686,19 @@ vhost_async_channel_register(int vid)if (dma_bind[vid2socketid[vid]].dmas[VIRTIO_RXQ ].dev_id !=INVALID_DMA_ID ) { rx_ret =rte_vhost_async_channel_register (vid,VIRTIO_RXQ ); -if (rx_ret ==0 ) +if (rx_ret ==0 ) { dma_bind[vid2socketid[vid]].dmas[VIRTIO_RXQ ].async_enabled = true; + dma_bind[vid2socketid[vid]].dmas[VIRTIO_RXQ ].dma_ref_count++; + } + }if (dma_bind[vid2socketid[vid]].dmas[VIRTIO_TXQ ].dev_id !=INVALID_DMA_ID ) { tx_ret =rte_vhost_async_channel_register (vid,VIRTIO_TXQ ); -if (tx_ret ==0 ) +if (tx_ret ==0 ) { dma_bind[vid2socketid[vid]].dmas[VIRTIO_TXQ ].async_enabled = true; + dma_bind[vid2socketid[vid]].dmas[VIRTIO_TXQ ].dma_ref_count++; + } }return rx_ret | tx_ret;
@@ -1886,6 +1904,7 @@ reset_dma(void)for (j =0 ; j <RTE_MAX_QUEUES_PER_PORT *2 ; j++) { dma_bind[i].dmas[j].dev_id =INVALID_DMA_ID ; dma_bind[i].dmas[j].async_enabled = false; + dma_bind[i].dmas[j].dma_ref_count =0 ; } }
examples/vhost/main.h +1 -0
@@ -96,6 +96,7 @@ struct dma_info {struct rte_pci_addr addr;int16_t dev_id;bool async_enabled; +uint16_t dma_ref_count; };struct dma_for_vhost {
Signature · 1 line
2.17.1
RE: [PATCH v3 2/2] examples/vhost: unconfigure DMA vchannel
Quoted 37 lines
-----Original Message-----
From: Ding, Xuan <[email protected]>
Sent: Thursday, September 29, 2022 9:33 AM
To: [email protected]; Xia, Chenbo <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; He, Xingguang <[email protected]>; Yang, YvonneX <[email protected]>; Jiang, Cheng1 <[email protected]>; Wang, YuanX <[email protected]>; Ma, WenwuX <[email protected]>; Ding, Xuan <[email protected]>
Subject: [PATCH v3 2/2] examples/vhost: unconfigure DMA vchannelFrom: Xuan Ding <[email protected]>
This patch applies
rte_vhost_async_dma_unconfigure()API to manually free DMA vchannels instead of waiting until the program ends to be released.Signed-off-by Xuan Ding <[email protected]>--- examples/vhost/main.c | 45 ++++++++++++++++++++++++++++++------------- examples/vhost/main.h | 1 + 2 files changed, 33 insertions(+), 13 deletions(-)examples/vhost/main.c +6
-0@@ -1558,6 +1558,28 @@ vhost_clear_queue(struct vhost_dev *vdev, uint16_t queue_id) } } +static void +vhost_clear_async (struct vhost_dev *vdev,int vid,uint16_t queue_id) +{ +int16_t dma_id; +uint16_t ref_count; +
examples/vhost/main.c vhost_clear_queue(struct vhost_dev *vdev, uint16_t · line 1577
Quoted 10 lines
examples/vhost/main.c +9
-0+if (dma_bind[vid].dmas[queue_id].async_enabled) { +vhost_clear_queue (vdev, queue_id); +rte_vhost_async_channel_unregister (vid, queue_id); + dma_bind[vid].dmas[queue_id].async_enabled = false; + } + + dma_id = dma_bind[vid2socketid[vdev->vid]].dmas[queue_id].dev_id; + dma_bind[vid2socketid[vdev->vid]].dmas[queue_id].dma_ref_count--; + ref_count = dma_bind[vid2socketid[vdev- >vid]].dmas[queue_id].dma_ref_count;
Above two lines could be combined like 'ref_count = --XXXX'
But I wonder is this dma_ref_count really needed? It's always 0/1 like Async_enabled, no? If it target at multi-port sharing same one dma case, it should not be implemented like this.
Thanks,
Chenbo
Quoted 81 lines
examples/vhost/main.c +7
-0+ +if (ref_count ==0 && dma_id !=INVALID_DMA_ID ) { +if (rte_vhost_async_dma_unconfigure (dma_id,0 ) <0 ) + RTE_LOG(ERR, VHOST_PORT, "Failed to unconfigure DMA in vhost.\n"); + } +} +/*
* Remove a device from the specific data core linked list and from the
* main linked list. Synchronization occurs through the use of theexamples/vhost/main.c +7
-13@@ -1614,17 +1636,8 @@ destroy_device(int vid)"(%d) device has been removed from data core\n" , vdev->vid); -if (dma_bind[vid].dmas[VIRTIO_RXQ ].async_enabled) { -vhost_clear_queue (vdev,VIRTIO_RXQ ); -rte_vhost_async_channel_unregister (vid,VIRTIO_RXQ ); - dma_bind[vid].dmas[VIRTIO_RXQ ].async_enabled = false; - } - -if (dma_bind[vid].dmas[VIRTIO_TXQ ].async_enabled) { -vhost_clear_queue (vdev,VIRTIO_TXQ ); -rte_vhost_async_channel_unregister (vid,VIRTIO_TXQ ); - dma_bind[vid].dmas[VIRTIO_TXQ ].async_enabled = false; - } +vhost_clear_async (vdev, vid,VIRTIO_RXQ ); +vhost_clear_async (vdev, vid,VIRTIO_TXQ );rte_free (vdev); }@@ -1673,14 +1686,19 @@ vhost_async_channel_register(int vid)if (dma_bind[vid2socketid[vid]].dmas[VIRTIO_RXQ ].dev_id !=INVALID_DMA_ID ) { rx_ret =rte_vhost_async_channel_register (vid,VIRTIO_RXQ ); -if (rx_ret ==0 ) +if (rx_ret ==0 ) { dma_bind[vid2socketid[vid]].dmas[VIRTIO_RXQ ].async_enabled = true; + dma_bind[vid2socketid[vid]].dmas[VIRTIO_RXQ ].dma_ref_count++; + } + }if (dma_bind[vid2socketid[vid]].dmas[VIRTIO_TXQ ].dev_id !=INVALID_DMA_ID ) { tx_ret =rte_vhost_async_channel_register (vid,VIRTIO_TXQ ); -if (tx_ret ==0 ) +if (tx_ret ==0 ) {dma_bind[vid2socketid[vid]].dmas[VIRTIO_TXQ].async_enabled = true; + dma_bind[vid2socketid[vid]].dmas[VIRTIO_TXQ].dma_ref_count++;examples/vhost/main.c +2
-0+ } }return rx_ret | tx_ret;@@ -1886,6 +1904,7 @@ reset_dma(void)for (j =0 ; j <RTE_MAX_QUEUES_PER_PORT *2 ; j++) { dma_bind[i].dmas[j].dev_id =INVALID_DMA_ID ; dma_bind[i].dmas[j].async_enabled = false; + dma_bind[i].dmas[j].dma_ref_count =0 ; } }examples/vhost/main.h +1
-1@@ -96,6 +96,7 @@ struct dma_info {struct rte_pci_addr addr;int16_t dev_id;bool async_enabled; +uint16_t dma_ref_count; };struct dma_for_vhost { --2.17.1
RE: [PATCH v3 2/2] examples/vhost: unconfigure DMA vchannel
Hi Chenbo,
Quoted 63 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Thursday, September 29, 2022 4:28 PM
To: Ding, Xuan <[email protected]>; [email protected]
Cc: [email protected]; Hu, Jiayu <[email protected]>; He, Xingguang <[email protected]>; Yang, YvonneX <[email protected]>; Jiang, Cheng1 <[email protected]>; Wang, YuanX <[email protected]>; Ma, WenwuX <[email protected]>
Subject: RE: [PATCH v3 2/2] examples/vhost: unconfigure DMA vchannelQuoted 8 lines
-----Original Message-----
From: Ding, Xuan <[email protected]>
Sent: Thursday, September 29, 2022 9:33 AM
To: [email protected]; Xia, Chenbo <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; He, Xingguang <[email protected]>; Yang, YvonneX <[email protected]>; Jiang,
Cheng1 <[email protected]>; Wang, YuanXQuoted 1 line
Ma, WenwuX <[email protected]>; Ding, Xuan
Quoted 42 lines
Subject: [PATCH v3 2/2] examples/vhost: unconfigure DMA vchannel
From: Xuan Ding <[email protected]>
This patch applies
rte_vhost_async_dma_unconfigure()API to manually free DMA vchannels instead of waiting until the program ends to be released.Signed-off-by Xuan Ding <[email protected]>--- examples/vhost/main.c | 45 ++++++++++++++++++++++++++++++------------- examples/vhost/main.h | 1 + 2 files changed, 33 insertions(+), 13 deletions(-)examples/vhost/main.c index renamed from examples/vhost/main.c +0
-0
0fa4753e70..32f396d88a100644examples/vhost/main.c +0
-0@@ -1558,6 +1558,28 @@ vhost_clear_queue(struct vhost_dev *vdev,
uint16_tqueue_id)examples/vhost/main.c +13
-0} } +static void +vhost_clear_async (struct vhost_dev *vdev,int vid,uint16_t queue_id) +{ +int16_t dma_id; +uint16_t ref_count; + +if (dma_bind[vid].dmas[queue_id].async_enabled) { +vhost_clear_queue (vdev, queue_id); +rte_vhost_async_channel_unregister (vid, queue_id); + dma_bind[vid].dmas[queue_id].async_enabled = false; + } + + dma_id = dma_bind[vid2socketid[vdev-vid]].dmas[queue_id].dev_id;
examples/vhost/main.c +1
-0+ dma_bind[vid2socketid[vdev-vid]].dmas[queue_id].dma_ref_count--;
examples/vhost/main.c +1
-0+ ref_count = dma_bind[vid2socketid[vdev-
Quoted 7 lines
Quoted 1 line
Quoted 1 line
vid]].dmas[queue_id].dma_ref_count;
Above two lines could be combined like '
ref_count= --XXXX'But I wonder is this
dma_ref_countreally needed? It's always 0/1 likeAsync_enabled, no? If it target at multi-port sharing same one dma case, it should not be implemented like this.
Yes, the dma_ref_count here is for multi-port sharing same one dma case. The intention was to unconfigure dma device when it is no longer used by any ports.
I will check the implementation.
Thanks,
Xuan
Quoted 90 lines
Thanks,
Chenboexamples/vhost/main.c vhost_clear_queue(struct vhost_dev *vdev, uint16_t queue_id)
Quoted 4 lines
examples/vhost/main.c +4
-0+ +if (ref_count ==0 && dma_id !=INVALID_DMA_ID ) { +if (rte_vhost_async_dma_unconfigure (dma_id,0 ) <0 ) + RTE_LOG(ERR, VHOST_PORT, "Failed to unconfigureDMA in
Quoted 28 lines
vhost.\n");
examples/vhost/main.c +5
-11+ } +} +/* * Remove a device from the specific data core linked list and from the * main linked list. Synchronization occurs through the use of the @@ -1614,17 +1636,8 @@ destroy_device(int vid)"(%d) device has been removed from data core\n" , vdev->vid); -if (dma_bind[vid].dmas[VIRTIO_RXQ ].async_enabled) { -vhost_clear_queue (vdev,VIRTIO_RXQ ); -rte_vhost_async_channel_unregister (vid,VIRTIO_RXQ ); - dma_bind[vid].dmas[VIRTIO_RXQ ].async_enabled = false; - } - -if (dma_bind[vid].dmas[VIRTIO_TXQ ].async_enabled) { -vhost_clear_queue (vdev,VIRTIO_TXQ ); -rte_vhost_async_channel_unregister (vid,VIRTIO_TXQ ); - dma_bind[vid].dmas[VIRTIO_TXQ ].async_enabled = false; - } +vhost_clear_async (vdev, vid,VIRTIO_RXQ ); +vhost_clear_async (vdev, vid,VIRTIO_TXQ );rte_free (vdev); }examples/vhost/main.c vhost_async_channel_register(int vid) · line 1689
Quoted 4 lines
examples/vhost/main.c +0
-0@@ -1673,14 +1686,19 @@ vhost_async_channel_register(int vid)if (dma_bind[vid2socketid[vid]].dmas[VIRTIO_RXQ ].dev_id !=INVALID_DMA_ID ) { rx_ret = rte_vhost_async_channel_register(vid,
VIRTIO_RXQ);examples/vhost/main.c vhost_async_channel_register(int vid)
Quoted 4 lines
examples/vhost/main.c +1
-1-if (rx_ret ==0 ) +if (rx_ret ==0 ) { dma_bind[vid2socketid[vid]].dmas[VIRTIO_RXQ].async_enabled =true;
examples/vhost/main.c vhost_async_channel_register(int vid) · line 1701
Quoted 9 lines
examples/vhost/main.c +3
-0+ dma_bind[vid2socketid[vid]].dmas[VIRTIO_RXQ ].dma_ref_count++; + } + }if (dma_bind[vid2socketid[vid]].dmas[VIRTIO_TXQ ].dev_id !=INVALID_DMA_ID ) { tx_ret = rte_vhost_async_channel_register(vid,
VIRTIO_TXQ);Quoted 4 lines
examples/vhost/main.c +1
-1-if (tx_ret ==0 ) +if (tx_ret ==0 ) {dma_bind[vid2socketid[vid]].dmas[VIRTIO_TXQ].async_enabled =
true;
Quoted 28 lines
+
dma_bind[vid2socketid[vid]].dmas[VIRTIO_TXQ].dma_ref_count++;examples/vhost/main.c +2
-0+ } }return rx_ret | tx_ret;@@ -1886,6 +1904,7 @@ reset_dma(void)for (j =0 ; j <RTE_MAX_QUEUES_PER_PORT *2 ; j++) { dma_bind[i].dmas[j].dev_id =INVALID_DMA_ID ; dma_bind[i].dmas[j].async_enabled = false; + dma_bind[i].dmas[j].dma_ref_count =0 ; } }examples/vhost/main.h index renamed from examples/vhost/main.h +0
-0
2fcb8376c5..2b2cf828d3100644examples/vhost/main.h +1
-1@@ -96,6 +96,7 @@ struct dma_info {struct rte_pci_addr addr;int16_t dev_id;bool async_enabled; +uint16_t dma_ref_count; };struct dma_for_vhost { --2.17.1
v4 cover vhost: introduce DMA vchannel unconfiguration
In reply to [email protected]From: Xuan Ding <[email protected]>
This patchset introduces a new API rte_vhost_async_dma_unconfigure() to help user to manually free DMA vchannels finished to use.
Note: this API should be called after async channel unregister.
v4:
* Rebase to 22.11 rc1.
* Fix the usage of 'dma_ref_count' to make sure the specified DMA device is not used by any vhost ports before unconfiguration.
v3:
* Rebase to latest DPDK.
* Refine some descriptions in the doc.
* Fix one bug in the vhost example.
v2:
* Add spinlock protection.
* Fix a memory leak issue.
* Refine the doc.
Xuan Ding (2):
vhost: introduce DMA vchannel unconfiguration
examples/vhost: unconfigure DMA vchannel
doc/guides/prog_guide/vhost_lib.rst | 6 +++ doc/guides/rel_notes/release_22_11.rst | 4 ++ examples/vhost/main.c | 38 +++++++++----- lib/vhost/rte_vhost_async.h | 18 +++++++ lib/vhost/version.map | 3 ++ lib/vhost/vhost.c | 69 ++++++++++++++++++++++++-- 6 files changed, 120 insertions(+), 18 deletions(-)
Signature · 1 line
2.17.1
PATCH v4 1/2 vhost: introduce DMA vchannel unconfiguration
From: Xuan Ding <[email protected]>
Add a new API rte_vhost_async_dma_unconfigure() to unconfigure DMA vchannels in vhost async data path. Lock protection are also added to protect DMA vchannels configuration and unconfiguration from concurrent calls.
--- doc/guides/prog_guide/vhost_lib.rst | 6 +++ doc/guides/rel_notes/release_22_11.rst | 4 ++ lib/vhost/rte_vhost_async.h | 18 +++++++ lib/vhost/version.map | 3 ++ lib/vhost/vhost.c | 69 ++++++++++++++++++++++++-- 5 files changed, 95 insertions(+), 5 deletions(-)
doc/guides/prog_guide/vhost_lib.rst +6 -0
@@ -323,6 +323,12 @@ The following is an overview of some key Vhost API functions: Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET, VDPA_DEVICE_TYPE_BLK. +* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)`` + + Clean DMA vChannels finished to use. This function needs to + be called after the deregistration of async DMA vchannel + has been finished. + Vhost-user Implementations --------------------------
doc/guides/rel_notes/release_22_11.rst +4 -0
@@ -236,6 +236,10 @@ New Features strings $dpdk_binary_or_driver | sed -n 's/^PMD_INFO_STRING= //p' +* **Added DMA vChannel unconfiguration for async vhost.** + + * Added support to unconfigure DMA vChannels that have been unregistered. + Removed Items -------------
lib/vhost/rte_vhost_async.h +18 -0
@@ -266,6 +266,24 @@ rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id, struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count, int *nr_inflight, int16_t dma_id, uint16_t vchan_id); +/** +* @warning +* @b EXPERIMENTAL: this API may change, or be removed, without prior notice. +* +* Unconfigure DMA vChannels in asynchronous data path. +* This function should be called after the DMA vChannel has been unregistered. +* +* @param dma_id +* the identifier of DMA device +* @param vchan_id +* the identifier of virtual DMA channel +* @return +* 0 on success, and -1 on failure +*/ +__rte_experimental +int +rte_vhost_async_dma_unconfigure (int16_t dma_id,uint16_t vchan_id); + #ifdef __cplusplus }#endif
lib/vhost/version.map +3 -0
@@ -94,6 +94,9 @@ EXPERIMENTAL { rte_vhost_async_try_dequeue_burst; rte_vhost_driver_get_vdpa_dev_type; rte_vhost_clear_queue; + + # added in 22.11 + rte_vhost_async_dma_unconfigure; }; INTERNAL {
lib/vhost/vhost.c +64 -5
121 lines not on this page · raw message
Signature · 1 line
2.17.1
Quoted 96 lines · On 10/13/22 08:40, [email protected] wrote:
From: Xuan Ding <[email protected]>
Add a new API
rte_vhost_async_dma_unconfigure()to unconfigure DMA vchannels in vhost async data path. Lock protection are also added to protect DMA vchannels configuration and unconfiguration from concurrent calls.Signed-off-by Xuan Ding <[email protected]>--- doc/guides/prog_guide/vhost_lib.rst | 6 +++ doc/guides/rel_notes/release_22_11.rst | 4 ++ lib/vhost/rte_vhost_async.h | 18 +++++++ lib/vhost/version.map | 3 ++ lib/vhost/vhost.c | 69 ++++++++++++++++++++++++-- 5 files changed, 95 insertions(+), 5 deletions(-)doc/guides/prog_guide/vhost_lib.rst +6
-0@@ -323,6 +323,12 @@ The following is an overview of some key Vhost API functions: Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET, VDPA_DEVICE_TYPE_BLK. +* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)`` + + Clean DMA vChannels finished to use. This function needs to + be called after the deregistration of async DMA vchannel + has been finished. + Vhost-user Implementations --------------------------doc/guides/rel_notes/release_22_11.rst +4
-0@@ -236,6 +236,10 @@ New Features strings $dpdk_binary_or_driver | sed -n 's/^PMD_INFO_STRING= //p' +* **Added DMA vChannel unconfiguration for async vhost.** + + * Added support to unconfigure DMA vChannels that have been unregistered. + Removed Items -------------lib/vhost/rte_vhost_async.h +18
-0@@ -266,6 +266,24 @@ rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id, struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count, int *nr_inflight, int16_t dma_id, uint16_t vchan_id); +/** +* @warning +* @b EXPERIMENTAL: this API may change, or be removed, without prior notice. +* +* Unconfigure DMA vChannels in asynchronous data path. +* This function should be called after the DMA vChannel has been unregistered. +* +* @param dma_id +* the identifier of DMA device +* @param vchan_id +* the identifier of virtual DMA channel +* @return +* 0 on success, and -1 on failure +*/ +__rte_experimental +int +rte_vhost_async_dma_unconfigure (int16_t dma_id,uint16_t vchan_id); + #ifdef __cplusplus }#endif lib/vhost/version.map +3
-0@@ -94,6 +94,9 @@ EXPERIMENTAL { rte_vhost_async_try_dequeue_burst; rte_vhost_driver_get_vdpa_dev_type; rte_vhost_clear_queue; + + # added in 22.11 + rte_vhost_async_dma_unconfigure; }; INTERNAL {lib/vhost/vhost.c +0
-0
lib/vhost/vhost.c line 26
Quoted 4 lines
lib/vhost/vhost.c +1
-0@@ -23,6 +23,7 @@struct virtio_net *vhost_devices[RTE_MAX_VHOST_DEVICE ];pthread_mutex_t vhost_dev_lock =PTHREAD_MUTEX_INITIALIZER ; +static rte_spinlock_t vhost_dma_lock =RTE_SPINLOCK_INITIALIZER ;
I think a mutex would be more appropiate.
Thanks,
Maxime
Hi Maxime,
Quoted 118 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Thursday, October 13, 2022 4:02 PM
To: Ding, Xuan <[email protected]>; Xia, Chenbo <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; He, Xingguang <[email protected]>; Yang, YvonneX <[email protected]>; Jiang, Cheng1 <[email protected]>; Wang, YuanX <[email protected]>; Ma, WenwuX <[email protected]>
Subject: Re: [PATCH v4 1/2] vhost: introduce DMA vchannel unconfigurationdoc/guides/prog_guide/vhost_lib.rst The following is an overview of some key Vhost
Quoted 22 lines · On 10/13/22 08:40, [email protected] wrote:
From: Xuan Ding <[email protected]>
Add a new API
rte_vhost_async_dma_unconfigure()to unconfigure DMA vchannels in vhost async data path. Lock protection are also added to protect DMA vchannels configuration and unconfiguration from concurrent calls.Signed-off-by Xuan Ding <[email protected]>--- doc/guides/prog_guide/vhost_lib.rst | 6 +++ doc/guides/rel_notes/release_22_11.rst | 4 ++ lib/vhost/rte_vhost_async.h | 18 +++++++ lib/vhost/version.map | 3 ++ lib/vhost/vhost.c | 69 ++++++++++++++++++++++++-- 5 files changed, 95 insertions(+), 5 deletions(-)doc/guides/prog_guide/vhost_lib.rst +0
-0@@ -323,6 +323,12 @@ The following is an overview of some key VhostAPI functions:
Quoted 18 lines
doc/guides/prog_guide/vhost_lib.rst +6
-0Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET, VDPA_DEVICE_TYPE_BLK. +* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)`` + + Clean DMA vChannels finished to use. This function needs to be + called after the deregistration of async DMA vchannel has been + finished. + Vhost-user Implementations --------------------------doc/guides/rel_notes/release_22_11.rst +0
-0doc/guides/rel_notes/release_22_11.rst New Features · line 241
Quoted 6 lines
doc/guides/rel_notes/release_22_11.rst +3
-0@@ -236,6 +236,10 @@ New Features strings $dpdk_binary_or_driver | sed -n 's/^PMD_INFO_STRING= //p' +* **Added DMA vChannel unconfiguration for async vhost.** + + * Added support to unconfigure DMA vChannels that have beenunregistered.
lib/vhost/rte_vhost_async.h rte_vhost_async_try_dequeue_burst(int vid,
Quoted 9 lines
doc/guides/rel_notes/release_22_11.rst +1
-0+ Removed Items -------------lib/vhost/rte_vhost_async.h +0
-0@@ -266,6 +266,24 @@ rte_vhost_async_try_dequeue_burst(int vid,
uint16_tqueue_id,lib/vhost/rte_vhost_async.h rte_vhost_async_try_dequeue_burst(int vid, · line 266
Quoted 1 line
lib/vhost/rte_vhost_async.h +0
-0struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_tcount,
lib/vhost/rte_vhost_async.h rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id,
Quoted 5 lines
lib/vhost/rte_vhost_async.h +3
-0int *nr_inflight, int16_t dma_id, uint16_t vchan_id); +/** +* @warning +* @b EXPERIMENTAL: this API may change, or be removed, without prior notice.
lib/vhost/rte_vhost_async.h rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id,
Quoted 3 lines
lib/vhost/rte_vhost_async.h +3
-0+* +* Unconfigure DMA vChannels in asynchronous data path. +* This function should be called after the DMA vChannel has been unregistered.
Quoted 35 lines
lib/vhost/rte_vhost_async.h +12
-0+* +* @param dma_id +* the identifier of DMA device +* @param vchan_id +* the identifier of virtual DMA channel +* @return +* 0 on success, and -1 on failure +*/ +__rte_experimental +int +rte_vhost_async_dma_unconfigure (int16_t dma_id,uint16_t vchan_id); + #ifdef __cplusplus }#endif lib/vhost/version.map index renamed from lib/vhost/version.map +0
-0
7a00b65740..0b61870870100644lib/vhost/version.map +3
-0@@ -94,6 +94,9 @@ EXPERIMENTAL { rte_vhost_async_try_dequeue_burst; rte_vhost_driver_get_vdpa_dev_type; rte_vhost_clear_queue; + + # added in 22.11 + rte_vhost_async_dma_unconfigure; }; INTERNAL {lib/vhost/vhost.c index renamed from lib/vhost/vhost.c +0
-0
8740aa2788..9fbc56229a100644lib/vhost/vhost.c +0
-0@@ -23,6 +23,7 @@
Quoted 5 lines
lib/vhost/vhost.c line 26
Quoted 3 lines
lib/vhost/vhost.c +1
-0struct virtio_net *vhost_devices[RTE_MAX_VHOST_DEVICE ];pthread_mutex_t vhost_dev_lock =PTHREAD_MUTEX_INITIALIZER ; +static rte_spinlock_t vhost_dma_lock =RTE_SPINLOCK_INITIALIZER ;I think a mutex would be more appropiate.
Thanks for your suggestion.
Maybe mutex is more efficient, I will switch to mutex in next version.
Regards,
Xuan
Quoted 3 lines
Thanks,
Maxime
PATCH v4 2/2 examples/vhost: unconfigure DMA vchannel
In reply to [email protected]From: Xuan Ding <[email protected]>
This patch applies rte_vhost_async_dma_unconfigure() to manually free DMA vchannels. Before unconfiguration, need make sure the specified DMA device is no longer used by any vhost ports.
--- examples/vhost/main.c | 38 +++++++++++++++++++++++++------------- 1 file changed, 25 insertions(+), 13 deletions(-)
examples/vhost/main.c +25 -13
@@ -73,6 +73,7 @@ static int total_num_mbufs = NUM_MBUFS_DEFAULT;struct dma_for_vhost dma_bind[RTE_MAX_VHOST_DEVICE ];int16_t dmas_id[RTE_DMADEV_DEFAULT_MAX ]; +int16_t dma_ref_count[RTE_DMADEV_DEFAULT_MAX ];static int dma_count;/* mask of enabled ports */
@@ -371,6 +372,7 @@ open_dma(const char *value) done: (dma_info + socketid)->dmas[vring_id].dev_id = dev_id; (dma_info + socketid)->async_flag |= async_flag; + dma_ref_count[dev_id]++; i++; } out:
@@ -1562,6 +1564,27 @@ vhost_clear_queue(struct vhost_dev *vdev, uint16_t queue_id) } } +static void +vhost_clear_async (struct vhost_dev *vdev,int vid,uint16_t queue_id) +{ +int16_t dma_id; + +if (dma_bind[vid].dmas[queue_id].async_enabled) { +vhost_clear_queue (vdev, queue_id); +rte_vhost_async_channel_unregister (vid, queue_id); + dma_bind[vid].dmas[queue_id].async_enabled = false; + + dma_id = dma_bind[vid2socketid[vdev->vid]].dmas[queue_id].dev_id; + dma_ref_count[dma_id]--; + +if (dma_ref_count[dma_id] ==0 ) { +if (rte_vhost_async_dma_unconfigure (dma_id,0 ) <0 ) +RTE_LOG (ERR ,VHOST_CONFIG , +"Failed to unconfigure DMA %d in vhost.\n" , dma_id); + } + } +} +/* * Remove a device from the specific data core linked list and from the * main linked list. Synchronization occurs through the use of the
@@ -1618,17 +1641,8 @@ destroy_device(int vid)"(%d) device has been removed from data core\n" , vdev->vid); -if (dma_bind[vid].dmas[VIRTIO_RXQ ].async_enabled) { -vhost_clear_queue (vdev,VIRTIO_RXQ ); -rte_vhost_async_channel_unregister (vid,VIRTIO_RXQ ); - dma_bind[vid].dmas[VIRTIO_RXQ ].async_enabled = false; - } - -if (dma_bind[vid].dmas[VIRTIO_TXQ ].async_enabled) { -vhost_clear_queue (vdev,VIRTIO_TXQ ); -rte_vhost_async_channel_unregister (vid,VIRTIO_TXQ ); - dma_bind[vid].dmas[VIRTIO_TXQ ].async_enabled = false; - } +vhost_clear_async (vdev, vid,VIRTIO_RXQ ); +vhost_clear_async (vdev, vid,VIRTIO_TXQ );rte_free (vdev); }
@@ -1690,8 +1704,6 @@ vhost_async_channel_register(int vid)return rx_ret | tx_ret; } - -/* * A new device is added to a data core. First the device is added to the main linked list * and then allocated to a specific data core.
Signature · 1 line
2.17.1
Re: [PATCH v4 2/2] examples/vhost: unconfigure DMA vchannel
Quoted 32 lines · On 10/13/22 08:40, [email protected] wrote:
From: Xuan Ding <[email protected]>
This patch applies
rte_vhost_async_dma_unconfigure()to manually free DMA vchannels. Before unconfiguration, need make sure the specified DMA device is no longer used by any vhost ports.Signed-off-by Xuan Ding <[email protected]>--- examples/vhost/main.c | 38 +++++++++++++++++++++++++------------- 1 file changed, 25 insertions(+), 13 deletions(-)examples/vhost/main.c +2
-0@@ -73,6 +73,7 @@ static int total_num_mbufs = NUM_MBUFS_DEFAULT;struct dma_for_vhost dma_bind[RTE_MAX_VHOST_DEVICE ];int16_t dmas_id[RTE_DMADEV_DEFAULT_MAX ]; +int16_t dma_ref_count[RTE_DMADEV_DEFAULT_MAX ];static int dma_count;/* mask of enabled ports */ @@ -371,6 +372,7 @@ open_dma(const char *value) done: (dma_info + socketid)->dmas[vring_id].dev_id = dev_id; (dma_info + socketid)->async_flag |= async_flag; + dma_ref_count[dev_id]++; i++; } out:
examples/vhost/main.c vhost_clear_queue(struct vhost_dev *vdev, uint16_t queue_id) · line 1572
Quoted 9 lines
examples/vhost/main.c +6
-0@@ -1562,6 +1564,27 @@ vhost_clear_queue(struct vhost_dev *vdev, uint16_t queue_id) } } +static void +vhost_clear_async (struct vhost_dev *vdev,int vid,uint16_t queue_id) +{ +int16_t dma_id; + +if (dma_bind[vid].dmas[queue_id].async_enabled) {
if (!dma_bind[vid].dmas[queue_id].async_enabled)
return;
examples/vhost/main.c vhost_clear_queue(struct vhost_dev *vdev, uint16_t queue_id) · line 1580
Quoted 8 lines
examples/vhost/main.c +8
-0+vhost_clear_queue (vdev, queue_id); +rte_vhost_async_channel_unregister (vid, queue_id); + dma_bind[vid].dmas[queue_id].async_enabled = false; + + dma_id = dma_bind[vid2socketid[vdev->vid]].dmas[queue_id].dev_id; + dma_ref_count[dma_id]--; + +if (dma_ref_count[dma_id] ==0 ) {
if (dma_ref_count[dma_id] > 0)
return;
Doing this should improve readability.
Quoted 39 lines
examples/vhost/main.c +9
-13+if (rte_vhost_async_dma_unconfigure (dma_id,0 ) <0 ) +RTE_LOG (ERR ,VHOST_CONFIG , +"Failed to unconfigure DMA %d in vhost.\n" , dma_id); + } + } +} +/* * Remove a device from the specific data core linked list and from the * main linked list. Synchronization occurs through the use of the @@ -1618,17 +1641,8 @@ destroy_device(int vid)"(%d) device has been removed from data core\n" , vdev->vid); -if (dma_bind[vid].dmas[VIRTIO_RXQ ].async_enabled) { -vhost_clear_queue (vdev,VIRTIO_RXQ ); -rte_vhost_async_channel_unregister (vid,VIRTIO_RXQ ); - dma_bind[vid].dmas[VIRTIO_RXQ ].async_enabled = false; - } - -if (dma_bind[vid].dmas[VIRTIO_TXQ ].async_enabled) { -vhost_clear_queue (vdev,VIRTIO_TXQ ); -rte_vhost_async_channel_unregister (vid,VIRTIO_TXQ ); - dma_bind[vid].dmas[VIRTIO_TXQ ].async_enabled = false; - } +vhost_clear_async (vdev, vid,VIRTIO_RXQ ); +vhost_clear_async (vdev, vid,VIRTIO_TXQ );rte_free (vdev); }@@ -1690,8 +1704,6 @@ vhost_async_channel_register(int vid)return rx_ret | tx_ret; } - -/* * A new device is added to a data core. First the device is added to the main linked list * and then allocated to a specific data core.
RE: [PATCH v4 2/2] examples/vhost: unconfigure DMA vchannel
Quoted 72 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Thursday, October 13, 2022 4:07 PM
To: Ding, Xuan <[email protected]>; Xia, Chenbo <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; He, Xingguang <[email protected]>; Yang, YvonneX <[email protected]>; Jiang, Cheng1 <[email protected]>; Wang, YuanX <[email protected]>; Ma, WenwuX <[email protected]>
Subject: Re: [PATCH v4 2/2] examples/vhost: unconfigure DMA vchannelexamples/vhost/main.c vhost_clear_queue(struct vhost_dev *vdev,
Quoted 32 lines · On 10/13/22 08:40, [email protected] wrote:
From: Xuan Ding <[email protected]>
This patch applies
rte_vhost_async_dma_unconfigure()to manually free DMA vchannels. Before unconfiguration, need make sure the specified DMA device is no longer used by any vhost ports.Signed-off-by Xuan Ding <[email protected]>--- examples/vhost/main.c | 38 +++++++++++++++++++++++++------------- 1 file changed, 25 insertions(+), 13 deletions(-)examples/vhost/main.c index renamed from examples/vhost/main.c +0
-0
ac78704d79..bfeb808dcc100644examples/vhost/main.c +2
-0@@ -73,6 +73,7 @@ static int total_num_mbufs = NUM_MBUFS_DEFAULT;struct dma_for_vhost dma_bind[RTE_MAX_VHOST_DEVICE ];int16_t dmas_id[RTE_DMADEV_DEFAULT_MAX ]; +int16_t dma_ref_count[RTE_DMADEV_DEFAULT_MAX ];static int dma_count;/* mask of enabled ports */ @@ -371,6 +372,7 @@ open_dma(const char *value) done: (dma_info + socketid)->dmas[vring_id].dev_id = dev_id; (dma_info + socketid)->async_flag |= async_flag; + dma_ref_count[dev_id]++; i++; } out:@@ -1562,6 +1564,27 @@ vhost_clear_queue(struct vhost_dev *vdev,
uint16_tqueue_id)examples/vhost/main.c vhost_clear_queue(struct vhost_dev *vdev, uint16_t queue_id) · line 1572
Quoted 9 lines
examples/vhost/main.c +6
-0} } +static void +vhost_clear_async (struct vhost_dev *vdev,int vid,uint16_t queue_id) +{ +int16_t dma_id; + +if (dma_bind[vid].dmas[queue_id].async_enabled) {if (!dma_bind[vid].dmas[queue_id].async_enabled)
return;examples/vhost/main.c vhost_clear_queue(struct vhost_dev *vdev, uint16_t queue_id) · line 1580
Quoted 9 lines
examples/vhost/main.c +8
-0+vhost_clear_queue (vdev, queue_id); +rte_vhost_async_channel_unregister (vid, queue_id); + dma_bind[vid].dmas[queue_id].async_enabled = false; + + dma_id = dma_bind[vid2socketid[vdev- vid]].dmas[queue_id].dev_id; + dma_ref_count[dma_id]--; + +if (dma_ref_count[dma_id] ==0 ) {if (dma_ref_count[dma_id] > 0)
Quoted 3 lines
return;Doing this should improve readability.
Good suggestion! Please see v5.
Thanks,
Xuan
Quoted 42 lines
examples/vhost/main.c vhost_clear_queue(struct vhost_dev *vdev, uint16_t queue_id)
Quoted 4 lines
examples/vhost/main.c +3
-0+if (rte_vhost_async_dma_unconfigure (dma_id,0 ) <0 ) + RTE_LOG(ERR, VHOST_CONFIG, + "Failed to unconfigure DMA %d invhost.\n",
dma_id);Quoted 28 lines
examples/vhost/main.c +4
-0+ } + } +} +/* * Remove a device from the specific data core linked list and from the * main linked list. Synchronization occurs through the use of the
examples/vhost/main.c +2
-11@@ -1618,17 +1641,8 @@ destroy_device(int vid)"(%d) device has been removed from data core\n" , vdev->vid); -if (dma_bind[vid].dmas[VIRTIO_RXQ ].async_enabled) { -vhost_clear_queue (vdev,VIRTIO_RXQ ); -rte_vhost_async_channel_unregister (vid,VIRTIO_RXQ ); - dma_bind[vid].dmas[VIRTIO_RXQ ].async_enabled = false; - } - -if (dma_bind[vid].dmas[VIRTIO_TXQ ].async_enabled) { -vhost_clear_queue (vdev,VIRTIO_TXQ ); -rte_vhost_async_channel_unregister (vid,VIRTIO_TXQ ); - dma_bind[vid].dmas[VIRTIO_TXQ ].async_enabled = false; - } +vhost_clear_async (vdev, vid,VIRTIO_RXQ ); +vhost_clear_async (vdev, vid,VIRTIO_TXQ );rte_free (vdev); }examples/vhost/main.c vhost_async_channel_register(int vid) · line 1708
Quoted 7 lines
examples/vhost/main.c +0
-2@@ -1690,8 +1704,6 @@ vhost_async_channel_register(int vid)return rx_ret | tx_ret; } - -/* * A new device is added to a data core. First the device is added to the main linked list
Quoted 1 line
examples/vhost/main.c +0
-0* and then allocated to a specific data core.
v5 cover vhost: introduce DMA vchannel unconfiguration
In reply to [email protected]From: Xuan Ding <[email protected]>
This patchset introduces a new API rte_vhost_async_dma_unconfigure() to help user to manually free DMA vchannels finished to use.
Note: this API should be called after async channel unregister.
v5:
* Use mutex instead of spinlock.
* Improve code readability.
v4:
* Rebase to 22.11 rc1.
* Fix the usage of 'dma_ref_count' to make sure the specified DMA device is not used by any vhost ports before unconfiguration.
v3:
* Rebase to latest DPDK.
* Refine some descriptions in the doc.
* Fix one bug in the vhost example.
v2:
* Add spinlock protection.
* Fix a memory leak issue.
* Refine the doc.
Xuan Ding (2):
vhost: introduce DMA vchannel unconfiguration
examples/vhost: unconfigure DMA vchannel
doc/guides/prog_guide/vhost_lib.rst | 6 +++ doc/guides/rel_notes/release_22_11.rst | 4 ++ examples/vhost/main.c | 40 ++++++++++----- lib/vhost/rte_vhost_async.h | 18 +++++++ lib/vhost/version.map | 3 ++ lib/vhost/vhost.c | 69 ++++++++++++++++++++++++-- 6 files changed, 122 insertions(+), 18 deletions(-)
Signature · 1 line
2.17.1
PATCH v5 1/2 vhost: introduce DMA vchannel unconfiguration
From: Xuan Ding <[email protected]>
Add a new API rte_vhost_async_dma_unconfigure() to unconfigure DMA vchannels in vhost async data path. Lock protection are also added to protect DMA vchannels configuration and unconfiguration from concurrent calls.
--- doc/guides/prog_guide/vhost_lib.rst | 6 +++ doc/guides/rel_notes/release_22_11.rst | 4 ++ lib/vhost/rte_vhost_async.h | 18 +++++++ lib/vhost/version.map | 3 ++ lib/vhost/vhost.c | 69 ++++++++++++++++++++++++-- 5 files changed, 95 insertions(+), 5 deletions(-)
doc/guides/prog_guide/vhost_lib.rst +6 -0
@@ -323,6 +323,12 @@ The following is an overview of some key Vhost API functions: Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET, VDPA_DEVICE_TYPE_BLK. +* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)`` + + Clean DMA vChannels finished to use. This function needs to + be called after the deregistration of async DMA vchannel + has been finished. + Vhost-user Implementations --------------------------
doc/guides/rel_notes/release_22_11.rst +4 -0
@@ -236,6 +236,10 @@ New Features strings $dpdk_binary_or_driver | sed -n 's/^PMD_INFO_STRING= //p' +* **Added DMA vChannel unconfiguration for async vhost.** + + * Added support to unconfigure DMA vChannels that have been unregistered. + Removed Items -------------
lib/vhost/rte_vhost_async.h +18 -0
@@ -266,6 +266,24 @@ rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id, struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count, int *nr_inflight, int16_t dma_id, uint16_t vchan_id); +/** +* @warning +* @b EXPERIMENTAL: this API may change, or be removed, without prior notice. +* +* Unconfigure DMA vChannels in asynchronous data path. +* This function should be called after the DMA vChannel has been unregistered. +* +* @param dma_id +* the identifier of DMA device +* @param vchan_id +* the identifier of virtual DMA channel +* @return +* 0 on success, and -1 on failure +*/ +__rte_experimental +int +rte_vhost_async_dma_unconfigure (int16_t dma_id,uint16_t vchan_id); + #ifdef __cplusplus }#endif
lib/vhost/version.map +3 -0
@@ -94,6 +94,9 @@ EXPERIMENTAL { rte_vhost_async_try_dequeue_burst; rte_vhost_driver_get_vdpa_dev_type; rte_vhost_clear_queue; + + # added in 22.11 + rte_vhost_async_dma_unconfigure; }; INTERNAL {
lib/vhost/vhost.c +64 -5
121 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v5 2/2 examples/vhost: unconfigure DMA vchannel
In reply to [email protected]From: Xuan Ding <[email protected]>
This patch applies rte_vhost_async_dma_unconfigure() to manually free DMA vchannels. Before unconfiguration, make sure the specified DMA device is no longer used by any vhost ports.
--- examples/vhost/main.c | 40 +++++++++++++++++++++++++++------------- 1 file changed, 27 insertions(+), 13 deletions(-)
examples/vhost/main.c +27 -13
@@ -73,6 +73,7 @@ static int total_num_mbufs = NUM_MBUFS_DEFAULT;struct dma_for_vhost dma_bind[RTE_MAX_VHOST_DEVICE ];int16_t dmas_id[RTE_DMADEV_DEFAULT_MAX ]; +int16_t dma_ref_count[RTE_DMADEV_DEFAULT_MAX ];static int dma_count;/* mask of enabled ports */
@@ -371,6 +372,7 @@ open_dma(const char *value) done: (dma_info + socketid)->dmas[vring_id].dev_id = dev_id; (dma_info + socketid)->async_flag |= async_flag; + dma_ref_count[dev_id]++; i++; } out:
@@ -1562,6 +1564,29 @@ vhost_clear_queue(struct vhost_dev *vdev, uint16_t queue_id) } } +static void +vhost_clear_async (struct vhost_dev *vdev,int vid,uint16_t queue_id) +{ +int16_t dma_id; + +if (!dma_bind[vid].dmas[queue_id].async_enabled) +return ; + +vhost_clear_queue (vdev, queue_id); +rte_vhost_async_channel_unregister (vid, queue_id); + dma_bind[vid].dmas[queue_id].async_enabled = false; + + dma_id = dma_bind[vid2socketid[vdev->vid]].dmas[queue_id].dev_id; + dma_ref_count[dma_id]--; + +if (dma_ref_count[dma_id] >0 ) +return ; + +if (rte_vhost_async_dma_unconfigure (dma_id,0 ) <0 ) +RTE_LOG (ERR ,VHOST_CONFIG , +"Failed to unconfigure DMA %d in vhost.\n" , dma_id); +} +/* * Remove a device from the specific data core linked list and from the * main linked list. Synchronization occurs through the use of the
@@ -1618,17 +1643,8 @@ destroy_device(int vid)"(%d) device has been removed from data core\n" , vdev->vid); -if (dma_bind[vid].dmas[VIRTIO_RXQ ].async_enabled) { -vhost_clear_queue (vdev,VIRTIO_RXQ ); -rte_vhost_async_channel_unregister (vid,VIRTIO_RXQ ); - dma_bind[vid].dmas[VIRTIO_RXQ ].async_enabled = false; - } - -if (dma_bind[vid].dmas[VIRTIO_TXQ ].async_enabled) { -vhost_clear_queue (vdev,VIRTIO_TXQ ); -rte_vhost_async_channel_unregister (vid,VIRTIO_TXQ ); - dma_bind[vid].dmas[VIRTIO_TXQ ].async_enabled = false; - } +vhost_clear_async (vdev, vid,VIRTIO_RXQ ); +vhost_clear_async (vdev, vid,VIRTIO_TXQ );rte_free (vdev); }
@@ -1690,8 +1706,6 @@ vhost_async_channel_register(int vid)return rx_ret | tx_ret; } - -/* * A new device is added to a data core. First the device is added to the main linked list * and then allocated to a specific data core.
Signature · 1 line
2.17.1
v6 cover vhost: introduce DMA vchannel unconfiguration
In reply to [email protected]From: Xuan Ding <[email protected]>
This patchset introduces a new API rte_vhost_async_dma_unconfigure() to help user to manually free DMA vchannels finished to use.
v6:
* Move DMA unconfiguration to the end due to DMA devices maybe reused after destroy_device().
* Refine the doc to claim the DMA device should not be used in vhost after unconfiguration.
v5:
* Use mutex instead of spinlock.
* Improve code readability.
v4:
* Rebase to 22.11 rc1.
* Fix the usage of 'dma_ref_count' to make sure the specified DMA device is not used by any vhost ports before unconfiguration.
v3:
* Rebase to latest DPDK.
* Refine some descriptions in the doc.
* Fix one bug in the vhost example.
v2:
* Add spinlock protection.
* Fix a memory leak issue.
* Refine the doc.
Xuan Ding (2):
vhost: introduce DMA vchannel unconfiguration
examples/vhost: unconfigure DMA vchannel
doc/guides/prog_guide/vhost_lib.rst | 7 +++ doc/guides/rel_notes/release_22_11.rst | 5 ++ examples/vhost/main.c | 8 +++ lib/vhost/rte_vhost_async.h | 20 ++++++++ lib/vhost/version.map | 3 ++ lib/vhost/vhost.c | 69 ++++++++++++++++++++++++-- 6 files changed, 107 insertions(+), 5 deletions(-)
Signature · 1 line
2.17.1
PATCH v6 1/2 vhost: introduce DMA vchannel unconfiguration
From: Xuan Ding <[email protected]>
Add a new API rte_vhost_async_dma_unconfigure() to unconfigure DMA vchannels in vhost async data path. Lock protection are also added to protect DMA vchannels configuration and unconfiguration from concurrent calls.
--- doc/guides/prog_guide/vhost_lib.rst | 7 +++ doc/guides/rel_notes/release_22_11.rst | 5 ++ lib/vhost/rte_vhost_async.h | 20 ++++++++ lib/vhost/version.map | 3 ++ lib/vhost/vhost.c | 69 ++++++++++++++++++++++++-- 5 files changed, 99 insertions(+), 5 deletions(-)
doc/guides/prog_guide/vhost_lib.rst +7 -0
@@ -323,6 +323,13 @@ The following is an overview of some key Vhost API functions: Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET, VDPA_DEVICE_TYPE_BLK. +* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)`` + + Clean DMA vChannels finished to use. This function needs to be called + after the deregistration of async DMA vchannel has been finished. + After this function is called, the specified DMA device should no + longer be used by the Vhost library. + Vhost-user Implementations --------------------------
doc/guides/rel_notes/release_22_11.rst +5 -0
@@ -236,6 +236,11 @@ New Features strings $dpdk_binary_or_driver | sed -n 's/^PMD_INFO_STRING= //p' +* **Added DMA vChannel unconfiguration for async vhost.** + + * Added support to unconfigure DMA vChannels that have been unregistered + and no longer used by Vhost library. + Removed Items -------------
lib/vhost/rte_vhost_async.h +20 -0
@@ -266,6 +266,26 @@ rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id, struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count, int *nr_inflight, int16_t dma_id, uint16_t vchan_id); +/** +* @warning +* @b EXPERIMENTAL: this API may change, or be removed, without prior notice. +* +* Unconfigure DMA vChannels in Vhost asynchronous data path. +* This function should be called after the DMA vChannel has been unregistered. +* After this function is called, the specified DMA device should no longer +* be used by the Vhost library. +* +* @param dma_id +* the identifier of DMA device +* @param vchan_id +* the identifier of virtual DMA channel +* @return +* 0 on success, and -1 on failure +*/ +__rte_experimental +int +rte_vhost_async_dma_unconfigure (int16_t dma_id,uint16_t vchan_id); + #ifdef __cplusplus }#endif
lib/vhost/version.map +3 -0
@@ -94,6 +94,9 @@ EXPERIMENTAL { rte_vhost_async_try_dequeue_burst; rte_vhost_driver_get_vdpa_dev_type; rte_vhost_clear_queue; + + # added in 22.11 + rte_vhost_async_dma_unconfigure; }; INTERNAL {
lib/vhost/vhost.c +64 -5
121 lines not on this page · raw message
Signature · 1 line
2.17.1
Quoted 58 lines
-----Original Message-----
From: Ding, Xuan <[email protected]>
Sent: Tuesday, October 18, 2022 11:22 PM
To: [email protected]; Xia, Chenbo <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; He, Xingguang <[email protected]>; Yang, YvonneX <[email protected]>; Jiang, Cheng1 <[email protected]>; Wang, YuanX <[email protected]>; Ma, WenwuX <[email protected]>; Ding, Xuan <[email protected]>
Subject: [PATCH v6 1/2] vhost: introduce DMA vchannel unconfigurationFrom: Xuan Ding <[email protected]>
Add a new API
rte_vhost_async_dma_unconfigure()to unconfigure DMA vchannels in vhost async data path. Lock protection are also added to protect DMA vchannels configuration and unconfiguration from concurrent calls.Signed-off-by Xuan Ding <[email protected]>--- doc/guides/prog_guide/vhost_lib.rst | 7 +++ doc/guides/rel_notes/release_22_11.rst | 5 ++ lib/vhost/rte_vhost_async.h | 20 ++++++++ lib/vhost/version.map | 3 ++ lib/vhost/vhost.c | 69 ++++++++++++++++++++++++-- 5 files changed, 99 insertions(+), 5 deletions(-)doc/guides/prog_guide/vhost_lib.rst +0
-0@@ -323,6 +323,13 @@ The following is an overview of some key Vhost APIfunctions:
Get device type of vDPA device, such asVDPA_DEVICE_TYPE_NET,VDPA_DEVICE_TYPE_BLK.doc/guides/prog_guide/vhost_lib.rst +7
-0+* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)`` + + Clean DMA vChannels finished to use. This function needs to be called + after the deregistration of async DMA vchannel has been finished. + After this function is called, the specified DMA device should no + longer be used by the Vhost library. + Vhost-user Implementations --------------------------doc/guides/rel_notes/release_22_11.rst +3
-0@@ -236,6 +236,11 @@ New Features strings $dpdk_binary_or_driver | sed -n 's/^PMD_INFO_STRING= //p' +* **Added DMA vChannel unconfiguration for async vhost.** + + * Added support to unconfigure DMA vChannels that have been
Quoted 2 lines
unregistered
+ and no longer used by Vhost library.
Somehow this is making release notes build fail.
Maxime, is this because of the '*' before 'Added support'?
Thanks,
Chenbo
Quoted 210 lines
doc/guides/rel_notes/release_22_11.rst +1
-0+ Removed Items -------------lib/vhost/rte_vhost_async.h +3
-0@@ -266,6 +266,26 @@ rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id, struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count, int *nr_inflight, int16_t dma_id, uint16_t vchan_id); +/** +* @warning +* @b EXPERIMENTAL: this API may change, or be removed, without prior notice.
+ *
+ * Unconfigure DMA vChannels in Vhost asynchronous data path.
+ * This function should be called after the DMA vChannel has been unregistered.
+ * After this function is called, the specified DMA device should no longer
+ * be used by the Vhost library.
+ *
+ * @paramdma_id
+ * the identifier of DMA device
+ * @paramvchan_id
+ * the identifier of virtual DMA channel
+ * @return
+ * 0 on success, and -1 on failure
+ */lib/vhost/rte_vhost_async.h +4
-0+__rte_experimental +int +rte_vhost_async_dma_unconfigure (int16_t dma_id,uint16_t vchan_id); + #ifdef __cplusplus }#endif lib/vhost/version.map +3
-0@@ -94,6 +94,9 @@ EXPERIMENTAL { rte_vhost_async_try_dequeue_burst; rte_vhost_driver_get_vdpa_dev_type; rte_vhost_clear_queue; + + # added in 22.11 + rte_vhost_async_dma_unconfigure; }; INTERNAL {lib/vhost/vhost.c +5
-3@@ -23,6 +23,7 @@struct virtio_net *vhost_devices[RTE_MAX_VHOST_DEVICE ];pthread_mutex_t vhost_dev_lock =PTHREAD_MUTEX_INITIALIZER ; +pthread_mutex_t vhost_dma_lock =PTHREAD_MUTEX_INITIALIZER ;struct vhost_vq_stats_name_off {char name[RTE_VHOST_STATS_NAME_SIZE ];@@ -1844,19 +1845,20 @@ rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id)void *pkts_cmpl_flag_addr;uint16_t max_desc; +pthread_mutex_lock (&vhost_dma_lock);if (!rte_dma_is_valid (dma_id)) {VHOST_LOG_CONFIG ("dma" ,ERR ,"DMA %d is not found.\n" , dma_id); -return -1 ; + goto error; }if (rte_dma_info_get (dma_id, &info) !=0 ) { VHOST_LOG_CONFIG("dma", ERR, "Fail to get DMA %d information.\n", dma_id); -return -1 ; + goto error; }if (vchan_id >= info.max_vchans) {VHOST_LOG_CONFIG ("dma" ,ERR ,"Invalid DMA %d vChannel %u.\n" , dma_id, vchan_id); -return -1 ; + goto error;lib/vhost/vhost.c +1
-1}if (!dma_copy_track[dma_id].vchans) {@@ -1868,7 +1870,7 @@ rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id) VHOST_LOG_CONFIG("dma", ERR, "Failed to allocate vchans for DMA %d vChannel %u.\n", dma_id, vchan_id); -return -1 ; + goto error; }lib/vhost/vhost.c +1
-0dma_copy_track[dma_id].vchans = vchans;@@ -1877,6 +1879,7 @@ rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id)if (dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr) { VHOST_LOG_CONFIG("dma", INFO, "DMA %d vChannel %u already registered.\n", dma_id, vchan_id); +pthread_mutex_unlock (&vhost_dma_lock);return 0 ;lib/vhost/vhost.c +1
-1}@@ -1894,7 +1897,7 @@ rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id) rte_free(dma_copy_track[dma_id].vchans); dma_copy_track[dma_id].vchans =NULL ; } -return -1 ; + goto error; }dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr =
pkts_cmpl_flag_addr;lib/vhost/vhost.c +5
-0@@ -1902,7 +1905,12 @@ rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id) dma_copy_track[dma_id].vchans[vchan_id].ring_mask = max_desc - 1; dma_copy_track[dma_id].nr_vchans++; +pthread_mutex_unlock (&vhost_dma_lock);return 0 ; + +error: +pthread_mutex_unlock (&vhost_dma_lock); +return -1 ; }int
lib/vhost/vhost.c +20
-0@@ -2091,5 +2099,56 @@ int rte_vhost_vring_stats_reset(int vid, uint16_t queue_id)return 0 ; } +int +rte_vhost_async_dma_unconfigure (int16_t dma_id,uint16_t vchan_id) +{ +struct rte_dma_info info; +uint16_t max_desc; +int i; + +pthread_mutex_lock (&vhost_dma_lock); +if (!rte_dma_is_valid (dma_id)) { +VHOST_LOG_CONFIG ("dma" ,ERR ,"DMA %d is not found.\n" , dma_id); + goto error; + } + +if (rte_dma_info_get (dma_id, &info) !=0 ) { + VHOST_LOG_CONFIG("dma", ERR, "Fail to get DMA %d information.\n", dma_id); + goto error; + } + +if (vchan_id >= info.max_vchans) { + VHOST_LOG_CONFIG("dma", ERR, "Invalid DMA %d vChannel %u.\n",
dma_id,vchan_id);lib/vhost/vhost.c +6
-0+ goto error; + } + + max_desc = info.max_desc; +for (i =0 ; i < max_desc; i++) { + if(dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr[i] != NULL) { +
rte_free(dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr
[i]);
+
dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr[i] =
NULL;lib/vhost/vhost.c +4
-0+ } + } + + if (dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr !=NULL) {
+
rte_free(dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr
);lib/vhost/vhost.c +1
-0+ dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr =NULL;
lib/vhost/vhost.c +17
-1+ } + +if (dma_copy_track[dma_id].vchans !=NULL ) { +rte_free (dma_copy_track[dma_id].vchans); + dma_copy_track[dma_id].vchans =NULL ; + } + + dma_copy_track[dma_id].nr_vchans--; + +pthread_mutex_unlock (&vhost_dma_lock); +return 0 ; + +error: +pthread_mutex_unlock (&vhost_dma_lock); +return -1 ; +} +RTE_LOG_REGISTER_SUFFIX (vhost_config_log_level, config,INFO );RTE_LOG_REGISTER_SUFFIX (vhost_data_log_level, data,WARNING ); --2.17.1
PATCH v6 2/2 examples/vhost: unconfigure DMA vchannel
In reply to [email protected]From: Xuan Ding <[email protected]>
This patch applies rte_vhost_async_dma_unconfigure() to manually free DMA vchannels. Before unconfiguration, make sure the specified DMA device is no longer used by any vhost ports.
--- examples/vhost/main.c | 8 ++++++++ 1 file changed, 8 insertions(+)
examples/vhost/main.c +8 -0
@@ -2066,6 +2066,14 @@ main(int argc, char *argv[])RTE_LCORE_FOREACH_WORKER (lcore_id)rte_eal_wait_lcore (lcore_id); +for (i =0 ; i < dma_count; i++) { +if (rte_vhost_async_dma_unconfigure (dmas_id[i],0 ) <0 ) { +RTE_LOG (ERR ,VHOST_PORT , +"Failed to unconfigure DMA %d in vhost.\n" , dmas_id[i]); +rte_exit (EXIT_FAILURE ,"Cannot use given DMA device\n" ); + } + } +/* clean up the EAL */ rte_eal_cleanup ();
Signature · 1 line
2.17.1
RE: [PATCH v6 2/2] examples/vhost: unconfigure DMA vchannel
Quoted 10 lines
-----Original Message-----
From: [email protected] <[email protected]>
Sent: Tuesday, October 18, 2022 11:22 PM
To: [email protected]; Xia, Chenbo <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; He, Xingguang <[email protected]>; Yang, YvonneX <[email protected]>; Jiang, Cheng1 <[email protected]>; Wang, YuanX <[email protected]>; Ma, WenwuX <[email protected]>; Ding, Xuan <[email protected]>
Subject: [PATCH v6 2/2] examples/vhost: unconfigure DMA vchannel
Quoted 8 lines
From: Xuan Ding <[email protected]>
This patch applies
rte_vhost_async_dma_unconfigure()to manually free DMA vchannels. Before unconfiguration, make sure the specified DMA device is no longer used by any vhost ports.Signed-off-by Xuan Ding <[email protected]>---
v7 cover vhost: introduce DMA vchannel unconfiguration
In reply to [email protected]From: Xuan Ding <[email protected]>
This patchset introduces a new API rte_vhost_async_dma_unconfigure() to help user to manually free DMA vchannels finished to use.
v7:
* Add inflight packets processing.
* Fix CI error.
v6:
* Move DMA unconfiguration to the end due to DMA devices maybe reused after destroy_device().
* Refine the doc to claim the DMA device should not be used in vhost after unconfiguration.
v5:
* Use mutex instead of spinlock.
* Improve code readability.
v4:
* Rebase to 22.11 rc1.
* Fix the usage of 'dma_ref_count' to make sure the specified DMA device is not used by any vhost ports before unconfiguration.
v3:
* Rebase to latest DPDK.
* Refine some descriptions in the doc.
* Fix one bug in the vhost example.
v2:
* Add spinlock protection.
* Fix a memory leak issue.
* Refine the doc.
Xuan Ding (2):
vhost: introduce DMA vchannel unconfiguration
examples/vhost: unconfigure DMA vchannel
doc/guides/prog_guide/vhost_lib.rst | 5 ++ doc/guides/rel_notes/release_22_11.rst | 5 ++ examples/vhost/main.c | 8 +++ lib/vhost/rte_vhost_async.h | 20 +++++++ lib/vhost/version.map | 3 ++ lib/vhost/vhost.c | 75 ++++++++++++++++++++++++-- 6 files changed, 111 insertions(+), 5 deletions(-)
Signature · 1 line
2.17.1
PATCH v7 1/2 vhost: introduce DMA vchannel unconfiguration
From: Xuan Ding <[email protected]>
Add a new API rte_vhost_async_dma_unconfigure() to unconfigure DMA vchannels in vhost async data path. Lock protection are also added to protect DMA vchannels configuration and unconfiguration from concurrent calls.
--- doc/guides/prog_guide/vhost_lib.rst | 5 ++ doc/guides/rel_notes/release_22_11.rst | 5 ++ lib/vhost/rte_vhost_async.h | 20 +++++++ lib/vhost/version.map | 3 ++ lib/vhost/vhost.c | 75 ++++++++++++++++++++++++-- 5 files changed, 103 insertions(+), 5 deletions(-)
doc/guides/prog_guide/vhost_lib.rst +5 -0
@@ -323,6 +323,11 @@ The following is an overview of some key Vhost API functions: Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET, VDPA_DEVICE_TYPE_BLK. +* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)`` + + Clean DMA vChannels finished to use. After this function is called, + the specified DMA vChannel should no longer be used by the Vhost library. + Vhost-user Implementations --------------------------
doc/guides/rel_notes/release_22_11.rst +5 -0
@@ -225,6 +225,11 @@ New Features sysfs entries to adjust the minimum and maximum uncore frequency values, which works on Linux with Intel hardware only. +* **Added DMA vChannel unconfiguration for async vhost.** + + Added support to unconfigure DMA vChannels that are no longer used + by the Vhost library. + * **Rewritten pmdinfo script.** The ``dpdk-pmdinfo.py`` script was rewritten to produce valid JSON only.
lib/vhost/rte_vhost_async.h +20 -0
@@ -266,6 +266,26 @@ rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id, struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count, int *nr_inflight, int16_t dma_id, uint16_t vchan_id); +/** +* @warning +* @b EXPERIMENTAL: this API may change, or be removed, without prior notice. +* +* Unconfigure DMA vChannels in Vhost asynchronous data path. +* This function should be called when the specified DMA device is no longer +* used by the Vhost library. Before this function is called, make sure there +* does not exist in-flight packets in DMA vChannel. +* +* @param dma_id +* the identifier of DMA device +* @param vchan_id +* the identifier of virtual DMA channel +* @return +* 0 on success, and -1 on failure +*/ +__rte_experimental +int +rte_vhost_async_dma_unconfigure (int16_t dma_id,uint16_t vchan_id); + #ifdef __cplusplus }#endif
lib/vhost/version.map +3 -0
@@ -94,6 +94,9 @@ EXPERIMENTAL { rte_vhost_async_try_dequeue_burst; rte_vhost_driver_get_vdpa_dev_type; rte_vhost_clear_queue; + + # added in 22.11 + rte_vhost_async_dma_unconfigure; }; INTERNAL {
lib/vhost/vhost.c +70 -5
127 lines not on this page · raw message
Signature · 1 line
2.17.1
Quoted 7 lines · On 10/20/22 11:11, [email protected] wrote:
From: Xuan Ding <[email protected]>
Add a new API
rte_vhost_async_dma_unconfigure()to unconfigure DMA vchannels in vhost async data path. Lock protection are also added to protect DMA vchannels configuration and unconfiguration from concurrent calls.
Quoted 9 lines
Signed-off-by Xuan Ding <[email protected]>--- doc/guides/prog_guide/vhost_lib.rst | 5 ++ doc/guides/rel_notes/release_22_11.rst | 5 ++ lib/vhost/rte_vhost_async.h | 20 +++++++ lib/vhost/version.map | 3 ++ lib/vhost/vhost.c | 75 ++++++++++++++++++++++++-- 5 files changed, 103 insertions(+), 5 deletions(-)
Thanks,
Maxime
Hi Maxime,
Quoted 30 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Friday, October 21, 2022 4:10 PM
To: Ding, Xuan <[email protected]>; Xia, Chenbo <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; He, Xingguang <[email protected]>; Yang, YvonneX <[email protected]>; Jiang, Cheng1 <[email protected]>; Wang, YuanX <[email protected]>; Ma, WenwuX <[email protected]>
Subject: Re: [PATCH v7 1/2] vhost: introduce DMA vchannel unconfigurationQuoted 7 lines · On 10/20/22 11:11, [email protected] wrote:
From: Xuan Ding <[email protected]>
Add a new API
rte_vhost_async_dma_unconfigure()to unconfigure DMA vchannels in vhost async data path. Lock protection are also added to protect DMA vchannels configuration and unconfiguration from concurrent calls.Quoted 9 lines
Signed-off-by Xuan Ding <[email protected]>--- doc/guides/prog_guide/vhost_lib.rst | 5 ++ doc/guides/rel_notes/release_22_11.rst | 5 ++ lib/vhost/rte_vhost_async.h | 20 +++++++ lib/vhost/version.map | 3 ++ lib/vhost/vhost.c | 75 ++++++++++++++++++++++++-- 5 files changed, 103 insertions(+), 5 deletions(-)
Quoted 1 line
Reviewed-by Maxime Coquelin <[email protected]>
This patch needs mirror changes.
Please help to review and wait until the v8 test passes before merging. Thanks a lot.
Regards,
Xuan
Quoted 3 lines
Thanks,
Maxime
PATCH v7 2/2 examples/vhost: unconfigure DMA vchannel
In reply to [email protected]From: Xuan Ding <[email protected]>
This patch applies rte_vhost_async_dma_unconfigure() to manually free DMA vchannels. Before unconfiguration, make sure the specified DMA device is no longer used by any vhost ports.
--- examples/vhost/main.c | 8 ++++++++ 1 file changed, 8 insertions(+)
examples/vhost/main.c +8 -0
@@ -2066,6 +2066,14 @@ main(int argc, char *argv[])RTE_LCORE_FOREACH_WORKER (lcore_id)rte_eal_wait_lcore (lcore_id); +for (i =0 ; i < dma_count; i++) { +if (rte_vhost_async_dma_unconfigure (dmas_id[i],0 ) <0 ) { +RTE_LOG (ERR ,VHOST_PORT , +"Failed to unconfigure DMA %d in vhost.\n" , dmas_id[i]); +rte_exit (EXIT_FAILURE ,"Cannot use given DMA device\n" ); + } + } +/* clean up the EAL */ rte_eal_cleanup ();
Signature · 1 line
2.17.1
Re: [PATCH v7 2/2] examples/vhost: unconfigure DMA vchannel
Quoted 20 lines · On 10/20/22 11:11, [email protected] wrote:
From: Xuan Ding <[email protected]>
This patch applies
rte_vhost_async_dma_unconfigure()to manually free DMA vchannels. Before unconfiguration, make sure the specified DMA device is no longer used by any vhost ports.Signed-off-by Xuan Ding <[email protected]>--- examples/vhost/main.c | 8 ++++++++ 1 file changed, 8 insertions(+)examples/vhost/main.c +1
-0@@ -2066,6 +2066,14 @@ main(int argc, char *argv[])RTE_LCORE_FOREACH_WORKER (lcore_id)rte_eal_wait_lcore (lcore_id); +for (i =0 ; i < dma_count; i++) {
examples/vhost/main.c main(int argc, char *argv[]) · line 2079
Quoted 10 lines
examples/vhost/main.c +7
-0+if (rte_vhost_async_dma_unconfigure (dmas_id[i],0 ) <0 ) { +RTE_LOG (ERR ,VHOST_PORT , +"Failed to unconfigure DMA %d in vhost.\n" , dmas_id[i]); +rte_exit (EXIT_FAILURE ,"Cannot use given DMA device\n" ); + } + } +/* clean up the EAL */ rte_eal_cleanup ();
Thanks,
Maxime
v8 cover vhost: introduce DMA vchannel unconfiguration
In reply to [email protected]From: Xuan Ding <[email protected]>
This patchset introduces a new API rte_vhost_async_dma_unconfigure() to help user to manually free DMA vChannels finished to use.
v8:
* Check inflight packets release virtual channel.
v7:
* Add inflight packets processing.
* Fix CI error.
v6:
* Move DMA unconfiguration to the end due to DMA devices maybe reused after destroy_device().
* Refine the doc to claim the DMA device should not be used in vhost after unconfiguration.
v5:
* Use mutex instead of spinlock.
* Improve code readability.
v4:
* Rebase to 22.11 rc1.
* Fix the usage of 'dma_ref_count' to make sure the specified DMA device is not used by any vhost ports before unconfiguration.
v3:
* Rebase to latest DPDK.
* Refine some descriptions in the doc.
* Fix one bug in the vhost example.
v2:
* Add spinlock protection.
* Fix a memory leak issue.
* Refine the doc.
Xuan Ding (2):
vhost: introduce DMA vchannel unconfiguration
examples/vhost: unconfigure DMA vchannel
doc/guides/prog_guide/vhost_lib.rst | 5 ++ doc/guides/rel_notes/release_22_11.rst | 5 ++ examples/vhost/main.c | 8 +++ lib/vhost/rte_vhost_async.h | 20 +++++++ lib/vhost/version.map | 3 ++ lib/vhost/vhost.c | 72 ++++++++++++++++++++++++-- 6 files changed, 108 insertions(+), 5 deletions(-)
Signature · 1 line
2.17.1
PATCH v8 1/2 vhost: introduce DMA vchannel unconfiguration
From: Xuan Ding <[email protected]>
Add a new API rte_vhost_async_dma_unconfigure() to unconfigure DMA vChannels in vhost async data path. Lock protection are also added to protect DMA vChannel configuration and unconfiguration from concurrent calls.
--- doc/guides/prog_guide/vhost_lib.rst | 5 ++ doc/guides/rel_notes/release_22_11.rst | 5 ++ lib/vhost/rte_vhost_async.h | 20 +++++++ lib/vhost/version.map | 3 ++ lib/vhost/vhost.c | 72 ++++++++++++++++++++++++-- 5 files changed, 100 insertions(+), 5 deletions(-)
doc/guides/prog_guide/vhost_lib.rst +5 -0
@@ -323,6 +323,11 @@ The following is an overview of some key Vhost API functions: Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET, VDPA_DEVICE_TYPE_BLK. +* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)`` + + Clean DMA vChannel finished to use. After this function is called, + the specified DMA vChannel should no longer be used by the Vhost library. + Vhost-user Implementations --------------------------
doc/guides/rel_notes/release_22_11.rst +5 -0
@@ -225,6 +225,11 @@ New Features sysfs entries to adjust the minimum and maximum uncore frequency values, which works on Linux with Intel hardware only. +* **Added DMA vChannel unconfiguration for async vhost.** + + Added support to unconfigure DMA vChannel that is no longer used + by the Vhost library. + * **Rewritten pmdinfo script.** The ``dpdk-pmdinfo.py`` script was rewritten to produce valid JSON only.
lib/vhost/rte_vhost_async.h +20 -0
@@ -266,6 +266,26 @@ rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id, struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count, int *nr_inflight, int16_t dma_id, uint16_t vchan_id); +/** +* @warning +* @b EXPERIMENTAL: this API may change, or be removed, without prior notice. +* +* Unconfigure DMA vChannel in Vhost asynchronous data path. +* This function should be called when the specified DMA vChannel is no longer +* used by the Vhost library. Before this function is called, make sure there +* does not exist in-flight packets in DMA vChannel. +* +* @param dma_id +* the identifier of DMA device +* @param vchan_id +* the identifier of virtual DMA channel +* @return +* 0 on success, and -1 on failure +*/ +__rte_experimental +int +rte_vhost_async_dma_unconfigure (int16_t dma_id,uint16_t vchan_id); + #ifdef __cplusplus }#endif
lib/vhost/version.map +3 -0
@@ -94,6 +94,9 @@ EXPERIMENTAL { rte_vhost_async_try_dequeue_burst; rte_vhost_driver_get_vdpa_dev_type; rte_vhost_clear_queue; + + # added in 22.11 + rte_vhost_async_dma_unconfigure; }; INTERNAL {
lib/vhost/vhost.c +67 -5
124 lines not on this page · raw message
Signature · 1 line
2.17.1
Quoted 7 lines · On 10/25/22 10:25, [email protected] wrote:
From: Xuan Ding <[email protected]>
Add a new API
rte_vhost_async_dma_unconfigure()to unconfigure DMA vChannels in vhost async data path. Lock protection are also added to protect DMA vChannel configuration and unconfiguration from concurrent calls.
Quoted 9 lines
Signed-off-by Xuan Ding <[email protected]>--- doc/guides/prog_guide/vhost_lib.rst | 5 ++ doc/guides/rel_notes/release_22_11.rst | 5 ++ lib/vhost/rte_vhost_async.h | 20 +++++++ lib/vhost/version.map | 3 ++ lib/vhost/vhost.c | 72 ++++++++++++++++++++++++-- 5 files changed, 100 insertions(+), 5 deletions(-)
Thanks,
Maxime
Quoted 266 lines
-----Original Message-----
From: Ding, Xuan <[email protected]>
Sent: Tuesday, October 25, 2022 4:26 PM
To: [email protected]; Xia, Chenbo <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; He, Xingguang <[email protected]>; Ling, WeiX <[email protected]>; Jiang, Cheng1 <[email protected]>; Wang, YuanX <[email protected]>; Ma, WenwuX <[email protected]>; Ding, Xuan <[email protected]>
Subject: [PATCH v8 1/2] vhost: introduce DMA vchannel unconfigurationFrom: Xuan Ding <[email protected]>
Add a new API
rte_vhost_async_dma_unconfigure()to unconfigure DMA vChannels in vhost async data path. Lock protection are also added to protect DMA vChannel configuration and unconfiguration from concurrent calls.Signed-off-by Xuan Ding <[email protected]>--- doc/guides/prog_guide/vhost_lib.rst | 5 ++ doc/guides/rel_notes/release_22_11.rst | 5 ++ lib/vhost/rte_vhost_async.h | 20 +++++++ lib/vhost/version.map | 3 ++ lib/vhost/vhost.c | 72 ++++++++++++++++++++++++-- 5 files changed, 100 insertions(+), 5 deletions(-)doc/guides/prog_guide/vhost_lib.rst +0
-0@@ -323,6 +323,11 @@ The following is an overview of some key Vhost APIfunctions:
Get device type of vDPA device, such asVDPA_DEVICE_TYPE_NET,VDPA_DEVICE_TYPE_BLK.doc/guides/prog_guide/vhost_lib.rst +4
-0+* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)`` + + Clean DMA vChannel finished to use. After this function is called, + the specified DMA vChannel should no longer be used by the Vhostlibrary.
doc/guides/prog_guide/vhost_lib.rst +1
-0+ Vhost-user Implementations --------------------------doc/guides/rel_notes/release_22_11.rst +5
-0@@ -225,6 +225,11 @@ New Features sysfs entries to adjust the minimum and maximum uncore frequency values, which works on Linux with Intel hardware only. +* **Added DMA vChannel unconfiguration for async vhost.** + + Added support to unconfigure DMA vChannel that is no longer used + by the Vhost library. + * **Rewritten pmdinfo script.** The ``dpdk-pmdinfo.py`` script was rewritten to produce valid JSON only.lib/vhost/rte_vhost_async.h +3
-0@@ -266,6 +266,26 @@ rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id, struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count, int *nr_inflight, int16_t dma_id, uint16_t vchan_id); +/** +* @warning +* @b EXPERIMENTAL: this API may change, or be removed, without prior notice.
+ *
+ * Unconfigure DMA vChannel in Vhost asynchronous data path.
+ * This function should be called when the specified DMA vChannel is no longer
+ * used by the Vhost library. Before this function is called, make sure there
+ * does not exist in-flight packets in DMA vChannel.
+ *
+ * @paramdma_id
+ * the identifier of DMA device
+ * @paramvchan_id
+ * the identifier of virtual DMA channel
+ * @return
+ * 0 on success, and -1 on failure
+ */lib/vhost/rte_vhost_async.h +4
-0+__rte_experimental +int +rte_vhost_async_dma_unconfigure (int16_t dma_id,uint16_t vchan_id); + #ifdef __cplusplus }#endif lib/vhost/version.map +3
-0@@ -94,6 +94,9 @@ EXPERIMENTAL { rte_vhost_async_try_dequeue_burst; rte_vhost_driver_get_vdpa_dev_type; rte_vhost_clear_queue; + + # added in 22.11 + rte_vhost_async_dma_unconfigure; }; INTERNAL {lib/vhost/vhost.c +6
-3@@ -23,6 +23,7 @@struct virtio_net *vhost_devices[RTE_MAX_VHOST_DEVICE ];pthread_mutex_t vhost_dev_lock =PTHREAD_MUTEX_INITIALIZER ; +pthread_mutex_t vhost_dma_lock =PTHREAD_MUTEX_INITIALIZER ;struct vhost_vq_stats_name_off {char name[RTE_VHOST_STATS_NAME_SIZE ];@@ -1844,19 +1845,21 @@ rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id)void *pkts_cmpl_flag_addr;uint16_t max_desc; +pthread_mutex_lock (&vhost_dma_lock); +if (!rte_dma_is_valid (dma_id)) {VHOST_LOG_CONFIG ("dma" ,ERR ,"DMA %d is not found.\n" , dma_id); -return -1 ; + goto error; }if (rte_dma_info_get (dma_id, &info) !=0 ) { VHOST_LOG_CONFIG("dma", ERR, "Fail to get DMA %d information.\n", dma_id); -return -1 ; + goto error; }if (vchan_id >= info.max_vchans) {VHOST_LOG_CONFIG ("dma" ,ERR ,"Invalid DMA %d vChannel %u.\n" , dma_id, vchan_id); -return -1 ; + goto error;lib/vhost/vhost.c +1
-1}if (!dma_copy_track[dma_id].vchans) {@@ -1868,7 +1871,7 @@ rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id) VHOST_LOG_CONFIG("dma", ERR, "Failed to allocate vchans for DMA %d vChannel %u.\n", dma_id, vchan_id); -return -1 ; + goto error; }lib/vhost/vhost.c +1
-0dma_copy_track[dma_id].vchans = vchans;@@ -1877,6 +1880,7 @@ rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id)if (dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr) { VHOST_LOG_CONFIG("dma", INFO, "DMA %d vChannel %u already registered.\n", dma_id, vchan_id); +pthread_mutex_unlock (&vhost_dma_lock);return 0 ;lib/vhost/vhost.c +1
-1}@@ -1894,7 +1898,7 @@ rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id) rte_free(dma_copy_track[dma_id].vchans); dma_copy_track[dma_id].vchans =NULL ; } -return -1 ; + goto error; }dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr =
pkts_cmpl_flag_addr;lib/vhost/vhost.c +5
-0@@ -1902,7 +1906,12 @@ rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id) dma_copy_track[dma_id].vchans[vchan_id].ring_mask = max_desc - 1; dma_copy_track[dma_id].nr_vchans++; +pthread_mutex_unlock (&vhost_dma_lock);return 0 ; + +error: +pthread_mutex_unlock (&vhost_dma_lock); +return -1 ; }int
lib/vhost/vhost.c +20
-0@@ -2091,5 +2100,58 @@ int rte_vhost_vring_stats_reset(int vid, uint16_t queue_id)return 0 ; } +int +rte_vhost_async_dma_unconfigure (int16_t dma_id,uint16_t vchan_id) +{ +struct rte_dma_info info; +struct rte_dma_stats stats = {0 }; + +pthread_mutex_lock (&vhost_dma_lock); + +if (!rte_dma_is_valid (dma_id)) { +VHOST_LOG_CONFIG ("dma" ,ERR ,"DMA %d is not found.\n" , dma_id); + goto error; + } + +if (rte_dma_info_get (dma_id, &info) !=0 ) { + VHOST_LOG_CONFIG("dma", ERR, "Fail to get DMA %d information.\n", dma_id); + goto error; + } + +if (vchan_id >= info.max_vchans || !dma_copy_track[dma_id].vchans || + !dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr){
lib/vhost/vhost.c +1
-0+ VHOST_LOG_CONFIG("dma", ERR, "Invalid channel %d:%u.\n",
dma_id,vchan_id);lib/vhost/vhost.c +6
-0+ goto error; + } + +if (rte_dma_stats_get (dma_id, vchan_id, &stats) !=0 ) { + VHOST_LOG_CONFIG("dma", ERR, + "Failed to get stats for DMA %d vChannel %u.\n",
dma_id,vchan_id);lib/vhost/vhost.c +6
-0+ goto error; + } + +if (stats.submitted - stats.completed !=0 ) { + VHOST_LOG_CONFIG("dma", ERR, + "Do not unconfigure when there are inflightpackets.\n");
lib/vhost/vhost.c +4
-0+ goto error; + } + +rte_free(dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr
);lib/vhost/vhost.c +16
-1+ dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr =NULL ; + dma_copy_track[dma_id].nr_vchans--; + +if (dma_copy_track[dma_id].nr_vchans ==0 ) { +rte_free (dma_copy_track[dma_id].vchans); + dma_copy_track[dma_id].vchans =NULL ; + } + +pthread_mutex_unlock (&vhost_dma_lock); +return 0 ; + +error: +pthread_mutex_unlock (&vhost_dma_lock); +return -1 ; +} +RTE_LOG_REGISTER_SUFFIX (vhost_config_log_level, config,INFO );RTE_LOG_REGISTER_SUFFIX (vhost_data_log_level, data,WARNING ); --
Quoted 1 line
2.17.1
PATCH v8 2/2 examples/vhost: unconfigure DMA vchannel
In reply to [email protected]From: Xuan Ding <[email protected]>
This patch applies rte_vhost_async_dma_unconfigure() to manually free DMA vChannels. Before unconfiguration, make sure the specified DMA vChannel is no longer used by any vhost ports.
--- examples/vhost/main.c | 8 ++++++++ 1 file changed, 8 insertions(+)
examples/vhost/main.c +8 -0
@@ -2066,6 +2066,14 @@ main(int argc, char *argv[])RTE_LCORE_FOREACH_WORKER (lcore_id)rte_eal_wait_lcore (lcore_id); +for (i =0 ; i < dma_count; i++) { +if (rte_vhost_async_dma_unconfigure (dmas_id[i],0 ) <0 ) { +RTE_LOG (ERR ,VHOST_PORT , +"Failed to unconfigure DMA %d in vhost.\n" , dmas_id[i]); +rte_exit (EXIT_FAILURE ,"Cannot use given DMA device\n" ); + } + } +/* clean up the EAL */ rte_eal_cleanup ();
Signature · 1 line
2.17.1
RE: [PATCH v8 2/2] examples/vhost: unconfigure DMA vchannel
Quoted 10 lines
-----Original Message-----
From: Ding, Xuan <[email protected]>
Sent: Tuesday, October 25, 2022 4:26 PM
To: [email protected]; Xia, Chenbo <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; He, Xingguang <[email protected]>; Ling, WeiX <[email protected]>; Jiang, Cheng1 <[email protected]>; Wang, YuanX <[email protected]>; Ma, WenwuX <[email protected]>; Ding, Xuan <[email protected]>
Subject: [PATCH v8 2/2] examples/vhost: unconfigure DMA vchannel
Quoted 8 lines
From: Xuan Ding <[email protected]>
This patch applies
rte_vhost_async_dma_unconfigure()to manually free DMA vChannels. Before unconfiguration, make sure the specified DMA vChannel is no longer used by any vhost ports.Signed-off-by Xuan Ding <[email protected]>---
Re: [PATCH v8 2/2] examples/vhost: unconfigure DMA vchannel
In reply to [email protected]Quoted 20 lines · On 10/25/22 10:25, [email protected] wrote:
From: Xuan Ding <[email protected]>
This patch applies
rte_vhost_async_dma_unconfigure()to manually free DMA vChannels. Before unconfiguration, make sure the specified DMA vChannel is no longer used by any vhost ports.Signed-off-by Xuan Ding <[email protected]>--- examples/vhost/main.c | 8 ++++++++ 1 file changed, 8 insertions(+)examples/vhost/main.c +1
-0@@ -2066,6 +2066,14 @@ main(int argc, char *argv[])RTE_LCORE_FOREACH_WORKER (lcore_id)rte_eal_wait_lcore (lcore_id); +for (i =0 ; i < dma_count; i++) {
examples/vhost/main.c main(int argc, char *argv[]) · line 2079
Quoted 10 lines
examples/vhost/main.c +7
-0+if (rte_vhost_async_dma_unconfigure (dmas_id[i],0 ) <0 ) { +RTE_LOG (ERR ,VHOST_PORT , +"Failed to unconfigure DMA %d in vhost.\n" , dmas_id[i]); +rte_exit (EXIT_FAILURE ,"Cannot use given DMA device\n" ); + } + } +/* clean up the EAL */ rte_eal_cleanup ();
Thanks,
Maxime
RE: [PATCH v8 2/2] examples/vhost: unconfigure DMA vchannel
In reply to [email protected]Quoted 42 lines
-----Original Message-----
From: Ding, Xuan <[email protected]>
Sent: Tuesday, October 25, 2022 4:26 PM
To: [email protected]; Xia, Chenbo <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; He, Xingguang <[email protected]>; Ling, WeiX <[email protected]>; Jiang, Cheng1 <[email protected]>; Wang, YuanX <[email protected]>; Ma, WenwuX <[email protected]>; Ding, Xuan <[email protected]>
Subject: [PATCH v8 2/2] examples/vhost: unconfigure DMA vchannelFrom: Xuan Ding <[email protected]>
This patch applies
rte_vhost_async_dma_unconfigure()to manually free DMA vChannels. Before unconfiguration, make sure the specified DMA vChannel is no longer used by any vhost ports.Signed-off-by Xuan Ding <[email protected]>--- examples/vhost/main.c | 8 ++++++++ 1 file changed, 8 insertions(+)examples/vhost/main.c +8
-1@@ -2066,6 +2066,14 @@ main(int argc, char *argv[])RTE_LCORE_FOREACH_WORKER (lcore_id)rte_eal_wait_lcore (lcore_id); +for (i =0 ; i < dma_count; i++) { +if (rte_vhost_async_dma_unconfigure (dmas_id[i],0 ) <0 ) { +RTE_LOG (ERR ,VHOST_PORT , +"Failed to unconfigure DMA %d in vhost.\n" , dmas_id[i]); +rte_exit (EXIT_FAILURE ,"Cannot use given DMA device\n" ); + } + } +/* clean up the EAL */ rte_eal_cleanup (); --
Quoted 1 line
2.17.1
Quoted 59 lines
-----Original Message-----
From: Ding, Xuan <[email protected]>
Sent: Tuesday, October 25, 2022 4:26 PM
To: [email protected]; Xia, Chenbo <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; He, Xingguang <[email protected]>; Ling, WeiX <[email protected]>; Jiang, Cheng1 <[email protected]>; Wang, YuanX <[email protected]>; Ma, WenwuX <[email protected]>; Ding, Xuan <[email protected]>
Subject: [PATCH v8 0/2] vhost: introduce DMA vchannel unconfigurationFrom: Xuan Ding <[email protected]>
This patchset introduces a new API
rte_vhost_async_dma_unconfigure()to help user to manually free DMA vChannels finished to use.v8:
* Check inflight packets release virtual channel.v7:
* Add inflight packets processing.
* Fix CI error.v6:
* Move DMA unconfiguration to the end due to DMA devices maybe reused afterdestroy_device().
* Refine the doc to claim the DMA device should not be used in vhost after unconfiguration.v5:
* Use mutex instead of spinlock.
* Improve code readability.v4:
* Rebase to 22.11 rc1.
* Fix the usage of 'dma_ref_count' to make sure the specified DMA device is not used by any vhost ports before unconfiguration.v3:
* Rebase to latest DPDK.
* Refine some descriptions in the doc.
* Fix one bug in the vhost example.v2:
* Add spinlock protection.
* Fix a memory leak issue.
* Refine the doc.Xuan Ding (2):
vhost: introduce DMA vchannel unconfiguration
examples/vhost: unconfigure DMA vchanneldoc/guides/prog_guide/vhost_lib.rst | 5 ++ doc/guides/rel_notes/release_22_11.rst | 5 ++ examples/vhost/main.c | 8 +++ lib/vhost/rte_vhost_async.h | 20 +++++++ lib/vhost/version.map | 3 ++ lib/vhost/vhost.c | 72 ++++++++++++++++++++++++-- 6 files changed, 108 insertions(+), 5 deletions(-)
Quoted 2 lines
--
2.17.1
Series applied to next-virtio/main, thanks