Skip to contentSkip to conversation

update docs for vhost async API

PATCH Superseded [email protected] 3 participants 15 messages Jul 22, 2020 – Jul 28, 2020

lore
Share & tools
lore
#0 lore raw

cover update docs for vhost async API

From: Patrick Fu <[email protected]>

update documentations for vhost async API

Patrick Fu (2):
doc: update guides for vhost async APIs
doc: update release notes for vhost async APIs

 doc/guides/prog_guide/vhost_lib.rst    | 86 +++++++++++++++++++++++---
 doc/guides/rel_notes/release_20_08.rst |  5 ++
 2 files changed, 82 insertions(+), 9 deletions(-)
Signature · 1 line
2.18.4
#1 lore raw

PATCH 1/2 doc: update guides for vhost async APIs

From: Patrick Fu <[email protected]>

Update vhost guides to document vhost async APIs

---
 doc/guides/prog_guide/vhost_lib.rst | 86 ++++++++++++++++++++++++++---
 1 file changed, 77 insertions(+), 9 deletions(-)
doc/guides/prog_guide/vhost_lib.rst +77 -9
index db921f922..cce8b6ae7 100644
@@ -147,6 +147,21 @@ The following is an overview of some key Vhost API functions:
 
     It is disabled by default.
 
+  - ``RTE_VHOST_USER_ASYNC_COPY``
+
+    Asynchronous data path will be enabled when this flag is set. Async data
+    path allows applications to register async copy devices (typically
+    hardware DMA channels) to the vhost queues. Vhost leverages the copy
+    device registered to offload CPU from memory copy operations. A set of
+    async data path APIs are defined for DPDK applications to make use of
+    the async capability. Only packets enqueued/dequeued by async APIs are
+    processed through the async data path.
+
+    Currently this feature is only implemented on split ring enqueue data
+    path.
+
+    It is disabled by default.
+
 * ``rte_vhost_driver_set_features(path, features)``
 
   This function sets the feature bits the vhost-user driver supports. The
@@ -235,6 +250,59 @@ The following is an overview of some key Vhost API functions:
 
   Enable or disable zero copy feature of the vhost crypto backend.
 
+* ``rte_vhost_async_channel_register(vid, queue_id, features, ops)``
+
+  Register a vhost queue with async copy device channel.
+  Following device ``features`` must be specified together with the
+  registration:
+
+  * ``async_inorder``
+
+    Async copy device can guarantee the ordering of copy completion
+    sequence. Copies are completed in the same order with that at
+    the submission time.
+
+    Currently, only ``async_inorder`` capable device is supported by vhost.
+
+  * ``async_threshold``
+
+    The copy length (in bytes) below which CPU copy will be used even if
+    applications call async vhost APIs to enqueue/dequeue data.
+
+    Typical value is 512~1024 depending on the async device capability.
+
+  Applications must provide following ``ops`` callbacks for vhost lib to
+  work with the async copye devices:
+
+  * ``transfer_data(vid, queue_id, descs, opaque_data, count)``
+
+    vhost invokes this function to submit copy data to the async devices.
+    For non-async_inorder capable devices, ``opaque_data`` could be used
+    for identifying the completed packets.
+
+  * ``check_completed_copies(vid, queue_id, opaque_data, max_packets)``
+
+    vhost invokes this function to get the copy data completed by async
+    devices.
+
+* ``rte_vhost_async_channel_unregister(vid, queue_id)``
+
+  Unregister the async copy device from a vhost queue.
+
+* ``rte_vhost_submit_enqueue_burst(vid, queue_id, pkts, count)``
+
+  Submit an enqueue request to transmit ``count`` packets from host to guest
+  by async data path. Enqueue is not guaranteed to finish upon the return of
+  this API call.
+
+  Applications must not free the packets submitted for enqueue until the
+  packets are completed.
+
+* ``rte_vhost_poll_enqueue_completed(vid, queue_id, pkts, count)``
+
+  Poll enqueue completion status from async data path. Completed packets
+  are returned to applications through ``pkts``.
+
 Vhost-user Implementations
 --------------------------
 
@@ -294,16 +362,16 @@ Guest memory requirement
 
 * Memory pre-allocation
 
-  For non-zerocopy, guest memory pre-allocation is not a must. This can help
-  save of memory. If users really want the guest memory to be pre-allocated
-  (e.g., for performance reason), we can add option ``-mem-prealloc`` when
-  starting QEMU. Or, we can lock all memory at vhost side which will force
-  memory to be allocated when mmap at vhost side; option --mlockall in
-  ovs-dpdk is an example in hand.
+  For non-zerocopy non-async data path, guest memory pre-allocation is not a
+  must. This can help save of memory. If users really want the guest memory
+  to be pre-allocated (e.g., for performance reason), we can add option
+  ``-mem-prealloc`` when starting QEMU. Or, we can lock all memory at vhost
+  side which will force memory to be allocated when mmap at vhost side;
+  option --mlockall in ovs-dpdk is an example in hand.
 
-  For zerocopy, we force the VM memory to be pre-allocated at vhost lib when
-  mapping the guest memory; and also we need to lock the memory to prevent
-  pages being swapped out to disk.
+  For async data path or zerocopy, we force the VM memory to be pre-allocated
+  at vhost lib when mapping the guest memory; and also we need to lock the
+  memory to prevent pages being swapped out to disk.
 
 * Memory sharing
 
Signature · 1 line
2.18.4
#2 lore raw

Re: [dpdk-dev] [PATCH v1 1/2] doc: update guides for vhost async APIs

Hi Patrick,

Quoted 27 lines

-----Original Message-----
From: Fu, Patrick <[email protected]>
Sent: Wednesday, July 22, 2020 6:58 PM
To: [email protected]; [email protected]; Xia, Chenbo <[email protected]>
Cc: Fu, Patrick <[email protected]>
Subject: [PATCH v1 1/2] doc: update guides for vhost async APIs

From: Patrick Fu <[email protected]>

Update vhost guides to document vhost async APIs

---
 doc/guides/prog_guide/vhost_lib.rst | 86 ++++++++++++++++++++++++++---
 1 file changed, 77 insertions(+), 9 deletions(-)
doc/guides/prog_guide/vhost_lib.rst +0 -0
index db921f922..cce8b6ae7 100644
@@ -147,6 +147,21 @@ The following is an overview of some key Vhost API

functions:

It is disabled by default.

doc/guides/prog_guide/vhost_lib.rst The following is an overview of some key Vhost API functions: · line 155

Quoted 6 lines
doc/guides/prog_guide/vhost_lib.rst +6 -0
+  - ``RTE_VHOST_USER_ASYNC_COPY``
+
+    Asynchronous data path will be enabled when this flag is set. Async data
+    path allows applications to register async copy devices (typically
+    hardware DMA channels) to the vhost queues. Vhost leverages the copy
+    device registered to offload CPU from memory copy operations. A set of

You mean 'free' CPU from memory copy?

Quoted 30 lines
doc/guides/prog_guide/vhost_lib.rst +9 -0
+    async data path APIs are defined for DPDK applications to make use of
+    the async capability. Only packets enqueued/dequeued by async APIs are
+    processed through the async data path.
+
+    Currently this feature is only implemented on split ring enqueue data
+    path.
+
+    It is disabled by default.
+
 * ``rte_vhost_driver_set_features(path, features)``
 
   This function sets the feature bits the vhost-user driver supports. The @@ -

235,6 +250,59 @@ The following is an overview of some key Vhost API functions:

Enable or disable zero copy feature of the vhost crypto backend.

doc/guides/prog_guide/vhost_lib.rst +13 -0
+* ``rte_vhost_async_channel_register(vid, queue_id, features, ops)``
+
+  Register a vhost queue with async copy device channel.
+  Following device ``features`` must be specified together with the
+  registration:
+
+  * ``async_inorder``
+
+    Async copy device can guarantee the ordering of copy completion
+    sequence. Copies are completed in the same order with that at
+    the submission time.
+
+    Currently, only ``async_inorder`` capable device is supported by vhost.

doc/guides/prog_guide/vhost_lib.rst The following is an overview of some key Vhost API functions:

Quoted 10 lines
doc/guides/prog_guide/vhost_lib.rst +10 -0
+
+  * ``async_threshold``
+
+    The copy length (in bytes) below which CPU copy will be used even if
+    applications call async vhost APIs to enqueue/dequeue data.
+
+    Typical value is 512~1024 depending on the async device capability.
+
+  Applications must provide following ``ops`` callbacks for vhost lib
+ to  work with the async copye devices:

s/copye/copy

Quoted 5 lines
doc/guides/prog_guide/vhost_lib.rst +5 -0
+
+  * ``transfer_data(vid, queue_id, descs, opaque_data, count)``
+
+    vhost invokes this function to submit copy data to the async devices.
+    For non-async_inorder capable devices, ``opaque_data`` could be used

doc/guides/prog_guide/vhost_lib.rst The following is an overview of some key Vhost API functions: · line 290

Quoted 10 lines
doc/guides/prog_guide/vhost_lib.rst +10 -0
+    for identifying the completed packets.
+
+  * ``check_completed_copies(vid, queue_id, opaque_data, max_packets)``
+
+    vhost invokes this function to get the copy data completed by async
+    devices.
+
+* ``rte_vhost_async_channel_unregister(vid, queue_id)``
+
+  Unregister the async copy device from a vhost queue.

'Copy device channel' may be more accurate?

Quoted 29 lines
doc/guides/prog_guide/vhost_lib.rst +16 -6
+
+* ``rte_vhost_submit_enqueue_burst(vid, queue_id, pkts, count)``
+
+  Submit an enqueue request to transmit ``count`` packets from host to
+ guest  by async data path. Enqueue is not guaranteed to finish upon
+ the return of  this API call.
+
+  Applications must not free the packets submitted for enqueue until
+ the  packets are completed.
+
+* ``rte_vhost_poll_enqueue_completed(vid, queue_id, pkts, count)``
+
+  Poll enqueue completion status from async data path. Completed
+ packets  are returned to applications through ``pkts``.
+
 Vhost-user Implementations
 --------------------------
@@ -294,16 +362,16 @@ Guest memory requirement
 
 * Memory pre-allocation
 
-  For non-zerocopy, guest memory pre-allocation is not a must. This can help
-  save of memory. If users really want the guest memory to be pre-allocated
-  (e.g., for performance reason), we can add option ``-mem-prealloc`` when
-  starting QEMU. Or, we can lock all memory at vhost side which will force
-  memory to be allocated when mmap at vhost side; option --mlockall in
-  ovs-dpdk is an example in hand.
+  For non-zerocopy non-async data path, guest memory pre-allocation is

doc/guides/prog_guide/vhost_lib.rst Guest memory requirement · line 372

Quoted 10 lines
doc/guides/prog_guide/vhost_lib.rst +6 -3
+ not a  must. This can help save of memory. If users really want the
+ guest memory  to be pre-allocated (e.g., for performance reason), we
+ can add option  ``-mem-prealloc`` when starting QEMU. Or, we can lock
+ all memory at vhost  side which will force memory to be allocated when
+ mmap at vhost side;  option --mlockall in ovs-dpdk is an example in hand.
 
-  For zerocopy, we force the VM memory to be pre-allocated at vhost lib when
-  mapping the guest memory; and also we need to lock the memory to prevent
-  pages being swapped out to disk.
+  For async data path or zerocopy, we force the VM memory to be

'For async or zerocopy data path' may be better?

Thanks!
Chenbo

Quoted 7 lines
doc/guides/prog_guide/vhost_lib.rst +2 -1
+ pre-allocated  at vhost lib when mapping the guest memory; and also we
+ need to lock the  memory to prevent pages being swapped out to disk.
 
 * Memory sharing
 
--

2.18.4

#3 lore raw

Re: [dpdk-dev] [PATCH v1 1/2] doc: update guides for vhost async APIs

Thanks for comments. v2 patch sent with all the changes suggested.

Thanks,

Patrick

Quoted 173 lines

-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Wednesday, July 22, 2020 7:21 PM
To: Fu, Patrick <[email protected]>; [email protected]; [email protected]
Subject: RE: [PATCH v1 1/2] doc: update guides for vhost async APIs

Hi Patrick,

Quoted 29 lines

-----Original Message-----
From: Fu, Patrick <[email protected]>
Sent: Wednesday, July 22, 2020 6:58 PM
To: [email protected]; [email protected]; Xia, Chenbo <[email protected]>
Cc: Fu, Patrick <[email protected]>
Subject: [PATCH v1 1/2] doc: update guides for vhost async APIs

From: Patrick Fu <[email protected]>

Update vhost guides to document vhost async APIs

---
 doc/guides/prog_guide/vhost_lib.rst | 86
++++++++++++++++++++++++++---
 1 file changed, 77 insertions(+), 9 deletions(-)
doc/guides/prog_guide/vhost_lib.rst +0 -0
index db921f922..cce8b6ae7 100644
@@ -147,6 +147,21 @@ The following is an overview of some key Vhost

API
functions:

It is disabled by default.

doc/guides/prog_guide/vhost_lib.rst The following is an overview of some key Vhost API functions:

Quoted 3 lines
doc/guides/prog_guide/vhost_lib.rst +3 -0
+  - ``RTE_VHOST_USER_ASYNC_COPY``
+
+    Asynchronous data path will be enabled when this flag is set. Async

data

doc/guides/prog_guide/vhost_lib.rst The following is an overview of some key Vhost API functions:

Quoted 4 lines
doc/guides/prog_guide/vhost_lib.rst +4 -0
+    path allows applications to register async copy devices (typically
+    hardware DMA channels) to the vhost queues. Vhost leverages the copy
+    device registered to offload CPU from memory copy operations. A
+ set of

You mean 'free' CPU from memory copy?

doc/guides/prog_guide/vhost_lib.rst

Quoted 2 lines
doc/guides/prog_guide/vhost_lib.rst +2 -0
+    async data path APIs are defined for DPDK applications to make use of
+    the async capability. Only packets enqueued/dequeued by async APIs

are

Quoted 29 lines
doc/guides/prog_guide/vhost_lib.rst +7 -0
+    processed through the async data path.
+
+    Currently this feature is only implemented on split ring enqueue data
+    path.
+
+    It is disabled by default.
+
 * ``rte_vhost_driver_set_features(path, features)``
 
   This function sets the feature bits the vhost-user driver supports.

The @@ -
235,6 +250,59 @@ The following is an overview of some key Vhost API functions:

Enable or disable zero copy feature of the vhost crypto backend.

doc/guides/prog_guide/vhost_lib.rst +13 -0
+* ``rte_vhost_async_channel_register(vid, queue_id, features, ops)``
+
+  Register a vhost queue with async copy device channel.
+  Following device ``features`` must be specified together with the
+  registration:
+
+  * ``async_inorder``
+
+    Async copy device can guarantee the ordering of copy completion
+    sequence. Copies are completed in the same order with that at
+    the submission time.
+
+    Currently, only ``async_inorder`` capable device is supported by vhost.

doc/guides/prog_guide/vhost_lib.rst The following is an overview of some key Vhost API functions:

Quoted 10 lines
doc/guides/prog_guide/vhost_lib.rst +10 -0
+
+  * ``async_threshold``
+
+    The copy length (in bytes) below which CPU copy will be used even if
+    applications call async vhost APIs to enqueue/dequeue data.
+
+    Typical value is 512~1024 depending on the async device capability.
+
+  Applications must provide following ``ops`` callbacks for vhost lib
+ to  work with the async copye devices:

s/copye/copy

Quoted 6 lines
doc/guides/prog_guide/vhost_lib.rst +6 -0
+
+  * ``transfer_data(vid, queue_id, descs, opaque_data, count)``
+
+    vhost invokes this function to submit copy data to the async devices.
+    For non-async_inorder capable devices, ``opaque_data`` could be used
+    for identifying the completed packets.

doc/guides/prog_guide/vhost_lib.rst The following is an overview of some key Vhost API functions: · line 290

Quoted 10 lines
doc/guides/prog_guide/vhost_lib.rst +10 -0
+
+  * ``check_completed_copies(vid, queue_id, opaque_data,
+ max_packets)``
+
+    vhost invokes this function to get the copy data completed by async
+    devices.
+
+* ``rte_vhost_async_channel_unregister(vid, queue_id)``
+
+  Unregister the async copy device from a vhost queue.

'Copy device channel' may be more accurate?

Quoted 32 lines
doc/guides/prog_guide/vhost_lib.rst +15 -1
+
+* ``rte_vhost_submit_enqueue_burst(vid, queue_id, pkts, count)``
+
+  Submit an enqueue request to transmit ``count`` packets from host
+ to guest  by async data path. Enqueue is not guaranteed to finish
+ upon the return of  this API call.
+
+  Applications must not free the packets submitted for enqueue until
+ the  packets are completed.
+
+* ``rte_vhost_poll_enqueue_completed(vid, queue_id, pkts, count)``
+
+  Poll enqueue completion status from async data path. Completed
+ packets  are returned to applications through ``pkts``.
+
 Vhost-user Implementations
 --------------------------
@@ -294,16 +362,16 @@ Guest memory requirement
 
 * Memory pre-allocation
 
-  For non-zerocopy, guest memory pre-allocation is not a must. This

can help
- save of memory. If users really want the guest memory to be pre-allocated
- (e.g., for performance reason), we can add option ``-mem-prealloc`` when
- starting QEMU. Or, we can lock all memory at vhost side which will force
- memory to be allocated when mmap at vhost side; option --mlockall in

doc/guides/prog_guide/vhost_lib.rst

Quoted 7 lines
doc/guides/prog_guide/vhost_lib.rst +6 -1
-  ovs-dpdk is an example in hand.
+  For non-zerocopy non-async data path, guest memory pre-allocation
+ is not a  must. This can help save of memory. If users really want
+ the guest memory  to be pre-allocated (e.g., for performance
+ reason), we can add option  ``-mem-prealloc`` when starting QEMU.
+ Or, we can lock all memory at vhost  side which will force memory to
+ be allocated when mmap at vhost side;  option --mlockall in ovs-dpdk is

an example in hand.

doc/guides/prog_guide/vhost_lib.rst

Quoted 7 lines

- For zerocopy, we force the VM memory to be pre-allocated at vhost lib when
- mapping the guest memory; and also we need to lock the memory to prevent

doc/guides/prog_guide/vhost_lib.rst +1 -1
-  pages being swapped out to disk.
+  For async data path or zerocopy, we force the VM memory to be

'For async or zerocopy data path' may be better?

Thanks!
Chenbo

Quoted 7 lines

+ pre-allocated at vhost lib when mapping the guest memory; and also
+ we need to lock the memory to prevent pages being swapped out to disk.

* Memory sharing

--
2.18.4

#4 lore raw

PATCH 2/2 doc: update release notes for vhost async APIs

In reply to [email protected]

From: Patrick Fu <[email protected]>

Update 20.08 release notes to include vhost async APIs

---
 doc/guides/rel_notes/release_20_08.rst | 5 +++++
 1 file changed, 5 insertions(+)
doc/guides/rel_notes/release_20_08.rst +5 -0
index f19b74872..47ed488ed 100644
@@ -225,6 +225,11 @@ New Features
   See the :doc:`../sample_app_ug/l2_forward_real_virtual` for more
   details of this parameter usage.
 
+* **Added vhost async data path APIs.**
+
+  4 new APIs has been added to enable vhost async data path. With the new
+  data path, applications can register vhost with copy devices, such as DMA
+  engines, to offload memory copy operations.
 
 Removed Items
 -------------
Signature · 1 line
2.18.4
#5 lore raw

Re: [dpdk-dev] [PATCH v1 2/2] doc: update release notes for vhost async APIs

Hi Patrick,

Quoted 23 lines

-----Original Message-----
From: Fu, Patrick <[email protected]>
Sent: Wednesday, July 22, 2020 6:58 PM
To: [email protected]; [email protected]; Xia, Chenbo <[email protected]>
Cc: Fu, Patrick <[email protected]>
Subject: [PATCH v1 2/2] doc: update release notes for vhost async APIs

From: Patrick Fu <[email protected]>

Update 20.08 release notes to include vhost async APIs

---
 doc/guides/rel_notes/release_20_08.rst | 5 +++++
 1 file changed, 5 insertions(+)
doc/guides/rel_notes/release_20_08.rst +0 -0
index f19b74872..47ed488ed 100644

doc/guides/rel_notes/release_20_08.rst New Features · line 230

Quoted 6 lines
doc/guides/rel_notes/release_20_08.rst +3 -0
@@ -225,6 +225,11 @@ New Features
   See the :doc:`../sample_app_ug/l2_forward_real_virtual` for more
   details of this parameter usage.
 
+* **Added vhost async data path APIs.**
+
+  4 new APIs has been added to enable vhost async data path. With the

s/has/have

Should we mention you only added enqueue support?

Thanks,
Chenbo

Quoted 7 lines
doc/guides/rel_notes/release_20_08.rst +2 -1
+ new  data path, applications can register vhost with copy devices,
+ such as DMA  engines, to offload memory copy operations.
 
 Removed Items
 -------------
--

2.18.4

#6 lore raw

Re: [dpdk-dev] [PATCH v1 2/2] doc: update release notes for vhost async APIs

Hi,

Quoted 35 lines

-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Wednesday, July 22, 2020 7:25 PM
To: Fu, Patrick <[email protected]>; [email protected]; [email protected]
Subject: RE: [PATCH v1 2/2] doc: update release notes for vhost async APIs

Hi Patrick,

Quoted 26 lines

-----Original Message-----
From: Fu, Patrick <[email protected]>
Sent: Wednesday, July 22, 2020 6:58 PM
To: [email protected]; [email protected]; Xia, Chenbo <[email protected]>
Cc: Fu, Patrick <[email protected]>
Subject: [PATCH v1 2/2] doc: update release notes for vhost async APIs

From: Patrick Fu <[email protected]>

Update 20.08 release notes to include vhost async APIs

---
 doc/guides/rel_notes/release_20_08.rst | 5 +++++
 1 file changed, 5 insertions(+)
doc/guides/rel_notes/release_20_08.rst +0 -0
index f19b74872..47ed488ed 100644
@@ -225,6 +225,11 @@ New Features
   See the :doc:`../sample_app_ug/l2_forward_real_virtual` for more
   details of this parameter usage.
 
Quoted 7 lines

doc/guides/rel_notes/release_20_08.rst New Features

Quoted 3 lines
doc/guides/rel_notes/release_20_08.rst +3 -0
+* **Added vhost async data path APIs.**
+
+  4 new APIs has been added to enable vhost async data path. With the

s/has/have

Should we mention you only added enqueue support?

I included more details in v2 patch for the release notes.

Thanks,

Patrick

#7 lore raw

v2 cover update docs for vhost async APIs

In reply to [email protected]

From: Patrick Fu <[email protected]>

update docs for vhost async APIs

v2:
- fix typos in vhost_lib.rst
- minor rewording in vhost_lib.rst
- add more detailed API notes in release notes
- rebase the patch to 20.08 rc2

Patrick Fu (2):
doc: update guides for vhost async APIs
doc: update release notes for vhost async APIs

 doc/guides/prog_guide/vhost_lib.rst    | 86 +++++++++++++++++++++++---
 doc/guides/rel_notes/release_20_08.rst |  6 ++
 2 files changed, 83 insertions(+), 9 deletions(-)
Signature · 1 line
2.18.4
#8 lore raw

PATCH v2 1/2 doc: update guides for vhost async APIs

From: Patrick Fu <[email protected]>

Update vhost guides to document vhost async APIs

---
 doc/guides/prog_guide/vhost_lib.rst | 86 ++++++++++++++++++++++++++---
 1 file changed, 77 insertions(+), 9 deletions(-)
doc/guides/prog_guide/vhost_lib.rst +77 -9
index db921f922..b892eec67 100644
@@ -147,6 +147,21 @@ The following is an overview of some key Vhost API functions:
 
     It is disabled by default.
 
+  - ``RTE_VHOST_USER_ASYNC_COPY``
+
+    Asynchronous data path will be enabled when this flag is set. Async data
+    path allows applications to register async copy devices (typically
+    hardware DMA channels) to the vhost queues. Vhost leverages the copy
+    device registered to free CPU from memory copy operations. A set of
+    async data path APIs are defined for DPDK applications to make use of
+    the async capability. Only packets enqueued/dequeued by async APIs are
+    processed through the async data path.
+
+    Currently this feature is only implemented on split ring enqueue data
+    path.
+
+    It is disabled by default.
+
 * ``rte_vhost_driver_set_features(path, features)``
 
   This function sets the feature bits the vhost-user driver supports. The
@@ -235,6 +250,59 @@ The following is an overview of some key Vhost API functions:
 
   Enable or disable zero copy feature of the vhost crypto backend.
 
+* ``rte_vhost_async_channel_register(vid, queue_id, features, ops)``
+
+  Register a vhost queue with async copy device channel.
+  Following device ``features`` must be specified together with the
+  registration:
+
+  * ``async_inorder``
+
+    Async copy device can guarantee the ordering of copy completion
+    sequence. Copies are completed in the same order with that at
+    the submission time.
+
+    Currently, only ``async_inorder`` capable device is supported by vhost.
+
+  * ``async_threshold``
+
+    The copy length (in bytes) below which CPU copy will be used even if
+    applications call async vhost APIs to enqueue/dequeue data.
+
+    Typical value is 512~1024 depending on the async device capability.
+
+  Applications must provide following ``ops`` callbacks for vhost lib to
+  work with the async copy devices:
+
+  * ``transfer_data(vid, queue_id, descs, opaque_data, count)``
+
+    vhost invokes this function to submit copy data to the async devices.
+    For non-async_inorder capable devices, ``opaque_data`` could be used
+    for identifying the completed packets.
+
+  * ``check_completed_copies(vid, queue_id, opaque_data, max_packets)``
+
+    vhost invokes this function to get the copy data completed by async
+    devices.
+
+* ``rte_vhost_async_channel_unregister(vid, queue_id)``
+
+  Unregister the async copy device channel from a vhost queue.
+
+* ``rte_vhost_submit_enqueue_burst(vid, queue_id, pkts, count)``
+
+  Submit an enqueue request to transmit ``count`` packets from host to guest
+  by async data path. Enqueue is not guaranteed to finish upon the return of
+  this API call.
+
+  Applications must not free the packets submitted for enqueue until the
+  packets are completed.
+
+* ``rte_vhost_poll_enqueue_completed(vid, queue_id, pkts, count)``
+
+  Poll enqueue completion status from async data path. Completed packets
+  are returned to applications through ``pkts``.
+
 Vhost-user Implementations
 --------------------------
 
@@ -294,16 +362,16 @@ Guest memory requirement
 
 * Memory pre-allocation
 
-  For non-zerocopy, guest memory pre-allocation is not a must. This can help
-  save of memory. If users really want the guest memory to be pre-allocated
-  (e.g., for performance reason), we can add option ``-mem-prealloc`` when
-  starting QEMU. Or, we can lock all memory at vhost side which will force
-  memory to be allocated when mmap at vhost side; option --mlockall in
-  ovs-dpdk is an example in hand.
+  For non-zerocopy non-async data path, guest memory pre-allocation is not a
+  must. This can help save of memory. If users really want the guest memory
+  to be pre-allocated (e.g., for performance reason), we can add option
+  ``-mem-prealloc`` when starting QEMU. Or, we can lock all memory at vhost
+  side which will force memory to be allocated when mmap at vhost side;
+  option --mlockall in ovs-dpdk is an example in hand.
 
-  For zerocopy, we force the VM memory to be pre-allocated at vhost lib when
-  mapping the guest memory; and also we need to lock the memory to prevent
-  pages being swapped out to disk.
+  For async and zerocopy data path, we force the VM memory to be
+  pre-allocated at vhost lib when mapping the guest memory; and also we need
+  to lock the memory to prevent pages being swapped out to disk.
 
 * Memory sharing
 
Signature · 1 line
2.18.4
Reviewed-by on v2 1/2
#9 lore raw

Re: [dpdk-dev] [PATCH v2 1/2] doc: update guides for vhost async APIs

Quoted 134 lines

-----Original Message-----
From: Fu, Patrick <[email protected]>
Sent: Wednesday, July 22, 2020 11:02 PM
To: [email protected]; [email protected]; Xia, Chenbo <[email protected]>
Cc: Fu, Patrick <[email protected]>
Subject: [PATCH v2 1/2] doc: update guides for vhost async APIs

From: Patrick Fu <[email protected]>

Update vhost guides to document vhost async APIs

---
 doc/guides/prog_guide/vhost_lib.rst | 86 ++++++++++++++++++++++++++---
 1 file changed, 77 insertions(+), 9 deletions(-)
doc/guides/prog_guide/vhost_lib.rst +0 -0
index db921f922..b892eec67 100644
@@ -147,6 +147,21 @@ The following is an overview of some key Vhost API

functions:

It is disabled by default.

doc/guides/prog_guide/vhost_lib.rst +15 -0
+  - ``RTE_VHOST_USER_ASYNC_COPY``
+
+    Asynchronous data path will be enabled when this flag is set. Async data
+    path allows applications to register async copy devices (typically
+    hardware DMA channels) to the vhost queues. Vhost leverages the copy
+    device registered to free CPU from memory copy operations. A set of
+    async data path APIs are defined for DPDK applications to make use of
+    the async capability. Only packets enqueued/dequeued by async APIs are
+    processed through the async data path.
+
+    Currently this feature is only implemented on split ring enqueue data
+    path.
+
+    It is disabled by default.
+
 * ``rte_vhost_driver_set_features(path, features)``
 
   This function sets the feature bits the vhost-user driver supports. The @@ -

235,6 +250,59 @@ The following is an overview of some key Vhost API functions:

Enable or disable zero copy feature of the vhost crypto backend.

doc/guides/prog_guide/vhost_lib.rst +62 -10
+* ``rte_vhost_async_channel_register(vid, queue_id, features, ops)``
+
+  Register a vhost queue with async copy device channel.
+  Following device ``features`` must be specified together with the
+  registration:
+
+  * ``async_inorder``
+
+    Async copy device can guarantee the ordering of copy completion
+    sequence. Copies are completed in the same order with that at
+    the submission time.
+
+    Currently, only ``async_inorder`` capable device is supported by vhost.
+
+  * ``async_threshold``
+
+    The copy length (in bytes) below which CPU copy will be used even if
+    applications call async vhost APIs to enqueue/dequeue data.
+
+    Typical value is 512~1024 depending on the async device capability.
+
+  Applications must provide following ``ops`` callbacks for vhost lib
+ to  work with the async copy devices:
+
+  * ``transfer_data(vid, queue_id, descs, opaque_data, count)``
+
+    vhost invokes this function to submit copy data to the async devices.
+    For non-async_inorder capable devices, ``opaque_data`` could be used
+    for identifying the completed packets.
+
+  * ``check_completed_copies(vid, queue_id, opaque_data, max_packets)``
+
+    vhost invokes this function to get the copy data completed by async
+    devices.
+
+* ``rte_vhost_async_channel_unregister(vid, queue_id)``
+
+  Unregister the async copy device channel from a vhost queue.
+
+* ``rte_vhost_submit_enqueue_burst(vid, queue_id, pkts, count)``
+
+  Submit an enqueue request to transmit ``count`` packets from host to
+ guest  by async data path. Enqueue is not guaranteed to finish upon
+ the return of  this API call.
+
+  Applications must not free the packets submitted for enqueue until
+ the  packets are completed.
+
+* ``rte_vhost_poll_enqueue_completed(vid, queue_id, pkts, count)``
+
+  Poll enqueue completion status from async data path. Completed
+ packets  are returned to applications through ``pkts``.
+
 Vhost-user Implementations
 --------------------------
@@ -294,16 +362,16 @@ Guest memory requirement
 
 * Memory pre-allocation
 
-  For non-zerocopy, guest memory pre-allocation is not a must. This can help
-  save of memory. If users really want the guest memory to be pre-allocated
-  (e.g., for performance reason), we can add option ``-mem-prealloc`` when
-  starting QEMU. Or, we can lock all memory at vhost side which will force
-  memory to be allocated when mmap at vhost side; option --mlockall in
-  ovs-dpdk is an example in hand.
+  For non-zerocopy non-async data path, guest memory pre-allocation is
+ not a  must. This can help save of memory. If users really want the
+ guest memory  to be pre-allocated (e.g., for performance reason), we
+ can add option  ``-mem-prealloc`` when starting QEMU. Or, we can lock
+ all memory at vhost  side which will force memory to be allocated when
+ mmap at vhost side;  option --mlockall in ovs-dpdk is an example in hand.
 
-  For zerocopy, we force the VM memory to be pre-allocated at vhost lib when
-  mapping the guest memory; and also we need to lock the memory to prevent
-  pages being swapped out to disk.
+  For async and zerocopy data path, we force the VM memory to be
+ pre-allocated at vhost lib when mapping the guest memory; and also we
+ need  to lock the memory to prevent pages being swapped out to disk.
 
 * Memory sharing
 
--
Quoted 1 line

2.18.4

Reviewed-by on v2 1/2
#10 lore raw

Re: [dpdk-dev] [PATCH v2 1/2] doc: update guides for vhost async APIs

In reply to [email protected]
Quoted 9 lines · On 7/22/20 5:01 PM, [email protected] wrote:

From: Patrick Fu <[email protected]>

Update vhost guides to document vhost async APIs

---
 doc/guides/prog_guide/vhost_lib.rst | 86 ++++++++++++++++++++++++++---
 1 file changed, 77 insertions(+), 9 deletions(-)

Thanks,
Maxime

#11 lore raw

PATCH v2 2/2 doc: update release notes for vhost async APIs

In reply to [email protected]

From: Patrick Fu <[email protected]>

Update 20.08 release notes to include vhost async APIs

---
 doc/guides/rel_notes/release_20_08.rst | 6 ++++++
 1 file changed, 6 insertions(+)
doc/guides/rel_notes/release_20_08.rst +6 -0
index d65f77250..dc8936056 100644
@@ -243,6 +243,12 @@ New Features
   of ingress packets to specific NIC queues.
   See the :doc:`../sample_app_ug/ipsec_secgw` for more details.
 
+* **Added vhost async data path APIs.**
+
+  4 new APIs have been added to enable vhost async data path, including:
+
+  * Async device channel register/unregister APIs
+  * Async packets enqueue/completion APIs (only split ring was implemented)
 
 Removed Items
 -------------
Signature · 1 line
2.18.4
Reviewed-by on v2 2/2
#12 lore raw

Re: [dpdk-dev] [PATCH v2 2/2] doc: update release notes for vhost async APIs

Quoted 37 lines

-----Original Message-----
From: Fu, Patrick <[email protected]>
Sent: Wednesday, July 22, 2020 11:02 PM
To: [email protected]; [email protected]; Xia, Chenbo <[email protected]>
Cc: Fu, Patrick <[email protected]>
Subject: [PATCH v2 2/2] doc: update release notes for vhost async APIs

From: Patrick Fu <[email protected]>

Update 20.08 release notes to include vhost async APIs

---
 doc/guides/rel_notes/release_20_08.rst | 6 ++++++
 1 file changed, 6 insertions(+)
doc/guides/rel_notes/release_20_08.rst +6 -1
index d65f77250..dc8936056 100644
@@ -243,6 +243,12 @@ New Features
   of ingress packets to specific NIC queues.
   See the :doc:`../sample_app_ug/ipsec_secgw` for more details.
 
+* **Added vhost async data path APIs.**
+
+  4 new APIs have been added to enable vhost async data path, including:
+
+  * Async device channel register/unregister APIs
+  * Async packets enqueue/completion APIs (only split ring was implemented)
 
 Removed Items
 -------------
--
Quoted 1 line

2.18.4

Reviewed-by on v2 2/2
#13 lore raw

Re: [dpdk-dev] [PATCH v2 2/2] doc: update release notes for vhost async APIs

In reply to [email protected]
Quoted 17 lines · On 7/22/20 5:01 PM, [email protected] wrote:

From: Patrick Fu <[email protected]>

Update 20.08 release notes to include vhost async APIs

---
 doc/guides/rel_notes/release_20_08.rst | 6 ++++++
 1 file changed, 6 insertions(+)
doc/guides/rel_notes/release_20_08.rst +0 -0
index d65f77250..dc8936056 100644
@@ -243,6 +243,12 @@ New Features
   of ingress packets to specific NIC queues.
   See the :doc:`../sample_app_ug/ipsec_secgw` for more details.

doc/guides/rel_notes/release_20_08.rst New Features · line 254

Quoted 10 lines
doc/guides/rel_notes/release_20_08.rst +6 -0
 
+* **Added vhost async data path APIs.**
+
+  4 new APIs have been added to enable vhost async data path, including:
+
+  * Async device channel register/unregister APIs
+  * Async packets enqueue/completion APIs (only split ring was implemented)
 
 Removed Items
 -------------

Thanks,
Maxime

#14 lore raw

Re: [dpdk-dev] [PATCH v2 0/2] update docs for vhost async APIs

In reply to [email protected]
Quoted 11 lines · On 7/22/20 5:01 PM, [email protected] wrote:

From: Patrick Fu <[email protected]>

update docs for vhost async APIs

v2:
- fix typos in vhost_lib.rst
- minor rewording in vhost_lib.rst
- add more detailed API notes in release notes
- rebase the patch to 20.08 rc2

Quoted 8 lines

Patrick Fu (2):
doc: update guides for vhost async APIs
doc: update release notes for vhost async APIs

 doc/guides/prog_guide/vhost_lib.rst    | 86 +++++++++++++++++++++++---
 doc/guides/rel_notes/release_20_08.rst |  6 ++
 2 files changed, 83 insertions(+), 9 deletions(-)

Applied to dpdk-next-virtio/master

Thanks!
Maxime