Drop vhost async (DMA-accelerated) datapath
cover Drop vhost async (DMA-accelerated) datapath
The integration of this feature in OVS has been postponed following feedback how the current API forced the complexity on the application side. There is no contributor working on making those changes.
The feature has no other identified opensource consumer.
On the other hand, this feature has been left unattended for some years now while the "normal" datapath is actively maintained.
It is highly likely the async datapath code, which never underwent a stabilisation phase, is riddled with bugs.
Given the craziness around finding security bugs with AI, and since the vhost library is a good target, I propose to drop this feature and reduce the library attack surface.
Signature · 29 lines
David Marchand David Marchand (3): examples: stop using vhost async datapath vhost: refuse async datapath vhost: drop async datapath doc/guides/prog_guide/vhost_lib.rst | 151 +- doc/guides/rel_notes/release_26_11.rst | 20 + examples/vhost/Makefile | 2 - examples/vhost/main.c | 436 +--- examples/vhost/main.h | 21 +- examples/vhost/meson.build | 2 - examples/vhost_crypto/Makefile | 1 - examples/vhost_crypto/main.c | 3 +- examples/vhost_crypto/meson.build | 1 - lib/vhost/meson.build | 3 +- lib/vhost/rte_vhost.h | 2 +- lib/vhost/rte_vhost_async.h | 293 --- lib/vhost/socket.c | 27 +- lib/vhost/vhost.c | 434 ---- lib/vhost/vhost.h | 115 - lib/vhost/vhost_user.c | 215 +- lib/vhost/virtio_net.c | 3275 +++++------------------- 17 files changed, 708 insertions(+), 4293 deletions(-) delete mode 100644 lib/vhost/rte_vhost_async.h -- 2.54.0
RFC 1/3 examples: stop using vhost async datapath
Stop using vhost async as this feature will be removed in a next commit. The vhost-crypto example was requesting async copy for no identified reason.
Thanks to this, we can drop use of experimental api in those examples.
--- examples/vhost/Makefile | 2 - examples/vhost/main.c | 436 +----------------------------- examples/vhost/main.h | 21 +- examples/vhost/meson.build | 2 - examples/vhost_crypto/Makefile | 1 - examples/vhost_crypto/main.c | 3 +- examples/vhost_crypto/meson.build | 1 - 7 files changed, 13 insertions(+), 453 deletions(-)
examples/vhost/Makefile +0 -2
@@ -28,8 +28,6 @@ CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk) LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk) -CFLAGS += -DALLOW_EXPERIMENTAL_API - build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build $(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
examples/vhost/main.c +10 -426
603 lines not on this page · raw message
examples/vhost/main.h +2 -19
@@ -8,7 +8,6 @@#include <sys/queue.h> #include <rte_ether.h> -#include <rte_pci.h> /* Macros for printing using RTE_LOG */ #define RTE_LOGTYPE_VHOST_CONFIG RTE_LOGTYPE_USER1
@@ -92,17 +91,6 @@ struct lcore_info {struct vhost_dev_tailq_list vdev_list; }; -struct dma_info { -struct rte_pci_addr addr; -int16_t dev_id; -bool async_enabled; -}; - -struct dma_for_vhost { -struct dma_info dmas[RTE_MAX_QUEUES_PER_PORT *2 ]; -uint32_t async_flag; -}; -/* we implement non-extra virtio net features */ #define VIRTIO_NET_FEATURES 0
@@ -116,14 +104,9 @@ uint16_t builtin_enqueue_pkts(struct vhost_dev *dev, uint16_t queue_id,uint16_t builtin_dequeue_pkts (struct vhost_dev *dev,uint16_t queue_id,struct rte_mempool *mbuf_pool,struct rte_mbuf **pkts,uint16_t count); -uint16_t sync_enqueue_pkts (struct vhost_dev *dev,uint16_t queue_id, -struct rte_mbuf **pkts,uint32_t count); -uint16_t sync_dequeue_pkts (struct vhost_dev *dev,uint16_t queue_id, -struct rte_mempool *mbuf_pool, -struct rte_mbuf **pkts,uint16_t count); -uint16_t async_enqueue_pkts (struct vhost_dev *dev,uint16_t queue_id, +uint16_t enqueue_pkts (struct vhost_dev *dev,uint16_t queue_id,struct rte_mbuf **pkts,uint32_t count); -uint16_t async_dequeue_pkts (struct vhost_dev *dev,uint16_t queue_id, +uint16_t dequeue_pkts (struct vhost_dev *dev,uint16_t queue_id,struct rte_mempool *mbuf_pool,struct rte_mbuf **pkts,uint16_t count);#endif /* _MAIN_H_ */
examples/vhost/meson.build +0 -2
@@ -12,8 +12,6 @@ if not is_linux endif deps += 'vhost' -deps += 'dmadev' -allow_experimental_apis = true sources = files( 'main.c', 'virtio_net.c',
examples/vhost_crypto/Makefile +0 -1
@@ -6,7 +6,6 @@ APP = vhost-crypto # all source are stored in SRCS-y SRCS-y := main.c -CFLAGS += -DALLOW_EXPERIMENTAL_API PKGCONF ?= pkg-config
examples/vhost_crypto/main.c +1 -2
@@ -622,8 +622,7 @@ main(int argc, char *argv[]) }for (j =0 ; j < lo->nb_sockets; j++) { - ret =rte_vhost_driver_register (lo->socket_files[j], -RTE_VHOST_USER_ASYNC_COPY ); + ret =rte_vhost_driver_register (lo->socket_files[j],0 );if (ret <0 ) {RTE_LOG (ERR ,USER1 ,"socket %s already exists\n" , lo->socket_files[j]);
examples/vhost_crypto/meson.build +0 -1
@@ -6,7 +6,6 @@ # To build this example as a standalone application with an already-installed # DPDK instance, use 'make' -allow_experimental_apis = true deps += ['vhost', 'cryptodev'] sources = files( 'main.c',
Signature · 1 line
2.54.0
Re: [RFC 1/3] examples: stop using vhost async datapath
Hello Maxime,
Quoted 4 lines · David Marchand
Stop using vhost async as this feature will be removed in a next commit. The vhost-crypto example was requesting async copy for no identified reason.
Well, unfortunately, this was the tree hiding the forest... I'll need more changes to avoid breaking vhost crypto zero copy stuff.
My understanding is that a call to rte_vhost_crypto_set_zero_copy() could be used to detect guest pages mapping is needed.. does it look ok to you?
Signature · 1 line
David Marchand
RFC 2/3 vhost: refuse async datapath
In reply to David MarchandAs a first step for dropping the whole feature, refuse initialisation when RTE_VHOST_USER_ASYNC_COPY is passed.
This is mainly make more visible those changes and isolate them from dropping all the code in the next commit.
--- doc/guides/prog_guide/vhost_lib.rst | 15 -- doc/guides/rel_notes/release_26_11.rst | 7 + lib/vhost/rte_vhost.h | 2 +- lib/vhost/socket.c | 27 +--- lib/vhost/vhost_user.c | 210 +------------------------ 5 files changed, 17 insertions(+), 244 deletions(-)
doc/guides/prog_guide/vhost_lib.rst +0 -15
@@ -103,21 +103,6 @@ The following is an overview of some key Vhost API functions: It is disabled by default. - - ``RTE_VHOST_USER_ASYNC_COPY`` - - Asynchronous data path will be enabled when this flag is set. Async - data path allows applications to enable DMA acceleration for vhost - queues. Vhost leverages the registered DMA channels to free CPU from - memory copy operations in data path. A set of async data path APIs are - defined for DPDK applications to make use of the async capability. Only - packets enqueued/dequeued by async APIs are processed through the async - data path. - - Currently this feature is only implemented on split ring enqueue data - path. - - It is disabled by default. - - ``RTE_VHOST_USER_NET_COMPLIANT_OL_FLAGS`` Since v16.04, the vhost library forwards checksum and gso requests for
doc/guides/rel_notes/release_26_11.rst +7 -0
@@ -68,6 +68,13 @@ Removed Items Also, make sure to start the actual text at the margin. ======================================================= +* **Removed vhost async (DMA-accelerated) data path.** + + The experimental async data path APIs have been removed from the vhost library. + The following functions and flag are no longer available: + + - ``RTE_VHOST_USER_ASYNC_COPY`` + API Changes -----------
lib/vhost/rte_vhost.h +1 -1
@@ -38,7 +38,7 @@ extern "C" {#define RTE_VHOST_USER_EXTBUF_SUPPORT (1ULL << 5)/* support only linear buffers (no chained mbufs) */ #define RTE_VHOST_USER_LINEARBUF_SUPPORT (1ULL << 6) -#define RTE_VHOST_USER_ASYNC_COPY (1ULL << 7) +/* Bit 7 was RTE_VHOST_USER_ASYNC_COPY, now unsupported */ #define RTE_VHOST_USER_NET_COMPLIANT_OL_FLAGS (1ULL << 8)#define RTE_VHOST_USER_NET_STATS_ENABLE (1ULL << 9)#define RTE_VHOST_USER_ASYNC_CONNECT (1ULL << 10)
lib/vhost/socket.c +6 -21
@@ -210,7 +210,6 @@ vhost_user_add_connection(int fd, struct vhost_user_socket *vsocket)size_t size;struct vhost_user_connection *conn;int ret; -struct virtio_net *dev;if (vsocket ==NULL )return ;
@@ -241,13 +240,6 @@ vhost_user_add_connection(int fd, struct vhost_user_socket *vsocket)if (vsocket->linearbuf)vhost_enable_linearbuf (vid); -if (vsocket->async_copy) { - dev =get_device (vid); - -if (dev) - dev->async_copy =1 ; - } -VHOST_CONFIG_LOG (vsocket->path,INFO ,"new device, handle is %d" , vid);if (vsocket->notify_ops->new_connection) {
@@ -911,6 +903,11 @@ rte_vhost_driver_register(const char *path, uint64_t flags)if (!path)return -1 ; +if (flags & (1ULL <<7 )) { +VHOST_CONFIG_LOG (path,ERR ,"async copy flag (bit 7) is no longer supported" ); +return -1 ; + } +pthread_mutex_lock (&vhost_user.mutex);if (vhost_user.vsocket_cnt ==MAX_VHOST_SOCKET ) {
@@ -938,7 +935,7 @@ rte_vhost_driver_register(const char *path, uint64_t flags) vsocket->max_queue_pairs =VHOST_MAX_QUEUE_PAIRS ; vsocket->extbuf = flags &RTE_VHOST_USER_EXTBUF_SUPPORT ; vsocket->linearbuf = flags &RTE_VHOST_USER_LINEARBUF_SUPPORT ; - vsocket->async_copy = flags &RTE_VHOST_USER_ASYNC_COPY ; + vsocket->async_copy = false; vsocket->net_compliant_ol_flags = flags &RTE_VHOST_USER_NET_COMPLIANT_OL_FLAGS ; vsocket->stats_enabled = flags &RTE_VHOST_USER_NET_STATS_ENABLE ; vsocket->async_connect = flags &RTE_VHOST_USER_ASYNC_CONNECT ;
@@ -947,12 +944,6 @@ rte_vhost_driver_register(const char *path, uint64_t flags)else vsocket->iommu_support = flags &RTE_VHOST_USER_IOMMU_SUPPORT ; -if (vsocket->async_copy && (vsocket->iommu_support || - (flags &RTE_VHOST_USER_POSTCOPY_SUPPORT ))) { -VHOST_CONFIG_LOG (path,ERR ,"async copy with IOMMU or post-copy not supported" ); - goto out_mutex; - } -/* * Set the supported features correctly for the builtin vhost-user * net driver.
@@ -975,12 +966,6 @@ rte_vhost_driver_register(const char *path, uint64_t flags) vsocket->protocol_features =VHOST_USER_PROTOCOL_FEATURES ; } -if (vsocket->async_copy) { - vsocket->supported_features &= ~(1ULL <<VHOST_F_LOG_ALL ); - vsocket->features &= ~(1ULL <<VHOST_F_LOG_ALL ); -VHOST_CONFIG_LOG (path,INFO ,"logging feature is disabled in async copy mode" ); - } -/* * We'll not be able to receive a buffer from guest in linear mode * without external buffer if it will not fit in a single mbuf, which is
lib/vhost/vhost_user.c +3 -207
287 lines not on this page · raw message
Signature · 1 line
2.54.0
RFC 3/3 vhost: drop async datapath
In reply to David MarchandAll the API related to this feature is experimental, allowing us by the project policy to drop the feature in one go.
--- doc/guides/prog_guide/vhost_lib.rst | 136 +- doc/guides/rel_notes/release_26_11.rst | 13 + lib/vhost/meson.build | 3 +- lib/vhost/rte_vhost_async.h | 293 --- lib/vhost/socket.c | 2 - lib/vhost/vhost.c | 434 ---- lib/vhost/vhost.h | 115 - lib/vhost/vhost_user.c | 5 - lib/vhost/virtio_net.c | 3275 +++++------------------- 9 files changed, 679 insertions(+), 3597 deletions(-) delete mode 100644 lib/vhost/rte_vhost_async.h
doc/guides/prog_guide/vhost_lib.rst +5 -131
@@ -212,76 +212,6 @@ The following is an overview of some key Vhost API functions: Enable or disable zero copy feature of the vhost crypto backend. -* ``rte_vhost_async_dma_configure(dma_id, vchan_id)`` - - Tell vhost which DMA vChannel is going to use. This function needs to - be called before register async data-path for vring. - -* ``rte_vhost_async_channel_register(vid, queue_id)`` - - Register async DMA acceleration for a vhost queue after vring is enabled. - -* ``rte_vhost_async_channel_register_thread_unsafe(vid, queue_id)`` - - Register async DMA acceleration for a vhost queue without performing - any locking. - - This function is only safe to call in vhost callback functions - (i.e., struct rte_vhost_device_ops). - -* ``rte_vhost_async_channel_unregister(vid, queue_id)`` - - Unregister the async DMA acceleration from a vhost queue. - Unregistration will fail, if the vhost queue has in-flight - packets that are not completed. - - Unregister async DMA acceleration in vring_state_changed() may - fail, as this API tries to acquire the spinlock of vhost - queue. The recommended way is to unregister async copy - devices for all vhost queues in destroy_device(), when a - virtio device is paused or shut down. - -* ``rte_vhost_async_channel_unregister_thread_unsafe(vid, queue_id)`` - - Unregister async DMA acceleration for a vhost queue without performing - any locking. - - This function is only safe to call in vhost callback functions - (i.e., struct rte_vhost_device_ops). - -* ``rte_vhost_submit_enqueue_burst(vid, queue_id, pkts, count, dma_id, vchan_id)`` - - Submit an enqueue request to transmit ``count`` packets from host to guest - by async data path. Applications must not free the packets submitted for - enqueue until the packets are completed. - -* ``rte_vhost_poll_enqueue_completed(vid, queue_id, pkts, count, dma_id, vchan_id)`` - - Poll enqueue completion status from async data path. Completed packets - are returned to applications through ``pkts``. - -* ``rte_vhost_async_get_inflight(vid, queue_id)`` - - This function returns the amount of in-flight packets for the vhost - queue using async acceleration. - - * ``rte_vhost_async_get_inflight_thread_unsafe(vid, queue_id)`` - - Get the number of inflight packets for a vhost queue without performing - any locking. It should only be used within the vhost ops, which already - holds the lock. - -* ``rte_vhost_clear_queue_thread_unsafe(vid, queue_id, **pkts, count, dma_id, vchan_id)`` - - Clear in-flight packets which are submitted to async channel in vhost - async data path without performing locking on virtqueue. Completed - packets are returned to applications through ``pkts``. - -* ``rte_vhost_clear_queue(vid, queue_id, **pkts, count, dma_id, vchan_id)`` - - Clear in-flight packets which are submitted to async channel in vhost async data - path. Completed packets are returned to applications through ``pkts``. - * ``rte_vhost_vring_call_nonblock(int vid, uint16_t vring_idx)`` Notify the guest that used descriptors have been added to the vring. This function
@@ -303,22 +233,11 @@ The following is an overview of some key Vhost API functions: This function resets the queue statistics. It requires statistics collection to be enabled at registration time. -* ``rte_vhost_async_try_dequeue_burst(vid, queue_id, mbuf_pool, pkts, count, - nr_inflight, dma_id, vchan_id)`` - - Receive ``count`` packets from guest to host in async data path, - and store them at ``pkts``. - * ``rte_vhost_driver_get_vdpa_dev_type(path, type)`` Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET, VDPA_DEVICE_TYPE_BLK. -* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)`` - - Clean up a DMA vChannel after use is finished. After this function is called, - the specified DMA vChannel should no longer be used by the Vhost library. - * ``rte_vhost_notify_guest(int vid, uint16_t queue_id)`` Inject the offloaded interrupt received by the 'guest_notify' callback,
@@ -387,16 +306,11 @@ Guest memory requirement * Memory pre-allocation - For non-async data path guest memory pre-allocation is not a - must but can help save memory. To do this we can add option - ``-mem-prealloc`` when starting QEMU, or we can lock all memory at vhost - side which will force memory to be allocated when it calls mmap - (option --mlockall in ovs-dpdk is an example in hand). - - - For async data path, we force the VM memory to be pre-allocated at vhost - lib when mapping the guest memory; and also we need to lock the memory to - prevent pages being swapped out to disk. + Guest memory pre-allocation is not a must but can help save memory. + To do this we can add option ``-mem-prealloc`` when starting QEMU, + or we can lock all memory at vhost side which will force memory to + be allocated when it calls mmap (option --mlockall in ovs-dpdk is + an example in hand). * Memory sharing
@@ -467,43 +381,3 @@ Finally, a set of device ops is defined for device specific operations: * ``get_notify_area`` Called to get the notify area info of the queue. - -Vhost asynchronous data path ----------------------------- - -Vhost asynchronous data path leverages DMA devices to offload memory -copies from the CPU and it is implemented in an asynchronous way. It -enables applications, like OVS, to save CPU cycles and hide memory copy -overhead, thus achieving higher throughput. - -Vhost doesn't manage DMA devices and applications, like OVS, need to -manage and configure DMA devices. Applications need to tell vhost what -DMA devices to use in every data path function call. This design enables -the flexibility for applications to dynamically use DMA channels in -different function modules, not limited in vhost. - -In addition, vhost supports M:N mapping between vrings and DMA virtual -channels. Specifically, one vring can use multiple different DMA channels -and one DMA channel can be shared by multiple vrings at the same time. -The reason of enabling one vring to use multiple DMA channels is that -it's possible that more than one dataplane threads enqueue packets to -the same vring with their own DMA virtual channels. Besides, the number -of DMA devices is limited. For the purpose of scaling, it's necessary to -support sharing DMA channels among vrings. - -* Async enqueue API usage - - In async enqueue path, rte_vhost_poll_enqueue_completed() needs to be - called in time to notify the guest of DMA copy completed packets. - Moreover, calling rte_vhost_submit_enqueue_burst() all the time but - not poll completed will cause the DMA ring to be full, which will - result in packet loss eventually. - -* Recommended IOVA mode in async datapath - - When DMA devices are bound to VFIO driver, VA mode is recommended. - For PA mode, page by page mapping may exceed IOMMU's max capability, - better to use 1G guest hugepage. - - For UIO driver or kernel driver, any VFIO related error messages - can be ignored.
doc/guides/rel_notes/release_26_11.rst +13 -0
@@ -73,6 +73,19 @@ Removed Items The experimental async data path APIs have been removed from the vhost library. The following functions and flag are no longer available: + - ``rte_vhost_async_channel_register`` + - ``rte_vhost_async_channel_register_thread_unsafe`` + - ``rte_vhost_async_channel_unregister`` + - ``rte_vhost_async_channel_unregister_thread_unsafe`` + - ``rte_vhost_async_dma_configure`` + - ``rte_vhost_async_dma_unconfigure`` + - ``rte_vhost_submit_enqueue_burst`` + - ``rte_vhost_poll_enqueue_completed`` + - ``rte_vhost_async_try_dequeue_burst`` + - ``rte_vhost_async_get_inflight`` + - ``rte_vhost_async_get_inflight_thread_unsafe`` + - ``rte_vhost_clear_queue`` + - ``rte_vhost_clear_queue_thread_unsafe`` - ``RTE_VHOST_USER_ASYNC_COPY``
lib/vhost/meson.build +1 -2
@@ -38,10 +38,9 @@ sources = files( headers = files( 'rte_vdpa.h', 'rte_vhost.h', - 'rte_vhost_async.h', 'rte_vhost_crypto.h', ) driver_sdk_headers = files( 'vdpa_driver.h', ) -deps += ['ethdev', 'cryptodev', 'hash', 'pci', 'dmadev'] +deps += ['ethdev', 'cryptodev', 'hash', 'pci']
lib/vhost/rte_vhost_async.h deleted +0 -293
293 lines not on this page · raw message
lib/vhost/socket.c +0 -2
@@ -43,7 +43,6 @@ struct vhost_user_socket {bool use_builtin_virtio_net;bool extbuf;bool linearbuf; -bool async_copy;bool net_compliant_ol_flags;bool stats_enabled;bool async_connect;
@@ -935,7 +934,6 @@ rte_vhost_driver_register(const char *path, uint64_t flags) vsocket->max_queue_pairs =VHOST_MAX_QUEUE_PAIRS ; vsocket->extbuf = flags &RTE_VHOST_USER_EXTBUF_SUPPORT ; vsocket->linearbuf = flags &RTE_VHOST_USER_LINEARBUF_SUPPORT ; - vsocket->async_copy = false; vsocket->net_compliant_ol_flags = flags &RTE_VHOST_USER_NET_COMPLIANT_OL_FLAGS ; vsocket->stats_enabled = flags &RTE_VHOST_USER_NET_STATS_ENABLE ; vsocket->async_connect = flags &RTE_VHOST_USER_ASYNC_CONNECT ;
lib/vhost/vhost.c +0 -434
463 lines not on this page · raw message
lib/vhost/vhost.h +0 -115
150 lines not on this page · raw message
lib/vhost/vhost_user.c +0 -5
@@ -2409,11 +2409,6 @@ vhost_user_set_vring_enable(struct virtio_net **pdev,if (!(dev->flags &VIRTIO_DEV_VDPA_CONFIGURED )) {/* vhost_user_lock_all_queue_pairs locked all qps */ VHOST_USER_ASSERT_LOCK (dev, vq,VHOST_USER_SET_VRING_ENABLE ); -if (enable && vq->async && vq->async->pkts_inflight_n) { -VHOST_CONFIG_LOG (dev->ifname,ERR , -"failed to enable vring. Inflight packets must be completed first" ); -return RTE_VHOST_MSG_RESULT_ERR ; - } } vq->enabled = enable;
lib/vhost/virtio_net.c +660 -2615
1,877 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.54.0
Part of the diff is not shown here. View the raw message.
Hi David,
Quoted 6 lines · David Marchand
The integration of this feature in OVS has been postponed following feedback how the current API forced the complexity on the application side. There is no contributor working on making those changes.
The feature has no other identified opensource consumer.
I am not aware of any either.
Quoted 8 lines
On the other hand, this feature has been left unattended for some years now while the "normal" datapath is actively maintained.
It is highly likely the async datapath code, which never underwent a stabilisation phase, is riddled with bugs.Given the craziness around finding security bugs with AI, and since the vhost library is a good target, I propose to drop this feature and reduce the library attack surface.
This feature was introduced by Intel a while back, so their acknowledgement that they have no ongoing work regarding Vhost async would be welcome.
Adding Bruce for intel (feel free to add relevant people), and tehcboard in case it requires a decision from it.
I favor dropping this feature.
Best regards,
Maxime
Quoted 32 lines
--
David MarchandDavid Marchand (3):
examples: stop using vhost async datapath
vhost: refuse async datapath
vhost: drop async datapathdoc/guides/prog_guide/vhost_lib.rst | 151 +- doc/guides/rel_notes/release_26_11.rst | 20 + examples/vhost/Makefile | 2 - examples/vhost/main.c | 436 +--- examples/vhost/main.h | 21 +- examples/vhost/meson.build | 2 - examples/vhost_crypto/Makefile | 1 - examples/vhost_crypto/main.c | 3 +- examples/vhost_crypto/meson.build | 1 - lib/vhost/meson.build | 3 +- lib/vhost/rte_vhost.h | 2 +- lib/vhost/rte_vhost_async.h | 293 --- lib/vhost/socket.c | 27 +- lib/vhost/vhost.c | 434 ---- lib/vhost/vhost.h | 115 - lib/vhost/vhost_user.c | 215 +- lib/vhost/virtio_net.c | 3275 +++++------------------- 17 files changed, 708 insertions(+), 4293 deletions(-) delete mode 100644 lib/vhost/rte_vhost_async.h--
2.54.0
cover Drop vhost async (DMA-accelerated) datapath
In reply to David MarchandThe integration of this feature in OVS has been postponed following feedback how the current API forced the complexity on the application side. There is no contributor working on making those changes.
The feature has no other identified opensource consumer.
On the other hand, this feature has been left unattended for some years now while the "normal" datapath is actively maintained.
It is highly likely the async datapath code, which never underwent a stabilisation phase, is riddled with bugs.
Given the craziness around finding security bugs with AI, and since the vhost library is a good target, I propose to drop this feature and reduce the library attack surface.
There is one complication with the vhost_crypto and some external applications that seemed to rely on the async flag to track and populate guest pages, so a new flag replacing async is added by this series.
Signature · 36 lines
David Marchand Changes since RFC: - maintained guest pages population in a new flag (vhost_crypto/SPDK), - removed remaining helpers which had sync_ in their names, David Marchand (5): vhost: decouple guest pages population from async datapath examples/vhost: stop using async datapath vhost: drop async datapath vhost: rename packed layout helpers for batches vhost: simplify some descriptor handling doc/guides/prog_guide/vhost_lib.rst | 152 +- doc/guides/rel_notes/release_26_11.rst | 20 + examples/vhost/Makefile | 2 - examples/vhost/main.c | 436 +-- examples/vhost/main.h | 21 +- examples/vhost/meson.build | 2 - examples/vhost_crypto/Makefile | 1 - examples/vhost_crypto/main.c | 2 +- examples/vhost_crypto/meson.build | 1 - lib/vhost/meson.build | 3 +- lib/vhost/rte_vhost.h | 2 +- lib/vhost/rte_vhost_async.h | 293 --- lib/vhost/socket.c | 20 +- lib/vhost/vhost.c | 434 --- lib/vhost/vhost.h | 116 +- lib/vhost/vhost_user.c | 115 +- lib/vhost/virtio_net.c | 3343 +++++------------------- 17 files changed, 736 insertions(+), 4227 deletions(-) delete mode 100644 lib/vhost/rte_vhost_async.h -- 2.54.0
PATCH 1/5 vhost: decouple guest pages population from async datapath
The vhost crypto zero-copy feature relies on the RTE_VHOST_USER_ASYNC_COPY flag to enable guest page tracking (guest_pages array population and MAP_POPULATE during mmap).
Introduce a RTE_VHOST_USER_MAP_POPULATE flag that maps to RTE_VHOST_USER_ASYNC_COPY at this point in time.
Adjust vhost_crypto example and remove the unnecessary ALLOW_EXPERIMENTAL_API flag as this example does not use any experimental APIs.
--- doc/guides/prog_guide/vhost_lib.rst | 9 +++++++++ examples/vhost_crypto/Makefile | 1 - examples/vhost_crypto/main.c | 2 +- examples/vhost_crypto/meson.build | 1 - lib/vhost/rte_vhost.h | 1 + lib/vhost/socket.c | 9 +++++++++ lib/vhost/vhost.h | 1 + lib/vhost/vhost_user.c | 4 ++-- 8 files changed, 23 insertions(+), 5 deletions(-)
doc/guides/prog_guide/vhost_lib.rst +9 -0
@@ -118,6 +118,15 @@ The following is an overview of some key Vhost API functions: It is disabled by default. + - ``RTE_VHOST_USER_MAP_POPULATE`` + + Guest memory regions will be mapped with ``MAP_POPULATE`` when this flag + is set, pre-faulting pages into memory. This is useful for applications + requiring direct access to guest memory, such as vhost-crypto zero-copy + operations. + + It is disabled by default. + - ``RTE_VHOST_USER_NET_COMPLIANT_OL_FLAGS`` Since v16.04, the vhost library forwards checksum and gso requests for
examples/vhost_crypto/Makefile +0 -1
@@ -6,7 +6,6 @@ APP = vhost-crypto # all source are stored in SRCS-y SRCS-y := main.c -CFLAGS += -DALLOW_EXPERIMENTAL_API PKGCONF ?= pkg-config
examples/vhost_crypto/main.c +1 -1
@@ -623,7 +623,7 @@ main(int argc, char *argv[])for (j =0 ; j < lo->nb_sockets; j++) { ret =rte_vhost_driver_register (lo->socket_files[j], -RTE_VHOST_USER_ASYNC_COPY ); +RTE_VHOST_USER_MAP_POPULATE );if (ret <0 ) {RTE_LOG (ERR ,USER1 ,"socket %s already exists\n" , lo->socket_files[j]);
examples/vhost_crypto/meson.build +0 -1
@@ -6,7 +6,6 @@ # To build this example as a standalone application with an already-installed # DPDK instance, use 'make' -allow_experimental_apis = true deps += ['vhost', 'cryptodev'] sources = files( 'main.c',
lib/vhost/rte_vhost.h +1 -0
@@ -39,6 +39,7 @@ extern "C" {/* support only linear buffers (no chained mbufs) */ #define RTE_VHOST_USER_LINEARBUF_SUPPORT (1ULL << 6)#define RTE_VHOST_USER_ASYNC_COPY (1ULL << 7) +#define RTE_VHOST_USER_MAP_POPULATE RTE_VHOST_USER_ASYNC_COPY#define RTE_VHOST_USER_NET_COMPLIANT_OL_FLAGS (1ULL << 8)#define RTE_VHOST_USER_NET_STATS_ENABLE (1ULL << 9)#define RTE_VHOST_USER_ASYNC_CONNECT (1ULL << 10)
lib/vhost/socket.c +9 -0
@@ -44,6 +44,7 @@ struct vhost_user_socket {bool extbuf;bool linearbuf;bool async_copy; +bool map_populate;bool net_compliant_ol_flags;bool stats_enabled;bool async_connect;
@@ -248,6 +249,13 @@ vhost_user_add_connection(int fd, struct vhost_user_socket *vsocket) dev->async_copy =1 ; } +if (vsocket->map_populate) { + dev =get_device (vid); + +if (dev) + dev->map_populate =1 ; + } +VHOST_CONFIG_LOG (vsocket->path,INFO ,"new device, handle is %d" , vid);if (vsocket->notify_ops->new_connection) {
@@ -939,6 +947,7 @@ rte_vhost_driver_register(const char *path, uint64_t flags) vsocket->extbuf = flags &RTE_VHOST_USER_EXTBUF_SUPPORT ; vsocket->linearbuf = flags &RTE_VHOST_USER_LINEARBUF_SUPPORT ; vsocket->async_copy = flags &RTE_VHOST_USER_ASYNC_COPY ; + vsocket->map_populate = flags &RTE_VHOST_USER_MAP_POPULATE ; vsocket->net_compliant_ol_flags = flags &RTE_VHOST_USER_NET_COMPLIANT_OL_FLAGS ; vsocket->stats_enabled = flags &RTE_VHOST_USER_NET_STATS_ENABLE ; vsocket->async_connect = flags &RTE_VHOST_USER_ASYNC_CONNECT ;
lib/vhost/vhost.h +1 -0
@@ -499,6 +499,7 @@ struct __rte_cache_aligned virtio_net {RTE_ATOMIC (int16_t ) broadcast_rarp;uint32_t nr_vring;int async_copy; +int map_populate;int extbuf;int linearbuf;
lib/vhost/vhost_user.c +2 -2
@@ -1384,7 +1384,7 @@ vhost_user_mmap_region(struct virtio_net *dev,return -1 ; } - populate = dev->async_copy ?MAP_POPULATE :0 ; + populate = dev->map_populate ?MAP_POPULATE :0 ; mmap_addr =mmap (NULL , mmap_size,PROT_READ |PROT_WRITE ,MAP_SHARED | populate, region->fd,0 );
@@ -1398,7 +1398,7 @@ vhost_user_mmap_region(struct virtio_net *dev, region->host_user_addr = (uint64_t )(uintptr_t )mmap_addr + mmap_offset;mem_set_dump (dev, mmap_addr, mmap_size, false, alignment); -if (dev->async_copy) { +if (dev->map_populate) {if (add_guest_pages (dev, region, alignment) <0 ) {VHOST_CONFIG_LOG (dev->ifname,ERR ,"adding guest pages to region failed." );
Signature · 1 line
2.54.0
PATCH 2/5 examples/vhost: stop using async datapath
In reply to David MarchandStop using vhost async as this feature will be removed in a next commit. Thanks to this, we can drop use of experimental api.
---
Changes since RFC:
- dropped vhost_crypto changes (now handled in patch 1),
--- examples/vhost/Makefile | 2 - examples/vhost/main.c | 436 +------------------------------------ examples/vhost/main.h | 21 +- examples/vhost/meson.build | 2 - 4 files changed, 12 insertions(+), 449 deletions(-)
examples/vhost/Makefile +0 -2
@@ -28,8 +28,6 @@ CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk) LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk) -CFLAGS += -DALLOW_EXPERIMENTAL_API - build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build $(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
examples/vhost/main.c +10 -426
603 lines not on this page · raw message
examples/vhost/main.h +2 -19
@@ -8,7 +8,6 @@#include <sys/queue.h> #include <rte_ether.h> -#include <rte_pci.h> /* Macros for printing using RTE_LOG */ #define RTE_LOGTYPE_VHOST_CONFIG RTE_LOGTYPE_USER1
@@ -92,17 +91,6 @@ struct lcore_info {struct vhost_dev_tailq_list vdev_list; }; -struct dma_info { -struct rte_pci_addr addr; -int16_t dev_id; -bool async_enabled; -}; - -struct dma_for_vhost { -struct dma_info dmas[RTE_MAX_QUEUES_PER_PORT *2 ]; -uint32_t async_flag; -}; -/* we implement non-extra virtio net features */ #define VIRTIO_NET_FEATURES 0
@@ -116,14 +104,9 @@ uint16_t builtin_enqueue_pkts(struct vhost_dev *dev, uint16_t queue_id,uint16_t builtin_dequeue_pkts (struct vhost_dev *dev,uint16_t queue_id,struct rte_mempool *mbuf_pool,struct rte_mbuf **pkts,uint16_t count); -uint16_t sync_enqueue_pkts (struct vhost_dev *dev,uint16_t queue_id, -struct rte_mbuf **pkts,uint32_t count); -uint16_t sync_dequeue_pkts (struct vhost_dev *dev,uint16_t queue_id, -struct rte_mempool *mbuf_pool, -struct rte_mbuf **pkts,uint16_t count); -uint16_t async_enqueue_pkts (struct vhost_dev *dev,uint16_t queue_id, +uint16_t enqueue_pkts (struct vhost_dev *dev,uint16_t queue_id,struct rte_mbuf **pkts,uint32_t count); -uint16_t async_dequeue_pkts (struct vhost_dev *dev,uint16_t queue_id, +uint16_t dequeue_pkts (struct vhost_dev *dev,uint16_t queue_id,struct rte_mempool *mbuf_pool,struct rte_mbuf **pkts,uint16_t count);#endif /* _MAIN_H_ */
examples/vhost/meson.build +0 -2
@@ -12,8 +12,6 @@ if not is_linux endif deps += 'vhost' -deps += 'dmadev' -allow_experimental_apis = true sources = files( 'main.c', 'virtio_net.c',
Signature · 1 line
2.54.0
PATCH 3/5 vhost: drop async datapath
In reply to David MarchandAll the API related to this feature is experimental, allowing us by the project policy to drop the feature in one go.
---
Changes since RFC:
- kept mmap populate behavior for vhost_crypto,
--- doc/guides/prog_guide/vhost_lib.rst | 151 +- doc/guides/rel_notes/release_26_11.rst | 20 + lib/vhost/meson.build | 3 +- lib/vhost/rte_vhost.h | 3 +- lib/vhost/rte_vhost_async.h | 293 --- lib/vhost/socket.c | 21 - lib/vhost/vhost.c | 434 ---- lib/vhost/vhost.h | 115 - lib/vhost/vhost_user.c | 111 - lib/vhost/virtio_net.c | 3275 +++++------------------- 10 files changed, 687 insertions(+), 3739 deletions(-) delete mode 100644 lib/vhost/rte_vhost_async.h
doc/guides/prog_guide/vhost_lib.rst +5 -146
@@ -103,21 +103,6 @@ The following is an overview of some key Vhost API functions: It is disabled by default. - - ``RTE_VHOST_USER_ASYNC_COPY`` - - Asynchronous data path will be enabled when this flag is set. Async - data path allows applications to enable DMA acceleration for vhost - queues. Vhost leverages the registered DMA channels to free CPU from - memory copy operations in data path. A set of async data path APIs are - defined for DPDK applications to make use of the async capability. Only - packets enqueued/dequeued by async APIs are processed through the async - data path. - - Currently this feature is only implemented on split ring enqueue data - path. - - It is disabled by default. - - ``RTE_VHOST_USER_MAP_POPULATE`` Guest memory regions will be mapped with ``MAP_POPULATE`` when this flag
@@ -236,76 +221,6 @@ The following is an overview of some key Vhost API functions: Enable or disable zero copy feature of the vhost crypto backend. -* ``rte_vhost_async_dma_configure(dma_id, vchan_id)`` - - Tell vhost which DMA vChannel is going to use. This function needs to - be called before register async data-path for vring. - -* ``rte_vhost_async_channel_register(vid, queue_id)`` - - Register async DMA acceleration for a vhost queue after vring is enabled. - -* ``rte_vhost_async_channel_register_thread_unsafe(vid, queue_id)`` - - Register async DMA acceleration for a vhost queue without performing - any locking. - - This function is only safe to call in vhost callback functions - (i.e., struct rte_vhost_device_ops). - -* ``rte_vhost_async_channel_unregister(vid, queue_id)`` - - Unregister the async DMA acceleration from a vhost queue. - Unregistration will fail, if the vhost queue has in-flight - packets that are not completed. - - Unregister async DMA acceleration in vring_state_changed() may - fail, as this API tries to acquire the spinlock of vhost - queue. The recommended way is to unregister async copy - devices for all vhost queues in destroy_device(), when a - virtio device is paused or shut down. - -* ``rte_vhost_async_channel_unregister_thread_unsafe(vid, queue_id)`` - - Unregister async DMA acceleration for a vhost queue without performing - any locking. - - This function is only safe to call in vhost callback functions - (i.e., struct rte_vhost_device_ops). - -* ``rte_vhost_submit_enqueue_burst(vid, queue_id, pkts, count, dma_id, vchan_id)`` - - Submit an enqueue request to transmit ``count`` packets from host to guest - by async data path. Applications must not free the packets submitted for - enqueue until the packets are completed. - -* ``rte_vhost_poll_enqueue_completed(vid, queue_id, pkts, count, dma_id, vchan_id)`` - - Poll enqueue completion status from async data path. Completed packets - are returned to applications through ``pkts``. - -* ``rte_vhost_async_get_inflight(vid, queue_id)`` - - This function returns the amount of in-flight packets for the vhost - queue using async acceleration. - - * ``rte_vhost_async_get_inflight_thread_unsafe(vid, queue_id)`` - - Get the number of inflight packets for a vhost queue without performing - any locking. It should only be used within the vhost ops, which already - holds the lock. - -* ``rte_vhost_clear_queue_thread_unsafe(vid, queue_id, **pkts, count, dma_id, vchan_id)`` - - Clear in-flight packets which are submitted to async channel in vhost - async data path without performing locking on virtqueue. Completed - packets are returned to applications through ``pkts``. - -* ``rte_vhost_clear_queue(vid, queue_id, **pkts, count, dma_id, vchan_id)`` - - Clear in-flight packets which are submitted to async channel in vhost async data - path. Completed packets are returned to applications through ``pkts``. - * ``rte_vhost_vring_call_nonblock(int vid, uint16_t vring_idx)`` Notify the guest that used descriptors have been added to the vring. This function
@@ -327,22 +242,11 @@ The following is an overview of some key Vhost API functions: This function resets the queue statistics. It requires statistics collection to be enabled at registration time. -* ``rte_vhost_async_try_dequeue_burst(vid, queue_id, mbuf_pool, pkts, count, - nr_inflight, dma_id, vchan_id)`` - - Receive ``count`` packets from guest to host in async data path, - and store them at ``pkts``. - * ``rte_vhost_driver_get_vdpa_dev_type(path, type)`` Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET, VDPA_DEVICE_TYPE_BLK. -* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)`` - - Clean up a DMA vChannel after use is finished. After this function is called, - the specified DMA vChannel should no longer be used by the Vhost library. - * ``rte_vhost_notify_guest(int vid, uint16_t queue_id)`` Inject the offloaded interrupt received by the 'guest_notify' callback,
@@ -411,16 +315,11 @@ Guest memory requirement * Memory pre-allocation - For non-async data path guest memory pre-allocation is not a - must but can help save memory. To do this we can add option - ``-mem-prealloc`` when starting QEMU, or we can lock all memory at vhost - side which will force memory to be allocated when it calls mmap - (option --mlockall in ovs-dpdk is an example in hand). - - - For async data path, we force the VM memory to be pre-allocated at vhost - lib when mapping the guest memory; and also we need to lock the memory to - prevent pages being swapped out to disk. + Guest memory pre-allocation is not a must but can help save memory. + To do this we can add option ``-mem-prealloc`` when starting QEMU, + or we can lock all memory at vhost side which will force memory to + be allocated when it calls mmap (option --mlockall in ovs-dpdk is + an example in hand). * Memory sharing
@@ -491,43 +390,3 @@ Finally, a set of device ops is defined for device specific operations: * ``get_notify_area`` Called to get the notify area info of the queue. - -Vhost asynchronous data path ----------------------------- - -Vhost asynchronous data path leverages DMA devices to offload memory -copies from the CPU and it is implemented in an asynchronous way. It -enables applications, like OVS, to save CPU cycles and hide memory copy -overhead, thus achieving higher throughput. - -Vhost doesn't manage DMA devices and applications, like OVS, need to -manage and configure DMA devices. Applications need to tell vhost what -DMA devices to use in every data path function call. This design enables -the flexibility for applications to dynamically use DMA channels in -different function modules, not limited in vhost. - -In addition, vhost supports M:N mapping between vrings and DMA virtual -channels. Specifically, one vring can use multiple different DMA channels -and one DMA channel can be shared by multiple vrings at the same time. -The reason of enabling one vring to use multiple DMA channels is that -it's possible that more than one dataplane threads enqueue packets to -the same vring with their own DMA virtual channels. Besides, the number -of DMA devices is limited. For the purpose of scaling, it's necessary to -support sharing DMA channels among vrings. - -* Async enqueue API usage - - In async enqueue path, rte_vhost_poll_enqueue_completed() needs to be - called in time to notify the guest of DMA copy completed packets. - Moreover, calling rte_vhost_submit_enqueue_burst() all the time but - not poll completed will cause the DMA ring to be full, which will - result in packet loss eventually. - -* Recommended IOVA mode in async datapath - - When DMA devices are bound to VFIO driver, VA mode is recommended. - For PA mode, page by page mapping may exceed IOMMU's max capability, - better to use 1G guest hugepage. - - For UIO driver or kernel driver, any VFIO related error messages - can be ignored.
doc/guides/rel_notes/release_26_11.rst +20 -0
@@ -79,6 +79,26 @@ Removed Items ``rte_rib6_is_equal`` * table: ``RTE_LPM_IPV6_ADDR_SIZE`` +* **Removed vhost async (DMA-accelerated) data path.** + + The experimental async data path APIs have been removed from the vhost library. + The following functions and flag are no longer available: + + - ``rte_vhost_async_channel_register`` + - ``rte_vhost_async_channel_register_thread_unsafe`` + - ``rte_vhost_async_channel_unregister`` + - ``rte_vhost_async_channel_unregister_thread_unsafe`` + - ``rte_vhost_async_dma_configure`` + - ``rte_vhost_async_dma_unconfigure`` + - ``rte_vhost_submit_enqueue_burst`` + - ``rte_vhost_poll_enqueue_completed`` + - ``rte_vhost_async_try_dequeue_burst`` + - ``rte_vhost_async_get_inflight`` + - ``rte_vhost_async_get_inflight_thread_unsafe`` + - ``rte_vhost_clear_queue`` + - ``rte_vhost_clear_queue_thread_unsafe`` + - ``RTE_VHOST_USER_ASYNC_COPY`` + API Changes -----------
lib/vhost/meson.build +1 -2
@@ -38,10 +38,9 @@ sources = files( headers = files( 'rte_vdpa.h', 'rte_vhost.h', - 'rte_vhost_async.h', 'rte_vhost_crypto.h', ) driver_sdk_headers = files( 'vdpa_driver.h', ) -deps += ['ethdev', 'cryptodev', 'hash', 'pci', 'dmadev'] +deps += ['ethdev', 'cryptodev', 'hash', 'pci']
lib/vhost/rte_vhost.h +1 -2
@@ -38,8 +38,7 @@ extern "C" {#define RTE_VHOST_USER_EXTBUF_SUPPORT (1ULL << 5)/* support only linear buffers (no chained mbufs) */ #define RTE_VHOST_USER_LINEARBUF_SUPPORT (1ULL << 6) -#define RTE_VHOST_USER_ASYNC_COPY (1ULL << 7) -#define RTE_VHOST_USER_MAP_POPULATE RTE_VHOST_USER_ASYNC_COPY +#define RTE_VHOST_USER_MAP_POPULATE (1ULL << 7)#define RTE_VHOST_USER_NET_COMPLIANT_OL_FLAGS (1ULL << 8)#define RTE_VHOST_USER_NET_STATS_ENABLE (1ULL << 9)#define RTE_VHOST_USER_ASYNC_CONNECT (1ULL << 10)
lib/vhost/rte_vhost_async.h deleted +0 -293
293 lines not on this page · raw message
lib/vhost/socket.c +0 -21
@@ -43,7 +43,6 @@ struct vhost_user_socket {bool use_builtin_virtio_net;bool extbuf;bool linearbuf; -bool async_copy;bool map_populate;bool net_compliant_ol_flags;bool stats_enabled;
@@ -242,13 +241,6 @@ vhost_user_add_connection(int fd, struct vhost_user_socket *vsocket)if (vsocket->linearbuf)vhost_enable_linearbuf (vid); -if (vsocket->async_copy) { - dev =get_device (vid); - -if (dev) - dev->async_copy =1 ; - } -if (vsocket->map_populate) { dev =get_device (vid);
@@ -946,7 +938,6 @@ rte_vhost_driver_register(const char *path, uint64_t flags) vsocket->max_queue_pairs =VHOST_MAX_QUEUE_PAIRS ; vsocket->extbuf = flags &RTE_VHOST_USER_EXTBUF_SUPPORT ; vsocket->linearbuf = flags &RTE_VHOST_USER_LINEARBUF_SUPPORT ; - vsocket->async_copy = flags &RTE_VHOST_USER_ASYNC_COPY ; vsocket->map_populate = flags &RTE_VHOST_USER_MAP_POPULATE ; vsocket->net_compliant_ol_flags = flags &RTE_VHOST_USER_NET_COMPLIANT_OL_FLAGS ; vsocket->stats_enabled = flags &RTE_VHOST_USER_NET_STATS_ENABLE ;
@@ -956,12 +947,6 @@ rte_vhost_driver_register(const char *path, uint64_t flags)else vsocket->iommu_support = flags &RTE_VHOST_USER_IOMMU_SUPPORT ; -if (vsocket->async_copy && (vsocket->iommu_support || - (flags &RTE_VHOST_USER_POSTCOPY_SUPPORT ))) { -VHOST_CONFIG_LOG (path,ERR ,"async copy with IOMMU or post-copy not supported" ); - goto out_mutex; - } -/* * Set the supported features correctly for the builtin vhost-user * net driver.
@@ -984,12 +969,6 @@ rte_vhost_driver_register(const char *path, uint64_t flags) vsocket->protocol_features =VHOST_USER_PROTOCOL_FEATURES ; } -if (vsocket->async_copy) { - vsocket->supported_features &= ~(1ULL <<VHOST_F_LOG_ALL ); - vsocket->features &= ~(1ULL <<VHOST_F_LOG_ALL ); -VHOST_CONFIG_LOG (path,INFO ,"logging feature is disabled in async copy mode" ); - } -/* * We'll not be able to receive a buffer from guest in linear mode * without external buffer if it will not fit in a single mbuf, which is
lib/vhost/vhost.c +0 -434
463 lines not on this page · raw message
lib/vhost/vhost.h +0 -115
150 lines not on this page · raw message
lib/vhost/vhost_user.c +0 -111
177 lines not on this page · raw message
lib/vhost/virtio_net.c +660 -2615
1,637 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.54.0
Part of the diff is not shown here. View the raw message.
PATCH 4/5 vhost: rename packed layout helpers for batches
In reply to David MarchandNow that the async datapath has been removed, the _sync in function names gives no useful information.
Besides, vhost_reserve_avail_batch_packed does not reserve anything and its name is confusing, rename it following the other rx/tx split/packed helpers convention.
--- lib/vhost/virtio_net.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-)
lib/vhost/virtio_net.c +6 -8
@@ -1111,7 +1111,7 @@ virtio_dev_rx_split(struct virtio_net *dev, struct vhost_virtqueue *vq, } static __rte_always_inline int -virtio_dev_rx_sync_batch_check(struct virtio_net *dev, +virtio_dev_rx_batch_check(struct virtio_net *dev, struct vhost_virtqueue *vq, struct rte_mbuf **pkts, uint64_t *desc_addrs,
@@ -1213,7 +1213,7 @@ virtio_dev_rx_batch_packed_copy(struct virtio_net *dev, } static __rte_always_inline int -virtio_dev_rx_sync_batch_packed(struct virtio_net *dev, +virtio_dev_rx_batch_packed(struct virtio_net *dev, struct vhost_virtqueue *vq, struct rte_mbuf **pkts) __rte_requires_shared_capability(&vq->iotlb_lock)
@@ -1221,7 +1221,7 @@ virtio_dev_rx_sync_batch_packed(struct virtio_net *dev,uint64_t desc_addrs[PACKED_BATCH_SIZE ];uint64_t lens[PACKED_BATCH_SIZE ]; -if (virtio_dev_rx_sync_batch_check (dev, vq, pkts, desc_addrs, lens) ==-1 ) +if (virtio_dev_rx_batch_check (dev, vq, pkts, desc_addrs, lens) ==-1 )return -1 ;if (vq->shadow_used_idx) {
@@ -1273,8 +1273,7 @@ virtio_dev_rx_packed(struct virtio_net *dev,rte_prefetch0 (&vq->desc_packed[vq->last_avail_idx]);if (count - pkt_idx >=PACKED_BATCH_SIZE ) { -if (!virtio_dev_rx_sync_batch_packed (dev, vq, - &pkts[pkt_idx])) { +if (!virtio_dev_rx_batch_packed (dev, vq, &pkts[pkt_idx])) { pkt_idx +=PACKED_BATCH_SIZE ;continue ; }
@@ -1971,7 +1970,7 @@ virtio_dev_tx_split_compliant(struct virtio_net *dev, } static __rte_always_inline int -vhost_reserve_avail_batch_packed(struct virtio_net *dev, +virtio_dev_tx_batch_packed_check(struct virtio_net *dev, struct vhost_virtqueue *vq, struct rte_mbuf **pkts, uint16_t avail_idx,
@@ -2055,8 +2054,7 @@ virtio_dev_tx_batch_packed(struct virtio_net *dev,uint16_t ids[PACKED_BATCH_SIZE ];uint16_t i; -if (vhost_reserve_avail_batch_packed (dev, vq, pkts, avail_idx, - desc_addrs, ids)) +if (virtio_dev_tx_batch_packed_check (dev, vq, pkts, avail_idx, desc_addrs, ids))return -1 ;vhost_for_each_try_unroll (i,0 ,PACKED_BATCH_SIZE )
Signature · 1 line
2.54.0
PATCH 5/5 vhost: simplify some descriptor handling
In reply to David MarchandNow that the async datapath has been removed, move back the descriptors copy/filling code in desc_to_mbuf/mbuf_to_desc and eliminate an intermediate helper.
Tracking buf_iova is not needed in the descriptor to mbuf case.
--- lib/vhost/virtio_net.c | 78 ++++++++++++++++-------------------------- 1 file changed, 30 insertions(+), 48 deletions(-)
lib/vhost/virtio_net.c +30 -48
@@ -76,6 +76,12 @@ vhost_queue_stats_update(const struct virtio_net *dev, struct vhost_virtqueue *v } } +static __rte_always_inline bool +vhost_can_batch_copy (struct vhost_virtqueue *vq,uint32_t cpy_len) +{ +return cpy_len <=MAX_BATCH_LEN && vq->batch_copy_nb_elems < vq->size; +} + static inline void do_data_copy_enqueue(struct virtio_net *dev, struct vhost_virtqueue *vq) __rte_requires_shared_capability(&vq->iotlb_lock)
@@ -843,44 +849,6 @@ copy_vnet_hdr_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, } } -static __rte_always_inline void -sync_fill_seg (struct virtio_net *dev,struct vhost_virtqueue *vq, -struct rte_mbuf *m,uint32_t mbuf_offset, -uint64_t buf_addr,uint64_t buf_iova,uint32_t cpy_len,bool to_desc) - __rte_requires_shared_capability(&vq->iotlb_lock) -{ -struct batch_copy_elem *batch_copy = vq->batch_copy_elems; - -if (likely (cpy_len >MAX_BATCH_LEN || vq->batch_copy_nb_elems >= vq->size)) { -if (to_desc) { -rte_memcpy ((void *)((uintptr_t )(buf_addr)), - rte_pktmbuf_mtod_offset(m, void *, mbuf_offset), - cpy_len); -vhost_log_cache_write_iova (dev, vq, buf_iova, cpy_len); -PRINT_PACKET (dev, (uintptr_t )(buf_addr), cpy_len,0 ); - }else { - rte_memcpy(rte_pktmbuf_mtod_offset(m, void *, mbuf_offset), - (void *)((uintptr_t)(buf_addr)), - cpy_len); - } - }else { -if (to_desc) { - batch_copy[vq->batch_copy_nb_elems].dst = - (void *)((uintptr_t )(buf_addr)); - batch_copy[vq->batch_copy_nb_elems].src = - rte_pktmbuf_mtod_offset(m, void *, mbuf_offset); - batch_copy[vq->batch_copy_nb_elems].log_addr = buf_iova; - }else { - batch_copy[vq->batch_copy_nb_elems].dst = - rte_pktmbuf_mtod_offset(m, void *, mbuf_offset); - batch_copy[vq->batch_copy_nb_elems].src = - (void *)((uintptr_t )(buf_addr)); - } - batch_copy[vq->batch_copy_nb_elems].len = cpy_len; - vq->batch_copy_nb_elems++; - } -} - static __rte_always_inline int mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, struct rte_mbuf *m, struct buf_vector *buf_vec,
@@ -976,9 +944,22 @@ mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, cpy_len =RTE_MIN (buf_avail, mbuf_avail); -sync_fill_seg (dev, vq, m, mbuf_offset, - buf_addr + buf_offset, - buf_iova + buf_offset, cpy_len, true); +if (likely (!vhost_can_batch_copy (vq, cpy_len))) { +rte_memcpy (RTE_PTR_ADD (buf_addr, buf_offset), + rte_pktmbuf_mtod_offset(m, void *, mbuf_offset), + cpy_len); +vhost_log_cache_write_iova (dev, vq, buf_iova + buf_offset, cpy_len); +PRINT_PACKET (dev,RTE_PTR_ADD (buf_addr, buf_offset), cpy_len,0 ); + }else { + vq->batch_copy_elems[vq->batch_copy_nb_elems].dst = +RTE_PTR_ADD (buf_addr, buf_offset); + vq->batch_copy_elems[vq->batch_copy_nb_elems].src = + rte_pktmbuf_mtod_offset(m, void *, mbuf_offset); + vq->batch_copy_elems[vq->batch_copy_nb_elems].log_addr = + buf_iova + buf_offset; + vq->batch_copy_elems[vq->batch_copy_nb_elems].len = cpy_len; + vq->batch_copy_nb_elems++; + } mbuf_avail -= cpy_len; mbuf_offset += cpy_len;
@@ -1679,7 +1660,7 @@ desc_to_mbuf(struct virtio_net *dev, struct vhost_virtqueue *vq, __rte_requires_shared_capability(&vq->iotlb_lock) {uint32_t buf_avail, buf_offset, buf_len; -uint64_t buf_addr, buf_iova; +uint64_t buf_addr;uint32_t mbuf_avail, mbuf_offset;uint32_t hdr_remain = dev->vhost_hlen;uint32_t cpy_len;
@@ -1710,7 +1691,6 @@ desc_to_mbuf(struct virtio_net *dev, struct vhost_virtqueue *vq, } buf_addr = buf_vec[vec_idx].buf_addr; - buf_iova = buf_vec[vec_idx].buf_iova; buf_len = buf_vec[vec_idx].buf_len; buf_offset = hdr_remain; buf_avail = buf_vec[vec_idx].buf_len - hdr_remain;
@@ -1725,14 +1705,17 @@ desc_to_mbuf(struct virtio_net *dev, struct vhost_virtqueue *vq,while (1 ) { cpy_len =RTE_MIN (buf_avail, mbuf_avail); -if (likely (hdr && cur == m)) { +if (likely ((hdr && cur == m) || !vhost_can_batch_copy (vq, cpy_len))) { rte_memcpy(rte_pktmbuf_mtod_offset(cur, void *, mbuf_offset), (void *)((uintptr_t)(buf_addr + buf_offset)), cpy_len); }else { -sync_fill_seg (dev, vq, cur, mbuf_offset, - buf_addr + buf_offset, - buf_iova + buf_offset, cpy_len, false); + vq->batch_copy_elems[vq->batch_copy_nb_elems].dst = + rte_pktmbuf_mtod_offset(cur, void *, mbuf_offset); + vq->batch_copy_elems[vq->batch_copy_nb_elems].src = +RTE_PTR_ADD (buf_addr, buf_offset); + vq->batch_copy_elems[vq->batch_copy_nb_elems].len = cpy_len; + vq->batch_copy_nb_elems++; } mbuf_avail -= cpy_len;
@@ -1746,7 +1729,6 @@ desc_to_mbuf(struct virtio_net *dev, struct vhost_virtqueue *vq,break ; buf_addr = buf_vec[vec_idx].buf_addr; - buf_iova = buf_vec[vec_idx].buf_iova; buf_len = buf_vec[vec_idx].buf_len; buf_offset =0 ;
Signature · 1 line
2.54.0
v2 cover Drop vhost async (DMA-accelerated) datapath
In reply to David MarchandThe integration of this feature in OVS has been postponed following feedback how the current API forced the complexity on the application side. There is no contributor working on making those changes.
The feature has no other identified opensource consumer.
On the other hand, this feature has been left unattended for some years now while the "normal" datapath is actively maintained.
It is highly likely the async datapath code, which never underwent a stabilisation phase, is riddled with bugs.
Given the craziness around finding security bugs with AI, and since the vhost library is a good target, I propose to drop this feature and reduce the library attack surface.
There is one complication with the vhost_crypto and some external applications that seemed to rely on the async flag to track and populate guest pages, so a new flag replacing async is added by this series.
Signature · 41 lines
David Marchand Changes since v1: - removed unneeded dependencies on the PCI library, Changes since RFC: - maintained guest pages population in a new flag (vhost_crypto/SPDK), - removed remaining helpers which had sync_ in their names, David Marchand (6): vhost: decouple guest pages population from async datapath examples/vhost: stop using async datapath examples/vdpa: remove dependency on PCI library vhost: drop async datapath vhost: rename packed layout helpers for batches vhost: simplify some descriptor handling doc/guides/prog_guide/vhost_lib.rst | 152 +- doc/guides/rel_notes/release_26_11.rst | 20 + examples/vdpa/main.c | 1 - examples/vhost/Makefile | 2 - examples/vhost/main.c | 436 +-- examples/vhost/main.h | 21 +- examples/vhost/meson.build | 2 - examples/vhost_crypto/Makefile | 1 - examples/vhost_crypto/main.c | 2 +- examples/vhost_crypto/meson.build | 1 - lib/vhost/meson.build | 3 +- lib/vhost/rte_vhost.h | 2 +- lib/vhost/rte_vhost_async.h | 293 --- lib/vhost/socket.c | 20 +- lib/vhost/vhost.c | 434 --- lib/vhost/vhost.h | 116 +- lib/vhost/vhost_user.c | 115 +- lib/vhost/virtio_net.c | 3343 +++++------------------- 18 files changed, 736 insertions(+), 4228 deletions(-) delete mode 100644 lib/vhost/rte_vhost_async.h -- 2.54.0
PATCH v2 1/6 vhost: decouple guest pages population from async datapath
The vhost crypto zero-copy feature relies on the RTE_VHOST_USER_ASYNC_COPY flag to enable guest page tracking (guest_pages array population and MAP_POPULATE during mmap).
Introduce a RTE_VHOST_USER_MAP_POPULATE flag that maps to RTE_VHOST_USER_ASYNC_COPY at this point in time.
Adjust vhost_crypto example and remove the unnecessary ALLOW_EXPERIMENTAL_API flag as this example does not use any experimental APIs.
--- doc/guides/prog_guide/vhost_lib.rst | 9 +++++++++ examples/vhost_crypto/Makefile | 1 - examples/vhost_crypto/main.c | 2 +- examples/vhost_crypto/meson.build | 1 - lib/vhost/rte_vhost.h | 1 + lib/vhost/socket.c | 9 +++++++++ lib/vhost/vhost.h | 1 + lib/vhost/vhost_user.c | 4 ++-- 8 files changed, 23 insertions(+), 5 deletions(-)
doc/guides/prog_guide/vhost_lib.rst +9 -0
@@ -118,6 +118,15 @@ The following is an overview of some key Vhost API functions: It is disabled by default. + - ``RTE_VHOST_USER_MAP_POPULATE`` + + Guest memory regions will be mapped with ``MAP_POPULATE`` when this flag + is set, pre-faulting pages into memory. This is useful for applications + requiring direct access to guest memory, such as vhost-crypto zero-copy + operations. + + It is disabled by default. + - ``RTE_VHOST_USER_NET_COMPLIANT_OL_FLAGS`` Since v16.04, the vhost library forwards checksum and gso requests for
examples/vhost_crypto/Makefile +0 -1
@@ -6,7 +6,6 @@ APP = vhost-crypto # all source are stored in SRCS-y SRCS-y := main.c -CFLAGS += -DALLOW_EXPERIMENTAL_API PKGCONF ?= pkg-config
examples/vhost_crypto/main.c +1 -1
@@ -623,7 +623,7 @@ main(int argc, char *argv[])for (j =0 ; j < lo->nb_sockets; j++) { ret =rte_vhost_driver_register (lo->socket_files[j], -RTE_VHOST_USER_ASYNC_COPY ); +RTE_VHOST_USER_MAP_POPULATE );if (ret <0 ) {RTE_LOG (ERR ,USER1 ,"socket %s already exists\n" , lo->socket_files[j]);
examples/vhost_crypto/meson.build +0 -1
@@ -6,7 +6,6 @@ # To build this example as a standalone application with an already-installed # DPDK instance, use 'make' -allow_experimental_apis = true deps += ['vhost', 'cryptodev'] sources = files( 'main.c',
lib/vhost/rte_vhost.h +1 -0
@@ -39,6 +39,7 @@ extern "C" {/* support only linear buffers (no chained mbufs) */ #define RTE_VHOST_USER_LINEARBUF_SUPPORT (1ULL << 6)#define RTE_VHOST_USER_ASYNC_COPY (1ULL << 7) +#define RTE_VHOST_USER_MAP_POPULATE RTE_VHOST_USER_ASYNC_COPY#define RTE_VHOST_USER_NET_COMPLIANT_OL_FLAGS (1ULL << 8)#define RTE_VHOST_USER_NET_STATS_ENABLE (1ULL << 9)#define RTE_VHOST_USER_ASYNC_CONNECT (1ULL << 10)
lib/vhost/socket.c +9 -0
@@ -44,6 +44,7 @@ struct vhost_user_socket {bool extbuf;bool linearbuf;bool async_copy; +bool map_populate;bool net_compliant_ol_flags;bool stats_enabled;bool async_connect;
@@ -248,6 +249,13 @@ vhost_user_add_connection(int fd, struct vhost_user_socket *vsocket) dev->async_copy =1 ; } +if (vsocket->map_populate) { + dev =get_device (vid); + +if (dev) + dev->map_populate =1 ; + } +VHOST_CONFIG_LOG (vsocket->path,INFO ,"new device, handle is %d" , vid);if (vsocket->notify_ops->new_connection) {
@@ -939,6 +947,7 @@ rte_vhost_driver_register(const char *path, uint64_t flags) vsocket->extbuf = flags &RTE_VHOST_USER_EXTBUF_SUPPORT ; vsocket->linearbuf = flags &RTE_VHOST_USER_LINEARBUF_SUPPORT ; vsocket->async_copy = flags &RTE_VHOST_USER_ASYNC_COPY ; + vsocket->map_populate = flags &RTE_VHOST_USER_MAP_POPULATE ; vsocket->net_compliant_ol_flags = flags &RTE_VHOST_USER_NET_COMPLIANT_OL_FLAGS ; vsocket->stats_enabled = flags &RTE_VHOST_USER_NET_STATS_ENABLE ; vsocket->async_connect = flags &RTE_VHOST_USER_ASYNC_CONNECT ;
lib/vhost/vhost.h +1 -0
@@ -499,6 +499,7 @@ struct __rte_cache_aligned virtio_net {RTE_ATOMIC (int16_t ) broadcast_rarp;uint32_t nr_vring;int async_copy; +int map_populate;int extbuf;int linearbuf;
lib/vhost/vhost_user.c +2 -2
@@ -1384,7 +1384,7 @@ vhost_user_mmap_region(struct virtio_net *dev,return -1 ; } - populate = dev->async_copy ?MAP_POPULATE :0 ; + populate = dev->map_populate ?MAP_POPULATE :0 ; mmap_addr =mmap (NULL , mmap_size,PROT_READ |PROT_WRITE ,MAP_SHARED | populate, region->fd,0 );
@@ -1398,7 +1398,7 @@ vhost_user_mmap_region(struct virtio_net *dev, region->host_user_addr = (uint64_t )(uintptr_t )mmap_addr + mmap_offset;mem_set_dump (dev, mmap_addr, mmap_size, false, alignment); -if (dev->async_copy) { +if (dev->map_populate) {if (add_guest_pages (dev, region, alignment) <0 ) {VHOST_CONFIG_LOG (dev->ifname,ERR ,"adding guest pages to region failed." );
Signature · 1 line
2.54.0
Re: [PATCH v2 1/6] vhost: decouple guest pages population from async datapath
Quoted 12 lines · David Marchand
The vhost crypto zero-copy feature relies on the
RTE_VHOST_USER_ASYNC_COPYflag to enable guest page tracking (guest_pagesarray population andMAP_POPULATEduring mmap).Introduce a
RTE_VHOST_USER_MAP_POPULATEflag that maps toRTE_VHOST_USER_ASYNC_COPYat this point in time.Adjust
vhost_cryptoexample and remove the unnecessaryALLOW_EXPERIMENTAL_APIflag as this example does not use any experimental APIs.Signed-off-by David Marchand <[email protected]>
Quoted 11 lines
--- doc/guides/prog_guide/vhost_lib.rst | 9 +++++++++ examples/vhost_crypto/Makefile | 1 - examples/vhost_crypto/main.c | 2 +- examples/vhost_crypto/meson.build | 1 - lib/vhost/rte_vhost.h | 1 + lib/vhost/socket.c | 9 +++++++++ lib/vhost/vhost.h | 1 + lib/vhost/vhost_user.c | 4 ++-- 8 files changed, 23 insertions(+), 5 deletions(-)
Sounds reasonable to keep the same value as RTE_VHOST_USER_ASYNC_COPY to not break compatibility for existing Vhsot-crypto applications.
Thanks,
Maxime
PATCH v2 2/6 examples/vhost: stop using async datapath
In reply to David MarchandStop using vhost async as this feature will be removed in a next commit. Thanks to this, we can drop use of experimental api.
---
Changes since RFC:
- dropped vhost_crypto changes (now handled in patch 1),
--- examples/vhost/Makefile | 2 - examples/vhost/main.c | 436 +------------------------------------ examples/vhost/main.h | 21 +- examples/vhost/meson.build | 2 - 4 files changed, 12 insertions(+), 449 deletions(-)
examples/vhost/Makefile +0 -2
@@ -28,8 +28,6 @@ CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk) LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk) -CFLAGS += -DALLOW_EXPERIMENTAL_API - build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build $(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
examples/vhost/main.c +10 -426
603 lines not on this page · raw message
examples/vhost/main.h +2 -19
@@ -8,7 +8,6 @@#include <sys/queue.h> #include <rte_ether.h> -#include <rte_pci.h> /* Macros for printing using RTE_LOG */ #define RTE_LOGTYPE_VHOST_CONFIG RTE_LOGTYPE_USER1
@@ -92,17 +91,6 @@ struct lcore_info {struct vhost_dev_tailq_list vdev_list; }; -struct dma_info { -struct rte_pci_addr addr; -int16_t dev_id; -bool async_enabled; -}; - -struct dma_for_vhost { -struct dma_info dmas[RTE_MAX_QUEUES_PER_PORT *2 ]; -uint32_t async_flag; -}; -/* we implement non-extra virtio net features */ #define VIRTIO_NET_FEATURES 0
@@ -116,14 +104,9 @@ uint16_t builtin_enqueue_pkts(struct vhost_dev *dev, uint16_t queue_id,uint16_t builtin_dequeue_pkts (struct vhost_dev *dev,uint16_t queue_id,struct rte_mempool *mbuf_pool,struct rte_mbuf **pkts,uint16_t count); -uint16_t sync_enqueue_pkts (struct vhost_dev *dev,uint16_t queue_id, -struct rte_mbuf **pkts,uint32_t count); -uint16_t sync_dequeue_pkts (struct vhost_dev *dev,uint16_t queue_id, -struct rte_mempool *mbuf_pool, -struct rte_mbuf **pkts,uint16_t count); -uint16_t async_enqueue_pkts (struct vhost_dev *dev,uint16_t queue_id, +uint16_t enqueue_pkts (struct vhost_dev *dev,uint16_t queue_id,struct rte_mbuf **pkts,uint32_t count); -uint16_t async_dequeue_pkts (struct vhost_dev *dev,uint16_t queue_id, +uint16_t dequeue_pkts (struct vhost_dev *dev,uint16_t queue_id,struct rte_mempool *mbuf_pool,struct rte_mbuf **pkts,uint16_t count);#endif /* _MAIN_H_ */
examples/vhost/meson.build +0 -2
@@ -12,8 +12,6 @@ if not is_linux endif deps += 'vhost' -deps += 'dmadev' -allow_experimental_apis = true sources = files( 'main.c', 'virtio_net.c',
Signature · 1 line
2.54.0
Re: [PATCH v2 2/6] examples/vhost: stop using async datapath
Quoted 8 lines · David Marchand
Stop using vhost async as this feature will be removed in a next commit. Thanks to this, we can drop use of experimental api.
Signed-off-by David Marchand <[email protected]>---
Changes since RFC:
- droppedvhost_cryptochanges (now handled in patch 1),
Quoted 7 lines
--- examples/vhost/Makefile | 2 - examples/vhost/main.c | 436 +------------------------------------ examples/vhost/main.h | 21 +- examples/vhost/meson.build | 2 - 4 files changed, 12 insertions(+), 449 deletions(-)
Thanks,
Maxime
PATCH v2 3/6 examples/vdpa: remove dependency on PCI library
In reply to David MarchandThis is some forgotten unneeded header include from a previous refactoring.
--- examples/vdpa/main.c | 1 - 1 file changed, 1 deletion(-)
examples/vdpa/main.c +0 -1
@@ -13,7 +13,6 @@#include <rte_malloc.h> #include <rte_vhost.h> #include <rte_vdpa.h> -#include <rte_pci.h> #include <rte_string_fns.h> #include <cmdline_socket.h>
Signature · 1 line
2.54.0
Re: [PATCH v2 3/6] examples/vdpa: remove dependency on PCI library
Quoted 15 lines · David Marchand
This is some forgotten unneeded header include from a previous refactoring.
Fixes 38f8ab0bbc8d ("vhost: make vDPA framework bus agnostic")Signed-off-by David Marchand <[email protected]>--- examples/vdpa/main.c | 1 - 1 file changed, 1 deletion(-)examples/vdpa/main.c +0
-0
examples/vdpa/main.c line 18
Quoted 7 lines
examples/vdpa/main.c +0
-1@@ -13,7 +13,6 @@#include <rte_malloc.h> #include <rte_vhost.h> #include <rte_vdpa.h> -#include <rte_pci.h> #include <rte_string_fns.h> #include <cmdline_socket.h>
Thanks,
Maxime
PATCH v2 4/6 vhost: drop async datapath
In reply to David MarchandAll the API related to this feature is experimental, allowing us by the project policy to drop the feature in one go.
---
Changes since v1:
- dropped dependency on PCI library,
Changes since RFC:
- kept mmap populate behavior for vhost_crypto,
--- doc/guides/prog_guide/vhost_lib.rst | 151 +- doc/guides/rel_notes/release_26_11.rst | 20 + lib/vhost/meson.build | 3 +- lib/vhost/rte_vhost.h | 3 +- lib/vhost/rte_vhost_async.h | 293 --- lib/vhost/socket.c | 21 - lib/vhost/vhost.c | 434 ---- lib/vhost/vhost.h | 115 - lib/vhost/vhost_user.c | 111 - lib/vhost/virtio_net.c | 3275 +++++------------------- 10 files changed, 687 insertions(+), 3739 deletions(-) delete mode 100644 lib/vhost/rte_vhost_async.h
doc/guides/prog_guide/vhost_lib.rst +5 -146
@@ -103,21 +103,6 @@ The following is an overview of some key Vhost API functions: It is disabled by default. - - ``RTE_VHOST_USER_ASYNC_COPY`` - - Asynchronous data path will be enabled when this flag is set. Async - data path allows applications to enable DMA acceleration for vhost - queues. Vhost leverages the registered DMA channels to free CPU from - memory copy operations in data path. A set of async data path APIs are - defined for DPDK applications to make use of the async capability. Only - packets enqueued/dequeued by async APIs are processed through the async - data path. - - Currently this feature is only implemented on split ring enqueue data - path. - - It is disabled by default. - - ``RTE_VHOST_USER_MAP_POPULATE`` Guest memory regions will be mapped with ``MAP_POPULATE`` when this flag
@@ -236,76 +221,6 @@ The following is an overview of some key Vhost API functions: Enable or disable zero copy feature of the vhost crypto backend. -* ``rte_vhost_async_dma_configure(dma_id, vchan_id)`` - - Tell vhost which DMA vChannel is going to use. This function needs to - be called before register async data-path for vring. - -* ``rte_vhost_async_channel_register(vid, queue_id)`` - - Register async DMA acceleration for a vhost queue after vring is enabled. - -* ``rte_vhost_async_channel_register_thread_unsafe(vid, queue_id)`` - - Register async DMA acceleration for a vhost queue without performing - any locking. - - This function is only safe to call in vhost callback functions - (i.e., struct rte_vhost_device_ops). - -* ``rte_vhost_async_channel_unregister(vid, queue_id)`` - - Unregister the async DMA acceleration from a vhost queue. - Unregistration will fail, if the vhost queue has in-flight - packets that are not completed. - - Unregister async DMA acceleration in vring_state_changed() may - fail, as this API tries to acquire the spinlock of vhost - queue. The recommended way is to unregister async copy - devices for all vhost queues in destroy_device(), when a - virtio device is paused or shut down. - -* ``rte_vhost_async_channel_unregister_thread_unsafe(vid, queue_id)`` - - Unregister async DMA acceleration for a vhost queue without performing - any locking. - - This function is only safe to call in vhost callback functions - (i.e., struct rte_vhost_device_ops). - -* ``rte_vhost_submit_enqueue_burst(vid, queue_id, pkts, count, dma_id, vchan_id)`` - - Submit an enqueue request to transmit ``count`` packets from host to guest - by async data path. Applications must not free the packets submitted for - enqueue until the packets are completed. - -* ``rte_vhost_poll_enqueue_completed(vid, queue_id, pkts, count, dma_id, vchan_id)`` - - Poll enqueue completion status from async data path. Completed packets - are returned to applications through ``pkts``. - -* ``rte_vhost_async_get_inflight(vid, queue_id)`` - - This function returns the amount of in-flight packets for the vhost - queue using async acceleration. - - * ``rte_vhost_async_get_inflight_thread_unsafe(vid, queue_id)`` - - Get the number of inflight packets for a vhost queue without performing - any locking. It should only be used within the vhost ops, which already - holds the lock. - -* ``rte_vhost_clear_queue_thread_unsafe(vid, queue_id, **pkts, count, dma_id, vchan_id)`` - - Clear in-flight packets which are submitted to async channel in vhost - async data path without performing locking on virtqueue. Completed - packets are returned to applications through ``pkts``. - -* ``rte_vhost_clear_queue(vid, queue_id, **pkts, count, dma_id, vchan_id)`` - - Clear in-flight packets which are submitted to async channel in vhost async data - path. Completed packets are returned to applications through ``pkts``. - * ``rte_vhost_vring_call_nonblock(int vid, uint16_t vring_idx)`` Notify the guest that used descriptors have been added to the vring. This function
@@ -327,22 +242,11 @@ The following is an overview of some key Vhost API functions: This function resets the queue statistics. It requires statistics collection to be enabled at registration time. -* ``rte_vhost_async_try_dequeue_burst(vid, queue_id, mbuf_pool, pkts, count, - nr_inflight, dma_id, vchan_id)`` - - Receive ``count`` packets from guest to host in async data path, - and store them at ``pkts``. - * ``rte_vhost_driver_get_vdpa_dev_type(path, type)`` Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET, VDPA_DEVICE_TYPE_BLK. -* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)`` - - Clean up a DMA vChannel after use is finished. After this function is called, - the specified DMA vChannel should no longer be used by the Vhost library. - * ``rte_vhost_notify_guest(int vid, uint16_t queue_id)`` Inject the offloaded interrupt received by the 'guest_notify' callback,
@@ -411,16 +315,11 @@ Guest memory requirement * Memory pre-allocation - For non-async data path guest memory pre-allocation is not a - must but can help save memory. To do this we can add option - ``-mem-prealloc`` when starting QEMU, or we can lock all memory at vhost - side which will force memory to be allocated when it calls mmap - (option --mlockall in ovs-dpdk is an example in hand). - - - For async data path, we force the VM memory to be pre-allocated at vhost - lib when mapping the guest memory; and also we need to lock the memory to - prevent pages being swapped out to disk. + Guest memory pre-allocation is not a must but can help save memory. + To do this we can add option ``-mem-prealloc`` when starting QEMU, + or we can lock all memory at vhost side which will force memory to + be allocated when it calls mmap (option --mlockall in ovs-dpdk is + an example in hand). * Memory sharing
@@ -491,43 +390,3 @@ Finally, a set of device ops is defined for device specific operations: * ``get_notify_area`` Called to get the notify area info of the queue. - -Vhost asynchronous data path ----------------------------- - -Vhost asynchronous data path leverages DMA devices to offload memory -copies from the CPU and it is implemented in an asynchronous way. It -enables applications, like OVS, to save CPU cycles and hide memory copy -overhead, thus achieving higher throughput. - -Vhost doesn't manage DMA devices and applications, like OVS, need to -manage and configure DMA devices. Applications need to tell vhost what -DMA devices to use in every data path function call. This design enables -the flexibility for applications to dynamically use DMA channels in -different function modules, not limited in vhost. - -In addition, vhost supports M:N mapping between vrings and DMA virtual -channels. Specifically, one vring can use multiple different DMA channels -and one DMA channel can be shared by multiple vrings at the same time. -The reason of enabling one vring to use multiple DMA channels is that -it's possible that more than one dataplane threads enqueue packets to -the same vring with their own DMA virtual channels. Besides, the number -of DMA devices is limited. For the purpose of scaling, it's necessary to -support sharing DMA channels among vrings. - -* Async enqueue API usage - - In async enqueue path, rte_vhost_poll_enqueue_completed() needs to be - called in time to notify the guest of DMA copy completed packets. - Moreover, calling rte_vhost_submit_enqueue_burst() all the time but - not poll completed will cause the DMA ring to be full, which will - result in packet loss eventually. - -* Recommended IOVA mode in async datapath - - When DMA devices are bound to VFIO driver, VA mode is recommended. - For PA mode, page by page mapping may exceed IOMMU's max capability, - better to use 1G guest hugepage. - - For UIO driver or kernel driver, any VFIO related error messages - can be ignored.
doc/guides/rel_notes/release_26_11.rst +20 -0
@@ -79,6 +79,26 @@ Removed Items ``rte_rib6_is_equal`` * table: ``RTE_LPM_IPV6_ADDR_SIZE`` +* **Removed vhost async (DMA-accelerated) data path.** + + The experimental async data path APIs have been removed from the vhost library. + The following functions and flag are no longer available: + + - ``rte_vhost_async_channel_register`` + - ``rte_vhost_async_channel_register_thread_unsafe`` + - ``rte_vhost_async_channel_unregister`` + - ``rte_vhost_async_channel_unregister_thread_unsafe`` + - ``rte_vhost_async_dma_configure`` + - ``rte_vhost_async_dma_unconfigure`` + - ``rte_vhost_submit_enqueue_burst`` + - ``rte_vhost_poll_enqueue_completed`` + - ``rte_vhost_async_try_dequeue_burst`` + - ``rte_vhost_async_get_inflight`` + - ``rte_vhost_async_get_inflight_thread_unsafe`` + - ``rte_vhost_clear_queue`` + - ``rte_vhost_clear_queue_thread_unsafe`` + - ``RTE_VHOST_USER_ASYNC_COPY`` + API Changes -----------
lib/vhost/meson.build +1 -2
@@ -38,10 +38,9 @@ sources = files( headers = files( 'rte_vdpa.h', 'rte_vhost.h', - 'rte_vhost_async.h', 'rte_vhost_crypto.h', ) driver_sdk_headers = files( 'vdpa_driver.h', ) -deps += ['ethdev', 'cryptodev', 'hash', 'pci', 'dmadev'] +deps += ['ethdev', 'cryptodev', 'hash']
lib/vhost/rte_vhost.h +1 -2
@@ -38,8 +38,7 @@ extern "C" {#define RTE_VHOST_USER_EXTBUF_SUPPORT (1ULL << 5)/* support only linear buffers (no chained mbufs) */ #define RTE_VHOST_USER_LINEARBUF_SUPPORT (1ULL << 6) -#define RTE_VHOST_USER_ASYNC_COPY (1ULL << 7) -#define RTE_VHOST_USER_MAP_POPULATE RTE_VHOST_USER_ASYNC_COPY +#define RTE_VHOST_USER_MAP_POPULATE (1ULL << 7)#define RTE_VHOST_USER_NET_COMPLIANT_OL_FLAGS (1ULL << 8)#define RTE_VHOST_USER_NET_STATS_ENABLE (1ULL << 9)#define RTE_VHOST_USER_ASYNC_CONNECT (1ULL << 10)
lib/vhost/rte_vhost_async.h deleted +0 -293
293 lines not on this page · raw message
lib/vhost/socket.c +0 -21
@@ -43,7 +43,6 @@ struct vhost_user_socket {bool use_builtin_virtio_net;bool extbuf;bool linearbuf; -bool async_copy;bool map_populate;bool net_compliant_ol_flags;bool stats_enabled;
@@ -242,13 +241,6 @@ vhost_user_add_connection(int fd, struct vhost_user_socket *vsocket)if (vsocket->linearbuf)vhost_enable_linearbuf (vid); -if (vsocket->async_copy) { - dev =get_device (vid); - -if (dev) - dev->async_copy =1 ; - } -if (vsocket->map_populate) { dev =get_device (vid);
@@ -946,7 +938,6 @@ rte_vhost_driver_register(const char *path, uint64_t flags) vsocket->max_queue_pairs =VHOST_MAX_QUEUE_PAIRS ; vsocket->extbuf = flags &RTE_VHOST_USER_EXTBUF_SUPPORT ; vsocket->linearbuf = flags &RTE_VHOST_USER_LINEARBUF_SUPPORT ; - vsocket->async_copy = flags &RTE_VHOST_USER_ASYNC_COPY ; vsocket->map_populate = flags &RTE_VHOST_USER_MAP_POPULATE ; vsocket->net_compliant_ol_flags = flags &RTE_VHOST_USER_NET_COMPLIANT_OL_FLAGS ; vsocket->stats_enabled = flags &RTE_VHOST_USER_NET_STATS_ENABLE ;
@@ -956,12 +947,6 @@ rte_vhost_driver_register(const char *path, uint64_t flags)else vsocket->iommu_support = flags &RTE_VHOST_USER_IOMMU_SUPPORT ; -if (vsocket->async_copy && (vsocket->iommu_support || - (flags &RTE_VHOST_USER_POSTCOPY_SUPPORT ))) { -VHOST_CONFIG_LOG (path,ERR ,"async copy with IOMMU or post-copy not supported" ); - goto out_mutex; - } -/* * Set the supported features correctly for the builtin vhost-user * net driver.
@@ -984,12 +969,6 @@ rte_vhost_driver_register(const char *path, uint64_t flags) vsocket->protocol_features =VHOST_USER_PROTOCOL_FEATURES ; } -if (vsocket->async_copy) { - vsocket->supported_features &= ~(1ULL <<VHOST_F_LOG_ALL ); - vsocket->features &= ~(1ULL <<VHOST_F_LOG_ALL ); -VHOST_CONFIG_LOG (path,INFO ,"logging feature is disabled in async copy mode" ); - } -/* * We'll not be able to receive a buffer from guest in linear mode * without external buffer if it will not fit in a single mbuf, which is
lib/vhost/vhost.c +0 -434
463 lines not on this page · raw message
lib/vhost/vhost.h +0 -115
150 lines not on this page · raw message
lib/vhost/vhost_user.c +0 -111
177 lines not on this page · raw message
lib/vhost/virtio_net.c +660 -2615
1,637 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.54.0
Part of the diff is not shown here. View the raw message.
PATCH v2 5/6 vhost: rename packed layout helpers for batches
In reply to David MarchandNow that the async datapath has been removed, the _sync in function names gives no useful information.
Besides, vhost_reserve_avail_batch_packed does not reserve anything and its name is confusing, rename it following the other rx/tx split/packed helpers convention.
--- lib/vhost/virtio_net.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-)
lib/vhost/virtio_net.c +6 -8
@@ -1111,7 +1111,7 @@ virtio_dev_rx_split(struct virtio_net *dev, struct vhost_virtqueue *vq, } static __rte_always_inline int -virtio_dev_rx_sync_batch_check(struct virtio_net *dev, +virtio_dev_rx_batch_check(struct virtio_net *dev, struct vhost_virtqueue *vq, struct rte_mbuf **pkts, uint64_t *desc_addrs,
@@ -1213,7 +1213,7 @@ virtio_dev_rx_batch_packed_copy(struct virtio_net *dev, } static __rte_always_inline int -virtio_dev_rx_sync_batch_packed(struct virtio_net *dev, +virtio_dev_rx_batch_packed(struct virtio_net *dev, struct vhost_virtqueue *vq, struct rte_mbuf **pkts) __rte_requires_shared_capability(&vq->iotlb_lock)
@@ -1221,7 +1221,7 @@ virtio_dev_rx_sync_batch_packed(struct virtio_net *dev,uint64_t desc_addrs[PACKED_BATCH_SIZE ];uint64_t lens[PACKED_BATCH_SIZE ]; -if (virtio_dev_rx_sync_batch_check (dev, vq, pkts, desc_addrs, lens) ==-1 ) +if (virtio_dev_rx_batch_check (dev, vq, pkts, desc_addrs, lens) ==-1 )return -1 ;if (vq->shadow_used_idx) {
@@ -1273,8 +1273,7 @@ virtio_dev_rx_packed(struct virtio_net *dev,rte_prefetch0 (&vq->desc_packed[vq->last_avail_idx]);if (count - pkt_idx >=PACKED_BATCH_SIZE ) { -if (!virtio_dev_rx_sync_batch_packed (dev, vq, - &pkts[pkt_idx])) { +if (!virtio_dev_rx_batch_packed (dev, vq, &pkts[pkt_idx])) { pkt_idx +=PACKED_BATCH_SIZE ;continue ; }
@@ -1971,7 +1970,7 @@ virtio_dev_tx_split_compliant(struct virtio_net *dev, } static __rte_always_inline int -vhost_reserve_avail_batch_packed(struct virtio_net *dev, +virtio_dev_tx_batch_packed_check(struct virtio_net *dev, struct vhost_virtqueue *vq, struct rte_mbuf **pkts, uint16_t avail_idx,
@@ -2055,8 +2054,7 @@ virtio_dev_tx_batch_packed(struct virtio_net *dev,uint16_t ids[PACKED_BATCH_SIZE ];uint16_t i; -if (vhost_reserve_avail_batch_packed (dev, vq, pkts, avail_idx, - desc_addrs, ids)) +if (virtio_dev_tx_batch_packed_check (dev, vq, pkts, avail_idx, desc_addrs, ids))return -1 ;vhost_for_each_try_unroll (i,0 ,PACKED_BATCH_SIZE )
Signature · 1 line
2.54.0
PATCH v2 6/6 vhost: simplify some descriptor handling
In reply to David MarchandNow that the async datapath has been removed, move back the descriptors copy/filling code in desc_to_mbuf/mbuf_to_desc and eliminate an intermediate helper.
Tracking buf_iova is not needed in the descriptor to mbuf case.
--- lib/vhost/virtio_net.c | 78 ++++++++++++++++-------------------------- 1 file changed, 30 insertions(+), 48 deletions(-)
lib/vhost/virtio_net.c +30 -48
@@ -76,6 +76,12 @@ vhost_queue_stats_update(const struct virtio_net *dev, struct vhost_virtqueue *v } } +static __rte_always_inline bool +vhost_can_batch_copy (struct vhost_virtqueue *vq,uint32_t cpy_len) +{ +return cpy_len <=MAX_BATCH_LEN && vq->batch_copy_nb_elems < vq->size; +} + static inline void do_data_copy_enqueue(struct virtio_net *dev, struct vhost_virtqueue *vq) __rte_requires_shared_capability(&vq->iotlb_lock)
@@ -843,44 +849,6 @@ copy_vnet_hdr_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, } } -static __rte_always_inline void -sync_fill_seg (struct virtio_net *dev,struct vhost_virtqueue *vq, -struct rte_mbuf *m,uint32_t mbuf_offset, -uint64_t buf_addr,uint64_t buf_iova,uint32_t cpy_len,bool to_desc) - __rte_requires_shared_capability(&vq->iotlb_lock) -{ -struct batch_copy_elem *batch_copy = vq->batch_copy_elems; - -if (likely (cpy_len >MAX_BATCH_LEN || vq->batch_copy_nb_elems >= vq->size)) { -if (to_desc) { -rte_memcpy ((void *)((uintptr_t )(buf_addr)), - rte_pktmbuf_mtod_offset(m, void *, mbuf_offset), - cpy_len); -vhost_log_cache_write_iova (dev, vq, buf_iova, cpy_len); -PRINT_PACKET (dev, (uintptr_t )(buf_addr), cpy_len,0 ); - }else { - rte_memcpy(rte_pktmbuf_mtod_offset(m, void *, mbuf_offset), - (void *)((uintptr_t)(buf_addr)), - cpy_len); - } - }else { -if (to_desc) { - batch_copy[vq->batch_copy_nb_elems].dst = - (void *)((uintptr_t )(buf_addr)); - batch_copy[vq->batch_copy_nb_elems].src = - rte_pktmbuf_mtod_offset(m, void *, mbuf_offset); - batch_copy[vq->batch_copy_nb_elems].log_addr = buf_iova; - }else { - batch_copy[vq->batch_copy_nb_elems].dst = - rte_pktmbuf_mtod_offset(m, void *, mbuf_offset); - batch_copy[vq->batch_copy_nb_elems].src = - (void *)((uintptr_t )(buf_addr)); - } - batch_copy[vq->batch_copy_nb_elems].len = cpy_len; - vq->batch_copy_nb_elems++; - } -} - static __rte_always_inline int mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, struct rte_mbuf *m, struct buf_vector *buf_vec,
@@ -976,9 +944,22 @@ mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, cpy_len =RTE_MIN (buf_avail, mbuf_avail); -sync_fill_seg (dev, vq, m, mbuf_offset, - buf_addr + buf_offset, - buf_iova + buf_offset, cpy_len, true); +if (likely (!vhost_can_batch_copy (vq, cpy_len))) { +rte_memcpy (RTE_PTR_ADD (buf_addr, buf_offset), + rte_pktmbuf_mtod_offset(m, void *, mbuf_offset), + cpy_len); +vhost_log_cache_write_iova (dev, vq, buf_iova + buf_offset, cpy_len); +PRINT_PACKET (dev,RTE_PTR_ADD (buf_addr, buf_offset), cpy_len,0 ); + }else { + vq->batch_copy_elems[vq->batch_copy_nb_elems].dst = +RTE_PTR_ADD (buf_addr, buf_offset); + vq->batch_copy_elems[vq->batch_copy_nb_elems].src = + rte_pktmbuf_mtod_offset(m, void *, mbuf_offset); + vq->batch_copy_elems[vq->batch_copy_nb_elems].log_addr = + buf_iova + buf_offset; + vq->batch_copy_elems[vq->batch_copy_nb_elems].len = cpy_len; + vq->batch_copy_nb_elems++; + } mbuf_avail -= cpy_len; mbuf_offset += cpy_len;
@@ -1679,7 +1660,7 @@ desc_to_mbuf(struct virtio_net *dev, struct vhost_virtqueue *vq, __rte_requires_shared_capability(&vq->iotlb_lock) {uint32_t buf_avail, buf_offset, buf_len; -uint64_t buf_addr, buf_iova; +uint64_t buf_addr;uint32_t mbuf_avail, mbuf_offset;uint32_t hdr_remain = dev->vhost_hlen;uint32_t cpy_len;
@@ -1710,7 +1691,6 @@ desc_to_mbuf(struct virtio_net *dev, struct vhost_virtqueue *vq, } buf_addr = buf_vec[vec_idx].buf_addr; - buf_iova = buf_vec[vec_idx].buf_iova; buf_len = buf_vec[vec_idx].buf_len; buf_offset = hdr_remain; buf_avail = buf_vec[vec_idx].buf_len - hdr_remain;
@@ -1725,14 +1705,17 @@ desc_to_mbuf(struct virtio_net *dev, struct vhost_virtqueue *vq,while (1 ) { cpy_len =RTE_MIN (buf_avail, mbuf_avail); -if (likely (hdr && cur == m)) { +if (likely ((hdr && cur == m) || !vhost_can_batch_copy (vq, cpy_len))) { rte_memcpy(rte_pktmbuf_mtod_offset(cur, void *, mbuf_offset), (void *)((uintptr_t)(buf_addr + buf_offset)), cpy_len); }else { -sync_fill_seg (dev, vq, cur, mbuf_offset, - buf_addr + buf_offset, - buf_iova + buf_offset, cpy_len, false); + vq->batch_copy_elems[vq->batch_copy_nb_elems].dst = + rte_pktmbuf_mtod_offset(cur, void *, mbuf_offset); + vq->batch_copy_elems[vq->batch_copy_nb_elems].src = +RTE_PTR_ADD (buf_addr, buf_offset); + vq->batch_copy_elems[vq->batch_copy_nb_elems].len = cpy_len; + vq->batch_copy_nb_elems++; } mbuf_avail -= cpy_len;
@@ -1746,7 +1729,6 @@ desc_to_mbuf(struct virtio_net *dev, struct vhost_virtqueue *vq,break ; buf_addr = buf_vec[vec_idx].buf_addr; - buf_iova = buf_vec[vec_idx].buf_iova; buf_len = buf_vec[vec_idx].buf_len; buf_offset =0 ;
Signature · 1 line
2.54.0
v3 cover Drop vhost async (DMA-accelerated) datapath
In reply to David MarchandThe integration of this feature in OVS has been postponed following feedback how the current API forced the complexity on the application side. There is no contributor working on making those changes.
The feature has no other identified opensource consumer.
On the other hand, this feature has been left unattended for some years now while the "normal" datapath is actively maintained.
It is highly likely the async datapath code, which never underwent a stabilisation phase, is riddled with bugs.
Given the craziness around finding security bugs with AI, and since the vhost library is a good target, I propose to drop this feature and reduce the library attack surface.
There is one complication with the vhost_crypto and some external applications that seemed to rely on the async flag to track and populate guest pages, so a new flag replacing async is added by this series.
Signature · 47 lines
David Marchand Changes since v2: - updated release notes and vhost example documentation, - removed some async statistics missed in previous revisions, - re-added checks on map populate and post copy or other virtio features, Changes since v1: - removed unneeded dependencies on the PCI library, Changes since RFC: - maintained guest pages population in a new flag (vhost_crypto/SPDK), - removed remaining helpers which had sync_ in their names, David Marchand (6): vhost: decouple guest pages population from async datapath examples/vhost: stop using async datapath examples/vdpa: remove dependency on PCI library vhost: drop async datapath vhost: rename packed layout helpers for batches vhost: simplify some descriptor handling doc/guides/prog_guide/vhost_lib.rst | 152 +- doc/guides/rel_notes/release_26_11.rst | 24 + doc/guides/sample_app_ug/vhost.rst | 10 - examples/vdpa/main.c | 1 - examples/vhost/Makefile | 2 - examples/vhost/main.c | 437 +--- examples/vhost/main.h | 21 +- examples/vhost/meson.build | 2 - examples/vhost_crypto/Makefile | 1 - examples/vhost_crypto/main.c | 2 +- examples/vhost_crypto/meson.build | 1 - lib/vhost/meson.build | 3 +- lib/vhost/rte_vhost.h | 2 +- lib/vhost/rte_vhost_async.h | 293 --- lib/vhost/socket.c | 18 +- lib/vhost/vhost.c | 436 ---- lib/vhost/vhost.h | 118 +- lib/vhost/vhost_user.c | 115 +- lib/vhost/virtio_net.c | 3317 +++++------------------- 19 files changed, 729 insertions(+), 4226 deletions(-) delete mode 100644 lib/vhost/rte_vhost_async.h -- 2.54.0
PATCH v3 1/6 vhost: decouple guest pages population from async datapath
The vhost crypto zero-copy feature relies on the RTE_VHOST_USER_ASYNC_COPY flag to enable guest page tracking (guest_pages array population and MAP_POPULATE during mmap).
Introduce a RTE_VHOST_USER_MAP_POPULATE flag that maps to RTE_VHOST_USER_ASYNC_COPY at this point in time.
Adjust vhost_crypto example and remove the unnecessary ALLOW_EXPERIMENTAL_API flag as this example does not use any experimental APIs.
---
Changes since v2:
- updated release notes for new flag,
- fixed checks on map populate feature,
--- doc/guides/prog_guide/vhost_lib.rst | 9 +++++++++ doc/guides/rel_notes/release_26_11.rst | 4 ++++ examples/vhost_crypto/Makefile | 1 - examples/vhost_crypto/main.c | 2 +- examples/vhost_crypto/meson.build | 1 - lib/vhost/rte_vhost.h | 1 + lib/vhost/socket.c | 19 +++++++++++++++---- lib/vhost/vhost.h | 1 + lib/vhost/vhost_user.c | 4 ++-- 9 files changed, 33 insertions(+), 9 deletions(-)
doc/guides/prog_guide/vhost_lib.rst +9 -0
@@ -118,6 +118,15 @@ The following is an overview of some key Vhost API functions: It is disabled by default. + - ``RTE_VHOST_USER_MAP_POPULATE`` + + Guest memory regions will be mapped with ``MAP_POPULATE`` when this flag + is set, pre-faulting pages into memory. This is useful for applications + requiring direct access to guest memory, such as vhost-crypto zero-copy + operations. + + It is disabled by default. + - ``RTE_VHOST_USER_NET_COMPLIANT_OL_FLAGS`` Since v16.04, the vhost library forwards checksum and gso requests for
doc/guides/rel_notes/release_26_11.rst +4 -0
@@ -309,6 +309,10 @@ API Changes ``rte_event_port_profile_links_get`` and ``rte_event_port_profile_unlink`` +* vhost: Introduced ``RTE_VHOST_USER_MAP_POPULATE`` as a replacement of + ``RTE_VHOST_USER_ASYNC_COPY`` for users that did not rely on the vhost + async copy feature but expected guest pages to be prefaulted. + * reorder: Promoted the following API from experimental to stable: ``rte_reorder_seqn``, ``rte_reorder_drain_up_to_seqn``, ``rte_reorder_min_seqn_set`` and ``rte_reorder_memory_footprint_get``.
examples/vhost_crypto/Makefile +0 -1
@@ -6,7 +6,6 @@ APP = vhost-crypto # all source are stored in SRCS-y SRCS-y := main.c -CFLAGS += -DALLOW_EXPERIMENTAL_API PKGCONF ?= pkg-config
examples/vhost_crypto/main.c +1 -1
@@ -623,7 +623,7 @@ main(int argc, char *argv[])for (j =0 ; j < lo->nb_sockets; j++) { ret =rte_vhost_driver_register (lo->socket_files[j], -RTE_VHOST_USER_ASYNC_COPY ); +RTE_VHOST_USER_MAP_POPULATE );if (ret <0 ) {RTE_LOG (ERR ,USER1 ,"socket %s already exists\n" , lo->socket_files[j]);
examples/vhost_crypto/meson.build +0 -1
@@ -6,7 +6,6 @@ # To build this example as a standalone application with an already-installed # DPDK instance, use 'make' -allow_experimental_apis = true deps += ['vhost', 'cryptodev'] sources = files( 'main.c',
lib/vhost/rte_vhost.h +1 -0
@@ -39,6 +39,7 @@ extern "C" {/* support only linear buffers (no chained mbufs) */ #define RTE_VHOST_USER_LINEARBUF_SUPPORT (1ULL << 6)#define RTE_VHOST_USER_ASYNC_COPY (1ULL << 7) +#define RTE_VHOST_USER_MAP_POPULATE RTE_VHOST_USER_ASYNC_COPY#define RTE_VHOST_USER_NET_COMPLIANT_OL_FLAGS (1ULL << 8)#define RTE_VHOST_USER_NET_STATS_ENABLE (1ULL << 9)#define RTE_VHOST_USER_ASYNC_CONNECT (1ULL << 10)
lib/vhost/socket.c +15 -4
@@ -44,6 +44,7 @@ struct vhost_user_socket {bool extbuf;bool linearbuf;bool async_copy; +bool map_populate;bool net_compliant_ol_flags;bool stats_enabled;bool async_connect;
@@ -246,6 +247,13 @@ vhost_user_add_connection(int fd, struct vhost_user_socket *vsocket) dev->async_copy =1 ; } +if (vsocket->map_populate) { + dev =get_device (vid); + +if (dev) + dev->map_populate =1 ; + } +VHOST_CONFIG_LOG (vsocket->path,INFO ,"new device, handle is %d" , vid);if (vsocket->notify_ops->new_connection) {
@@ -937,6 +945,7 @@ rte_vhost_driver_register(const char *path, uint64_t flags) vsocket->extbuf = flags &RTE_VHOST_USER_EXTBUF_SUPPORT ; vsocket->linearbuf = flags &RTE_VHOST_USER_LINEARBUF_SUPPORT ; vsocket->async_copy = flags &RTE_VHOST_USER_ASYNC_COPY ; + vsocket->map_populate = flags &RTE_VHOST_USER_MAP_POPULATE ; vsocket->net_compliant_ol_flags = flags &RTE_VHOST_USER_NET_COMPLIANT_OL_FLAGS ; vsocket->stats_enabled = flags &RTE_VHOST_USER_NET_STATS_ENABLE ; vsocket->async_connect = flags &RTE_VHOST_USER_ASYNC_CONNECT ;
@@ -945,9 +954,10 @@ rte_vhost_driver_register(const char *path, uint64_t flags)else vsocket->iommu_support = flags &RTE_VHOST_USER_IOMMU_SUPPORT ; -if (vsocket->async_copy && (vsocket->iommu_support || +if (vsocket->map_populate && (vsocket->iommu_support || (flags &RTE_VHOST_USER_POSTCOPY_SUPPORT ))) { -VHOST_CONFIG_LOG (path,ERR ,"async copy with IOMMU or post-copy not supported" ); +VHOST_CONFIG_LOG (path,ERR , +"guest pages prefaulting not supported with IOMMU or post-copy" ); goto out_mutex; }
@@ -973,10 +983,11 @@ rte_vhost_driver_register(const char *path, uint64_t flags) vsocket->protocol_features =VHOST_USER_PROTOCOL_FEATURES ; } -if (vsocket->async_copy) { +if (vsocket->map_populate) { vsocket->supported_features &= ~(1ULL <<VHOST_F_LOG_ALL ); vsocket->features &= ~(1ULL <<VHOST_F_LOG_ALL ); -VHOST_CONFIG_LOG (path,INFO ,"logging feature is disabled in async copy mode" ); +VHOST_CONFIG_LOG (path,INFO , +"logging feature is disabled with guest pages prefaulting" ); }/*
lib/vhost/vhost.h +1 -0
@@ -499,6 +499,7 @@ struct __rte_cache_aligned virtio_net {RTE_ATOMIC (int16_t ) broadcast_rarp;uint32_t nr_vring;int async_copy; +int map_populate;int extbuf;int linearbuf;
lib/vhost/vhost_user.c +2 -2
@@ -1384,7 +1384,7 @@ vhost_user_mmap_region(struct virtio_net *dev,return -1 ; } - populate = dev->async_copy ?MAP_POPULATE :0 ; + populate = dev->map_populate ?MAP_POPULATE :0 ; mmap_addr =mmap (NULL , mmap_size,PROT_READ |PROT_WRITE ,MAP_SHARED | populate, region->fd,0 );
@@ -1398,7 +1398,7 @@ vhost_user_mmap_region(struct virtio_net *dev, region->host_user_addr = (uint64_t )(uintptr_t )mmap_addr + mmap_offset;mem_set_dump (dev, mmap_addr, mmap_size, false, alignment); -if (dev->async_copy) { +if (dev->map_populate) {if (add_guest_pages (dev, region, alignment) <0 ) {VHOST_CONFIG_LOG (dev->ifname,ERR ,"adding guest pages to region failed." );
Signature · 1 line
2.54.0
PATCH v3 2/6 examples/vhost: stop using async datapath
In reply to David MarchandStop using vhost async as this feature will be removed in a next commit. Thanks to this, we can drop use of experimental api.
---
Changes since v2:
- removed --dmas option in vhost example doc,
Changes since RFC:
- dropped vhost_crypto changes (now handled in patch 1),
--- doc/guides/sample_app_ug/vhost.rst | 10 - examples/vhost/Makefile | 2 - examples/vhost/main.c | 437 +---------------------------- examples/vhost/main.h | 21 +- examples/vhost/meson.build | 2 - 5 files changed, 12 insertions(+), 460 deletions(-)
doc/guides/sample_app_ug/vhost.rst +0 -10
@@ -149,16 +149,6 @@ default value is 15. A very simple vhost-user net driver which demonstrates how to use the generic vhost APIs will be used when this option is given. It is disabled by default. -**--dmas** -This parameter is used to specify the assigned DMA device of a vhost device. -Async vhost-user net driver will be used if --dmas is set. For example ---dmas [txd0@00:04.0,txd1@00:04.1,rxd0@00:04.2,rxd1@00:04.3] means use -DMA channel 00:04.0/00:04.2 for vhost device 0 enqueue/dequeue operation -and use DMA channel 00:04.1/00:04.3 for vhost device 1 enqueue/dequeue -operation. The index of the device corresponds to the socket file in order, -that means vhost device 0 is created through the first socket file, vhost -device 1 is created through the second socket file, and so on. - **--total-num-mbufs 0-N** This parameter sets the number of mbufs to be allocated in mbuf pools, the default value is 147456. This is can be used if launch of a port fails
examples/vhost/Makefile +0 -2
@@ -28,8 +28,6 @@ CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk) LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk) -CFLAGS += -DALLOW_EXPERIMENTAL_API - build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build $(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
examples/vhost/main.c +10 -427
604 lines not on this page · raw message
examples/vhost/main.h +2 -19
@@ -8,7 +8,6 @@#include <sys/queue.h> #include <rte_ether.h> -#include <rte_pci.h> /* Macros for printing using RTE_LOG */ #define RTE_LOGTYPE_VHOST_CONFIG RTE_LOGTYPE_USER1
@@ -92,17 +91,6 @@ struct lcore_info {struct vhost_dev_tailq_list vdev_list; }; -struct dma_info { -struct rte_pci_addr addr; -int16_t dev_id; -bool async_enabled; -}; - -struct dma_for_vhost { -struct dma_info dmas[RTE_MAX_QUEUES_PER_PORT *2 ]; -uint32_t async_flag; -}; -/* we implement non-extra virtio net features */ #define VIRTIO_NET_FEATURES 0
@@ -116,14 +104,9 @@ uint16_t builtin_enqueue_pkts(struct vhost_dev *dev, uint16_t queue_id,uint16_t builtin_dequeue_pkts (struct vhost_dev *dev,uint16_t queue_id,struct rte_mempool *mbuf_pool,struct rte_mbuf **pkts,uint16_t count); -uint16_t sync_enqueue_pkts (struct vhost_dev *dev,uint16_t queue_id, -struct rte_mbuf **pkts,uint32_t count); -uint16_t sync_dequeue_pkts (struct vhost_dev *dev,uint16_t queue_id, -struct rte_mempool *mbuf_pool, -struct rte_mbuf **pkts,uint16_t count); -uint16_t async_enqueue_pkts (struct vhost_dev *dev,uint16_t queue_id, +uint16_t enqueue_pkts (struct vhost_dev *dev,uint16_t queue_id,struct rte_mbuf **pkts,uint32_t count); -uint16_t async_dequeue_pkts (struct vhost_dev *dev,uint16_t queue_id, +uint16_t dequeue_pkts (struct vhost_dev *dev,uint16_t queue_id,struct rte_mempool *mbuf_pool,struct rte_mbuf **pkts,uint16_t count);#endif /* _MAIN_H_ */
examples/vhost/meson.build +0 -2
@@ -12,8 +12,6 @@ if not is_linux endif deps += 'vhost' -deps += 'dmadev' -allow_experimental_apis = true sources = files( 'main.c', 'virtio_net.c',
Signature · 1 line
2.54.0
PATCH v3 3/6 examples/vdpa: remove dependency on PCI library
In reply to David MarchandThis is some forgotten unneeded header include from a previous refactoring.
--- examples/vdpa/main.c | 1 - 1 file changed, 1 deletion(-)
examples/vdpa/main.c +0 -1
@@ -13,7 +13,6 @@#include <rte_malloc.h> #include <rte_vhost.h> #include <rte_vdpa.h> -#include <rte_pci.h> #include <rte_string_fns.h> #include <cmdline_socket.h>
Signature · 1 line
2.54.0
PATCH v3 4/6 vhost: drop async datapath
In reply to David MarchandAll the API related to this feature is experimental, allowing us by the project policy to drop the feature in one go.
---
Changes since v2:
- removed inflight async statistics,
Changes since v1:
- dropped dependency on PCI library,
Changes since RFC:
- kept mmap populate behavior for vhost_crypto,
--- doc/guides/prog_guide/vhost_lib.rst | 151 +- doc/guides/rel_notes/release_26_11.rst | 20 + lib/vhost/meson.build | 3 +- lib/vhost/rte_vhost.h | 3 +- lib/vhost/rte_vhost_async.h | 293 --- lib/vhost/socket.c | 9 - lib/vhost/vhost.c | 436 ---- lib/vhost/vhost.h | 117 - lib/vhost/vhost_user.c | 111 - lib/vhost/virtio_net.c | 3275 +++++------------------- 10 files changed, 687 insertions(+), 3731 deletions(-) delete mode 100644 lib/vhost/rte_vhost_async.h
doc/guides/prog_guide/vhost_lib.rst +5 -146
@@ -103,21 +103,6 @@ The following is an overview of some key Vhost API functions: It is disabled by default. - - ``RTE_VHOST_USER_ASYNC_COPY`` - - Asynchronous data path will be enabled when this flag is set. Async - data path allows applications to enable DMA acceleration for vhost - queues. Vhost leverages the registered DMA channels to free CPU from - memory copy operations in data path. A set of async data path APIs are - defined for DPDK applications to make use of the async capability. Only - packets enqueued/dequeued by async APIs are processed through the async - data path. - - Currently this feature is only implemented on split ring enqueue data - path. - - It is disabled by default. - - ``RTE_VHOST_USER_MAP_POPULATE`` Guest memory regions will be mapped with ``MAP_POPULATE`` when this flag
@@ -236,76 +221,6 @@ The following is an overview of some key Vhost API functions: Enable or disable zero copy feature of the vhost crypto backend. -* ``rte_vhost_async_dma_configure(dma_id, vchan_id)`` - - Tell vhost which DMA vChannel is going to use. This function needs to - be called before register async data-path for vring. - -* ``rte_vhost_async_channel_register(vid, queue_id)`` - - Register async DMA acceleration for a vhost queue after vring is enabled. - -* ``rte_vhost_async_channel_register_thread_unsafe(vid, queue_id)`` - - Register async DMA acceleration for a vhost queue without performing - any locking. - - This function is only safe to call in vhost callback functions - (i.e., struct rte_vhost_device_ops). - -* ``rte_vhost_async_channel_unregister(vid, queue_id)`` - - Unregister the async DMA acceleration from a vhost queue. - Unregistration will fail, if the vhost queue has in-flight - packets that are not completed. - - Unregister async DMA acceleration in vring_state_changed() may - fail, as this API tries to acquire the spinlock of vhost - queue. The recommended way is to unregister async copy - devices for all vhost queues in destroy_device(), when a - virtio device is paused or shut down. - -* ``rte_vhost_async_channel_unregister_thread_unsafe(vid, queue_id)`` - - Unregister async DMA acceleration for a vhost queue without performing - any locking. - - This function is only safe to call in vhost callback functions - (i.e., struct rte_vhost_device_ops). - -* ``rte_vhost_submit_enqueue_burst(vid, queue_id, pkts, count, dma_id, vchan_id)`` - - Submit an enqueue request to transmit ``count`` packets from host to guest - by async data path. Applications must not free the packets submitted for - enqueue until the packets are completed. - -* ``rte_vhost_poll_enqueue_completed(vid, queue_id, pkts, count, dma_id, vchan_id)`` - - Poll enqueue completion status from async data path. Completed packets - are returned to applications through ``pkts``. - -* ``rte_vhost_async_get_inflight(vid, queue_id)`` - - This function returns the amount of in-flight packets for the vhost - queue using async acceleration. - - * ``rte_vhost_async_get_inflight_thread_unsafe(vid, queue_id)`` - - Get the number of inflight packets for a vhost queue without performing - any locking. It should only be used within the vhost ops, which already - holds the lock. - -* ``rte_vhost_clear_queue_thread_unsafe(vid, queue_id, **pkts, count, dma_id, vchan_id)`` - - Clear in-flight packets which are submitted to async channel in vhost - async data path without performing locking on virtqueue. Completed - packets are returned to applications through ``pkts``. - -* ``rte_vhost_clear_queue(vid, queue_id, **pkts, count, dma_id, vchan_id)`` - - Clear in-flight packets which are submitted to async channel in vhost async data - path. Completed packets are returned to applications through ``pkts``. - * ``rte_vhost_vring_call_nonblock(int vid, uint16_t vring_idx)`` Notify the guest that used descriptors have been added to the vring. This function
@@ -327,22 +242,11 @@ The following is an overview of some key Vhost API functions: This function resets the queue statistics. It requires statistics collection to be enabled at registration time. -* ``rte_vhost_async_try_dequeue_burst(vid, queue_id, mbuf_pool, pkts, count, - nr_inflight, dma_id, vchan_id)`` - - Receive ``count`` packets from guest to host in async data path, - and store them at ``pkts``. - * ``rte_vhost_driver_get_vdpa_dev_type(path, type)`` Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET, VDPA_DEVICE_TYPE_BLK. -* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)`` - - Clean up a DMA vChannel after use is finished. After this function is called, - the specified DMA vChannel should no longer be used by the Vhost library. - * ``rte_vhost_notify_guest(int vid, uint16_t queue_id)`` Inject the offloaded interrupt received by the 'guest_notify' callback,
@@ -411,16 +315,11 @@ Guest memory requirement * Memory pre-allocation - For non-async data path guest memory pre-allocation is not a - must but can help save memory. To do this we can add option - ``-mem-prealloc`` when starting QEMU, or we can lock all memory at vhost - side which will force memory to be allocated when it calls mmap - (option --mlockall in ovs-dpdk is an example in hand). - - - For async data path, we force the VM memory to be pre-allocated at vhost - lib when mapping the guest memory; and also we need to lock the memory to - prevent pages being swapped out to disk. + Guest memory pre-allocation is not a must but can help save memory. + To do this we can add option ``-mem-prealloc`` when starting QEMU, + or we can lock all memory at vhost side which will force memory to + be allocated when it calls mmap (option --mlockall in ovs-dpdk is + an example in hand). * Memory sharing
@@ -491,43 +390,3 @@ Finally, a set of device ops is defined for device specific operations: * ``get_notify_area`` Called to get the notify area info of the queue. - -Vhost asynchronous data path ----------------------------- - -Vhost asynchronous data path leverages DMA devices to offload memory -copies from the CPU and it is implemented in an asynchronous way. It -enables applications, like OVS, to save CPU cycles and hide memory copy -overhead, thus achieving higher throughput. - -Vhost doesn't manage DMA devices and applications, like OVS, need to -manage and configure DMA devices. Applications need to tell vhost what -DMA devices to use in every data path function call. This design enables -the flexibility for applications to dynamically use DMA channels in -different function modules, not limited in vhost. - -In addition, vhost supports M:N mapping between vrings and DMA virtual -channels. Specifically, one vring can use multiple different DMA channels -and one DMA channel can be shared by multiple vrings at the same time. -The reason of enabling one vring to use multiple DMA channels is that -it's possible that more than one dataplane threads enqueue packets to -the same vring with their own DMA virtual channels. Besides, the number -of DMA devices is limited. For the purpose of scaling, it's necessary to -support sharing DMA channels among vrings. - -* Async enqueue API usage - - In async enqueue path, rte_vhost_poll_enqueue_completed() needs to be - called in time to notify the guest of DMA copy completed packets. - Moreover, calling rte_vhost_submit_enqueue_burst() all the time but - not poll completed will cause the DMA ring to be full, which will - result in packet loss eventually. - -* Recommended IOVA mode in async datapath - - When DMA devices are bound to VFIO driver, VA mode is recommended. - For PA mode, page by page mapping may exceed IOMMU's max capability, - better to use 1G guest hugepage. - - For UIO driver or kernel driver, any VFIO related error messages - can be ignored.
doc/guides/rel_notes/release_26_11.rst +20 -0
@@ -197,6 +197,26 @@ Removed Items The only remaining method for detecting LLDP packets is by using the mbuf packet type in conjunction with the ``enable_lldp`` devarg. +* Removed vhost async (DMA-accelerated) data path. + + The experimental async data path APIs have been removed from the vhost library. + The following functions and flag are no longer available: + + - ``rte_vhost_async_channel_register`` + - ``rte_vhost_async_channel_register_thread_unsafe`` + - ``rte_vhost_async_channel_unregister`` + - ``rte_vhost_async_channel_unregister_thread_unsafe`` + - ``rte_vhost_async_dma_configure`` + - ``rte_vhost_async_dma_unconfigure`` + - ``rte_vhost_submit_enqueue_burst`` + - ``rte_vhost_poll_enqueue_completed`` + - ``rte_vhost_async_try_dequeue_burst`` + - ``rte_vhost_async_get_inflight`` + - ``rte_vhost_async_get_inflight_thread_unsafe`` + - ``rte_vhost_clear_queue`` + - ``rte_vhost_clear_queue_thread_unsafe`` + - ``RTE_VHOST_USER_ASYNC_COPY`` + API Changes -----------
lib/vhost/meson.build +1 -2
@@ -38,10 +38,9 @@ sources = files( headers = files( 'rte_vdpa.h', 'rte_vhost.h', - 'rte_vhost_async.h', 'rte_vhost_crypto.h', ) driver_sdk_headers = files( 'vdpa_driver.h', ) -deps += ['ethdev', 'cryptodev', 'hash', 'pci', 'dmadev'] +deps += ['ethdev', 'cryptodev', 'hash']
lib/vhost/rte_vhost.h +1 -2
@@ -38,8 +38,7 @@ extern "C" {#define RTE_VHOST_USER_EXTBUF_SUPPORT (1ULL << 5)/* support only linear buffers (no chained mbufs) */ #define RTE_VHOST_USER_LINEARBUF_SUPPORT (1ULL << 6) -#define RTE_VHOST_USER_ASYNC_COPY (1ULL << 7) -#define RTE_VHOST_USER_MAP_POPULATE RTE_VHOST_USER_ASYNC_COPY +#define RTE_VHOST_USER_MAP_POPULATE (1ULL << 7)#define RTE_VHOST_USER_NET_COMPLIANT_OL_FLAGS (1ULL << 8)#define RTE_VHOST_USER_NET_STATS_ENABLE (1ULL << 9)#define RTE_VHOST_USER_ASYNC_CONNECT (1ULL << 10)
lib/vhost/rte_vhost_async.h deleted +0 -293
293 lines not on this page · raw message
lib/vhost/socket.c +0 -9
@@ -43,7 +43,6 @@ struct vhost_user_socket {bool use_builtin_virtio_net;bool extbuf;bool linearbuf; -bool async_copy;bool map_populate;bool net_compliant_ol_flags;bool stats_enabled;
@@ -240,13 +239,6 @@ vhost_user_add_connection(int fd, struct vhost_user_socket *vsocket)if (vsocket->linearbuf)vhost_enable_linearbuf (vid); -if (vsocket->async_copy) { - dev =get_device (vid); - -if (dev) - dev->async_copy =1 ; - } -if (vsocket->map_populate) { dev =get_device (vid);
@@ -944,7 +936,6 @@ rte_vhost_driver_register(const char *path, uint64_t flags) vsocket->max_queue_pairs =VHOST_MAX_QUEUE_PAIRS ; vsocket->extbuf = flags &RTE_VHOST_USER_EXTBUF_SUPPORT ; vsocket->linearbuf = flags &RTE_VHOST_USER_LINEARBUF_SUPPORT ; - vsocket->async_copy = flags &RTE_VHOST_USER_ASYNC_COPY ; vsocket->map_populate = flags &RTE_VHOST_USER_MAP_POPULATE ; vsocket->net_compliant_ol_flags = flags &RTE_VHOST_USER_NET_COMPLIANT_OL_FLAGS ; vsocket->stats_enabled = flags &RTE_VHOST_USER_NET_STATS_ENABLE ;
lib/vhost/vhost.c +0 -436
471 lines not on this page · raw message
lib/vhost/vhost.h +0 -117
158 lines not on this page · raw message
lib/vhost/vhost_user.c +0 -111
177 lines not on this page · raw message
lib/vhost/virtio_net.c +660 -2615
1,645 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.54.0
Part of the diff is not shown here. View the raw message.
Re: [PATCH v3 4/6] vhost: drop async datapath
Quoted 26 lines · David Marchand
All the API related to this feature is experimental, allowing us by the project policy to drop the feature in one go.
Signed-off-by David Marchand <[email protected]>---
Changes since v2:
- removed inflight async statistics,Changes since v1:
- dropped dependency on PCI library,Changes since RFC:
- kept mmap populate behavior forvhost_crypto,--- doc/guides/prog_guide/vhost_lib.rst | 151 +- doc/guides/rel_notes/release_26_11.rst | 20 + lib/vhost/meson.build | 3 +- lib/vhost/rte_vhost.h | 3 +- lib/vhost/rte_vhost_async.h | 293 --- lib/vhost/socket.c | 9 - lib/vhost/vhost.c | 436 ---- lib/vhost/vhost.h | 117 - lib/vhost/vhost_user.c | 111 - lib/vhost/virtio_net.c | 3275 +++++------------------- 10 files changed, 687 insertions(+), 3731 deletions(-)
Quoted 2 lines
delete mode 100644
lib/vhost/rte_vhost_async.h
Great, looks good to me:
Thanks,
Maxime
PATCH v3 5/6 vhost: rename packed layout helpers for batches
In reply to David MarchandNow that the async datapath has been removed, the _sync in function names gives no useful information.
Besides, vhost_reserve_avail_batch_packed does not reserve anything and its name is confusing, rename it following the other rx/tx split/packed helpers convention.
---
Changes since v2:
- cleaned up indent,
--- lib/vhost/virtio_net.c | 28 +++++++++------------------- 1 file changed, 9 insertions(+), 19 deletions(-)
lib/vhost/virtio_net.c +9 -19
@@ -1111,11 +1111,8 @@ virtio_dev_rx_split(struct virtio_net *dev, struct vhost_virtqueue *vq, } static __rte_always_inline int -virtio_dev_rx_sync_batch_check (struct virtio_net *dev, -struct vhost_virtqueue *vq, -struct rte_mbuf **pkts, -uint64_t *desc_addrs, -uint64_t *lens) +virtio_dev_rx_batch_check (struct virtio_net *dev,struct vhost_virtqueue *vq, +struct rte_mbuf **pkts,uint64_t *desc_addrs,uint64_t *lens) __rte_requires_shared_capability(&vq->iotlb_lock) {bool wrap_counter = vq->avail_wrap_counter;
@@ -1213,15 +1210,14 @@ virtio_dev_rx_batch_packed_copy(struct virtio_net *dev, } static __rte_always_inline int -virtio_dev_rx_sync_batch_packed (struct virtio_net *dev, -struct vhost_virtqueue *vq, -struct rte_mbuf **pkts) +virtio_dev_rx_batch_packed (struct virtio_net *dev,struct vhost_virtqueue *vq, +struct rte_mbuf **pkts) __rte_requires_shared_capability(&vq->iotlb_lock) {uint64_t desc_addrs[PACKED_BATCH_SIZE ];uint64_t lens[PACKED_BATCH_SIZE ]; -if (virtio_dev_rx_sync_batch_check (dev, vq, pkts, desc_addrs, lens) ==-1 ) +if (virtio_dev_rx_batch_check (dev, vq, pkts, desc_addrs, lens) ==-1 )return -1 ;if (vq->shadow_used_idx) {
@@ -1273,8 +1269,7 @@ virtio_dev_rx_packed(struct virtio_net *dev,rte_prefetch0 (&vq->desc_packed[vq->last_avail_idx]);if (count - pkt_idx >=PACKED_BATCH_SIZE ) { -if (!virtio_dev_rx_sync_batch_packed (dev, vq, - &pkts[pkt_idx])) { +if (!virtio_dev_rx_batch_packed (dev, vq, &pkts[pkt_idx])) { pkt_idx +=PACKED_BATCH_SIZE ;continue ; }
@@ -1971,12 +1966,8 @@ virtio_dev_tx_split_compliant(struct virtio_net *dev, } static __rte_always_inline int -vhost_reserve_avail_batch_packed (struct virtio_net *dev, -struct vhost_virtqueue *vq, -struct rte_mbuf **pkts, -uint16_t avail_idx, -uintptr_t *desc_addrs, -uint16_t *ids) +virtio_dev_tx_batch_packed_check (struct virtio_net *dev,struct vhost_virtqueue *vq, +struct rte_mbuf **pkts,uint16_t avail_idx,uintptr_t *desc_addrs,uint16_t *ids) __rte_requires_shared_capability(&vq->iotlb_lock) {bool wrap = vq->avail_wrap_counter;
@@ -2055,8 +2046,7 @@ virtio_dev_tx_batch_packed(struct virtio_net *dev,uint16_t ids[PACKED_BATCH_SIZE ];uint16_t i; -if (vhost_reserve_avail_batch_packed (dev, vq, pkts, avail_idx, - desc_addrs, ids)) +if (virtio_dev_tx_batch_packed_check (dev, vq, pkts, avail_idx, desc_addrs, ids))return -1 ;vhost_for_each_try_unroll (i,0 ,PACKED_BATCH_SIZE )
Signature · 1 line
2.54.0
Re: [PATCH v3 5/6] vhost: rename packed layout helpers for batches
Quoted 7 lines · David Marchand
Now that the async datapath has been removed, the
_syncin function names gives no useful information.Besides,
vhost_reserve_avail_batch_packeddoes not reserve anything and its name is confusing, rename it following the other rx/tx split/packed helpers convention.
Quoted 9 lines
Signed-off-by David Marchand <[email protected]>---
Changes since v2:
- cleaned up indent,--- lib/vhost/virtio_net.c | 28 +++++++++------------------- 1 file changed, 9 insertions(+), 19 deletions(-)
Thanks,
Maxime
PATCH v3 6/6 vhost: simplify some descriptor handling
In reply to David MarchandNow that the async datapath has been removed, move back the descriptors copy/filling code in desc_to_mbuf/mbuf_to_desc and eliminate an intermediate helper.
Tracking buf_iova is not needed in the descriptor to mbuf case.
---
Changes since v2:
- used RTE_PTR_ADD for consistency,
--- lib/vhost/virtio_net.c | 80 ++++++++++++++++-------------------------- 1 file changed, 31 insertions(+), 49 deletions(-)
lib/vhost/virtio_net.c +31 -49
@@ -76,6 +76,12 @@ vhost_queue_stats_update(const struct virtio_net *dev, struct vhost_virtqueue *v } } +static __rte_always_inline bool +vhost_can_batch_copy (struct vhost_virtqueue *vq,uint32_t cpy_len) +{ +return cpy_len <=MAX_BATCH_LEN && vq->batch_copy_nb_elems < vq->size; +} + static inline void do_data_copy_enqueue(struct virtio_net *dev, struct vhost_virtqueue *vq) __rte_requires_shared_capability(&vq->iotlb_lock)
@@ -843,44 +849,6 @@ copy_vnet_hdr_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, } } -static __rte_always_inline void -sync_fill_seg (struct virtio_net *dev,struct vhost_virtqueue *vq, -struct rte_mbuf *m,uint32_t mbuf_offset, -uint64_t buf_addr,uint64_t buf_iova,uint32_t cpy_len,bool to_desc) - __rte_requires_shared_capability(&vq->iotlb_lock) -{ -struct batch_copy_elem *batch_copy = vq->batch_copy_elems; - -if (likely (cpy_len >MAX_BATCH_LEN || vq->batch_copy_nb_elems >= vq->size)) { -if (to_desc) { -rte_memcpy ((void *)((uintptr_t )(buf_addr)), - rte_pktmbuf_mtod_offset(m, void *, mbuf_offset), - cpy_len); -vhost_log_cache_write_iova (dev, vq, buf_iova, cpy_len); -PRINT_PACKET (dev, (uintptr_t )(buf_addr), cpy_len,0 ); - }else { - rte_memcpy(rte_pktmbuf_mtod_offset(m, void *, mbuf_offset), - (void *)((uintptr_t)(buf_addr)), - cpy_len); - } - }else { -if (to_desc) { - batch_copy[vq->batch_copy_nb_elems].dst = - (void *)((uintptr_t )(buf_addr)); - batch_copy[vq->batch_copy_nb_elems].src = - rte_pktmbuf_mtod_offset(m, void *, mbuf_offset); - batch_copy[vq->batch_copy_nb_elems].log_addr = buf_iova; - }else { - batch_copy[vq->batch_copy_nb_elems].dst = - rte_pktmbuf_mtod_offset(m, void *, mbuf_offset); - batch_copy[vq->batch_copy_nb_elems].src = - (void *)((uintptr_t )(buf_addr)); - } - batch_copy[vq->batch_copy_nb_elems].len = cpy_len; - vq->batch_copy_nb_elems++; - } -} - static __rte_always_inline int mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, struct rte_mbuf *m, struct buf_vector *buf_vec,
@@ -976,9 +944,22 @@ mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, cpy_len =RTE_MIN (buf_avail, mbuf_avail); -sync_fill_seg (dev, vq, m, mbuf_offset, - buf_addr + buf_offset, - buf_iova + buf_offset, cpy_len, true); +if (likely (!vhost_can_batch_copy (vq, cpy_len))) { +rte_memcpy (RTE_PTR_ADD (buf_addr, buf_offset), + rte_pktmbuf_mtod_offset(m, void *, mbuf_offset), + cpy_len); +vhost_log_cache_write_iova (dev, vq, buf_iova + buf_offset, cpy_len); +PRINT_PACKET (dev,RTE_PTR_ADD (buf_addr, buf_offset), cpy_len,0 ); + }else { + vq->batch_copy_elems[vq->batch_copy_nb_elems].dst = +RTE_PTR_ADD (buf_addr, buf_offset); + vq->batch_copy_elems[vq->batch_copy_nb_elems].src = + rte_pktmbuf_mtod_offset(m, void *, mbuf_offset); + vq->batch_copy_elems[vq->batch_copy_nb_elems].log_addr = + buf_iova + buf_offset; + vq->batch_copy_elems[vq->batch_copy_nb_elems].len = cpy_len; + vq->batch_copy_nb_elems++; + } mbuf_avail -= cpy_len; mbuf_offset += cpy_len;
@@ -1675,7 +1656,7 @@ desc_to_mbuf(struct virtio_net *dev, struct vhost_virtqueue *vq, __rte_requires_shared_capability(&vq->iotlb_lock) {uint32_t buf_avail, buf_offset, buf_len; -uint64_t buf_addr, buf_iova; +uint64_t buf_addr;uint32_t mbuf_avail, mbuf_offset;uint32_t hdr_remain = dev->vhost_hlen;uint32_t cpy_len;
@@ -1706,7 +1687,6 @@ desc_to_mbuf(struct virtio_net *dev, struct vhost_virtqueue *vq, } buf_addr = buf_vec[vec_idx].buf_addr; - buf_iova = buf_vec[vec_idx].buf_iova; buf_len = buf_vec[vec_idx].buf_len; buf_offset = hdr_remain; buf_avail = buf_vec[vec_idx].buf_len - hdr_remain;
@@ -1721,14 +1701,17 @@ desc_to_mbuf(struct virtio_net *dev, struct vhost_virtqueue *vq,while (1 ) { cpy_len =RTE_MIN (buf_avail, mbuf_avail); -if (likely (hdr && cur == m)) { +if (likely ((hdr && cur == m) || !vhost_can_batch_copy (vq, cpy_len))) { rte_memcpy(rte_pktmbuf_mtod_offset(cur, void *, mbuf_offset), - (void *)((uintptr_t)(buf_addr + buf_offset)), +RTE_PTR_ADD (buf_addr ,buf_offset ), cpy_len); }else { -sync_fill_seg (dev, vq, cur, mbuf_offset, - buf_addr + buf_offset, - buf_iova + buf_offset, cpy_len, false); + vq->batch_copy_elems[vq->batch_copy_nb_elems].dst = + rte_pktmbuf_mtod_offset(cur, void *, mbuf_offset); + vq->batch_copy_elems[vq->batch_copy_nb_elems].src = +RTE_PTR_ADD (buf_addr, buf_offset); + vq->batch_copy_elems[vq->batch_copy_nb_elems].len = cpy_len; + vq->batch_copy_nb_elems++; } mbuf_avail -= cpy_len;
@@ -1742,7 +1725,6 @@ desc_to_mbuf(struct virtio_net *dev, struct vhost_virtqueue *vq,break ; buf_addr = buf_vec[vec_idx].buf_addr; - buf_iova = buf_vec[vec_idx].buf_iova; buf_len = buf_vec[vec_idx].buf_len; buf_offset =0 ;
Signature · 1 line
2.54.0
Re: [PATCH v3 6/6] vhost: simplify some descriptor handling
Quoted 6 lines · David Marchand
Now that the async datapath has been removed, move back the descriptors copy/filling code in desc_to_mbuf/mbuf_to_desc and eliminate an intermediate helper.
Tracking
buf_iovais not needed in the descriptor to mbuf case.
Quoted 9 lines
Signed-off-by David Marchand <[email protected]>---
Changes since v2:
- usedRTE_PTR_ADDfor consistency,--- lib/vhost/virtio_net.c | 80 ++++++++++++++++-------------------------- 1 file changed, 31 insertions(+), 49 deletions(-)
Thanks,
Maxime
v4 cover Drop vhost async (DMA-accelerated) datapath
In reply to David MarchandThe integration of this feature in OVS has been postponed following feedback how the current API forced the complexity on the application side. There is no contributor working on making those changes.
The feature has no other identified opensource consumer.
On the other hand, this feature has been left unattended for some years now while the "normal" datapath is actively maintained.
It is highly likely the async datapath code, which never underwent a stabilisation phase, is riddled with bugs.
Given the craziness around finding security bugs with AI, and since the vhost library is a good target, I propose to drop this feature and reduce the library attack surface.
There is one complication with the vhost_crypto and some external applications that seemed to rely on the async flag to track and populate guest pages, so a new flag replacing async is added by this series.
Signature · 51 lines
David Marchand Changes since v3: - rebased after CVE fixes and various changes: RTE_PTR_ADD, VFIO, signal cleanup in examples, Changes since v2: - updated release notes and vhost example documentation, - removed some async statistics missed in previous revisions, - re-added checks on map populate and post copy or other virtio features, Changes since v1: - removed unneeded dependencies on the PCI library, Changes since RFC: - maintained guest pages population in a new flag (vhost_crypto/SPDK), - removed remaining helpers which had sync_ in their names, David Marchand (6): vhost: decouple guest pages population from async datapath examples/vhost: stop using async datapath examples/vdpa: remove dependency on PCI library vhost: drop async datapath vhost: rename packed layout helpers for batches vhost: simplify some descriptor handling doc/guides/prog_guide/vhost_lib.rst | 152 +- doc/guides/rel_notes/release_26_11.rst | 24 + doc/guides/sample_app_ug/vhost.rst | 10 - examples/vdpa/main.c | 1 - examples/vhost/Makefile | 2 - examples/vhost/main.c | 437 +--- examples/vhost/main.h | 21 +- examples/vhost/meson.build | 2 - examples/vhost_crypto/Makefile | 1 - examples/vhost_crypto/main.c | 2 +- examples/vhost_crypto/meson.build | 1 - lib/vhost/meson.build | 3 +- lib/vhost/rte_vhost.h | 2 +- lib/vhost/rte_vhost_async.h | 293 --- lib/vhost/socket.c | 23 +- lib/vhost/vhost.c | 436 ---- lib/vhost/vhost.h | 119 +- lib/vhost/vhost_user.c | 115 +- lib/vhost/virtio_net.c | 3318 +++++------------------- 19 files changed, 731 insertions(+), 4231 deletions(-) delete mode 100644 lib/vhost/rte_vhost_async.h -- 2.54.0
PATCH v4 1/6 vhost: decouple guest pages population from async datapath
The vhost crypto zero-copy feature relies on the RTE_VHOST_USER_ASYNC_COPY flag to enable guest page tracking (guest_pages array population and MAP_POPULATE during mmap).
Introduce a RTE_VHOST_USER_MAP_POPULATE flag that maps to RTE_VHOST_USER_ASYNC_COPY at this point in time.
Adjust vhost_crypto example and remove the unnecessary ALLOW_EXPERIMENTAL_API flag as this example does not use any experimental APIs.
---
Changes since v3:
- rebased after VFIO changes,
Changes since v2:
- updated release notes for new flag,
- fixed checks on map populate feature,
--- doc/guides/prog_guide/vhost_lib.rst | 9 +++++++++ doc/guides/rel_notes/release_26_11.rst | 4 ++++ examples/vhost_crypto/Makefile | 1 - examples/vhost_crypto/main.c | 2 +- examples/vhost_crypto/meson.build | 1 - lib/vhost/rte_vhost.h | 1 + lib/vhost/socket.c | 19 +++++++++++++++---- lib/vhost/vhost.h | 1 + lib/vhost/vhost_user.c | 4 ++-- 9 files changed, 33 insertions(+), 9 deletions(-)
doc/guides/prog_guide/vhost_lib.rst +9 -0
@@ -118,6 +118,15 @@ The following is an overview of some key Vhost API functions: It is disabled by default. + - ``RTE_VHOST_USER_MAP_POPULATE`` + + Guest memory regions will be mapped with ``MAP_POPULATE`` when this flag + is set, pre-faulting pages into memory. This is useful for applications + requiring direct access to guest memory, such as vhost-crypto zero-copy + operations. + + It is disabled by default. + - ``RTE_VHOST_USER_NET_COMPLIANT_OL_FLAGS`` Since v16.04, the vhost library forwards checksum and gso requests for
doc/guides/rel_notes/release_26_11.rst +4 -0
@@ -448,6 +448,10 @@ API Changes ``rte_event_port_profile_links_get`` and ``rte_event_port_profile_unlink`` +* vhost: Introduced ``RTE_VHOST_USER_MAP_POPULATE`` as a replacement of + ``RTE_VHOST_USER_ASYNC_COPY`` for users that did not rely on the vhost + async copy feature but expected guest pages to be prefaulted. + * reorder: Promoted the following API from experimental to stable: ``rte_reorder_seqn``, ``rte_reorder_drain_up_to_seqn``, ``rte_reorder_min_seqn_set`` and ``rte_reorder_memory_footprint_get``.
examples/vhost_crypto/Makefile +0 -1
@@ -6,7 +6,6 @@ APP = vhost-crypto # all source are stored in SRCS-y SRCS-y := main.c -CFLAGS += -DALLOW_EXPERIMENTAL_API PKGCONF ?= pkg-config
examples/vhost_crypto/main.c +1 -1
@@ -623,7 +623,7 @@ main(int argc, char *argv[])for (j =0 ; j < lo->nb_sockets; j++) { ret =rte_vhost_driver_register (lo->socket_files[j], -RTE_VHOST_USER_ASYNC_COPY ); +RTE_VHOST_USER_MAP_POPULATE );if (ret <0 ) {RTE_LOG (ERR ,USER1 ,"socket %s already exists\n" , lo->socket_files[j]);
examples/vhost_crypto/meson.build +0 -1
@@ -6,7 +6,6 @@ # To build this example as a standalone application with an already-installed # DPDK instance, use 'make' -allow_experimental_apis = true deps += ['vhost', 'cryptodev'] sources = files( 'main.c',
lib/vhost/rte_vhost.h +1 -0
@@ -39,6 +39,7 @@ extern "C" {/* support only linear buffers (no chained mbufs) */ #define RTE_VHOST_USER_LINEARBUF_SUPPORT (1ULL << 6)#define RTE_VHOST_USER_ASYNC_COPY (1ULL << 7) +#define RTE_VHOST_USER_MAP_POPULATE RTE_VHOST_USER_ASYNC_COPY#define RTE_VHOST_USER_NET_COMPLIANT_OL_FLAGS (1ULL << 8)#define RTE_VHOST_USER_NET_STATS_ENABLE (1ULL << 9)#define RTE_VHOST_USER_ASYNC_CONNECT (1ULL << 10)
lib/vhost/socket.c +15 -4
@@ -45,6 +45,7 @@ struct vhost_user_socket {bool extbuf;bool linearbuf;bool async_copy; +bool map_populate;bool net_compliant_ol_flags;bool stats_enabled;bool async_connect;
@@ -249,6 +250,13 @@ vhost_user_add_connection(int fd, struct vhost_user_socket *vsocket) } } +if (vsocket->map_populate) { + dev =get_device (vid); + +if (dev) + dev->map_populate =1 ; + } +VHOST_CONFIG_LOG (vsocket->path,INFO ,"new device, handle is %d" , vid);if (vsocket->notify_ops->new_connection) {
@@ -940,6 +948,7 @@ rte_vhost_driver_register(const char *path, uint64_t flags) vsocket->extbuf = flags &RTE_VHOST_USER_EXTBUF_SUPPORT ; vsocket->linearbuf = flags &RTE_VHOST_USER_LINEARBUF_SUPPORT ; vsocket->async_copy = flags &RTE_VHOST_USER_ASYNC_COPY ; + vsocket->map_populate = flags &RTE_VHOST_USER_MAP_POPULATE ; vsocket->net_compliant_ol_flags = flags &RTE_VHOST_USER_NET_COMPLIANT_OL_FLAGS ; vsocket->stats_enabled = flags &RTE_VHOST_USER_NET_STATS_ENABLE ; vsocket->async_connect = flags &RTE_VHOST_USER_ASYNC_CONNECT ;
@@ -948,9 +957,10 @@ rte_vhost_driver_register(const char *path, uint64_t flags)else vsocket->iommu_support = flags &RTE_VHOST_USER_IOMMU_SUPPORT ; -if (vsocket->async_copy && (vsocket->iommu_support || +if (vsocket->map_populate && (vsocket->iommu_support || (flags &RTE_VHOST_USER_POSTCOPY_SUPPORT ))) { -VHOST_CONFIG_LOG (path,ERR ,"async copy with IOMMU or post-copy not supported" ); +VHOST_CONFIG_LOG (path,ERR , +"guest pages prefaulting not supported with IOMMU or post-copy" ); goto out_mutex; }
@@ -976,10 +986,11 @@ rte_vhost_driver_register(const char *path, uint64_t flags) vsocket->protocol_features =VHOST_USER_PROTOCOL_FEATURES ; } -if (vsocket->async_copy) { +if (vsocket->map_populate) { vsocket->supported_features &= ~(1ULL <<VHOST_F_LOG_ALL ); vsocket->features &= ~(1ULL <<VHOST_F_LOG_ALL ); -VHOST_CONFIG_LOG (path,INFO ,"logging feature is disabled in async copy mode" ); +VHOST_CONFIG_LOG (path,INFO , +"logging feature is disabled with guest pages prefaulting" ); }/*
lib/vhost/vhost.h +1 -0
@@ -500,6 +500,7 @@ struct __rte_cache_aligned virtio_net {uint32_t nr_vring;int async_copy;bool dma_map_available; +int map_populate;int extbuf;int linearbuf;
lib/vhost/vhost_user.c +2 -2
@@ -1406,7 +1406,7 @@ vhost_user_mmap_region(struct virtio_net *dev,return -1 ; } - populate = dev->async_copy ?MAP_POPULATE :0 ; + populate = dev->map_populate ?MAP_POPULATE :0 ; mmap_addr =mmap (NULL , mmap_size,PROT_READ |PROT_WRITE ,MAP_SHARED | populate, region->fd,0 );
@@ -1420,7 +1420,7 @@ vhost_user_mmap_region(struct virtio_net *dev, region->host_user_addr = (uint64_t )(uintptr_t )mmap_addr + mmap_offset;mem_set_dump (dev, mmap_addr, mmap_size, false, alignment); -if (dev->async_copy) { +if (dev->map_populate) {if (add_guest_pages (dev, region, alignment) <0 ) {VHOST_CONFIG_LOG (dev->ifname,ERR ,"adding guest pages to region failed." );
Signature · 1 line
2.54.0
PATCH v4 2/6 examples/vhost: stop using async datapath
In reply to David MarchandStop using vhost async as this feature will be removed in a next commit. Thanks to this, we can drop use of experimental api.
---
Changes since v3:
- rebased after signal cleanup changes,
Changes since v2:
- removed --dmas option in vhost example doc,
Changes since RFC:
- dropped vhost_crypto changes (now handled in patch 1),
--- doc/guides/sample_app_ug/vhost.rst | 10 - examples/vhost/Makefile | 2 - examples/vhost/main.c | 437 +---------------------------- examples/vhost/main.h | 21 +- examples/vhost/meson.build | 2 - 5 files changed, 12 insertions(+), 460 deletions(-)
doc/guides/sample_app_ug/vhost.rst +0 -10
@@ -149,16 +149,6 @@ default value is 15. A very simple vhost-user net driver which demonstrates how to use the generic vhost APIs will be used when this option is given. It is disabled by default. -**--dmas** -This parameter is used to specify the assigned DMA device of a vhost device. -Async vhost-user net driver will be used if --dmas is set. For example ---dmas [txd0@00:04.0,txd1@00:04.1,rxd0@00:04.2,rxd1@00:04.3] means use -DMA channel 00:04.0/00:04.2 for vhost device 0 enqueue/dequeue operation -and use DMA channel 00:04.1/00:04.3 for vhost device 1 enqueue/dequeue -operation. The index of the device corresponds to the socket file in order, -that means vhost device 0 is created through the first socket file, vhost -device 1 is created through the second socket file, and so on. - **--total-num-mbufs 0-N** This parameter sets the number of mbufs to be allocated in mbuf pools, the default value is 147456. This is can be used if launch of a port fails
examples/vhost/Makefile +0 -2
@@ -28,8 +28,6 @@ CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk) LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk) -CFLAGS += -DALLOW_EXPERIMENTAL_API - build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build $(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
examples/vhost/main.c +10 -427
604 lines not on this page · raw message
examples/vhost/main.h +2 -19
@@ -8,7 +8,6 @@#include <sys/queue.h> #include <rte_ether.h> -#include <rte_pci.h> /* Macros for printing using RTE_LOG */ #define RTE_LOGTYPE_VHOST_CONFIG RTE_LOGTYPE_USER1
@@ -92,17 +91,6 @@ struct lcore_info {struct vhost_dev_tailq_list vdev_list; }; -struct dma_info { -struct rte_pci_addr addr; -int16_t dev_id; -bool async_enabled; -}; - -struct dma_for_vhost { -struct dma_info dmas[RTE_MAX_QUEUES_PER_PORT *2 ]; -uint32_t async_flag; -}; -/* we implement non-extra virtio net features */ #define VIRTIO_NET_FEATURES 0
@@ -116,14 +104,9 @@ uint16_t builtin_enqueue_pkts(struct vhost_dev *dev, uint16_t queue_id,uint16_t builtin_dequeue_pkts (struct vhost_dev *dev,uint16_t queue_id,struct rte_mempool *mbuf_pool,struct rte_mbuf **pkts,uint16_t count); -uint16_t sync_enqueue_pkts (struct vhost_dev *dev,uint16_t queue_id, -struct rte_mbuf **pkts,uint32_t count); -uint16_t sync_dequeue_pkts (struct vhost_dev *dev,uint16_t queue_id, -struct rte_mempool *mbuf_pool, -struct rte_mbuf **pkts,uint16_t count); -uint16_t async_enqueue_pkts (struct vhost_dev *dev,uint16_t queue_id, +uint16_t enqueue_pkts (struct vhost_dev *dev,uint16_t queue_id,struct rte_mbuf **pkts,uint32_t count); -uint16_t async_dequeue_pkts (struct vhost_dev *dev,uint16_t queue_id, +uint16_t dequeue_pkts (struct vhost_dev *dev,uint16_t queue_id,struct rte_mempool *mbuf_pool,struct rte_mbuf **pkts,uint16_t count);#endif /* _MAIN_H_ */
examples/vhost/meson.build +0 -2
@@ -12,8 +12,6 @@ if not is_linux endif deps += 'vhost' -deps += 'dmadev' -allow_experimental_apis = true sources = files( 'main.c', 'virtio_net.c',
Signature · 1 line
2.54.0
PATCH v4 3/6 examples/vdpa: remove dependency on PCI library
In reply to David MarchandThis is some forgotten unneeded header include from a previous refactoring.
--- examples/vdpa/main.c | 1 - 1 file changed, 1 deletion(-)
examples/vdpa/main.c +0 -1
@@ -13,7 +13,6 @@#include <rte_malloc.h> #include <rte_vhost.h> #include <rte_vdpa.h> -#include <rte_pci.h> #include <rte_string_fns.h> #include <cmdline_socket.h>
Signature · 1 line
2.54.0
PATCH v4 4/6 vhost: drop async datapath
In reply to David MarchandAll the API related to this feature is experimental, allowing us by the project policy to drop the feature in one go.
---
Changes since v3:
- rebased after VFIO changes,
Changes since v2:
- removed inflight async statistics,
Changes since v1:
- dropped dependency on PCI library,
Changes since RFC:
- kept mmap populate behavior for vhost_crypto,
--- doc/guides/prog_guide/vhost_lib.rst | 151 +- doc/guides/rel_notes/release_26_11.rst | 20 + lib/vhost/meson.build | 3 +- lib/vhost/rte_vhost.h | 3 +- lib/vhost/rte_vhost_async.h | 293 --- lib/vhost/socket.c | 12 - lib/vhost/vhost.c | 436 ---- lib/vhost/vhost.h | 118 - lib/vhost/vhost_user.c | 111 - lib/vhost/virtio_net.c | 3275 +++++------------------- 10 files changed, 687 insertions(+), 3735 deletions(-) delete mode 100644 lib/vhost/rte_vhost_async.h
doc/guides/prog_guide/vhost_lib.rst +5 -146
@@ -103,21 +103,6 @@ The following is an overview of some key Vhost API functions: It is disabled by default. - - ``RTE_VHOST_USER_ASYNC_COPY`` - - Asynchronous data path will be enabled when this flag is set. Async - data path allows applications to enable DMA acceleration for vhost - queues. Vhost leverages the registered DMA channels to free CPU from - memory copy operations in data path. A set of async data path APIs are - defined for DPDK applications to make use of the async capability. Only - packets enqueued/dequeued by async APIs are processed through the async - data path. - - Currently this feature is only implemented on split ring enqueue data - path. - - It is disabled by default. - - ``RTE_VHOST_USER_MAP_POPULATE`` Guest memory regions will be mapped with ``MAP_POPULATE`` when this flag
@@ -236,76 +221,6 @@ The following is an overview of some key Vhost API functions: Enable or disable zero copy feature of the vhost crypto backend. -* ``rte_vhost_async_dma_configure(dma_id, vchan_id)`` - - Tell vhost which DMA vChannel is going to use. This function needs to - be called before register async data-path for vring. - -* ``rte_vhost_async_channel_register(vid, queue_id)`` - - Register async DMA acceleration for a vhost queue after vring is enabled. - -* ``rte_vhost_async_channel_register_thread_unsafe(vid, queue_id)`` - - Register async DMA acceleration for a vhost queue without performing - any locking. - - This function is only safe to call in vhost callback functions - (i.e., struct rte_vhost_device_ops). - -* ``rte_vhost_async_channel_unregister(vid, queue_id)`` - - Unregister the async DMA acceleration from a vhost queue. - Unregistration will fail, if the vhost queue has in-flight - packets that are not completed. - - Unregister async DMA acceleration in vring_state_changed() may - fail, as this API tries to acquire the spinlock of vhost - queue. The recommended way is to unregister async copy - devices for all vhost queues in destroy_device(), when a - virtio device is paused or shut down. - -* ``rte_vhost_async_channel_unregister_thread_unsafe(vid, queue_id)`` - - Unregister async DMA acceleration for a vhost queue without performing - any locking. - - This function is only safe to call in vhost callback functions - (i.e., struct rte_vhost_device_ops). - -* ``rte_vhost_submit_enqueue_burst(vid, queue_id, pkts, count, dma_id, vchan_id)`` - - Submit an enqueue request to transmit ``count`` packets from host to guest - by async data path. Applications must not free the packets submitted for - enqueue until the packets are completed. - -* ``rte_vhost_poll_enqueue_completed(vid, queue_id, pkts, count, dma_id, vchan_id)`` - - Poll enqueue completion status from async data path. Completed packets - are returned to applications through ``pkts``. - -* ``rte_vhost_async_get_inflight(vid, queue_id)`` - - This function returns the amount of in-flight packets for the vhost - queue using async acceleration. - - * ``rte_vhost_async_get_inflight_thread_unsafe(vid, queue_id)`` - - Get the number of inflight packets for a vhost queue without performing - any locking. It should only be used within the vhost ops, which already - holds the lock. - -* ``rte_vhost_clear_queue_thread_unsafe(vid, queue_id, **pkts, count, dma_id, vchan_id)`` - - Clear in-flight packets which are submitted to async channel in vhost - async data path without performing locking on virtqueue. Completed - packets are returned to applications through ``pkts``. - -* ``rte_vhost_clear_queue(vid, queue_id, **pkts, count, dma_id, vchan_id)`` - - Clear in-flight packets which are submitted to async channel in vhost async data - path. Completed packets are returned to applications through ``pkts``. - * ``rte_vhost_vring_call_nonblock(int vid, uint16_t vring_idx)`` Notify the guest that used descriptors have been added to the vring. This function
@@ -327,22 +242,11 @@ The following is an overview of some key Vhost API functions: This function resets the queue statistics. It requires statistics collection to be enabled at registration time. -* ``rte_vhost_async_try_dequeue_burst(vid, queue_id, mbuf_pool, pkts, count, - nr_inflight, dma_id, vchan_id)`` - - Receive ``count`` packets from guest to host in async data path, - and store them at ``pkts``. - * ``rte_vhost_driver_get_vdpa_dev_type(path, type)`` Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET, VDPA_DEVICE_TYPE_BLK. -* ``rte_vhost_async_dma_unconfigure(dma_id, vchan_id)`` - - Clean up a DMA vChannel after use is finished. After this function is called, - the specified DMA vChannel should no longer be used by the Vhost library. - * ``rte_vhost_notify_guest(int vid, uint16_t queue_id)`` Inject the offloaded interrupt received by the 'guest_notify' callback,
@@ -411,16 +315,11 @@ Guest memory requirement * Memory pre-allocation - For non-async data path guest memory pre-allocation is not a - must but can help save memory. To do this we can add option - ``-mem-prealloc`` when starting QEMU, or we can lock all memory at vhost - side which will force memory to be allocated when it calls mmap - (option --mlockall in ovs-dpdk is an example in hand). - - - For async data path, we force the VM memory to be pre-allocated at vhost - lib when mapping the guest memory; and also we need to lock the memory to - prevent pages being swapped out to disk. + Guest memory pre-allocation is not a must but can help save memory. + To do this we can add option ``-mem-prealloc`` when starting QEMU, + or we can lock all memory at vhost side which will force memory to + be allocated when it calls mmap (option --mlockall in ovs-dpdk is + an example in hand). * Memory sharing
@@ -487,43 +386,3 @@ Finally, a set of device ops is defined for device specific operations: * ``get_notify_area`` Called to get the notify area info of the queue. - -Vhost asynchronous data path ----------------------------- - -Vhost asynchronous data path leverages DMA devices to offload memory -copies from the CPU and it is implemented in an asynchronous way. It -enables applications, like OVS, to save CPU cycles and hide memory copy -overhead, thus achieving higher throughput. - -Vhost doesn't manage DMA devices and applications, like OVS, need to -manage and configure DMA devices. Applications need to tell vhost what -DMA devices to use in every data path function call. This design enables -the flexibility for applications to dynamically use DMA channels in -different function modules, not limited in vhost. - -In addition, vhost supports M:N mapping between vrings and DMA virtual -channels. Specifically, one vring can use multiple different DMA channels -and one DMA channel can be shared by multiple vrings at the same time. -The reason of enabling one vring to use multiple DMA channels is that -it's possible that more than one dataplane threads enqueue packets to -the same vring with their own DMA virtual channels. Besides, the number -of DMA devices is limited. For the purpose of scaling, it's necessary to -support sharing DMA channels among vrings. - -* Async enqueue API usage - - In async enqueue path, rte_vhost_poll_enqueue_completed() needs to be - called in time to notify the guest of DMA copy completed packets. - Moreover, calling rte_vhost_submit_enqueue_burst() all the time but - not poll completed will cause the DMA ring to be full, which will - result in packet loss eventually. - -* Recommended IOVA mode in async datapath - - When DMA devices are bound to VFIO driver, VA mode is recommended. - For PA mode, page by page mapping may exceed IOMMU's max capability, - better to use 1G guest hugepage. - - For UIO driver or kernel driver, any VFIO related error messages - can be ignored.
doc/guides/rel_notes/release_26_11.rst +20 -0
@@ -294,6 +294,26 @@ Removed Items The only remaining method for detecting LLDP packets is by using the mbuf packet type in conjunction with the ``enable_lldp`` devarg. +* Removed vhost async (DMA-accelerated) data path. + + The experimental async data path APIs have been removed from the vhost library. + The following functions and flag are no longer available: + + - ``rte_vhost_async_channel_register`` + - ``rte_vhost_async_channel_register_thread_unsafe`` + - ``rte_vhost_async_channel_unregister`` + - ``rte_vhost_async_channel_unregister_thread_unsafe`` + - ``rte_vhost_async_dma_configure`` + - ``rte_vhost_async_dma_unconfigure`` + - ``rte_vhost_submit_enqueue_burst`` + - ``rte_vhost_poll_enqueue_completed`` + - ``rte_vhost_async_try_dequeue_burst`` + - ``rte_vhost_async_get_inflight`` + - ``rte_vhost_async_get_inflight_thread_unsafe`` + - ``rte_vhost_clear_queue`` + - ``rte_vhost_clear_queue_thread_unsafe`` + - ``RTE_VHOST_USER_ASYNC_COPY`` + API Changes -----------
lib/vhost/meson.build +1 -2
@@ -38,10 +38,9 @@ sources = files( headers = files( 'rte_vdpa.h', 'rte_vhost.h', - 'rte_vhost_async.h', 'rte_vhost_crypto.h', ) driver_sdk_headers = files( 'vdpa_driver.h', ) -deps += ['ethdev', 'cryptodev', 'hash', 'pci', 'dmadev'] +deps += ['ethdev', 'cryptodev', 'hash']
lib/vhost/rte_vhost.h +1 -2
@@ -38,8 +38,7 @@ extern "C" {#define RTE_VHOST_USER_EXTBUF_SUPPORT (1ULL << 5)/* support only linear buffers (no chained mbufs) */ #define RTE_VHOST_USER_LINEARBUF_SUPPORT (1ULL << 6) -#define RTE_VHOST_USER_ASYNC_COPY (1ULL << 7) -#define RTE_VHOST_USER_MAP_POPULATE RTE_VHOST_USER_ASYNC_COPY +#define RTE_VHOST_USER_MAP_POPULATE (1ULL << 7)#define RTE_VHOST_USER_NET_COMPLIANT_OL_FLAGS (1ULL << 8)#define RTE_VHOST_USER_NET_STATS_ENABLE (1ULL << 9)#define RTE_VHOST_USER_ASYNC_CONNECT (1ULL << 10)
lib/vhost/rte_vhost_async.h deleted +0 -293
293 lines not on this page · raw message
lib/vhost/socket.c +0 -12
@@ -17,7 +17,6 @@#include <eal_export.h> #include <rte_thread.h> #include <rte_log.h> -#include <dev_vfio.h> #include "fd_man.h" #include "vduse.h"
@@ -44,7 +43,6 @@ struct vhost_user_socket {bool use_builtin_virtio_net;bool extbuf;bool linearbuf; -bool async_copy;bool map_populate;bool net_compliant_ol_flags;bool stats_enabled;
@@ -241,15 +239,6 @@ vhost_user_add_connection(int fd, struct vhost_user_socket *vsocket)if (vsocket->linearbuf)vhost_enable_linearbuf (vid); -if (vsocket->async_copy) { - dev =get_device (vid); - -if (dev !=NULL ) { - dev->async_copy =1 ; - dev->dma_map_available =dev_vfio_is_enabled (); - } - } -if (vsocket->map_populate) { dev =get_device (vid);
@@ -947,7 +936,6 @@ rte_vhost_driver_register(const char *path, uint64_t flags) vsocket->max_queue_pairs =VHOST_MAX_QUEUE_PAIRS ; vsocket->extbuf = flags &RTE_VHOST_USER_EXTBUF_SUPPORT ; vsocket->linearbuf = flags &RTE_VHOST_USER_LINEARBUF_SUPPORT ; - vsocket->async_copy = flags &RTE_VHOST_USER_ASYNC_COPY ; vsocket->map_populate = flags &RTE_VHOST_USER_MAP_POPULATE ; vsocket->net_compliant_ol_flags = flags &RTE_VHOST_USER_NET_COMPLIANT_OL_FLAGS ; vsocket->stats_enabled = flags &RTE_VHOST_USER_NET_STATS_ENABLE ;
lib/vhost/vhost.c +0 -436
471 lines not on this page · raw message
lib/vhost/vhost.h +0 -118
159 lines not on this page · raw message
lib/vhost/vhost_user.c +0 -111
177 lines not on this page · raw message
lib/vhost/virtio_net.c +660 -2615
1,635 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.54.0
Part of the diff is not shown here. View the raw message.
Re: [PATCH v4 4/6] vhost: drop async datapath
Quoted 29 lines · David Marchand
All the API related to this feature is experimental, allowing us by the project policy to drop the feature in one go.
Signed-off-by David Marchand <[email protected]>---
Changes since v3:
- rebased after VFIO changes,Changes since v2:
- removed inflight async statistics,Changes since v1:
- dropped dependency on PCI library,Changes since RFC:
- kept mmap populate behavior forvhost_crypto,--- doc/guides/prog_guide/vhost_lib.rst | 151 +- doc/guides/rel_notes/release_26_11.rst | 20 + lib/vhost/meson.build | 3 +- lib/vhost/rte_vhost.h | 3 +- lib/vhost/rte_vhost_async.h | 293 --- lib/vhost/socket.c | 12 - lib/vhost/vhost.c | 436 ---- lib/vhost/vhost.h | 118 - lib/vhost/vhost_user.c | 111 - lib/vhost/virtio_net.c | 3275 +++++------------------- 10 files changed, 687 insertions(+), 3735 deletions(-)
Quoted 2 lines
delete mode 100644
lib/vhost/rte_vhost_async.h
Already provided my R-by on previous revision:
PATCH v4 5/6 vhost: rename packed layout helpers for batches
In reply to David MarchandNow that the async datapath has been removed, the _sync in function names gives no useful information.
Besides, vhost_reserve_avail_batch_packed does not reserve anything and its name is confusing, rename it following the other rx/tx split/packed helpers convention.
---
Changes since v2:
- cleaned up indent,
--- lib/vhost/virtio_net.c | 28 +++++++++------------------- 1 file changed, 9 insertions(+), 19 deletions(-)
lib/vhost/virtio_net.c +9 -19
@@ -1118,11 +1118,8 @@ virtio_dev_rx_split(struct virtio_net *dev, struct vhost_virtqueue *vq, } static __rte_always_inline int -virtio_dev_rx_sync_batch_check (struct virtio_net *dev, -struct vhost_virtqueue *vq, -struct rte_mbuf **pkts, -uint64_t *desc_addrs, -uint64_t *lens) +virtio_dev_rx_batch_check (struct virtio_net *dev,struct vhost_virtqueue *vq, +struct rte_mbuf **pkts,uint64_t *desc_addrs,uint64_t *lens) __rte_requires_shared_capability(&vq->iotlb_lock) {bool wrap_counter = vq->avail_wrap_counter;
@@ -1220,15 +1217,14 @@ virtio_dev_rx_batch_packed_copy(struct virtio_net *dev, } static __rte_always_inline int -virtio_dev_rx_sync_batch_packed (struct virtio_net *dev, -struct vhost_virtqueue *vq, -struct rte_mbuf **pkts) +virtio_dev_rx_batch_packed (struct virtio_net *dev,struct vhost_virtqueue *vq, +struct rte_mbuf **pkts) __rte_requires_shared_capability(&vq->iotlb_lock) {uint64_t desc_addrs[PACKED_BATCH_SIZE ];uint64_t lens[PACKED_BATCH_SIZE ]; -if (virtio_dev_rx_sync_batch_check (dev, vq, pkts, desc_addrs, lens) ==-1 ) +if (virtio_dev_rx_batch_check (dev, vq, pkts, desc_addrs, lens) ==-1 )return -1 ;if (vq->shadow_used_idx) {
@@ -1280,8 +1276,7 @@ virtio_dev_rx_packed(struct virtio_net *dev,rte_prefetch0 (&vq->desc_packed[vq->last_avail_idx]);if (count - pkt_idx >=PACKED_BATCH_SIZE ) { -if (!virtio_dev_rx_sync_batch_packed (dev, vq, - &pkts[pkt_idx])) { +if (!virtio_dev_rx_batch_packed (dev, vq, &pkts[pkt_idx])) { pkt_idx +=PACKED_BATCH_SIZE ;continue ; }
@@ -1978,12 +1973,8 @@ virtio_dev_tx_split_compliant(struct virtio_net *dev, } static __rte_always_inline int -vhost_reserve_avail_batch_packed (struct virtio_net *dev, -struct vhost_virtqueue *vq, -struct rte_mbuf **pkts, -uint16_t avail_idx, -uintptr_t *desc_addrs, -uint16_t *ids) +virtio_dev_tx_batch_packed_check (struct virtio_net *dev,struct vhost_virtqueue *vq, +struct rte_mbuf **pkts,uint16_t avail_idx,uintptr_t *desc_addrs,uint16_t *ids) __rte_requires_shared_capability(&vq->iotlb_lock) {bool wrap = vq->avail_wrap_counter;
@@ -2062,8 +2053,7 @@ virtio_dev_tx_batch_packed(struct virtio_net *dev,uint16_t ids[PACKED_BATCH_SIZE ];uint16_t i; -if (vhost_reserve_avail_batch_packed (dev, vq, pkts, avail_idx, - desc_addrs, ids)) +if (virtio_dev_tx_batch_packed_check (dev, vq, pkts, avail_idx, desc_addrs, ids))return -1 ;vhost_for_each_try_unroll (i,0 ,PACKED_BATCH_SIZE )
Signature · 1 line
2.54.0
Re: [PATCH v4 5/6] vhost: rename packed layout helpers for batches
Quoted 7 lines · David Marchand
Now that the async datapath has been removed, the
_syncin function names gives no useful information.Besides,
vhost_reserve_avail_batch_packeddoes not reserve anything and its name is confusing, rename it following the other rx/tx split/packed helpers convention.
Quoted 9 lines
Signed-off-by David Marchand <[email protected]>---
Changes since v2:
- cleaned up indent,--- lib/vhost/virtio_net.c | 28 +++++++++------------------- 1 file changed, 9 insertions(+), 19 deletions(-)
Already provided my R-by on previous revision:
PATCH v4 6/6 vhost: simplify some descriptor handling
In reply to David MarchandNow that the async datapath has been removed, move back the descriptors copy/filling code in desc_to_mbuf/mbuf_to_desc and eliminate an intermediate helper.
Tracking buf_iova is not needed in the descriptor to mbuf case.
---
Changes since v3:
- rebased after RTE_PTR_ADD changes,
Changes since v2:
- used RTE_PTR_ADD for consistency,
--- lib/vhost/virtio_net.c | 81 +++++++++++++++++------------------------- 1 file changed, 32 insertions(+), 49 deletions(-)
lib/vhost/virtio_net.c +32 -49
@@ -76,6 +76,12 @@ vhost_queue_stats_update(const struct virtio_net *dev, struct vhost_virtqueue *v } } +static __rte_always_inline bool +vhost_can_batch_copy (struct vhost_virtqueue *vq,uint32_t cpy_len) +{ +return cpy_len <=MAX_BATCH_LEN && vq->batch_copy_nb_elems < vq->size; +} + static inline void do_data_copy_enqueue(struct virtio_net *dev, struct vhost_virtqueue *vq) __rte_requires_shared_capability(&vq->iotlb_lock)
@@ -861,44 +867,6 @@ copy_vnet_hdr_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,return 0 ; } -static __rte_always_inline void -sync_fill_seg (struct virtio_net *dev,struct vhost_virtqueue *vq, -struct rte_mbuf *m,uint32_t mbuf_offset, -uint64_t buf_addr,uint64_t buf_iova,uint32_t cpy_len,bool to_desc) - __rte_requires_shared_capability(&vq->iotlb_lock) -{ -struct batch_copy_elem *batch_copy = vq->batch_copy_elems; - -if (likely (cpy_len >MAX_BATCH_LEN || vq->batch_copy_nb_elems >= vq->size)) { -if (to_desc) { -rte_memcpy ((void *)((uintptr_t )(buf_addr)), - rte_pktmbuf_mtod_offset(m, void *, mbuf_offset), - cpy_len); -vhost_log_cache_write_iova (dev, vq, buf_iova, cpy_len); -PRINT_PACKET (dev, (uintptr_t )(buf_addr), cpy_len,0 ); - }else { - rte_memcpy(rte_pktmbuf_mtod_offset(m, void *, mbuf_offset), - (void *)((uintptr_t)(buf_addr)), - cpy_len); - } - }else { -if (to_desc) { - batch_copy[vq->batch_copy_nb_elems].dst = - (void *)((uintptr_t )(buf_addr)); - batch_copy[vq->batch_copy_nb_elems].src = - rte_pktmbuf_mtod_offset(m, void *, mbuf_offset); - batch_copy[vq->batch_copy_nb_elems].log_addr = buf_iova; - }else { - batch_copy[vq->batch_copy_nb_elems].dst = - rte_pktmbuf_mtod_offset(m, void *, mbuf_offset); - batch_copy[vq->batch_copy_nb_elems].src = - (void *)((uintptr_t )(buf_addr)); - } - batch_copy[vq->batch_copy_nb_elems].len = cpy_len; - vq->batch_copy_nb_elems++; - } -} - static __rte_always_inline int mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, struct rte_mbuf *m, struct buf_vector *buf_vec,
@@ -983,9 +951,23 @@ mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, cpy_len =RTE_MIN (buf_avail, mbuf_avail); -sync_fill_seg (dev, vq, m, mbuf_offset, - buf_addr + buf_offset, - buf_iova + buf_offset, cpy_len, true); +if (likely (!vhost_can_batch_copy (vq, cpy_len))) { +rte_memcpy (RTE_PTR_ADD (RTE_PTR_UNQUAL (buf_addr), buf_offset), + rte_pktmbuf_mtod_offset(m, void *, mbuf_offset), + cpy_len); +vhost_log_cache_write_iova (dev, vq, buf_iova + buf_offset, cpy_len); +PRINT_PACKET (dev,RTE_PTR_ADD (RTE_PTR_UNQUAL (buf_addr), buf_offset), + cpy_len,0 ); + }else { + vq->batch_copy_elems[vq->batch_copy_nb_elems].dst = +RTE_PTR_ADD (RTE_PTR_UNQUAL (buf_addr), buf_offset); + vq->batch_copy_elems[vq->batch_copy_nb_elems].src = + rte_pktmbuf_mtod_offset(m, void *, mbuf_offset); + vq->batch_copy_elems[vq->batch_copy_nb_elems].log_addr = + buf_iova + buf_offset; + vq->batch_copy_elems[vq->batch_copy_nb_elems].len = cpy_len; + vq->batch_copy_nb_elems++; + } mbuf_avail -= cpy_len; mbuf_offset += cpy_len;
@@ -1682,7 +1664,7 @@ desc_to_mbuf(struct virtio_net *dev, struct vhost_virtqueue *vq, __rte_requires_shared_capability(&vq->iotlb_lock) {uint32_t buf_avail, buf_offset, buf_len; -uint64_t buf_addr, buf_iova; +uint64_t buf_addr;uint32_t mbuf_avail, mbuf_offset;uint32_t hdr_remain = dev->vhost_hlen;uint32_t cpy_len;
@@ -1713,7 +1695,6 @@ desc_to_mbuf(struct virtio_net *dev, struct vhost_virtqueue *vq, } buf_addr = buf_vec[vec_idx].buf_addr; - buf_iova = buf_vec[vec_idx].buf_iova; buf_len = buf_vec[vec_idx].buf_len; buf_offset = hdr_remain; buf_avail = buf_vec[vec_idx].buf_len - hdr_remain;
@@ -1728,14 +1709,17 @@ desc_to_mbuf(struct virtio_net *dev, struct vhost_virtqueue *vq,while (1 ) { cpy_len =RTE_MIN (buf_avail, mbuf_avail); -if (likely (hdr && cur == m)) { +if (likely ((hdr && cur == m) || !vhost_can_batch_copy (vq, cpy_len))) { rte_memcpy(rte_pktmbuf_mtod_offset(cur, void *, mbuf_offset), - (void *)((uintptr_t)(buf_addr + buf_offset)), +RTE_PTR_ADD (RTE_PTR_UNQUAL (buf_addr ),buf_offset ), cpy_len); }else { -sync_fill_seg (dev, vq, cur, mbuf_offset, - buf_addr + buf_offset, - buf_iova + buf_offset, cpy_len, false); + vq->batch_copy_elems[vq->batch_copy_nb_elems].dst = + rte_pktmbuf_mtod_offset(cur, void *, mbuf_offset); + vq->batch_copy_elems[vq->batch_copy_nb_elems].src = +RTE_PTR_ADD (RTE_PTR_UNQUAL (buf_addr), buf_offset); + vq->batch_copy_elems[vq->batch_copy_nb_elems].len = cpy_len; + vq->batch_copy_nb_elems++; } mbuf_avail -= cpy_len;
@@ -1749,7 +1733,6 @@ desc_to_mbuf(struct virtio_net *dev, struct vhost_virtqueue *vq,break ; buf_addr = buf_vec[vec_idx].buf_addr; - buf_iova = buf_vec[vec_idx].buf_iova; buf_len = buf_vec[vec_idx].buf_len; buf_offset =0 ;
Signature · 1 line
2.54.0
Re: [PATCH v4 6/6] vhost: simplify some descriptor handling
Quoted 11 lines · David Marchand
Now that the async datapath has been removed, move back the descriptors copy/filling code in desc_to_mbuf/mbuf_to_desc and eliminate an intermediate helper.
Tracking
buf_iovais not needed in the descriptor to mbuf case.Signed-off-by David Marchand <[email protected]>---
Changes since v3:
- rebased afterRTE_PTR_ADDchanges,
Quoted 7 lines
Changes since v2:
- usedRTE_PTR_ADDfor consistency,--- lib/vhost/virtio_net.c | 81 +++++++++++++++++------------------------- 1 file changed, 32 insertions(+), 49 deletions(-)
Already provided my R-by on previous revision:
Maxime Coquelin
AppliedQuoted 11 lines · David Marchand
The integration of this feature in OVS has been postponed following feedback how the current API forced the complexity on the application side. There is no contributor working on making those changes.
The feature has no other identified opensource consumer.
On the other hand, this feature has been left unattended for some years now while the "normal" datapath is actively maintained.
It is highly likely the async datapath code, which never underwent a stabilisation phase, is riddled with bugs.
Quoted 9 lines
Given the craziness around finding security bugs with AI, and since the vhost library is a good target, I propose to drop this feature and reduce the library attack surface.
There is one complication with the
vhost_cryptoand some external applications that seemed to rely on the async flag to track and populate guest pages, so a new flag replacing async is added by this series.
Applied to next-virtio/for-next-net.
Thanks!
Maxime